clawhdf5/CHANGELOG.md

# Changelog

## Unreleased

### Memory
- `clawhdf5-agent`: **`MemoryConfig::quantized_index`** stores the vector
  index's own copy of the embeddings as `i8` rather than `f32`, which at 100k
  384-dim entries takes the index from 266 to 123 MiB and the whole reopened
  store from 399 to 256 MiB (2.72x -> **1.74x** the raw vectors). Quantised
  distances are approximate and `ef` cannot compensate — recall@10 tops out at
  0.967 against f32's 0.9995 — so the query path re-scores the candidate pool
  against the exact embeddings the store already holds, which restores recall
  (0.9940 vs 0.9945 at ef=64) for about 13% of QPS. **Off by default**: it
  trades query speed for memory, and which side is worth more depends on the
  deployment. The setting is persisted, so a reopened store does not silently
  revert to four times the index memory.
- `clawhdf5-ann`: `Storage::Int8` and the `build_with` / `new_with` /
  `from_graph_bytes_with` constructors that select it. The scale is per row,
  not global — a fixed `[-1, 1]` scale spends fewer than 12 of the 255 levels
  on a unit-length 128-dim vector and is unusable (0.35 top-10 overlap against
  an exact ranking, versus 0.99 per row). `compact()` keeps the storage it was
  given; serialized indexes still carry f32 vectors, so a quantised index is
  rebuilt rather than loaded.

### Memory
- `clawhdf5-agent`: **a loaded store holds ~30% less memory** (100k 384-dim
  entries: 505 -> 357 MiB, 3.44x -> 2.43x the raw vectors). The cache kept
  every embedding twice — a `Vec<Vec<f32>>` and a flattened copy for the
  batched kernels, maintained in lock-step — so it now stores only the flat
  buffer and indexes into it. Recall and query latency are unchanged.
  **Breaking:** `MemoryCache::embeddings` is a `cache::Embeddings` rather than
  a `Vec<Vec<f32>>` (indexing still yields a `&[f32]` row); `embeddings_flat`
  is gone, replaced by `flat_embeddings()`; `rebuild_flat()` is a deprecated
  no-op. Rows are now always exactly `dim` long — shorter ones are
  zero-padded — which makes the ragged-row case that used to silently
  misalign the flattened copy unrepresentable.
- `clawhdf5-bench`: `search_harness --footprint` reports live heap use per
  stage, measured with a counting allocator (RSS cannot see a structure freed
  into the allocator's own pool).

## Unreleased

### Retrieval quality
- `clawhdf5-agent`: **re-ranking discarded the retrieval score.**
  `reranker::rerank` built its combined score from temporal decay, source
  authority and Hebbian activation only — `RerankInput` had no relevance field
  — so re-ranking a candidate pool reordered it by age and threw the
  retriever's ordering away. The OpenClaw backend re-ranked every search, so
  this was its shipping behaviour: measured over the full LongMemEval haystack
  it cost **40.6pp of Hit@1** (11.0% vs 51.6%) and two thirds of MRR (0.183 vs
  0.643). `RerankInput::relevance` and `ReRankConfig::relevance_weight` (1.0 by
  default) fix it: relevance leads and the metadata signals break near-ties,
  which restores retrieval (Hit@1 +0.4pp vs no re-ranking) and improves
  recency discrimination by 6–7pp. **Breaking:** `RerankInput` and
  `ReRankConfig` gained fields, so literal constructions need updating;
  `..Default::default()` does not.
- `clawhdf5-bench`: the LongMemEval harness feeds the dataset's real session
  dates to the store instead of a synthetic counter (decay needs true
  intervals, not just the right order), and reports `newest_gold_first` — on a
  `knowledge-update` question, did the newest gold session outrank the stale
  one it supersedes? Plain recall cannot see this, because both are labelled
  gold. New `--rerank-sweep`.

## v2.5.0 (2026-09-19)

### Upgrade Notes
- **Retrieval rankings change, for the better.** The default fusion weights
  move from `0.7/0.3` to `0.4/0.6` (`hybrid::DEFAULT_FUSION`), measured over the
  full LongMemEval haystack: turn-level Hit@1 51.6% vs 44.2%, MRR 0.643 vs
  0.586. `unified_search` and the OpenClaw backend pick this up automatically;
  callers passing weights to `hybrid_search` explicitly are unaffected.
- **Out-of-range selections are now errors.** `read_*_selection` used to return
  data for a selection that ran past a dataset edge — a hyperslab came back
  zero-padded, and a point with an out-of-range coordinate wrapped into the
  next row. Both are now `FormatError::SelectionOutOfBounds`. Code relying on
  the old (wrong) values will start seeing errors.
- **Large compressed datasets written without explicit chunk dimensions get a
  different layout.** They used to be stored as one chunk; they are now split
  to ~1 MiB chunks. The files stay standard and h5py-readable, and explicit
  `with_chunks` is unaffected.
- `rayon` is now a default dependency of `clawhdf5-agent` (the parallel index
  build). Opt out with `--no-default-features --features float16,hnsw`.
- `clawhdf5-ann` search results no longer shrink when records near the query
  have been deleted, so a search that previously returned fewer than `k`
  results now returns `k`.

### Retrieval quality
- `clawhdf5-agent`: optional keyword stemming — `bm25::TokenFilter::Stemmed`
  and `HDF5Memory::set_token_filter`, so "training" and "trains" match. **Off
  by default**, on measurement rather than principle: over the full LongMemEval
  haystack it buys depth and costs the top rank (BM25 alone: Hit@5 +2.8pp,
  Hit@10 +2.4pp, Hit@1 −1.8pp, MRR unchanged), and on the shipping hybrid
  configuration the trade is narrower still. See `BENCHMARKS.md`.
- `clawhdf5-agent`: **`QueryExpander::expand` panicked on ordinary non-ASCII
  input** — `"İ AI"` was enough. It searched a lowercased copy of the query and
  then sliced the *original* with those offsets, which only works while
  lowercasing preserves byte length (Turkish `İ` is 2 bytes and lowercases to
  3). Depending on where the offsets drifted it either corrupted the output
  ("İstanbul AI trip" lost a character) or panicked. Matching now walks the
  original string.
- `clawhdf5-agent`: query expansion no longer rewrites text inside words.
  `replace_word_case_insensitive` did a plain substring replace despite its
  name, so "training" became "trArtificial Intelligencening" and "programming"
  became "Pull Requestogramming" — every acronym expansion of ordinary prose
  was corrupt. Matches now require word boundaries; genuine acronyms
  (`API`, `database`) still expand.
- `clawhdf5-agent`: **the default fusion weights are now the measured ones.**
  A sweep of every 0.1 step over the full LongMemEval haystack (500 questions,
  real MiniLM embeddings) shows the long-standing `0.7/0.3` default is
  *strictly dominated* by `0.4/0.6` — turn-level Hit@1 51.6% vs 44.2%, Hit@5
  81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.643 vs 0.586, and better at
  session level too. The finding was recorded in `BENCHMARKS.md` but had never
  been applied: `unified_search` and the OpenClaw backend both hardcoded
  `0.7/0.3`. They now use `hybrid::DEFAULT_FUSION`. **Callers passing weights
  to `hybrid_search` explicitly are unaffected** — pass `0.4`/`0.6` (or use
  `hybrid_search_with`) to get the tuned behaviour.
- `clawhdf5-agent`: fusion is now selectable. New `hybrid::Fusion`
  (`Weighted { vector, keyword }` or `Rrf { k }`), `hybrid::fuse`,
  `hybrid::hybrid_search_fused` and `HDF5Memory::hybrid_search_with`.
  Reciprocal rank fusion existed but was unreachable from the store, so it had
  never been measured against the weighted sum; the LongMemEval bench now has
  an `RRF` mode.

### HDF5 Read Path
- **Selection reads cost what the selection costs.** `read_*_selection` decoded
  the *entire* dataset and then picked elements out, so a 64 x 64 window of a
  64 MB compressed dataset took 105 ms - about as long as reading all of it.
  Now only the rows (contiguous) or chunks that overlap the selection's
  bounding box are read and decompressed: that window takes 0.39 ms, one row
  2.7 ms, one column 5.2 ms. Results are identical to the full-read path
  (equivalence-tested over random hyperslabs and point lists, ranks 1-3,
  contiguous / chunked / deflate). New `read_harness` bench binary.
- **Faster full reads** (same-moment A/B, 64 MB `f64`): chunked + deflate
  110 -> 69 ms, chunked 72 -> 60 ms, contiguous 56 -> 30 ms. The facade's
  cached read path now decompresses cache misses in parallel batches (it was
  sequential; only the uncached reader was parallel) and caches only datasets
  that fit the chunk cache; unfiltered chunks are copied straight from the file
  bytes; a contiguous dataset is converted straight from the file bytes; and
  the native-endian conversions no longer zero a buffer before overwriting it.
- **Datasets indexed by a version-2 B-tree now read** (layout v4, chunk index
  type 5 — what `libver='latest'` uses for two or more unlimited dimensions;
  previously "unsupported chunked layout"). The four copies of the chunk-index
  dispatch are now one shared function, so every read path gets it.
- **`H5T_STD_REF` references** (HDF5 1.12+, datatype message version 4) parse:
  `ReferenceType` gains `Object2`, `DatasetRegion2` and `Attribute`, and
  `read_object_references` decodes the new object references. Previously any
  dataset of this type failed with `InvalidReferenceType(2)`. Tested against a
  file written by HDF5 2.0 itself (fixture + generator script committed).
- **Automatic chunk sizes.** Asking for compression (or any filter) without
  `with_chunks` used to store the whole dataset as one chunk, so any read had
  to decompress everything and nothing could be decoded in parallel. Datasets up
  to 1 MiB stay a single chunk, as before; larger ones are split by halving the
  dimensions in turn until a chunk is at most 1 MiB (the approach h5py takes).
  **Behaviour change:** large compressed datasets written without explicit
  chunk dimensions get a different (standard, h5py-readable) layout. Explicit
  `with_chunks` is unaffected.
- **Out-of-range selections are errors.** They used to return data: a hyperslab
  past an edge came back padded with zeros, and a point whose column was out of
  range wrapped into the next row and returned that element. Now
  `FormatError::SelectionOutOfBounds` (also for a rank mismatch or overlapping
  blocks).

### Search
- `clawhdf5-ann`: **faster index builds.** Back-link pruning is 90% of a
  build's distance evaluations; the bulk build now inserts in batches and
  prunes each overflowing neighbour list once per batch (10K: 1676 -> 1074 ms).
  With the `parallel` feature, planning and pruning run on a thread pool (10K:
  388 ms, 100K: ~21 s -> 5.9 s on 16 cores). The graph is deterministic and
  identical with or without the feature. `clawhdf5-agent`'s `parallel` feature
  enables it for the agent's index and is now **on by default** (adds `rayon`
  to the default dependency set; build with `--no-default-features --features
  float16,hnsw` to opt out).
- `clawhdf5-ann`: `HnswIndex::search` returned fewer than `k` results — often
  none — when the records nearest the query had been deleted: it collected `ef`
  candidates, *then* dropped the deleted ones, *then* took `k`. Deleted nodes
  are now traversed as waypoints but never occupy a result slot, so a search
  returns the `k` nearest live records. Matters for any store that deletes or
  supersedes memories without compacting straight away.

## v2.4.0 (2026-09-19)

### Upgrade Notes
- **Search results improve on upgrade.** The HNSW index now reaches true
  neighbours it previously could not (recall@10 0.31 -> 0.98 at 100K records on
  clustered data), so `hybrid_search` rankings change for the better. The agent
  rebuilds its index from the store automatically; a standalone `HnswIndex`
  persisted with `to_hdf5_bytes` keeps its old graph until rebuilt.
- **`hybrid_search` no longer writes the store.** Hebbian activation boosts are
  persisted by the next checkpoint (any flushing write, `flush_wal`, or when
  the `HDF5Memory` is dropped) instead of inside every query; a crash before
  then forgets only the boosts since the last checkpoint. Activation weights
  are now capped at 16.
- A new sidecar file, `<store>.h5.ann`, holds the vector index graph. It is
  derived data: safe to delete (the index is rebuilt), copied by `snapshot()`,
  and worth including when copying a store by hand to avoid a rebuild.
- `BM25Index` no longer caches IDF and gained `add_document`,
  `remove_document`, `pad_to`, `scores`, `len` and `is_empty`; results are now
  deterministic (ties break by record id).

### Search
- `clawhdf5-ann`: **HNSW recall fix.** Neighbours were chosen as the plain
  closest-M, which on clustered data (what embeddings look like) turns each
  cluster into an island: recall@10 was 0.87 / 0.67 / 0.31 at 1K / 10K / 100K
  vectors and did not improve with `ef`. The index now uses the HNSW paper's
  diversity heuristic (Algorithm 4 with kept pruned connections) when linking a
  new node and when pruning back-links: recall@10 at `ef = 64` is 1.00 / 1.00 /
  0.98 and responds to `ef`. Builds are slower (~3.5x at 10K). Existing
  persisted indexes keep their old graph until rebuilt; the agent rebuilds its
  index from the cache, so stores pick this up automatically.
- `clawhdf5-agent`: **`hybrid_search` is 23-39x faster in steady state** (p50
  5.5 -> 0.24 ms at 1K records, 49 -> 2.1 ms at 10K, 884 -> 23 ms at 100K).
  Every query used to rebuild the BM25 index from scratch and rewrite the whole
  `.h5` file. The keyword index now lives for the life of the store and is
  updated incrementally (add / remove / in-place update, exactly equivalent to
  a fresh build - property-tested), and a query no longer writes the store.
  **Behaviour change:** Hebbian activation boosts are persisted by the next
  checkpoint (any flushing write, `flush_wal`, or drop) rather than
  immediately; a crash in between forgets only the boosts since the last
  checkpoint. Activation weights are now capped (16.0) - they grew without
  bound.
- `clawhdf5-agent`: **the vector index is persisted**, so `open()` no longer
  rebuilds it on the first search (first query after open: 2627 -> 15 ms at 10K
  records, 36 s -> 159 ms at 100K). The HNSW graph — not the vectors, which the
  store already holds — is written to `<store>.h5.ann` at each checkpoint and
  tied to it by a generation id in `/meta`; a missing, stale, damaged or
  structurally invalid sidecar is ignored and the index rebuilt. Records
  replayed from the WAL join the loaded index incrementally; a replayed update
  or delete invalidates it. `snapshot()` copies it. Batch saves no longer force
  a full index rebuild.
- `clawhdf5-ann`: faster HNSW build and search with identical recall. The
  cosine metric stores unit vectors and compares them with a plain dot product
  (it re-derived both norms on every distance evaluation), and the per-call
  `HashSet` of visited nodes is a reusable epoch-stamped array. Build 2.75 ->
  1.89 s at 10K and ~38 -> 21 s at 100K; QPS at `ef = 64` 22.7K -> 39K at 10K.
  Distances returned by `search` are unchanged (1 - cosine). Indexes loaded
  from older HDF5 files are normalised on load.
- `clawhdf5-accel`: the SIMD backend is detected once per process instead of
  on every kernel call.
- `clawhdf5-ann`: `HnswIndex::graph_to_bytes` / `from_graph_bytes` — graph-only
  serialization (checksummed, every neighbour id and level validated on load).
- `clawhdf5-agent`: a further 4-5x on `hybrid_search` with **identical
  rankings** (p50 now 0.07 / 0.49 / 4.65 ms at 1K / 10K / 100K — 79x / 100x /
  190x faster than v2.3.0). Fusion needs every keyword score but not their
  ranking: new `BM25Index::scores` returns them unsorted from a dense
  accumulator (it hashed every posting, then sorted every match), and
  `merge_vector_keyword` selects its top k instead of sorting every candidate.
  Capping the keyword candidate pool was measured and rejected: it changes the
  top-10 for most queries (`search_harness --fusion-study`).
- `clawhdf5-agent`: BM25 results are deterministic (ties break by record id),
  top-k uses a bounded heap, and the "WAND early termination" that computed a
  bound and then ignored it is gone. IDF is computed per query.
- `clawhdf5-bench`: new `search_harness` binary — HNSW recall@10 / QPS / latency
  per `ef` against an exact scan, and end-to-end `hybrid_search` timings, on
  deterministic clustered (or `--uniform`) data. Baseline in `BENCHMARKS.md`.

## v2.3.0 (2026-09-19)

### Upgrade Notes
- **A memory store now has a single writer.** `HDF5Memory::create`/`open` take
  an exclusive lock (`<store>.h5.lock`); a second open of the same store — in
  the same or another process — returns `MemoryError::Locked`. Code that opened
  a second handle just to read should use `HDF5Memory::open_read_only`.
- **Unsigned array attributes arrive as `AttrValue::U64Array`**, not
  `I64Array`, and `attrs()` may now return `AttrValue::Raw`. Exhaustive matches
  on `AttrValue` need the two new arms.
- **WAL header version 3 → 4.** v3 files are read and upgraded in place, but a
  store written by 2.3.0 with a pending WAL cannot be opened by 2.2.0 or
  earlier (it is refused, not corrupted). Checkpoint first
  (`flush_wal`) if you need to downgrade.
- `MemoryConfig::compression` now uses deflate unless the agent's new `zstd`
  feature is enabled; it previously failed outright in a default build.
- `MemoryError` gained `Locked`; `FormatError` gained `UnresolvedSharedMessage`,
  `ExternalDataFilesUnsupported` and `ExternalLinkUnsupported`; `MessageType`
  gained `ExternalDataFiles`.

### Bug Fixes
- `clawhdf5-format`: compound datatypes written with **default libver bounds**
  (datatype message version 1 — what plain `h5py.File(path, 'w')` produces)
  were mis-parsed. The v1 member layout carries 28 bytes of legacy array
  fields after the byte offset (the parser skipped 24), and v2 pads member
  names to 8 bytes and has no array fields at all (the parser did neither), so
  every member after the first byte offset was read from the wrong position —
  typically surfacing as `Overflow("compound member ...")` on read. Found by
  adding a default-libver axis to the h5py interop tests; byte-level regression
  tests for v1 and v2 added.
- `clawhdf5-gpu`: `gpu_tests` could hang forever under the default parallel
  test runner — every test created its own wgpu instance and device at once.
  Tests now serialise GPU access, and GPU→CPU readback waits are bounded
  (30 s) so a wedged driver returns `GpuError::BufferMap` instead of blocking.
- `clawhdf5-agent`: `benches/bench.rs` and `benches/memory_bench.rs` no longer
  compiled against the current `strategy`/`consolidation` APIs.

### HDF5 Compatibility
- `clawhdf5-format`/`clawhdf5`: datasets and attributes that use a **committed
  (named) datatype** now read correctly. They store a shared-message reference;
  the facade parsed the reference bytes as the datatype (`Time { size: 0 }`,
  unreadable data) and silently dropped such attributes. The shared-reference
  parser itself was wrong for real files: version 2 has no reserved bytes, and
  the version 3 types were inverted (1 = SOHM heap, 2 = committed).
- **Fill values are applied on read.** There was no Fill Value message parser:
  the holes of a sparse chunked dataset read as zeros even when the fill value
  was not zero (silently wrong data), and a dataset that was created but never
  written failed with `NoDataAllocated` where h5py returns a filled array.
  Messages v1–v3 and the old 0x0004 form are parsed; the fill value is written
  into exactly the chunk-grid cells missing from the chunk index.
- **Soft links are followed** during path resolution, in old- and new-style
  groups (absolute/relative targets, links to groups, links through links),
  with a depth limit so a link cycle is an error rather than a hang. A dangling
  link reports the target it could not find.
- Things the reader does not follow are now explicit errors instead of wrong
  answers: an external link is `ExternalLinkUnsupported { filename,
  object_path }` (was `PathNotFound`), and a dataset whose raw data lives in
  external files (message 0x0007, now a known `MessageType`) is
  `ExternalDataFilesUnsupported` (it would otherwise read as fill values).
- **`attrs()` no longer drops attributes.** Any attribute whose datatype had
  no `AttrValue` variant was omitted with no error — including every Python
  `bool` (h5py stores `attrs["flag"] = True` as an enum), complex numbers,
  compound values and object references. Now:
  - numpy/h5py-style booleans (an enum of exactly `FALSE`=0 / `TRUE`=1) decode
    as `I64` / `I64Array` of 0/1;
  - new `AttrValue::U64Array` keeps unsigned arrays unsigned (they were cast to
    `I64Array`, so values above `i64::MAX` came back negative). **Behaviour
    change:** code matching `I64Array` for an unsigned attribute must also
    match `U64Array` (the netCDF-4 CF helpers and Python bindings do);
  - new `AttrValue::Raw { datatype, shape, data }` carries everything else
    verbatim, decodable with `clawhdf5_format::data_read` against `datatype`.
  Both new variants are writable, so an attribute can be copied between files
  unchanged. Python receives `Raw` as `{"dtype", "shape", "data"}`.
- All of the above are covered by h5py interop tests under both default and
  `libver='latest'` bounds, compared against h5py's own readback.

### Security
- `clawhdf5`: virtual-dataset source file names are untrusted input but were
  joined straight onto the opened file's directory, so a crafted file could
  make the reader open any path the process can reach (absolute path, or `..`
  components). Only plain relative paths inside that directory are accepted.

### Durability & Integrity
- `clawhdf5-agent`: a crash between writing a checkpoint and truncating the WAL
  no longer **duplicates every pending entry** on the next open. Each
  checkpoint records a `WalMark` (byte length + chained CRC of the WAL prefix it
  folded in) in `/meta`; `open()` skips exactly that prefix when it is still
  present. No WAL format change for this; older files behave as before.
- `clawhdf5-agent`: checkpoints and snapshots are durable as a unit — the temp
  file is synced before the rename and the directory after it. Individual WAL
  appends remain unsynced by design (documented in `CLAUDE.md`).
- `clawhdf5-agent`: `save_or_update` hits are logged as a new `Update` WAL
  record, so replay updates in place instead of appending a duplicate. WAL
  header version 3 → 4 (so older builds refuse the file rather than truncating
  a record they can't parse); v3 files are read and upgraded in place.
- `clawhdf5-agent`: loading validates every per-record dataset length (a
  truncated store is now `MemoryError::Schema`, not a later panic), fixes the
  `n.len() == n.len()` tautology that trusted a norms dataset of any length,
  and rejects `embedding_dim == 0` with records present.
- `clawhdf5-agent`: eight behavioural `MemoryConfig` fields are now persisted in
  `/meta`. Previously they reset to defaults on every open — a compressed store
  was rewritten uncompressed, `wal_enabled = false` flipped back to `true`.
- `clawhdf5-agent`: `compression = true` never worked in a default build (it
  requested Zstd without enabling the feature, so every checkpoint failed with
  `unsupported filter: 32015`). Default builds now use deflate; Zstd is the new
  opt-in `zstd` feature.
- `clawhdf5-agent`: **single-writer lock** (`<store>.h5.lock`,
  `MemoryError::Locked`) — two handles on one store used to silently destroy
  each other's data. New `HDF5Memory::open_read_only` gives a lock-free,
  never-writing view; the CLI's read-only subcommands use it.
- `clawhdf5-agent`: an unreadable WAL (torn header / bad magic) is quarantined
  (`HDF5Memory::quarantined_wal()`) instead of blocking `open()` of a healthy
  store. A WAL from an unknown newer version still fails and is left intact.
- `clawhdf5-agent`: provenance records are renumbered on compaction (they
  weren't, so every later `save_or_update` raised a false High integrity
  alert); pending anomaly alerts and tracked sessions are bounded;
  `snapshot()` includes entries still in the WAL.
- `clawhdf5-agent`: hybrid ranking is deterministic (index tie-breaks instead
  of `HashMap` order); a set of identical positive scores — including a single
  candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer
  reinforces zero-score filler results.
- `clawhdf5-format`: chunked/VDS/hyperslab reads size their buffers with
  overflow-checked arithmetic and fallible allocation, so crafted dimensions
  are `FormatError::Overflow` instead of a wrapped size or a process abort;
  `parallel_read` bounds checks use `checked_add`.
- `clawhdf5`: a malformed filter-pipeline message is an error instead of being
  treated as "no filters" (which returned compressed bytes as data);
  `FileBuilder::write` is atomic and synced instead of truncating the
  destination first.

### CI / Testing
- CI now lints every target (`cargo clippy --all-targets`) plus
  `clawhdf5-format`'s optional features, compiles all benches, and tests the
  format feature matrix. Previously test/bench code and feature-gated modules
  were never linted; the accumulated clippy backlog is fixed.
- CI installs python3 + h5py/numpy/netCDF4/xarray and sets
  `CLAWHDF5_REQUIRE_INTEROP=1`, which turns a missing interop dependency into a
  test **failure**. Until now every h5py/netCDF4 interop test silently skipped
  in CI, which is how the HDF5 2.0 compound bug fixed in v2.2.0 reached a user.
  The `#[ignore]`d `writer_h5py_tests` suite is run explicitly.
- h5py-generated-file tests now cover default libver bounds as well as
  `libver='latest'` (HDF5 2.0 raised the default low bound to 1.8).
- `clawhdf5-agent`: WAL property tests (round trip; after any corruption the
  entries read back are an exact prefix of what was written — 1500 seeded
  cases), a crash-recovery matrix (an on-disk image after every operation, the
  checkpoint window, and the WAL torn at every byte length, each reopened and
  checked against a model), and a WAL fuzz target.
- Optional fuzz smoke run (`CLAWHDF5_FUZZ_SECONDS=N scripts/ci-test.sh`); new
  datatype corpus seeds for v1 compound and native complex messages.

## v2.2.0 (2026-09-18)

### Security
- `clawhdf5-format`: bounded decompression output (`MAX_DECOMPRESS_SIZE`) for
  deflate/lz4/zstd/pcodec so a crafted compressed chunk can't drive an
  unbounded allocation (memory-exhaustion DoS).
- `clawhdf5-format`: `chunked_read.rs`/`data_read.rs`/`local_heap.rs` bounds
  audit — added `ensure_len` overflow guards at every plain-arithmetic
  offset+size check, a recursion-depth guard against a crafted
  self-referencing/cyclic B-tree chunk index, a fix for an unguarded
  compound-datatype `byte_offset` overrun in `read_compound_fields`, and an
  `ndims - 1` underflow guard for degenerate zero-dimension chunked layouts.
  Added a new `fuzz_dataset_read` cargo-fuzz target (walks every dataset in a
  parsed file and exercises the contiguous/chunked/compact raw-data read
  paths) which found and fixed 3 real crash bugs — an integer-multiply
  overflow in `copy_chunk_to_output`'s N-D assembly path, the `ndims - 1`
  underflow above, and an overflow in `local_heap.rs` — within the first few
  fuzzing runs.
- `clawhdf5-format`: `btree_v1.rs` overflow-safe bounds checks via a local
  `ensure_len` helper, closing a `usize`-overflow panic reachable from a
  crafted near-`usize::MAX` B-tree offset.
- `clawhdf5-agent`: WAL length-prefix caps (`MAX_WAL_FIELD_LEN`, 64 MiB) reject
  a corrupted/truncated length claim before allocating. Followed by a full
  per-entry CRC32 trailer (`WAL_VERSION` bumped to 2) — a bit-flip inside an
  entry now stops replay cleanly instead of silently accepting corrupted
  data. Old-format WAL files are still read correctly and migrated to the new
  format on next open.
- `clawhdf5-android`: validate `embedding_len`/`query_embedding_len` against
  the handle's configured `embedding_dim` (and reject null pointers) before
  constructing a slice from a raw pointer in `edgehdf5_save` /
  `edgehdf5_hybrid_search`.
- `clawhdf5-py`: bump pyo3/numpy `0.28` → `0.29`, clearing two RUSTSEC
  advisories (OOB read in `PyList`/`PyTuple` iterator; missing `Sync` bound on
  `PyCFunction::new_closure`).
- Clarified that the integrity hashes in `clawhdf5-agent::provenance`
  (FNV-1a) and `clawhdf5-format::provenance` (SHA-256) are unkeyed and detect
  only accidental corruption, not tampering — doc-only change, no behavior
  change.

### Performance
- `clawhdf5-format`: chunk cache lookup is now O(1) (`slot_index: HashMap`)
  instead of a linear scan, and cache hits return a shared `Arc` instead of
  cloning the decompressed buffer — the hottest path in chunked reads.
- `clawhdf5-ann`: optional `parallel` feature (rayon) parallelizes HNSW's
  `prune_connections` neighbor-distance computation. The outer build/insert
  loop is deliberately left sequential — it has genuine cross-iteration data
  dependencies and needs its own correctness-focused design pass.
- `clawhdf5-format/chunked_read.rs`: removed 12 unnecessary
  `chunk_dimensions[..rank].to_vec()` allocations where callees already
  accept `&[u32]`.

### Architecture
- Added `.gitea/workflows/ci.yml`, actually wiring the long-existing
  `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) into CI on every
  push/PR to `main`. Fixed stale package names in `ci-test.sh`/
  `check-nostd.sh` that had been silently no-op'ing the `clawhdf5-py`
  exclusion and the no_std check.
- Fixed a genuine no_std build break in `clawhdf5-format` (uncovered once the
  no_std CI check actually started running): `core::sync::atomic::AtomicU64`
  doesn't exist on `thumbv7em-none-eabihf` (switched to `portable-atomic`),
  missing `alloc` imports for `Box`/`Vec`/`format!` on a few no_std paths, and
  `f64::powi` (std/libm-only) replaced with a local exponentiation-by-squaring
  helper in the scale-offset filter.
- Added `[workspace.dependencies]` for `tempfile`/`criterion`/`half`/`serde`,
  fixing a real version skew on `half` (`2` vs `2.7` across crates).
- Fixed version skew: `clawhdf5-py` (`pyproject.toml`) and
  `packages/clawhdf5-node` (`package.json`) were both behind the actual crate
  version (2.1.0).
- Documented that the `mpi-io` feature's read/write paths are root-read
  +broadcast / gather-to-rank-0, not true collective I/O.

### Documentation
- BENCHMARKS.md: re-ran the previously-undated "LongMemEval Results", "SIMD &
  Parallelism", and "Vector Search Latency"/"Comparison to MemX" sections on
  a second machine (tank, Ryzen 7 7800X3D) with explicit dates and reproduce
  commands. Found and corrected a methodology issue in the SIMD/Parallelism
  benchmark selection (several originally-compared benchmarks didn't actually
  isolate the scalar/SIMD/parallel axis).
- README.md / ROADMAP.md / CLAUDE.md: corrected several stale facts —
  the `clawhdf5-types` crate (removed earlier) was still listed in the
  README crate map; the LongMemEval numbers in the README badge and table
  didn't match the actual (much better) benchmark results in BENCHMARKS.md;
  total line-of-code and test-count figures were stale; `clawhdf5-gpu`'s
  CubeCL→wgpu correction; documented the new `clawhdf5-ann` `parallel`
  feature flag, which had no entry in the Feature Flags table.

### New Features
- `clawhdf5-migrate`: substantial engine improvements:
  - **Real content validation** — the post-migration check now reads the written
    HDF5 back and compares actual content (chunk text, embeddings, and every
    session/entity/relation field) against the source, not just row counts. A
    representative sample of chunk rows is verified by default; `--validate-full`
    checks every row. A corrupt migration that preserves counts no longer passes.
  - **Configurable schema** — table names are no longer hardcoded; queries are
    built from a `SchemaConfig` (table + ordered column names, defaulting to the
    ZeroClaw layout) with `--chunks-table` / `--sessions-table` /
    `--entities-table` / `--relations-table` overrides.
  - **Streaming count pass** — `--dry-run` now does a `COUNT(*)`-only pass per
    table instead of loading every row into memory.
  - **Incremental migration** — `--incremental` reads the existing output, reads
    only source chunks newer than the last migrated id, and appends them
    (refreshing the metadata groups), instead of re-migrating everything.
- `clawhdf5-format`: read **IEEE-754 half-precision (f16)** floats. `read_as_f32`
  / `read_as_f64` previously only handled 4- and 8-byte floats; 2-byte floats
  (e.g. float16-stored embeddings) now decode via a no_std-safe bit conversion.
- `clawhdf5-format`: **write multi-block fractal heaps** (root indirect block).
  Dense attribute and dense link storage previously capped at a single direct
  block (~64 KiB of heap data — a few thousand attributes/links). When the
  objects exceed one direct block, the heap now lays out a root indirect block
  (FHIB) over multiple direct blocks sized by the doubling table, distributing
  objects across blocks with correct per-block heap offsets. Validated
  end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
  through our reader and are read correctly by h5py. (Objects still may not
  span a block — no huge-object path.)
- `clawhdf5-format`: **write dense group link storage** (fractal heap + v2
  B-tree). A group with more than 8 links (libhdf5's compact `max_compact`
  default) is now written densely — its links live in a fractal heap indexed by
  a v2 B-tree of type 5 (link-name index) referenced from the group's LinkInfo
  message — instead of as inline Link messages. This matches libhdf5's
  compact→dense switchover and keeps large groups out of the object header.
  Reverse-engineered against libhdf5: link heaps use `heap_id_length` 7 /
  `max_heap_size` 32 (vs 8 / 40 for attributes). The shared single-direct-block
  fractal-heap builder is now parameterized and used by both dense attributes
  and dense links. Validated end-to-end: our reader round-trips, and h5py reads
  the dense groups we write. (Single direct block — up to ~a couple thousand
  links per group; beyond that needs indirect blocks, still unsupported.)

### Robustness
- `clawhdf5-format`: harden the readers added this cycle against malformed /
  hostile input — they parse untrusted bytes and must return errors, never
  panic, OOM, or recurse without bound. Fixed concrete vectors found by audit
  and locked in with adversarial tests:
  - **Paged Fixed Array**: `1 << max_nelmts_bits` shift overflow (a `u8` ≥ 64);
    element/page offset multiplications now checked; element count bounded by
    file size.
  - **H5S selection decoder**: `ALL`/`NONE` no longer claim 16 bytes they don't
    have; hyperslab `rank` capped at 32 (`H5S_MAX_RANK`) to stop a giant
    allocation; `iter_linear` coordinate/stride/product arithmetic is checked.
  - **VDS mapping parser**: no pre-allocation from the untrusted `nused`; all
    selection slicing is bounds-checked.
  - **scale-offset / N-Bit filters**: `1 << minbits` overflow at `minbits == 64`;
    N-Bit `bit_offset + precision` overflow; N-Bit type-tree recursion depth
    capped (no stack overflow from a crafted nested tree); element counts
    bounded by the chunk's expected decompressed size so a bogus count can't
    drive a huge allocation.
  - **Virtual Dataset assembly**: a virtual dataset whose source is itself
    virtual (a cycle) now errors instead of recursing into a stack overflow.

### New Features
- `clawhdf5-agent`: **compress fixed-length string datasets** (memory text
  chunks, session summaries, ids, tags, entity/relation names, …). These were
  always stored uncompressed with a "chunked compound not yet supported" note
  that was simply stale — chunked writes work for fixed-size string/compound
  datatypes like any other. `write_string_dataset` now chunks + deflates a
  string dataset once its payload reaches 4 KiB, so large, highly-redundant
  NullPad content shrinks substantially while tiny metadata stays contiguous
  (no chunk-overhead bloat).
- `clawhdf5-format`: decode the **scale-offset filter** (id 6) — both the
  integer variant (`H5Z_SO_INT`) and the floating-point **D-scale** variant
  (`H5Z_SO_FLOAT_DSCALE`). Handles signed/unsigned int sizes, f32/f64, negative
  minima, decimal scale factors and fill values; reverse-engineered against
  HDF5 2.0 and validated end-to-end. The float E-scale variant remains
  unsupported.
- `clawhdf5-format`: decode the **N-Bit filter** (id 5) — atomic, **compound**
  and **array** layouts (the full recursive type tree, nestable to any depth),
  previously unsupported. Signed and unsigned reduced-precision integers and
  float members all read end-to-end, validated against HDF5 2.0.

### New Features
- `clawhdf5` / `clawhdf5-format`: read **external-file Virtual Datasets (VDS)**.
  The format layer gains `read_raw_data_full_with_resolver` and a
  `VdsSourceResolver` callback (`Fn(&str) -> Option<Vec<u8>>`) that maps a
  stored source file name to its bytes, so the pure-byte reader can pull in
  external sources without a filesystem of its own. The `clawhdf5` `File` API
  wires a default resolver that reads sibling source files relative to the
  opened file's directory, so `File::open(...).dataset(...).read_*()` now
  transparently assembles cross-file VDS. A source file the resolver cannot
  supply leaves its region at the fill value (matching HDF5); an external
  source with no resolver at all is a clean error. In-memory files
  (`File::from_bytes`) have no directory, so only same-file VDS resolves there.
- `clawhdf5-format`: assemble **same-file Virtual Datasets (VDS)** of any rank.
  Previously a virtual layout returned `UnsupportedVersion`. The reader now
  decodes the global-heap mapping block (reverse-engineered against HDF5 2.0:
  `version · nused · [source-file · source-dataset · source-selection ·
  virtual-selection]* · checksum`, including the block-version-1 same-file
  marker), decodes the `H5S` source/virtual dataspace **selections** (ALL,
  NONE, and version-3 regular hyperslabs), reads each same-file source dataset,
  and scatters its selected elements into the virtual buffer in row-major order
  (so multi-dimensional block mappings land correctly); unmapped regions are
  left at the zero fill value. External-file sources return a clean unsupported
  error. The previous `parse_vds_mappings` used a guessed layout that did not
  match real files and is replaced.

### Tests
- `clawhdf5-format`: regression test for **scale-offset float E-scale**
  datasets. The HDF5 library does not implement E-scale encoding — when asked
  for it (`cd_values[0] = 1`) it stores the chunk raw and sets the chunk filter
  mask to skip the filter — so these files read back verbatim purely by
  honoring the per-chunk filter mask. The test locks in that behavior against a
  fixture produced via the HDF5 low-level API; no E-scale decoder is needed.

### Bug Fixes
- `clawhdf5-format`: **read multi-direct-block fractal heaps**. The reader split
  direct vs indirect block rows using the FRHP "Starting # of Rows in Root
  Indirect Block" field (a constant, typically 1), so any heap whose data spans
  more than one direct block — common in libhdf5 files with a large group or
  many dense attributes — was misread as having indirect blocks and failed with
  `InvalidFractalHeapSignature`. The split is now derived from the heap geometry
  (`max_direct_rows = log2(max_direct / start) + 2`). Validated against an
  h5py-written 400-dense-attribute group (root indirect block, 4 rows, 13 direct
  blocks).
- `clawhdf5-format`: scope the per-file **chunk cache by dataset**. The shared
  `ChunkCache` built its chunk index once and reused it for every chunked
  dataset in the file, keyed only by chunk coordinate with no dataset
  discrimination. With a single chunked dataset per file this was latent; once a
  file holds two chunked datasets of different rank (e.g. a 1-D compressed
  string array and the 2-D embeddings matrix), the first dataset's index was
  reused for the second, panicking with an out-of-bounds chunk coordinate. The
  cache now rebinds (dropping its index, chunk-index map, layout, and
  decompressed slots) whenever the dataset being read changes, while still
  caching repeated/sequential access to the same dataset.
- `clawhdf5-format`: read **paged Fixed Array** chunk indexes. A filtered,
  fixed-dimension dataset with more than one data-block page (>1024 chunks by
  default) previously failed with "paged Fixed Array data blocks not yet
  supported". The reader now walks the page-init bitmap (MSB-first), skips
  uninitialized pages, and resolves each page's fixed full-size slot (including
  the short final page). Reverse-engineered and validated end-to-end against an
  HDF5 2.0 file.
- `clawhdf5-format`: read **array-typed datatypes** (e.g. an array-typed
  compound member) via `read_as_i32/i64/u64/f32/f64` — previously a
  `TypeMismatch`. The array is read as a flat sequence of its base elements
  (recursing for nested arrays), applying base-type precision rules.
- `clawhdf5-format`: **sign-extend reduced-precision fixed-point integers** on
  read. A signed integer whose datatype precision is smaller than its storage
  size is stored zero-filled, so e.g. a 16-bit-precision `-1` previously read as
  `65535`. The integer read paths now extract the precision field and
  sign-extend (full-width types are unchanged). Completes signed N-Bit reads and
  also fixes un-filtered reduced-precision integer datasets.
- `clawhdf5-format`: read datasets written by modern HDF5 (1.14+/2.0, i.e.
  `libver=latest`). Compound (class 6) and array (class 10) datatype **version 5**
  messages and data layout **version 5** messages were rejected as invalid; they
  reuse the v3/v4 binary structure, so they are now accepted. This unblocks
  reading compound types and — critically — every chunked/compressed dataset
  written by HDF5 2.0. Found by running the h5py interop tests against
  h5py 3.16 / HDF5 2.0.
  Independently reported (with a patch) against the v2.1.0 tag by
  M. Scot Breitenfeld (The HDF Group) — v2.1.0 predates this fix.
- `clawhdf5-format`: parse HDF5 2.0 native complex datatypes (class 11,
  datatype version 5, e.g. `H5T_COMPLEX_IEEE_F64LE`). The properties are a
  single base floating-point datatype, not a compound-style member list; the
  old parser read the base type's bytes as member names, producing a garbage
  datatype, and failed with `UnexpectedEof` when a complex type was nested in
  a compound. It is now surfaced as the equivalent `{r, i}` compound (the
  shape h5py writes for numpy complex dtypes), with a size check against the
  base type. Validated end-to-end against an HDF5 2.0-written file.

### Performance
- `clawhdf5-format`: chunked writes now compress all chunks up front via
  `compress_all_chunks`, running across rayon threads under the `parallel`
  feature when there are more than 4 filtered chunks. On-disk layout is
  unchanged. Speeds up compressed embedding writes in `clawhdf5-agent` (which
  enables `parallel`).

### Documentation
- Fix stale package names across all 13 per-crate READMEs (`rustyhdf5-*` /
  `edgehdf5-*` → `clawhdf5-*`, usage versions → 2.1.0).
- Correct README workspace/test/crate stats and the CLAUDE.md CLI subcommand
  list; document the `hnsw` and format compression/checksum feature flags and
  the `entity_extract` / `async_memory` modules.

## v2.1.0 (2026-06-03)

### New Features
- `clawhdf5-agent`: HNSW now backs the vector stage of `hybrid_search`. The
  `hnsw` feature is **on by default**, so semantic search uses the approximate
  `clawhdf5-ann` index instead of a linear cosine scan. The index mirrors the
  memory cache (node id == cache index) and self-heals — it rebuilds whenever it
  drifts from the cache length, so no mutation path can desync it. Non-indexable
  stores (no/zero-dim/mixed embeddings) and dimension-mismatched queries fall
  back to the exact linear scan. Disable with
  `--no-default-features --features float16` for exact search.
- `clawhdf5-ann`: HNSW is now a live, mutable index — added `insert`,
  `mark_deleted` (soft-delete bitset; deleted nodes are traversed for
  connectivity but never returned), `compact` (drops deleted vectors and
  renumbers survivors), and `new` (empty index). Serialization gains a format
  version tag (`HNSW_FORMAT_VERSION` = 2) and persists the deleted bitset;
  pre-existing v1 files still load.
- `clawhdf5-agent`: `hybrid::merge_vector_keyword` exposes the shared
  normalize-and-fuse step used by both the linear and HNSW vector paths.
- Expose `max_dimensions()` API on Dataset, MmapDataset, and LazyDataset
- NetCDF-4 unlimited dimension detection now works correctly
- Python bindings (`clawhdf5-py`) build and link on macOS with system Python

### Bug Fixes
- `clawhdf5-py`: upgrade PyO3 and numpy `0.23` → `0.28` so the bindings build on
  Python 3.14 (PyO3 0.23 capped at 3.13 and hard-failed `cargo build
  --workspace`). Updated for the removed `PyObject` alias (`Py<PyAny>`) and the
  `Python::allow_threads` → `Python::detach` rename.
- Fix GPU L2 distance test (squared vs actual L2 mismatch in test helper)
- Mark Android JNI functions as `unsafe` for Rust 2024 edition compliance
- Add `# Safety` documentation to all public unsafe extern functions
- Fix all clippy warnings: needless_range_loop, manual_strip, ptr_arg, etc.
- Rename `RelationType::from_str` to `from_label` to avoid trait confusion
- Isolate h5py interop tests with `#[ignore]` when h5py unavailable

### Code Quality
- Full rustfmt pass across workspace (61 files)
- Refine inner unsafe blocks for Rust 2024 edition style
- Zero clippy warnings, zero clippy errors across entire workspace
- 1,546 tests passing, 0 failures

## v2.0.0 (2026-03-19)

- Unified rustyhdf5 (11 crates) and edgehdf5 (4 crates) into a single workspace
- All crates renamed to clawhdf5-* prefix
- Version bumped to 2.0.0 across all crates
- Git dependencies replaced with in-workspace path dependencies
- Added `agent` feature flag to clawhdf5-agent