clawhdf5/CHANGELOG.md

Changelog

Unreleased

Testing

• The Python interop suites honour CLAWHDF5_PYTHON, and ci-test.sh picks up a .venv/bin/python automatically. On a PEP 668 "externally managed" system h5py cannot be installed into the system interpreter at all, so every interop suite — the h5py writer round-trips, the facade, netCDF4 and the reference files — was skipping silently. A silent skip here is exactly how the v5 compound-datatype bug reached a release. CLAWHDF5_REQUIRE_INTEROP=1 still turns a skip into a failure.

Memory

• clawhdf5-agent: MemoryConfig::quantized_index stores the vector index's own copy of the embeddings as i8 rather than f32, which at 100k 384-dim entries takes the index from 266 to 123 MiB and the whole reopened store from 399 to 256 MiB (2.72x -> 1.74x the raw vectors). Quantised distances are approximate and ef cannot compensate — recall@10 tops out at 0.967 against f32's 0.9995 — so the query path re-scores the candidate pool against the exact embeddings the store already holds, which restores recall (0.9940 vs 0.9945 at ef=64) for about 13% of QPS. Off by default: it trades query speed for memory, and which side is worth more depends on the deployment. The setting is persisted, so a reopened store does not silently revert to four times the index memory.
• clawhdf5-ann: Storage::Int8 and the build_with / new_with / from_graph_bytes_with constructors that select it. The scale is per row, not global — a fixed [-1, 1] scale spends fewer than 12 of the 255 levels on a unit-length 128-dim vector and is unusable (0.35 top-10 overlap against an exact ranking, versus 0.99 per row). compact() keeps the storage it was given; serialized indexes still carry f32 vectors, so a quantised index is rebuilt rather than loaded.

Memory

• clawhdf5-agent: a loaded store holds ~30% less memory (100k 384-dim entries: 505 -> 357 MiB, 3.44x -> 2.43x the raw vectors). The cache kept every embedding twice — a Vec<Vec<f32>> and a flattened copy for the batched kernels, maintained in lock-step — so it now stores only the flat buffer and indexes into it. Recall and query latency are unchanged. Breaking: MemoryCache::embeddings is a cache::Embeddings rather than a Vec<Vec<f32>> (indexing still yields a &[f32] row); embeddings_flat is gone, replaced by flat_embeddings(); rebuild_flat() is a deprecated no-op. Rows are now always exactly dim long — shorter ones are zero-padded — which makes the ragged-row case that used to silently misalign the flattened copy unrepresentable.
• clawhdf5-bench: search_harness --footprint reports live heap use per stage, measured with a counting allocator (RSS cannot see a structure freed into the allocator's own pool).

Unreleased

Retrieval quality

• clawhdf5-agent: re-ranking discarded the retrieval score. reranker::rerank built its combined score from temporal decay, source authority and Hebbian activation only — RerankInput had no relevance field — so re-ranking a candidate pool reordered it by age and threw the retriever's ordering away. The OpenClaw backend re-ranked every search, so this was its shipping behaviour: measured over the full LongMemEval haystack it cost 40.6pp of Hit@1 (11.0% vs 51.6%) and two thirds of MRR (0.183 vs 0.643). RerankInput::relevance and ReRankConfig::relevance_weight (1.0 by default) fix it: relevance leads and the metadata signals break near-ties, which restores retrieval (Hit@1 +0.4pp vs no re-ranking) and improves recency discrimination by 6–7pp. Breaking: RerankInput and ReRankConfig gained fields, so literal constructions need updating; ..Default::default() does not.
• clawhdf5-bench: the LongMemEval harness feeds the dataset's real session dates to the store instead of a synthetic counter (decay needs true intervals, not just the right order), and reports newest_gold_first — on a knowledge-update question, did the newest gold session outrank the stale one it supersedes? Plain recall cannot see this, because both are labelled gold. New --rerank-sweep.

v2.5.0 (2026-09-19)

Upgrade Notes

• Retrieval rankings change, for the better. The default fusion weights move from 0.7/0.3 to 0.4/0.6 (hybrid::DEFAULT_FUSION), measured over the full LongMemEval haystack: turn-level Hit@1 51.6% vs 44.2%, MRR 0.643 vs 0.586. unified_search and the OpenClaw backend pick this up automatically; callers passing weights to hybrid_search explicitly are unaffected.
• Out-of-range selections are now errors. read_*_selection used to return data for a selection that ran past a dataset edge — a hyperslab came back zero-padded, and a point with an out-of-range coordinate wrapped into the next row. Both are now FormatError::SelectionOutOfBounds. Code relying on the old (wrong) values will start seeing errors.
• Large compressed datasets written without explicit chunk dimensions get a different layout. They used to be stored as one chunk; they are now split to ~1 MiB chunks. The files stay standard and h5py-readable, and explicit with_chunks is unaffected.
• rayon is now a default dependency of clawhdf5-agent (the parallel index build). Opt out with --no-default-features --features float16,hnsw.
• clawhdf5-ann search results no longer shrink when records near the query have been deleted, so a search that previously returned fewer than k results now returns k.

Retrieval quality

• clawhdf5-agent: optional keyword stemming — bm25::TokenFilter::Stemmed and HDF5Memory::set_token_filter, so "training" and "trains" match. Off by default, on measurement rather than principle: over the full LongMemEval haystack it buys depth and costs the top rank (BM25 alone: Hit@5 +2.8pp, Hit@10 +2.4pp, Hit@1 −1.8pp, MRR unchanged), and on the shipping hybrid configuration the trade is narrower still. See BENCHMARKS.md.
• clawhdf5-agent: QueryExpander::expand panicked on ordinary non-ASCII input — "İ AI" was enough. It searched a lowercased copy of the query and then sliced the original with those offsets, which only works while lowercasing preserves byte length (Turkish İ is 2 bytes and lowercases to 3). Depending on where the offsets drifted it either corrupted the output ("İstanbul AI trip" lost a character) or panicked. Matching now walks the original string.
• clawhdf5-agent: query expansion no longer rewrites text inside words. replace_word_case_insensitive did a plain substring replace despite its name, so "training" became "trArtificial Intelligencening" and "programming" became "Pull Requestogramming" — every acronym expansion of ordinary prose was corrupt. Matches now require word boundaries; genuine acronyms (API, database) still expand.
• clawhdf5-agent: the default fusion weights are now the measured ones. A sweep of every 0.1 step over the full LongMemEval haystack (500 questions, real MiniLM embeddings) shows the long-standing 0.7/0.3 default is strictly dominated by 0.4/0.6 — turn-level Hit@1 51.6% vs 44.2%, Hit@5 81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.643 vs 0.586, and better at session level too. The finding was recorded in BENCHMARKS.md but had never been applied: unified_search and the OpenClaw backend both hardcoded 0.7/0.3. They now use hybrid::DEFAULT_FUSION. Callers passing weights to hybrid_search explicitly are unaffected — pass 0.4/0.6 (or use hybrid_search_with) to get the tuned behaviour.
• clawhdf5-agent: fusion is now selectable. New hybrid::Fusion (Weighted { vector, keyword } or Rrf { k }), hybrid::fuse, hybrid::hybrid_search_fused and HDF5Memory::hybrid_search_with. Reciprocal rank fusion existed but was unreachable from the store, so it had never been measured against the weighted sum; the LongMemEval bench now has an RRF mode.

HDF5 Read Path

• Selection reads cost what the selection costs. read_*_selection decoded the entire dataset and then picked elements out, so a 64 x 64 window of a 64 MB compressed dataset took 105 ms - about as long as reading all of it. Now only the rows (contiguous) or chunks that overlap the selection's bounding box are read and decompressed: that window takes 0.39 ms, one row 2.7 ms, one column 5.2 ms. Results are identical to the full-read path (equivalence-tested over random hyperslabs and point lists, ranks 1-3, contiguous / chunked / deflate). New read_harness bench binary.
• Faster full reads (same-moment A/B, 64 MB f64): chunked + deflate 110 -> 69 ms, chunked 72 -> 60 ms, contiguous 56 -> 30 ms. The facade's cached read path now decompresses cache misses in parallel batches (it was sequential; only the uncached reader was parallel) and caches only datasets that fit the chunk cache; unfiltered chunks are copied straight from the file bytes; a contiguous dataset is converted straight from the file bytes; and the native-endian conversions no longer zero a buffer before overwriting it.
• Datasets indexed by a version-2 B-tree now read (layout v4, chunk index type 5 — what libver='latest' uses for two or more unlimited dimensions; previously "unsupported chunked layout"). The four copies of the chunk-index dispatch are now one shared function, so every read path gets it.
• H5T_STD_REF references (HDF5 1.12+, datatype message version 4) parse: ReferenceType gains Object2, DatasetRegion2 and Attribute, and read_object_references decodes the new object references. Previously any dataset of this type failed with InvalidReferenceType(2). Tested against a file written by HDF5 2.0 itself (fixture + generator script committed).
• Automatic chunk sizes. Asking for compression (or any filter) without with_chunks used to store the whole dataset as one chunk, so any read had to decompress everything and nothing could be decoded in parallel. Datasets up to 1 MiB stay a single chunk, as before; larger ones are split by halving the dimensions in turn until a chunk is at most 1 MiB (the approach h5py takes). Behaviour change: large compressed datasets written without explicit chunk dimensions get a different (standard, h5py-readable) layout. Explicit with_chunks is unaffected.
• Out-of-range selections are errors. They used to return data: a hyperslab past an edge came back padded with zeros, and a point whose column was out of range wrapped into the next row and returned that element. Now FormatError::SelectionOutOfBounds (also for a rank mismatch or overlapping blocks).

Search

• clawhdf5-ann: faster index builds. Back-link pruning is 90% of a build's distance evaluations; the bulk build now inserts in batches and prunes each overflowing neighbour list once per batch (10K: 1676 -> 1074 ms). With the parallel feature, planning and pruning run on a thread pool (10K: 388 ms, 100K: ~21 s -> 5.9 s on 16 cores). The graph is deterministic and identical with or without the feature. clawhdf5-agent's parallel feature enables it for the agent's index and is now on by default (adds rayon to the default dependency set; build with --no-default-features --features float16,hnsw to opt out).
• clawhdf5-ann: HnswIndex::search returned fewer than k results — often none — when the records nearest the query had been deleted: it collected ef candidates, then dropped the deleted ones, then took k. Deleted nodes are now traversed as waypoints but never occupy a result slot, so a search returns the k nearest live records. Matters for any store that deletes or supersedes memories without compacting straight away.

v2.4.0 (2026-09-19)

Upgrade Notes

• Search results improve on upgrade. The HNSW index now reaches true neighbours it previously could not (recall@10 0.31 -> 0.98 at 100K records on clustered data), so hybrid_search rankings change for the better. The agent rebuilds its index from the store automatically; a standalone HnswIndex persisted with to_hdf5_bytes keeps its old graph until rebuilt.
• hybrid_search no longer writes the store. Hebbian activation boosts are persisted by the next checkpoint (any flushing write, flush_wal, or when the HDF5Memory is dropped) instead of inside every query; a crash before then forgets only the boosts since the last checkpoint. Activation weights are now capped at 16.
• A new sidecar file, <store>.h5.ann, holds the vector index graph. It is derived data: safe to delete (the index is rebuilt), copied by snapshot(), and worth including when copying a store by hand to avoid a rebuild.
• BM25Index no longer caches IDF and gained add_document, remove_document, pad_to, scores, len and is_empty; results are now deterministic (ties break by record id).

Search

• clawhdf5-ann: HNSW recall fix. Neighbours were chosen as the plain closest-M, which on clustered data (what embeddings look like) turns each cluster into an island: recall@10 was 0.87 / 0.67 / 0.31 at 1K / 10K / 100K vectors and did not improve with ef. The index now uses the HNSW paper's diversity heuristic (Algorithm 4 with kept pruned connections) when linking a new node and when pruning back-links: recall@10 at ef = 64 is 1.00 / 1.00 / 0.98 and responds to ef. Builds are slower (~3.5x at 10K). Existing persisted indexes keep their old graph until rebuilt; the agent rebuilds its index from the cache, so stores pick this up automatically.
• clawhdf5-agent: hybrid_search is 23-39x faster in steady state (p50 5.5 -> 0.24 ms at 1K records, 49 -> 2.1 ms at 10K, 884 -> 23 ms at 100K). Every query used to rebuild the BM25 index from scratch and rewrite the whole .h5 file. The keyword index now lives for the life of the store and is updated incrementally (add / remove / in-place update, exactly equivalent to a fresh build - property-tested), and a query no longer writes the store. Behaviour change: Hebbian activation boosts are persisted by the next checkpoint (any flushing write, flush_wal, or drop) rather than immediately; a crash in between forgets only the boosts since the last checkpoint. Activation weights are now capped (16.0) - they grew without bound.
• clawhdf5-agent: the vector index is persisted, so open() no longer rebuilds it on the first search (first query after open: 2627 -> 15 ms at 10K records, 36 s -> 159 ms at 100K). The HNSW graph — not the vectors, which the store already holds — is written to <store>.h5.ann at each checkpoint and tied to it by a generation id in /meta; a missing, stale, damaged or structurally invalid sidecar is ignored and the index rebuilt. Records replayed from the WAL join the loaded index incrementally; a replayed update or delete invalidates it. snapshot() copies it. Batch saves no longer force a full index rebuild.
• clawhdf5-ann: faster HNSW build and search with identical recall. The cosine metric stores unit vectors and compares them with a plain dot product (it re-derived both norms on every distance evaluation), and the per-call HashSet of visited nodes is a reusable epoch-stamped array. Build 2.75 -> 1.89 s at 10K and ~38 -> 21 s at 100K; QPS at ef = 64 22.7K -> 39K at 10K. Distances returned by search are unchanged (1 - cosine). Indexes loaded from older HDF5 files are normalised on load.
• clawhdf5-accel: the SIMD backend is detected once per process instead of on every kernel call.
• clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes — graph-only serialization (checksummed, every neighbour id and level validated on load).
• clawhdf5-agent: a further 4-5x on hybrid_search with identical rankings (p50 now 0.07 / 0.49 / 4.65 ms at 1K / 10K / 100K — 79x / 100x / 190x faster than v2.3.0). Fusion needs every keyword score but not their ranking: new BM25Index::scores returns them unsorted from a dense accumulator (it hashed every posting, then sorted every match), and merge_vector_keyword selects its top k instead of sorting every candidate. Capping the keyword candidate pool was measured and rejected: it changes the top-10 for most queries (search_harness --fusion-study).
• clawhdf5-agent: BM25 results are deterministic (ties break by record id), top-k uses a bounded heap, and the "WAND early termination" that computed a bound and then ignored it is gone. IDF is computed per query.
• clawhdf5-bench: new search_harness binary — HNSW recall@10 / QPS / latency per ef against an exact scan, and end-to-end hybrid_search timings, on deterministic clustered (or --uniform) data. Baseline in BENCHMARKS.md.

v2.3.0 (2026-09-19)

Upgrade Notes

• A memory store now has a single writer. HDF5Memory::create/open take an exclusive lock (<store>.h5.lock); a second open of the same store — in the same or another process — returns MemoryError::Locked. Code that opened a second handle just to read should use HDF5Memory::open_read_only.
• Unsigned array attributes arrive as AttrValue::U64Array, not I64Array, and attrs() may now return AttrValue::Raw. Exhaustive matches on AttrValue need the two new arms.
• WAL header version 3 → 4. v3 files are read and upgraded in place, but a store written by 2.3.0 with a pending WAL cannot be opened by 2.2.0 or earlier (it is refused, not corrupted). Checkpoint first (flush_wal) if you need to downgrade.
• MemoryConfig::compression now uses deflate unless the agent's new zstd feature is enabled; it previously failed outright in a default build.
• MemoryError gained Locked; FormatError gained UnresolvedSharedMessage, ExternalDataFilesUnsupported and ExternalLinkUnsupported; MessageType gained ExternalDataFiles.

Bug Fixes

• clawhdf5-format: compound datatypes written with default libver bounds (datatype message version 1 — what plain h5py.File(path, 'w') produces) were mis-parsed. The v1 member layout carries 28 bytes of legacy array fields after the byte offset (the parser skipped 24), and v2 pads member names to 8 bytes and has no array fields at all (the parser did neither), so every member after the first byte offset was read from the wrong position — typically surfacing as Overflow("compound member ...") on read. Found by adding a default-libver axis to the h5py interop tests; byte-level regression tests for v1 and v2 added.
• clawhdf5-gpu: gpu_tests could hang forever under the default parallel test runner — every test created its own wgpu instance and device at once. Tests now serialise GPU access, and GPU→CPU readback waits are bounded (30 s) so a wedged driver returns GpuError::BufferMap instead of blocking.
• clawhdf5-agent: benches/bench.rs and benches/memory_bench.rs no longer compiled against the current strategy/consolidation APIs.

HDF5 Compatibility

• clawhdf5-format/clawhdf5: datasets and attributes that use a committed (named) datatype now read correctly. They store a shared-message reference; the facade parsed the reference bytes as the datatype (Time { size: 0 }, unreadable data) and silently dropped such attributes. The shared-reference parser itself was wrong for real files: version 2 has no reserved bytes, and the version 3 types were inverted (1 = SOHM heap, 2 = committed).
• Fill values are applied on read. There was no Fill Value message parser: the holes of a sparse chunked dataset read as zeros even when the fill value was not zero (silently wrong data), and a dataset that was created but never written failed with NoDataAllocated where h5py returns a filled array. Messages v1–v3 and the old 0x0004 form are parsed; the fill value is written into exactly the chunk-grid cells missing from the chunk index.
• Soft links are followed during path resolution, in old- and new-style groups (absolute/relative targets, links to groups, links through links), with a depth limit so a link cycle is an error rather than a hang. A dangling link reports the target it could not find.
• Things the reader does not follow are now explicit errors instead of wrong answers: an external link is ExternalLinkUnsupported { filename, object_path } (was PathNotFound), and a dataset whose raw data lives in external files (message 0x0007, now a known MessageType) is ExternalDataFilesUnsupported (it would otherwise read as fill values).
• attrs() no longer drops attributes. Any attribute whose datatype had no AttrValue variant was omitted with no error — including every Python bool (h5py stores attrs["flag"] = True as an enum), complex numbers, compound values and object references. Now:
  • numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1) decode as I64 / I64Array of 0/1;
  • new AttrValue::U64Array keeps unsigned arrays unsigned (they were cast to I64Array, so values above i64::MAX came back negative). Behaviour change: code matching I64Array for an unsigned attribute must also match U64Array (the netCDF-4 CF helpers and Python bindings do);
  • new AttrValue::Raw { datatype, shape, data } carries everything else verbatim, decodable with clawhdf5_format::data_read against datatype. Both new variants are writable, so an attribute can be copied between files unchanged. Python receives Raw as {"dtype", "shape", "data"}.
• All of the above are covered by h5py interop tests under both default and libver='latest' bounds, compared against h5py's own readback.

Security

• clawhdf5: virtual-dataset source file names are untrusted input but were joined straight onto the opened file's directory, so a crafted file could make the reader open any path the process can reach (absolute path, or .. components). Only plain relative paths inside that directory are accepted.

Durability & Integrity

• clawhdf5-agent: a crash between writing a checkpoint and truncating the WAL no longer duplicates every pending entry on the next open. Each checkpoint records a WalMark (byte length + chained CRC of the WAL prefix it folded in) in /meta; open() skips exactly that prefix when it is still present. No WAL format change for this; older files behave as before.
• clawhdf5-agent: checkpoints and snapshots are durable as a unit — the temp file is synced before the rename and the directory after it. Individual WAL appends remain unsynced by design (documented in CLAUDE.md).
• clawhdf5-agent: save_or_update hits are logged as a new Update WAL record, so replay updates in place instead of appending a duplicate. WAL header version 3 → 4 (so older builds refuse the file rather than truncating a record they can't parse); v3 files are read and upgraded in place.
• clawhdf5-agent: loading validates every per-record dataset length (a truncated store is now MemoryError::Schema, not a later panic), fixes the n.len() == n.len() tautology that trusted a norms dataset of any length, and rejects embedding_dim == 0 with records present.
• clawhdf5-agent: eight behavioural MemoryConfig fields are now persisted in /meta. Previously they reset to defaults on every open — a compressed store was rewritten uncompressed, wal_enabled = false flipped back to true.
• clawhdf5-agent: compression = true never worked in a default build (it requested Zstd without enabling the feature, so every checkpoint failed with unsupported filter: 32015). Default builds now use deflate; Zstd is the new opt-in zstd feature.
• clawhdf5-agent: single-writer lock (<store>.h5.lock, MemoryError::Locked) — two handles on one store used to silently destroy each other's data. New HDF5Memory::open_read_only gives a lock-free, never-writing view; the CLI's read-only subcommands use it.
• clawhdf5-agent: an unreadable WAL (torn header / bad magic) is quarantined (HDF5Memory::quarantined_wal()) instead of blocking open() of a healthy store. A WAL from an unknown newer version still fails and is left intact.
• clawhdf5-agent: provenance records are renumbered on compaction (they weren't, so every later save_or_update raised a false High integrity alert); pending anomaly alerts and tracked sessions are bounded; snapshot() includes entries still in the WAL.
• clawhdf5-agent: hybrid ranking is deterministic (index tie-breaks instead of HashMap order); a set of identical positive scores — including a single candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer reinforces zero-score filler results.
• clawhdf5-format: chunked/VDS/hyperslab reads size their buffers with overflow-checked arithmetic and fallible allocation, so crafted dimensions are FormatError::Overflow instead of a wrapped size or a process abort; parallel_read bounds checks use checked_add.
• clawhdf5: a malformed filter-pipeline message is an error instead of being treated as "no filters" (which returned compressed bytes as data); FileBuilder::write is atomic and synced instead of truncating the destination first.

CI / Testing

• CI now lints every target (cargo clippy --all-targets) plus clawhdf5-format's optional features, compiles all benches, and tests the format feature matrix. Previously test/bench code and feature-gated modules were never linted; the accumulated clippy backlog is fixed.
• CI installs python3 + h5py/numpy/netCDF4/xarray and sets CLAWHDF5_REQUIRE_INTEROP=1, which turns a missing interop dependency into a test failure. Until now every h5py/netCDF4 interop test silently skipped in CI, which is how the HDF5 2.0 compound bug fixed in v2.2.0 reached a user. The #[ignore]d writer_h5py_tests suite is run explicitly.
• h5py-generated-file tests now cover default libver bounds as well as libver='latest' (HDF5 2.0 raised the default low bound to 1.8).
• clawhdf5-agent: WAL property tests (round trip; after any corruption the entries read back are an exact prefix of what was written — 1500 seeded cases), a crash-recovery matrix (an on-disk image after every operation, the checkpoint window, and the WAL torn at every byte length, each reopened and checked against a model), and a WAL fuzz target.
• Optional fuzz smoke run (CLAWHDF5_FUZZ_SECONDS=N scripts/ci-test.sh); new datatype corpus seeds for v1 compound and native complex messages.

v2.2.0 (2026-09-18)

Security

• clawhdf5-format: bounded decompression output (MAX_DECOMPRESS_SIZE) for deflate/lz4/zstd/pcodec so a crafted compressed chunk can't drive an unbounded allocation (memory-exhaustion DoS).
• clawhdf5-format: chunked_read.rs/data_read.rs/local_heap.rs bounds audit — added ensure_len overflow guards at every plain-arithmetic offset+size check, a recursion-depth guard against a crafted self-referencing/cyclic B-tree chunk index, a fix for an unguarded compound-datatype byte_offset overrun in read_compound_fields, and an ndims - 1 underflow guard for degenerate zero-dimension chunked layouts. Added a new fuzz_dataset_read cargo-fuzz target (walks every dataset in a parsed file and exercises the contiguous/chunked/compact raw-data read paths) which found and fixed 3 real crash bugs — an integer-multiply overflow in copy_chunk_to_output's N-D assembly path, the ndims - 1 underflow above, and an overflow in local_heap.rs — within the first few fuzzing runs.
• clawhdf5-format: btree_v1.rs overflow-safe bounds checks via a local ensure_len helper, closing a usize-overflow panic reachable from a crafted near-usize::MAX B-tree offset.
• clawhdf5-agent: WAL length-prefix caps (MAX_WAL_FIELD_LEN, 64 MiB) reject a corrupted/truncated length claim before allocating. Followed by a full per-entry CRC32 trailer (WAL_VERSION bumped to 2) — a bit-flip inside an entry now stops replay cleanly instead of silently accepting corrupted data. Old-format WAL files are still read correctly and migrated to the new format on next open.
• clawhdf5-android: validate embedding_len/query_embedding_len against the handle's configured embedding_dim (and reject null pointers) before constructing a slice from a raw pointer in edgehdf5_save / edgehdf5_hybrid_search.
• clawhdf5-py: bump pyo3/numpy 0.28 → 0.29, clearing two RUSTSEC advisories (OOB read in PyList/PyTuple iterator; missing Sync bound on PyCFunction::new_closure).
• Clarified that the integrity hashes in clawhdf5-agent::provenance (FNV-1a) and clawhdf5-format::provenance (SHA-256) are unkeyed and detect only accidental corruption, not tampering — doc-only change, no behavior change.

Performance

• clawhdf5-format: chunk cache lookup is now O(1) (slot_index: HashMap) instead of a linear scan, and cache hits return a shared Arc instead of cloning the decompressed buffer — the hottest path in chunked reads.
• clawhdf5-ann: optional parallel feature (rayon) parallelizes HNSW's prune_connections neighbor-distance computation. The outer build/insert loop is deliberately left sequential — it has genuine cross-iteration data dependencies and needs its own correctness-focused design pass.
• clawhdf5-format/chunked_read.rs: removed 12 unnecessary chunk_dimensions[..rank].to_vec() allocations where callees already accept &[u32].

Architecture

• Added .gitea/workflows/ci.yml, actually wiring the long-existing scripts/ci-test.sh (fmt, clippy, tests, no_std check) into CI on every push/PR to main. Fixed stale package names in ci-test.sh/ check-nostd.sh that had been silently no-op'ing the clawhdf5-py exclusion and the no_std check.
• Fixed a genuine no_std build break in clawhdf5-format (uncovered once the no_std CI check actually started running): core::sync::atomic::AtomicU64 doesn't exist on thumbv7em-none-eabihf (switched to portable-atomic), missing alloc imports for Box/Vec/format! on a few no_std paths, and f64::powi (std/libm-only) replaced with a local exponentiation-by-squaring helper in the scale-offset filter.
• Added [workspace.dependencies] for tempfile/criterion/half/serde, fixing a real version skew on half (2 vs 2.7 across crates).
• Fixed version skew: clawhdf5-py (pyproject.toml) and packages/clawhdf5-node (package.json) were both behind the actual crate version (2.1.0).
• Documented that the mpi-io feature's read/write paths are root-read +broadcast / gather-to-rank-0, not true collective I/O.

Documentation

• BENCHMARKS.md: re-ran the previously-undated "LongMemEval Results", "SIMD & Parallelism", and "Vector Search Latency"/"Comparison to MemX" sections on a second machine (tank, Ryzen 7 7800X3D) with explicit dates and reproduce commands. Found and corrected a methodology issue in the SIMD/Parallelism benchmark selection (several originally-compared benchmarks didn't actually isolate the scalar/SIMD/parallel axis).
• README.md / ROADMAP.md / CLAUDE.md: corrected several stale facts — the clawhdf5-types crate (removed earlier) was still listed in the README crate map; the LongMemEval numbers in the README badge and table didn't match the actual (much better) benchmark results in BENCHMARKS.md; total line-of-code and test-count figures were stale; clawhdf5-gpu's CubeCL→wgpu correction; documented the new clawhdf5-ann parallel feature flag, which had no entry in the Feature Flags table.

New Features

• clawhdf5-migrate: substantial engine improvements:
  • Real content validation — the post-migration check now reads the written HDF5 back and compares actual content (chunk text, embeddings, and every session/entity/relation field) against the source, not just row counts. A representative sample of chunk rows is verified by default; --validate-full checks every row. A corrupt migration that preserves counts no longer passes.
  • Configurable schema — table names are no longer hardcoded; queries are built from a SchemaConfig (table + ordered column names, defaulting to the ZeroClaw layout) with --chunks-table / --sessions-table / --entities-table / --relations-table overrides.
  • Streaming count pass — --dry-run now does a COUNT(*)-only pass per table instead of loading every row into memory.
  • Incremental migration — --incremental reads the existing output, reads only source chunks newer than the last migrated id, and appends them (refreshing the metadata groups), instead of re-migrating everything.
• clawhdf5-format: read IEEE-754 half-precision (f16) floats. read_as_f32 / read_as_f64 previously only handled 4- and 8-byte floats; 2-byte floats (e.g. float16-stored embeddings) now decode via a no_std-safe bit conversion.
• clawhdf5-format: write multi-block fractal heaps (root indirect block). Dense attribute and dense link storage previously capped at a single direct block (~64 KiB of heap data — a few thousand attributes/links). When the objects exceed one direct block, the heap now lays out a root indirect block (FHIB) over multiple direct blocks sized by the doubling table, distributing objects across blocks with correct per-block heap offsets. Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip through our reader and are read correctly by h5py. (Objects still may not span a block — no huge-object path.)
• clawhdf5-format: write dense group link storage (fractal heap + v2 B-tree). A group with more than 8 links (libhdf5's compact max_compact default) is now written densely — its links live in a fractal heap indexed by a v2 B-tree of type 5 (link-name index) referenced from the group's LinkInfo message — instead of as inline Link messages. This matches libhdf5's compact→dense switchover and keeps large groups out of the object header. Reverse-engineered against libhdf5: link heaps use heap_id_length 7 / max_heap_size 32 (vs 8 / 40 for attributes). The shared single-direct-block fractal-heap builder is now parameterized and used by both dense attributes and dense links. Validated end-to-end: our reader round-trips, and h5py reads the dense groups we write. (Single direct block — up to ~a couple thousand links per group; beyond that needs indirect blocks, still unsupported.)

Robustness

• clawhdf5-format: harden the readers added this cycle against malformed / hostile input — they parse untrusted bytes and must return errors, never panic, OOM, or recurse without bound. Fixed concrete vectors found by audit and locked in with adversarial tests:
  • Paged Fixed Array: 1 << max_nelmts_bits shift overflow (a u8 ≥ 64); element/page offset multiplications now checked; element count bounded by file size.
  • H5S selection decoder: ALL/NONE no longer claim 16 bytes they don't have; hyperslab rank capped at 32 (H5S_MAX_RANK) to stop a giant allocation; iter_linear coordinate/stride/product arithmetic is checked.
  • VDS mapping parser: no pre-allocation from the untrusted nused; all selection slicing is bounds-checked.
  • scale-offset / N-Bit filters: 1 << minbits overflow at minbits == 64; N-Bit bit_offset + precision overflow; N-Bit type-tree recursion depth capped (no stack overflow from a crafted nested tree); element counts bounded by the chunk's expected decompressed size so a bogus count can't drive a huge allocation.
  • Virtual Dataset assembly: a virtual dataset whose source is itself virtual (a cycle) now errors instead of recursing into a stack overflow.

New Features

• clawhdf5-agent: compress fixed-length string datasets (memory text chunks, session summaries, ids, tags, entity/relation names, …). These were always stored uncompressed with a "chunked compound not yet supported" note that was simply stale — chunked writes work for fixed-size string/compound datatypes like any other. write_string_dataset now chunks + deflates a string dataset once its payload reaches 4 KiB, so large, highly-redundant NullPad content shrinks substantially while tiny metadata stays contiguous (no chunk-overhead bloat).
• clawhdf5-format: decode the scale-offset filter (id 6) — both the integer variant (H5Z_SO_INT) and the floating-point D-scale variant (H5Z_SO_FLOAT_DSCALE). Handles signed/unsigned int sizes, f32/f64, negative minima, decimal scale factors and fill values; reverse-engineered against HDF5 2.0 and validated end-to-end. The float E-scale variant remains unsupported.
• clawhdf5-format: decode the N-Bit filter (id 5) — atomic, compound and array layouts (the full recursive type tree, nestable to any depth), previously unsupported. Signed and unsigned reduced-precision integers and float members all read end-to-end, validated against HDF5 2.0.

New Features

• clawhdf5 / clawhdf5-format: read external-file Virtual Datasets (VDS). The format layer gains read_raw_data_full_with_resolver and a VdsSourceResolver callback (Fn(&str) -> Option<Vec<u8>>) that maps a stored source file name to its bytes, so the pure-byte reader can pull in external sources without a filesystem of its own. The clawhdf5 File API wires a default resolver that reads sibling source files relative to the opened file's directory, so File::open(...).dataset(...).read_*() now transparently assembles cross-file VDS. A source file the resolver cannot supply leaves its region at the fill value (matching HDF5); an external source with no resolver at all is a clean error. In-memory files (File::from_bytes) have no directory, so only same-file VDS resolves there.
• clawhdf5-format: assemble same-file Virtual Datasets (VDS) of any rank. Previously a virtual layout returned UnsupportedVersion. The reader now decodes the global-heap mapping block (reverse-engineered against HDF5 2.0: version · nused · [source-file · source-dataset · source-selection · virtual-selection]* · checksum, including the block-version-1 same-file marker), decodes the H5S source/virtual dataspace selections (ALL, NONE, and version-3 regular hyperslabs), reads each same-file source dataset, and scatters its selected elements into the virtual buffer in row-major order (so multi-dimensional block mappings land correctly); unmapped regions are left at the zero fill value. External-file sources return a clean unsupported error. The previous parse_vds_mappings used a guessed layout that did not match real files and is replaced.

Tests

• clawhdf5-format: regression test for scale-offset float E-scale datasets. The HDF5 library does not implement E-scale encoding — when asked for it (cd_values[0] = 1) it stores the chunk raw and sets the chunk filter mask to skip the filter — so these files read back verbatim purely by honoring the per-chunk filter mask. The test locks in that behavior against a fixture produced via the HDF5 low-level API; no E-scale decoder is needed.

Bug Fixes

• clawhdf5-format: read multi-direct-block fractal heaps. The reader split direct vs indirect block rows using the FRHP "Starting # of Rows in Root Indirect Block" field (a constant, typically 1), so any heap whose data spans more than one direct block — common in libhdf5 files with a large group or many dense attributes — was misread as having indirect blocks and failed with InvalidFractalHeapSignature. The split is now derived from the heap geometry (max_direct_rows = log2(max_direct / start) + 2). Validated against an h5py-written 400-dense-attribute group (root indirect block, 4 rows, 13 direct blocks).
• clawhdf5-format: scope the per-file chunk cache by dataset. The shared ChunkCache built its chunk index once and reused it for every chunked dataset in the file, keyed only by chunk coordinate with no dataset discrimination. With a single chunked dataset per file this was latent; once a file holds two chunked datasets of different rank (e.g. a 1-D compressed string array and the 2-D embeddings matrix), the first dataset's index was reused for the second, panicking with an out-of-bounds chunk coordinate. The cache now rebinds (dropping its index, chunk-index map, layout, and decompressed slots) whenever the dataset being read changes, while still caching repeated/sequential access to the same dataset.
• clawhdf5-format: read paged Fixed Array chunk indexes. A filtered, fixed-dimension dataset with more than one data-block page (>1024 chunks by default) previously failed with "paged Fixed Array data blocks not yet supported". The reader now walks the page-init bitmap (MSB-first), skips uninitialized pages, and resolves each page's fixed full-size slot (including the short final page). Reverse-engineered and validated end-to-end against an HDF5 2.0 file.
• clawhdf5-format: read array-typed datatypes (e.g. an array-typed compound member) via read_as_i32/i64/u64/f32/f64 — previously a TypeMismatch. The array is read as a flat sequence of its base elements (recursing for nested arrays), applying base-type precision rules.
• clawhdf5-format: sign-extend reduced-precision fixed-point integers on read. A signed integer whose datatype precision is smaller than its storage size is stored zero-filled, so e.g. a 16-bit-precision -1 previously read as 65535. The integer read paths now extract the precision field and sign-extend (full-width types are unchanged). Completes signed N-Bit reads and also fixes un-filtered reduced-precision integer datasets.
• clawhdf5-format: read datasets written by modern HDF5 (1.14+/2.0, i.e. libver=latest). Compound (class 6) and array (class 10) datatype version 5 messages and data layout version 5 messages were rejected as invalid; they reuse the v3/v4 binary structure, so they are now accepted. This unblocks reading compound types and — critically — every chunked/compressed dataset written by HDF5 2.0. Found by running the h5py interop tests against h5py 3.16 / HDF5 2.0. Independently reported (with a patch) against the v2.1.0 tag by M. Scot Breitenfeld (The HDF Group) — v2.1.0 predates this fix.
• clawhdf5-format: parse HDF5 2.0 native complex datatypes (class 11, datatype version 5, e.g. H5T_COMPLEX_IEEE_F64LE). The properties are a single base floating-point datatype, not a compound-style member list; the old parser read the base type's bytes as member names, producing a garbage datatype, and failed with UnexpectedEof when a complex type was nested in a compound. It is now surfaced as the equivalent {r, i} compound (the shape h5py writes for numpy complex dtypes), with a size check against the base type. Validated end-to-end against an HDF5 2.0-written file.

Performance

• clawhdf5-format: chunked writes now compress all chunks up front via compress_all_chunks, running across rayon threads under the parallel feature when there are more than 4 filtered chunks. On-disk layout is unchanged. Speeds up compressed embedding writes in clawhdf5-agent (which enables parallel).

Documentation

• Fix stale package names across all 13 per-crate READMEs (rustyhdf5-* / edgehdf5-* → clawhdf5-*, usage versions → 2.1.0).
• Correct README workspace/test/crate stats and the CLAUDE.md CLI subcommand list; document the hnsw and format compression/checksum feature flags and the entity_extract / async_memory modules.

v2.1.0 (2026-06-03)

New Features

• clawhdf5-agent: HNSW now backs the vector stage of hybrid_search. The hnsw feature is on by default, so semantic search uses the approximate clawhdf5-ann index instead of a linear cosine scan. The index mirrors the memory cache (node id == cache index) and self-heals — it rebuilds whenever it drifts from the cache length, so no mutation path can desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and dimension-mismatched queries fall back to the exact linear scan. Disable with --no-default-features --features float16 for exact search.
• clawhdf5-ann: HNSW is now a live, mutable index — added insert, mark_deleted (soft-delete bitset; deleted nodes are traversed for connectivity but never returned), compact (drops deleted vectors and renumbers survivors), and new (empty index). Serialization gains a format version tag (HNSW_FORMAT_VERSION = 2) and persists the deleted bitset; pre-existing v1 files still load.
• clawhdf5-agent: hybrid::merge_vector_keyword exposes the shared normalize-and-fuse step used by both the linear and HNSW vector paths.
• Expose max_dimensions() API on Dataset, MmapDataset, and LazyDataset
• NetCDF-4 unlimited dimension detection now works correctly
• Python bindings (clawhdf5-py) build and link on macOS with system Python

Bug Fixes

• clawhdf5-py: upgrade PyO3 and numpy 0.23 → 0.28 so the bindings build on Python 3.14 (PyO3 0.23 capped at 3.13 and hard-failed cargo build --workspace). Updated for the removed PyObject alias (Py<PyAny>) and the Python::allow_threads → Python::detach rename.
• Fix GPU L2 distance test (squared vs actual L2 mismatch in test helper)
• Mark Android JNI functions as unsafe for Rust 2024 edition compliance
• Add # Safety documentation to all public unsafe extern functions
• Fix all clippy warnings: needless_range_loop, manual_strip, ptr_arg, etc.
• Rename RelationType::from_str to from_label to avoid trait confusion
• Isolate h5py interop tests with #[ignore] when h5py unavailable

Code Quality

• Full rustfmt pass across workspace (61 files)
• Refine inner unsafe blocks for Rust 2024 edition style
• Zero clippy warnings, zero clippy errors across entire workspace
• 1,546 tests passing, 0 failures

v2.0.0 (2026-03-19)

• Unified rustyhdf5 (11 crates) and edgehdf5 (4 crates) into a single workspace
• All crates renamed to clawhdf5-* prefix
• Version bumped to 2.0.0 across all crates
• Git dependencies replaced with in-workspace path dependencies
• Added agent feature flag to clawhdf5-agent