clawhdf5/CHANGELOG.md

Changelog

Unreleased

Upgrade Notes

• HDF5 correctness audit (2026-09-25). A sweep of 686 public files (the libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c, netcdf4-python, h5wasm, h5py and xarray corpora), a 567-case read matrix and a 96-case write matrix against HDF5 1.10–2.0 found bugs that returned wrong values with no error, and files we wrote that libhdf5 rejects. The fixes are listed under Correctness and Interop. What changes for callers:
  • Chunked datasets whose max shape is larger than their current shape, or whose unlimited dimension is not the first, were indexed by the current shape instead of the max shape, both when read and when written. Files from libhdf5 now read correctly. Files clawhdf5 wrote with such a max shape were laid out wrongly and now read the way libhdf5 always read them — rewrite them. Agent stores and ClawBrainHub files have no max shape and are unaffected.
  • Integer reads (read_i32/read_i64/read_u64/...) of float data now convert (truncate toward zero, saturate at the type's range, NaN reads as 0) instead of returning the IEEE bit pattern, and out-of-range integers saturate instead of keeping the low bits.
  • FileWriter::finish() now returns an error instead of writing a corrupt file for: a header message over 64 KiB (e.g. an attribute larger than ~64 KiB), a group/dataset/link name that is empty, . or contains / (nested paths were written as one literal link), a max shape smaller than the shape, a page size outside 512 B–1 GiB, and more than 65 535 chunks in a dataset with several unlimited dimensions.
  • Breaking (format crate): ObjectHeaderWriter::serialize, BatchObjectHeaderWriter::compute_sizes/serialize_all and build_chunked_data_from_precompressed return Result; read_fixed_array_chunks/read_extensible_array_chunks take max_dims; build_fixed_array_at/ea_writer::build_extensible_array_at take one Option<WrittenChunk> per index slot; fill_value::dataset_fill_value returns UnresolvedSharedMessage for a shared message it cannot resolve instead of None. FillTime::default() is IfSet (libhdf5's default; default files are byte-identical).
• ZeroClaw does not use clawhdf5. The project described itself as ZeroClaw's memory backend ("imported as a clawhdf5 Cargo feature"). Checked against ZeroClaw v0.8.5 (the latest release), the osobh/zeroclaw fork and their full history: no such feature or backend has ever existed. And clawhdf5-migrate's "ZeroClaw layout" (memory_chunks, sessions, entities, relations) is not ZeroClaw's schema — ZeroClaw uses a single memories table — so the migrator cannot read a ZeroClaw database. The claims are withdrawn; the migrator's layout is documented as its own.
• OpenClaw is not supported, and never was. The docs described a "drop-in" OpenClaw memory backend enabled with memory.backend = "clawhdf5". That config was never valid in any OpenClaw release (v2026.2–v2026.7 accepted only builtin/qmd and rejected unknown keys, so a Gateway given it refuses to start; OpenClaw 2.0 removed the key), no plugin was ever built, and @redclaw/clawhdf5 was never published. The integration docs (openclaw-integration.md, openclaw-config.md, migration-guide.md) are removed; docs/openclaw.md explains the status and what a real plugin would need against OpenClaw v2026.9.6. ClawhdfBackend stays as a library API.
• Breaking: MemoryError is now #[non_exhaustive] and gained SigningKeyRequired; a match on it needs a wildcard arm. Future variants will no longer be breaking.
• Breaking: clawhdf5-agent's agent feature is removed. It enabled nothing — the agent layer is always built — but the README and guides told people to pass it; drop agent from features = [...].
• clawhdf5-migrate now writes a real agent store. Its output used to be a layout of its own (/chunks, /sessions, /entities, /relations, no /meta) that HDF5Memory::open rejected, so a migrated file could not be used as agent memory. Files it wrote before this release are not agent stores; re-run the migration. Also: embeddings default to float16 like any new store (--f32 opts out; --float16 is a hidden no-op); a row with the wrong embedding length is an error instead of being truncated or padded; --incremental now matches rows by content against an existing store and follows the source's deleted flags; a source with no memory rows needs --embedding-dim. The per-dataset SHA-256 provenance attributes of the old layout are gone (the agent schema has no place for them).
• Files written by clawhdf5 now open in h5py and libhdf5. Every f32 dataset we wrote — including every agent store's embeddings — was refused with "sign bit position out of bounds", and every empty dataset with "invalid dataset size". Both were write-side bugs present in every release; clawhdf5's own reader was unaffected. An agent store is rewritten in full at each checkpoint, so it becomes readable at its next checkpoint on this version; other files with f32 or empty datasets need rewriting. Details in docs/known-issues.md.
• New stores store embeddings as half precision by default. MemoryConfig::float16 was persisted and otherwise ignored; it now writes float16 embeddings (48% smaller files at 100K) and rounds each embedding to half precision as it is saved — and it defaults to true for new stores. On the full LongMemEval haystack with real MiniLM embeddings every retrieval metric matched f32. Existing stores are unaffected: every agent store has recorded float16 = false, and keeps it (a v2.5.0 fixture guards this). A store that already had float16 = true rounds its embeddings when next opened and writes them as float16 at its next checkpoint. Opt out with float16 = false or create --f32; the CLI's --float16 is still accepted and now a no-op. Values beyond ±65504 are refused, so keep f32 for unnormalised vectors.
• Breaking: MemoryError gained InvalidEntry, returned when a float16 store is given an embedding value beyond ±65504. Exhaustive matches need the new arm.
• The default build no longer compiles any C. Deflate now defaults to the pure-Rust zlib-rs instead of zlib-ng, so building the core crates needs neither cmake nor a C compiler. Speed on HDF5 reads and writes is within 6% of zlib-ng, and compressed output is byte-identical. To keep zlib-ng, enable fast-deflate (on clawhdf5, clawhdf5-format or clawhdf5-filters); it overrides zlib-rs wherever it is on.
• A truncated deflate chunk is now an error. It used to read back short, with no error.
• Minimum supported Rust is 1.92, now declared in every crate's rust-version and checked in CI.
• New stores use the int8 vector index by default. MemoryConfig::quantized_index now defaults to true: a quarter of the index memory, builds 1.8x (x86-64) and 2.3x (Raspberry Pi 5) faster, and searches 1.63x and 1.18x faster at equal recall, measured on every configuration tested. Existing stores are unaffected — a store written with v2.6.0 or later keeps its persisted setting, and one written before the setting existed opens as false and keeps its f32 index. Set quantized_index = false, or pass create --f32-index to the CLI, to opt out. The CLI's --quantized-index is still accepted but is now a no-op.

Tools

• New crate clawhdf5-tools with the binary h5rs: HDF5 command-line tools without libhdf5, built only on the clawhdf5 facade and clawhdf5-format (no C, so it also builds as a static musl binary).
  • h5rs ls [-r] [-v] FILE[/path] lists objects like h5ls (its first two columns are h5ls's text on the test files) plus the datatype; -v adds address, link count, layout and chunk index, chunk size, storage, filters, datatype and attributes.
  • h5rs dump [--json] [-A] [-p] [-d PATH] FILE prints DDL text that is byte-identical to h5dump 1.14.6's on the test files (all layouts and chunk indexes, v1/v2 groups, compound, enum, strings, links, named types, attributes), or JSON in the HDF Group's hdf5-json layout (schema in the crate README).
  • h5rs stat FILE reports h5stat's object, link, rank, layout, filter, attribute, raw-data and file-size figures (equal to h5stat's on the test files); metadata space is one figure, not broken down.
  • h5rs diff [-r] [-q] [-d D] [-p R] A B [OBJ1 [OBJ2]] compares objects, kinds, datatypes, shapes, attributes, values and link targets; exit status 0/1/2 as h5diff's. Every path is compared, including every name of a hard-linked object and the members of a hard-linked group. Objects that cannot be compared count as a difference (h5diff exits 0 for them), and NaN equals NaN.
  • h5rs check [--data] FILE is a structural validator: it walks every object, parses every header message, verifies the checksums of every version 2+ structure it meets (superblock, object headers and continuation chunks, v2 B-tree nodes, fractal heap headers and — which the library's reads do not — every direct and indirect heap block, and extensible/fixed array chunk indexes), checks each chunk index against its dataset (aligned, in-extent, unique, plausibly sized chunks), and that raw data lies inside the file without overlaps. Every problem is printed with its address; exit 1 when there are any. libhdf5's h5check reads only the 1.8 format. On the conformance corpus it passes all 418 files that both clawhdf5 and h5py read in full, and check --data flags 147 of the 180 files of the CVE corpus (tank, 2026-09-26). It inherits the library's tolerance, though: 26 of the 33 it passes are files h5dump 1.14.6 rejects (see docs/known-issues.md, header checks).
  • Values over --max-bytes (default 1 GiB) are reported instead of read; a panic is caught and reported as an internal error (exit 3). scripts/h5rs-fuzz.sh runs every subcommand over a corpus (default the CVE reproducers, optionally with byte-flipped copies) with overflow checks, a timeout and a memory limit, and fails on any panic, crash or hang; scripts/h5rs-check-ok-files.sh runs check --data over the fully-read conformance files.
  • Because the library does not verify fractal heap block checksums when it reads a dense group's links or dense attributes, h5rs verifies a heap's blocks before reading from it and refuses a damaged one, as libhdf5 does, instead of printing what the damaged block holds.

Signing

• clawhdf5-agent: Ed25519-signed checkpoints — the README's "cryptographically verifiable memory", now true. With HDF5Memory::set_signing_key(key), every checkpoint stores a signed manifest: a SHA-256 per record (text, embedding as stored, channel, timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus hashes of the settings (and WAL mark), sessions and knowledge graph, with the per-record hashes in /integrity/record_hashes. HDF5Memory::verify(path, &public_key) recomputes everything from the file and reports which part changed and which records (changed_records); a forged manifest fails the signature. The key is never persisted; a signed store refuses to checkpoint without it (MemoryError::SigningKeyRequired), and remove_signature() is the deliberate way back to unsigned. Saves still in the WAL are not covered (wal_entries_unsigned). Tests include every kind of edit, and an edit made with h5py in place, which verify pinpoints. Cost: ~20% of a checkpoint, 32 bytes per record (BENCHMARKS.md, "Signed checkpoints"). New dependencies ed25519-dalek, sha2, rand_core — pure Rust; the no-C check still passes.
• clawhdf5-cli: keygen --out <file> (owner-only key file), --signing-key <file> / CLAWHDF5_SIGNING_KEY on writing commands (create signs immediately), verify --public-key <hex|file> (JSON report; exit status 2 if not valid), and signed in create/stats output.

Migration

• clawhdf5-migrate: writes through the agent's own API (HDF5Memory::create / open, save_batch, the session cache and knowledge graph), so there is no second copy of the schema. Sessions and entities/relations carry over; deleted rows become deleted records (or are left out with --skip-deleted). Every source row is checked before the output is created, so a source that cannot be migrated leaves an existing store untouched. Validation reads the result back with HDF5Memory::open_read_only, compares every field (embeddings bit for bit — round_to_f16 of the source for a float16 store) and checks that a migrated record is found by search. The half-based conversion is gone; clawhdf5_format::float16 is the only one. 42 tests, including h5py opening a migrated store; an adversarial review's two blocker and four major findings are fixed with regression tests.
• clawhdf5-agent: HDF5Memory::sessions() / sessions_mut(), HDF5Memory::delete_batch(&[usize]) (one save, all-or-nothing, never auto-compacts), SessionCache::add_at, and SessionCache / SessionEntry re-exported from the crate root.

Search

• clawhdf5-agent: HDF5Memory::search with SearchOptions — source filtering, re-ranking and confidence rejection in the store's own search path. Re-ranking and confidence rejection used to be reachable only through the OpenClaw backend, which now calls search with both on.
  • with_sources([..]) restricts a search to records from those source channels. It applies before ranking, so a filtered search still returns up to k results, normalised over what it can return. Measured at 100K: the exact filtered top 10 for filters keeping 50%, 10% and 1% of the store and for records far from the query, and never slower than an unfiltered search (2.3 ms for a 1% filter vs 4.6 ms unfiltered). See BENCHMARKS.md, "Search options".
  • with_rerank(ReRankConfig) re-ranks a pool of max(3k, 10) candidates (rerank_pool to change it) by relevance, recency, source authority and activation; with_confidence(ConfidenceConfig) drops low-confidence results; at_time(now) pins the clock for recency. About 3% on latency.
  • hybrid_search and hybrid_search_with are unchanged (tested bit for bit against search with default options).
• clawhdf5-agent: the OpenClaw backend's search now boosts the Hebbian activation of the k results it returns, not of the whole 3k candidate pool it re-ranks.

Documentation

• OpenClaw claims withdrawn across the README, QUICKSTART, USE_CASES, ROADMAP (Track 7 marked withdrawn) and the openclaw module docs; the dead github.com/redclawsystems/openclaw link is gone. The Node package is marked unpublished and broken (now "private": true so it cannot be published by accident), with its bugs recorded in docs/known-issues.md.

Benchmarks

• Every undated or pre-September section of BENCHMARKS.md re-run on one machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for each and every number traced back to the raw output by a separate check. Where a figure moved, the section says so. Two apparent regressions were isolated rather than published: knowledge-graph traversal (a real bug, fixed above) and the write path, which measures the same at v2.3.0 on this machine — the old 18 µs / 6.17 ms figures came from an undated run on other hardware; float16 adds ~2 µs per save and the int8 index nothing.
• New multimodal_bench: cross-modal search at 1K and 10K records, which the README claimed but nothing measured.
• footprint_bench reports whether it built float16 or f32 stores and takes --f32; it had kept printing "f32" after the default changed.

Interop

• Conformance sweep in the repo (conformance/, report in CONFORMANCE.md). conformance/run.sh fetches eight public HDF5 corpora pinned by commit (libhdf5's test files, the HDF Group's CVE reproducers, pyfive, netcdf-c, netcdf4-python, h5wasm, h5py, xarray-data) into a gitignored cache, reads every file with clawhdf5 and with h5py/libhdf5 (and the CVE files with h5dump) under a timeout and memory limit, compares them object by object and regenerates the report — about 30 s once the corpus is cached. A nightly Gitea job (.gitea/workflows/conformance.yml) runs it and fails on any panic, hang, crash or out-of-memory, or when a file in conformance/baseline.json stops reading identically. First report, on 42b81d9: 467 of 697 files identical to h5py, 123 our-error, 15 mismatch (2 of them an h5py bug), 92 that libhdf5 cannot read, no panics, hangs or crashes. Compared with the ad-hoc audit sweep, the probe now compares N-Bit floats (and integers with a bit offset) as the values libhdf5 converts them to rather than raw file bytes — 8 files that were reported as mismatches read identically — and the reference side no longer flips between runs when libhdf5 aborts while freeing h5py objects.
• clawhdf5-format: every f32 dataset was unreadable by h5py and libhdf5. The float datatype encoder hard-coded the sign bit's position to 63, correct only for f64; libhdf5 validates it and refused the dataset. It is now derived from the type (15 / 31 / 63). Our reader ignores the field, and the interop suites only wrote f64, which is how it went unnoticed.
• clawhdf5-format: every empty dataset was unreadable by h5py and libhdf5. It was written with a real address and zero bytes, which trips libhdf5's addr + size <= addr overflow check. An empty contiguous dataset now gets the undefined address, as libhdf5 writes it. This affected every agent store without sessions or a knowledge graph.
• New interop tests: f32 and float16 datasets in both directions (our float16 rounding matches numpy's bit for bit on 4 020 probe values, including ties, subnormals and the overflow boundary), and an agent store — f32 and float16 — opened by h5py with every dataset decoded.
• clawhdf5-format filters, checked against libhdf5 + hdf5plugin:
  • LZ4 (32004) now uses the registered HDF5 LZ4 format (8-byte BE size, 4-byte BE block size, BE-length-prefixed blocks). Our old framing (4-byte LE size + one block) was readable only by clawhdf5, and we could not read libhdf5's (h5ex_d_lz4.h5). Old clawhdf5 LZ4 chunks still read; they are told apart unambiguously (a registered chunk starts with four zero bytes).
  • Zstd (32015) frames now record the content size, which libhdf5's zstd plugin needs; h5py could not read our zstd datasets.
  • Pcodec moved from filter ID 32023 to 480. 32023 is registered to Granular BitRound, whose decode is a pass-through — libhdf5 with that plugin would have returned compressed bytes as data. Pcodec has no registered ID; 480 is in the registry's private range (256–511) and only clawhdf5 can read it. Chunks written under 32023 with the filter name pcodec (clawhdf5 ≤ 2.7.0) still read.
  • SZIP decode matches libhdf5. It returned garbage or zeros with no error for libhdf5-written files (the 4-byte size prefix, 32/64-bit byte-plane interleaving, reference interval, scanline padding and byte order were all handled wrongly) and rejected 64-bit data.
  • N-Bit honours libhdf5's "need not compress" flag (multi-filter pipelines such as tfilters.h5 failed) and reads enum/no-op members.
  • Scale-offset float decode uses libhdf5's single-precision arithmetic (was 1 ULP off for some values).
  • A pipeline with Fletcher32 ahead of the compressor (h5py set_fletcher32() then set_deflate()) no longer fails with "deflate: output exceeds size limit".
• clawhdf5-format: HDF5 1.4/1.6-era files are readable. Data Layout message versions 1 and 2 (compact, contiguous, and chunked through the version-1 B-tree) failed with InvalidLayoutVersion — 84 of the 686 files in the 2026-09-25 audit sweep, 205 datasets. They now read as libhdf5 does; checked byte for byte against h5py on HDF5's own test files (tests/legacy_format_interop.rs).

Storage

• clawhdf5-format: half-precision datasets. DatasetBuilder::with_f16_data writes IEEE binary16 (numpy float16), rounding to nearest-even; make_f16_type, and clawhdf5_format::float16 with the conversions, which are checked against the half crate on 16.7M values and round-trip all 65 536 half values. Reading float16 as f32 gained a little-endian fast path.
• clawhdf5-agent: MemoryConfig::float16 stores embeddings as half precision. At 100K x 384 the file goes from 154.0 to 80.8 MiB (−48%), a checkpoint from 752 to 512 ms and open from 300 to 252 ms, with the same vector recall@10 against an exact scan (0.999 vs 0.994) and the same hybrid_search latency; at 10K open is 3 ms slower. On the full LongMemEval haystack with real MiniLM embeddings every retrieval metric is identical to f32 (longmemeval_bench --float16). The cache rounds each embedding as it is saved, so memory and file agree bit for bit and a store returns the same results before and after a reopen (tested). Out-of-range values are refused with MemoryError::InvalidEntry rather than stored as infinity; batches are all or nothing. CLI: create --float16. See BENCHMARKS.md, "float16 embedding storage".

Build

• Pure-Rust default. clawhdf5-format, clawhdf5-filters and the clawhdf5 facade default to the zlib-rs deflate backend; fast-deflate (zlib-ng) is opt-in. No crate in the default dependency tree of the core crates compiles C, and ci-test.sh now fails if one appears. The facade's fast-deflate was on by default and is now off. See BENCHMARKS.md, "Deflate backend".
• zlib-rs also enables flate2's runtime_detection. Without it zlib-rs has no std, cannot detect SIMD at runtime, and inflates 3.5x slower; the workspace builds flate2 with default-features = false, which had been switching it off.
• rust-version = "1.92" for the whole workspace (the floor: wgpu requires it), and CI checks the workspace on exactly that toolchain.
• CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.

Correctness

• clawhdf5-format VDS: variable-length and reference data from a source in another file is refused. Those elements are global-heap IDs and object addresses in the source file; copied into the virtual dataset they would be decoded against the wrong file and name another object.
• clawhdf5-agent: a store whose /meta has an attribute that cannot be decoded fails to open (MemoryError::Schema). With attrs() now leaving unreadable attributes out, it would otherwise have opened with defaults in place of its settings (float16, compression, the WAL mark, ...).
• clawhdf5-format reader: an old-style group whose local heap has a free list pointing outside the heap was listed with names read from the broken heap (garbage names on cve-2021-36977.h5 once its user block was applied). libhdf5 refuses such a heap ("bad heap free list"); so do we now, with FormatError::InvalidLocalHeapFreeList. As in libhdf5 the free list is checked when the first name is read (LocalHeap::validate_free_list, new), so an empty group with a damaged heap still lists as empty.
• Files with a user block (h5py.File(..., userblock_size=N), h5jam; the superblock at 512, 1024, …) could not be read: every address in the file is relative to the superblock, but it was applied from byte 0 (InvalidObjectHeaderVersion on the root group). File (mmap, buffered, from_bytes), MmapFile, LazyFile, AsyncHDF5File, the VOL readers, the HNSW loader and external VDS sources now view the file from the superblock on, using the signature's position as the base address as libhdf5 does; user_block_size() reports the user block (h5py's userblock_size), and as_bytes() returns the bytes from the superblock on. Breaking (format crate): Superblock::parse refuses a non-zero signature offset with FormatError::UserBlockNotStripped, since the addresses it returns would be applied to the wrong bytes; pass the slice from signature::split_user_block (new) and parse at offset 0.
• clawhdf5-format reader: version-1 shared messages (HDF5 1.6-era files, e.g. a dataset using a committed datatype in libhdf5's tcompound.h5) read the heap-offset field of the embedded symbol-table entry as the target address and failed with InvalidObjectHeaderVersion. The address is now read after it, as libhdf5 does. Breaking (format crate): shared_message::parse_shared_ref takes length_size. A reference whose target header has no message of the referenced type is now FormatError::SharedMessageTargetMissing instead of returning the first other message found there (which decoded as garbage).
• clawhdf5-format reader: array members of version-1 compound datatypes (HDF5 1.6-era files, e.g. libhdf5's tcompound.h5) were read as a single element: a [4] i32 member came back as one i32, with the wrong size. The legacy per-member dimension fields are now decoded into an array type, as libhdf5 does; more than four dimensions, or a zero-sized one, is an error.
• clawhdf5-format virtual datasets (VDS), checked against HDF5 2.0 through h5py (crates/clawhdf5/tests/vds_interop.rs):
  • Wrong data: elements no mapping supplies — unmapped regions, and mappings whose source file or dataset is missing — read as 0 instead of the virtual dataset's fill value (e.g. h5py fillvalue=-1). Assembly moved to the new vds module: vds::read_virtual_dataset takes the fill value and a resolver that can refuse a name (VdsFileResolver), and File passes the dataset's fill value. A missing source dataset read as an error; it is fill now, as in libhdf5. Source datasets are read with their own fill value for unallocated chunks, and a source whose datatype differs from the virtual dataset's is an error (libhdf5 converts; we do not). File now refuses a source name that leaves the virtual file's directory (../x.h5, absolute paths), or any external source of a File::from_bytes file, with an error — these used to read as fill. Behaviour change: the raw-read API (read_raw_data_full*), which has no fill value, now returns an error for a virtual dataset with unmapped elements instead of zeros.
  • Unlimited and printf-style mappings are supported (all 7 VDS files in the libhdf5 test set are such mappings, e.g. Eiger/Percival detector layouts). %b in a source file or dataset name is the block number and %% a literal % (other % sequences are an error, as in libhdf5); block j is read from the source named with j, probing from 0 up to the first missing source. Unlimited source/virtual selections cover as much as the source's current extent fills, including a partial last block. As libhdf5 does on H5Dget_space, the extent is recomputed from the sources present (default "last available" view, printf gap 0) — vds::virtual_dataset_extent, used by Dataset::shape() — so e.g. vds-eiger.h5 is [5, 10, 10], not its stored [20, 10, 10]. A source stored in the other byte order is byte-swapped (libhdf5 converts); other type conversions remain an error.
  • Hyperslab selection versions 1 and 2 were refused ("only version-3 hyperslab selections are supported"). Version 1 is what libhdf5 writes for every VDS created with the default format bounds (h5py's default), so those could not be read at all; version 2 is its encoding of an unlimited selection. Both are decoded now, as are irregular hyperslabs (a union of blocks, read in row-major order as libhdf5 iterates them). SerializedSelection exposes the raw form, including unlimited counts.
  • The version-1 mapping list HDF5 2.0 writes (low version bound 2.0) was misparsed: each entry's flags byte was read as the start of the source file name, and names shared with an earlier entry (stored as that entry's index) were not followed. Now decoded as H5D__virtual_load_layout does.
• clawhdf5-format reader — values returned wrong with no error:
  • Fixed Array and Extensible Array chunk indexes were laid out by the dataset's current shape instead of its max shape (23 libhdf5 test files, and any h5py file with e.g. maxshape=(10, None) or (20, 10) under libver='latest').
  • Files with 4-byte offsets: unfiltered chunked datasets read as zeros. Chunk B-tree keys store offsets in 8 bytes whatever the file's offset size.
  • A chunk's filter mask skipped the whole pipeline when any bit was set; only the flagged filters are skipped now.
  • Float data read as an integer returned the bit pattern; narrowing integer reads kept the low bits; bfloat16 was decoded as IEEE half. Floats are now decoded from their datatype fields (bf16, FP8 E4M3/E5M2, IEEE half, single and double).
  • vl_data::read_vl_bytes truncated sequences of non-byte base types.
  • A shared fill-value message read as zero fill; it is resolved now, including from the file's shared-message (SOHM) table, which could never resolve because its index version byte was skipped.
  • Two threads reading two chunked datasets through one File could get each other's chunks (the shared chunk cache was switched between datasets across separate lock acquisitions). The cache is now keyed by dataset.
  • Compound datatype version 1 members with legacy array dimensions (HDF5 before 1.4, which had no array class) were read as a single scalar at the member's offset; they are now array members, as in libhdf5 (tarrold.h5, tcompound.h5). Only reachable once layout versions 1/2 were readable, since the files that use it are that old.
• clawhdf5-format reader — errors on valid files: a version-1 shared message (a committed datatype in HDF5 1.4/1.6-era files) was read as if the object header address followed the reserved bytes; it follows a link-name offset (the reference is an old-style symbol table entry), so the reader followed the name offset and failed with InvalidObjectHeaderVersion (tcompound.h5). New shared_message::parse_shared_ref_sized takes the superblock's length size; parse_shared_ref assumes it equals the offset size.
• clawhdf5-format reader — errors on valid files: enum and bool datasets through the numeric readers; the "don't filter partial edge chunks" layout flag; Fletcher32 ahead of deflate (NetCDF-4's order). Unknown-message flags follow libhdf5 (tbogus.h5): "fail if unknown" is refused, "fail if unknown and writing" is ignored by a reader.
• clawhdf5-format reader — dense groups and attributes (links or attributes kept in a fractal heap indexed by a v2 B-tree):
  • A link heap larger than the root indirect block's direct rows (512 KiB with libhdf5's defaults: a few thousand long link names, or ~20 000 short ones) could not be listed: child indirect blocks were given the wrong number of rows, so every link stored in one was unreachable.
  • v2 B-trees of depth 3 or more (a dense group of ~22 000+ links) were misparsed: internal-node child pointers were read with widths from an estimate instead of libhdf5's per-depth record capacities, and the listing failed. The same B-tree code indexes dense attributes, shared messages and chunks.
  • Fractal-heap "huge" objects (larger than the heap's managed-object limit, 4 KiB by default — e.g. an 8 KiB dense attribute or a link with a very long name) and "tiny" objects are now read; the ID type was taken from the wrong bits (6-7, the version, instead of 4-5), so a huge object failed and took every attribute on its object down with it (NetCDF-4 files such as netcdf4-python's issue671.nc). Huge objects are found directly from the ID or through the huge-object v2 B-tree, filtered or not.
  • Heaps with an I/O filter pipeline (a group created with a filter on its creation property list compresses its link heap) are now read: the header's pipeline was skipped with the wrong size, so its checksum was looked for in the wrong place, and filtered direct blocks were read raw.
  • A user-defined link (link class 65-255, e.g. 187 in libhdf5's tall.h5/tudlink.h5) made its whole group unlistable. Such links cannot be followed without the application that registered the class, so they are now left out of datasets()/groups() and path lookup, as h5py leaves out links it cannot open; reserved link types are still an error.
• clawhdf5 — soft links are listed, as h5py lists them: datasets() and groups() on Group/MmapGroup/LazyGroup include each soft link under its own name as the kind of object it resolves to, and dataset(name) / group(name) open through it. Relative targets resolve from the group holding the link. Dangling or cyclic soft links, external links and user-defined links are left out (h5py lists their names but cannot open them). Previously soft links were missing from the listings, and in old-style (symbol table) groups a soft link made the listing fail. New group_v2::resolve_group_children / resolve_path_from and group_v1::v1_soft_links in clawhdf5-format.
• clawhdf5 — one unreadable attribute no longer fails attrs() for every attribute on its object: it is left out of the map, and the new attrs_with_errors() (on every group and dataset handle) returns the map plus one error per attribute left out. Returned values are always complete. An error in the attribute index itself (attribute info message, dense heap header or B-tree) still fails the call. clawhdf5-format gains attribute::extract_attributes_tolerant; extract_attributes_full stays strict.
• clawhdf5-format reader — files with shared object header messages (SOHM, H5Pset_shared_mesg_index): a datatype, dataspace, filter pipeline or attribute stored in the file's SOHM heap failed with "invalid shared message version: 2" — only shared fill values loaded the SOHM table — so such files' datasets and attributes could not be read. shared_message::resolve_shared_message now loads the table when a reference needs it (36 cases of the audit's read matrix).
• clawhdf5-format writer — files libhdf5 rejects or reads wrong:
  • Extensible Array (one unlimited dimension): chunks from index 244 on were written but never indexed and read as 0, by libhdf5 and by us.
  • Fixed Array: more than 1 024 chunks gave checksum errors (data blocks were never paged).
  • A finite max shape larger than the shape gave libhdf5 "addr overflow"; an unlimited dimension that is not the first scrambled the data; several unlimited dimensions ((None, None)) broke the whole file. These now write the index libhdf5 writes (swizzled Extensible Array, or a B-tree v2 index for several unlimited dimensions).
  • Header messages over 64 KiB (the size field is 16 bits) and compact datasets at 65 534–65 535 bytes produced corrupt files.
  • Reference, Opaque, BitField and Time datatypes were written as empty messages; they now encode as HDF5 2.0 does.
  • with_page_size wrote a nonexistent superblock version 4; it now writes the v3 superblock and File Space Info message libhdf5 writes.
  • FillTime values were rotated on disk (NEVER was written as ALLOC, and so on). New DatasetBuilder::with_fill_value.
  • An empty-string attribute got a zero-size datatype, which made every attribute on the object unreadable in libhdf5.
  • maxshape equal to the shape no longer forces chunked layout.
• clawhdf5-format: a truncated deflate chunk read back short, with no error. The deflate filter used flate2's streaming reader, which returns the bytes it has when the input runs out before the end-of-stream marker. It now decodes in one pass into a buffer sized to the chunk and reports a truncated stream as DecompressionError. Same fix in clawhdf5-filters, where output longer than the stated size was also silently cut off; it is now an error.

Defaults

• clawhdf5-agent: MemoryConfig::float16 defaults to true for new stores, measured rather than assumed: identical LongMemEval retrieval on real embeddings, 48% smaller files and faster checkpoints and opens at 100K. clawhdf5-cli create --f32 opts out; like --f32-index, it only ever switches the default off.
• clawhdf5-agent: MemoryConfig::quantized_index defaults to true for new stores. The reason it had been off — that int8 search was slower on ARM — did not survive measurement (see Corrections). Stores that predate the setting still load it as false, so reopening one never changes how its index is held; a store written by the v2.5.0 CLI is now a test fixture that guards exactly that, and the test fails if the load default is changed.
• clawhdf5-cli: create --f32-index opts out. create used to assign --quantized-index straight into the config, which under the new default would have forced every CLI-created store back to f32 unless the caller knew to ask; it now only ever switches the default off.

Performance

• clawhdf5-agent: consolidation's novelty scoring (each add_memory against the whole working tier) computes the new record's norm once, takes each comparison in one vectorised pass instead of three, and splits a working tier of 4 096+ records across threads — same results, tested against the old formula. It had made consolidation_efficiency stall at 100K; the complete run now takes 8 min and fills in the 100K cycle row (46.66 ms) and the memory-reduction table.
• clawhdf5-bench: consolidation_efficiency no longer prints a record-count ratio as a "BM25 Speedup" (it was never measured), nor claims cycle time grows sub-linearly (its own numbers grow slightly faster than linearly).
• clawhdf5-agent: knowledge-graph traversal was 6.5x slower than it should be. bfs_neighbors and spreading_activation built an adjacency index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K entities took 155 µs. The index is now cached on KnowledgeCache and checked against a fingerprint of the graph on each use — one pass over entity ids and relation endpoints, no allocation — so any change, including direct edits of its public Vecs, still rebuilds it (tested). BFS over 1K entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
• clawhdf5-format, clawhdf5-filters: both deflate paths hand the codec the whole chunk in one call, into a buffer allocated once, instead of streaming it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's 1 MB inflate.
• clawhdf5-accel: dot_i8 has aarch64 kernels — SDOT for CPUs with the ARMv8.2 dot-product extension (Cortex-A76 and later, Neoverse-N1, every Apple Silicon generation) and plain NEON (vmull_s8 + vpadalq_s16) for the rest, selected at runtime. SDOT is issued through inline assembly, because the vdotq_s32 intrinsic is still behind the unstable stdarch_neon_dotprod feature. On a Raspberry Pi 5 at N = 100 000 and equal recall, the quantised index answers 1.18x the queries per second of f32 (7 267 vs 6 164) and builds 2.3x faster (14 464 vs 33 413 ms). Both kernels are tested bit-for-bit against scalar on real hardware, each explicitly — dispatch only ever takes one path on a given CPU, so testing through it alone would have left the plain-NEON fallback unexercised on any machine with SDOT.

Corrections

• The v2.7.0 entry for dot_i8 said quantized_index stayed off by default because "aarch64 falls back to the scalar loop", implying the ~13% search penalty measured on x86 applied on ARM too. It did not. That figure came from scalar int8 against hand-written AVX2 f32 kernels on x86, whose portable baseline is SSE2; on aarch64 NEON is the baseline, and measured on a Pi 5 the scalar int8 loop already matched f32 for search while building 1.76x faster. The claim was extrapolated rather than measured.

v2.7.0 (2026-09-20)

Upgrade Notes

• Two read-path bugs fixed, one of them silent. Datasets indexed by an Extensible Array (any dataset with one unlimited dimension) returned data from the wrong chunks past their first few dozen. If you have readings taken from such a dataset with an earlier release, they may be wrong; re-read them.
• A corrupt chunk index is now an error. Fixed and Extensible Array structures carry checksums that were previously ignored, so damage surfaced as plausible data from the wrong offset. Code that read a damaged file and got numbers will now get ChecksumMismatch instead. That is the point.
• Breaking: MemoryConfig gained hnsw_m, hnsw_ef_construction and hnsw_ef_search, so literal constructions need updating; ..Default::default() does not. All three default to the previous behaviour.

Correctness

• clawhdf5-format: datasets indexed by an Extensible Array returned wrong data beyond their first few dozen chunks. One unlimited dimension gives a dataset an Extensible Array chunk index, whose first elements (4 by default) sit inline in the index block and whose rest live in data blocks sized by a formula the reader got wrong. In the default layout everything through the 36th chunk happened to line up and the 37th onwards did not: a 400-chunk dataset silently returned wrong values from chunk 37, and datasets past about a thousand chunks failed outright with "invalid Extensible Array data block signature". Reads were wrong, not merely refused — the caller got plausible numbers from the wrong chunks. Four separate layout errors, each checked against files written by HDF5 2.0 and against the library source:
  • the number of data blocks in super block u is 2^(u/2), not 2^u;
  • each holds 2^((u+1)/2) * data_blk_min_elmts elements, which doubles every other level rather than every level;
  • a super block carries a block-offset field before its data block addresses, which was not skipped;
  • the page-init bitmap belongs to the super block, one bit per page packed across all its data blocks (MSB first), and was being read from inside the data block instead; a paged data block also ends its prefix with a checksum before the first page. Covered now by interop tests at 4, 37, 400, 5 000 and 200 000 chunks (the last large enough for paged data blocks), plus sparse, gzip-filtered and 2-D cases. Writing is unaffected; this is a read-path bug.
• clawhdf5-format: the sibling Fixed Array index (fixed dimensions written with libver='latest') was checked against the same range and is correct, including paged data blocks and sparse datasets — it really does keep its page-init bitmap in the data block, where the Extensible Array does not. It had no real-file coverage above the inline sizes either, so it now has the same tests.

Security

• clawhdf5-format: a crafted file could crash any reader through B-tree v2 traversal. Recursion was bounded only by the depth the file claimed (a u16), and child addresses were never checked for sharing. A node listing itself as its own child under a header claiming 65 535 levels — under 100 bytes — overflowed the stack and aborted the process (SIGABRT, not a catchable error). Levels whose children all point at one shared node below reached it fan-out^depth times: 29.5 million records from ~5 KB, and one more level would exhaust memory. Both are now errors, returned in under a millisecond: depth is capped at 64 (as the fractal heap already was), and traversal stops once it has produced more records than the file has bytes to hold. Every B-tree v2 user goes through this path — dense attributes, v2 groups, shared messages and chunk indexes. Valid files are unaffected, including a depth-2 HDF5 2.0 chunk index with 40 000 records, now covered by an interop test.

Integrity

• clawhdf5-format: Fixed and Extensible Array chunk indexes now verify their checksums (the checksum feature, on by default). Every structure in both — header, index block, super block, data block and each data block page — carries a Jenkins lookup3 checksum that was parsed past and ignored. The consequence of skipping it is not a missing warning but wrong data: a single flipped bit in a chunk address still parses, still points inside the file, and the reader hands back whatever bytes now sit there as the chunk's contents. Verified in both directions — the checksums accept files written by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged, and an interop test corrupts an address to confirm the read now fails instead of returning data (it does return data when the check is removed).

Performance

• clawhdf5-agent: opening a store is ~28% faster (455 ms -> 327 ms at 100k x 384). read_from_disk memory-mapped the file and then copied the entire mapping into a Vec for File::from_bytes, when File::open memory-maps it directly — so every open paid a full-file memcpy for nothing. Process peak memory is unchanged: the peak falls after the parse, during the index build, so the transient never reached the high-water mark. The footprint harness now reports that peak next to the retained figure, which is how this was checked rather than assumed.
• clawhdf5-accel: dot_i8, a runtime-dispatched int8 dot product (AVX2: sign-extend each half to i16, then madd_epi16; scalar fallback elsewhere). The quantised HNSW index used a scalar loop while the f32 path it was measured against ran AVX2, so the ~13% throughput cost recorded for MemoryConfig::quantized_index was a missing kernel rather than a property of int8. With the kernel, at N = 100 000 x 384 and equal recall, the quantised index answers 1.63x as many queries per second (21 848 vs 13 399 at ef=64, recall 0.9940 vs 0.9945) and builds 1.8x faster (1778 vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three alternating runs. It remains off by default only because the kernel is AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so the SIMD path is tested to agree with scalar bit for bit.

Tuning

• clawhdf5-agent: the HNSW parameters are configurable — MemoryConfig::hnsw_m, hnsw_ef_construction and hnsw_ef_search (defaults 16, 64, and 0 meaning "scale with k", i.e. today's behaviour). They were constants, so a deployment could not trade recall against memory or query speed at all. All three are persisted with the store. Values are clamped where the index requires it: clawhdf5-ann asserts a graph degree of at least 2, so a configured 0 — from a file, or from a caller who took 0 to mean "default" — used to abort the process inside the builder. Lowering ef_search also no longer narrows the candidate pool that fusion sees. Breaking: MemoryConfig gained fields, so literal constructions need updating; ..Default::default() does not.

Documentation

• clawhdf5-agent: BM25Index::search claimed to use Block-Max WAND for early termination. It never did; it scores every match exhaustively. It now says so, and why no pruning would help the store: hybrid_search uses scores(), since fusion normalises over every match.

v2.6.0 (2026-09-20)

Upgrade Notes

• Re-ranked results change, substantially for the better. RerankInput and ReRankConfig gained fields (relevance, relevance_weight), so literal constructions need updating; ..Default::default() does not. Any caller that re-ranked was previously getting results ordered by age with the retrieval score discarded — see below.
• Breaking: MemoryCache::embeddings is a cache::Embeddings rather than a Vec<Vec<f32>> (indexing still yields a &[f32] row); embeddings_flat is gone, replaced by flat_embeddings(); rebuild_flat() is a deprecated no-op.
• MemoryConfig gained quantized_index (default false, so behaviour is unchanged unless you opt in); literal constructions need the field.

Retrieval quality

• clawhdf5-agent: re-ranking discarded the retrieval score. reranker::rerank built its combined score from temporal decay, source authority and Hebbian activation only — RerankInput had no relevance field — so re-ranking a candidate pool reordered it by age and threw the retriever's ordering away. The OpenClaw backend re-ranked every search, so this was its shipping behaviour: measured over the full LongMemEval haystack it cost 40.6pp of Hit@1 (11.0% vs 51.6%) and two thirds of MRR (0.183 vs 0.643). RerankInput::relevance and ReRankConfig::relevance_weight (1.0 by default) fix it: relevance leads and the metadata signals break near-ties, which restores retrieval (Hit@1 +0.4pp vs no re-ranking) and improves recency discrimination by 6–7pp. Breaking: RerankInput and ReRankConfig gained fields, so literal constructions need updating; ..Default::default() does not.
• clawhdf5-bench: the LongMemEval harness feeds the dataset's real session dates to the store instead of a synthetic counter (decay needs true intervals, not just the right order), and reports newest_gold_first — on a knowledge-update question, did the newest gold session outrank the stale one it supersedes? Plain recall cannot see this, because both are labelled gold. New --rerank-sweep.

Memory

• clawhdf5-agent: MemoryConfig::quantized_index stores the vector index's own copy of the embeddings as i8 rather than f32, which at 100k 384-dim entries takes the index from 266 to 123 MiB and the whole reopened store from 399 to 256 MiB (2.72x -> 1.74x the raw vectors). Quantised distances are approximate and ef cannot compensate — recall@10 tops out at 0.967 against f32's 0.9995 — so the query path re-scores the candidate pool against the exact embeddings the store already holds, which restores recall (0.9940 vs 0.9945 at ef=64) for about 13% of QPS. Off by default: it trades query speed for memory, and which side is worth more depends on the deployment. The setting is persisted, so a reopened store does not silently revert to four times the index memory.
• clawhdf5-ann: Storage::Int8 and the build_with / new_with / from_graph_bytes_with constructors that select it. The scale is per row, not global — a fixed [-1, 1] scale spends fewer than 12 of the 255 levels on a unit-length 128-dim vector and is unusable (0.35 top-10 overlap against an exact ranking, versus 0.99 per row). compact() keeps the storage it was given; serialized indexes still carry f32 vectors, so a quantised index is rebuilt rather than loaded.
• clawhdf5-agent: a loaded store holds ~30% less memory (100k 384-dim entries: 505 -> 357 MiB, 3.44x -> 2.43x the raw vectors). The cache kept every embedding twice — a Vec<Vec<f32>> and a flattened copy for the batched kernels, maintained in lock-step — so it now stores only the flat buffer and indexes into it. Recall and query latency are unchanged. Breaking: MemoryCache::embeddings is a cache::Embeddings rather than a Vec<Vec<f32>> (indexing still yields a &[f32] row); embeddings_flat is gone, replaced by flat_embeddings(); rebuild_flat() is a deprecated no-op. Rows are now always exactly dim long — shorter ones are zero-padded — which makes the ragged-row case that used to silently misalign the flattened copy unrepresentable.
• clawhdf5-bench: search_harness --footprint reports live heap use per stage, measured with a counting allocator (RSS cannot see a structure freed into the allocator's own pool).

Testing

• The Python interop suites honour CLAWHDF5_PYTHON, and ci-test.sh picks up a .venv/bin/python automatically. On a PEP 668 "externally managed" system h5py cannot be installed into the system interpreter at all, so every interop suite — the h5py writer round-trips, the facade, netCDF4 and the reference files — was skipping silently. A silent skip here is exactly how the v5 compound-datatype bug reached a release. CLAWHDF5_REQUIRE_INTEROP=1 still turns a skip into a failure.

v2.5.0 (2026-09-19)

Upgrade Notes

• Retrieval rankings change, for the better. The default fusion weights move from 0.7/0.3 to 0.4/0.6 (hybrid::DEFAULT_FUSION), measured over the full LongMemEval haystack: turn-level Hit@1 51.6% vs 44.2%, MRR 0.643 vs 0.586. unified_search and the OpenClaw backend pick this up automatically; callers passing weights to hybrid_search explicitly are unaffected.
• Out-of-range selections are now errors. read_*_selection used to return data for a selection that ran past a dataset edge — a hyperslab came back zero-padded, and a point with an out-of-range coordinate wrapped into the next row. Both are now FormatError::SelectionOutOfBounds. Code relying on the old (wrong) values will start seeing errors.
• Large compressed datasets written without explicit chunk dimensions get a different layout. They used to be stored as one chunk; they are now split to ~1 MiB chunks. The files stay standard and h5py-readable, and explicit with_chunks is unaffected.
• rayon is now a default dependency of clawhdf5-agent (the parallel index build). Opt out with --no-default-features --features float16,hnsw.
• clawhdf5-ann search results no longer shrink when records near the query have been deleted, so a search that previously returned fewer than k results now returns k.

Retrieval quality

• clawhdf5-agent: optional keyword stemming — bm25::TokenFilter::Stemmed and HDF5Memory::set_token_filter, so "training" and "trains" match. Off by default, on measurement rather than principle: over the full LongMemEval haystack it buys depth and costs the top rank (BM25 alone: Hit@5 +2.8pp, Hit@10 +2.4pp, Hit@1 −1.8pp, MRR unchanged), and on the shipping hybrid configuration the trade is narrower still. See BENCHMARKS.md.
• clawhdf5-agent: QueryExpander::expand panicked on ordinary non-ASCII input — "İ AI" was enough. It searched a lowercased copy of the query and then sliced the original with those offsets, which only works while lowercasing preserves byte length (Turkish İ is 2 bytes and lowercases to 3). Depending on where the offsets drifted it either corrupted the output ("İstanbul AI trip" lost a character) or panicked. Matching now walks the original string.
• clawhdf5-agent: query expansion no longer rewrites text inside words. replace_word_case_insensitive did a plain substring replace despite its name, so "training" became "trArtificial Intelligencening" and "programming" became "Pull Requestogramming" — every acronym expansion of ordinary prose was corrupt. Matches now require word boundaries; genuine acronyms (API, database) still expand.
• clawhdf5-agent: the default fusion weights are now the measured ones. A sweep of every 0.1 step over the full LongMemEval haystack (500 questions, real MiniLM embeddings) shows the long-standing 0.7/0.3 default is strictly dominated by 0.4/0.6 — turn-level Hit@1 51.6% vs 44.2%, Hit@5 81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.643 vs 0.586, and better at session level too. The finding was recorded in BENCHMARKS.md but had never been applied: unified_search and the OpenClaw backend both hardcoded 0.7/0.3. They now use hybrid::DEFAULT_FUSION. Callers passing weights to hybrid_search explicitly are unaffected — pass 0.4/0.6 (or use hybrid_search_with) to get the tuned behaviour.
• clawhdf5-agent: fusion is now selectable. New hybrid::Fusion (Weighted { vector, keyword } or Rrf { k }), hybrid::fuse, hybrid::hybrid_search_fused and HDF5Memory::hybrid_search_with. Reciprocal rank fusion existed but was unreachable from the store, so it had never been measured against the weighted sum; the LongMemEval bench now has an RRF mode.

HDF5 Read Path

• Selection reads cost what the selection costs. read_*_selection decoded the entire dataset and then picked elements out, so a 64 x 64 window of a 64 MB compressed dataset took 105 ms - about as long as reading all of it. Now only the rows (contiguous) or chunks that overlap the selection's bounding box are read and decompressed: that window takes 0.39 ms, one row 2.7 ms, one column 5.2 ms. Results are identical to the full-read path (equivalence-tested over random hyperslabs and point lists, ranks 1-3, contiguous / chunked / deflate). New read_harness bench binary.
• Faster full reads (same-moment A/B, 64 MB f64): chunked + deflate 110 -> 69 ms, chunked 72 -> 60 ms, contiguous 56 -> 30 ms. The facade's cached read path now decompresses cache misses in parallel batches (it was sequential; only the uncached reader was parallel) and caches only datasets that fit the chunk cache; unfiltered chunks are copied straight from the file bytes; a contiguous dataset is converted straight from the file bytes; and the native-endian conversions no longer zero a buffer before overwriting it.
• Datasets indexed by a version-2 B-tree now read (layout v4, chunk index type 5 — what libver='latest' uses for two or more unlimited dimensions; previously "unsupported chunked layout"). The four copies of the chunk-index dispatch are now one shared function, so every read path gets it.
• H5T_STD_REF references (HDF5 1.12+, datatype message version 4) parse: ReferenceType gains Object2, DatasetRegion2 and Attribute, and read_object_references decodes the new object references. Previously any dataset of this type failed with InvalidReferenceType(2). Tested against a file written by HDF5 2.0 itself (fixture + generator script committed).
• Automatic chunk sizes. Asking for compression (or any filter) without with_chunks used to store the whole dataset as one chunk, so any read had to decompress everything and nothing could be decoded in parallel. Datasets up to 1 MiB stay a single chunk, as before; larger ones are split by halving the dimensions in turn until a chunk is at most 1 MiB (the approach h5py takes). Behaviour change: large compressed datasets written without explicit chunk dimensions get a different (standard, h5py-readable) layout. Explicit with_chunks is unaffected.
• Out-of-range selections are errors. They used to return data: a hyperslab past an edge came back padded with zeros, and a point whose column was out of range wrapped into the next row and returned that element. Now FormatError::SelectionOutOfBounds (also for a rank mismatch or overlapping blocks).

Search

• clawhdf5-ann: faster index builds. Back-link pruning is 90% of a build's distance evaluations; the bulk build now inserts in batches and prunes each overflowing neighbour list once per batch (10K: 1676 -> 1074 ms). With the parallel feature, planning and pruning run on a thread pool (10K: 388 ms, 100K: ~21 s -> 5.9 s on 16 cores). The graph is deterministic and identical with or without the feature. clawhdf5-agent's parallel feature enables it for the agent's index and is now on by default (adds rayon to the default dependency set; build with --no-default-features --features float16,hnsw to opt out).
• clawhdf5-ann: HnswIndex::search returned fewer than k results — often none — when the records nearest the query had been deleted: it collected ef candidates, then dropped the deleted ones, then took k. Deleted nodes are now traversed as waypoints but never occupy a result slot, so a search returns the k nearest live records. Matters for any store that deletes or supersedes memories without compacting straight away.

v2.4.0 (2026-09-19)

Upgrade Notes

• Search results improve on upgrade. The HNSW index now reaches true neighbours it previously could not (recall@10 0.31 -> 0.98 at 100K records on clustered data), so hybrid_search rankings change for the better. The agent rebuilds its index from the store automatically; a standalone HnswIndex persisted with to_hdf5_bytes keeps its old graph until rebuilt.
• hybrid_search no longer writes the store. Hebbian activation boosts are persisted by the next checkpoint (any flushing write, flush_wal, or when the HDF5Memory is dropped) instead of inside every query; a crash before then forgets only the boosts since the last checkpoint. Activation weights are now capped at 16.
• A new sidecar file, <store>.h5.ann, holds the vector index graph. It is derived data: safe to delete (the index is rebuilt), copied by snapshot(), and worth including when copying a store by hand to avoid a rebuild.
• BM25Index no longer caches IDF and gained add_document, remove_document, pad_to, scores, len and is_empty; results are now deterministic (ties break by record id).

Search

• clawhdf5-ann: HNSW recall fix. Neighbours were chosen as the plain closest-M, which on clustered data (what embeddings look like) turns each cluster into an island: recall@10 was 0.87 / 0.67 / 0.31 at 1K / 10K / 100K vectors and did not improve with ef. The index now uses the HNSW paper's diversity heuristic (Algorithm 4 with kept pruned connections) when linking a new node and when pruning back-links: recall@10 at ef = 64 is 1.00 / 1.00 / 0.98 and responds to ef. Builds are slower (~3.5x at 10K). Existing persisted indexes keep their old graph until rebuilt; the agent rebuilds its index from the cache, so stores pick this up automatically.
• clawhdf5-agent: hybrid_search is 23-39x faster in steady state (p50 5.5 -> 0.24 ms at 1K records, 49 -> 2.1 ms at 10K, 884 -> 23 ms at 100K). Every query used to rebuild the BM25 index from scratch and rewrite the whole .h5 file. The keyword index now lives for the life of the store and is updated incrementally (add / remove / in-place update, exactly equivalent to a fresh build - property-tested), and a query no longer writes the store. Behaviour change: Hebbian activation boosts are persisted by the next checkpoint (any flushing write, flush_wal, or drop) rather than immediately; a crash in between forgets only the boosts since the last checkpoint. Activation weights are now capped (16.0) - they grew without bound.
• clawhdf5-agent: the vector index is persisted, so open() no longer rebuilds it on the first search (first query after open: 2627 -> 15 ms at 10K records, 36 s -> 159 ms at 100K). The HNSW graph — not the vectors, which the store already holds — is written to <store>.h5.ann at each checkpoint and tied to it by a generation id in /meta; a missing, stale, damaged or structurally invalid sidecar is ignored and the index rebuilt. Records replayed from the WAL join the loaded index incrementally; a replayed update or delete invalidates it. snapshot() copies it. Batch saves no longer force a full index rebuild.
• clawhdf5-ann: faster HNSW build and search with identical recall. The cosine metric stores unit vectors and compares them with a plain dot product (it re-derived both norms on every distance evaluation), and the per-call HashSet of visited nodes is a reusable epoch-stamped array. Build 2.75 -> 1.89 s at 10K and ~38 -> 21 s at 100K; QPS at ef = 64 22.7K -> 39K at 10K. Distances returned by search are unchanged (1 - cosine). Indexes loaded from older HDF5 files are normalised on load.
• clawhdf5-accel: the SIMD backend is detected once per process instead of on every kernel call.
• clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes — graph-only serialization (checksummed, every neighbour id and level validated on load).
• clawhdf5-agent: a further 4-5x on hybrid_search with identical rankings (p50 now 0.07 / 0.49 / 4.65 ms at 1K / 10K / 100K — 79x / 100x / 190x faster than v2.3.0). Fusion needs every keyword score but not their ranking: new BM25Index::scores returns them unsorted from a dense accumulator (it hashed every posting, then sorted every match), and merge_vector_keyword selects its top k instead of sorting every candidate. Capping the keyword candidate pool was measured and rejected: it changes the top-10 for most queries (search_harness --fusion-study).
• clawhdf5-agent: BM25 results are deterministic (ties break by record id), top-k uses a bounded heap, and the "WAND early termination" that computed a bound and then ignored it is gone. IDF is computed per query.
• clawhdf5-bench: new search_harness binary — HNSW recall@10 / QPS / latency per ef against an exact scan, and end-to-end hybrid_search timings, on deterministic clustered (or --uniform) data. Baseline in BENCHMARKS.md.

v2.3.0 (2026-09-19)

Upgrade Notes

• A memory store now has a single writer. HDF5Memory::create/open take an exclusive lock (<store>.h5.lock); a second open of the same store — in the same or another process — returns MemoryError::Locked. Code that opened a second handle just to read should use HDF5Memory::open_read_only.
• Unsigned array attributes arrive as AttrValue::U64Array, not I64Array, and attrs() may now return AttrValue::Raw. Exhaustive matches on AttrValue need the two new arms.
• WAL header version 3 → 4. v3 files are read and upgraded in place, but a store written by 2.3.0 with a pending WAL cannot be opened by 2.2.0 or earlier (it is refused, not corrupted). Checkpoint first (flush_wal) if you need to downgrade.
• MemoryConfig::compression now uses deflate unless the agent's new zstd feature is enabled; it previously failed outright in a default build.
• MemoryError gained Locked; FormatError gained UnresolvedSharedMessage, ExternalDataFilesUnsupported and ExternalLinkUnsupported; MessageType gained ExternalDataFiles.

Bug Fixes

• clawhdf5-format: compound datatypes written with default libver bounds (datatype message version 1 — what plain h5py.File(path, 'w') produces) were mis-parsed. The v1 member layout carries 28 bytes of legacy array fields after the byte offset (the parser skipped 24), and v2 pads member names to 8 bytes and has no array fields at all (the parser did neither), so every member after the first byte offset was read from the wrong position — typically surfacing as Overflow("compound member ...") on read. Found by adding a default-libver axis to the h5py interop tests; byte-level regression tests for v1 and v2 added.
• clawhdf5-gpu: gpu_tests could hang forever under the default parallel test runner — every test created its own wgpu instance and device at once. Tests now serialise GPU access, and GPU→CPU readback waits are bounded (30 s) so a wedged driver returns GpuError::BufferMap instead of blocking.
• clawhdf5-agent: benches/bench.rs and benches/memory_bench.rs no longer compiled against the current strategy/consolidation APIs.

HDF5 Compatibility

• clawhdf5-format/clawhdf5: datasets and attributes that use a committed (named) datatype now read correctly. They store a shared-message reference; the facade parsed the reference bytes as the datatype (Time { size: 0 }, unreadable data) and silently dropped such attributes. The shared-reference parser itself was wrong for real files: version 2 has no reserved bytes, and the version 3 types were inverted (1 = SOHM heap, 2 = committed).
• Fill values are applied on read. There was no Fill Value message parser: the holes of a sparse chunked dataset read as zeros even when the fill value was not zero (silently wrong data), and a dataset that was created but never written failed with NoDataAllocated where h5py returns a filled array. Messages v1–v3 and the old 0x0004 form are parsed; the fill value is written into exactly the chunk-grid cells missing from the chunk index.
• Soft links are followed during path resolution, in old- and new-style groups (absolute/relative targets, links to groups, links through links), with a depth limit so a link cycle is an error rather than a hang. A dangling link reports the target it could not find.
• Things the reader does not follow are now explicit errors instead of wrong answers: an external link is ExternalLinkUnsupported { filename, object_path } (was PathNotFound), and a dataset whose raw data lives in external files (message 0x0007, now a known MessageType) is ExternalDataFilesUnsupported (it would otherwise read as fill values).
• attrs() no longer drops attributes. Any attribute whose datatype had no AttrValue variant was omitted with no error — including every Python bool (h5py stores attrs["flag"] = True as an enum), complex numbers, compound values and object references. Now:
  • numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1) decode as I64 / I64Array of 0/1;
  • new AttrValue::U64Array keeps unsigned arrays unsigned (they were cast to I64Array, so values above i64::MAX came back negative). Behaviour change: code matching I64Array for an unsigned attribute must also match U64Array (the netCDF-4 CF helpers and Python bindings do);
  • new AttrValue::Raw { datatype, shape, data } carries everything else verbatim, decodable with clawhdf5_format::data_read against datatype. Both new variants are writable, so an attribute can be copied between files unchanged. Python receives Raw as {"dtype", "shape", "data"}.
• All of the above are covered by h5py interop tests under both default and libver='latest' bounds, compared against h5py's own readback.

Security

• clawhdf5: virtual-dataset source file names are untrusted input but were joined straight onto the opened file's directory, so a crafted file could make the reader open any path the process can reach (absolute path, or .. components). Only plain relative paths inside that directory are accepted.

Durability & Integrity

• clawhdf5-agent: a crash between writing a checkpoint and truncating the WAL no longer duplicates every pending entry on the next open. Each checkpoint records a WalMark (byte length + chained CRC of the WAL prefix it folded in) in /meta; open() skips exactly that prefix when it is still present. No WAL format change for this; older files behave as before.
• clawhdf5-agent: checkpoints and snapshots are durable as a unit — the temp file is synced before the rename and the directory after it. Individual WAL appends remain unsynced by design (documented in CLAUDE.md).
• clawhdf5-agent: save_or_update hits are logged as a new Update WAL record, so replay updates in place instead of appending a duplicate. WAL header version 3 → 4 (so older builds refuse the file rather than truncating a record they can't parse); v3 files are read and upgraded in place.
• clawhdf5-agent: loading validates every per-record dataset length (a truncated store is now MemoryError::Schema, not a later panic), fixes the n.len() == n.len() tautology that trusted a norms dataset of any length, and rejects embedding_dim == 0 with records present.
• clawhdf5-agent: eight behavioural MemoryConfig fields are now persisted in /meta. Previously they reset to defaults on every open — a compressed store was rewritten uncompressed, wal_enabled = false flipped back to true.
• clawhdf5-agent: compression = true never worked in a default build (it requested Zstd without enabling the feature, so every checkpoint failed with unsupported filter: 32015). Default builds now use deflate; Zstd is the new opt-in zstd feature.
• clawhdf5-agent: single-writer lock (<store>.h5.lock, MemoryError::Locked) — two handles on one store used to silently destroy each other's data. New HDF5Memory::open_read_only gives a lock-free, never-writing view; the CLI's read-only subcommands use it.
• clawhdf5-agent: an unreadable WAL (torn header / bad magic) is quarantined (HDF5Memory::quarantined_wal()) instead of blocking open() of a healthy store. A WAL from an unknown newer version still fails and is left intact.
• clawhdf5-agent: provenance records are renumbered on compaction (they weren't, so every later save_or_update raised a false High integrity alert); pending anomaly alerts and tracked sessions are bounded; snapshot() includes entries still in the WAL.
• clawhdf5-agent: hybrid ranking is deterministic (index tie-breaks instead of HashMap order); a set of identical positive scores — including a single candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer reinforces zero-score filler results.
• clawhdf5-format: chunked/VDS/hyperslab reads size their buffers with overflow-checked arithmetic and fallible allocation, so crafted dimensions are FormatError::Overflow instead of a wrapped size or a process abort; parallel_read bounds checks use checked_add.
• clawhdf5: a malformed filter-pipeline message is an error instead of being treated as "no filters" (which returned compressed bytes as data); FileBuilder::write is atomic and synced instead of truncating the destination first.

CI / Testing

• CI now lints every target (cargo clippy --all-targets) plus clawhdf5-format's optional features, compiles all benches, and tests the format feature matrix. Previously test/bench code and feature-gated modules were never linted; the accumulated clippy backlog is fixed.
• CI installs python3 + h5py/numpy/netCDF4/xarray and sets CLAWHDF5_REQUIRE_INTEROP=1, which turns a missing interop dependency into a test failure. Until now every h5py/netCDF4 interop test silently skipped in CI, which is how the HDF5 2.0 compound bug fixed in v2.2.0 reached a user. The #[ignore]d writer_h5py_tests suite is run explicitly.
• h5py-generated-file tests now cover default libver bounds as well as libver='latest' (HDF5 2.0 raised the default low bound to 1.8).
• clawhdf5-agent: WAL property tests (round trip; after any corruption the entries read back are an exact prefix of what was written — 1500 seeded cases), a crash-recovery matrix (an on-disk image after every operation, the checkpoint window, and the WAL torn at every byte length, each reopened and checked against a model), and a WAL fuzz target.
• Optional fuzz smoke run (CLAWHDF5_FUZZ_SECONDS=N scripts/ci-test.sh); new datatype corpus seeds for v1 compound and native complex messages.

v2.2.0 (2026-09-18)

Security

• clawhdf5-format: bounded decompression output (MAX_DECOMPRESS_SIZE) for deflate/lz4/zstd/pcodec so a crafted compressed chunk can't drive an unbounded allocation (memory-exhaustion DoS).
• clawhdf5-format: chunked_read.rs/data_read.rs/local_heap.rs bounds audit — added ensure_len overflow guards at every plain-arithmetic offset+size check, a recursion-depth guard against a crafted self-referencing/cyclic B-tree chunk index, a fix for an unguarded compound-datatype byte_offset overrun in read_compound_fields, and an ndims - 1 underflow guard for degenerate zero-dimension chunked layouts. Added a new fuzz_dataset_read cargo-fuzz target (walks every dataset in a parsed file and exercises the contiguous/chunked/compact raw-data read paths) which found and fixed 3 real crash bugs — an integer-multiply overflow in copy_chunk_to_output's N-D assembly path, the ndims - 1 underflow above, and an overflow in local_heap.rs — within the first few fuzzing runs.
• clawhdf5-format: btree_v1.rs overflow-safe bounds checks via a local ensure_len helper, closing a usize-overflow panic reachable from a crafted near-usize::MAX B-tree offset.
• clawhdf5-agent: WAL length-prefix caps (MAX_WAL_FIELD_LEN, 64 MiB) reject a corrupted/truncated length claim before allocating. Followed by a full per-entry CRC32 trailer (WAL_VERSION bumped to 2) — a bit-flip inside an entry now stops replay cleanly instead of silently accepting corrupted data. Old-format WAL files are still read correctly and migrated to the new format on next open.
• clawhdf5-android: validate embedding_len/query_embedding_len against the handle's configured embedding_dim (and reject null pointers) before constructing a slice from a raw pointer in edgehdf5_save / edgehdf5_hybrid_search.
• clawhdf5-py: bump pyo3/numpy 0.28 → 0.29, clearing two RUSTSEC advisories (OOB read in PyList/PyTuple iterator; missing Sync bound on PyCFunction::new_closure).
• Clarified that the integrity hashes in clawhdf5-agent::provenance (FNV-1a) and clawhdf5-format::provenance (SHA-256) are unkeyed and detect only accidental corruption, not tampering — doc-only change, no behavior change.

Performance

• clawhdf5-format: chunk cache lookup is now O(1) (slot_index: HashMap) instead of a linear scan, and cache hits return a shared Arc instead of cloning the decompressed buffer — the hottest path in chunked reads.
• clawhdf5-ann: optional parallel feature (rayon) parallelizes HNSW's prune_connections neighbor-distance computation. The outer build/insert loop is deliberately left sequential — it has genuine cross-iteration data dependencies and needs its own correctness-focused design pass.
• clawhdf5-format/chunked_read.rs: removed 12 unnecessary chunk_dimensions[..rank].to_vec() allocations where callees already accept &[u32].

Architecture

• Added .gitea/workflows/ci.yml, actually wiring the long-existing scripts/ci-test.sh (fmt, clippy, tests, no_std check) into CI on every push/PR to main. Fixed stale package names in ci-test.sh/ check-nostd.sh that had been silently no-op'ing the clawhdf5-py exclusion and the no_std check.
• Fixed a genuine no_std build break in clawhdf5-format (uncovered once the no_std CI check actually started running): core::sync::atomic::AtomicU64 doesn't exist on thumbv7em-none-eabihf (switched to portable-atomic), missing alloc imports for Box/Vec/format! on a few no_std paths, and f64::powi (std/libm-only) replaced with a local exponentiation-by-squaring helper in the scale-offset filter.
• Added [workspace.dependencies] for tempfile/criterion/half/serde, fixing a real version skew on half (2 vs 2.7 across crates).
• Fixed version skew: clawhdf5-py (pyproject.toml) and packages/clawhdf5-node (package.json) were both behind the actual crate version (2.1.0).
• Documented that the mpi-io feature's read/write paths are root-read +broadcast / gather-to-rank-0, not true collective I/O.

Documentation

• BENCHMARKS.md: re-ran the previously-undated "LongMemEval Results", "SIMD & Parallelism", and "Vector Search Latency"/"Comparison to MemX" sections on a second machine (tank, Ryzen 7 7800X3D) with explicit dates and reproduce commands. Found and corrected a methodology issue in the SIMD/Parallelism benchmark selection (several originally-compared benchmarks didn't actually isolate the scalar/SIMD/parallel axis).
• README.md / ROADMAP.md / CLAUDE.md: corrected several stale facts — the clawhdf5-types crate (removed earlier) was still listed in the README crate map; the LongMemEval numbers in the README badge and table didn't match the actual (much better) benchmark results in BENCHMARKS.md; total line-of-code and test-count figures were stale; clawhdf5-gpu's CubeCL→wgpu correction; documented the new clawhdf5-ann parallel feature flag, which had no entry in the Feature Flags table.

New Features

• clawhdf5-migrate: substantial engine improvements:
  • Real content validation — the post-migration check now reads the written HDF5 back and compares actual content (chunk text, embeddings, and every session/entity/relation field) against the source, not just row counts. A representative sample of chunk rows is verified by default; --validate-full checks every row. A corrupt migration that preserves counts no longer passes.
  • Configurable schema — table names are no longer hardcoded; queries are built from a SchemaConfig (table + ordered column names, defaulting to the ZeroClaw layout) with --chunks-table / --sessions-table / --entities-table / --relations-table overrides.
  • Streaming count pass — --dry-run now does a COUNT(*)-only pass per table instead of loading every row into memory.
  • Incremental migration — --incremental reads the existing output, reads only source chunks newer than the last migrated id, and appends them (refreshing the metadata groups), instead of re-migrating everything.
• clawhdf5-format: read IEEE-754 half-precision (f16) floats. read_as_f32 / read_as_f64 previously only handled 4- and 8-byte floats; 2-byte floats (e.g. float16-stored embeddings) now decode via a no_std-safe bit conversion.
• clawhdf5-format: write multi-block fractal heaps (root indirect block). Dense attribute and dense link storage previously capped at a single direct block (~64 KiB of heap data — a few thousand attributes/links). When the objects exceed one direct block, the heap now lays out a root indirect block (FHIB) over multiple direct blocks sized by the doubling table, distributing objects across blocks with correct per-block heap offsets. Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip through our reader and are read correctly by h5py. (Objects still may not span a block — no huge-object path.)
• clawhdf5-format: write dense group link storage (fractal heap + v2 B-tree). A group with more than 8 links (libhdf5's compact max_compact default) is now written densely — its links live in a fractal heap indexed by a v2 B-tree of type 5 (link-name index) referenced from the group's LinkInfo message — instead of as inline Link messages. This matches libhdf5's compact→dense switchover and keeps large groups out of the object header. Reverse-engineered against libhdf5: link heaps use heap_id_length 7 / max_heap_size 32 (vs 8 / 40 for attributes). The shared single-direct-block fractal-heap builder is now parameterized and used by both dense attributes and dense links. Validated end-to-end: our reader round-trips, and h5py reads the dense groups we write. (Single direct block — up to ~a couple thousand links per group; beyond that needs indirect blocks, still unsupported.)

Robustness

• clawhdf5-format: harden the readers added this cycle against malformed / hostile input — they parse untrusted bytes and must return errors, never panic, OOM, or recurse without bound. Fixed concrete vectors found by audit and locked in with adversarial tests:
  • Paged Fixed Array: 1 << max_nelmts_bits shift overflow (a u8 ≥ 64); element/page offset multiplications now checked; element count bounded by file size.
  • H5S selection decoder: ALL/NONE no longer claim 16 bytes they don't have; hyperslab rank capped at 32 (H5S_MAX_RANK) to stop a giant allocation; iter_linear coordinate/stride/product arithmetic is checked.
  • VDS mapping parser: no pre-allocation from the untrusted nused; all selection slicing is bounds-checked.
  • scale-offset / N-Bit filters: 1 << minbits overflow at minbits == 64; N-Bit bit_offset + precision overflow; N-Bit type-tree recursion depth capped (no stack overflow from a crafted nested tree); element counts bounded by the chunk's expected decompressed size so a bogus count can't drive a huge allocation.
  • Virtual Dataset assembly: a virtual dataset whose source is itself virtual (a cycle) now errors instead of recursing into a stack overflow.

New Features

• clawhdf5-agent: compress fixed-length string datasets (memory text chunks, session summaries, ids, tags, entity/relation names, …). These were always stored uncompressed with a "chunked compound not yet supported" note that was simply stale — chunked writes work for fixed-size string/compound datatypes like any other. write_string_dataset now chunks + deflates a string dataset once its payload reaches 4 KiB, so large, highly-redundant NullPad content shrinks substantially while tiny metadata stays contiguous (no chunk-overhead bloat).
• clawhdf5-format: decode the scale-offset filter (id 6) — both the integer variant (H5Z_SO_INT) and the floating-point D-scale variant (H5Z_SO_FLOAT_DSCALE). Handles signed/unsigned int sizes, f32/f64, negative minima, decimal scale factors and fill values; reverse-engineered against HDF5 2.0 and validated end-to-end. The float E-scale variant remains unsupported.
• clawhdf5-format: decode the N-Bit filter (id 5) — atomic, compound and array layouts (the full recursive type tree, nestable to any depth), previously unsupported. Signed and unsigned reduced-precision integers and float members all read end-to-end, validated against HDF5 2.0.

New Features

• clawhdf5 / clawhdf5-format: read external-file Virtual Datasets (VDS). The format layer gains read_raw_data_full_with_resolver and a VdsSourceResolver callback (Fn(&str) -> Option<Vec<u8>>) that maps a stored source file name to its bytes, so the pure-byte reader can pull in external sources without a filesystem of its own. The clawhdf5 File API wires a default resolver that reads sibling source files relative to the opened file's directory, so File::open(...).dataset(...).read_*() now transparently assembles cross-file VDS. A source file the resolver cannot supply leaves its region at the fill value (matching HDF5); an external source with no resolver at all is a clean error. In-memory files (File::from_bytes) have no directory, so only same-file VDS resolves there.
• clawhdf5-format: assemble same-file Virtual Datasets (VDS) of any rank. Previously a virtual layout returned UnsupportedVersion. The reader now decodes the global-heap mapping block (reverse-engineered against HDF5 2.0: version · nused · [source-file · source-dataset · source-selection · virtual-selection]* · checksum, including the block-version-1 same-file marker), decodes the H5S source/virtual dataspace selections (ALL, NONE, and version-3 regular hyperslabs), reads each same-file source dataset, and scatters its selected elements into the virtual buffer in row-major order (so multi-dimensional block mappings land correctly); unmapped regions are left at the zero fill value. External-file sources return a clean unsupported error. The previous parse_vds_mappings used a guessed layout that did not match real files and is replaced.

Tests

• clawhdf5-format: regression test for scale-offset float E-scale datasets. The HDF5 library does not implement E-scale encoding — when asked for it (cd_values[0] = 1) it stores the chunk raw and sets the chunk filter mask to skip the filter — so these files read back verbatim purely by honoring the per-chunk filter mask. The test locks in that behavior against a fixture produced via the HDF5 low-level API; no E-scale decoder is needed.

Bug Fixes

• clawhdf5-format: read multi-direct-block fractal heaps. The reader split direct vs indirect block rows using the FRHP "Starting # of Rows in Root Indirect Block" field (a constant, typically 1), so any heap whose data spans more than one direct block — common in libhdf5 files with a large group or many dense attributes — was misread as having indirect blocks and failed with InvalidFractalHeapSignature. The split is now derived from the heap geometry (max_direct_rows = log2(max_direct / start) + 2). Validated against an h5py-written 400-dense-attribute group (root indirect block, 4 rows, 13 direct blocks).
• clawhdf5-format: scope the per-file chunk cache by dataset. The shared ChunkCache built its chunk index once and reused it for every chunked dataset in the file, keyed only by chunk coordinate with no dataset discrimination. With a single chunked dataset per file this was latent; once a file holds two chunked datasets of different rank (e.g. a 1-D compressed string array and the 2-D embeddings matrix), the first dataset's index was reused for the second, panicking with an out-of-bounds chunk coordinate. The cache now rebinds (dropping its index, chunk-index map, layout, and decompressed slots) whenever the dataset being read changes, while still caching repeated/sequential access to the same dataset.
• clawhdf5-format: read paged Fixed Array chunk indexes. A filtered, fixed-dimension dataset with more than one data-block page (>1024 chunks by default) previously failed with "paged Fixed Array data blocks not yet supported". The reader now walks the page-init bitmap (MSB-first), skips uninitialized pages, and resolves each page's fixed full-size slot (including the short final page). Reverse-engineered and validated end-to-end against an HDF5 2.0 file.
• clawhdf5-format: read array-typed datatypes (e.g. an array-typed compound member) via read_as_i32/i64/u64/f32/f64 — previously a TypeMismatch. The array is read as a flat sequence of its base elements (recursing for nested arrays), applying base-type precision rules.
• clawhdf5-format: sign-extend reduced-precision fixed-point integers on read. A signed integer whose datatype precision is smaller than its storage size is stored zero-filled, so e.g. a 16-bit-precision -1 previously read as 65535. The integer read paths now extract the precision field and sign-extend (full-width types are unchanged). Completes signed N-Bit reads and also fixes un-filtered reduced-precision integer datasets.
• clawhdf5-format: read datasets written by modern HDF5 (1.14+/2.0, i.e. libver=latest). Compound (class 6) and array (class 10) datatype version 5 messages and data layout version 5 messages were rejected as invalid; they reuse the v3/v4 binary structure, so they are now accepted. This unblocks reading compound types and — critically — every chunked/compressed dataset written by HDF5 2.0. Found by running the h5py interop tests against h5py 3.16 / HDF5 2.0. Independently reported (with a patch) against the v2.1.0 tag by M. Scot Breitenfeld (The HDF Group) — v2.1.0 predates this fix.
• clawhdf5-format: parse HDF5 2.0 native complex datatypes (class 11, datatype version 5, e.g. H5T_COMPLEX_IEEE_F64LE). The properties are a single base floating-point datatype, not a compound-style member list; the old parser read the base type's bytes as member names, producing a garbage datatype, and failed with UnexpectedEof when a complex type was nested in a compound. It is now surfaced as the equivalent {r, i} compound (the shape h5py writes for numpy complex dtypes), with a size check against the base type. Validated end-to-end against an HDF5 2.0-written file.

Performance

• clawhdf5-format: chunked writes now compress all chunks up front via compress_all_chunks, running across rayon threads under the parallel feature when there are more than 4 filtered chunks. On-disk layout is unchanged. Speeds up compressed embedding writes in clawhdf5-agent (which enables parallel).

Documentation

• Fix stale package names across all 13 per-crate READMEs (rustyhdf5-* / edgehdf5-* → clawhdf5-*, usage versions → 2.1.0).
• Correct README workspace/test/crate stats and the CLAUDE.md CLI subcommand list; document the hnsw and format compression/checksum feature flags and the entity_extract / async_memory modules.

v2.1.0 (2026-06-03)

New Features

• clawhdf5-agent: HNSW now backs the vector stage of hybrid_search. The hnsw feature is on by default, so semantic search uses the approximate clawhdf5-ann index instead of a linear cosine scan. The index mirrors the memory cache (node id == cache index) and self-heals — it rebuilds whenever it drifts from the cache length, so no mutation path can desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and dimension-mismatched queries fall back to the exact linear scan. Disable with --no-default-features --features float16 for exact search.
• clawhdf5-ann: HNSW is now a live, mutable index — added insert, mark_deleted (soft-delete bitset; deleted nodes are traversed for connectivity but never returned), compact (drops deleted vectors and renumbers survivors), and new (empty index). Serialization gains a format version tag (HNSW_FORMAT_VERSION = 2) and persists the deleted bitset; pre-existing v1 files still load.
• clawhdf5-agent: hybrid::merge_vector_keyword exposes the shared normalize-and-fuse step used by both the linear and HNSW vector paths.
• Expose max_dimensions() API on Dataset, MmapDataset, and LazyDataset
• NetCDF-4 unlimited dimension detection now works correctly
• Python bindings (clawhdf5-py) build and link on macOS with system Python

Bug Fixes

• clawhdf5-py: upgrade PyO3 and numpy 0.23 → 0.28 so the bindings build on Python 3.14 (PyO3 0.23 capped at 3.13 and hard-failed cargo build --workspace). Updated for the removed PyObject alias (Py<PyAny>) and the Python::allow_threads → Python::detach rename.
• Fix GPU L2 distance test (squared vs actual L2 mismatch in test helper)
• Mark Android JNI functions as unsafe for Rust 2024 edition compliance
• Add # Safety documentation to all public unsafe extern functions
• Fix all clippy warnings: needless_range_loop, manual_strip, ptr_arg, etc.
• Rename RelationType::from_str to from_label to avoid trait confusion
• Isolate h5py interop tests with #[ignore] when h5py unavailable

Code Quality

• Full rustfmt pass across workspace (61 files)
• Refine inner unsafe blocks for Rust 2024 edition style
• Zero clippy warnings, zero clippy errors across entire workspace
• 1,546 tests passing, 0 failures

v2.0.0 (2026-03-19)

• Unified rustyhdf5 (11 crates) and edgehdf5 (4 crates) into a single workspace
• All crates renamed to clawhdf5-* prefix
• Version bumped to 2.0.0 across all crates
• Git dependencies replaced with in-workspace path dependencies
• Added agent feature flag to clawhdf5-agent