clawhdf5/CHANGELOG.md

Changelog

Unreleased

Name lookups through the name index (2026-09-26)

• Finding one link or attribute by name reads the name index, not every entry. In a dense group (links in a fractal heap) the v2 B-tree name index (record type 5, lookup3 hash of the name) is descended to the records with the name's hash and only their links are read — O(log n) instead of all n. Path resolution (File::dataset, resolve_path_any, soft-link targets) and Group::dataset/Group::group (on File, MmapFile and LazyFile, which listed the whole group per call) use it; names whose hashes collide are all compared, so the order libhdf5 gives them does not matter. New clawhdf5_format::group_v2::resolve_child, btree_v2::find_btree_v2_records (records in one key range), and a lookup-stats feature counting heap objects read, for tests. Huge heap objects are found through their index the same way.
• attr(name) on the facade's groups and datasets (all three file types): one attribute, found in dense storage through its name index (record type 8) instead of reading every attribute (clawhdf5_format::attribute::find_attribute_in_file).
• Two links of one name: the first wins everywhere. A group cannot validly hold two links of one name, but a damaged or hand-made one can. The listing, resolve_child (Group::dataset/group) and path resolution now all use only the first link of a name (header message order in a compact group, name index order in a dense one) and ignore the rest, even if the first dangles — libhdf5's rule for compact groups (h5py fails to open a dangling first link although a later one resolves). Before, the listing skipped a dangling first link and listed the name via a later one that lookup did not follow, and path resolution followed the last.
• B-tree v2 internal nodes are checksum-verified. Lookups prune children by internal-node keys, so a corrupted internal node could hide a name with no error; a mismatch is now ChecksumMismatch, as in libhdf5, for lookups and listings alike.
• Group::entries() and File::group_at(address): a listing's (name, address) pairs, to open children without looking names up again.
• Test: crates/clawhdf5/tests/indexed_lookup_interop.rs — every child of an h5py-written 35 001-link group (with colliding hashes) opened by name reads at most two links per lookup (before: 35 001), matches h5py, and every link kind (soft, relative, dangling, external) resolves as h5py resolves it in dense and compact groups.

Checked address conversion (2026-09-26)

• No 64-bit file value is truncated on a 32-bit target. All 119 truncating u64 as usize casts in clawhdf5-format that clippy's cast_possible_truncation reports, under every feature the crate is built with in CI except szip (115 with default features and plugin-filters, 4 more behind parallel), are gone: file addresses, lengths and counts go through addr::to_usize, which fails with FormatError::Overflow where the value does not fit (wasm32 and other 32-bit targets; it used to wrap onto another part of the file), and in-memory counts through addr::saturating_usize. On 64-bit targets nothing changes. scripts/check-32bit-casts.sh (run by ci-test.sh) lints the crate with no default features, with default features, and with every optional feature but szip (for wasm32; the set with zstd, which does not build for wasm32, for the host), and fails on any new truncating cast. The facade, clawhdf5-io and clawhdf5-ann are not covered.

Chunked full reads (2026-09-26)

• Chunks are decoded straight into the output, into reused buffers. A full read of a chunked dataset faulted in about three times its size in fresh pages: every chunk was decoded into a new buffer per filter stage (the cached reader behind read_* decoded 128 chunks at a time before placing any; the uncached one behind MmapFile, LazyFile and verify_provenance decoded the whole dataset first), then assembled into a byte buffer, which the typed readers copied once more. Now each chunk is decoded into buffers the thread keeps between chunks and reads (clawhdf5_format::filters::DecodeScratch, decompress_chunk_exact_with: deflate inflates into a kept buffer with a reset inflater, shuffle into the other one, Fletcher32 is checked in place; other filters go through the registry as before) and copied directly to its place in the output. Chunks still go into the file's chunk cache when the whole dataset fits. Selection reads decode the chunks they touch the same way.
• Typed full reads of chunked data skip the byte buffer. read_f32, read_f64, read_i32, read_i64 and read_u64 (on File, MmapFile and LazyFile) of a chunked dataset stored as that type in native byte order decode every chunk into the returned Vec (huge-page backed when large, like the byte readers' output); other types and byte orders convert as before. New public clawhdf5_format::data_read::read_chunked_native.
• Reading threads no longer wait for a busy rayon pool. A full read handed its chunks to rayon and the calling thread slept until the pool had decoded them, so with a small pool (2-4 threads) readers outside it queued behind its workers. The calling thread now decodes too, and pool workers join in only when free; a helper the pool starts after the read has finished returns at once. A single read still spreads over the default pool. This replaces the one-thread-pool special case below for full reads. Chunks are placed from several threads only when the chunk index puts them on the chunk grid at distinct places; a corrupt index is read one chunk at a time, and the error reported is still the first failing chunk's. New test crates/clawhdf5/tests/busy_decode_pool.rs.
• Fixed: in a filtered dataset, a chunk stored with every filter skipped (filter mask) and shorter than a chunk read with zeros in place of its missing part through File's read_*; it is now an error naming the chunk, as MmapFile/LazyFile already made it.
• New h5py comparison crates/clawhdf5/tests/chunked_read_paths_interop.rs: every chunked read path (cached and uncached full reads, MmapFile, LazyFile, small, strided and point selections, with and without the parallel feature) for 1-8-byte integers and 2-8-byte floats in both byte orders, through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc, with partial edge chunks, sparse datasets and fill values, and datasets larger than the chunk cache.

Writer: large dense indexes (2026-09-26)

• track_order orders attributes too, as h5py's track_order=True does. It tracked link creation order only, so h5py listed a tracked object's attributes by name. A tracking object's header now has the attribute creation order tracked and indexed flags, an Attribute Info message with the next creation order (also for inline attributes), and a creation order on each inline attribute message; dense attribute storage gets a creation-order index (B-tree type 9). FileWriter::track_order / FileBuilder::track_order now apply to datasets' attributes as well, and DatasetBuilder::track_order sets it per dataset. Groups and datasets that track order are written differently from before; others are unchanged. More than 65 535 attributes on a tracking object is an error (libhdf5's creation order counter is 2 bytes). The reader (attribute::extract_attributes*) lists a tracking object's attributes in creation order. Test track_order_lists_attributes_in_creation_order (h5py lists, reads and extends them in "r+" mode, including libhdf5's move from inline to dense storage).
• No more 65 535-record limit on the writer's v2 B-trees. Dense link storage (name index and creation-order index), dense attribute storage and the chunk index of datasets with more than one unlimited dimension were written as a single leaf, so a group with more than 65 535 links, an object with more than 65 535 dense attributes, or such a dataset with more than 65 535 chunks was an error. The writer now builds internal nodes to any depth (clawhdf5_format::btree_v2_write), with node capacities and child-pointer widths from the same arithmetic as libhdf5's H5B2__hdr_init (shared with the reader, btree_v2::node_info) and libhdf5's node sizes (512 bytes for dense storage, 2048 for chunks). Indexes that fit the old one-leaf layout are written byte for byte as before. New tests crates/clawhdf5/tests/deep_btree_interop.rs (100 000 links, 70 000 attributes, 200 000 chunks; h5py, h5dump, clawhdf5, and h5py "r+" edits) and check_files_with_deep_btrees (h5rs check).
• Links and attributes whose name hashes collide are found by name. The dense name indexes (a group's links: B-tree v2 type 5; an object's attributes: type 8) are ordered by the name's lookup3 hash and, when two hashes are equal, by the name itself, as libhdf5 compares them. The writer broke ties by insertion order, so libhdf5 could not open one of two colliding names ("k69209" and "k155448" share hash 0x3a0b13e6; collisions are likely from about 77 000 names). Regression test names_whose_hashes_collide_are_found_by_name in crates/clawhdf5/tests/writer_groups_interop.rs.

Blosc2 (2026-09-26)

• Blosc2 (filter 32026) reads, in pure Rust. Files written with hdf5plugin's Blosc2 failed with UnsupportedFilter. New feature blosc2 (clawhdf5-format and clawhdf5, included in plugin-filters) decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered back into C order), Blosc2 chunks with their special values (zeros, NaN, uninitialised, one repeated value), and the shuffle, bit-shuffle, delta and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2 encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined codecs and registered Blosc2 filters (e.g. bytedelta) are errors; uninitialised chunks read as zeros. Tested against h5py 3.16 + hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with partial edge chunks; every integer width and f4/f8; datasets of zeros, one value and NaN; Fletcher32 before Blosc2) and against frames from python-blosc2 4.13.1 for what hdf5plugin never writes (crates/clawhdf5-format/tests/fixtures/blosc2/); the decoder is fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
• A crafted Blosc2 chunk cannot allocate more than a few times its HDF5 chunk size. Found in review before release: the sizes a frame declares sized the decoder's buffers. A 173-byte frame whose offsets chunk claimed 2 GiB was decoded in full for a 1 MiB chunk; an empty chunk allocated its declared block size twice (about 1 GiB); a B2ND chunk was decoded whole with its padding (up to 16x the chunk); and a Zstandard stream's declared window (up to 100 MiB) was reserved as the decoder was reused, which also affected Blosc 1 and bitshuffle with Zstandard. Now the offsets chunk is capped at the chunk size, block sizes are clamped to the chunk, B2ND blocks are placed as they are decoded (padding is never held), and the Zstandard window is capped at twice the stream's output (at least 128 KiB). A B2ND chunk may no longer be larger than its array, which hdf5-blosc2 never writes. tests/blosc2_alloc_bounds.rs measures peak allocation for these frames and for 45,000 fuzzed ones: at most 6x the chunk size, twice the input and 2 MiB of Zstandard state.

Remaining conformance errors (2026-09-26)

Conformance on tank, conformance/run.sh --no-fetch: 598 of 697 files ok (575 before). Of the 5 our-errors left, 3 are corrupt data HDF5 2.0 reads only through a bug (listed in CONFORMANCE.md), 2 are the Blosc2 and ZFP filters; the 2 mismatches are the known h5py big-endian VL bug. The five files whose cache image libhdf5 cannot load (cve-2025-6269-*, cve-2025-6516) count as ok because the library, like libhdf5, opens them and fails their objects (see below).

• Metadata cache images are read. A file written with a metadata cache image keeps its metadata cache entries in an image block the superblock extension points at, and libhdf5 reads them in place of the file's own bytes; in h5clear_mdc_image.h5 the root group exists only there, and every reader failed with InvalidObjectHeaderVersion(0). File, MmapFile and LazyFile (and h5rs) now apply the image at open (clawhdf5_format::superblock_ext::CacheImage), with libhdf5's checks. The file is not copied to do it: a mapped file gets the image's entries written into a private copy-on-write mapping (clawhdf5_io::HDF5Read::private_copy, MAP_PRIVATE), so only the pages they land on are copied, and a buffer the opener owns (File::from_bytes, open_buffered) is patched in place; files without an image are read from the mapping exactly as before. (An interim version copied the whole file onto the heap: 2 GB of memory to open a 1 GiB sparse file with an image, and an abort for an 8 GiB one; tests/cache_image_memory.rs guards it.) An image entry that runs past the end of file is refused (libhdf5 checks only its start; the images it writes never do this). A file whose image libhdf5 cannot load (cve-2025-6269-*, cve-2025-6516) opens, as in libhdf5, and every object lookup fails with the image's error (File, MmapFile; LazyFile reads the root group at open, so its open fails). libhdf5 fails only its first metadata read and then reads the file's own, possibly stale, bytes; those are never read here. An interim version refused such a file at File::open while the conformance probe reported it as libhdf5 does, so the gate counted five files as agreeing with h5py that the library did not open; probe and library now take the decision from the same superblock_ext::cache_image_state.
• Every other opener applies the superblock extension and the cache image too (superblock_ext::apply_cache_image_in_place, writing into the buffer each already owns): clawhdf5_io's NativeVol (at open, and on read for from_bytes), AsyncHDF5File, MpiVol (a minimal edit through the same vol::load_hdf5; the mpi-io feature cannot be built without an MPI installation, so it was not compiled), and the external source files of a virtual dataset. They read a file with an image from its own bytes — stale metadata, or none (h5clear_mdc_image.h5 failed with InvalidObjectHeaderVersion(0)) — and skipped the extension checks File::open makes. These readers cannot open a file and fail each object, so an image libhdf5 cannot load is refused with the image's error.
• The superblock extension is decoded at open, as libhdf5 does: a File Space Info or Metadata Cache Image message libhdf5 cannot decode makes the open fail (cve-2020-10810, cve-2020-10812 were opened). FormatError::InvalidSuperblockExtension, InvalidCacheImage.
• Dataset storage libhdf5 refuses at open is refused at open (FormatError::InvalidDatasetStorage, data_read::check_dataset_storage): an element count times element size that overflows (cve-2024-32624 /Dset_OBJREF opened and reported its shape), contiguous storage past the end of the file, compact data of the wrong size. An empty contiguous dataset at a defined address, which clawhdf5 up to v2.7.0 wrote, still opens.
• Wrong or missing data fixed:
  • a simple dataspace of rank 0 holds one element (it held 0; cve-2020-18494), and contiguous storage larger than the dataset reads (cve-2024-32623, cve-2025-2309; libhdf5 ignores the excess);
  • scale-offset returned wrong values for ordinary h5py files — see Correctness below; E-scale is refused, as in libhdf5; codes past the end of the chunk stay an error (cve-2025-2308, where HDF5 2.0 reads past its buffer);
  • shuffle uses its own parameter as the element size, as libhdf5 does (cve-2025-44905);
  • an unfiltered chunk the index records at other than the chunk's size is refused (it read with zeros for the missing bytes; cve-2025-44904);
  • a v1 B-tree chunk index is read as libhdf5 reads it: each chunk is looked up the way H5B_find / H5D__btree_cmp3 / H5D__btree_found look it up, and a chunk that lookup does not find reads as fill values. A key with a non-zero element-size coordinate is found in a 1-D dataset and not in one of rank 2 or more (cve-2025-44905 /Shuffle_float_data_le, which read the chunk's data where h5py reads fill values); an interim fix refused every such key, including 1-D files libhdf5 reads correctly.
• Refused as libhdf5 refuses them: a v1 group with an empty link name fails its listing (FormatError::InvalidLinkName; lookups still work, cve-2021-46244); dataspaces with more than 32 dimensions, a rank on a scalar or null dataspace, or a dimension over its maximum (FormatError::InvalidDataspace).
• ObjectHeader::object_class classifies a header as libhdf5 does (a dataset needs a datatype and a dataspace).
• Conformance harness: user-defined links were listed as objects by the reference, unopenable objects were not deduplicated, nested array types were hashed wrong (tarray3.h5), and the attributes of objects h5py cannot open were compared; all fixed. CONFORMANCE.md lists the corrupt objects HDF5 2.0 reads through a bug (bad_nbit_parms_walk.h5 among them: libhdf5's own test now requires that read to fail).

Concurrent reads (2026-09-26)

• Full reads of chunked datasets scale with threads again when rayon's pool has one thread. Each full read handed its chunks to rayon to decode; with a one-thread pool (RAYON_NUM_THREADS=1, or concurrent_read --decode-threads 1) every thread reading through a File queued behind that single worker, so N readers decoded on one core and throughput stopped at about 2x one thread. Such reads, and verify_provenance's uncached reader, now decode on the calling thread (clawhdf5_format::parallel_read::pool_can_parallelise). The File's chunk cache, the suspect in docs/known-issues.md, was not the cause: reads of datasets larger than its budget already skipped inserting, and its lookups cost a few percent at 16 threads. Throughput with the default pool is unchanged, and still short of an h5py process pool.

Contiguous read speed (2026-09-26)

• Large read buffers are backed by transparent huge pages. A full read of a contiguous dataset was one memcpy from the mapped file, yet ran at a quarter of h5py's speed on one thread: the fresh output Vec took a page fault (and a kernel page clear) for every 4 KiB page it was written to, 16384 of them for 64 MiB, and those cost several times the copy. numpy, and so h5py, asks for transparent huge pages on every allocation of 4 MiB or more; clawhdf5-format's read buffers now do too (madvise(MADV_HUGEPAGE) on Linux, libc added as a Linux-only dependency; a no-op elsewhere or when THP is disabled). It applies to the typed readers' output (read_f32, read_f64, read_i32, read_i64, read_u64, both byte orders), the raw contiguous read and the chunk assembly buffer. Values are unchanged; new h5py comparison crates/clawhdf5/tests/contiguous_read_interop.rs covers every 1-8-byte integer and float type in both byte orders, ranks 1-4, and datasets past the 4 MiB threshold.
• Hyperslab and point reads of contiguous data copy runs, not elements. A 256 x 256 hyperslab of a contiguous f32 dataset read at an eighth of h5py's speed: the selection's bounding box was copied out of the file, then walked element by element (a recursive call and two bounds checks per element) into a second buffer, which read_f32_selection converted into a third. Selections of contiguous data are now copied straight from the file, one memcpy per run of elements that is contiguous in the file (a block along the last dimension, blocks that touch, and whole rows when the inner dimensions are selected in full, merged), with no zero-filled intermediate; a selection covering most of the dataset no longer makes a full copy first. The typed selection readers (read_f32_selection, read_f64_selection, read_i32_selection, read_i64_selection) copy directly into their output when the dataset stores that type natively, and convert as before otherwise (big-endian, other widths). The chunked paths use the same run-based extraction. New public clawhdf5_format::data_read::read_selection_native and the sealed NativeElement trait (also used by the read_as_* fast paths, which gained one for native u64). Values are unchanged: checked against h5py by contiguous_read_interop.rs (strided, blocked, adjacent-block and whole-row hyperslabs, points, empty selections; every type, both byte orders, ranks 1-4).

Variable-length data (2026-09-26)

• VL values in files with 4-byte offsets (sizeof_addr = 4). A VL string attribute came back as AttrValue::Raw, a VL member of a compound failed with GlobalHeapObjectNotFound, and VL datasets failed with a size mismatch. Two causes: Datatype::type_size() reported 16 for every VL type (the element is 4 + offset size + 4 bytes: 12 here), and the global heap was parsed without the padding libhdf5 puts after its collection and object headers (H5HG_SIZEOF_HDR/H5HG_SIZEOF_OBJHDR round up to 8), so with 4-byte lengths every object was looked up 4 bytes early. Datatype::VariableLength now carries the element size stored in the datatype message (breaking for code that builds or exhaustively destructures that variant; patterns with .. are unaffected), and it is written back as stored. Tested against h5py (crates/clawhdf5/tests/vl_offset4_interop.rs).

• Wrong data: VL strings with an embedded NUL, and VL elements whose heap object has the wrong size. libhdf5 hands VL strings over as C strings, so h5py reads "a\0b" as "a"; read_vl_strings returned the NUL and what followed. An element whose heap object is not exactly length × base size bytes is refused by libhdf5 ("Expected global heap object size does not match"); we returned the object cut or padded to the length. Both now behave as libhdf5, and a heap address of 0 is a null element (empty) whatever its length. The new clawhdf5_format::vl_data::VlResolver does this and parses each global heap collection once per read: read_vl_strings parsed the whole collection again for every element. vl_data::check_element_size refuses a VL datatype whose stored size is not 4 + offset size + 4 (libhdf5 ignores the stored size). The conformance probe resolves VL elements with VlResolver too; conformance unchanged at 575 of 697.

• VL data through the facade. VL-string datasets (h5py's default str dtype) failed read_string with "type mismatch: expected String, got VariableLength". Dataset::read_string now reads fixed- and variable-length strings; new read_string_bytes (a VL string's exact bytes, as h5py's Dataset[()] returns them), read_string_selection, read_vlen::<T>() / read_vlen_selection::<T>() for VL sequences of numbers (T = f64, f32, i64, i32, u64; converted like the other typed readers), and File::decode_strings / decode_string_bytes / decode_vlen for VL values in compound fields and AttrValue::Raw attributes. MmapDataset and LazyDataset gain read_string for VL strings, read_string_bytes and read_vlen. Checked against h5py with 8- and 4-byte offsets: scalar and 1-/2-D, ASCII and UTF-8, empty strings, contiguous, compact, chunked with gzip/shuffle, never-written and partly written chunks, hyperslab selections, VL members of compound datasets and attributes (crates/clawhdf5/tests/vl_data_interop.rs). NetCDF-4 string variables now read through clawhdf5_netcdf4::Variable::read_string (checked against netCDF4-python in crates/clawhdf5-netcdf4/tests/interop_tests.rs).

• Crafted global heaps could exhaust memory. VlResolver kept an owned copy of every object of every heap collection it read, so collections nested inside each other's object data made a 744 KB file take 1.58 GB (and read_vl_strings before it did the same). The cache now records where objects lie instead of copying them, is dropped past a 32 MiB budget, and a collection overlapping one already read is an error (libhdf5 never writes one). New GlobalHeapCollection::parse_index locates a collection's objects without copying them; parse and parse_index refuse a collection running past the end of the file or an object running past its collection. Conformance unchanged at 575 of 697 (crates/clawhdf5-format/tests/vl_heap_bounds.rs).

• Every reader resolves VL data the same way. h5rs (dump, ls, diff, check --data) had its own lenient VL decoder: a heap object longer than the element's length was cut to it (h5py refuses it), a null string printed "" where h5dump prints NULL, the stored element size was trusted, and each heap collection was kept as a copy for the whole run. It now resolves through VlResolver, so dump matches h5dump byte for byte on VL strings ("a\0b" as "a", null as NULL), VL sequences and 4-byte-offset files, dump --json gives h5py's values, and check --data reports any heap object whose size is not exactly the element's length × base size. clawhdf5-wasm already resolved VL strings with read_vl_strings; it now uses VlResolver and refuses a VL type whose stored element size disagrees with the file, as File does (crates/clawhdf5-tools/tests/h5rs_interop.rs, crates/clawhdf5-wasm/tests/vl_strings.rs). New VlResolver::element / string_element resolve one element in place.

• A VL element at the undefined heap address is an error, as in libhdf5 ("addr undefined"). One of length 0 read as "" in every reader (File, h5rs, clawhdf5-wasm, read_vl_strings, read_vl_bytes). libhdf5 writes a null element with heap address 0, which still reads as empty, and h5py writes "" as a zero-size heap object at a real address, so no file libhdf5 or h5py writes is affected (a_vl_element_at_the_undefined_heap_address_fails_like_h5py in crates/clawhdf5/tests/vl_data_interop.rs). read_vl_bytes now also treats address 0 as null whatever the length, as VlResolver does.

Writer: groups and links (2026-09-26)

• Nested groups, to any depth. FileWriter/FileBuilder wrote the root group plus one level, and refused path-like names. Now a name may be a path (create_dataset("a/b/x"), create_group("a/b"), a leading / at the root) and missing intermediate groups are created, as h5py does; groups also nest through the new GroupBuilder::create_group/add_group. A group added at a path that already holds a group is merged into it (h5py's require_group); a name used twice otherwise, an empty or "." component ("a//b", "a/") or an absolute path below the root is an error. Datasets, attributes, dense attribute storage and dense link storage work at every level.
• Soft, hard and external links at any depth: add_soft_link(name, target) (h5py's SoftLink; the target may dangle), add_hard_link(name, target) (h5py's f[name] = f[target]; the target path is resolved when the file is written, may go through other hard links, and a missing target, a soft link on the way or a cycle of hard-link paths is an error) and add_external_link, on FileWriter, FileBuilder and GroupBuilder. An object with several hard links gets an Object Reference Count message, so libhdf5 can delete one of the links without freeing the object.
• Link creation order: track_order(true) on a GroupBuilder, or on FileWriter/FileBuilder for every group that does not set its own, tracks and indexes link creation order (h5py's track_order=True): the Link Info message carries the flags, each link its order, and a dense group a creation-order B-tree (type 6). h5py then lists members in insertion order. Attribute creation order is not tracked.
• A group holds at most 65 535 links (its link index is one B-tree leaf), and in a group of more than 8 links (dense storage) each link message must be at most 65 515 bytes (one fractal heap block; huge heap objects are not written); more is an error. Measured at the limit: 65 535 links with 100-byte names (a 7 MB heap) read in h5py, h5dump and clawhdf5, and h5py can add to the group. GroupBuilder's fields changed (they were crate-private); FinishedGroup is unchanged for callers.
• Files that use one level of groups and no new link kinds are laid out as before: byte-identical to the writer with the Group Info fix below (compared on simple, mixed dense/chunked/compact/external-link and paged files). Tests: h5py and clawhdf5 read the same tree (every path, attribute and value) from a 5-level file; soft, hard, external and cyclic hard links; 10 000, 20 000 and 65 535 links in one group, with and without creation order; libhdf5 adding and deleting links in our groups; h5rs check passes and h5rs dump equals h5dump (crates/clawhdf5/tests/writer_groups_interop.rs, crates/clawhdf5-tools/tests/h5rs_interop.rs).
• Big dense groups and attribute sets were unreadable. The fractal heap holding dense links or attributes wrote every doubling-table row as direct blocks, but past the 512 KiB the root's direct blocks hold, rows are child indirect blocks, and libhdf5 and h5rs check read them as such: a group with 20 000 links of 20-byte names was written without error and h5py could not list it ("incorrect metadata checksum"); 150 dense attributes of up to 56 KB could not be opened. This was in 2.7.0 too. The heap writer now writes child indirect blocks, nested as deep as needed. Found on the way: an object bigger than the next block's space was cut off (it now goes in the first block big enough), and h5py adding a link to a heap over 64 KiB overwrote its first block (the header's next-block offset was 0).
• h5py crashed adding a link to a group of more than about 47 700 links (35 000 with creation order tracked). The link index leaf's node size gave libhdf5 room for more than 65 535 records, which overflows the leaf's 2-byte count. The node is now capped at 65 535 records. Dense attributes use the same index builder: more than 65 535 on one object used to be written with the count modulo 65 536, and are now an error.
• A dense link or attribute message over 65 515 bytes (e.g. a soft link with a long target in a group of more than 8 links) was written cut off, and libhdf5 could not list the group ("object overruns end of direct block"). It is now an error.
• Chained hard links took exponential time to resolve. A hard-link target going through other hard links resolved them again on every path through them: 26 links whose targets each named the previous one twice took 46 s. Each hard link is now resolved once, and a cycle is reported by the link's name.
• A dataset attribute set twice read back as its first value, as for groups below (h5py listed the name twice). The later value now replaces the earlier one; a hand-set attribute named like a provenance attribute is replaced by the computed one.
• A group attribute set twice read back as its first value. Setting a group (or root) attribute again wrote a second attribute message with the same name, and h5py returned the first value. The later value now replaces the earlier one, as attrs[name] = v does in h5py — also when a group is merged from two builders.
• Non-ASCII link names were marked ASCII. A group or dataset name such as größe was written with the ASCII character set flag (h5py reported cset 0 for it); it is now flagged UTF-8, as h5py writes it.
• libhdf5 could not add links to groups we wrote. h5py in "r+" mode failed with "Unable to create link (message type not found)" on every group FileWriter wrote: libhdf5 reads a group's Group Info message before inserting a link, and none was written. Every group now carries one (version 0, default thresholds: 6 more bytes per group header, so files are not byte-identical to earlier versions). Regression test: crates/clawhdf5/tests/writer_groups_interop.rs.

Python bindings (2026-09-26)

• Panic: selections of v4 implicit-index chunked datasets (pre-existing, facade Dataset::read_selection, Rust callers too). A hyperslab whose bounding box covered more than half of a chunked dataset with the implicit index (libver='latest', early allocation, no filters) panicked with "index out of bounds" in generate_implicit_chunks: the fallback in data_read::read_raw_data_selection passed the layout's chunk dimensions, element-size dimension included, and then decoded the whole dataset anyway. That arm now decodes and extracts directly, for every chunk index. crates/clawhdf5/tests/v4_chunk_index_selection.rs reads small and large hyperslabs of all five v4 indexes (single chunk, implicit, fixed array, extensible array, B-tree v2) and compares them with h5py; it panicked before. The Python bindings made this easy to reach (ds[0:3] on libhdf5's h5fc_ext*.h5 test files).
• pip install / maturin develop now gives import clawhdf5. The distribution in crates/clawhdf5-py/pyproject.toml was still called rustyhdf5 while the extension module was clawhdf5, and the package's tests imported rustyhdf5, so they failed at collection. Distribution, module and tests now all say clawhdf5, and the module has __version__.
• h5py-style reads that read the selection, not the dataset. ds[...] used to read the whole dataset and slice it in numpy, and knew six dtypes. Now integers (negative from the end), slices with positive steps, ..., one increasing list of integers per key and compound field names map onto the facade's hyperslab selection (a list is read one group of neighbouring chunks at a time and picked from in memory), with h5py's results (numpy scalar for an all-integer key, 0-d array for scalar[...]) and h5py's errors for everything else (negative steps, None, boolean masks, out-of-range indices). Dataset.dtype is the numpy dtype h5py reports, for every integer and IEEE float width (incl. float16) in either byte order, bool, enums (base integer with metadata['enum']), complex (r/i compounds), fixed strings (S<n>), variable-length strings (object of bytes, as h5py), variable-length sequences (object of arrays), opaque (V<n>), HDF5 array types and compounds (numpy structured, offsets and padding kept, nested). The bytes the library returns become the numpy array's buffer without a copy. Types the mapping cannot describe exactly (references, bitfields, time, non-IEEE floats, integers with padding bits, variable-length members inside compounds) raise TypeError rather than return guessed data. Attributes come back as h5py returns them (numpy scalars and arrays with the stored dtype, str for variable-length strings, numpy.bytes_ for fixed ones — a change: string attributes written by this package are fixed-length and used to come back as str — and clawhdf5.Empty for a null dataspace, which datasets return too). Group/File gain get, values, items, iteration, len, name, absolute and relative paths (g['/a/b'], g['c/d'], f['/']); Dataset gains ndim, size, maxshape, name, len() and numpy.asarray(ds). File access and decoding run with the GIL released, so Python threads read in parallel. crates/clawhdf5-py/tests/test_read_vs_h5py.py compares every read with h5py 3.16 (HDF5 2.0) on a file h5py writes. One difference is h5py's: it returns variable-length sequences of big-endian floats unswapped; this package returns the stored values.
• A panic in the library is an ordinary Python exception. PyO3 turns a Rust panic into PanicException, a BaseException that except Exception does not catch. Every call from the bindings into the library is now guarded and a panic becomes clawhdf5.InternalError (a RuntimeError) naming the object; with the implicit-index panic above restored, ds[0:30] raises it.
• Wrong data: uninitialised padding in compound results of index lists. ds[[0, 3, 6]] joined one read per run with np.concatenate, which copies structured dtypes field by field into an np.empty result, so the padding bytes held whatever was in memory (pointers were seen) and leaked through tobytes(), hashes and write-backs. The runs' bytes are now joined in Rust, whole elements at a time, so the result carries the bytes read from the file (h5py's, zero for files it wrote) and stays zero-copy. The h5py comparisons now also compare every byte of structured values (test_compound_padding_bytes_match_h5py and assert_same).
• Index lists no longer decode the same chunks once per run. A list index was one uncached hyperslab read per run of consecutive indices, so on a chunked, compressed dataset every run decoded its chunk again: d[list(range(0, 200000, 40))] over 20 gzip chunks took 8 s (h5py: 0.014 s). The list is now read in groups — for a chunked dataset a group ends only where a whole chunk holds no selected index, so each chunk is decoded once; otherwise at a gap of more than 64 KiB — and the selected rows are picked from each group in Rust. The same read now takes 3.8 ms (h5py 4.1 ms; release build on tank, best of 5). test_a_long_index_list_decodes_each_chunk_once compares 1-D, 2-D and contiguous cases with h5py under a 2 s bound (5.8 s before, debug build).
• Groups and datasets remember where they are. Every ds[...], and every g[k], resolved its path from the root again (two or three times per open), and in a large group each resolution scans the group's links, so visiting a group was quadratic: 4000 scalar datasets in one group took 39 s (libver='earliest') and 131 s ('latest') to list, read and re-read in test_big_groups_are_not_quadratic; now 0.3 s each (debug build). A Dataset keeps its object's address, and a Group (and the file's root) its address and, once listed, its link table. New facade API: File::dataset_at(address) opens a dataset without resolving a path. libhdf5's h5stat_newgrat.h5 (35001 members in the root): listing takes 0.03 s and 2000 opens 1 ms (h5py: 0.022 s).
• ds[np.array(1)] is an integer index, as in h5py; a 0-d integer array went down the index-list path and raised a confusing TypeError. The h5py comparison keys now include 0-d arrays on every axis.
• Tests that would notice a held GIL, and our extra errors. test_reads_release_the_gil times a Python thread spinning while another reads: with the read made to hold the GIL it stalls for the whole read (0.062 s of a 0.064 s read) and the test fails; released, its longest stall is about 3 ms. (The existing threads test only checked values.) test_errors_match_h5py now also requires that every key h5py reads reads here too, with the same result, and covers more keys (0-d arrays, repeated and empty lists, (), ...).
• Docs say when a selection reads more than itself. The README and the package README said ds[...] reads only the selected elements, without condition. The library decodes the whole dataset when the selection's bounding box covers more than half of it, and for compact, virtual, unwritten and non-default-fill chunked datasets; the READMEs, the facade's read_selection docs and docs/known-issues.md now say so.
• CI builds and tests the Python package. It was excluded from CI. scripts/ci-test.sh now lints clawhdf5-py, builds the wheel with maturin, unpacks it under target/ and runs the pytest suite; skipped without maturin/pytest in $CLAWHDF5_PYTHON, a failure then under CLAWHDF5_REQUIRE_INTEROP=1. The CI interop venv installs both.

Plugin filters (2026-09-26)

• LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust. Files written by h5py with compression="lzf", or with hdf5plugin's Bitshuffle, BZip2 and Blosc, failed with UnsupportedFilter. New clawhdf5-format/clawhdf5 features: lzf (32000, on by default, no dependencies), bitshuffle (32008: transpose only, LZ4 and Zstandard modes), bzip2 (307), blosc (32001: Blosc 1 frames with BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard codecs and byte/bit shuffle; BloscLZ is decoded by a port of c-blosc 1.21's decoder, and cannot be written), and plugin-filters for all four. None compiles C: Zstandard is ruzstd, bzip2 is libbz2-rs-sys. Write with DatasetBuilder::with_lzf(), with_bitshuffle(..), with_bzip2(..), with_blosc(..) or with_plugin_filter(PluginFilter::..); ChunkOptions gains a plugin field (breaking for code that builds ChunkOptions with a struct literal and no ..Default::default()). Tested both ways against h5py 3.16
  • hdf5plugin 7.1 over 1-3-D shapes with partial edge chunks, 1-8-byte types in both byte orders and incompressible data (crates/clawhdf5/tests/plugin_filters_interop.rs). Conformance: 573 of 697 files ok (was 569) — h5ex_d_lzf/bshuf/bzip2/blosc.
• Filter registry. Filters are looked up by ID in clawhdf5_format::filter_registry instead of a match: the built-in table (per build), then codecs registered at run time with register_filter(id, codec) — a decoding closure or a FilterCodec that can also encode. Built-in IDs cannot be overridden; a registered decoder's output is held to the chunk-size bound. Unknown IDs still fail with UnsupportedFilter(id), whose message now names known filters and the missing feature ("unsupported filter: 32026 (Blosc2, not implemented by clawhdf5)").
• Not implemented: Blosc2 (32026) and ZFP (32013) remain a clear error. (Blosc2 reads since the blosc2 feature, see above.)
• Wrong data: a chunk that decodes short read as zeros (pre-existing, every filter). HDF5 stores every chunk at the full chunk size, so a filter pipeline that decodes to fewer bytes means a corrupt chunk; every chunk reader (full, cached, selection, parallel, partial) padded it with zeros. It is now an error naming the chunk ("chunk at [16] decoded to 16 bytes, expected 32"), via the new filters::decompress_chunk_exact. libhdf5 returns the rest of such a chunk uninitialised, or fails when the filter checks. A Blosc frame declaring no data for a non-empty chunk is an error too. Legitimate edge chunks are unaffected (they are stored full-size, filtered or not); conformance is unchanged at 573 of 697, with no file changing class.
• Crash: a hostile Blosc chunk panicked in builds with overflow checks (debug builds, cargo test, maturin develop): a frame size below the 16-byte header underflowed. It is now an error. Every new decoder (LZF, bitshuffle, bzip2, Blosc/BloscLZ) is fuzzed with random and mutated frames in the unit tests.
• register_filter(32023, ..) works with the pcodec feature. 32023 is Granular BitRound's ID; the built-in entry there only reads clawhdf5 <= 2.7.0's pcodec chunks (filter name "pcodec"), so a registered codec now handles every other chunk with that ID, and writes. It was refused as "built in".

Upgrade Notes

• HDF5 correctness audit (2026-09-25). A sweep of 686 public files (the libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c, netcdf4-python, h5wasm, h5py and xarray corpora), a 567-case read matrix and a 96-case write matrix against HDF5 1.10–2.0 found bugs that returned wrong values with no error, and files we wrote that libhdf5 rejects. The fixes are listed under Correctness and Interop. What changes for callers:
  • Chunked datasets whose max shape is larger than their current shape, or whose unlimited dimension is not the first, were indexed by the current shape instead of the max shape, both when read and when written. Files from libhdf5 now read correctly. Files clawhdf5 wrote with such a max shape were laid out wrongly and now read the way libhdf5 always read them — rewrite them. Agent stores and ClawBrainHub files have no max shape and are unaffected.
  • Integer reads (read_i32/read_i64/read_u64/...) of float data now convert (truncate toward zero, saturate at the type's range, NaN reads as 0) instead of returning the IEEE bit pattern, and out-of-range integers saturate instead of keeping the low bits.
  • FileWriter::finish() now returns an error instead of writing a corrupt file for: a header message over 64 KiB (e.g. an attribute larger than ~64 KiB), a group/dataset/link name that is empty, . or contains / (nested paths were written as one literal link), a max shape smaller than the shape, a page size outside 512 B–1 GiB, and more than 65 535 chunks in a dataset with several unlimited dimensions.
  • Breaking (format crate): ObjectHeaderWriter::serialize, BatchObjectHeaderWriter::compute_sizes/serialize_all and build_chunked_data_from_precompressed return Result; read_fixed_array_chunks/read_extensible_array_chunks take max_dims; build_fixed_array_at/ea_writer::build_extensible_array_at take one Option<WrittenChunk> per index slot; fill_value::dataset_fill_value returns UnresolvedSharedMessage for a shared message it cannot resolve instead of None. FillTime::default() is IfSet (libhdf5's default; default files are byte-identical).
• ZeroClaw does not use clawhdf5. The project described itself as ZeroClaw's memory backend ("imported as a clawhdf5 Cargo feature"). Checked against ZeroClaw v0.8.5 (the latest release), the osobh/zeroclaw fork and their full history: no such feature or backend has ever existed. And clawhdf5-migrate's "ZeroClaw layout" (memory_chunks, sessions, entities, relations) is not ZeroClaw's schema — ZeroClaw uses a single memories table — so the migrator cannot read a ZeroClaw database. The claims are withdrawn; the migrator's layout is documented as its own.
• OpenClaw is not supported, and never was. The docs described a "drop-in" OpenClaw memory backend enabled with memory.backend = "clawhdf5". That config was never valid in any OpenClaw release (v2026.2–v2026.7 accepted only builtin/qmd and rejected unknown keys, so a Gateway given it refuses to start; OpenClaw 2.0 removed the key), no plugin was ever built, and @redclaw/clawhdf5 was never published. The integration docs (openclaw-integration.md, openclaw-config.md, migration-guide.md) are removed; docs/openclaw.md explains the status and what a real plugin would need against OpenClaw v2026.9.6. ClawhdfBackend stays as a library API.
• Breaking: MemoryError is now #[non_exhaustive] and gained SigningKeyRequired; a match on it needs a wildcard arm. Future variants will no longer be breaking.
• Breaking: clawhdf5-agent's agent feature is removed. It enabled nothing — the agent layer is always built — but the README and guides told people to pass it; drop agent from features = [...].
• clawhdf5-migrate now writes a real agent store. Its output used to be a layout of its own (/chunks, /sessions, /entities, /relations, no /meta) that HDF5Memory::open rejected, so a migrated file could not be used as agent memory. Files it wrote before this release are not agent stores; re-run the migration. Also: embeddings default to float16 like any new store (--f32 opts out; --float16 is a hidden no-op); a row with the wrong embedding length is an error instead of being truncated or padded; --incremental now matches rows by content against an existing store and follows the source's deleted flags; a source with no memory rows needs --embedding-dim. The per-dataset SHA-256 provenance attributes of the old layout are gone (the agent schema has no place for them).
• Files written by clawhdf5 now open in h5py and libhdf5. Every f32 dataset we wrote — including every agent store's embeddings — was refused with "sign bit position out of bounds", and every empty dataset with "invalid dataset size". Both were write-side bugs present in every release; clawhdf5's own reader was unaffected. An agent store is rewritten in full at each checkpoint, so it becomes readable at its next checkpoint on this version; other files with f32 or empty datasets need rewriting. Details in docs/known-issues.md.
• New stores store embeddings as half precision by default. MemoryConfig::float16 was persisted and otherwise ignored; it now writes float16 embeddings (48% smaller files at 100K) and rounds each embedding to half precision as it is saved — and it defaults to true for new stores. On the full LongMemEval haystack with real MiniLM embeddings every retrieval metric matched f32. Existing stores are unaffected: every agent store has recorded float16 = false, and keeps it (a v2.5.0 fixture guards this). A store that already had float16 = true rounds its embeddings when next opened and writes them as float16 at its next checkpoint. Opt out with float16 = false or create --f32; the CLI's --float16 is still accepted and now a no-op. Values beyond ±65504 are refused, so keep f32 for unnormalised vectors.
• Breaking: MemoryError gained InvalidEntry, returned when a float16 store is given an embedding value beyond ±65504. Exhaustive matches need the new arm.
• The default build no longer compiles any C. Deflate now defaults to the pure-Rust zlib-rs instead of zlib-ng, so building the core crates needs neither cmake nor a C compiler. Speed on HDF5 reads and writes is within 6% of zlib-ng, and compressed output is byte-identical. To keep zlib-ng, enable fast-deflate (on clawhdf5, clawhdf5-format or clawhdf5-filters); it overrides zlib-rs wherever it is on.
• A truncated deflate chunk is now an error. It used to read back short, with no error.
• Minimum supported Rust is 1.92, now declared in every crate's rust-version and checked in CI.
• New stores use the int8 vector index by default. MemoryConfig::quantized_index now defaults to true: a quarter of the index memory, builds 1.8x (x86-64) and 2.3x (Raspberry Pi 5) faster, and searches 1.63x and 1.18x faster at equal recall, measured on every configuration tested. Existing stores are unaffected — a store written with v2.6.0 or later keeps its persisted setting, and one written before the setting existed opens as false and keeps its f32 index. Set quantized_index = false, or pass create --f32-index to the CLI, to opt out. The CLI's --quantized-index is still accepted but is now a no-op.

Tools

• New crate clawhdf5-tools with the binary h5rs: HDF5 command-line tools without libhdf5, built only on the clawhdf5 facade and clawhdf5-format (no C, so it also builds as a static musl binary).
  • h5rs ls [-r] [-v] FILE[/path] lists objects like h5ls (its first two columns are h5ls's text on the test files) plus the datatype; -v adds address, link count, layout and chunk index, chunk size, storage, filters, datatype and attributes.
  • h5rs dump [--json] [-A] [-p] [-d PATH] FILE prints DDL text that is byte-identical to h5dump 1.14.6's (and to Debian's 1.14.5, which CI uses) on the test files (all layouts and chunk indexes, v1/v2 groups, compound, enum, strings, links, named types, attributes; null-padded strings show their NULs at any depth), or JSON in the HDF Group's hdf5-json layout (schema in the crate README). Nested compounds print inline and long double values as errors (exit 1); both are listed in the README.
  • h5rs stat FILE reports h5stat's object, link, rank, layout, filter, attribute, raw-data and file-size figures (equal to h5stat's on the test files); metadata space is one figure, not broken down.
  • h5rs diff [-r] [-q] [-n N] [-d D] [-p R] [--follow-symlinks] A B [OBJ1 [OBJ2]] (option names as h5diff's: -c is --compare, the count is -n/--count=N) compares objects, kinds, datatypes, shapes, attributes, values and link targets; exit status 0/1/2 as h5diff's. Soft links are compared by target path, as h5diff's default, or with --follow-symlinks by the objects they lead to (external links are never followed). Every path is compared, including every name of a hard-linked object and the members of a hard-linked group; with a -d/-p tolerance, integers are compared exactly in integer arithmetic (no loss above 2^53), and a -p below the f64 epsilon compares exactly, as h5diff's. Objects that cannot be compared count as a difference (h5diff exits 0 for them), and NaN equals NaN.
  • h5rs check [--data] FILE is a structural validator: it walks every object, parses every header message, verifies the checksums of every version 2+ structure it meets (superblock, object headers and continuation chunks, v2 B-tree nodes, fractal heap headers and — which the library's reads do not — every direct and indirect heap block, and extensible/fixed array chunk indexes), checks each chunk index against its dataset (aligned, in-extent, unique, plausibly sized chunks), and that raw data lies inside the file without overlaps. Every problem is printed with its address; exit 1 when there are any. libhdf5's h5check reads only the 1.8 format. On the conformance corpus it passes all 418 files that both clawhdf5 and h5py read in full, and check --data flags 135 of the 150 CVE and fuzzer files of the cve_hdf5 corpus (tank, 2026-09-26). --data also follows variable-length data into its global heap collections and reports a damaged one at its address. It inherits the library's tolerance, though: 8 of the 15 it passes are files h5dump 1.14.6 rejects (see docs/known-issues.md, header checks).
  • Values over --max-bytes (default 1 GiB) are reported instead of read; a panic is caught and reported as an internal error (exit 3). scripts/h5rs-fuzz.sh runs every subcommand over a corpus (default the CVE reproducers, optionally with byte-flipped copies) with overflow checks, a timeout and a memory limit, and fails on any panic, crash or hang; scripts/h5rs-check-ok-files.sh runs check --data over the fully-read conformance files.
  • Because the library does not verify fractal heap block checksums when it reads a dense group's links or dense attributes, h5rs verifies a heap's blocks before reading from it and refuses a damaged one, as libhdf5 does, instead of printing what the damaged block holds.

Signing

• clawhdf5-agent: Ed25519-signed checkpoints — the README's "cryptographically verifiable memory", now true. With HDF5Memory::set_signing_key(key), every checkpoint stores a signed manifest: a SHA-256 per record (text, embedding as stored, channel, timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus hashes of the settings (and WAL mark), sessions and knowledge graph, with the per-record hashes in /integrity/record_hashes. HDF5Memory::verify(path, &public_key) recomputes everything from the file and reports which part changed and which records (changed_records); a forged manifest fails the signature. The key is never persisted; a signed store refuses to checkpoint without it (MemoryError::SigningKeyRequired), and remove_signature() is the deliberate way back to unsigned. Saves still in the WAL are not covered (wal_entries_unsigned). Tests include every kind of edit, and an edit made with h5py in place, which verify pinpoints. Cost: ~20% of a checkpoint, 32 bytes per record (BENCHMARKS.md, "Signed checkpoints"). New dependencies ed25519-dalek, sha2, rand_core — pure Rust; the no-C check still passes.
• clawhdf5-cli: keygen --out <file> (owner-only key file), --signing-key <file> / CLAWHDF5_SIGNING_KEY on writing commands (create signs immediately), verify --public-key <hex|file> (JSON report; exit status 2 if not valid), and signed in create/stats output.

Migration

• clawhdf5-migrate: writes through the agent's own API (HDF5Memory::create / open, save_batch, the session cache and knowledge graph), so there is no second copy of the schema. Sessions and entities/relations carry over; deleted rows become deleted records (or are left out with --skip-deleted). Every source row is checked before the output is created, so a source that cannot be migrated leaves an existing store untouched. Validation reads the result back with HDF5Memory::open_read_only, compares every field (embeddings bit for bit — round_to_f16 of the source for a float16 store) and checks that a migrated record is found by search. The half-based conversion is gone; clawhdf5_format::float16 is the only one. 42 tests, including h5py opening a migrated store; an adversarial review's two blocker and four major findings are fixed with regression tests.
• clawhdf5-agent: HDF5Memory::sessions() / sessions_mut(), HDF5Memory::delete_batch(&[usize]) (one save, all-or-nothing, never auto-compacts), SessionCache::add_at, and SessionCache / SessionEntry re-exported from the crate root.

Search

• clawhdf5-agent: HDF5Memory::search with SearchOptions — source filtering, re-ranking and confidence rejection in the store's own search path. Re-ranking and confidence rejection used to be reachable only through the OpenClaw backend, which now calls search with both on.
  • with_sources([..]) restricts a search to records from those source channels. It applies before ranking, so a filtered search still returns up to k results, normalised over what it can return. Measured at 100K: the exact filtered top 10 for filters keeping 50%, 10% and 1% of the store and for records far from the query, and never slower than an unfiltered search (2.3 ms for a 1% filter vs 4.6 ms unfiltered). See BENCHMARKS.md, "Search options".
  • with_rerank(ReRankConfig) re-ranks a pool of max(3k, 10) candidates (rerank_pool to change it) by relevance, recency, source authority and activation; with_confidence(ConfidenceConfig) drops low-confidence results; at_time(now) pins the clock for recency. About 3% on latency.
  • hybrid_search and hybrid_search_with are unchanged (tested bit for bit against search with default options).
• clawhdf5-agent: the OpenClaw backend's search now boosts the Hebbian activation of the k results it returns, not of the whole 3k candidate pool it re-ranks.

Documentation

• OpenClaw claims withdrawn across the README, QUICKSTART, USE_CASES, ROADMAP (Track 7 marked withdrawn) and the openclaw module docs; the dead github.com/redclawsystems/openclaw link is gone. The Node package is marked unpublished and broken (now "private": true so it cannot be published by accident), with its bugs recorded in docs/known-issues.md.

Benchmarks

• Every undated or pre-September section of BENCHMARKS.md re-run on one machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for each and every number traced back to the raw output by a separate check. Where a figure moved, the section says so. Two apparent regressions were isolated rather than published: knowledge-graph traversal (a real bug, fixed above) and the write path, which measures the same at v2.3.0 on this machine — the old 18 µs / 6.17 ms figures came from an undated run on other hardware; float16 adds ~2 µs per save and the int8 index nothing.
• New multimodal_bench: cross-modal search at 1K and 10K records, which the README claimed but nothing measured.
• footprint_bench reports whether it built float16 or f32 stores and takes --f32; it had kept printing "f32" after the default changed.
• New concurrent_read harness, with an h5py counterpart (crates/clawhdf5-bench/scripts/concurrent_read_h5py.py, threads or processes) and compare_concurrent_read.py: decoded read throughput and scaling efficiency at 1-16 threads on one open file, full reads of distinct datasets and random hyperslabs of one dataset, deflate and contiguous, warm or --cold page cache, JSON output. Not yet measured — BENCHMARKS.md ("Concurrent reads") has the commands and no numbers.

Interop

• h5py could not open chunked datasets we wrote with a chunk dimension from 65 536 to 16 777 215. A version-4 layout must store its chunk dimensions in the fewest bytes that hold the largest (3 for 70 000); the writer rounded 3 up to 4, and HDF5 2.0.0 (h5py 3.16) refuses that ("stored chunk dimension encoding length does not match value calculated from chunk dimensions"). Newer libhdf5 and clawhdf5 read those files; new files use the exact width. Test: we_write_chunk_dimensions_in_the_fewest_bytes.
• Conformance sweep in the repo (conformance/, report in CONFORMANCE.md). conformance/run.sh fetches eight public HDF5 corpora pinned by commit (libhdf5's test files, the HDF Group's CVE reproducers, pyfive, netcdf-c, netcdf4-python, h5wasm, h5py, xarray-data) into a gitignored cache, reads every file with clawhdf5 and with h5py/libhdf5 (and the CVE files with h5dump) under a timeout and memory limit, compares them object by object and regenerates the report — about 30 s once the corpus is cached. A nightly Gitea job (.gitea/workflows/conformance.yml) runs it and fails on any panic, hang, crash or out-of-memory, or when a file in conformance/baseline.json stops reading identically. First report, on 42b81d9: 467 of 697 files identical to h5py, 123 our-error, 15 mismatch (2 of them an h5py bug), 92 that libhdf5 cannot read, no panics, hangs or crashes. Compared with the ad-hoc audit sweep, the probe now compares N-Bit floats (and integers with a bit offset) as the values libhdf5 converts them to rather than raw file bytes — 8 files that were reported as mismatches read identically — and the reference side no longer flips between runs when libhdf5 aborts while freeing h5py objects.
• clawhdf5-format: every f32 dataset was unreadable by h5py and libhdf5. The float datatype encoder hard-coded the sign bit's position to 63, correct only for f64; libhdf5 validates it and refused the dataset. It is now derived from the type (15 / 31 / 63). Our reader ignores the field, and the interop suites only wrote f64, which is how it went unnoticed.
• clawhdf5-format: every empty dataset was unreadable by h5py and libhdf5. It was written with a real address and zero bytes, which trips libhdf5's addr + size <= addr overflow check. An empty contiguous dataset now gets the undefined address, as libhdf5 writes it. This affected every agent store without sessions or a knowledge graph.
• New interop tests: f32 and float16 datasets in both directions (our float16 rounding matches numpy's bit for bit on 4 020 probe values, including ties, subnormals and the overflow boundary), and an agent store — f32 and float16 — opened by h5py with every dataset decoded.
• clawhdf5-format filters, checked against libhdf5 + hdf5plugin:
  • LZ4 (32004) now uses the registered HDF5 LZ4 format (8-byte BE size, 4-byte BE block size, BE-length-prefixed blocks). Our old framing (4-byte LE size + one block) was readable only by clawhdf5, and we could not read libhdf5's (h5ex_d_lz4.h5). Old clawhdf5 LZ4 chunks still read; they are told apart unambiguously (a registered chunk starts with four zero bytes).
  • Zstd (32015) frames now record the content size, which libhdf5's zstd plugin needs; h5py could not read our zstd datasets.
  • Pcodec moved from filter ID 32023 to 480. 32023 is registered to Granular BitRound, whose decode is a pass-through — libhdf5 with that plugin would have returned compressed bytes as data. Pcodec has no registered ID; 480 is in the registry's private range (256–511) and only clawhdf5 can read it. Chunks written under 32023 with the filter name pcodec (clawhdf5 ≤ 2.7.0) still read.
  • SZIP decode matches libhdf5. It returned garbage or zeros with no error for libhdf5-written files (the 4-byte size prefix, 32/64-bit byte-plane interleaving, reference interval, scanline padding and byte order were all handled wrongly) and rejected 64-bit data.
  • N-Bit honours libhdf5's "need not compress" flag (multi-filter pipelines such as tfilters.h5 failed) and reads enum/no-op members.
  • Scale-offset float decode uses libhdf5's single-precision arithmetic (was 1 ULP off for some values).
  • A pipeline with Fletcher32 ahead of the compressor (h5py set_fletcher32() then set_deflate()) no longer fails with "deflate: output exceeds size limit".
• clawhdf5-format: HDF5 1.4/1.6-era files are readable. Data Layout message versions 1 and 2 (compact, contiguous, and chunked through the version-1 B-tree) failed with InvalidLayoutVersion — 84 of the 686 files in the 2026-09-25 audit sweep, 205 datasets. They now read as libhdf5 does; checked byte for byte against h5py on HDF5's own test files (tests/legacy_format_interop.rs).

Storage

• clawhdf5-format: half-precision datasets. DatasetBuilder::with_f16_data writes IEEE binary16 (numpy float16), rounding to nearest-even; make_f16_type, and clawhdf5_format::float16 with the conversions, which are checked against the half crate on 16.7M values and round-trip all 65 536 half values. Reading float16 as f32 gained a little-endian fast path.
• clawhdf5-agent: MemoryConfig::float16 stores embeddings as half precision. At 100K x 384 the file goes from 154.0 to 80.8 MiB (−48%), a checkpoint from 752 to 512 ms and open from 300 to 252 ms, with the same vector recall@10 against an exact scan (0.999 vs 0.994) and the same hybrid_search latency; at 10K open is 3 ms slower. On the full LongMemEval haystack with real MiniLM embeddings every retrieval metric is identical to f32 (longmemeval_bench --float16). The cache rounds each embedding as it is saved, so memory and file agree bit for bit and a store returns the same results before and after a reopen (tested). Out-of-range values are refused with MemoryError::InvalidEntry rather than stored as infinity; batches are all or nothing. CLI: create --float16. See BENCHMARKS.md, "float16 embedding storage".

Browser (WebAssembly)

• New crate clawhdf5-wasm: the reader compiled to wasm32-unknown-unknown with a wasm-bindgen JavaScript API — open(bytes), list, info, attrs, read, readHyperslab — returning typed arrays of the stored width (BigInt64Array for 64-bit integers), string arrays for strings and enums, and a thrown Error for types with no typed-array form (compound, reference, opaque, VL sequences) or filters the build lacks (Zstd, SZIP). Read-only; the file is held in memory.
• examples/wasm-viewer/: a drop-a-file HDF5/NetCDF-4 viewer page (tree, type/shape/attributes, values paged as hyperslabs; ?file=&path= opens a URL). build.sh produces the package; test/run.sh checks it under Node (251 checks against values h5py/libhdf5 read back from an h5py- and a netCDF4-written file) and renders the page in headless Chromium. Size, measured 2026-09-26 on tank (gzip -9 -n): 627,501 B of wasm, 191,639 B gzipped, plus 21,826 B (4,487 B) of JS glue; h5wasm 0.10.3's embedded wasm is 3,544,184 B (907,096 B) — full libhdf5, so not equal functionality. See examples/wasm-viewer/README.md.
• The facade's read path already built for wasm32-unknown-unknown (nothing needed gating); ci-test.sh now builds it (--no-default-features) and lints clawhdf5-wasm for that target, and CI installs the target. The Node and browser tests run in ci-test.sh only where node and wasm-bindgen exist (not the CI container); CI checks the same expectations natively (clawhdf5-wasm's h5py_interop test).
• Dataset::raw_datatype() (facade) returns the full stored datatype, for decoding read_selection bytes with clawhdf5_format::data_read.

Build

• Pure-Rust default. clawhdf5-format, clawhdf5-filters and the clawhdf5 facade default to the zlib-rs deflate backend; fast-deflate (zlib-ng) is opt-in. No crate in the default dependency tree of the core crates compiles C, and ci-test.sh now fails if one appears. The facade's fast-deflate was on by default and is now off. See BENCHMARKS.md, "Deflate backend".
• zlib-rs also enables flate2's runtime_detection. Without it zlib-rs has no std, cannot detect SIMD at runtime, and inflates 3.5x slower; the workspace builds flate2 with default-features = false, which had been switching it off.
• rust-version = "1.92" for the whole workspace (the floor: wgpu requires it), and CI checks the workspace on exactly that toolchain.
• CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.

Correctness

• Scale-offset data read wrong values in every release that decoded it (v2.2.0 to v2.7.0), silently, on ordinary h5py files (fixed 2026-09-26). Of 1480 scale-offset datasets h5py writes across every integer type (i1 .. u8), f4 and f8, both byte orders, with and without a fill value, and scaleoffset from 0 to the full width, 332 did not read as h5py reads them: 151 returned wrong values with no error and 181 failed to read. The common cause was a chunk libhdf5 stores at full width (minbits equal to the type's width), which it does for any full-width scaleoffset and on its own whenever a chunk's values span most of the type's range: scaleoffset=0 integer data with a wide range (82 datasets, all wrong values), full-width u4/i4/u8/i8 (51 wrong values; the narrower types and the rest failed with "truncated minval" or "implausible minbits"), and f4 D-scale data with a large range (18, wrong values). Such a chunk holds the elements as they are; they were decoded as offsets from minval. Also fixed, found on crafted files: the packed codes start at byte 21 whatever size the chunk records for minval (cve-2025-44905 /Scale_offset_short_data_be), and a chunk with minbits 0 and a fill value is all fill values (it read as minval). The whole matrix is now an interop test (crates/clawhdf5/tests/scaleoffset_interop.rs, generated by h5py at test time, every dataset compared); on v2.7.0's decoder it reports the 332. See docs/known-issues.md.

• Corrupt files libhdf5 refuses are now refused instead of read. On the HDF Group's CVE reproducers, 18 objects that libhdf5 (HDF5 2.0, through h5py) refuses to open were read by clawhdf5, some as wrong data (a chunk dimension of 0 read as all fill values; chunks read at offsets off the chunk grid). The parser now makes libhdf5's checks, with libhdf5's error text:

  • object headers (FormatError::InvalidObjectHeader): every message of a v1 chunk is read and more than the prefix's count is refused (the rest used to be dropped); v1 message sizes must be multiples of 8 and a v1 chunk cannot end in a gap; a message running past its chunk is an error (it used to end the chunk quietly); contradictory message flags; a message of a class that cannot be shared flagged shareable; a reference-count message in a v1 header; malformed continuation, reference-count and modification-time messages; unknown v2 header flags.
  • datatypes (FormatError::InvalidDatatype): size 0; integer bits outside the type; float exponent/mantissa outside the type, empty or overlapping; a compound with no members, a member outside the compound, a duplicate name or overlapping members; an enum whose size differs from its base type's or with an empty name; array rank over 32 or a zero dimension; an opaque tag length that is not a multiple of 8; in a version-1 (unchecksummed) header, a numeric type that leaves more than half its bits unused (Datatype::parse_in_header, Datatype::check_unused_bits). A v1/v2 float's class bit 6 was read as VAX byte order; libhdf5 ignores it before version 3, and so does this. The overlap check measures each earlier member by its stored size, as libhdf5 does, so a variable-length member (4 + offset size + 4 bytes) in a file with 4-byte offsets does not overlap the member after it.
  • chunked layouts (FormatError::InvalidChunkDimensions): a zero chunk dimension, a chunk rank that does not match the dataspace, a chunk of 4 GiB or more indexed by a v1 B-tree (layout version 3 or earlier; 0x80000000-sized chunks hung the reader — layout versions 4 and 5 allow larger chunks, and HDF5 2.0 writes them), an element size in the layout that differs from the datatype's stored size (the chunks were laid out with the wrong element size), and v1 B-tree chunk keys whose offsets are not multiples of the chunk dimensions, including the keys that only bound a node (chunked_read::collect_chunk_info_checked).
  • truncated files (FormatError::TruncatedFile, Superblock::data_end): a file shorter than the end of file its superblock records is refused ("truncated file"), and nothing past that end is read. Every reader does this: File, LazyFile and MmapFile, and in clawhdf5-io NativeVol (at open, and on read for from_bytes), AsyncHDF5File and MpiVol (the MPI path is not built in CI: it needs an MPI installation).
  • the writer: FileWriter::finish() / FileBuilder::finish() refuse a datatype the reader would refuse (FormatError::SerializationError, "datatype cannot be written: ..."), such as a compound with a repeated field name or no fields, or an enum member with an empty name (CompoundTypeBuilder and EnumTypeBuilder build them without complaint). These were never valid HDF5 — h5py refuses them — and clawhdf5 wrote them, which made files it could not read back.

  Checks newer libhdf5 releases make but HDF5 2.0 does not (bit-field offsets, the variable-length kind, array sizes) are left out, so files h5py opens still open. Two libhdf5 checks are skipped on purpose because clawhdf5 up to v2.7.0 wrote files that fail them without being wrong: the sign bit of every float at position 63, and a size-0 string type for an empty-string attribute (new fixtures written by v2.7.0 guard this). Conformance: 569 -> 571 ok (h5stat_err_refcount.h5, h5clear_fsm_persist_less.h5), and 17 of the 18 CVE objects now fail as in libhdf5 (see docs/known-issues.md for the one left), as do 10 files h5py refuses as truncated. Tests: header_validation_interop.rs (h5py writes, the test damages a copy, both libraries must refuse it), legacy_writer_files.rs, and unit tests next to each check. Breaking (format crate): FormatError gained InvalidObjectHeader, InvalidDatatype, InvalidChunkDimensions and TruncatedFile; an exhaustive match on it needs the new arms.

• Chunked datasets whose chunk dimensions take 3, 5, 6 or 7 bytes did not open. A version-4 layout (libver="latest") stores each chunk dimension in the fewest bytes that hold the largest one, so a chunk dimension from 65 536 to 16 777 215 (e.g. h5py chunks=(70000,)) takes 3 bytes; only 1, 2, 4 and 8 were read, and the rest failed with UnexpectedEof. Widths 1-8 are read now, and 0 or more than 8 is refused as libhdf5 refuses it. A width larger than needed is accepted: HDF5 2.0.0 refuses one ("stored chunk dimension encoding length does not match"), but libhdf5 since HDFGroup/hdf5@e124c36 (2026-06-05) reads it, and clawhdf5 itself wrote such layouts.

• clawhdf5-format VDS: variable-length and reference data from a source in another file is refused. Those elements are global-heap IDs and object addresses in the source file; copied into the virtual dataset they would be decoded against the wrong file and name another object.

• clawhdf5-agent: a store whose /meta has an attribute that cannot be decoded fails to open (MemoryError::Schema). With attrs() now leaving unreadable attributes out, it would otherwise have opened with defaults in place of its settings (float16, compression, the WAL mark, ...).

• clawhdf5-format reader: an old-style group whose local heap has a free list pointing outside the heap was listed with names read from the broken heap (garbage names on cve-2021-36977.h5 once its user block was applied). libhdf5 refuses such a heap ("bad heap free list"); so do we now, with FormatError::InvalidLocalHeapFreeList. As in libhdf5 the free list is checked when the first name is read (LocalHeap::validate_free_list, new), so an empty group with a damaged heap still lists as empty.

• Files with a user block (h5py.File(..., userblock_size=N), h5jam; the superblock at 512, 1024, …) could not be read: every address in the file is relative to the superblock, but it was applied from byte 0 (InvalidObjectHeaderVersion on the root group). File (mmap, buffered, from_bytes), MmapFile, LazyFile, AsyncHDF5File, the VOL readers, the HNSW loader and external VDS sources now view the file from the superblock on, using the signature's position as the base address as libhdf5 does; user_block_size() reports the user block (h5py's userblock_size), and as_bytes() returns the bytes from the superblock on. Breaking (format crate): Superblock::parse refuses a non-zero signature offset with FormatError::UserBlockNotStripped, since the addresses it returns would be applied to the wrong bytes; pass the slice from signature::split_user_block (new) and parse at offset 0.

• clawhdf5-format reader: version-1 shared messages (HDF5 1.6-era files, e.g. a dataset using a committed datatype in libhdf5's tcompound.h5) read the heap-offset field of the embedded symbol-table entry as the target address and failed with InvalidObjectHeaderVersion. The address is now read after it, as libhdf5 does. Breaking (format crate): shared_message::parse_shared_ref takes length_size. A reference whose target header has no message of the referenced type is now FormatError::SharedMessageTargetMissing instead of returning the first other message found there (which decoded as garbage).

• clawhdf5-format reader: array members of version-1 compound datatypes (HDF5 1.6-era files, e.g. libhdf5's tcompound.h5) were read as a single element: a [4] i32 member came back as one i32, with the wrong size. The legacy per-member dimension fields are now decoded into an array type, as libhdf5 does; more than four dimensions, or a zero-sized one, is an error.

• clawhdf5-format virtual datasets (VDS), checked against HDF5 2.0 through h5py (crates/clawhdf5/tests/vds_interop.rs):

  • Wrong data: elements no mapping supplies — unmapped regions, and mappings whose source file or dataset is missing — read as 0 instead of the virtual dataset's fill value (e.g. h5py fillvalue=-1). Assembly moved to the new vds module: vds::read_virtual_dataset takes the fill value and a resolver that can refuse a name (VdsFileResolver), and File passes the dataset's fill value. A missing source dataset read as an error; it is fill now, as in libhdf5. Source datasets are read with their own fill value for unallocated chunks, and a source whose datatype differs from the virtual dataset's is an error (libhdf5 converts; we do not). File now refuses a source name that leaves the virtual file's directory (../x.h5, absolute paths), or any external source of a File::from_bytes file, with an error — these used to read as fill. Behaviour change: the raw-read API (read_raw_data_full*), which has no fill value, now returns an error for a virtual dataset with unmapped elements instead of zeros.
  • Unlimited and printf-style mappings are supported (all 7 VDS files in the libhdf5 test set are such mappings, e.g. Eiger/Percival detector layouts). %b in a source file or dataset name is the block number and %% a literal % (other % sequences are an error, as in libhdf5); block j is read from the source named with j, probing from 0 up to the first missing source. Unlimited source/virtual selections cover as much as the source's current extent fills, including a partial last block. As libhdf5 does on H5Dget_space, the extent is recomputed from the sources present (default "last available" view, printf gap 0) — vds::virtual_dataset_extent, used by Dataset::shape() — so e.g. vds-eiger.h5 is [5, 10, 10], not its stored [20, 10, 10]. A source stored in the other byte order is byte-swapped (libhdf5 converts); other type conversions remain an error.
  • Hyperslab selection versions 1 and 2 were refused ("only version-3 hyperslab selections are supported"). Version 1 is what libhdf5 writes for every VDS created with the default format bounds (h5py's default), so those could not be read at all; version 2 is its encoding of an unlimited selection. Both are decoded now, as are irregular hyperslabs (a union of blocks, read in row-major order as libhdf5 iterates them). SerializedSelection exposes the raw form, including unlimited counts.
  • The version-1 mapping list HDF5 2.0 writes (low version bound 2.0) was misparsed: each entry's flags byte was read as the start of the source file name, and names shared with an earlier entry (stored as that entry's index) were not followed. Now decoded as H5D__virtual_load_layout does.

• clawhdf5-format reader — values returned wrong with no error:

  • Fixed Array and Extensible Array chunk indexes were laid out by the dataset's current shape instead of its max shape (23 libhdf5 test files, and any h5py file with e.g. maxshape=(10, None) or (20, 10) under libver='latest').
  • Files with 4-byte offsets: unfiltered chunked datasets read as zeros. Chunk B-tree keys store offsets in 8 bytes whatever the file's offset size.
  • A chunk's filter mask skipped the whole pipeline when any bit was set; only the flagged filters are skipped now.
  • Float data read as an integer returned the bit pattern; narrowing integer reads kept the low bits; bfloat16 was decoded as IEEE half. Floats are now decoded from their datatype fields (bf16, FP8 E4M3/E5M2, IEEE half, single and double).
  • vl_data::read_vl_bytes truncated sequences of non-byte base types.
  • A shared fill-value message read as zero fill; it is resolved now, including from the file's shared-message (SOHM) table, which could never resolve because its index version byte was skipped.
  • Two threads reading two chunked datasets through one File could get each other's chunks (the shared chunk cache was switched between datasets across separate lock acquisitions). The cache is now keyed by dataset.
  • Compound datatype version 1 members with legacy array dimensions (HDF5 before 1.4, which had no array class) were read as a single scalar at the member's offset; they are now array members, as in libhdf5 (tarrold.h5, tcompound.h5). Only reachable once layout versions 1/2 were readable, since the files that use it are that old.

• clawhdf5-format reader — errors on valid files: a version-1 shared message (a committed datatype in HDF5 1.4/1.6-era files) was read as if the object header address followed the reserved bytes; it follows a link-name offset (the reference is an old-style symbol table entry), so the reader followed the name offset and failed with InvalidObjectHeaderVersion (tcompound.h5). New shared_message::parse_shared_ref_sized takes the superblock's length size; parse_shared_ref assumes it equals the offset size.

• clawhdf5-format reader — errors on valid files: enum and bool datasets through the numeric readers; the "don't filter partial edge chunks" layout flag; Fletcher32 ahead of deflate (NetCDF-4's order). Unknown-message flags follow libhdf5 (tbogus.h5): "fail if unknown" is refused, "fail if unknown and writing" is ignored by a reader.

• clawhdf5-format reader — dense groups and attributes (links or attributes kept in a fractal heap indexed by a v2 B-tree):

  • A link heap larger than the root indirect block's direct rows (512 KiB with libhdf5's defaults: a few thousand long link names, or ~20 000 short ones) could not be listed: child indirect blocks were given the wrong number of rows, so every link stored in one was unreachable.
  • v2 B-trees of depth 3 or more (a dense group of ~22 000+ links) were misparsed: internal-node child pointers were read with widths from an estimate instead of libhdf5's per-depth record capacities, and the listing failed. The same B-tree code indexes dense attributes, shared messages and chunks.
  • Fractal-heap "huge" objects (larger than the heap's managed-object limit, 4 KiB by default — e.g. an 8 KiB dense attribute or a link with a very long name) and "tiny" objects are now read; the ID type was taken from the wrong bits (6-7, the version, instead of 4-5), so a huge object failed and took every attribute on its object down with it (NetCDF-4 files such as netcdf4-python's issue671.nc). Huge objects are found directly from the ID or through the huge-object v2 B-tree, filtered or not.
  • Heaps with an I/O filter pipeline (a group created with a filter on its creation property list compresses its link heap) are now read: the header's pipeline was skipped with the wrong size, so its checksum was looked for in the wrong place, and filtered direct blocks were read raw.
  • A user-defined link (link class 65-255, e.g. 187 in libhdf5's tall.h5/tudlink.h5) made its whole group unlistable. Such links cannot be followed without the application that registered the class, so they are now left out of datasets()/groups() and path lookup, as h5py leaves out links it cannot open; reserved link types are still an error.

• clawhdf5 — soft links are listed, as h5py lists them: datasets() and groups() on Group/MmapGroup/LazyGroup include each soft link under its own name as the kind of object it resolves to, and dataset(name) / group(name) open through it. Relative targets resolve from the group holding the link. Dangling or cyclic soft links, external links and user-defined links are left out (h5py lists their names but cannot open them). Previously soft links were missing from the listings, and in old-style (symbol table) groups a soft link made the listing fail. New group_v2::resolve_group_children / resolve_path_from and group_v1::v1_soft_links in clawhdf5-format.

• clawhdf5 — one unreadable attribute no longer fails attrs() for every attribute on its object: it is left out of the map, and the new attrs_with_errors() (on every group and dataset handle) returns the map plus one error per attribute left out. Returned values are always complete. An error in the attribute index itself (attribute info message, dense heap header or B-tree) still fails the call. clawhdf5-format gains attribute::extract_attributes_tolerant; extract_attributes_full stays strict.

• clawhdf5-format reader — files with shared object header messages (SOHM, H5Pset_shared_mesg_index): a datatype, dataspace, filter pipeline or attribute stored in the file's SOHM heap failed with "invalid shared message version: 2" — only shared fill values loaded the SOHM table — so such files' datasets and attributes could not be read. shared_message::resolve_shared_message now loads the table when a reference needs it (36 cases of the audit's read matrix).

• clawhdf5-format writer — files libhdf5 rejects or reads wrong:

  • Extensible Array (one unlimited dimension): chunks from index 244 on were written but never indexed and read as 0, by libhdf5 and by us.
  • Fixed Array: more than 1 024 chunks gave checksum errors (data blocks were never paged).
  • A finite max shape larger than the shape gave libhdf5 "addr overflow"; an unlimited dimension that is not the first scrambled the data; several unlimited dimensions ((None, None)) broke the whole file. These now write the index libhdf5 writes (swizzled Extensible Array, or a B-tree v2 index for several unlimited dimensions).
  • Header messages over 64 KiB (the size field is 16 bits) and compact datasets at 65 534–65 535 bytes produced corrupt files.
  • Reference, Opaque, BitField and Time datatypes were written as empty messages; they now encode as HDF5 2.0 does.
  • with_page_size wrote a nonexistent superblock version 4; it now writes the v3 superblock and File Space Info message libhdf5 writes.
  • FillTime values were rotated on disk (NEVER was written as ALLOC, and so on). New DatasetBuilder::with_fill_value.
  • An empty-string attribute got a zero-size datatype, which made every attribute on the object unreadable in libhdf5.
  • maxshape equal to the shape no longer forces chunked layout.

• clawhdf5-format: a truncated deflate chunk read back short, with no error. The deflate filter used flate2's streaming reader, which returns the bytes it has when the input runs out before the end-of-stream marker. It now decodes in one pass into a buffer sized to the chunk and reports a truncated stream as DecompressionError. Same fix in clawhdf5-filters, where output longer than the stated size was also silently cut off; it is now an error.

Defaults

• clawhdf5-agent: MemoryConfig::float16 defaults to true for new stores, measured rather than assumed: identical LongMemEval retrieval on real embeddings, 48% smaller files and faster checkpoints and opens at 100K. clawhdf5-cli create --f32 opts out; like --f32-index, it only ever switches the default off.
• clawhdf5-agent: MemoryConfig::quantized_index defaults to true for new stores. The reason it had been off — that int8 search was slower on ARM — did not survive measurement (see Corrections). Stores that predate the setting still load it as false, so reopening one never changes how its index is held; a store written by the v2.5.0 CLI is now a test fixture that guards exactly that, and the test fails if the load default is changed.
• clawhdf5-cli: create --f32-index opts out. create used to assign --quantized-index straight into the config, which under the new default would have forced every CLI-created store back to f32 unless the caller knew to ask; it now only ever switches the default off.

Performance

• clawhdf5-agent: consolidation's novelty scoring (each add_memory against the whole working tier) computes the new record's norm once, takes each comparison in one vectorised pass instead of three, and splits a working tier of 4 096+ records across threads — same results, tested against the old formula. It had made consolidation_efficiency stall at 100K; the complete run now takes 8 min and fills in the 100K cycle row (46.66 ms) and the memory-reduction table.
• clawhdf5-bench: consolidation_efficiency no longer prints a record-count ratio as a "BM25 Speedup" (it was never measured), nor claims cycle time grows sub-linearly (its own numbers grow slightly faster than linearly).
• clawhdf5-agent: knowledge-graph traversal was 6.5x slower than it should be. bfs_neighbors and spreading_activation built an adjacency index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K entities took 155 µs. The index is now cached on KnowledgeCache and checked against a fingerprint of the graph on each use — one pass over entity ids and relation endpoints, no allocation — so any change, including direct edits of its public Vecs, still rebuilds it (tested). BFS over 1K entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
• clawhdf5-format, clawhdf5-filters: both deflate paths hand the codec the whole chunk in one call, into a buffer allocated once, instead of streaming it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's 1 MB inflate.
• clawhdf5-accel: dot_i8 has aarch64 kernels — SDOT for CPUs with the ARMv8.2 dot-product extension (Cortex-A76 and later, Neoverse-N1, every Apple Silicon generation) and plain NEON (vmull_s8 + vpadalq_s16) for the rest, selected at runtime. SDOT is issued through inline assembly, because the vdotq_s32 intrinsic is still behind the unstable stdarch_neon_dotprod feature. On a Raspberry Pi 5 at N = 100 000 and equal recall, the quantised index answers 1.18x the queries per second of f32 (7 267 vs 6 164) and builds 2.3x faster (14 464 vs 33 413 ms). Both kernels are tested bit-for-bit against scalar on real hardware, each explicitly — dispatch only ever takes one path on a given CPU, so testing through it alone would have left the plain-NEON fallback unexercised on any machine with SDOT.

Corrections

• The v2.7.0 entry for dot_i8 said quantized_index stayed off by default because "aarch64 falls back to the scalar loop", implying the ~13% search penalty measured on x86 applied on ARM too. It did not. That figure came from scalar int8 against hand-written AVX2 f32 kernels on x86, whose portable baseline is SSE2; on aarch64 NEON is the baseline, and measured on a Pi 5 the scalar int8 loop already matched f32 for search while building 1.76x faster. The claim was extrapolated rather than measured.

v2.7.0 (2026-09-20)

Upgrade Notes

• Two read-path bugs fixed, one of them silent. Datasets indexed by an Extensible Array (any dataset with one unlimited dimension) returned data from the wrong chunks past their first few dozen. If you have readings taken from such a dataset with an earlier release, they may be wrong; re-read them.
• A corrupt chunk index is now an error. Fixed and Extensible Array structures carry checksums that were previously ignored, so damage surfaced as plausible data from the wrong offset. Code that read a damaged file and got numbers will now get ChecksumMismatch instead. That is the point.
• Breaking: MemoryConfig gained hnsw_m, hnsw_ef_construction and hnsw_ef_search, so literal constructions need updating; ..Default::default() does not. All three default to the previous behaviour.

Correctness

• clawhdf5-format: datasets indexed by an Extensible Array returned wrong data beyond their first few dozen chunks. One unlimited dimension gives a dataset an Extensible Array chunk index, whose first elements (4 by default) sit inline in the index block and whose rest live in data blocks sized by a formula the reader got wrong. In the default layout everything through the 36th chunk happened to line up and the 37th onwards did not: a 400-chunk dataset silently returned wrong values from chunk 37, and datasets past about a thousand chunks failed outright with "invalid Extensible Array data block signature". Reads were wrong, not merely refused — the caller got plausible numbers from the wrong chunks. Four separate layout errors, each checked against files written by HDF5 2.0 and against the library source:
  • the number of data blocks in super block u is 2^(u/2), not 2^u;
  • each holds 2^((u+1)/2) * data_blk_min_elmts elements, which doubles every other level rather than every level;
  • a super block carries a block-offset field before its data block addresses, which was not skipped;
  • the page-init bitmap belongs to the super block, one bit per page packed across all its data blocks (MSB first), and was being read from inside the data block instead; a paged data block also ends its prefix with a checksum before the first page. Covered now by interop tests at 4, 37, 400, 5 000 and 200 000 chunks (the last large enough for paged data blocks), plus sparse, gzip-filtered and 2-D cases. Writing is unaffected; this is a read-path bug.
• clawhdf5-format: the sibling Fixed Array index (fixed dimensions written with libver='latest') was checked against the same range and is correct, including paged data blocks and sparse datasets — it really does keep its page-init bitmap in the data block, where the Extensible Array does not. It had no real-file coverage above the inline sizes either, so it now has the same tests.

Security

• clawhdf5-format: a crafted file could crash any reader through B-tree v2 traversal. Recursion was bounded only by the depth the file claimed (a u16), and child addresses were never checked for sharing. A node listing itself as its own child under a header claiming 65 535 levels — under 100 bytes — overflowed the stack and aborted the process (SIGABRT, not a catchable error). Levels whose children all point at one shared node below reached it fan-out^depth times: 29.5 million records from ~5 KB, and one more level would exhaust memory. Both are now errors, returned in under a millisecond: depth is capped at 64 (as the fractal heap already was), and traversal stops once it has produced more records than the file has bytes to hold. Every B-tree v2 user goes through this path — dense attributes, v2 groups, shared messages and chunk indexes. Valid files are unaffected, including a depth-2 HDF5 2.0 chunk index with 40 000 records, now covered by an interop test.

Integrity

• clawhdf5-format: Fixed and Extensible Array chunk indexes now verify their checksums (the checksum feature, on by default). Every structure in both — header, index block, super block, data block and each data block page — carries a Jenkins lookup3 checksum that was parsed past and ignored. The consequence of skipping it is not a missing warning but wrong data: a single flipped bit in a chunk address still parses, still points inside the file, and the reader hands back whatever bytes now sit there as the chunk's contents. Verified in both directions — the checksums accept files written by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged, and an interop test corrupts an address to confirm the read now fails instead of returning data (it does return data when the check is removed).

Performance

• clawhdf5-agent: opening a store is ~28% faster (455 ms -> 327 ms at 100k x 384). read_from_disk memory-mapped the file and then copied the entire mapping into a Vec for File::from_bytes, when File::open memory-maps it directly — so every open paid a full-file memcpy for nothing. Process peak memory is unchanged: the peak falls after the parse, during the index build, so the transient never reached the high-water mark. The footprint harness now reports that peak next to the retained figure, which is how this was checked rather than assumed.
• clawhdf5-accel: dot_i8, a runtime-dispatched int8 dot product (AVX2: sign-extend each half to i16, then madd_epi16; scalar fallback elsewhere). The quantised HNSW index used a scalar loop while the f32 path it was measured against ran AVX2, so the ~13% throughput cost recorded for MemoryConfig::quantized_index was a missing kernel rather than a property of int8. With the kernel, at N = 100 000 x 384 and equal recall, the quantised index answers 1.63x as many queries per second (21 848 vs 13 399 at ef=64, recall 0.9940 vs 0.9945) and builds 1.8x faster (1778 vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three alternating runs. It remains off by default only because the kernel is AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so the SIMD path is tested to agree with scalar bit for bit.

Tuning

• clawhdf5-agent: the HNSW parameters are configurable — MemoryConfig::hnsw_m, hnsw_ef_construction and hnsw_ef_search (defaults 16, 64, and 0 meaning "scale with k", i.e. today's behaviour). They were constants, so a deployment could not trade recall against memory or query speed at all. All three are persisted with the store. Values are clamped where the index requires it: clawhdf5-ann asserts a graph degree of at least 2, so a configured 0 — from a file, or from a caller who took 0 to mean "default" — used to abort the process inside the builder. Lowering ef_search also no longer narrows the candidate pool that fusion sees. Breaking: MemoryConfig gained fields, so literal constructions need updating; ..Default::default() does not.

Documentation

• clawhdf5-agent: BM25Index::search claimed to use Block-Max WAND for early termination. It never did; it scores every match exhaustively. It now says so, and why no pruning would help the store: hybrid_search uses scores(), since fusion normalises over every match.

v2.6.0 (2026-09-20)

Upgrade Notes

• Re-ranked results change, substantially for the better. RerankInput and ReRankConfig gained fields (relevance, relevance_weight), so literal constructions need updating; ..Default::default() does not. Any caller that re-ranked was previously getting results ordered by age with the retrieval score discarded — see below.
• Breaking: MemoryCache::embeddings is a cache::Embeddings rather than a Vec<Vec<f32>> (indexing still yields a &[f32] row); embeddings_flat is gone, replaced by flat_embeddings(); rebuild_flat() is a deprecated no-op.
• MemoryConfig gained quantized_index (default false, so behaviour is unchanged unless you opt in); literal constructions need the field.

Retrieval quality

• clawhdf5-agent: re-ranking discarded the retrieval score. reranker::rerank built its combined score from temporal decay, source authority and Hebbian activation only — RerankInput had no relevance field — so re-ranking a candidate pool reordered it by age and threw the retriever's ordering away. The OpenClaw backend re-ranked every search, so this was its shipping behaviour: measured over the full LongMemEval haystack it cost 40.6pp of Hit@1 (11.0% vs 51.6%) and two thirds of MRR (0.183 vs 0.643). RerankInput::relevance and ReRankConfig::relevance_weight (1.0 by default) fix it: relevance leads and the metadata signals break near-ties, which restores retrieval (Hit@1 +0.4pp vs no re-ranking) and improves recency discrimination by 6–7pp. Breaking: RerankInput and ReRankConfig gained fields, so literal constructions need updating; ..Default::default() does not.
• clawhdf5-bench: the LongMemEval harness feeds the dataset's real session dates to the store instead of a synthetic counter (decay needs true intervals, not just the right order), and reports newest_gold_first — on a knowledge-update question, did the newest gold session outrank the stale one it supersedes? Plain recall cannot see this, because both are labelled gold. New --rerank-sweep.

Memory

• clawhdf5-agent: MemoryConfig::quantized_index stores the vector index's own copy of the embeddings as i8 rather than f32, which at 100k 384-dim entries takes the index from 266 to 123 MiB and the whole reopened store from 399 to 256 MiB (2.72x -> 1.74x the raw vectors). Quantised distances are approximate and ef cannot compensate — recall@10 tops out at 0.967 against f32's 0.9995 — so the query path re-scores the candidate pool against the exact embeddings the store already holds, which restores recall (0.9940 vs 0.9945 at ef=64) for about 13% of QPS. Off by default: it trades query speed for memory, and which side is worth more depends on the deployment. The setting is persisted, so a reopened store does not silently revert to four times the index memory.
• clawhdf5-ann: Storage::Int8 and the build_with / new_with / from_graph_bytes_with constructors that select it. The scale is per row, not global — a fixed [-1, 1] scale spends fewer than 12 of the 255 levels on a unit-length 128-dim vector and is unusable (0.35 top-10 overlap against an exact ranking, versus 0.99 per row). compact() keeps the storage it was given; serialized indexes still carry f32 vectors, so a quantised index is rebuilt rather than loaded.
• clawhdf5-agent: a loaded store holds ~30% less memory (100k 384-dim entries: 505 -> 357 MiB, 3.44x -> 2.43x the raw vectors). The cache kept every embedding twice — a Vec<Vec<f32>> and a flattened copy for the batched kernels, maintained in lock-step — so it now stores only the flat buffer and indexes into it. Recall and query latency are unchanged. Breaking: MemoryCache::embeddings is a cache::Embeddings rather than a Vec<Vec<f32>> (indexing still yields a &[f32] row); embeddings_flat is gone, replaced by flat_embeddings(); rebuild_flat() is a deprecated no-op. Rows are now always exactly dim long — shorter ones are zero-padded — which makes the ragged-row case that used to silently misalign the flattened copy unrepresentable.
• clawhdf5-bench: search_harness --footprint reports live heap use per stage, measured with a counting allocator (RSS cannot see a structure freed into the allocator's own pool).

Testing

• The Python interop suites honour CLAWHDF5_PYTHON, and ci-test.sh picks up a .venv/bin/python automatically. On a PEP 668 "externally managed" system h5py cannot be installed into the system interpreter at all, so every interop suite — the h5py writer round-trips, the facade, netCDF4 and the reference files — was skipping silently. A silent skip here is exactly how the v5 compound-datatype bug reached a release. CLAWHDF5_REQUIRE_INTEROP=1 still turns a skip into a failure.

v2.5.0 (2026-09-19)

Upgrade Notes

• Retrieval rankings change, for the better. The default fusion weights move from 0.7/0.3 to 0.4/0.6 (hybrid::DEFAULT_FUSION), measured over the full LongMemEval haystack: turn-level Hit@1 51.6% vs 44.2%, MRR 0.643 vs 0.586. unified_search and the OpenClaw backend pick this up automatically; callers passing weights to hybrid_search explicitly are unaffected.
• Out-of-range selections are now errors. read_*_selection used to return data for a selection that ran past a dataset edge — a hyperslab came back zero-padded, and a point with an out-of-range coordinate wrapped into the next row. Both are now FormatError::SelectionOutOfBounds. Code relying on the old (wrong) values will start seeing errors.
• Large compressed datasets written without explicit chunk dimensions get a different layout. They used to be stored as one chunk; they are now split to ~1 MiB chunks. The files stay standard and h5py-readable, and explicit with_chunks is unaffected.
• rayon is now a default dependency of clawhdf5-agent (the parallel index build). Opt out with --no-default-features --features float16,hnsw.
• clawhdf5-ann search results no longer shrink when records near the query have been deleted, so a search that previously returned fewer than k results now returns k.

Retrieval quality

• clawhdf5-agent: optional keyword stemming — bm25::TokenFilter::Stemmed and HDF5Memory::set_token_filter, so "training" and "trains" match. Off by default, on measurement rather than principle: over the full LongMemEval haystack it buys depth and costs the top rank (BM25 alone: Hit@5 +2.8pp, Hit@10 +2.4pp, Hit@1 −1.8pp, MRR unchanged), and on the shipping hybrid configuration the trade is narrower still. See BENCHMARKS.md.
• clawhdf5-agent: QueryExpander::expand panicked on ordinary non-ASCII input — "İ AI" was enough. It searched a lowercased copy of the query and then sliced the original with those offsets, which only works while lowercasing preserves byte length (Turkish İ is 2 bytes and lowercases to 3). Depending on where the offsets drifted it either corrupted the output ("İstanbul AI trip" lost a character) or panicked. Matching now walks the original string.
• clawhdf5-agent: query expansion no longer rewrites text inside words. replace_word_case_insensitive did a plain substring replace despite its name, so "training" became "trArtificial Intelligencening" and "programming" became "Pull Requestogramming" — every acronym expansion of ordinary prose was corrupt. Matches now require word boundaries; genuine acronyms (API, database) still expand.
• clawhdf5-agent: the default fusion weights are now the measured ones. A sweep of every 0.1 step over the full LongMemEval haystack (500 questions, real MiniLM embeddings) shows the long-standing 0.7/0.3 default is strictly dominated by 0.4/0.6 — turn-level Hit@1 51.6% vs 44.2%, Hit@5 81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.643 vs 0.586, and better at session level too. The finding was recorded in BENCHMARKS.md but had never been applied: unified_search and the OpenClaw backend both hardcoded 0.7/0.3. They now use hybrid::DEFAULT_FUSION. Callers passing weights to hybrid_search explicitly are unaffected — pass 0.4/0.6 (or use hybrid_search_with) to get the tuned behaviour.
• clawhdf5-agent: fusion is now selectable. New hybrid::Fusion (Weighted { vector, keyword } or Rrf { k }), hybrid::fuse, hybrid::hybrid_search_fused and HDF5Memory::hybrid_search_with. Reciprocal rank fusion existed but was unreachable from the store, so it had never been measured against the weighted sum; the LongMemEval bench now has an RRF mode.

HDF5 Read Path

• Selection reads cost what the selection costs. read_*_selection decoded the entire dataset and then picked elements out, so a 64 x 64 window of a 64 MB compressed dataset took 105 ms - about as long as reading all of it. Now only the rows (contiguous) or chunks that overlap the selection's bounding box are read and decompressed: that window takes 0.39 ms, one row 2.7 ms, one column 5.2 ms. Results are identical to the full-read path (equivalence-tested over random hyperslabs and point lists, ranks 1-3, contiguous / chunked / deflate). New read_harness bench binary.
• Faster full reads (same-moment A/B, 64 MB f64): chunked + deflate 110 -> 69 ms, chunked 72 -> 60 ms, contiguous 56 -> 30 ms. The facade's cached read path now decompresses cache misses in parallel batches (it was sequential; only the uncached reader was parallel) and caches only datasets that fit the chunk cache; unfiltered chunks are copied straight from the file bytes; a contiguous dataset is converted straight from the file bytes; and the native-endian conversions no longer zero a buffer before overwriting it.
• Datasets indexed by a version-2 B-tree now read (layout v4, chunk index type 5 — what libver='latest' uses for two or more unlimited dimensions; previously "unsupported chunked layout"). The four copies of the chunk-index dispatch are now one shared function, so every read path gets it.
• H5T_STD_REF references (HDF5 1.12+, datatype message version 4) parse: ReferenceType gains Object2, DatasetRegion2 and Attribute, and read_object_references decodes the new object references. Previously any dataset of this type failed with InvalidReferenceType(2). Tested against a file written by HDF5 2.0 itself (fixture + generator script committed).
• Automatic chunk sizes. Asking for compression (or any filter) without with_chunks used to store the whole dataset as one chunk, so any read had to decompress everything and nothing could be decoded in parallel. Datasets up to 1 MiB stay a single chunk, as before; larger ones are split by halving the dimensions in turn until a chunk is at most 1 MiB (the approach h5py takes). Behaviour change: large compressed datasets written without explicit chunk dimensions get a different (standard, h5py-readable) layout. Explicit with_chunks is unaffected.
• Out-of-range selections are errors. They used to return data: a hyperslab past an edge came back padded with zeros, and a point whose column was out of range wrapped into the next row and returned that element. Now FormatError::SelectionOutOfBounds (also for a rank mismatch or overlapping blocks).

Search

• clawhdf5-ann: faster index builds. Back-link pruning is 90% of a build's distance evaluations; the bulk build now inserts in batches and prunes each overflowing neighbour list once per batch (10K: 1676 -> 1074 ms). With the parallel feature, planning and pruning run on a thread pool (10K: 388 ms, 100K: ~21 s -> 5.9 s on 16 cores). The graph is deterministic and identical with or without the feature. clawhdf5-agent's parallel feature enables it for the agent's index and is now on by default (adds rayon to the default dependency set; build with --no-default-features --features float16,hnsw to opt out).
• clawhdf5-ann: HnswIndex::search returned fewer than k results — often none — when the records nearest the query had been deleted: it collected ef candidates, then dropped the deleted ones, then took k. Deleted nodes are now traversed as waypoints but never occupy a result slot, so a search returns the k nearest live records. Matters for any store that deletes or supersedes memories without compacting straight away.

v2.4.0 (2026-09-19)

Upgrade Notes

• Search results improve on upgrade. The HNSW index now reaches true neighbours it previously could not (recall@10 0.31 -> 0.98 at 100K records on clustered data), so hybrid_search rankings change for the better. The agent rebuilds its index from the store automatically; a standalone HnswIndex persisted with to_hdf5_bytes keeps its old graph until rebuilt.
• hybrid_search no longer writes the store. Hebbian activation boosts are persisted by the next checkpoint (any flushing write, flush_wal, or when the HDF5Memory is dropped) instead of inside every query; a crash before then forgets only the boosts since the last checkpoint. Activation weights are now capped at 16.
• A new sidecar file, <store>.h5.ann, holds the vector index graph. It is derived data: safe to delete (the index is rebuilt), copied by snapshot(), and worth including when copying a store by hand to avoid a rebuild.
• BM25Index no longer caches IDF and gained add_document, remove_document, pad_to, scores, len and is_empty; results are now deterministic (ties break by record id).

Search

• clawhdf5-ann: HNSW recall fix. Neighbours were chosen as the plain closest-M, which on clustered data (what embeddings look like) turns each cluster into an island: recall@10 was 0.87 / 0.67 / 0.31 at 1K / 10K / 100K vectors and did not improve with ef. The index now uses the HNSW paper's diversity heuristic (Algorithm 4 with kept pruned connections) when linking a new node and when pruning back-links: recall@10 at ef = 64 is 1.00 / 1.00 / 0.98 and responds to ef. Builds are slower (~3.5x at 10K). Existing persisted indexes keep their old graph until rebuilt; the agent rebuilds its index from the cache, so stores pick this up automatically.
• clawhdf5-agent: hybrid_search is 23-39x faster in steady state (p50 5.5 -> 0.24 ms at 1K records, 49 -> 2.1 ms at 10K, 884 -> 23 ms at 100K). Every query used to rebuild the BM25 index from scratch and rewrite the whole .h5 file. The keyword index now lives for the life of the store and is updated incrementally (add / remove / in-place update, exactly equivalent to a fresh build - property-tested), and a query no longer writes the store. Behaviour change: Hebbian activation boosts are persisted by the next checkpoint (any flushing write, flush_wal, or drop) rather than immediately; a crash in between forgets only the boosts since the last checkpoint. Activation weights are now capped (16.0) - they grew without bound.
• clawhdf5-agent: the vector index is persisted, so open() no longer rebuilds it on the first search (first query after open: 2627 -> 15 ms at 10K records, 36 s -> 159 ms at 100K). The HNSW graph — not the vectors, which the store already holds — is written to <store>.h5.ann at each checkpoint and tied to it by a generation id in /meta; a missing, stale, damaged or structurally invalid sidecar is ignored and the index rebuilt. Records replayed from the WAL join the loaded index incrementally; a replayed update or delete invalidates it. snapshot() copies it. Batch saves no longer force a full index rebuild.
• clawhdf5-ann: faster HNSW build and search with identical recall. The cosine metric stores unit vectors and compares them with a plain dot product (it re-derived both norms on every distance evaluation), and the per-call HashSet of visited nodes is a reusable epoch-stamped array. Build 2.75 -> 1.89 s at 10K and ~38 -> 21 s at 100K; QPS at ef = 64 22.7K -> 39K at 10K. Distances returned by search are unchanged (1 - cosine). Indexes loaded from older HDF5 files are normalised on load.
• clawhdf5-accel: the SIMD backend is detected once per process instead of on every kernel call.
• clawhdf5-ann: HnswIndex::graph_to_bytes / from_graph_bytes — graph-only serialization (checksummed, every neighbour id and level validated on load).
• clawhdf5-agent: a further 4-5x on hybrid_search with identical rankings (p50 now 0.07 / 0.49 / 4.65 ms at 1K / 10K / 100K — 79x / 100x / 190x faster than v2.3.0). Fusion needs every keyword score but not their ranking: new BM25Index::scores returns them unsorted from a dense accumulator (it hashed every posting, then sorted every match), and merge_vector_keyword selects its top k instead of sorting every candidate. Capping the keyword candidate pool was measured and rejected: it changes the top-10 for most queries (search_harness --fusion-study).
• clawhdf5-agent: BM25 results are deterministic (ties break by record id), top-k uses a bounded heap, and the "WAND early termination" that computed a bound and then ignored it is gone. IDF is computed per query.
• clawhdf5-bench: new search_harness binary — HNSW recall@10 / QPS / latency per ef against an exact scan, and end-to-end hybrid_search timings, on deterministic clustered (or --uniform) data. Baseline in BENCHMARKS.md.

v2.3.0 (2026-09-19)

Upgrade Notes

• A memory store now has a single writer. HDF5Memory::create/open take an exclusive lock (<store>.h5.lock); a second open of the same store — in the same or another process — returns MemoryError::Locked. Code that opened a second handle just to read should use HDF5Memory::open_read_only.
• Unsigned array attributes arrive as AttrValue::U64Array, not I64Array, and attrs() may now return AttrValue::Raw. Exhaustive matches on AttrValue need the two new arms.
• WAL header version 3 → 4. v3 files are read and upgraded in place, but a store written by 2.3.0 with a pending WAL cannot be opened by 2.2.0 or earlier (it is refused, not corrupted). Checkpoint first (flush_wal) if you need to downgrade.
• MemoryConfig::compression now uses deflate unless the agent's new zstd feature is enabled; it previously failed outright in a default build.
• MemoryError gained Locked; FormatError gained UnresolvedSharedMessage, ExternalDataFilesUnsupported and ExternalLinkUnsupported; MessageType gained ExternalDataFiles.

Bug Fixes

• clawhdf5-format: compound datatypes written with default libver bounds (datatype message version 1 — what plain h5py.File(path, 'w') produces) were mis-parsed. The v1 member layout carries 28 bytes of legacy array fields after the byte offset (the parser skipped 24), and v2 pads member names to 8 bytes and has no array fields at all (the parser did neither), so every member after the first byte offset was read from the wrong position — typically surfacing as Overflow("compound member ...") on read. Found by adding a default-libver axis to the h5py interop tests; byte-level regression tests for v1 and v2 added.
• clawhdf5-gpu: gpu_tests could hang forever under the default parallel test runner — every test created its own wgpu instance and device at once. Tests now serialise GPU access, and GPU→CPU readback waits are bounded (30 s) so a wedged driver returns GpuError::BufferMap instead of blocking.
• clawhdf5-agent: benches/bench.rs and benches/memory_bench.rs no longer compiled against the current strategy/consolidation APIs.

HDF5 Compatibility

• clawhdf5-format/clawhdf5: datasets and attributes that use a committed (named) datatype now read correctly. They store a shared-message reference; the facade parsed the reference bytes as the datatype (Time { size: 0 }, unreadable data) and silently dropped such attributes. The shared-reference parser itself was wrong for real files: version 2 has no reserved bytes, and the version 3 types were inverted (1 = SOHM heap, 2 = committed).
• Fill values are applied on read. There was no Fill Value message parser: the holes of a sparse chunked dataset read as zeros even when the fill value was not zero (silently wrong data), and a dataset that was created but never written failed with NoDataAllocated where h5py returns a filled array. Messages v1–v3 and the old 0x0004 form are parsed; the fill value is written into exactly the chunk-grid cells missing from the chunk index.
• Soft links are followed during path resolution, in old- and new-style groups (absolute/relative targets, links to groups, links through links), with a depth limit so a link cycle is an error rather than a hang. A dangling link reports the target it could not find.
• Things the reader does not follow are now explicit errors instead of wrong answers: an external link is ExternalLinkUnsupported { filename, object_path } (was PathNotFound), and a dataset whose raw data lives in external files (message 0x0007, now a known MessageType) is ExternalDataFilesUnsupported (it would otherwise read as fill values).
• attrs() no longer drops attributes. Any attribute whose datatype had no AttrValue variant was omitted with no error — including every Python bool (h5py stores attrs["flag"] = True as an enum), complex numbers, compound values and object references. Now:
  • numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1) decode as I64 / I64Array of 0/1;
  • new AttrValue::U64Array keeps unsigned arrays unsigned (they were cast to I64Array, so values above i64::MAX came back negative). Behaviour change: code matching I64Array for an unsigned attribute must also match U64Array (the netCDF-4 CF helpers and Python bindings do);
  • new AttrValue::Raw { datatype, shape, data } carries everything else verbatim, decodable with clawhdf5_format::data_read against datatype. Both new variants are writable, so an attribute can be copied between files unchanged. Python receives Raw as {"dtype", "shape", "data"}.
• All of the above are covered by h5py interop tests under both default and libver='latest' bounds, compared against h5py's own readback.

Security

• clawhdf5: virtual-dataset source file names are untrusted input but were joined straight onto the opened file's directory, so a crafted file could make the reader open any path the process can reach (absolute path, or .. components). Only plain relative paths inside that directory are accepted.

Durability & Integrity

• clawhdf5-agent: a crash between writing a checkpoint and truncating the WAL no longer duplicates every pending entry on the next open. Each checkpoint records a WalMark (byte length + chained CRC of the WAL prefix it folded in) in /meta; open() skips exactly that prefix when it is still present. No WAL format change for this; older files behave as before.
• clawhdf5-agent: checkpoints and snapshots are durable as a unit — the temp file is synced before the rename and the directory after it. Individual WAL appends remain unsynced by design (documented in CLAUDE.md).
• clawhdf5-agent: save_or_update hits are logged as a new Update WAL record, so replay updates in place instead of appending a duplicate. WAL header version 3 → 4 (so older builds refuse the file rather than truncating a record they can't parse); v3 files are read and upgraded in place.
• clawhdf5-agent: loading validates every per-record dataset length (a truncated store is now MemoryError::Schema, not a later panic), fixes the n.len() == n.len() tautology that trusted a norms dataset of any length, and rejects embedding_dim == 0 with records present.
• clawhdf5-agent: eight behavioural MemoryConfig fields are now persisted in /meta. Previously they reset to defaults on every open — a compressed store was rewritten uncompressed, wal_enabled = false flipped back to true.
• clawhdf5-agent: compression = true never worked in a default build (it requested Zstd without enabling the feature, so every checkpoint failed with unsupported filter: 32015). Default builds now use deflate; Zstd is the new opt-in zstd feature.
• clawhdf5-agent: single-writer lock (<store>.h5.lock, MemoryError::Locked) — two handles on one store used to silently destroy each other's data. New HDF5Memory::open_read_only gives a lock-free, never-writing view; the CLI's read-only subcommands use it.
• clawhdf5-agent: an unreadable WAL (torn header / bad magic) is quarantined (HDF5Memory::quarantined_wal()) instead of blocking open() of a healthy store. A WAL from an unknown newer version still fails and is left intact.
• clawhdf5-agent: provenance records are renumbered on compaction (they weren't, so every later save_or_update raised a false High integrity alert); pending anomaly alerts and tracked sessions are bounded; snapshot() includes entries still in the WAL.
• clawhdf5-agent: hybrid ranking is deterministic (index tie-breaks instead of HashMap order); a set of identical positive scores — including a single candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer reinforces zero-score filler results.
• clawhdf5-format: chunked/VDS/hyperslab reads size their buffers with overflow-checked arithmetic and fallible allocation, so crafted dimensions are FormatError::Overflow instead of a wrapped size or a process abort; parallel_read bounds checks use checked_add.
• clawhdf5: a malformed filter-pipeline message is an error instead of being treated as "no filters" (which returned compressed bytes as data); FileBuilder::write is atomic and synced instead of truncating the destination first.

CI / Testing

• CI now lints every target (cargo clippy --all-targets) plus clawhdf5-format's optional features, compiles all benches, and tests the format feature matrix. Previously test/bench code and feature-gated modules were never linted; the accumulated clippy backlog is fixed.
• CI installs python3 + h5py/numpy/netCDF4/xarray and sets CLAWHDF5_REQUIRE_INTEROP=1, which turns a missing interop dependency into a test failure. Until now every h5py/netCDF4 interop test silently skipped in CI, which is how the HDF5 2.0 compound bug fixed in v2.2.0 reached a user. The #[ignore]d writer_h5py_tests suite is run explicitly.
• h5py-generated-file tests now cover default libver bounds as well as libver='latest' (HDF5 2.0 raised the default low bound to 1.8).
• clawhdf5-agent: WAL property tests (round trip; after any corruption the entries read back are an exact prefix of what was written — 1500 seeded cases), a crash-recovery matrix (an on-disk image after every operation, the checkpoint window, and the WAL torn at every byte length, each reopened and checked against a model), and a WAL fuzz target.
• Optional fuzz smoke run (CLAWHDF5_FUZZ_SECONDS=N scripts/ci-test.sh); new datatype corpus seeds for v1 compound and native complex messages.

v2.2.0 (2026-09-18)

Security

• clawhdf5-format: bounded decompression output (MAX_DECOMPRESS_SIZE) for deflate/lz4/zstd/pcodec so a crafted compressed chunk can't drive an unbounded allocation (memory-exhaustion DoS).
• clawhdf5-format: chunked_read.rs/data_read.rs/local_heap.rs bounds audit — added ensure_len overflow guards at every plain-arithmetic offset+size check, a recursion-depth guard against a crafted self-referencing/cyclic B-tree chunk index, a fix for an unguarded compound-datatype byte_offset overrun in read_compound_fields, and an ndims - 1 underflow guard for degenerate zero-dimension chunked layouts. Added a new fuzz_dataset_read cargo-fuzz target (walks every dataset in a parsed file and exercises the contiguous/chunked/compact raw-data read paths) which found and fixed 3 real crash bugs — an integer-multiply overflow in copy_chunk_to_output's N-D assembly path, the ndims - 1 underflow above, and an overflow in local_heap.rs — within the first few fuzzing runs.
• clawhdf5-format: btree_v1.rs overflow-safe bounds checks via a local ensure_len helper, closing a usize-overflow panic reachable from a crafted near-usize::MAX B-tree offset.
• clawhdf5-agent: WAL length-prefix caps (MAX_WAL_FIELD_LEN, 64 MiB) reject a corrupted/truncated length claim before allocating. Followed by a full per-entry CRC32 trailer (WAL_VERSION bumped to 2) — a bit-flip inside an entry now stops replay cleanly instead of silently accepting corrupted data. Old-format WAL files are still read correctly and migrated to the new format on next open.
• clawhdf5-android: validate embedding_len/query_embedding_len against the handle's configured embedding_dim (and reject null pointers) before constructing a slice from a raw pointer in edgehdf5_save / edgehdf5_hybrid_search.
• clawhdf5-py: bump pyo3/numpy 0.28 → 0.29, clearing two RUSTSEC advisories (OOB read in PyList/PyTuple iterator; missing Sync bound on PyCFunction::new_closure).
• Clarified that the integrity hashes in clawhdf5-agent::provenance (FNV-1a) and clawhdf5-format::provenance (SHA-256) are unkeyed and detect only accidental corruption, not tampering — doc-only change, no behavior change.

Performance

• clawhdf5-format: chunk cache lookup is now O(1) (slot_index: HashMap) instead of a linear scan, and cache hits return a shared Arc instead of cloning the decompressed buffer — the hottest path in chunked reads.
• clawhdf5-ann: optional parallel feature (rayon) parallelizes HNSW's prune_connections neighbor-distance computation. The outer build/insert loop is deliberately left sequential — it has genuine cross-iteration data dependencies and needs its own correctness-focused design pass.
• clawhdf5-format/chunked_read.rs: removed 12 unnecessary chunk_dimensions[..rank].to_vec() allocations where callees already accept &[u32].

Architecture

• Added .gitea/workflows/ci.yml, actually wiring the long-existing scripts/ci-test.sh (fmt, clippy, tests, no_std check) into CI on every push/PR to main. Fixed stale package names in ci-test.sh/ check-nostd.sh that had been silently no-op'ing the clawhdf5-py exclusion and the no_std check.
• Fixed a genuine no_std build break in clawhdf5-format (uncovered once the no_std CI check actually started running): core::sync::atomic::AtomicU64 doesn't exist on thumbv7em-none-eabihf (switched to portable-atomic), missing alloc imports for Box/Vec/format! on a few no_std paths, and f64::powi (std/libm-only) replaced with a local exponentiation-by-squaring helper in the scale-offset filter.
• Added [workspace.dependencies] for tempfile/criterion/half/serde, fixing a real version skew on half (2 vs 2.7 across crates).
• Fixed version skew: clawhdf5-py (pyproject.toml) and packages/clawhdf5-node (package.json) were both behind the actual crate version (2.1.0).
• Documented that the mpi-io feature's read/write paths are root-read +broadcast / gather-to-rank-0, not true collective I/O.

Documentation

• BENCHMARKS.md: re-ran the previously-undated "LongMemEval Results", "SIMD & Parallelism", and "Vector Search Latency"/"Comparison to MemX" sections on a second machine (tank, Ryzen 7 7800X3D) with explicit dates and reproduce commands. Found and corrected a methodology issue in the SIMD/Parallelism benchmark selection (several originally-compared benchmarks didn't actually isolate the scalar/SIMD/parallel axis).
• README.md / ROADMAP.md / CLAUDE.md: corrected several stale facts — the clawhdf5-types crate (removed earlier) was still listed in the README crate map; the LongMemEval numbers in the README badge and table didn't match the actual (much better) benchmark results in BENCHMARKS.md; total line-of-code and test-count figures were stale; clawhdf5-gpu's CubeCL→wgpu correction; documented the new clawhdf5-ann parallel feature flag, which had no entry in the Feature Flags table.

New Features

• clawhdf5-migrate: substantial engine improvements:
  • Real content validation — the post-migration check now reads the written HDF5 back and compares actual content (chunk text, embeddings, and every session/entity/relation field) against the source, not just row counts. A representative sample of chunk rows is verified by default; --validate-full checks every row. A corrupt migration that preserves counts no longer passes.
  • Configurable schema — table names are no longer hardcoded; queries are built from a SchemaConfig (table + ordered column names, defaulting to the ZeroClaw layout) with --chunks-table / --sessions-table / --entities-table / --relations-table overrides.
  • Streaming count pass — --dry-run now does a COUNT(*)-only pass per table instead of loading every row into memory.
  • Incremental migration — --incremental reads the existing output, reads only source chunks newer than the last migrated id, and appends them (refreshing the metadata groups), instead of re-migrating everything.
• clawhdf5-format: read IEEE-754 half-precision (f16) floats. read_as_f32 / read_as_f64 previously only handled 4- and 8-byte floats; 2-byte floats (e.g. float16-stored embeddings) now decode via a no_std-safe bit conversion.
• clawhdf5-format: write multi-block fractal heaps (root indirect block). Dense attribute and dense link storage previously capped at a single direct block (~64 KiB of heap data — a few thousand attributes/links). When the objects exceed one direct block, the heap now lays out a root indirect block (FHIB) over multiple direct blocks sized by the doubling table, distributing objects across blocks with correct per-block heap offsets. Validated end-to-end: a 2,500-attribute object and a 2,500-link group round-trip through our reader and are read correctly by h5py. (Objects still may not span a block — no huge-object path.)
• clawhdf5-format: write dense group link storage (fractal heap + v2 B-tree). A group with more than 8 links (libhdf5's compact max_compact default) is now written densely — its links live in a fractal heap indexed by a v2 B-tree of type 5 (link-name index) referenced from the group's LinkInfo message — instead of as inline Link messages. This matches libhdf5's compact→dense switchover and keeps large groups out of the object header. Reverse-engineered against libhdf5: link heaps use heap_id_length 7 / max_heap_size 32 (vs 8 / 40 for attributes). The shared single-direct-block fractal-heap builder is now parameterized and used by both dense attributes and dense links. Validated end-to-end: our reader round-trips, and h5py reads the dense groups we write. (Single direct block — up to ~a couple thousand links per group; beyond that needs indirect blocks, still unsupported.)

Robustness

• clawhdf5-format: harden the readers added this cycle against malformed / hostile input — they parse untrusted bytes and must return errors, never panic, OOM, or recurse without bound. Fixed concrete vectors found by audit and locked in with adversarial tests:
  • Paged Fixed Array: 1 << max_nelmts_bits shift overflow (a u8 ≥ 64); element/page offset multiplications now checked; element count bounded by file size.
  • H5S selection decoder: ALL/NONE no longer claim 16 bytes they don't have; hyperslab rank capped at 32 (H5S_MAX_RANK) to stop a giant allocation; iter_linear coordinate/stride/product arithmetic is checked.
  • VDS mapping parser: no pre-allocation from the untrusted nused; all selection slicing is bounds-checked.
  • scale-offset / N-Bit filters: 1 << minbits overflow at minbits == 64; N-Bit bit_offset + precision overflow; N-Bit type-tree recursion depth capped (no stack overflow from a crafted nested tree); element counts bounded by the chunk's expected decompressed size so a bogus count can't drive a huge allocation.
  • Virtual Dataset assembly: a virtual dataset whose source is itself virtual (a cycle) now errors instead of recursing into a stack overflow.

New Features

• clawhdf5-agent: compress fixed-length string datasets (memory text chunks, session summaries, ids, tags, entity/relation names, …). These were always stored uncompressed with a "chunked compound not yet supported" note that was simply stale — chunked writes work for fixed-size string/compound datatypes like any other. write_string_dataset now chunks + deflates a string dataset once its payload reaches 4 KiB, so large, highly-redundant NullPad content shrinks substantially while tiny metadata stays contiguous (no chunk-overhead bloat).
• clawhdf5-format: decode the scale-offset filter (id 6) — both the integer variant (H5Z_SO_INT) and the floating-point D-scale variant (H5Z_SO_FLOAT_DSCALE). Handles signed/unsigned int sizes, f32/f64, negative minima, decimal scale factors and fill values; reverse-engineered against HDF5 2.0 and validated end-to-end. The float E-scale variant remains unsupported.
• clawhdf5-format: decode the N-Bit filter (id 5) — atomic, compound and array layouts (the full recursive type tree, nestable to any depth), previously unsupported. Signed and unsigned reduced-precision integers and float members all read end-to-end, validated against HDF5 2.0.

New Features

• clawhdf5 / clawhdf5-format: read external-file Virtual Datasets (VDS). The format layer gains read_raw_data_full_with_resolver and a VdsSourceResolver callback (Fn(&str) -> Option<Vec<u8>>) that maps a stored source file name to its bytes, so the pure-byte reader can pull in external sources without a filesystem of its own. The clawhdf5 File API wires a default resolver that reads sibling source files relative to the opened file's directory, so File::open(...).dataset(...).read_*() now transparently assembles cross-file VDS. A source file the resolver cannot supply leaves its region at the fill value (matching HDF5); an external source with no resolver at all is a clean error. In-memory files (File::from_bytes) have no directory, so only same-file VDS resolves there.
• clawhdf5-format: assemble same-file Virtual Datasets (VDS) of any rank. Previously a virtual layout returned UnsupportedVersion. The reader now decodes the global-heap mapping block (reverse-engineered against HDF5 2.0: version · nused · [source-file · source-dataset · source-selection · virtual-selection]* · checksum, including the block-version-1 same-file marker), decodes the H5S source/virtual dataspace selections (ALL, NONE, and version-3 regular hyperslabs), reads each same-file source dataset, and scatters its selected elements into the virtual buffer in row-major order (so multi-dimensional block mappings land correctly); unmapped regions are left at the zero fill value. External-file sources return a clean unsupported error. The previous parse_vds_mappings used a guessed layout that did not match real files and is replaced.

Tests

• clawhdf5-format: regression test for scale-offset float E-scale datasets. The HDF5 library does not implement E-scale encoding — when asked for it (cd_values[0] = 1) it stores the chunk raw and sets the chunk filter mask to skip the filter — so these files read back verbatim purely by honoring the per-chunk filter mask. The test locks in that behavior against a fixture produced via the HDF5 low-level API; no E-scale decoder is needed.

Bug Fixes

• clawhdf5-format: read multi-direct-block fractal heaps. The reader split direct vs indirect block rows using the FRHP "Starting # of Rows in Root Indirect Block" field (a constant, typically 1), so any heap whose data spans more than one direct block — common in libhdf5 files with a large group or many dense attributes — was misread as having indirect blocks and failed with InvalidFractalHeapSignature. The split is now derived from the heap geometry (max_direct_rows = log2(max_direct / start) + 2). Validated against an h5py-written 400-dense-attribute group (root indirect block, 4 rows, 13 direct blocks).
• clawhdf5-format: scope the per-file chunk cache by dataset. The shared ChunkCache built its chunk index once and reused it for every chunked dataset in the file, keyed only by chunk coordinate with no dataset discrimination. With a single chunked dataset per file this was latent; once a file holds two chunked datasets of different rank (e.g. a 1-D compressed string array and the 2-D embeddings matrix), the first dataset's index was reused for the second, panicking with an out-of-bounds chunk coordinate. The cache now rebinds (dropping its index, chunk-index map, layout, and decompressed slots) whenever the dataset being read changes, while still caching repeated/sequential access to the same dataset.
• clawhdf5-format: read paged Fixed Array chunk indexes. A filtered, fixed-dimension dataset with more than one data-block page (>1024 chunks by default) previously failed with "paged Fixed Array data blocks not yet supported". The reader now walks the page-init bitmap (MSB-first), skips uninitialized pages, and resolves each page's fixed full-size slot (including the short final page). Reverse-engineered and validated end-to-end against an HDF5 2.0 file.
• clawhdf5-format: read array-typed datatypes (e.g. an array-typed compound member) via read_as_i32/i64/u64/f32/f64 — previously a TypeMismatch. The array is read as a flat sequence of its base elements (recursing for nested arrays), applying base-type precision rules.
• clawhdf5-format: sign-extend reduced-precision fixed-point integers on read. A signed integer whose datatype precision is smaller than its storage size is stored zero-filled, so e.g. a 16-bit-precision -1 previously read as 65535. The integer read paths now extract the precision field and sign-extend (full-width types are unchanged). Completes signed N-Bit reads and also fixes un-filtered reduced-precision integer datasets.
• clawhdf5-format: read datasets written by modern HDF5 (1.14+/2.0, i.e. libver=latest). Compound (class 6) and array (class 10) datatype version 5 messages and data layout version 5 messages were rejected as invalid; they reuse the v3/v4 binary structure, so they are now accepted. This unblocks reading compound types and — critically — every chunked/compressed dataset written by HDF5 2.0. Found by running the h5py interop tests against h5py 3.16 / HDF5 2.0. Independently reported (with a patch) against the v2.1.0 tag by M. Scot Breitenfeld (The HDF Group) — v2.1.0 predates this fix.
• clawhdf5-format: parse HDF5 2.0 native complex datatypes (class 11, datatype version 5, e.g. H5T_COMPLEX_IEEE_F64LE). The properties are a single base floating-point datatype, not a compound-style member list; the old parser read the base type's bytes as member names, producing a garbage datatype, and failed with UnexpectedEof when a complex type was nested in a compound. It is now surfaced as the equivalent {r, i} compound (the shape h5py writes for numpy complex dtypes), with a size check against the base type. Validated end-to-end against an HDF5 2.0-written file.

Performance

• clawhdf5-format: chunked writes now compress all chunks up front via compress_all_chunks, running across rayon threads under the parallel feature when there are more than 4 filtered chunks. On-disk layout is unchanged. Speeds up compressed embedding writes in clawhdf5-agent (which enables parallel).

Documentation

• Fix stale package names across all 13 per-crate READMEs (rustyhdf5-* / edgehdf5-* → clawhdf5-*, usage versions → 2.1.0).
• Correct README workspace/test/crate stats and the CLAUDE.md CLI subcommand list; document the hnsw and format compression/checksum feature flags and the entity_extract / async_memory modules.

v2.1.0 (2026-06-03)

New Features

• clawhdf5-agent: HNSW now backs the vector stage of hybrid_search. The hnsw feature is on by default, so semantic search uses the approximate clawhdf5-ann index instead of a linear cosine scan. The index mirrors the memory cache (node id == cache index) and self-heals — it rebuilds whenever it drifts from the cache length, so no mutation path can desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and dimension-mismatched queries fall back to the exact linear scan. Disable with --no-default-features --features float16 for exact search.
• clawhdf5-ann: HNSW is now a live, mutable index — added insert, mark_deleted (soft-delete bitset; deleted nodes are traversed for connectivity but never returned), compact (drops deleted vectors and renumbers survivors), and new (empty index). Serialization gains a format version tag (HNSW_FORMAT_VERSION = 2) and persists the deleted bitset; pre-existing v1 files still load.
• clawhdf5-agent: hybrid::merge_vector_keyword exposes the shared normalize-and-fuse step used by both the linear and HNSW vector paths.
• Expose max_dimensions() API on Dataset, MmapDataset, and LazyDataset
• NetCDF-4 unlimited dimension detection now works correctly
• Python bindings (clawhdf5-py) build and link on macOS with system Python

Bug Fixes

• clawhdf5-py: upgrade PyO3 and numpy 0.23 → 0.28 so the bindings build on Python 3.14 (PyO3 0.23 capped at 3.13 and hard-failed cargo build --workspace). Updated for the removed PyObject alias (Py<PyAny>) and the Python::allow_threads → Python::detach rename.
• Fix GPU L2 distance test (squared vs actual L2 mismatch in test helper)
• Mark Android JNI functions as unsafe for Rust 2024 edition compliance
• Add # Safety documentation to all public unsafe extern functions
• Fix all clippy warnings: needless_range_loop, manual_strip, ptr_arg, etc.
• Rename RelationType::from_str to from_label to avoid trait confusion
• Isolate h5py interop tests with #[ignore] when h5py unavailable

Code Quality

• Full rustfmt pass across workspace (61 files)
• Refine inner unsafe blocks for Rust 2024 edition style
• Zero clippy warnings, zero clippy errors across entire workspace
• 1,546 tests passing, 0 failures

v2.0.0 (2026-03-19)

• Unified rustyhdf5 (11 crates) and edgehdf5 (4 crates) into a single workspace
• All crates renamed to clawhdf5-* prefix
• Version bumped to 2.0.0 across all crates
• Git dependencies replaced with in-workspace path dependencies
• Added agent feature flag to clawhdf5-agent