clawhdf5/docs/known-issues.md

Known Issues

Bugs found during development or downstream use, tracked here because this repository's issue tracker is disabled. One entry per bug; when an entry is fixed, record the fix in CHANGELOG.md and update its status here rather than deleting it.

---

Concurrent and contiguous read performance (measured 2026-09-26)

Status: open for chunked full reads (one cause fixed 2026-09-26); the contiguous item is fixed (2026-09-26). Measured on tank with concurrent_read against h5py 3.16 / HDF5 2.0 (BENCHMARKS.md, "Concurrent reads"):

• Partly fixed 2026-09-26. Full reads of chunked datasets from several threads through one File stop scaling at about 4 threads (880 MB/s on deflate data vs 4424 MB/s for 16 h5py processes). Hyperslab reads, which skip the chunk cache, scale to 1244 MB/s, so the File's shared chunk cache is the suspect. Cause: not the cache. Those numbers were taken with --decode-threads 1, a one-thread rayon pool, and every full read handed its chunks to that pool, so all reader threads queued behind its single worker (per-thread CPU time: one thread did all the decoding, the 16 readers almost none). Hyperslab reads touch one chunk each and never used the pool. Reads now decode on the calling thread when the pool has one thread (tests/single_thread_decode_pool.rs). Still open: this fixes only a one-thread pool. With the default pool, 16 reader threads ran at about 2900 MB/s before and after the change, still short of 16 h5py processes (4424 MB/s); with a small pool (2-4 threads) readers outside it still wait on its workers. Datasets larger than the cache's budget were already read without inserting into it, and skipping its lookups entirely gained only a few percent at 16 threads. Remaining per-read overhead, not yet addressed: each full read_f32 of a chunked dataset faults in about three times its size in fresh pages (the output, the f32 copy of it, and a new buffer per decoded chunk).
• Contiguous datasets read 4x slower than h5py on one thread (2.5 vs 9.8 GB/s full, 0.12x for 256 x 256 hyperslabs). Fixed 2026-09-26 (not yet re-measured for BENCHMARKS.md): full reads were dominated by 4 KiB page faults on the fresh output buffer, which is now backed by transparent huge pages as numpy's is; hyperslab reads copied the selection three times, element by element, and now copy each contiguous run once, straight from the file into the output (see CHANGELOG.md). The chunked-read scaling item above is still open. Values are correct; this is speed only.

Silent wrong data found by the 2026-09-25 HDF5 audit

Status: fixed after v2.7.0 (2026-09-25). Every release up to and including v2.7.0 is affected.

An audit on tank checked clawhdf5 against libhdf5 in three ways:

• a sweep of 686 public files: the libhdf5 test files, the HDF Group's cve_hdf5 reproducers, and the pyfive, netcdf-c, netcdf4-python, h5wasm, h5py and xarray corpora;
• 567 read cases generated with h5py 3.16 / HDF5 2.0;
• 96 write cases checked with h5py builds linking HDF5 1.10, 1.12, 1.14 and 2.0, plus h5dump 1.14.6.

It found these cases where a value came back wrong without an error:

Area	What happened	Who is affected
Chunk index (read)	Fixed/Extensible Array indexes laid out by the current shape, not the max shape: chunks returned from the wrong place	any file with a max shape larger than its shape and libver='latest' (h5py maxshape=(10, None), (20, 10))
Chunk index (write)	Extensible Array chunks from index 244 on never indexed (read as 0); unlimited dimension not first: data scrambled	files we wrote with one unlimited dimension and > 244 chunks, or e.g. maxshape=(20, None)
4-byte offsets	unfiltered chunked datasets read as zeros	files created with sizeof_addr = 4
Filter mask	any skipped filter skipped the whole pipeline	files with partially filtered chunks (optional filters, direct chunk writes)
Numeric reads	float read as integer returned the bit pattern; narrowing integer reads kept the low bits; bfloat16 decoded as IEEE half	read_i32/read_i64/read_u64 callers on float or wider data; HDF5 2.0 bf16 data
SZIP	garbage or zeros	every libhdf5-written SZIP dataset
Scale-offset	float values 1 ULP off	libhdf5 D-scale float data
Shared fill value	read as zero fill	fill values stored as shared messages
VL sequences	read_vl_bytes truncated non-byte base types	VL int/float sequences
Chunk cache	two threads reading two chunked datasets through one File could get each other's chunks	multi-threaded readers, including Python with the GIL released

The audit also found files we wrote that libhdf5 refuses, now fixed:

• Fixed Array datasets with more than 1 024 chunks.
• Header messages over 64 KiB (large attributes).
• Reference, Opaque, BitField and Time datatypes.
• Files written with with_page_size.
• Several unlimited dimensions.
• A finite max shape larger than the shape.
• An empty-string attribute, which broke every attribute on its object.
• FillTime codes, which were rotated.

Our LZ4 and Zstd output could not be read by libhdf5's registered plugins, and our pcodec filter used Granular BitRound's ID. The details are in CHANGELOG.md under Correctness and Interop.

Before the fix, 419 of the 686 files read correctly and 43 differed from h5py. After it, 448 read correctly and 23 differ. Of those 23:

• 17 are N-Bit float files. The probe compares raw file-type bytes; the typed reader returns libhdf5's values (nbit_custom_float_decodes_like_libhdf5).
• 2 are an h5py bug: VL data with a big-endian base type comes back byte-swapped in h5py, and h5dump agrees with us.
• The rest are object or attribute listing differences.

Update 2026-09-25: the sweep is now in the repo (conformance/run.sh, corpora pinned by commit) and its current numbers are in CONFORMANCE.md, regenerated nightly by .gitea/workflows/conformance.yml. The probe now compares N-Bit floats as the values libhdf5 converts them to, so the N-Bit files above count as identical. Its file list is defined by conformance/list_files.py (697 files: netCDF classic files are left out, and 11 HDF5 files the ad-hoc sweep missed are in). On 42b81d9: 467 identical, 123 our-error, 15 mismatch (2 are the h5py bug above), 92 that libhdf5 cannot read, and no panics, hangs or crashes.

There were no panics, hangs or crashes before or after, including on all 147 CVE and fuzzer files. On some of those files, h5dump 1.14.6 and h5py/HDF5 2.0 segfault or abort.

Gaps found by the 2026-09-25 HDF5 audit (open)

Status: open. These fail with an error; none returns wrong data (the VDS fill-value item that did is fixed).

• Layout message versions 1 and 2 (HDF5 1.6-era files): 84 of the 686 sweep files, InvalidLayoutVersion. This is the largest single gap. Fixed 2026-09-25: versions 1 and 2 are parsed (compact, contiguous, chunked via the v1 B-tree).
• Compound datatype version 1 array members (found with the layout fix; pre-1.4 files such as tarrold.h5): wrong data — the legacy per-member dimensions were skipped, so an array member read as one scalar. Fixed 2026-09-25.
• Virtual datasets:
  • Wrong data: unmapped regions read as 0 instead of the fill value. Fixed 2026-09-25: unmapped elements and missing sources read as the virtual dataset's fill value.
  • %b printf-style source names are not expanded. Fixed 2026-09-25: printf-style and unlimited mappings are read, and the extent is recomputed from the sources as libhdf5 does. Still open: the "first missing" view and a printf gap other than 0 (libhdf5 access properties we always read at their defaults), source-to-virtual type conversion other than a byte swap, nested virtual sources, and source files outside the virtual file's directory (refused with an error).
  • Hyperslab selection versions 1 and 2 are refused. Fixed 2026-09-25: versions 1-3 and irregular hyperslabs are decoded.
  • The version-1 mapping list written with a 2.0 low bound (flags byte, shared names) was misparsed. Found and fixed 2026-09-25.
• Files with a user block: the base address is not applied. Fixed 2026-09-25: every reader views the file from the superblock on (twithub.h5, twithub513.h5, h5clear_fsm_persist_user_*.h5; the twithub files still stop at the user-defined link type below).
• Old-style shared messages (version 1) read the wrong address. Fixed 2026-09-25: the address follows the length-sized link-name offset of the embedded symbol table entry (tcompound.h5, tcompound2.h5).
• Groups and links:
  • Groups with a user-defined link type (e.g. 187) cannot be listed. Fixed 2026-09-25: user-defined links are skipped; the rest of the group lists.
  • Dense groups with more than about 22 000 links cannot be listed. Fixed 2026-09-25: two bugs — fractal-heap child indirect blocks had the wrong row count, and v2 B-tree internal nodes at depth 3+ were read with the wrong pointer widths.
  • Soft links are left out of datasets(). Fixed 2026-09-25: soft links are listed as their targets; dangling ones are left out.
  • Wrong data (found while fixing user blocks): an old-style group whose local-heap free list points outside the heap listed garbage names where libhdf5 refuses the heap. Fixed 2026-09-25 (InvalidLocalHeapFreeList, checked when a name is first read, as libhdf5 does).
• Dense attributes: a large attribute stored as a fractal-heap "huge" object makes every attribute on the object fail. This affects real NetCDF files (issue671.nc). Fixed 2026-09-25: huge and tiny heap objects, and filtered heaps, are read; and an attribute that still cannot be read is left out of attrs() (reported by attrs_with_errors()) instead of failing the others.
• Other readers:
  • VL-string datasets are not readable through File. Fixed 2026-09-26: read_string reads them (also read_string_bytes, read_string_selection, and on MmapFile/LazyFile), with h5py's values: strings end at a NUL, null elements (heap address 0) are "", and an element at the undefined heap address is an error as in libhdf5 (it read as "" until 2026-09-26); VL sequences of numbers read with read_vlen::<T>(), and VL values inside compounds or AttrValue::Raw attributes decode with File::decode_strings / File::decode_vlen (crates/clawhdf5/tests/vl_data_interop.rs).
  • Variable-length values inside a compound (and VL-string attributes) in a file with 4-byte offsets (sizeof_addr = 4) fail with GlobalHeapObjectNotFound or come back as Raw: these paths assume the 16-byte element of an 8-byte-offset file. The datatype itself reads (it was refused as "member overlaps with previous member" until 2026-09-26). Fixed 2026-09-26: a VL type's element size is the one its datatype message stores (12 with 4-byte offsets), and the global heap is read with libhdf5's header padding (crates/clawhdf5/tests/vl_offset4_interop.rs).
  • Metadata cache images are not supported.
  • x87 long double and binary128 are refused.
  • N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
• Filters: blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not implemented. Fixed 2026-09-26 for LZF (default-on lzf feature), bitshuffle, bzip2 and Blosc 1 (bitshuffle, bzip2, blosc, or plugin-filters for all), read and write, pure Rust; h5ex_d_lzf, h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of 697 ok). Still open: Blosc2 (32026 — hdf5plugin stores each chunk as a Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013); both fail with an UnsupportedFilter error that names the filter, and either can be plugged in with filter_registry::register_filter (32023, Granular BitRound, too, since 2026-09-26 even with the pcodec feature).
• Wrong data: a chunk whose filters decode to fewer bytes than the chunk read with zeros for the missing bytes (any filter; found reviewing the plugin filters). Fixed 2026-09-26: it is an error naming the chunk. A corrupt chunk must never read as zeros. Unfiltered chunks are read at their stored size and are not checked this way.
• Crash: a hostile Blosc chunk (frame size below its header) panicked in builds with overflow checks. Fixed 2026-09-26; the new decoders are fuzzed in the unit tests.
• Header checks: on 12 CVE datasets libhdf5 rejects a corrupt header and we read data anyway. We need stricter header checks. Fixed 2026-09-26 (counted again: 18 objects on the CVE corpus that libhdf5 refuses; some read as wrong data, e.g. a zero chunk dimension read as all fill values): object headers, datatypes, chunk dimensions and chunk-index offsets are checked as libhdf5 checks them, and truncated files are refused. 17 of the 18 now fail as in libhdf5 (conformance on tank, conformance/run.sh --no-fetch, 2026-09-26: 571 of 697 ok). Still read where libhdf5 refuses:
  • cve-2024-32624.h5 /Dset_OBJREF: a dataspace whose storage size overflows 64 bits. File::dataset and shape() succeed (libhdf5 refuses at open); reading the values fails.
  • cve-2020-10810.h5, cve-2020-10812.h5 (whole files libhdf5 cannot open, not among the 18): libhdf5 decodes the superblock extension's File Space Info and metadata-cache-image messages at open and refuses these files; we do not decode those messages at open.
  • Deliberately not refused, because clawhdf5 up to v2.7.0 wrote them: a float sign bit position outside the type, and a size-0 string type.
  • Not refused because current libhdf5 reads it though HDF5 2.0.0 (h5py 3.16) refuses it: a v4 chunked layout whose dimensions are encoded in more bytes than they need (HDFGroup/hdf5@e124c36, 2026-06-05, relaxed that check; clawhdf5 wrote such layouts until 2026-09-26).
  • Not refused because HDF5 2.0 (h5py 3.16) reads them though newer libhdf5 refuses them: bit-field offset/precision outside the type, an unknown variable-length kind, an array type whose stored size is not its element count times its base size.
  • (cve-2024-32616 /group1/dset3 and cve-2025-2309's Comp_OBJREF attribute are h5py/numpy type-mapping failures, not libhdf5 refusals.)
  • h5rs check validates with the library's parsers, so it inherits what they accept: of the 150 CVE and fuzzer files, check --data passes 15, and h5dump 1.14.6 rejects 8 of those (tank, 2026-09-26; 28 and 21 before these checks, 16 and 9 before a VL type's stored element size was checked, which flags cve-2024-32608).
• Writer:
  • Nested groups beyond one level: path-like names are now refused, not created. Fixed 2026-09-26: groups nest to any depth (path names create intermediate groups, as h5py does), with soft, extra hard and external links at any depth and optional creation-order tracking; h5py, h5dump and h5rs check --data read them (crates/clawhdf5/tests/writer_groups_interop.rs, crates/clawhdf5-tools/tests/h5rs_interop.rs). Still missing: a group with more than 65 535 links, or an object with more than 65 535 dense attributes, is an error (the index is one B-tree leaf), and attribute creation order is not tracked.
  • Dense link or attribute storage past 512 KiB of messages was written unreadable (child indirect blocks of the fractal heap written as direct blocks). Fixed 2026-09-26 (it affected 2.7.0 too): tested with 20 000 and 65 535 links and with 8 MB of dense attributes, read by h5py, h5dump, h5rs check and clawhdf5, and h5py can add links to such groups.
  • libhdf5 could not add a link to a group we wrote (no Group Info message). Fixed 2026-09-26.
  • Huge fractal heap objects: in dense storage (more than 8 attributes on an object, or more than 8 links in a group) one attribute or link message over 65 515 bytes is an error.
  • Output that HDF5 1.8 can read.
  • A B-tree v2 chunk index larger than one leaf, so datasets with several unlimited dimensions are limited to 65 535 chunks.

---

Compound datatype message version 5 is not parsed (HDF5 2.0)

Status: fixed on main in a13ff51 (2026-06-03); not in the v2.1.0 tag, which was cut five commits earlier. Ships in the next release.

Reported by: M. Scot Breitenfeld (The HDF Group), 2026-09-08, against v2.1.0.

Summary: clawhdf5-format v2.1.0 rejects any dataset with a compound (struct) datatype written by an HDF5 2.0 library in libver='latest' mode: InvalidDatatypeVersion { class: 6, version: 5 }.

Reproduction (h5py 3.16.0 / HDF5 2.0.0):

import h5py, numpy as np
dt = np.dtype([('x', 'f8'), ('y', 'f8'), ('id', 'i4')])
data = np.array([(1.0, 2.0, 10), (3.0, 4.0, 20)], dtype=dt)
f = h5py.File('compound.h5', 'w', libver='latest')
f.create_dataset('particles', data=data)
f.close()

Committed as crates/clawhdf5-format/tests/writer_h5py_tests.rs::read_h5py_generated_compound (#[ignore]d; needs python3 with h5py on PATH). Run with cargo test -p clawhdf5-format --test writer_h5py_tests -- --include-ignored: v2.1.0 gives 25 passed / 1 failed; main passes everything.

Root cause: the compound (class 6) branch of Datatype::parse (crates/clawhdf5-format/src/datatype.rs) accepted only versions 1–4. Datatype message versions 4 and 5 changed only the Reference and Complex classes, so a v5-tagged compound uses the unchanged v3 member-list layout.

Fix: versions 3–5 are accepted for compound (class 6) and array (class 10) datatypes, and data layout message version 5 is accepted too (needed for every chunked dataset written by HDF5 2.0). Byte-level regression tests: test_compound_v5_from_hdf5_2_0, test_array_v5_from_hdf5_2_0.

Native complex datatype (class 11) is mis-parsed (HDF5 2.0)

Status: fixed 2026-09-18. Found while validating the report above.

Summary: HDF5 2.0 native complex types (H5T_COMPLEX_IEEE_F64LE etc.) were parsed as if they carried a compound-style member list. The properties are actually a single base floating-point datatype, so the parser produced a garbage datatype, or UnexpectedEof when the complex type was a compound member. h5py's default numpy-complex mapping is unaffected (it writes a {r, i} compound); only files using the native type through the C API / h5py low-level API hit this.

Fix: class 11 parses its base type and is surfaced as the equivalent {r, i} compound. Tests: test_complex_v5_from_hdf5_2_0, test_compound_with_complex_member_from_hdf5_2_0, writer_h5py_tests.rs::read_h5py_generated_native_complex.

Revised reference datatype (class 7, version 4) is not parsed

Status: fixed 2026-09-19 for object references; region and attribute references are recognised but not decoded.

Summary: HDF5 1.12+ H5T_STD_REF references use datatype message version 4 with reference types 2-4 (object / region / attribute), which Datatype::parse rejected with InvalidReferenceType. h5py still writes the legacy references, so no file had been available to test against.

Fix: a real file was produced by driving the libhdf5 bundled in the h5py wheel through ctypes (tests/fixtures/gen_std_ref.py -> std_ref_hdf5_2_0.h5). The three new types parse as ReferenceType::{Object2, DatasetRegion2, Attribute}, and read_object_references decodes Object2 elements (type, flags, token size, token = target object header address). External references (flag bit 0) and the region/attribute payloads are errors rather than misreads.

clawhdf5-gpu gpu_tests can hang under the default parallel test runner

Status: fixed 2026-09-19.

Summary: during cargo test --workspace the gpu_tests binary sat idle for 25+ minutes. Every test created its own wgpu::Instance + device (requesting adapter-maximum limits) concurrently, and readback used an unbounded device.poll(Wait).

Fix: tests hold a process-wide lock while they own a device, and GpuAccelerator readback waits time out after 30 s with GpuError::BufferMap.

Compound datatype versions 1 and 2 are mis-parsed (default libver files)

Status: fixed 2026-09-19. Found by adding a default-libver axis to the h5py interop tests.

Summary: any compound dataset written with default libver bounds (plain h5py.File(path, 'w'), datatype message version 1) failed to read, typically with Overflow("compound member 'x': byte_offset(0) + field_size(4136977) ..."). Only libver='latest' files (version 3+) and files written by clawhdf5 itself worked, which is why the existing tests never caught it.

Root cause: Datatype::parse skipped 24 bytes of legacy per-member array fields for v1 where the format has 28 (dimensionality 1 + reserved 3 + permutation 4 + reserved 4 + 4 dimension sizes 16), and treated v2 like v1 minus name padding, whereas v2 keeps the 8-byte name padding and has no array fields.

Attributes with unsupported datatypes are silently dropped

Status: fixed 2026-09-19.

Summary: Dataset::attrs() / Group::attrs() returned only attributes convertible to AttrValue and omitted the rest without any indication — every Python bool (an HDF5 enum), complex, compound and reference attributes. Unsigned 64-bit arrays were also cast to I64Array, turning values above i64::MAX negative.

Fix: booleans decode as 0/1 integers, AttrValue::U64Array keeps unsigned arrays unsigned, and AttrValue::Raw { datatype, shape, data } carries any other attribute verbatim. Both new variants are writable. Still lossy: a multi-dimensional numeric attribute is returned as a flat array (its shape is not reported).

B-tree v2 chunk index (layout v4, index type 5) is not supported

Status: fixed 2026-09-19.

Summary: a chunked dataset with two or more unlimited dimensions written with libver='latest' indexes its chunks with a version-2 B-tree, and reading it failed with unsupported chunked layout version=4, index_type=Some(5).

Fix: record types 10 (unfiltered) and 11 (filtered) are decoded — address, stored size, filter mask, scaled offsets — through the shared chunk-listing function, so full reads, cached reads, partial reads and fill-value handling all work. Covered by an h5py interop test (plain, gzip+shuffle, a 2500-chunk tree with internal nodes, a sparse dataset with a fill value, a hyperslab).

External links and external raw data are not followed

Status: open (by design for now); both are explicit errors.

Summary: a path through an external link returns FormatError::ExternalLinkUnsupported { filename, object_path }, and a dataset created with external=[...] storage returns FormatError::ExternalDataFilesUnsupported. Neither is resolved. If support is added, file names must be confined to the opened file's directory, as the virtual-dataset resolver now does.

---

Python interop suites skip silently when no interpreter has h5py

Status: fixed on main in a29c1b2 (2026-09-19).

On a system where python3 is a PEP 668 "externally managed" interpreter, h5py cannot be installed into it at all, and every interop suite — the h5py writer round-trips, the facade suite, netCDF4, and the reference files — returned false from its availability probe and skipped without failing. CI reported SKIP and a green run. This is the same class of gap that let the compound-datatype v5 bug above reach a release.

The probes now read CLAWHDF5_PYTHON, and scripts/ci-test.sh picks up .venv/bin/python automatically. To restore the coverage on a fresh checkout:

python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4

Set CLAWHDF5_REQUIRE_INTEROP=1 in any automated runner so a missing interpreter is a failure rather than a skip.

---

Crafted B-tree v2 structures crash or exhaust the reader

Status: fixed on main (2026-09-20), after v2.6.0. Every release up to and including v2.6.0 is affected.

B-tree v2 traversal (clawhdf5-format, btree_v2::collect_btree_v2_records) recursed one frame per level with the depth taken from the file, and followed child addresses without checking whether they were shared. Two consequences for anyone reading untrusted files:

• A node that is its own child, under a header claiming 65 535 levels, overflows the stack and aborts the process. The file is under 100 bytes.
• Levels whose children all point at one node below make the traversal visit it fan-out^depth times: ~30 million records from ~5 KB, and memory exhaustion one level deeper.

B-tree v2 backs dense attribute storage, v2 groups, shared object header messages and chunk indexes, so opening an object that uses any of them is enough. Both are now errors: depth is capped at 64, and traversal stops once it has produced more records than the file could physically hold.

---

Crafted global heaps exhaust the variable-length reader's memory

Status: fixed on feat/p2-vl-strings (2026-09-26). Not a regression of that branch: every earlier release is affected through read_vl_strings.

Reading variable-length values kept an owned copy of every object of every global heap collection visited, for the whole read. A file whose collections nest inside one another's object data (32 bytes apart, each element pointing at a different one) made retained memory O(elements × file size): a 744 KB file reached 1.58 GB. Letting every collection's object chain jump to one shared run of tiny objects made the parse time O(elements × objects) too. libhdf5 refuses such files.

Now VlResolver caches where each object lies instead of a copy, drops its cache past a 32 MiB budget, and refuses a collection that overlaps one it has already read (libhdf5 gives each collection its own block, so only a crafted file has them). GlobalHeapCollection::parse (and the new parse_index) also refuse a collection that runs past the end of the file, or an object that runs past the end of its collection. Guarded by crates/clawhdf5-format/tests/vl_heap_bounds.rs, which measures peak heap use with a counting allocator. Still open: a file may point many elements at one large heap object, and a VL-sequence read then returns that object once per element, as h5py would.

---

Extensible Array chunk indexes read back wrong data past the inline elements

Status: fixed on main (2026-09-20), after v2.6.0. Every release up to and including v2.6.0 is affected.

A dataset created with exactly one unlimited dimension (maxshape=(None, ...), the usual append-only/resizable case) is indexed by an Extensible Array. Its index block holds the first idx_blk_elmts chunk entries inline — 4 by default — and everything after that lives in data blocks and super blocks whose layout clawhdf5-format computed incorrectly.

Consequences, by dataset size (1 chunk per element):

chunks	result before the fix
<= 36	correct (inline, plus two data blocks that happened to line up)
37	1 element wrong
400	364 elements wrong
>= ~1000	invalid Extensible Array data block signature

The dangerous case is the middle one: values were returned from the wrong chunks rather than an error being raised. Any reader that accepted the data at face value saw plausible but incorrect numbers.

The root causes were the super block sizing formulas (ndblks and dblk_nelmts each double every other level, a half-step apart), a missing block-offset field in the super block, and a page-init bitmap read from the wrong structure. All four are fixed and covered by interop tests against HDF5 2.0 at sizes that cross each boundary, including paged data blocks.

Files written by this crate were not affected by this read bug, but the writer had its own: it indexed only the first 244 chunks, so later chunks read back as 0 in libhdf5 and in clawhdf5. See "Silent wrong data found by the 2026-09-25 HDF5 audit" below.

Every f32 dataset we wrote was unreadable by h5py / libhdf5

Status: fixed 2026-09-23, after v2.7.0. Every release up to and including v2.7.0 is affected — the encoder was already wrong in v2.1.0.

The floating-point datatype message carries the position of the sign bit (bits 8–15 of its class bit field). clawhdf5-format wrote 63 for every float, which is correct only for f64. libhdf5 validates the field, so opening any f32 dataset written by this crate failed:

KeyError: 'Unable to synchronously open object (sign bit position out of bounds)'

That covers every agent store (/memory/embeddings, norms and activation_weights are f32). clawhdf5 itself ignores the field on read, and the interop suites only ever wrote f64 from our side, so nothing here noticed.

Fix: the sign position is computed from the type (bit_offset + bit_precision - 1: 15, 31, 63 for half, single, double). Regression tests: float_sign_location_is_the_top_bit_of_the_value (byte level), clawhdf5_writes_f32_h5py_reads and the agent's h5py_reads_every_dataset_of_an_agent_store.

Existing files: an agent store is rewritten in full at every checkpoint, so it becomes readable by h5py at its next checkpoint with a fixed build. Other files with f32 datasets need to be rewritten.

Empty datasets we wrote were unreadable by h5py / libhdf5

Status: fixed 2026-09-23, after v2.7.0. Every release up to and including v2.7.0 is affected.

A dataset with no elements was written with a real file address and a storage size of 0. libhdf5 guards contiguous storage with an overflow check (addr + size <= addr) that is always true when the size is 0, so it rejected the dataset:

KeyError: 'Unable to synchronously open object (invalid dataset size, likely file corruption)'

In practice: every agent store without sessions or a knowledge graph — the /sessions and /knowledge_graph datasets are empty until something is added — could not be read by h5py even once the f32 bug above was fixed. Found by the same agent-store interop test.

Fix: an empty contiguous dataset gets the undefined address (all 0xff), which is what libhdf5 itself writes.

clawhdf5-wasm (browser) limits

Status: open (by design for now; added 2026-09-26).

• The whole file is held in memory: open() takes its bytes. There are no HTTP range reads, so a multi-GB file does not fit a browser tab.
• Compound, reference, opaque, bitfield, time and VL-sequence datasets are refused with an error naming the type; attributes of those types come back as value: null with their dtype.
• No Zstd or SZIP (both link C): such datasets fail with unsupported filter: 32015 / : 4. pcodec is not enabled either.
• External links and virtual-dataset sources in other files cannot be followed (no file system).
• Variable-length string datasets are read by decoding read_selection's bytes with clawhdf5_format::vl_data in the wasm crate; File itself still cannot (see the audit gaps above). (File can since 2026-09-26. Since 2026-09-26 the wasm crate resolves them with the same VlResolver as File and h5rs, so all three return h5py's values.)

The Node.js package (packages/clawhdf5-node) does not work

Status: open (found 2026-09-25). Unpublished; not built or tested in CI.

The TypeScript wrapper over crates/clawhdf5-napi has never run successfully:

• napi-rs converts #[napi(object)] fields to camelCase, but the wrapper reads snake_case (r.line_range, s.total_records, s.working_count, …), so every stats and consolidation field comes back undefined (src/index.ts:76-120).
• It loads ../clawhdf5.node, but napi build --platform produces clawhdf5.<triple>.node; main points at index.js while tsc writes to dist/; napi prepublish expects per-platform packages that are not defined.
• save/saveBatch exist in the napi layer but not in the wrapper, so a TypeScript caller cannot store an embedding at all.
• The WAL for agent.brain is agent.h5.wal (the store uses with_extension("h5.wal")), not agent.brain.wal as the old docs and the test cleanup assume.

It was written for an OpenClaw integration that is not being pursued (see docs/openclaw.md). Fix and add CI, or remove it, before anyone depends on it.