clawhdf5/docs/known-issues.md

Known Issues

Bugs found during development or downstream use, tracked here because this repository's issue tracker is disabled. One entry per bug; when an entry is fixed, record the fix in CHANGELOG.md and update its status here rather than deleting it.

---

Silent wrong data found by the 2026-09-25 HDF5 audit

Status: fixed after v2.7.0 (2026-09-25). Every release up to and including v2.7.0 is affected.

An audit on tank checked clawhdf5 against libhdf5 in three ways:

• a sweep of 686 public files: the libhdf5 test files, the HDF Group's cve_hdf5 reproducers, and the pyfive, netcdf-c, netcdf4-python, h5wasm, h5py and xarray corpora;
• 567 read cases generated with h5py 3.16 / HDF5 2.0;
• 96 write cases checked with h5py builds linking HDF5 1.10, 1.12, 1.14 and 2.0, plus h5dump 1.14.6.

It found these cases where a value came back wrong without an error:

Area	What happened	Who is affected
Chunk index (read)	Fixed/Extensible Array indexes laid out by the current shape, not the max shape: chunks returned from the wrong place	any file with a max shape larger than its shape and libver='latest' (h5py maxshape=(10, None), (20, 10))
Chunk index (write)	Extensible Array chunks from index 244 on never indexed (read as 0); unlimited dimension not first: data scrambled	files we wrote with one unlimited dimension and > 244 chunks, or e.g. maxshape=(20, None)
4-byte offsets	unfiltered chunked datasets read as zeros	files created with sizeof_addr = 4
Filter mask	any skipped filter skipped the whole pipeline	files with partially filtered chunks (optional filters, direct chunk writes)
Numeric reads	float read as integer returned the bit pattern; narrowing integer reads kept the low bits; bfloat16 decoded as IEEE half	read_i32/read_i64/read_u64 callers on float or wider data; HDF5 2.0 bf16 data
SZIP	garbage or zeros	every libhdf5-written SZIP dataset
Scale-offset	float values 1 ULP off	libhdf5 D-scale float data
Shared fill value	read as zero fill	fill values stored as shared messages
VL sequences	read_vl_bytes truncated non-byte base types	VL int/float sequences
Chunk cache	two threads reading two chunked datasets through one File could get each other's chunks	multi-threaded readers, including Python with the GIL released

The audit also found files we wrote that libhdf5 refuses, now fixed:

• Fixed Array datasets with more than 1 024 chunks.
• Header messages over 64 KiB (large attributes).
• Reference, Opaque, BitField and Time datatypes.
• Files written with with_page_size.
• Several unlimited dimensions.
• A finite max shape larger than the shape.
• An empty-string attribute, which broke every attribute on its object.
• FillTime codes, which were rotated.

Our LZ4 and Zstd output could not be read by libhdf5's registered plugins, and our pcodec filter used Granular BitRound's ID. The details are in CHANGELOG.md under Correctness and Interop.

Before the fix, 419 of the 686 files read correctly and 43 differed from h5py. After it, 448 read correctly and 23 differ. Of those 23:

• 17 are N-Bit float files. The probe compares raw file-type bytes; the typed reader returns libhdf5's values (nbit_custom_float_decodes_like_libhdf5).
• 2 are an h5py bug: VL data with a big-endian base type comes back byte-swapped in h5py, and h5dump agrees with us.
• The rest are object or attribute listing differences.

There were no panics, hangs or crashes before or after, including on all 147 CVE and fuzzer files. On some of those files, h5dump 1.14.6 and h5py/HDF5 2.0 segfault or abort.

Gaps found by the 2026-09-25 HDF5 audit (open)

Status: open. These fail with an error; none returns wrong data, except the VDS item, which is marked.

• Layout message versions 1 and 2 (HDF5 1.6-era files): 84 of the 686 sweep files, InvalidLayoutVersion. This is the largest single gap.
• Virtual datasets:
  • Wrong data: unmapped regions read as 0 instead of the fill value.
  • %b printf-style source names are not expanded.
  • Hyperslab selection versions 1 and 2 are refused. Fixed 2026-09-25: versions 1-3 and irregular hyperslabs are decoded.
• Files with a user block: the base address is not applied.
• Old-style shared messages (version 1) read the wrong address.
• Groups and links:
  • Groups with a user-defined link type (e.g. 187) cannot be listed.
  • Dense groups with more than about 22 000 links cannot be listed.
  • Soft links are left out of datasets().
• Dense attributes: a large attribute stored as a fractal-heap "huge" object makes every attribute on the object fail. This affects real NetCDF files (issue671.nc).
• Other readers:
  • VL-string datasets are not readable through File.
  • Metadata cache images are not supported.
  • x87 long double and binary128 are refused.
  • N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
• Filters: blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not implemented.
• Header checks: on 12 CVE datasets libhdf5 rejects a corrupt header and we read data anyway. We need stricter header checks.
• Writer:
  • Nested groups beyond one level: path-like names are now refused, not created.
  • Dense attribute storage for attributes over 64 KiB.
  • Output that HDF5 1.8 can read.
  • A B-tree v2 chunk index larger than one leaf, so datasets with several unlimited dimensions are limited to 65 535 chunks.

---

Compound datatype message version 5 is not parsed (HDF5 2.0)

Status: fixed on main in a13ff51 (2026-06-03); not in the v2.1.0 tag, which was cut five commits earlier. Ships in the next release.

Reported by: M. Scot Breitenfeld (The HDF Group), 2026-09-08, against v2.1.0.

Summary: clawhdf5-format v2.1.0 rejects any dataset with a compound (struct) datatype written by an HDF5 2.0 library in libver='latest' mode: InvalidDatatypeVersion { class: 6, version: 5 }.

Reproduction (h5py 3.16.0 / HDF5 2.0.0):

import h5py, numpy as np
dt = np.dtype([('x', 'f8'), ('y', 'f8'), ('id', 'i4')])
data = np.array([(1.0, 2.0, 10), (3.0, 4.0, 20)], dtype=dt)
f = h5py.File('compound.h5', 'w', libver='latest')
f.create_dataset('particles', data=data)
f.close()

Committed as crates/clawhdf5-format/tests/writer_h5py_tests.rs::read_h5py_generated_compound (#[ignore]d; needs python3 with h5py on PATH). Run with cargo test -p clawhdf5-format --test writer_h5py_tests -- --include-ignored: v2.1.0 gives 25 passed / 1 failed; main passes everything.

Root cause: the compound (class 6) branch of Datatype::parse (crates/clawhdf5-format/src/datatype.rs) accepted only versions 1–4. Datatype message versions 4 and 5 changed only the Reference and Complex classes, so a v5-tagged compound uses the unchanged v3 member-list layout.

Fix: versions 3–5 are accepted for compound (class 6) and array (class 10) datatypes, and data layout message version 5 is accepted too (needed for every chunked dataset written by HDF5 2.0). Byte-level regression tests: test_compound_v5_from_hdf5_2_0, test_array_v5_from_hdf5_2_0.

Native complex datatype (class 11) is mis-parsed (HDF5 2.0)

Status: fixed 2026-09-18. Found while validating the report above.

Summary: HDF5 2.0 native complex types (H5T_COMPLEX_IEEE_F64LE etc.) were parsed as if they carried a compound-style member list. The properties are actually a single base floating-point datatype, so the parser produced a garbage datatype, or UnexpectedEof when the complex type was a compound member. h5py's default numpy-complex mapping is unaffected (it writes a {r, i} compound); only files using the native type through the C API / h5py low-level API hit this.

Fix: class 11 parses its base type and is surfaced as the equivalent {r, i} compound. Tests: test_complex_v5_from_hdf5_2_0, test_compound_with_complex_member_from_hdf5_2_0, writer_h5py_tests.rs::read_h5py_generated_native_complex.

Revised reference datatype (class 7, version 4) is not parsed

Status: fixed 2026-09-19 for object references; region and attribute references are recognised but not decoded.

Summary: HDF5 1.12+ H5T_STD_REF references use datatype message version 4 with reference types 2-4 (object / region / attribute), which Datatype::parse rejected with InvalidReferenceType. h5py still writes the legacy references, so no file had been available to test against.

Fix: a real file was produced by driving the libhdf5 bundled in the h5py wheel through ctypes (tests/fixtures/gen_std_ref.py -> std_ref_hdf5_2_0.h5). The three new types parse as ReferenceType::{Object2, DatasetRegion2, Attribute}, and read_object_references decodes Object2 elements (type, flags, token size, token = target object header address). External references (flag bit 0) and the region/attribute payloads are errors rather than misreads.

clawhdf5-gpu gpu_tests can hang under the default parallel test runner

Status: fixed 2026-09-19.

Summary: during cargo test --workspace the gpu_tests binary sat idle for 25+ minutes. Every test created its own wgpu::Instance + device (requesting adapter-maximum limits) concurrently, and readback used an unbounded device.poll(Wait).

Fix: tests hold a process-wide lock while they own a device, and GpuAccelerator readback waits time out after 30 s with GpuError::BufferMap.

Compound datatype versions 1 and 2 are mis-parsed (default libver files)

Status: fixed 2026-09-19. Found by adding a default-libver axis to the h5py interop tests.

Summary: any compound dataset written with default libver bounds (plain h5py.File(path, 'w'), datatype message version 1) failed to read, typically with Overflow("compound member 'x': byte_offset(0) + field_size(4136977) ..."). Only libver='latest' files (version 3+) and files written by clawhdf5 itself worked, which is why the existing tests never caught it.

Root cause: Datatype::parse skipped 24 bytes of legacy per-member array fields for v1 where the format has 28 (dimensionality 1 + reserved 3 + permutation 4 + reserved 4 + 4 dimension sizes 16), and treated v2 like v1 minus name padding, whereas v2 keeps the 8-byte name padding and has no array fields.

Attributes with unsupported datatypes are silently dropped

Status: fixed 2026-09-19.

Summary: Dataset::attrs() / Group::attrs() returned only attributes convertible to AttrValue and omitted the rest without any indication — every Python bool (an HDF5 enum), complex, compound and reference attributes. Unsigned 64-bit arrays were also cast to I64Array, turning values above i64::MAX negative.

Fix: booleans decode as 0/1 integers, AttrValue::U64Array keeps unsigned arrays unsigned, and AttrValue::Raw { datatype, shape, data } carries any other attribute verbatim. Both new variants are writable. Still lossy: a multi-dimensional numeric attribute is returned as a flat array (its shape is not reported).

B-tree v2 chunk index (layout v4, index type 5) is not supported

Status: fixed 2026-09-19.

Summary: a chunked dataset with two or more unlimited dimensions written with libver='latest' indexes its chunks with a version-2 B-tree, and reading it failed with unsupported chunked layout version=4, index_type=Some(5).

Fix: record types 10 (unfiltered) and 11 (filtered) are decoded — address, stored size, filter mask, scaled offsets — through the shared chunk-listing function, so full reads, cached reads, partial reads and fill-value handling all work. Covered by an h5py interop test (plain, gzip+shuffle, a 2500-chunk tree with internal nodes, a sparse dataset with a fill value, a hyperslab).

External links and external raw data are not followed

Status: open (by design for now); both are explicit errors.

Summary: a path through an external link returns FormatError::ExternalLinkUnsupported { filename, object_path }, and a dataset created with external=[...] storage returns FormatError::ExternalDataFilesUnsupported. Neither is resolved. If support is added, file names must be confined to the opened file's directory, as the virtual-dataset resolver now does.

---

Python interop suites skip silently when no interpreter has h5py

Status: fixed on main in a29c1b2 (2026-09-19).

On a system where python3 is a PEP 668 "externally managed" interpreter, h5py cannot be installed into it at all, and every interop suite — the h5py writer round-trips, the facade suite, netCDF4, and the reference files — returned false from its availability probe and skipped without failing. CI reported SKIP and a green run. This is the same class of gap that let the compound-datatype v5 bug above reach a release.

The probes now read CLAWHDF5_PYTHON, and scripts/ci-test.sh picks up .venv/bin/python automatically. To restore the coverage on a fresh checkout:

python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4

Set CLAWHDF5_REQUIRE_INTEROP=1 in any automated runner so a missing interpreter is a failure rather than a skip.

---

Crafted B-tree v2 structures crash or exhaust the reader

Status: fixed on main (2026-09-20), after v2.6.0. Every release up to and including v2.6.0 is affected.

B-tree v2 traversal (clawhdf5-format, btree_v2::collect_btree_v2_records) recursed one frame per level with the depth taken from the file, and followed child addresses without checking whether they were shared. Two consequences for anyone reading untrusted files:

• A node that is its own child, under a header claiming 65 535 levels, overflows the stack and aborts the process. The file is under 100 bytes.
• Levels whose children all point at one node below make the traversal visit it fan-out^depth times: ~30 million records from ~5 KB, and memory exhaustion one level deeper.

B-tree v2 backs dense attribute storage, v2 groups, shared object header messages and chunk indexes, so opening an object that uses any of them is enough. Both are now errors: depth is capped at 64, and traversal stops once it has produced more records than the file could physically hold.

---

Extensible Array chunk indexes read back wrong data past the inline elements

Status: fixed on main (2026-09-20), after v2.6.0. Every release up to and including v2.6.0 is affected.

A dataset created with exactly one unlimited dimension (maxshape=(None, ...), the usual append-only/resizable case) is indexed by an Extensible Array. Its index block holds the first idx_blk_elmts chunk entries inline — 4 by default — and everything after that lives in data blocks and super blocks whose layout clawhdf5-format computed incorrectly.

Consequences, by dataset size (1 chunk per element):

chunks	result before the fix
<= 36	correct (inline, plus two data blocks that happened to line up)
37	1 element wrong
400	364 elements wrong
>= ~1000	invalid Extensible Array data block signature

The dangerous case is the middle one: values were returned from the wrong chunks rather than an error being raised. Any reader that accepted the data at face value saw plausible but incorrect numbers.

The root causes were the super block sizing formulas (ndblks and dblk_nelmts each double every other level, a half-step apart), a missing block-offset field in the super block, and a page-init bitmap read from the wrong structure. All four are fixed and covered by interop tests against HDF5 2.0 at sizes that cross each boundary, including paged data blocks.

Files written by this crate were not affected by this read bug, but the writer had its own: it indexed only the first 244 chunks, so later chunks read back as 0 in libhdf5 and in clawhdf5. See "Silent wrong data found by the 2026-09-25 HDF5 audit" below.

Every f32 dataset we wrote was unreadable by h5py / libhdf5

Status: fixed 2026-09-23, after v2.7.0. Every release up to and including v2.7.0 is affected — the encoder was already wrong in v2.1.0.

The floating-point datatype message carries the position of the sign bit (bits 8–15 of its class bit field). clawhdf5-format wrote 63 for every float, which is correct only for f64. libhdf5 validates the field, so opening any f32 dataset written by this crate failed:

KeyError: 'Unable to synchronously open object (sign bit position out of bounds)'

That covers every agent store (/memory/embeddings, norms and activation_weights are f32). clawhdf5 itself ignores the field on read, and the interop suites only ever wrote f64 from our side, so nothing here noticed.

Fix: the sign position is computed from the type (bit_offset + bit_precision - 1: 15, 31, 63 for half, single, double). Regression tests: float_sign_location_is_the_top_bit_of_the_value (byte level), clawhdf5_writes_f32_h5py_reads and the agent's h5py_reads_every_dataset_of_an_agent_store.

Existing files: an agent store is rewritten in full at every checkpoint, so it becomes readable by h5py at its next checkpoint with a fixed build. Other files with f32 datasets need to be rewritten.

Empty datasets we wrote were unreadable by h5py / libhdf5

Status: fixed 2026-09-23, after v2.7.0. Every release up to and including v2.7.0 is affected.

A dataset with no elements was written with a real file address and a storage size of 0. libhdf5 guards contiguous storage with an overflow check (addr + size <= addr) that is always true when the size is 0, so it rejected the dataset:

KeyError: 'Unable to synchronously open object (invalid dataset size, likely file corruption)'

In practice: every agent store without sessions or a knowledge graph — the /sessions and /knowledge_graph datasets are empty until something is added — could not be read by h5py even once the f32 bug above was fixed. Found by the same agent-store interop test.

Fix: an empty contiguous dataset gets the undefined address (all 0xff), which is what libhdf5 itself writes.

The Node.js package (packages/clawhdf5-node) does not work

Status: open (found 2026-09-25). Unpublished; not built or tested in CI.

The TypeScript wrapper over crates/clawhdf5-napi has never run successfully:

• napi-rs converts #[napi(object)] fields to camelCase, but the wrapper reads snake_case (r.line_range, s.total_records, s.working_count, …), so every stats and consolidation field comes back undefined (src/index.ts:76-120).
• It loads ../clawhdf5.node, but napi build --platform produces clawhdf5.<triple>.node; main points at index.js while tsc writes to dist/; napi prepublish expects per-platform packages that are not defined.
• save/saveBatch exist in the napi layer but not in the wrapper, so a TypeScript caller cannot store an embedding at all.
• The WAL for agent.brain is agent.h5.wal (the store uses with_extension("h5.wal")), not agent.brain.wal as the old docs and the test cleanup assume.

It was written for an OpenClaw integration that is not being pursued (see docs/openclaw.md). Fix and add CI, or remove it, before anyone depends on it.