clawhdf5/docs/known-issues.md

# Known Issues

Bugs found during development or downstream use, tracked here because this
repository's issue tracker is disabled. One entry per bug; when an entry is
fixed, record the fix in `CHANGELOG.md` and update its status here rather than
deleting it.

---

## Silent wrong data found by the 2026-09-25 HDF5 audit

**Status:** fixed after v2.7.0 (2026-09-25). **Every release up
to and including v2.7.0 is affected.**

An audit on tank checked clawhdf5 against libhdf5 in three ways:
- a sweep of 686 public files: the libhdf5 test files, the HDF Group's
  `cve_hdf5` reproducers, and the pyfive, netcdf-c, netcdf4-python, h5wasm,
  h5py and xarray corpora;
- 567 read cases generated with h5py 3.16 / HDF5 2.0;
- 96 write cases checked with h5py builds linking HDF5 1.10, 1.12, 1.14 and
  2.0, plus h5dump 1.14.6.

It found these cases where a value came back wrong **without an error**:

| Area | What happened | Who is affected |
|---|---|---|
| Chunk index (read) | Fixed/Extensible Array indexes laid out by the current shape, not the max shape: chunks returned from the wrong place | any file with a max shape larger than its shape and `libver='latest'` (h5py `maxshape=(10, None)`, `(20, 10)`) |
| Chunk index (write) | Extensible Array chunks from index 244 on never indexed (read as 0); unlimited dimension not first: data scrambled | files we wrote with one unlimited dimension and > 244 chunks, or e.g. `maxshape=(20, None)` |
| 4-byte offsets | unfiltered chunked datasets read as zeros | files created with `sizeof_addr = 4` |
| Filter mask | any skipped filter skipped the whole pipeline | files with partially filtered chunks (optional filters, direct chunk writes) |
| Numeric reads | float read as integer returned the bit pattern; narrowing integer reads kept the low bits; bfloat16 decoded as IEEE half | `read_i32`/`read_i64`/`read_u64` callers on float or wider data; HDF5 2.0 bf16 data |
| SZIP | garbage or zeros | every libhdf5-written SZIP dataset |
| Scale-offset | float values 1 ULP off | libhdf5 D-scale float data |
| Shared fill value | read as zero fill | fill values stored as shared messages |
| VL sequences | `read_vl_bytes` truncated non-byte base types | VL int/float sequences |
| Chunk cache | two threads reading two chunked datasets through one `File` could get each other's chunks | multi-threaded readers, including Python with the GIL released |

The audit also found files we wrote that libhdf5 **refuses**, now fixed:
- Fixed Array datasets with more than 1 024 chunks.
- Header messages over 64 KiB (large attributes).
- Reference, Opaque, BitField and Time datatypes.
- Files written with `with_page_size`.
- Several unlimited dimensions.
- A finite max shape larger than the shape.
- An empty-string attribute, which broke every attribute on its object.
- `FillTime` codes, which were rotated.

Our LZ4 and Zstd output could not be read by libhdf5's registered plugins, and
our pcodec filter used Granular BitRound's ID. The details are in
`CHANGELOG.md` under Correctness and Interop.

Before the fix, 419 of the 686 files read correctly and 43 differed from h5py.
After it, 448 read correctly and 23 differ. Of those 23:
- 17 are N-Bit float files. The probe compares raw file-type bytes; the typed
  reader returns libhdf5's values (`nbit_custom_float_decodes_like_libhdf5`).
- 2 are an h5py bug: VL data with a big-endian base type comes back
  byte-swapped in h5py, and h5dump agrees with us.
- The rest are object or attribute listing differences.

**Update 2026-09-25:** the sweep is now in the repo (`conformance/run.sh`,
corpora pinned by commit) and its current numbers are in `CONFORMANCE.md`,
regenerated nightly by `.gitea/workflows/conformance.yml`. The probe now
compares N-Bit floats as the values libhdf5 converts them to, so the N-Bit
files above count as identical. Its file list is defined by
`conformance/list_files.py` (697 files: netCDF classic files are left out,
and 11 HDF5 files the ad-hoc sweep missed are in). On 42b81d9: 467 identical,
123 our-error, 15 mismatch (2 are the h5py bug above), 92 that libhdf5 cannot
read, and no panics, hangs or crashes.

There were no panics, hangs or crashes before or after, including on all 147
CVE and fuzzer files. On some of those files, h5dump 1.14.6 and h5py/HDF5 2.0
segfault or abort.

## Gaps found by the 2026-09-25 HDF5 audit (open)

**Status:** open. These fail with an error; none returns wrong data, except
the VDS item, which is marked.

- **Layout message versions 1 and 2** (HDF5 1.6-era files): 84 of the 686
  sweep files, `InvalidLayoutVersion`. This is the largest single gap.
  **Fixed 2026-09-25:** versions 1 and 2 are parsed (compact, contiguous,
  chunked via the v1 B-tree).
- **Compound datatype version 1 array members** (found with the layout
  fix; pre-1.4 files such as `tarrold.h5`): **wrong data** — the legacy
  per-member dimensions were skipped, so an array member read as one scalar.
  **Fixed 2026-09-25.**
- **Virtual datasets:**
  - **Wrong data:** unmapped regions read as 0 instead of the fill value.
  - `%b` printf-style source names are not expanded.
  - Hyperslab selection versions 1 and 2 are refused.
- **Files with a user block:** the base address is not applied.
  **Fixed 2026-09-25:** every reader views the file from the superblock on
  (`twithub.h5`, `twithub513.h5`, `h5clear_fsm_persist_user_*.h5`; the
  `twithub` files still stop at the user-defined link type below).
- **Old-style shared messages (version 1)** read the wrong address.
  **Fixed 2026-09-25:** the address follows the length-sized link-name
  offset of the embedded symbol table entry (`tcompound.h5`, `tcompound2.h5`).
- **Array members of version-1 compound datatypes** (found while fixing the
  items above) were read as one element — wrong data. **Fixed 2026-09-25.**
- **Groups and links:**
  - Groups with a user-defined link type (e.g. 187) cannot be listed.
  - Dense groups with more than about 22 000 links cannot be listed.
  - Soft links are left out of `datasets()`.
  - **Wrong data (found while fixing user blocks):** an old-style group whose
    local-heap free list points outside the heap listed garbage names where
    libhdf5 refuses the heap. **Fixed 2026-09-25**
    (`InvalidLocalHeapFreeList`, checked when a name is first read, as
    libhdf5 does).
- **Dense attributes:** a large attribute stored as a fractal-heap "huge"
  object makes every attribute on the object fail. This affects real NetCDF
  files (`issue671.nc`).
- **Other readers:**
  - VL-string datasets are not readable through `File`.
  - Metadata cache images are not supported.
  - x87 long double and binary128 are refused.
  - N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
- **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not
  implemented.
- **Header checks:** on 12 CVE datasets libhdf5 rejects a corrupt header and
  we read data anyway. We need stricter header checks.
- **Writer:**
  - Nested groups beyond one level: path-like names are now refused, not
    created.
  - Dense attribute storage for attributes over 64 KiB.
  - Output that HDF5 1.8 can read.
  - A B-tree v2 chunk index larger than one leaf, so datasets with several
    unlimited dimensions are limited to 65 535 chunks.

---

## Compound datatype message version 5 is not parsed (HDF5 2.0)

**Status:** fixed on `main` in `a13ff51` (2026-06-03); **not in the v2.1.0
tag**, which was cut five commits earlier. Ships in the next release.

**Reported by:** M. Scot Breitenfeld (The HDF Group), 2026-09-08, against v2.1.0.

**Summary:** `clawhdf5-format` v2.1.0 rejects any dataset with a compound
(struct) datatype written by an HDF5 2.0 library in `libver='latest'` mode:
`InvalidDatatypeVersion { class: 6, version: 5 }`.

**Reproduction** (h5py 3.16.0 / HDF5 2.0.0):

```python
import h5py, numpy as np
dt = np.dtype([('x', 'f8'), ('y', 'f8'), ('id', 'i4')])
data = np.array([(1.0, 2.0, 10), (3.0, 4.0, 20)], dtype=dt)
f = h5py.File('compound.h5', 'w', libver='latest')
f.create_dataset('particles', data=data)
f.close()
```

Committed as `crates/clawhdf5-format/tests/writer_h5py_tests.rs::read_h5py_generated_compound`
(`#[ignore]`d; needs `python3` with h5py on `PATH`). Run with
`cargo test -p clawhdf5-format --test writer_h5py_tests -- --include-ignored`:
v2.1.0 gives 25 passed / 1 failed; `main` passes everything.

**Root cause:** the compound (class 6) branch of `Datatype::parse`
(`crates/clawhdf5-format/src/datatype.rs`) accepted only versions 1–4. Datatype
message versions 4 and 5 changed only the Reference and Complex classes, so a
v5-tagged compound uses the unchanged v3 member-list layout.

**Fix:** versions 3–5 are accepted for compound (class 6) and array (class 10)
datatypes, and data layout message version 5 is accepted too (needed for every
chunked dataset written by HDF5 2.0). Byte-level regression tests:
`test_compound_v5_from_hdf5_2_0`, `test_array_v5_from_hdf5_2_0`.

## Native complex datatype (class 11) is mis-parsed (HDF5 2.0)

**Status:** fixed 2026-09-18. Found while validating the report above.

**Summary:** HDF5 2.0 native complex types (`H5T_COMPLEX_IEEE_F64LE` etc.)
were parsed as if they carried a compound-style member list. The properties are
actually a single base floating-point datatype, so the parser produced a garbage
datatype, or `UnexpectedEof` when the complex type was a compound member. h5py's
default numpy-complex mapping is unaffected (it writes a `{r, i}` compound);
only files using the native type through the C API / h5py low-level API hit this.

**Fix:** class 11 parses its base type and is surfaced as the equivalent
`{r, i}` compound. Tests: `test_complex_v5_from_hdf5_2_0`,
`test_compound_with_complex_member_from_hdf5_2_0`,
`writer_h5py_tests.rs::read_h5py_generated_native_complex`.

## Revised reference datatype (class 7, version 4) is not parsed

**Status:** fixed 2026-09-19 for object references; region and attribute
references are recognised but not decoded.

**Summary:** HDF5 1.12+ `H5T_STD_REF` references use datatype message version 4
with reference types 2-4 (object / region / attribute), which `Datatype::parse`
rejected with `InvalidReferenceType`. h5py still writes the legacy references,
so no file had been available to test against.

**Fix:** a real file was produced by driving the libhdf5 bundled in the h5py
wheel through ctypes (`tests/fixtures/gen_std_ref.py` ->
`std_ref_hdf5_2_0.h5`). The three new types parse as
`ReferenceType::{Object2, DatasetRegion2, Attribute}`, and
`read_object_references` decodes `Object2` elements (type, flags, token size,
token = target object header address). External references (flag bit 0) and
the region/attribute payloads are errors rather than misreads.

## `clawhdf5-gpu` `gpu_tests` can hang under the default parallel test runner

**Status:** fixed 2026-09-19.

**Summary:** during `cargo test --workspace` the `gpu_tests` binary sat idle for
25+ minutes. Every test created its own `wgpu::Instance` + device (requesting
adapter-maximum limits) concurrently, and readback used an unbounded
`device.poll(Wait)`.

**Fix:** tests hold a process-wide lock while they own a device, and
`GpuAccelerator` readback waits time out after 30 s with `GpuError::BufferMap`.

## Compound datatype versions 1 and 2 are mis-parsed (default libver files)

**Status:** fixed 2026-09-19. Found by adding a default-libver axis to the h5py
interop tests.

**Summary:** any compound dataset written with default libver bounds (plain
`h5py.File(path, 'w')`, datatype message version 1) failed to read, typically
with `Overflow("compound member 'x': byte_offset(0) + field_size(4136977) ...")`.
Only `libver='latest'` files (version 3+) and files written by clawhdf5 itself
worked, which is why the existing tests never caught it.

**Root cause:** `Datatype::parse` skipped 24 bytes of legacy per-member array
fields for v1 where the format has 28 (dimensionality 1 + reserved 3 +
permutation 4 + reserved 4 + 4 dimension sizes 16), and treated v2 like v1 minus
name padding, whereas v2 keeps the 8-byte name padding and has no array fields.

## Attributes with unsupported datatypes are silently dropped

**Status:** fixed 2026-09-19.

**Summary:** `Dataset::attrs()` / `Group::attrs()` returned only attributes
convertible to `AttrValue` and omitted the rest without any indication — every
Python `bool` (an HDF5 enum), complex, compound and reference attributes. Unsigned
64-bit arrays were also cast to `I64Array`, turning values above `i64::MAX`
negative.

**Fix:** booleans decode as 0/1 integers, `AttrValue::U64Array` keeps unsigned
arrays unsigned, and `AttrValue::Raw { datatype, shape, data }` carries any other
attribute verbatim. Both new variants are writable. Still lossy: a
multi-dimensional numeric attribute is returned as a flat array (its shape is
not reported).

## B-tree v2 chunk index (layout v4, index type 5) is not supported

**Status:** fixed 2026-09-19.

**Summary:** a chunked dataset with **two or more unlimited dimensions** written
with `libver='latest'` indexes its chunks with a version-2 B-tree, and reading it
failed with `unsupported chunked layout version=4, index_type=Some(5)`.

**Fix:** record types 10 (unfiltered) and 11 (filtered) are decoded — address,
stored size, filter mask, scaled offsets — through the shared chunk-listing
function, so full reads, cached reads, partial reads and fill-value handling
all work. Covered by an h5py interop test (plain, gzip+shuffle, a 2500-chunk
tree with internal nodes, a sparse dataset with a fill value, a hyperslab).

## External links and external raw data are not followed

**Status:** open (by design for now); both are explicit errors.

**Summary:** a path through an external link returns
`FormatError::ExternalLinkUnsupported { filename, object_path }`, and a dataset
created with `external=[...]` storage returns
`FormatError::ExternalDataFilesUnsupported`. Neither is resolved. If support is
added, file names must be confined to the opened file's directory, as the
virtual-dataset resolver now does.

---

## Python interop suites skip silently when no interpreter has h5py

**Status:** fixed on `main` in `a29c1b2` (2026-09-19).

On a system where `python3` is a PEP 668 "externally managed" interpreter,
h5py cannot be installed into it at all, and every interop suite — the h5py
writer round-trips, the facade suite, netCDF4, and the reference files —
returned `false` from its availability probe and skipped without failing. CI
reported `SKIP` and a green run. This is the same class of gap that let the
compound-datatype v5 bug above reach a release.

The probes now read `CLAWHDF5_PYTHON`, and `scripts/ci-test.sh` picks up
`.venv/bin/python` automatically. To restore the coverage on a fresh checkout:

```bash
python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4
```

Set `CLAWHDF5_REQUIRE_INTEROP=1` in any automated runner so a missing
interpreter is a failure rather than a skip.


---

## Crafted B-tree v2 structures crash or exhaust the reader

**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
and including v2.6.0 is affected.**

B-tree v2 traversal (`clawhdf5-format`, `btree_v2::collect_btree_v2_records`)
recursed one frame per level with the depth taken from the file, and followed
child addresses without checking whether they were shared. Two consequences
for anyone reading untrusted files:

- A node that is its own child, under a header claiming 65 535 levels, overflows
  the stack and aborts the process. The file is under 100 bytes.
- Levels whose children all point at one node below make the traversal visit it
  fan-out^depth times: ~30 million records from ~5 KB, and memory exhaustion one
  level deeper.

B-tree v2 backs dense attribute storage, v2 groups, shared object header
messages and chunk indexes, so opening an object that uses any of them is
enough. Both are now errors: depth is capped at 64, and traversal stops once it
has produced more records than the file could physically hold.

---

## Extensible Array chunk indexes read back wrong data past the inline elements

**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
and including v2.6.0 is affected.**

A dataset created with exactly one unlimited dimension (`maxshape=(None, ...)`,
the usual append-only/resizable case) is indexed by an Extensible Array. Its
index block holds the first `idx_blk_elmts` chunk entries inline — 4 by
default — and everything after that lives in data blocks and super blocks whose
layout `clawhdf5-format` computed incorrectly.

Consequences, by dataset size (1 chunk per element):

| chunks | result before the fix |
|---|---|
| <= 36 | correct (inline, plus two data blocks that happened to line up) |
| 37 | 1 element wrong |
| 400 | 364 elements wrong |
| >= ~1000 | `invalid Extensible Array data block signature` |

The dangerous case is the middle one: values were returned from the wrong
chunks rather than an error being raised. Any reader that accepted the data at
face value saw plausible but incorrect numbers.

The root causes were the super block sizing formulas (`ndblks` and
`dblk_nelmts` each double every *other* level, a half-step apart), a missing
block-offset field in the super block, and a page-init bitmap read from the
wrong structure. All four are fixed and covered by interop tests against
HDF5 2.0 at sizes that cross each boundary, including paged data blocks.

Files written by this crate were not affected by *this* read bug, but the
writer had its own: it indexed only the first 244 chunks, so later chunks
read back as 0 in libhdf5 and in clawhdf5. See "Silent wrong data found by
the 2026-09-25 HDF5 audit" below.

## Every `f32` dataset we wrote was unreadable by h5py / libhdf5

**Status:** fixed 2026-09-23, after v2.7.0. **Every
release up to and including v2.7.0 is affected** — the encoder was already
wrong in v2.1.0.

The floating-point datatype message carries the position of the sign bit
(bits 8–15 of its class bit field). `clawhdf5-format` wrote 63 for every
float, which is correct only for `f64`. libhdf5 validates the field, so opening
any `f32` dataset written by this crate failed:

```
KeyError: 'Unable to synchronously open object (sign bit position out of bounds)'
```

That covers every agent store (`/memory/embeddings`, `norms` and
`activation_weights` are `f32`). `clawhdf5` itself ignores the field on read,
and the interop suites only ever wrote `f64` from our side, so nothing here
noticed.

**Fix:** the sign position is computed from the type (`bit_offset +
bit_precision - 1`: 15, 31, 63 for half, single, double). Regression tests:
`float_sign_location_is_the_top_bit_of_the_value` (byte level),
`clawhdf5_writes_f32_h5py_reads` and the agent's
`h5py_reads_every_dataset_of_an_agent_store`.

**Existing files:** an agent store is rewritten in full at every checkpoint, so
it becomes readable by h5py at its next checkpoint with a fixed build. Other
files with `f32` datasets need to be rewritten.

## Empty datasets we wrote were unreadable by h5py / libhdf5

**Status:** fixed 2026-09-23, after v2.7.0. Every
release up to and including v2.7.0 is affected.

A dataset with no elements was written with a real file address and a storage
size of 0. libhdf5 guards contiguous storage with an overflow check
(`addr + size <= addr`) that is always true when the size is 0, so it rejected
the dataset:

```
KeyError: 'Unable to synchronously open object (invalid dataset size, likely file corruption)'
```

In practice: every agent store without sessions or a knowledge graph — the
`/sessions` and `/knowledge_graph` datasets are empty until something is added
— could not be read by h5py even once the `f32` bug above was fixed. Found by
the same agent-store interop test.

**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
which is what libhdf5 itself writes.

## The Node.js package (`packages/clawhdf5-node`) does not work

**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.

The TypeScript wrapper over `crates/clawhdf5-napi` has never run successfully:

- napi-rs converts `#[napi(object)]` fields to camelCase, but the wrapper reads
  snake_case (`r.line_range`, `s.total_records`, `s.working_count`, …), so
  every stats and consolidation field comes back `undefined`
  (`src/index.ts:76-120`).
- It loads `../clawhdf5.node`, but `napi build --platform` produces
  `clawhdf5.<triple>.node`; `main` points at `index.js` while `tsc` writes to
  `dist/`; `napi prepublish` expects per-platform packages that are not
  defined.
- `save`/`saveBatch` exist in the napi layer but not in the wrapper, so a
  TypeScript caller cannot store an embedding at all.
- The WAL for `agent.brain` is `agent.h5.wal` (the store uses
  `with_extension("h5.wal")`), not `agent.brain.wal` as the old docs and the
  test cleanup assume.

It was written for an OpenClaw integration that is not being pursued (see
`docs/openclaw.md`). Fix and add CI, or remove it, before anyone depends on it.