osobhandClaude Opus 5.5
dda28d6c72
Idle tank at 408f69e, h5py re-run in the same session. Contiguous reads
went from 0.25x to 1.44x h5py (full) and 0.12x to 6.3x (256x256
hyperslabs) on one thread; deflate full reads at 8 threads 887 -> 2943
MB/s (h5py processes 3042). Full chunked reads at 16 threads are still
0.69x-0.76x h5py processes; the issue stays open.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
631 lines
33 KiB
Markdown
631 lines
33 KiB
Markdown
# Known Issues
|
||
|
||
Bugs found during development or downstream use, tracked here because this
|
||
repository's issue tracker is disabled. One entry per bug; when an entry is
|
||
fixed, record the fix in `CHANGELOG.md` and update its status here rather than
|
||
deleting it.
|
||
|
||
---
|
||
|
||
## Selection reads that decode more than the selection
|
||
|
||
**Status:** open (documented 2026-09-26). `Dataset::read_selection` (and so
|
||
the Python `ds[...]`) materialises only the selection's bounding box when
|
||
that box covers at most half the dataset (`partial_read`). It decodes the
|
||
whole dataset and extracts the selection instead when:
|
||
- the bounding box covers more than half the dataset — which a strided
|
||
selection across a chunked dataset (`ds[::100]`) always does, although
|
||
it may touch few chunks;
|
||
- the dataset is compact or virtual, or has no storage;
|
||
- it is chunked with a non-default fill value (the box path does not fill
|
||
unallocated chunks, so the fill-aware full read is used).
|
||
Values are correct in every case; this is cost only. Selections other than
|
||
`Selection::All` also bypass the file's chunk cache. The bounding-box
|
||
heuristic's other cost is measured under "Concurrent and contiguous read
|
||
performance" below.
|
||
|
||
## Concurrent and contiguous read performance (measured 2026-09-26)
|
||
|
||
**Status:** open for chunked full reads (one cause fixed 2026-09-26); the
|
||
contiguous item is fixed (2026-09-26). Measured on
|
||
tank with `concurrent_read` against h5py 3.16 / HDF5 2.0 (`BENCHMARKS.md`,
|
||
"Concurrent reads"):
|
||
- **Partly fixed 2026-09-26.** Full reads of chunked datasets from several threads
|
||
through one `File` stop scaling at about 4 threads (880 MB/s on deflate
|
||
data vs 4424 MB/s for 16 h5py processes). Hyperslab reads, which skip the
|
||
chunk cache, scale to 1244 MB/s, so the `File`'s shared chunk cache is the
|
||
suspect. *Cause:* not the cache. Those numbers were taken with
|
||
`--decode-threads 1`, a one-thread rayon pool, and every full read handed
|
||
its chunks to that pool, so all reader threads queued behind its single
|
||
worker (per-thread CPU time: one thread did all the decoding, the 16
|
||
readers almost none). Hyperslab reads touch one chunk each and never used
|
||
the pool. Reads now decode on the calling thread when the pool has one
|
||
thread (`tests/single_thread_decode_pool.rs`); re-measured at
|
||
`408f69e`, 8 threads went from 887 to 2943 MB/s (h5py processes: 3042).
|
||
**Still open:** at 16 threads full chunked reads reach 2142-2341 MB/s,
|
||
0.69x-0.76x 16 h5py processes (3083 MB/s in the same run), with the
|
||
default pool as with a one-thread one; with a small pool (2-4 threads)
|
||
readers outside it still wait on its workers. Datasets larger than the
|
||
cache's budget were already read without inserting into it, and skipping
|
||
its lookups entirely gained only a few percent at 16 threads. Remaining
|
||
per-read overhead, not yet addressed: each full `read_f32` of a chunked
|
||
dataset faults in about three times its size in fresh pages (the output,
|
||
the `f32` copy of it, and a new buffer per decoded chunk).
|
||
- Contiguous datasets read 4x slower than h5py on one thread (2.5 vs
|
||
9.8 GB/s full, 0.12x for 256 x 256 hyperslabs).
|
||
**Fixed 2026-09-26** (re-measured on tank at `408f69e`: 13665 MB/s
|
||
full and 31991 MB/s for 256 x 256 hyperslabs on one thread, 1.44x and
|
||
6.3x h5py; `BENCHMARKS.md`): full
|
||
reads were dominated by 4 KiB page faults on the fresh output buffer,
|
||
which is now backed by transparent huge pages as numpy's is; hyperslab
|
||
reads copied the selection three times, element by element, and now copy
|
||
each contiguous run once, straight from the file into the output (see
|
||
`CHANGELOG.md`). The chunked-read scaling item above is still open.
|
||
Values are correct; this is speed only.
|
||
|
||
## Silent wrong data found by the 2026-09-25 HDF5 audit
|
||
|
||
**Status:** fixed after v2.7.0 (2026-09-25). **Every release up
|
||
to and including v2.7.0 is affected.**
|
||
|
||
An audit on tank checked clawhdf5 against libhdf5 in three ways:
|
||
- a sweep of 686 public files: the libhdf5 test files, the HDF Group's
|
||
`cve_hdf5` reproducers, and the pyfive, netcdf-c, netcdf4-python, h5wasm,
|
||
h5py and xarray corpora;
|
||
- 567 read cases generated with h5py 3.16 / HDF5 2.0;
|
||
- 96 write cases checked with h5py builds linking HDF5 1.10, 1.12, 1.14 and
|
||
2.0, plus h5dump 1.14.6.
|
||
|
||
It found these cases where a value came back wrong **without an error**:
|
||
|
||
| Area | What happened | Who is affected |
|
||
|---|---|---|
|
||
| Chunk index (read) | Fixed/Extensible Array indexes laid out by the current shape, not the max shape: chunks returned from the wrong place | any file with a max shape larger than its shape and `libver='latest'` (h5py `maxshape=(10, None)`, `(20, 10)`) |
|
||
| Chunk index (write) | Extensible Array chunks from index 244 on never indexed (read as 0); unlimited dimension not first: data scrambled | files we wrote with one unlimited dimension and > 244 chunks, or e.g. `maxshape=(20, None)` |
|
||
| 4-byte offsets | unfiltered chunked datasets read as zeros | files created with `sizeof_addr = 4` |
|
||
| Filter mask | any skipped filter skipped the whole pipeline | files with partially filtered chunks (optional filters, direct chunk writes) |
|
||
| Numeric reads | float read as integer returned the bit pattern; narrowing integer reads kept the low bits; bfloat16 decoded as IEEE half | `read_i32`/`read_i64`/`read_u64` callers on float or wider data; HDF5 2.0 bf16 data |
|
||
| SZIP | garbage or zeros | every libhdf5-written SZIP dataset |
|
||
| Scale-offset | float values 1 ULP off | libhdf5 D-scale float data |
|
||
| Shared fill value | read as zero fill | fill values stored as shared messages |
|
||
| VL sequences | `read_vl_bytes` truncated non-byte base types | VL int/float sequences |
|
||
| Chunk cache | two threads reading two chunked datasets through one `File` could get each other's chunks | multi-threaded readers, including Python with the GIL released |
|
||
|
||
The audit also found files we wrote that libhdf5 **refuses**, now fixed:
|
||
- Fixed Array datasets with more than 1 024 chunks.
|
||
- Header messages over 64 KiB (large attributes).
|
||
- Reference, Opaque, BitField and Time datatypes.
|
||
- Files written with `with_page_size`.
|
||
- Several unlimited dimensions.
|
||
- A finite max shape larger than the shape.
|
||
- An empty-string attribute, which broke every attribute on its object.
|
||
- `FillTime` codes, which were rotated.
|
||
|
||
Our LZ4 and Zstd output could not be read by libhdf5's registered plugins, and
|
||
our pcodec filter used Granular BitRound's ID. The details are in
|
||
`CHANGELOG.md` under Correctness and Interop.
|
||
|
||
Before the fix, 419 of the 686 files read correctly and 43 differed from h5py.
|
||
After it, 448 read correctly and 23 differ. Of those 23:
|
||
- 17 are N-Bit float files. The probe compares raw file-type bytes; the typed
|
||
reader returns libhdf5's values (`nbit_custom_float_decodes_like_libhdf5`).
|
||
- 2 are an h5py bug: VL data with a big-endian base type comes back
|
||
byte-swapped in h5py, and h5dump agrees with us.
|
||
- The rest are object or attribute listing differences.
|
||
|
||
**Update 2026-09-25:** the sweep is now in the repo (`conformance/run.sh`,
|
||
corpora pinned by commit) and its current numbers are in `CONFORMANCE.md`,
|
||
regenerated nightly by `.gitea/workflows/conformance.yml`. The probe now
|
||
compares N-Bit floats as the values libhdf5 converts them to, so the N-Bit
|
||
files above count as identical. Its file list is defined by
|
||
`conformance/list_files.py` (697 files: netCDF classic files are left out,
|
||
and 11 HDF5 files the ad-hoc sweep missed are in). On 42b81d9: 467 identical,
|
||
123 our-error, 15 mismatch (2 are the h5py bug above), 92 that libhdf5 cannot
|
||
read, and no panics, hangs or crashes.
|
||
|
||
There were no panics, hangs or crashes before or after, including on all 147
|
||
CVE and fuzzer files. On some of those files, h5dump 1.14.6 and h5py/HDF5 2.0
|
||
segfault or abort.
|
||
|
||
## Gaps found by the 2026-09-25 HDF5 audit (open)
|
||
|
||
**Status:** open. These fail with an error; none returns wrong data (the VDS
|
||
fill-value item that did is fixed).
|
||
|
||
- **Layout message versions 1 and 2** (HDF5 1.6-era files): 84 of the 686
|
||
sweep files, `InvalidLayoutVersion`. This is the largest single gap.
|
||
**Fixed 2026-09-25:** versions 1 and 2 are parsed (compact, contiguous,
|
||
chunked via the v1 B-tree).
|
||
- **Compound datatype version 1 array members** (found with the layout
|
||
fix; pre-1.4 files such as `tarrold.h5`): **wrong data** — the legacy
|
||
per-member dimensions were skipped, so an array member read as one scalar.
|
||
**Fixed 2026-09-25.**
|
||
- **Virtual datasets:**
|
||
- ~~**Wrong data:** unmapped regions read as 0 instead of the fill value.~~
|
||
Fixed 2026-09-25: unmapped elements and missing sources read as the
|
||
virtual dataset's fill value.
|
||
- ~~`%b` printf-style source names are not expanded.~~ Fixed 2026-09-25:
|
||
printf-style and unlimited mappings are read, and the extent is
|
||
recomputed from the sources as libhdf5 does. Still open: the
|
||
"first missing" view and a printf gap other than 0 (libhdf5 access
|
||
properties we always read at their defaults), source-to-virtual type
|
||
conversion other than a byte swap, nested virtual sources, and source
|
||
files outside the virtual file's directory (refused with an error).
|
||
- ~~Hyperslab selection versions 1 and 2 are refused.~~ Fixed 2026-09-25:
|
||
versions 1-3 and irregular hyperslabs are decoded.
|
||
- ~~The version-1 mapping list written with a 2.0 low bound (flags byte,
|
||
shared names) was misparsed.~~ Found and fixed 2026-09-25.
|
||
- **Files with a user block:** the base address is not applied.
|
||
**Fixed 2026-09-25:** every reader views the file from the superblock on
|
||
(`twithub.h5`, `twithub513.h5`, `h5clear_fsm_persist_user_*.h5`; the
|
||
`twithub` files still stop at the user-defined link type below).
|
||
- **Old-style shared messages (version 1)** read the wrong address.
|
||
**Fixed 2026-09-25:** the address follows the length-sized link-name
|
||
offset of the embedded symbol table entry (`tcompound.h5`, `tcompound2.h5`).
|
||
- **Groups and links:**
|
||
- Groups with a user-defined link type (e.g. 187) cannot be listed.
|
||
**Fixed 2026-09-25:** user-defined links are skipped; the rest of the
|
||
group lists.
|
||
- Dense groups with more than about 22 000 links cannot be listed.
|
||
**Fixed 2026-09-25:** two bugs — fractal-heap child indirect blocks had
|
||
the wrong row count, and v2 B-tree internal nodes at depth 3+ were read
|
||
with the wrong pointer widths.
|
||
- Soft links are left out of `datasets()`. **Fixed 2026-09-25:** soft
|
||
links are listed as their targets; dangling ones are left out.
|
||
- **Wrong data (found while fixing user blocks):** an old-style group whose
|
||
local-heap free list points outside the heap listed garbage names where
|
||
libhdf5 refuses the heap. **Fixed 2026-09-25**
|
||
(`InvalidLocalHeapFreeList`, checked when a name is first read, as
|
||
libhdf5 does).
|
||
- **Dense attributes:** a large attribute stored as a fractal-heap "huge"
|
||
object makes every attribute on the object fail. This affects real NetCDF
|
||
files (`issue671.nc`). **Fixed 2026-09-25:** huge and tiny heap objects,
|
||
and filtered heaps, are read; and an attribute that still cannot be read
|
||
is left out of `attrs()` (reported by `attrs_with_errors()`) instead of
|
||
failing the others.
|
||
- **Other readers:**
|
||
- VL-string datasets are not readable through `File`. **Fixed
|
||
2026-09-26:** `read_string` reads them (also `read_string_bytes`,
|
||
`read_string_selection`, and on `MmapFile`/`LazyFile`), with h5py's
|
||
values: strings end at a NUL, null elements (heap address 0) are `""`,
|
||
and an element at the undefined heap address is an error as in libhdf5
|
||
(it read as `""` until 2026-09-26); VL sequences of
|
||
numbers read with `read_vlen::<T>()`, and VL values inside compounds or
|
||
`AttrValue::Raw` attributes decode with `File::decode_strings` /
|
||
`File::decode_vlen` (`crates/clawhdf5/tests/vl_data_interop.rs`).
|
||
- Variable-length values inside a compound (and VL-string attributes) in
|
||
a file with 4-byte offsets (`sizeof_addr = 4`) fail with
|
||
`GlobalHeapObjectNotFound` or come back as `Raw`: these paths assume
|
||
the 16-byte element of an 8-byte-offset file. The datatype itself reads
|
||
(it was refused as "member overlaps with previous member" until
|
||
2026-09-26). **Fixed 2026-09-26:** a VL type's element size is the one
|
||
its datatype message stores (12 with 4-byte offsets), and the global
|
||
heap is read with libhdf5's header padding
|
||
(`crates/clawhdf5/tests/vl_offset4_interop.rs`).
|
||
- Metadata cache images are not supported.
|
||
- x87 long double and binary128 are refused.
|
||
- N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
|
||
- **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not
|
||
implemented. **Fixed 2026-09-26** for LZF (default-on `lzf` feature),
|
||
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
|
||
`plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf,
|
||
h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of
|
||
697 ok). **Still open:** Blosc2 (32026 — hdf5plugin stores each chunk as a
|
||
Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013);
|
||
both fail with an `UnsupportedFilter` error that names the filter, and
|
||
either can be plugged in with `filter_registry::register_filter` (32023,
|
||
Granular BitRound, too, since 2026-09-26 even with the `pcodec` feature).
|
||
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
|
||
read with zeros for the missing bytes** (any filter; found reviewing the plugin
|
||
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
|
||
chunk must never read as zeros. Unfiltered chunks are read at their stored
|
||
size and are not checked this way.
|
||
- **Crash:** a hostile Blosc chunk (frame size below its header) panicked in
|
||
builds with overflow checks. **Fixed 2026-09-26**; the new decoders are
|
||
fuzzed in the unit tests.
|
||
- ~~**Header checks:** on 12 CVE datasets libhdf5 rejects a corrupt header and
|
||
we read data anyway. We need stricter header checks.~~ **Fixed
|
||
2026-09-26** (counted again: 18 objects on the CVE corpus that libhdf5
|
||
refuses; some read as wrong data, e.g. a zero chunk dimension read as all
|
||
fill values): object headers, datatypes, chunk dimensions and chunk-index
|
||
offsets are checked as libhdf5 checks them, and truncated files are
|
||
refused. 17 of the 18 now
|
||
fail as in libhdf5 (conformance on tank, `conformance/run.sh --no-fetch`,
|
||
2026-09-26: 571 of 697 ok). Still read where libhdf5 refuses:
|
||
- `cve-2024-32624.h5` `/Dset_OBJREF`: a dataspace whose storage size
|
||
overflows 64 bits. `File::dataset` and `shape()` succeed (libhdf5
|
||
refuses at open); reading the values fails.
|
||
- `cve-2020-10810.h5`, `cve-2020-10812.h5` (whole files libhdf5 cannot
|
||
open, not among the 18): libhdf5 decodes the superblock extension's File
|
||
Space Info and metadata-cache-image messages at open and refuses these
|
||
files; we do not decode those messages at open.
|
||
- Deliberately not refused, because clawhdf5 up to v2.7.0 wrote them: a
|
||
float sign bit position outside the type, and a size-0 string type.
|
||
- Not refused because current libhdf5 reads it though HDF5 2.0.0
|
||
(h5py 3.16) refuses it: a v4 chunked layout whose dimensions are
|
||
encoded in more bytes than they need (HDFGroup/hdf5@e124c36,
|
||
2026-06-05, relaxed that check; clawhdf5 wrote such layouts until
|
||
2026-09-26).
|
||
- Not refused because HDF5 2.0 (h5py 3.16) reads them though newer
|
||
libhdf5 refuses them: bit-field offset/precision outside the type, an
|
||
unknown variable-length kind, an array type whose stored size is not
|
||
its element count times its base size.
|
||
- (`cve-2024-32616` `/group1/dset3` and `cve-2025-2309`'s `Comp_OBJREF`
|
||
attribute are h5py/numpy type-mapping failures, not libhdf5 refusals.)
|
||
- `h5rs check` validates with the library's parsers, so it inherits what
|
||
they accept: of the 150 CVE and fuzzer files, `check --data` passes 15,
|
||
and h5dump 1.14.6 rejects 8 of those (tank, 2026-09-26; 28 and 21
|
||
before these checks, 16 and 9 before a VL type's stored element size
|
||
was checked, which flags `cve-2024-32608`).
|
||
- **Writer:**
|
||
- ~~Nested groups beyond one level: path-like names are now refused, not
|
||
created.~~ **Fixed 2026-09-26:** groups nest to any depth (path names
|
||
create intermediate groups, as h5py does), with soft, extra hard and
|
||
external links at any depth and optional creation-order tracking;
|
||
h5py, h5dump and `h5rs check --data` read them
|
||
(`crates/clawhdf5/tests/writer_groups_interop.rs`,
|
||
`crates/clawhdf5-tools/tests/h5rs_interop.rs`). Still missing: a group
|
||
with more than 65 535 links, or an object with more than 65 535 dense
|
||
attributes, is an error (the index is one B-tree leaf), and attribute
|
||
creation order is not tracked.
|
||
- ~~Dense link or attribute storage past 512 KiB of messages was written
|
||
unreadable (child indirect blocks of the fractal heap written as direct
|
||
blocks).~~ **Fixed 2026-09-26** (it affected 2.7.0 too): tested with
|
||
20 000 and 65 535 links and with 8 MB of dense attributes, read by
|
||
h5py, h5dump, `h5rs check` and clawhdf5, and h5py can add links to
|
||
such groups.
|
||
- ~~libhdf5 could not add a link to a group we wrote (no Group Info
|
||
message).~~ **Fixed 2026-09-26.**
|
||
- Huge fractal heap objects: in dense storage (more than 8 attributes on
|
||
an object, or more than 8 links in a group) one attribute or link
|
||
message over 65 515 bytes is an error.
|
||
- Output that HDF5 1.8 can read.
|
||
- A B-tree v2 chunk index larger than one leaf, so datasets with several
|
||
unlimited dimensions are limited to 65 535 chunks.
|
||
|
||
---
|
||
|
||
## Compound datatype message version 5 is not parsed (HDF5 2.0)
|
||
|
||
**Status:** fixed on `main` in `a13ff51` (2026-06-03); **not in the v2.1.0
|
||
tag**, which was cut five commits earlier. Ships in the next release.
|
||
|
||
**Reported by:** M. Scot Breitenfeld (The HDF Group), 2026-09-08, against v2.1.0.
|
||
|
||
**Summary:** `clawhdf5-format` v2.1.0 rejects any dataset with a compound
|
||
(struct) datatype written by an HDF5 2.0 library in `libver='latest'` mode:
|
||
`InvalidDatatypeVersion { class: 6, version: 5 }`.
|
||
|
||
**Reproduction** (h5py 3.16.0 / HDF5 2.0.0):
|
||
|
||
```python
|
||
import h5py, numpy as np
|
||
dt = np.dtype([('x', 'f8'), ('y', 'f8'), ('id', 'i4')])
|
||
data = np.array([(1.0, 2.0, 10), (3.0, 4.0, 20)], dtype=dt)
|
||
f = h5py.File('compound.h5', 'w', libver='latest')
|
||
f.create_dataset('particles', data=data)
|
||
f.close()
|
||
```
|
||
|
||
Committed as `crates/clawhdf5-format/tests/writer_h5py_tests.rs::read_h5py_generated_compound`
|
||
(`#[ignore]`d; needs `python3` with h5py on `PATH`). Run with
|
||
`cargo test -p clawhdf5-format --test writer_h5py_tests -- --include-ignored`:
|
||
v2.1.0 gives 25 passed / 1 failed; `main` passes everything.
|
||
|
||
**Root cause:** the compound (class 6) branch of `Datatype::parse`
|
||
(`crates/clawhdf5-format/src/datatype.rs`) accepted only versions 1–4. Datatype
|
||
message versions 4 and 5 changed only the Reference and Complex classes, so a
|
||
v5-tagged compound uses the unchanged v3 member-list layout.
|
||
|
||
**Fix:** versions 3–5 are accepted for compound (class 6) and array (class 10)
|
||
datatypes, and data layout message version 5 is accepted too (needed for every
|
||
chunked dataset written by HDF5 2.0). Byte-level regression tests:
|
||
`test_compound_v5_from_hdf5_2_0`, `test_array_v5_from_hdf5_2_0`.
|
||
|
||
## Native complex datatype (class 11) is mis-parsed (HDF5 2.0)
|
||
|
||
**Status:** fixed 2026-09-18. Found while validating the report above.
|
||
|
||
**Summary:** HDF5 2.0 native complex types (`H5T_COMPLEX_IEEE_F64LE` etc.)
|
||
were parsed as if they carried a compound-style member list. The properties are
|
||
actually a single base floating-point datatype, so the parser produced a garbage
|
||
datatype, or `UnexpectedEof` when the complex type was a compound member. h5py's
|
||
default numpy-complex mapping is unaffected (it writes a `{r, i}` compound);
|
||
only files using the native type through the C API / h5py low-level API hit this.
|
||
|
||
**Fix:** class 11 parses its base type and is surfaced as the equivalent
|
||
`{r, i}` compound. Tests: `test_complex_v5_from_hdf5_2_0`,
|
||
`test_compound_with_complex_member_from_hdf5_2_0`,
|
||
`writer_h5py_tests.rs::read_h5py_generated_native_complex`.
|
||
|
||
## Revised reference datatype (class 7, version 4) is not parsed
|
||
|
||
**Status:** fixed 2026-09-19 for object references; region and attribute
|
||
references are recognised but not decoded.
|
||
|
||
**Summary:** HDF5 1.12+ `H5T_STD_REF` references use datatype message version 4
|
||
with reference types 2-4 (object / region / attribute), which `Datatype::parse`
|
||
rejected with `InvalidReferenceType`. h5py still writes the legacy references,
|
||
so no file had been available to test against.
|
||
|
||
**Fix:** a real file was produced by driving the libhdf5 bundled in the h5py
|
||
wheel through ctypes (`tests/fixtures/gen_std_ref.py` ->
|
||
`std_ref_hdf5_2_0.h5`). The three new types parse as
|
||
`ReferenceType::{Object2, DatasetRegion2, Attribute}`, and
|
||
`read_object_references` decodes `Object2` elements (type, flags, token size,
|
||
token = target object header address). External references (flag bit 0) and
|
||
the region/attribute payloads are errors rather than misreads.
|
||
|
||
## `clawhdf5-gpu` `gpu_tests` can hang under the default parallel test runner
|
||
|
||
**Status:** fixed 2026-09-19.
|
||
|
||
**Summary:** during `cargo test --workspace` the `gpu_tests` binary sat idle for
|
||
25+ minutes. Every test created its own `wgpu::Instance` + device (requesting
|
||
adapter-maximum limits) concurrently, and readback used an unbounded
|
||
`device.poll(Wait)`.
|
||
|
||
**Fix:** tests hold a process-wide lock while they own a device, and
|
||
`GpuAccelerator` readback waits time out after 30 s with `GpuError::BufferMap`.
|
||
|
||
## Compound datatype versions 1 and 2 are mis-parsed (default libver files)
|
||
|
||
**Status:** fixed 2026-09-19. Found by adding a default-libver axis to the h5py
|
||
interop tests.
|
||
|
||
**Summary:** any compound dataset written with default libver bounds (plain
|
||
`h5py.File(path, 'w')`, datatype message version 1) failed to read, typically
|
||
with `Overflow("compound member 'x': byte_offset(0) + field_size(4136977) ...")`.
|
||
Only `libver='latest'` files (version 3+) and files written by clawhdf5 itself
|
||
worked, which is why the existing tests never caught it.
|
||
|
||
**Root cause:** `Datatype::parse` skipped 24 bytes of legacy per-member array
|
||
fields for v1 where the format has 28 (dimensionality 1 + reserved 3 +
|
||
permutation 4 + reserved 4 + 4 dimension sizes 16), and treated v2 like v1 minus
|
||
name padding, whereas v2 keeps the 8-byte name padding and has no array fields.
|
||
|
||
## Attributes with unsupported datatypes are silently dropped
|
||
|
||
**Status:** fixed 2026-09-19.
|
||
|
||
**Summary:** `Dataset::attrs()` / `Group::attrs()` returned only attributes
|
||
convertible to `AttrValue` and omitted the rest without any indication — every
|
||
Python `bool` (an HDF5 enum), complex, compound and reference attributes. Unsigned
|
||
64-bit arrays were also cast to `I64Array`, turning values above `i64::MAX`
|
||
negative.
|
||
|
||
**Fix:** booleans decode as 0/1 integers, `AttrValue::U64Array` keeps unsigned
|
||
arrays unsigned, and `AttrValue::Raw { datatype, shape, data }` carries any other
|
||
attribute verbatim. Both new variants are writable. Still lossy: a
|
||
multi-dimensional numeric attribute is returned as a flat array (its shape is
|
||
not reported).
|
||
|
||
## B-tree v2 chunk index (layout v4, index type 5) is not supported
|
||
|
||
**Status:** fixed 2026-09-19.
|
||
|
||
**Summary:** a chunked dataset with **two or more unlimited dimensions** written
|
||
with `libver='latest'` indexes its chunks with a version-2 B-tree, and reading it
|
||
failed with `unsupported chunked layout version=4, index_type=Some(5)`.
|
||
|
||
**Fix:** record types 10 (unfiltered) and 11 (filtered) are decoded — address,
|
||
stored size, filter mask, scaled offsets — through the shared chunk-listing
|
||
function, so full reads, cached reads, partial reads and fill-value handling
|
||
all work. Covered by an h5py interop test (plain, gzip+shuffle, a 2500-chunk
|
||
tree with internal nodes, a sparse dataset with a fill value, a hyperslab).
|
||
|
||
## External links and external raw data are not followed
|
||
|
||
**Status:** open (by design for now); both are explicit errors.
|
||
|
||
**Summary:** a path through an external link returns
|
||
`FormatError::ExternalLinkUnsupported { filename, object_path }`, and a dataset
|
||
created with `external=[...]` storage returns
|
||
`FormatError::ExternalDataFilesUnsupported`. Neither is resolved. If support is
|
||
added, file names must be confined to the opened file's directory, as the
|
||
virtual-dataset resolver now does.
|
||
|
||
---
|
||
|
||
## Python interop suites skip silently when no interpreter has h5py
|
||
|
||
**Status:** fixed on `main` in `a29c1b2` (2026-09-19).
|
||
|
||
On a system where `python3` is a PEP 668 "externally managed" interpreter,
|
||
h5py cannot be installed into it at all, and every interop suite — the h5py
|
||
writer round-trips, the facade suite, netCDF4, and the reference files —
|
||
returned `false` from its availability probe and skipped without failing. CI
|
||
reported `SKIP` and a green run. This is the same class of gap that let the
|
||
compound-datatype v5 bug above reach a release.
|
||
|
||
The probes now read `CLAWHDF5_PYTHON`, and `scripts/ci-test.sh` picks up
|
||
`.venv/bin/python` automatically. To restore the coverage on a fresh checkout:
|
||
|
||
```bash
|
||
python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4
|
||
```
|
||
|
||
Set `CLAWHDF5_REQUIRE_INTEROP=1` in any automated runner so a missing
|
||
interpreter is a failure rather than a skip.
|
||
|
||
|
||
---
|
||
|
||
## Crafted B-tree v2 structures crash or exhaust the reader
|
||
|
||
**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
|
||
and including v2.6.0 is affected.**
|
||
|
||
B-tree v2 traversal (`clawhdf5-format`, `btree_v2::collect_btree_v2_records`)
|
||
recursed one frame per level with the depth taken from the file, and followed
|
||
child addresses without checking whether they were shared. Two consequences
|
||
for anyone reading untrusted files:
|
||
|
||
- A node that is its own child, under a header claiming 65 535 levels, overflows
|
||
the stack and aborts the process. The file is under 100 bytes.
|
||
- Levels whose children all point at one node below make the traversal visit it
|
||
fan-out^depth times: ~30 million records from ~5 KB, and memory exhaustion one
|
||
level deeper.
|
||
|
||
B-tree v2 backs dense attribute storage, v2 groups, shared object header
|
||
messages and chunk indexes, so opening an object that uses any of them is
|
||
enough. Both are now errors: depth is capped at 64, and traversal stops once it
|
||
has produced more records than the file could physically hold.
|
||
|
||
---
|
||
|
||
## Crafted global heaps exhaust the variable-length reader's memory
|
||
|
||
**Status:** fixed on `feat/p2-vl-strings` (2026-09-26). Not a regression of
|
||
that branch: every earlier release is affected through `read_vl_strings`.
|
||
|
||
Reading variable-length values kept an owned copy of every object of every
|
||
global heap collection visited, for the whole read. A file whose collections
|
||
nest inside one another's object data (32 bytes apart, each element pointing
|
||
at a different one) made retained memory O(elements × file size): a 744 KB
|
||
file reached 1.58 GB. Letting every collection's object chain jump to one
|
||
shared run of tiny objects made the parse time O(elements × objects) too.
|
||
libhdf5 refuses such files.
|
||
|
||
Now `VlResolver` caches where each object lies instead of a copy, drops its
|
||
cache past a 32 MiB budget, and refuses a collection that overlaps one it
|
||
has already read (libhdf5 gives each collection its own block, so only a
|
||
crafted file has them). `GlobalHeapCollection::parse` (and the new
|
||
`parse_index`) also refuse a collection that runs past the end of the file,
|
||
or an object that runs past the end of its collection. Guarded by
|
||
`crates/clawhdf5-format/tests/vl_heap_bounds.rs`, which measures peak heap
|
||
use with a counting allocator. Still open: a file may point many elements
|
||
at one large heap object, and a VL-*sequence* read then returns that
|
||
object once per element, as h5py would.
|
||
|
||
---
|
||
|
||
## Extensible Array chunk indexes read back wrong data past the inline elements
|
||
|
||
**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
|
||
and including v2.6.0 is affected.**
|
||
|
||
A dataset created with exactly one unlimited dimension (`maxshape=(None, ...)`,
|
||
the usual append-only/resizable case) is indexed by an Extensible Array. Its
|
||
index block holds the first `idx_blk_elmts` chunk entries inline — 4 by
|
||
default — and everything after that lives in data blocks and super blocks whose
|
||
layout `clawhdf5-format` computed incorrectly.
|
||
|
||
Consequences, by dataset size (1 chunk per element):
|
||
|
||
| chunks | result before the fix |
|
||
|---|---|
|
||
| <= 36 | correct (inline, plus two data blocks that happened to line up) |
|
||
| 37 | 1 element wrong |
|
||
| 400 | 364 elements wrong |
|
||
| >= ~1000 | `invalid Extensible Array data block signature` |
|
||
|
||
The dangerous case is the middle one: values were returned from the wrong
|
||
chunks rather than an error being raised. Any reader that accepted the data at
|
||
face value saw plausible but incorrect numbers.
|
||
|
||
The root causes were the super block sizing formulas (`ndblks` and
|
||
`dblk_nelmts` each double every *other* level, a half-step apart), a missing
|
||
block-offset field in the super block, and a page-init bitmap read from the
|
||
wrong structure. All four are fixed and covered by interop tests against
|
||
HDF5 2.0 at sizes that cross each boundary, including paged data blocks.
|
||
|
||
Files written by this crate were not affected by *this* read bug, but the
|
||
writer had its own: it indexed only the first 244 chunks, so later chunks
|
||
read back as 0 in libhdf5 and in clawhdf5. See "Silent wrong data found by
|
||
the 2026-09-25 HDF5 audit" below.
|
||
|
||
## Every `f32` dataset we wrote was unreadable by h5py / libhdf5
|
||
|
||
**Status:** fixed 2026-09-23, after v2.7.0. **Every
|
||
release up to and including v2.7.0 is affected** — the encoder was already
|
||
wrong in v2.1.0.
|
||
|
||
The floating-point datatype message carries the position of the sign bit
|
||
(bits 8–15 of its class bit field). `clawhdf5-format` wrote 63 for every
|
||
float, which is correct only for `f64`. libhdf5 validates the field, so opening
|
||
any `f32` dataset written by this crate failed:
|
||
|
||
```
|
||
KeyError: 'Unable to synchronously open object (sign bit position out of bounds)'
|
||
```
|
||
|
||
That covers every agent store (`/memory/embeddings`, `norms` and
|
||
`activation_weights` are `f32`). `clawhdf5` itself ignores the field on read,
|
||
and the interop suites only ever wrote `f64` from our side, so nothing here
|
||
noticed.
|
||
|
||
**Fix:** the sign position is computed from the type (`bit_offset +
|
||
bit_precision - 1`: 15, 31, 63 for half, single, double). Regression tests:
|
||
`float_sign_location_is_the_top_bit_of_the_value` (byte level),
|
||
`clawhdf5_writes_f32_h5py_reads` and the agent's
|
||
`h5py_reads_every_dataset_of_an_agent_store`.
|
||
|
||
**Existing files:** an agent store is rewritten in full at every checkpoint, so
|
||
it becomes readable by h5py at its next checkpoint with a fixed build. Other
|
||
files with `f32` datasets need to be rewritten.
|
||
|
||
## Empty datasets we wrote were unreadable by h5py / libhdf5
|
||
|
||
**Status:** fixed 2026-09-23, after v2.7.0. Every
|
||
release up to and including v2.7.0 is affected.
|
||
|
||
A dataset with no elements was written with a real file address and a storage
|
||
size of 0. libhdf5 guards contiguous storage with an overflow check
|
||
(`addr + size <= addr`) that is always true when the size is 0, so it rejected
|
||
the dataset:
|
||
|
||
```
|
||
KeyError: 'Unable to synchronously open object (invalid dataset size, likely file corruption)'
|
||
```
|
||
|
||
In practice: every agent store without sessions or a knowledge graph — the
|
||
`/sessions` and `/knowledge_graph` datasets are empty until something is added
|
||
— could not be read by h5py even once the `f32` bug above was fixed. Found by
|
||
the same agent-store interop test.
|
||
|
||
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
||
which is what libhdf5 itself writes.
|
||
|
||
## `clawhdf5-wasm` (browser) limits
|
||
|
||
**Status:** open (by design for now; added 2026-09-26).
|
||
|
||
- The whole file is held in memory: `open()` takes its bytes. There are no
|
||
HTTP range reads, so a multi-GB file does not fit a browser tab.
|
||
- Compound, reference, opaque, bitfield, time and VL-sequence datasets are
|
||
refused with an error naming the type; attributes of those types come back
|
||
as `value: null` with their `dtype`.
|
||
- No Zstd or SZIP (both link C): such datasets fail with
|
||
`unsupported filter: 32015` / `: 4`. pcodec is not enabled either.
|
||
- External links and virtual-dataset sources in other files cannot be
|
||
followed (no file system).
|
||
- Variable-length string datasets are read by decoding `read_selection`'s
|
||
bytes with `clawhdf5_format::vl_data` in the wasm crate; `File` itself still
|
||
cannot (see the audit gaps above). (`File` can since 2026-09-26. Since
|
||
2026-09-26 the wasm crate resolves them with the same `VlResolver` as
|
||
`File` and `h5rs`, so all three return h5py's values.)
|
||
|
||
## The Node.js package (`packages/clawhdf5-node`) does not work
|
||
|
||
**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
|
||
|
||
The TypeScript wrapper over `crates/clawhdf5-napi` has never run successfully:
|
||
|
||
- napi-rs converts `#[napi(object)]` fields to camelCase, but the wrapper reads
|
||
snake_case (`r.line_range`, `s.total_records`, `s.working_count`, …), so
|
||
every stats and consolidation field comes back `undefined`
|
||
(`src/index.ts:76-120`).
|
||
- It loads `../clawhdf5.node`, but `napi build --platform` produces
|
||
`clawhdf5.<triple>.node`; `main` points at `index.js` while `tsc` writes to
|
||
`dist/`; `napi prepublish` expects per-platform packages that are not
|
||
defined.
|
||
- `save`/`saveBatch` exist in the napi layer but not in the wrapper, so a
|
||
TypeScript caller cannot store an embedding at all.
|
||
- The WAL for `agent.brain` is `agent.h5.wal` (the store uses
|
||
`with_extension("h5.wal")`), not `agent.brain.wal` as the old docs and the
|
||
test cleanup assume.
|
||
|
||
It was written for an OpenClaw integration that is not being pursued (see
|
||
`docs/openclaw.md`). Fix and add CI, or remove it, before anyone depends on it.
|