osobhandClaude Opus 5
367faad7f7
A dataset with exactly one unlimited dimension — the ordinary append-only case — is indexed by an Extensible Array. Only its first few chunk entries (4 by default) sit inline in the index block, and everything past them was read with the wrong layout. In the default shape the 37th chunk onward came back from the wrong place: a 400-chunk dataset returned 364 wrong values while reporting success, and beyond about a thousand chunks the read failed outright. Silently wrong data is the worse half of that. It survived because the only Extensible Array fixture in the suite had three chunks — inside the inline limit — so no test ever reached a data block. Four layout errors, each confirmed against files written by HDF5 2.0 and against the library source rather than inferred: - super block `u` owns 2^(u/2) data blocks, not 2^u; - each holds 2^((u+1)/2) * data_blk_min_elmts elements — the two quantities double every *other* level, a half step apart; - a super block carries a block-offset field before its data block addresses, which was not skipped; - the page-init bitmap belongs to the super block, one bit per page packed across all of its data blocks and read MSB-first, rather than living inside the data block; a paged data block also ends its prefix with a checksum before the first page. Where the spec left room for doubt the file settled it: decoding a paged block's elements and reading the chunk values they address identifies the mapping exactly, and the bitmap's 68 set bits matched the 34 data blocks x 2 pages that 200 000 elements need, which only holds MSB-first. New interop tests cross every boundary — 4, 37, 400, 5 000 and 200 000 chunks, the last with paged data blocks — plus sparse (uninitialised pages taking fill values), gzip-filtered elements and a 2-D dataset. All three fail against the old traversal. Writing is untouched; this was a read-path bug. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
10 KiB
Known Issues
Bugs found during development or downstream use, tracked here because this
repository's issue tracker is disabled. One entry per bug; when an entry is
fixed, record the fix in CHANGELOG.md and update its status here rather than
deleting it.
Compound datatype message version 5 is not parsed (HDF5 2.0)
Status: fixed on main in a13ff51 (2026-06-03); not in the v2.1.0
tag, which was cut five commits earlier. Ships in the next release.
Reported by: M. Scot Breitenfeld (The HDF Group), 2026-09-08, against v2.1.0.
Summary: clawhdf5-format v2.1.0 rejects any dataset with a compound
(struct) datatype written by an HDF5 2.0 library in libver='latest' mode:
InvalidDatatypeVersion { class: 6, version: 5 }.
Reproduction (h5py 3.16.0 / HDF5 2.0.0):
import h5py, numpy as np
dt = np.dtype([('x', 'f8'), ('y', 'f8'), ('id', 'i4')])
data = np.array([(1.0, 2.0, 10), (3.0, 4.0, 20)], dtype=dt)
f = h5py.File('compound.h5', 'w', libver='latest')
f.create_dataset('particles', data=data)
f.close()
Committed as crates/clawhdf5-format/tests/writer_h5py_tests.rs::read_h5py_generated_compound
(#[ignore]d; needs python3 with h5py on PATH). Run with
cargo test -p clawhdf5-format --test writer_h5py_tests -- --include-ignored:
v2.1.0 gives 25 passed / 1 failed; main passes everything.
Root cause: the compound (class 6) branch of Datatype::parse
(crates/clawhdf5-format/src/datatype.rs) accepted only versions 1–4. Datatype
message versions 4 and 5 changed only the Reference and Complex classes, so a
v5-tagged compound uses the unchanged v3 member-list layout.
Fix: versions 3–5 are accepted for compound (class 6) and array (class 10)
datatypes, and data layout message version 5 is accepted too (needed for every
chunked dataset written by HDF5 2.0). Byte-level regression tests:
test_compound_v5_from_hdf5_2_0, test_array_v5_from_hdf5_2_0.
Native complex datatype (class 11) is mis-parsed (HDF5 2.0)
Status: fixed 2026-09-18. Found while validating the report above.
Summary: HDF5 2.0 native complex types (H5T_COMPLEX_IEEE_F64LE etc.)
were parsed as if they carried a compound-style member list. The properties are
actually a single base floating-point datatype, so the parser produced a garbage
datatype, or UnexpectedEof when the complex type was a compound member. h5py's
default numpy-complex mapping is unaffected (it writes a {r, i} compound);
only files using the native type through the C API / h5py low-level API hit this.
Fix: class 11 parses its base type and is surfaced as the equivalent
{r, i} compound. Tests: test_complex_v5_from_hdf5_2_0,
test_compound_with_complex_member_from_hdf5_2_0,
writer_h5py_tests.rs::read_h5py_generated_native_complex.
Revised reference datatype (class 7, version 4) is not parsed
Status: fixed 2026-09-19 for object references; region and attribute references are recognised but not decoded.
Summary: HDF5 1.12+ H5T_STD_REF references use datatype message version 4
with reference types 2-4 (object / region / attribute), which Datatype::parse
rejected with InvalidReferenceType. h5py still writes the legacy references,
so no file had been available to test against.
Fix: a real file was produced by driving the libhdf5 bundled in the h5py
wheel through ctypes (tests/fixtures/gen_std_ref.py ->
std_ref_hdf5_2_0.h5). The three new types parse as
ReferenceType::{Object2, DatasetRegion2, Attribute}, and
read_object_references decodes Object2 elements (type, flags, token size,
token = target object header address). External references (flag bit 0) and
the region/attribute payloads are errors rather than misreads.
clawhdf5-gpu gpu_tests can hang under the default parallel test runner
Status: fixed 2026-09-19.
Summary: during cargo test --workspace the gpu_tests binary sat idle for
25+ minutes. Every test created its own wgpu::Instance + device (requesting
adapter-maximum limits) concurrently, and readback used an unbounded
device.poll(Wait).
Fix: tests hold a process-wide lock while they own a device, and
GpuAccelerator readback waits time out after 30 s with GpuError::BufferMap.
Compound datatype versions 1 and 2 are mis-parsed (default libver files)
Status: fixed 2026-09-19. Found by adding a default-libver axis to the h5py interop tests.
Summary: any compound dataset written with default libver bounds (plain
h5py.File(path, 'w'), datatype message version 1) failed to read, typically
with Overflow("compound member 'x': byte_offset(0) + field_size(4136977) ...").
Only libver='latest' files (version 3+) and files written by clawhdf5 itself
worked, which is why the existing tests never caught it.
Root cause: Datatype::parse skipped 24 bytes of legacy per-member array
fields for v1 where the format has 28 (dimensionality 1 + reserved 3 +
permutation 4 + reserved 4 + 4 dimension sizes 16), and treated v2 like v1 minus
name padding, whereas v2 keeps the 8-byte name padding and has no array fields.
Attributes with unsupported datatypes are silently dropped
Status: fixed 2026-09-19.
Summary: Dataset::attrs() / Group::attrs() returned only attributes
convertible to AttrValue and omitted the rest without any indication — every
Python bool (an HDF5 enum), complex, compound and reference attributes. Unsigned
64-bit arrays were also cast to I64Array, turning values above i64::MAX
negative.
Fix: booleans decode as 0/1 integers, AttrValue::U64Array keeps unsigned
arrays unsigned, and AttrValue::Raw { datatype, shape, data } carries any other
attribute verbatim. Both new variants are writable. Still lossy: a
multi-dimensional numeric attribute is returned as a flat array (its shape is
not reported).
B-tree v2 chunk index (layout v4, index type 5) is not supported
Status: fixed 2026-09-19.
Summary: a chunked dataset with two or more unlimited dimensions written
with libver='latest' indexes its chunks with a version-2 B-tree, and reading it
failed with unsupported chunked layout version=4, index_type=Some(5).
Fix: record types 10 (unfiltered) and 11 (filtered) are decoded — address, stored size, filter mask, scaled offsets — through the shared chunk-listing function, so full reads, cached reads, partial reads and fill-value handling all work. Covered by an h5py interop test (plain, gzip+shuffle, a 2500-chunk tree with internal nodes, a sparse dataset with a fill value, a hyperslab).
External links and external raw data are not followed
Status: open (by design for now); both are explicit errors.
Summary: a path through an external link returns
FormatError::ExternalLinkUnsupported { filename, object_path }, and a dataset
created with external=[...] storage returns
FormatError::ExternalDataFilesUnsupported. Neither is resolved. If support is
added, file names must be confined to the opened file's directory, as the
virtual-dataset resolver now does.
Python interop suites skip silently when no interpreter has h5py
Status: fixed on main in a29c1b2 (2026-09-19).
On a system where python3 is a PEP 668 "externally managed" interpreter,
h5py cannot be installed into it at all, and every interop suite — the h5py
writer round-trips, the facade suite, netCDF4, and the reference files —
returned false from its availability probe and skipped without failing. CI
reported SKIP and a green run. This is the same class of gap that let the
compound-datatype v5 bug above reach a release.
The probes now read CLAWHDF5_PYTHON, and scripts/ci-test.sh picks up
.venv/bin/python automatically. To restore the coverage on a fresh checkout:
python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4
Set CLAWHDF5_REQUIRE_INTEROP=1 in any automated runner so a missing
interpreter is a failure rather than a skip.
Crafted B-tree v2 structures crash or exhaust the reader
Status: fixed on main (2026-09-20), after v2.6.0. Every release up to
and including v2.6.0 is affected.
B-tree v2 traversal (clawhdf5-format, btree_v2::collect_btree_v2_records)
recursed one frame per level with the depth taken from the file, and followed
child addresses without checking whether they were shared. Two consequences
for anyone reading untrusted files:
- A node that is its own child, under a header claiming 65 535 levels, overflows the stack and aborts the process. The file is under 100 bytes.
- Levels whose children all point at one node below make the traversal visit it fan-out^depth times: ~30 million records from ~5 KB, and memory exhaustion one level deeper.
B-tree v2 backs dense attribute storage, v2 groups, shared object header messages and chunk indexes, so opening an object that uses any of them is enough. Both are now errors: depth is capped at 64, and traversal stops once it has produced more records than the file could physically hold.
Extensible Array chunk indexes read back wrong data past the inline elements
Status: fixed on main (2026-09-20), after v2.6.0. Every release up to
and including v2.6.0 is affected.
A dataset created with exactly one unlimited dimension (maxshape=(None, ...),
the usual append-only/resizable case) is indexed by an Extensible Array. Its
index block holds the first idx_blk_elmts chunk entries inline — 4 by
default — and everything after that lives in data blocks and super blocks whose
layout clawhdf5-format computed incorrectly.
Consequences, by dataset size (1 chunk per element):
| chunks | result before the fix |
|---|---|
| <= 36 | correct (inline, plus two data blocks that happened to line up) |
| 37 | 1 element wrong |
| 400 | 364 elements wrong |
| >= ~1000 | invalid Extensible Array data block signature |
The dangerous case is the middle one: values were returned from the wrong chunks rather than an error being raised. Any reader that accepted the data at face value saw plausible but incorrect numbers.
The root causes were the super block sizing formulas (ndblks and
dblk_nelmts each double every other level, a half-step apart), a missing
block-offset field in the super block, and a page-init bitmap read from the
wrong structure. All four are fixed and covered by interop tests against
HDF5 2.0 at sizes that cross each boundary, including paged data blocks.
Files written by this crate are unaffected — this was purely a read-path bug.