osobhandClaude Opus 5
e9aeb110b7
Traversal recursed one frame per level with the depth taken from the file (a u16), and followed child addresses without asking whether they were shared. Two crafted inputs, both reproduced before fixing: - A node listing itself as its own child, under a header claiming 65 535 levels, overflowed the stack and aborted the process — SIGABRT, not an error a caller can handle — from under 100 bytes. - Levels whose children all point at one shared node below reached it fan-out^depth times: 29.5 million records in 8 s from ~5 KB, and one more level would exhaust memory. Depth is now capped at 64, as the fractal heap already was; no real tree approaches it, since even at the minimum fan-out of two that is over 2^64 records. And traversal stops once it has produced more records than the file has bytes to hold them — a valid tree stores each record once in its own bytes, so this bounds shared subtrees without trusting the header's own `total_records`. Both inputs now fail in under a millisecond. Every B-tree v2 user goes through this collector: dense attributes, v2 groups, shared messages and chunk indexes. To show the budget never refuses a real file, a new interop test has HDF5 2.0 write a depth-2 chunk index with 40 000 records and reads back all 160 000 values; it fails when the budget is deliberately made too tight. Also corrects `BM25Index::search`, which claimed to use Block-Max WAND. It scores exhaustively, and pruning would not help the store: `hybrid_search` needs every score because fusion normalises over them. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
8.8 KiB
Known Issues
Bugs found during development or downstream use, tracked here because this
repository's issue tracker is disabled. One entry per bug; when an entry is
fixed, record the fix in CHANGELOG.md and update its status here rather than
deleting it.
Compound datatype message version 5 is not parsed (HDF5 2.0)
Status: fixed on main in a13ff51 (2026-06-03); not in the v2.1.0
tag, which was cut five commits earlier. Ships in the next release.
Reported by: M. Scot Breitenfeld (The HDF Group), 2026-09-08, against v2.1.0.
Summary: clawhdf5-format v2.1.0 rejects any dataset with a compound
(struct) datatype written by an HDF5 2.0 library in libver='latest' mode:
InvalidDatatypeVersion { class: 6, version: 5 }.
Reproduction (h5py 3.16.0 / HDF5 2.0.0):
import h5py, numpy as np
dt = np.dtype([('x', 'f8'), ('y', 'f8'), ('id', 'i4')])
data = np.array([(1.0, 2.0, 10), (3.0, 4.0, 20)], dtype=dt)
f = h5py.File('compound.h5', 'w', libver='latest')
f.create_dataset('particles', data=data)
f.close()
Committed as crates/clawhdf5-format/tests/writer_h5py_tests.rs::read_h5py_generated_compound
(#[ignore]d; needs python3 with h5py on PATH). Run with
cargo test -p clawhdf5-format --test writer_h5py_tests -- --include-ignored:
v2.1.0 gives 25 passed / 1 failed; main passes everything.
Root cause: the compound (class 6) branch of Datatype::parse
(crates/clawhdf5-format/src/datatype.rs) accepted only versions 1–4. Datatype
message versions 4 and 5 changed only the Reference and Complex classes, so a
v5-tagged compound uses the unchanged v3 member-list layout.
Fix: versions 3–5 are accepted for compound (class 6) and array (class 10)
datatypes, and data layout message version 5 is accepted too (needed for every
chunked dataset written by HDF5 2.0). Byte-level regression tests:
test_compound_v5_from_hdf5_2_0, test_array_v5_from_hdf5_2_0.
Native complex datatype (class 11) is mis-parsed (HDF5 2.0)
Status: fixed 2026-09-18. Found while validating the report above.
Summary: HDF5 2.0 native complex types (H5T_COMPLEX_IEEE_F64LE etc.)
were parsed as if they carried a compound-style member list. The properties are
actually a single base floating-point datatype, so the parser produced a garbage
datatype, or UnexpectedEof when the complex type was a compound member. h5py's
default numpy-complex mapping is unaffected (it writes a {r, i} compound);
only files using the native type through the C API / h5py low-level API hit this.
Fix: class 11 parses its base type and is surfaced as the equivalent
{r, i} compound. Tests: test_complex_v5_from_hdf5_2_0,
test_compound_with_complex_member_from_hdf5_2_0,
writer_h5py_tests.rs::read_h5py_generated_native_complex.
Revised reference datatype (class 7, version 4) is not parsed
Status: fixed 2026-09-19 for object references; region and attribute references are recognised but not decoded.
Summary: HDF5 1.12+ H5T_STD_REF references use datatype message version 4
with reference types 2-4 (object / region / attribute), which Datatype::parse
rejected with InvalidReferenceType. h5py still writes the legacy references,
so no file had been available to test against.
Fix: a real file was produced by driving the libhdf5 bundled in the h5py
wheel through ctypes (tests/fixtures/gen_std_ref.py ->
std_ref_hdf5_2_0.h5). The three new types parse as
ReferenceType::{Object2, DatasetRegion2, Attribute}, and
read_object_references decodes Object2 elements (type, flags, token size,
token = target object header address). External references (flag bit 0) and
the region/attribute payloads are errors rather than misreads.
clawhdf5-gpu gpu_tests can hang under the default parallel test runner
Status: fixed 2026-09-19.
Summary: during cargo test --workspace the gpu_tests binary sat idle for
25+ minutes. Every test created its own wgpu::Instance + device (requesting
adapter-maximum limits) concurrently, and readback used an unbounded
device.poll(Wait).
Fix: tests hold a process-wide lock while they own a device, and
GpuAccelerator readback waits time out after 30 s with GpuError::BufferMap.
Compound datatype versions 1 and 2 are mis-parsed (default libver files)
Status: fixed 2026-09-19. Found by adding a default-libver axis to the h5py interop tests.
Summary: any compound dataset written with default libver bounds (plain
h5py.File(path, 'w'), datatype message version 1) failed to read, typically
with Overflow("compound member 'x': byte_offset(0) + field_size(4136977) ...").
Only libver='latest' files (version 3+) and files written by clawhdf5 itself
worked, which is why the existing tests never caught it.
Root cause: Datatype::parse skipped 24 bytes of legacy per-member array
fields for v1 where the format has 28 (dimensionality 1 + reserved 3 +
permutation 4 + reserved 4 + 4 dimension sizes 16), and treated v2 like v1 minus
name padding, whereas v2 keeps the 8-byte name padding and has no array fields.
Attributes with unsupported datatypes are silently dropped
Status: fixed 2026-09-19.
Summary: Dataset::attrs() / Group::attrs() returned only attributes
convertible to AttrValue and omitted the rest without any indication — every
Python bool (an HDF5 enum), complex, compound and reference attributes. Unsigned
64-bit arrays were also cast to I64Array, turning values above i64::MAX
negative.
Fix: booleans decode as 0/1 integers, AttrValue::U64Array keeps unsigned
arrays unsigned, and AttrValue::Raw { datatype, shape, data } carries any other
attribute verbatim. Both new variants are writable. Still lossy: a
multi-dimensional numeric attribute is returned as a flat array (its shape is
not reported).
B-tree v2 chunk index (layout v4, index type 5) is not supported
Status: fixed 2026-09-19.
Summary: a chunked dataset with two or more unlimited dimensions written
with libver='latest' indexes its chunks with a version-2 B-tree, and reading it
failed with unsupported chunked layout version=4, index_type=Some(5).
Fix: record types 10 (unfiltered) and 11 (filtered) are decoded — address, stored size, filter mask, scaled offsets — through the shared chunk-listing function, so full reads, cached reads, partial reads and fill-value handling all work. Covered by an h5py interop test (plain, gzip+shuffle, a 2500-chunk tree with internal nodes, a sparse dataset with a fill value, a hyperslab).
External links and external raw data are not followed
Status: open (by design for now); both are explicit errors.
Summary: a path through an external link returns
FormatError::ExternalLinkUnsupported { filename, object_path }, and a dataset
created with external=[...] storage returns
FormatError::ExternalDataFilesUnsupported. Neither is resolved. If support is
added, file names must be confined to the opened file's directory, as the
virtual-dataset resolver now does.
Python interop suites skip silently when no interpreter has h5py
Status: fixed on main in a29c1b2 (2026-09-19).
On a system where python3 is a PEP 668 "externally managed" interpreter,
h5py cannot be installed into it at all, and every interop suite — the h5py
writer round-trips, the facade suite, netCDF4, and the reference files —
returned false from its availability probe and skipped without failing. CI
reported SKIP and a green run. This is the same class of gap that let the
compound-datatype v5 bug above reach a release.
The probes now read CLAWHDF5_PYTHON, and scripts/ci-test.sh picks up
.venv/bin/python automatically. To restore the coverage on a fresh checkout:
python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4
Set CLAWHDF5_REQUIRE_INTEROP=1 in any automated runner so a missing
interpreter is a failure rather than a skip.
Crafted B-tree v2 structures crash or exhaust the reader
Status: fixed on main (2026-09-20), after v2.6.0. Every release up to
and including v2.6.0 is affected.
B-tree v2 traversal (clawhdf5-format, btree_v2::collect_btree_v2_records)
recursed one frame per level with the depth taken from the file, and followed
child addresses without checking whether they were shared. Two consequences
for anyone reading untrusted files:
- A node that is its own child, under a header claiming 65 535 levels, overflows the stack and aborts the process. The file is under 100 bytes.
- Levels whose children all point at one node below make the traversal visit it fan-out^depth times: ~30 million records from ~5 KB, and memory exhaustion one level deeper.
B-tree v2 backs dense attribute storage, v2 groups, shared object header messages and chunk indexes, so opening an object that uses any of them is enough. Both are now errors: depth is capped at 64, and traversal stops once it has produced more records than the file could physically hold.