Merge branch 'feat/p2-vl-strings' into feat/p2-perf-coverage

# Conflicts:
#	CHANGELOG.md
This commit is contained in:
osobh
2026-09-26 09:10:35 -05:00
24 changed files with 2405 additions and 300 deletions
+46 -6
View File
@@ -164,13 +164,24 @@ fill-value item that did is fixed).
is left out of `attrs()` (reported by `attrs_with_errors()`) instead of
failing the others.
- **Other readers:**
- VL-string datasets are not readable through `File`.
- VL-string datasets are not readable through `File`. **Fixed
2026-09-26:** `read_string` reads them (also `read_string_bytes`,
`read_string_selection`, and on `MmapFile`/`LazyFile`), with h5py's
values: strings end at a NUL, null elements (heap address 0) are `""`,
and an element at the undefined heap address is an error as in libhdf5
(it read as `""` until 2026-09-26); VL sequences of
numbers read with `read_vlen::<T>()`, and VL values inside compounds or
`AttrValue::Raw` attributes decode with `File::decode_strings` /
`File::decode_vlen` (`crates/clawhdf5/tests/vl_data_interop.rs`).
- Variable-length values inside a compound (and VL-string attributes) in
a file with 4-byte offsets (`sizeof_addr = 4`) fail with
`GlobalHeapObjectNotFound` or come back as `Raw`: these paths assume
the 16-byte element of an 8-byte-offset file. The datatype itself reads
(it was refused as "member overlaps with previous member" until
2026-09-26).
2026-09-26). **Fixed 2026-09-26:** a VL type's element size is the one
its datatype message stores (12 with 4-byte offsets), and the global
heap is read with libhdf5's header padding
(`crates/clawhdf5/tests/vl_offset4_interop.rs`).
- Metadata cache images are not supported.
- x87 long double and binary128 are refused.
- N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
@@ -222,9 +233,10 @@ fill-value item that did is fixed).
- (`cve-2024-32616` `/group1/dset3` and `cve-2025-2309`'s `Comp_OBJREF`
attribute are h5py/numpy type-mapping failures, not libhdf5 refusals.)
- `h5rs check` validates with the library's parsers, so it inherits what
they accept: of the 150 CVE and fuzzer files, `check --data` passes 16,
and h5dump 1.14.6 rejects 9 of those (tank, 2026-09-26; 28 and 21
before these checks).
they accept: of the 150 CVE and fuzzer files, `check --data` passes 15,
and h5dump 1.14.6 rejects 8 of those (tank, 2026-09-26; 28 and 21
before these checks, 16 and 9 before a VL type's stored element size
was checked, which flags `cve-2024-32608`).
- **Writer:**
- Nested groups beyond one level: path-like names are now refused, not
created.
@@ -424,6 +436,32 @@ has produced more records than the file could physically hold.
---
## Crafted global heaps exhaust the variable-length reader's memory
**Status:** fixed on `feat/p2-vl-strings` (2026-09-26). Not a regression of
that branch: every earlier release is affected through `read_vl_strings`.
Reading variable-length values kept an owned copy of every object of every
global heap collection visited, for the whole read. A file whose collections
nest inside one another's object data (32 bytes apart, each element pointing
at a different one) made retained memory O(elements × file size): a 744 KB
file reached 1.58 GB. Letting every collection's object chain jump to one
shared run of tiny objects made the parse time O(elements × objects) too.
libhdf5 refuses such files.
Now `VlResolver` caches where each object lies instead of a copy, drops its
cache past a 32 MiB budget, and refuses a collection that overlaps one it
has already read (libhdf5 gives each collection its own block, so only a
crafted file has them). `GlobalHeapCollection::parse` (and the new
`parse_index`) also refuse a collection that runs past the end of the file,
or an object that runs past the end of its collection. Guarded by
`crates/clawhdf5-format/tests/vl_heap_bounds.rs`, which measures peak heap
use with a counting allocator. Still open: a file may point many elements
at one large heap object, and a VL-*sequence* read then returns that
object once per element, as h5py would.
---
## Extensible Array chunk indexes read back wrong data past the inline elements
**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
@@ -526,7 +564,9 @@ which is what libhdf5 itself writes.
followed (no file system).
- Variable-length string datasets are read by decoding `read_selection`'s
bytes with `clawhdf5_format::vl_data` in the wasm crate; `File` itself still
cannot (see the audit gaps above).
cannot (see the audit gaps above). (`File` can since 2026-09-26. Since
2026-09-26 the wasm crate resolves them with the same `VlResolver` as
`File` and `h5rs`, so all three return h5py's values.)
## The Node.js package (`packages/clawhdf5-node`) does not work