Merge branch 'feat/p2-vl-strings' into feat/p2-perf-coverage
# Conflicts: # CHANGELOG.md
This commit is contained in:
+46
-6
@@ -164,13 +164,24 @@ fill-value item that did is fixed).
|
||||
is left out of `attrs()` (reported by `attrs_with_errors()`) instead of
|
||||
failing the others.
|
||||
- **Other readers:**
|
||||
- VL-string datasets are not readable through `File`.
|
||||
- VL-string datasets are not readable through `File`. **Fixed
|
||||
2026-09-26:** `read_string` reads them (also `read_string_bytes`,
|
||||
`read_string_selection`, and on `MmapFile`/`LazyFile`), with h5py's
|
||||
values: strings end at a NUL, null elements (heap address 0) are `""`,
|
||||
and an element at the undefined heap address is an error as in libhdf5
|
||||
(it read as `""` until 2026-09-26); VL sequences of
|
||||
numbers read with `read_vlen::<T>()`, and VL values inside compounds or
|
||||
`AttrValue::Raw` attributes decode with `File::decode_strings` /
|
||||
`File::decode_vlen` (`crates/clawhdf5/tests/vl_data_interop.rs`).
|
||||
- Variable-length values inside a compound (and VL-string attributes) in
|
||||
a file with 4-byte offsets (`sizeof_addr = 4`) fail with
|
||||
`GlobalHeapObjectNotFound` or come back as `Raw`: these paths assume
|
||||
the 16-byte element of an 8-byte-offset file. The datatype itself reads
|
||||
(it was refused as "member overlaps with previous member" until
|
||||
2026-09-26).
|
||||
2026-09-26). **Fixed 2026-09-26:** a VL type's element size is the one
|
||||
its datatype message stores (12 with 4-byte offsets), and the global
|
||||
heap is read with libhdf5's header padding
|
||||
(`crates/clawhdf5/tests/vl_offset4_interop.rs`).
|
||||
- Metadata cache images are not supported.
|
||||
- x87 long double and binary128 are refused.
|
||||
- N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
|
||||
@@ -222,9 +233,10 @@ fill-value item that did is fixed).
|
||||
- (`cve-2024-32616` `/group1/dset3` and `cve-2025-2309`'s `Comp_OBJREF`
|
||||
attribute are h5py/numpy type-mapping failures, not libhdf5 refusals.)
|
||||
- `h5rs check` validates with the library's parsers, so it inherits what
|
||||
they accept: of the 150 CVE and fuzzer files, `check --data` passes 16,
|
||||
and h5dump 1.14.6 rejects 9 of those (tank, 2026-09-26; 28 and 21
|
||||
before these checks).
|
||||
they accept: of the 150 CVE and fuzzer files, `check --data` passes 15,
|
||||
and h5dump 1.14.6 rejects 8 of those (tank, 2026-09-26; 28 and 21
|
||||
before these checks, 16 and 9 before a VL type's stored element size
|
||||
was checked, which flags `cve-2024-32608`).
|
||||
- **Writer:**
|
||||
- Nested groups beyond one level: path-like names are now refused, not
|
||||
created.
|
||||
@@ -424,6 +436,32 @@ has produced more records than the file could physically hold.
|
||||
|
||||
---
|
||||
|
||||
## Crafted global heaps exhaust the variable-length reader's memory
|
||||
|
||||
**Status:** fixed on `feat/p2-vl-strings` (2026-09-26). Not a regression of
|
||||
that branch: every earlier release is affected through `read_vl_strings`.
|
||||
|
||||
Reading variable-length values kept an owned copy of every object of every
|
||||
global heap collection visited, for the whole read. A file whose collections
|
||||
nest inside one another's object data (32 bytes apart, each element pointing
|
||||
at a different one) made retained memory O(elements × file size): a 744 KB
|
||||
file reached 1.58 GB. Letting every collection's object chain jump to one
|
||||
shared run of tiny objects made the parse time O(elements × objects) too.
|
||||
libhdf5 refuses such files.
|
||||
|
||||
Now `VlResolver` caches where each object lies instead of a copy, drops its
|
||||
cache past a 32 MiB budget, and refuses a collection that overlaps one it
|
||||
has already read (libhdf5 gives each collection its own block, so only a
|
||||
crafted file has them). `GlobalHeapCollection::parse` (and the new
|
||||
`parse_index`) also refuse a collection that runs past the end of the file,
|
||||
or an object that runs past the end of its collection. Guarded by
|
||||
`crates/clawhdf5-format/tests/vl_heap_bounds.rs`, which measures peak heap
|
||||
use with a counting allocator. Still open: a file may point many elements
|
||||
at one large heap object, and a VL-*sequence* read then returns that
|
||||
object once per element, as h5py would.
|
||||
|
||||
---
|
||||
|
||||
## Extensible Array chunk indexes read back wrong data past the inline elements
|
||||
|
||||
**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
|
||||
@@ -526,7 +564,9 @@ which is what libhdf5 itself writes.
|
||||
followed (no file system).
|
||||
- Variable-length string datasets are read by decoding `read_selection`'s
|
||||
bytes with `clawhdf5_format::vl_data` in the wasm crate; `File` itself still
|
||||
cannot (see the audit gaps above).
|
||||
cannot (see the audit gaps above). (`File` can since 2026-09-26. Since
|
||||
2026-09-26 the wasm crate resolves them with the same `VlResolver` as
|
||||
`File` and `h5rs`, so all three return h5py's values.)
|
||||
|
||||
## The Node.js package (`packages/clawhdf5-node`) does not work
|
||||
|
||||
|
||||
Reference in New Issue
Block a user