Merge branch 'feat/p2-vl-strings' into feat/p2-perf-coverage

# Conflicts:
#	CHANGELOG.md
This commit is contained in:
osobh
2026-09-26 09:10:35 -05:00
24 changed files with 2405 additions and 300 deletions
+87 -2
View File
@@ -54,6 +54,91 @@
whole-row hyperslabs, points, empty selections; every type, both byte
orders, ranks 1-4).
### Variable-length data (2026-09-26)
- **VL values in files with 4-byte offsets** (`sizeof_addr = 4`). A VL
string attribute came back as `AttrValue::Raw`, a VL member of a compound
failed with `GlobalHeapObjectNotFound`, and VL datasets failed with a
size mismatch. Two causes: `Datatype::type_size()` reported 16 for every
VL type (the element is 4 + offset size + 4 bytes: 12 here), and the
global heap was parsed without the padding libhdf5 puts after its
collection and object headers (`H5HG_SIZEOF_HDR`/`H5HG_SIZEOF_OBJHDR`
round up to 8), so with 4-byte lengths every object was looked up 4
bytes early. `Datatype::VariableLength` now carries the element `size`
stored in the datatype message (**breaking** for code that builds or
exhaustively destructures that variant; patterns with `..` are
unaffected), and it is written back as stored. Tested against h5py
(`crates/clawhdf5/tests/vl_offset4_interop.rs`).
- **Wrong data: VL strings with an embedded NUL, and VL elements whose heap
object has the wrong size.** libhdf5 hands VL strings over as C strings,
so h5py reads `"a\0b"` as `"a"`; `read_vl_strings` returned the NUL and
what followed. An element whose heap object is not exactly
`length × base size` bytes is refused by libhdf5 ("Expected global heap
object size does not match"); we returned the object cut or padded to
the length. Both now behave as libhdf5, and a heap address of 0 is a null
element (empty) whatever its length. The new
`clawhdf5_format::vl_data::VlResolver` does this and parses each global
heap collection once per read: `read_vl_strings` parsed the whole
collection again for every element. `vl_data::check_element_size` refuses
a VL datatype whose stored size is not 4 + offset size + 4 (libhdf5
ignores the stored size). The conformance probe resolves VL elements
with `VlResolver` too; conformance unchanged at 575 of 697.
- **VL data through the facade.** VL-string datasets (h5py's default `str`
dtype) failed `read_string` with "type mismatch: expected String, got
VariableLength". `Dataset::read_string` now reads fixed- and
variable-length strings; new `read_string_bytes` (a VL string's exact
bytes, as h5py's `Dataset[()]` returns them), `read_string_selection`,
`read_vlen::<T>()` / `read_vlen_selection::<T>()` for VL sequences of
numbers (`T` = `f64`, `f32`, `i64`, `i32`, `u64`; converted like the
other typed readers), and `File::decode_strings` / `decode_string_bytes`
/ `decode_vlen` for VL values in compound fields and `AttrValue::Raw`
attributes. `MmapDataset` and `LazyDataset` gain `read_string` for VL
strings, `read_string_bytes` and `read_vlen`. Checked against h5py with
8- and 4-byte offsets: scalar and 1-/2-D, ASCII and UTF-8, empty strings,
contiguous, compact, chunked with gzip/shuffle, never-written and
partly written chunks, hyperslab selections, VL members of compound
datasets and attributes (`crates/clawhdf5/tests/vl_data_interop.rs`).
NetCDF-4 `string` variables now read through
`clawhdf5_netcdf4::Variable::read_string` (checked against netCDF4-python
in `crates/clawhdf5-netcdf4/tests/interop_tests.rs`).
- **Crafted global heaps could exhaust memory.** `VlResolver` kept an owned
copy of every object of every heap collection it read, so collections
nested inside each other's object data made a 744 KB file take 1.58 GB
(and `read_vl_strings` before it did the same). The cache now records
where objects lie instead of copying them, is dropped past a 32 MiB
budget, and a collection overlapping one already read is an error
(libhdf5 never writes one). New `GlobalHeapCollection::parse_index`
locates a collection's objects without copying them; `parse` and
`parse_index` refuse a collection running past the end of the file or an
object running past its collection. Conformance unchanged at 575 of 697
(`crates/clawhdf5-format/tests/vl_heap_bounds.rs`).
- **Every reader resolves VL data the same way.** `h5rs` (`dump`, `ls`,
`diff`, `check --data`) had its own lenient VL decoder: a heap object
longer than the element's length was cut to it (h5py refuses it), a null
string printed `""` where h5dump prints `NULL`, the stored element size
was trusted, and each heap collection was kept as a copy for the whole
run. It now resolves through `VlResolver`, so `dump` matches h5dump byte
for byte on VL strings (`"a\0b"` as `"a"`, null as `NULL`), VL sequences
and 4-byte-offset files, `dump --json` gives h5py's values, and
`check --data` reports any heap object whose size is not exactly the
element's length × base size. `clawhdf5-wasm` already resolved VL strings
with `read_vl_strings`; it now uses `VlResolver` and refuses a VL type
whose stored element size disagrees with the file, as `File` does
(`crates/clawhdf5-tools/tests/h5rs_interop.rs`,
`crates/clawhdf5-wasm/tests/vl_strings.rs`). New
`VlResolver::element` / `string_element` resolve one element in place.
- **A VL element at the undefined heap address is an error**, as in
libhdf5 ("addr undefined"). One of length 0 read as `""` in every reader
(`File`, `h5rs`, `clawhdf5-wasm`, `read_vl_strings`, `read_vl_bytes`).
libhdf5 writes a null element with heap address 0, which still reads as
empty, and h5py writes `""` as a zero-size heap object at a real address,
so no file libhdf5 or h5py writes is affected
(`a_vl_element_at_the_undefined_heap_address_fails_like_h5py` in
`crates/clawhdf5/tests/vl_data_interop.rs`). `read_vl_bytes` now also
treats address 0 as null whatever the length, as `VlResolver` does.
### Plugin filters (2026-09-26)
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
written by h5py with `compression="lzf"`, or with hdf5plugin's
@@ -257,10 +342,10 @@
printed with its address; exit 1 when there are any. libhdf5's h5check
reads only the 1.8 format. On the conformance corpus it passes all 418
files that both clawhdf5 and h5py read in full, and `check --data` flags
134 of the 150 CVE and fuzzer files of the `cve_hdf5` corpus (tank,
135 of the 150 CVE and fuzzer files of the `cve_hdf5` corpus (tank,
2026-09-26). `--data` also follows variable-length data into its global
heap collections and reports a damaged one at its address. It inherits
the library's tolerance, though: 9 of the 16 it passes are files h5dump
the library's tolerance, though: 8 of the 15 it passes are files h5dump
1.14.6 rejects (see `docs/known-issues.md`, header checks).
- Values over `--max-bytes` (default 1 GiB) are reported instead of read;
a panic is caught and reported as an internal error (exit 3).