Merge branch 'feat/p2-vl-strings' into feat/p2-perf-coverage
# Conflicts: # CHANGELOG.md
This commit is contained in:
+87
-2
@@ -54,6 +54,91 @@
|
||||
whole-row hyperslabs, points, empty selections; every type, both byte
|
||||
orders, ranks 1-4).
|
||||
|
||||
### Variable-length data (2026-09-26)
|
||||
- **VL values in files with 4-byte offsets** (`sizeof_addr = 4`). A VL
|
||||
string attribute came back as `AttrValue::Raw`, a VL member of a compound
|
||||
failed with `GlobalHeapObjectNotFound`, and VL datasets failed with a
|
||||
size mismatch. Two causes: `Datatype::type_size()` reported 16 for every
|
||||
VL type (the element is 4 + offset size + 4 bytes: 12 here), and the
|
||||
global heap was parsed without the padding libhdf5 puts after its
|
||||
collection and object headers (`H5HG_SIZEOF_HDR`/`H5HG_SIZEOF_OBJHDR`
|
||||
round up to 8), so with 4-byte lengths every object was looked up 4
|
||||
bytes early. `Datatype::VariableLength` now carries the element `size`
|
||||
stored in the datatype message (**breaking** for code that builds or
|
||||
exhaustively destructures that variant; patterns with `..` are
|
||||
unaffected), and it is written back as stored. Tested against h5py
|
||||
(`crates/clawhdf5/tests/vl_offset4_interop.rs`).
|
||||
- **Wrong data: VL strings with an embedded NUL, and VL elements whose heap
|
||||
object has the wrong size.** libhdf5 hands VL strings over as C strings,
|
||||
so h5py reads `"a\0b"` as `"a"`; `read_vl_strings` returned the NUL and
|
||||
what followed. An element whose heap object is not exactly
|
||||
`length × base size` bytes is refused by libhdf5 ("Expected global heap
|
||||
object size does not match"); we returned the object cut or padded to
|
||||
the length. Both now behave as libhdf5, and a heap address of 0 is a null
|
||||
element (empty) whatever its length. The new
|
||||
`clawhdf5_format::vl_data::VlResolver` does this and parses each global
|
||||
heap collection once per read: `read_vl_strings` parsed the whole
|
||||
collection again for every element. `vl_data::check_element_size` refuses
|
||||
a VL datatype whose stored size is not 4 + offset size + 4 (libhdf5
|
||||
ignores the stored size). The conformance probe resolves VL elements
|
||||
with `VlResolver` too; conformance unchanged at 575 of 697.
|
||||
- **VL data through the facade.** VL-string datasets (h5py's default `str`
|
||||
dtype) failed `read_string` with "type mismatch: expected String, got
|
||||
VariableLength". `Dataset::read_string` now reads fixed- and
|
||||
variable-length strings; new `read_string_bytes` (a VL string's exact
|
||||
bytes, as h5py's `Dataset[()]` returns them), `read_string_selection`,
|
||||
`read_vlen::<T>()` / `read_vlen_selection::<T>()` for VL sequences of
|
||||
numbers (`T` = `f64`, `f32`, `i64`, `i32`, `u64`; converted like the
|
||||
other typed readers), and `File::decode_strings` / `decode_string_bytes`
|
||||
/ `decode_vlen` for VL values in compound fields and `AttrValue::Raw`
|
||||
attributes. `MmapDataset` and `LazyDataset` gain `read_string` for VL
|
||||
strings, `read_string_bytes` and `read_vlen`. Checked against h5py with
|
||||
8- and 4-byte offsets: scalar and 1-/2-D, ASCII and UTF-8, empty strings,
|
||||
contiguous, compact, chunked with gzip/shuffle, never-written and
|
||||
partly written chunks, hyperslab selections, VL members of compound
|
||||
datasets and attributes (`crates/clawhdf5/tests/vl_data_interop.rs`).
|
||||
NetCDF-4 `string` variables now read through
|
||||
`clawhdf5_netcdf4::Variable::read_string` (checked against netCDF4-python
|
||||
in `crates/clawhdf5-netcdf4/tests/interop_tests.rs`).
|
||||
|
||||
- **Crafted global heaps could exhaust memory.** `VlResolver` kept an owned
|
||||
copy of every object of every heap collection it read, so collections
|
||||
nested inside each other's object data made a 744 KB file take 1.58 GB
|
||||
(and `read_vl_strings` before it did the same). The cache now records
|
||||
where objects lie instead of copying them, is dropped past a 32 MiB
|
||||
budget, and a collection overlapping one already read is an error
|
||||
(libhdf5 never writes one). New `GlobalHeapCollection::parse_index`
|
||||
locates a collection's objects without copying them; `parse` and
|
||||
`parse_index` refuse a collection running past the end of the file or an
|
||||
object running past its collection. Conformance unchanged at 575 of 697
|
||||
(`crates/clawhdf5-format/tests/vl_heap_bounds.rs`).
|
||||
|
||||
- **Every reader resolves VL data the same way.** `h5rs` (`dump`, `ls`,
|
||||
`diff`, `check --data`) had its own lenient VL decoder: a heap object
|
||||
longer than the element's length was cut to it (h5py refuses it), a null
|
||||
string printed `""` where h5dump prints `NULL`, the stored element size
|
||||
was trusted, and each heap collection was kept as a copy for the whole
|
||||
run. It now resolves through `VlResolver`, so `dump` matches h5dump byte
|
||||
for byte on VL strings (`"a\0b"` as `"a"`, null as `NULL`), VL sequences
|
||||
and 4-byte-offset files, `dump --json` gives h5py's values, and
|
||||
`check --data` reports any heap object whose size is not exactly the
|
||||
element's length × base size. `clawhdf5-wasm` already resolved VL strings
|
||||
with `read_vl_strings`; it now uses `VlResolver` and refuses a VL type
|
||||
whose stored element size disagrees with the file, as `File` does
|
||||
(`crates/clawhdf5-tools/tests/h5rs_interop.rs`,
|
||||
`crates/clawhdf5-wasm/tests/vl_strings.rs`). New
|
||||
`VlResolver::element` / `string_element` resolve one element in place.
|
||||
|
||||
- **A VL element at the undefined heap address is an error**, as in
|
||||
libhdf5 ("addr undefined"). One of length 0 read as `""` in every reader
|
||||
(`File`, `h5rs`, `clawhdf5-wasm`, `read_vl_strings`, `read_vl_bytes`).
|
||||
libhdf5 writes a null element with heap address 0, which still reads as
|
||||
empty, and h5py writes `""` as a zero-size heap object at a real address,
|
||||
so no file libhdf5 or h5py writes is affected
|
||||
(`a_vl_element_at_the_undefined_heap_address_fails_like_h5py` in
|
||||
`crates/clawhdf5/tests/vl_data_interop.rs`). `read_vl_bytes` now also
|
||||
treats address 0 as null whatever the length, as `VlResolver` does.
|
||||
|
||||
### Plugin filters (2026-09-26)
|
||||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||||
written by h5py with `compression="lzf"`, or with hdf5plugin's
|
||||
@@ -257,10 +342,10 @@
|
||||
printed with its address; exit 1 when there are any. libhdf5's h5check
|
||||
reads only the 1.8 format. On the conformance corpus it passes all 418
|
||||
files that both clawhdf5 and h5py read in full, and `check --data` flags
|
||||
134 of the 150 CVE and fuzzer files of the `cve_hdf5` corpus (tank,
|
||||
135 of the 150 CVE and fuzzer files of the `cve_hdf5` corpus (tank,
|
||||
2026-09-26). `--data` also follows variable-length data into its global
|
||||
heap collections and reports a damaged one at its address. It inherits
|
||||
the library's tolerance, though: 9 of the 16 it passes are files h5dump
|
||||
the library's tolerance, though: 8 of the 15 it passes are files h5dump
|
||||
1.14.6 rejects (see `docs/known-issues.md`, header checks).
|
||||
- Values over `--max-bytes` (default 1 GiB) are reported instead of read;
|
||||
a panic is caught and reported as an internal error (exit 3).
|
||||
|
||||
Reference in New Issue
Block a user