Merge branch 'feat/p3-storage-trait' into feat/p3-range-zfp-edit
# Conflicts: # CHANGELOG.md # crates/clawhdf5-format/src/attribute.rs # crates/clawhdf5-format/src/btree_v1.rs # crates/clawhdf5-format/src/data_layout.rs # crates/clawhdf5-format/src/extensible_array.rs # crates/clawhdf5-format/src/fixed_array.rs # crates/clawhdf5-format/src/fractal_heap.rs # crates/clawhdf5-format/src/local_heap.rs # crates/clawhdf5-format/src/shared_message.rs
This commit is contained in:
osobh
@@ -58,6 +58,97 @@
|
||||
truncating cast. The facade, `clawhdf5-io` and `clawhdf5-ann` are not
|
||||
covered.
|
||||
|
||||
### Range reads, milestone M1: the `Storage` trait (2026-09-26)
|
||||
- **Breaking: `clawhdf5_format::error::FormatError` and `clawhdf5::Error`
|
||||
are now `#[non_exhaustive]`.** An exhaustive `match` on either needs a
|
||||
wildcard arm. `FormatError` has two new variants: `Storage(String)` (a
|
||||
storage backend failed to serve a read) and
|
||||
`ContiguousStorageRequired(&'static str)` (an operation not yet converted
|
||||
to range reads was asked of a backend without the whole file in memory).
|
||||
- New `clawhdf5_format::storage::Storage`, the synchronous, `no_std` read
|
||||
interface of `docs/design/range-reads.md` option (a): `read_at(offset:
|
||||
u64, len) -> Cow<[u8]>`, `read_ranges`, `len()` and an `as_contiguous()`
|
||||
fast path; implemented for `[u8]`, `Vec<u8>`, and references, `Box`es
|
||||
and (with `std`) `Arc`s of a `Storage`. Slices and `Vec`s serve borrowed
|
||||
bytes, so parsing an in-memory file costs no copy.
|
||||
- **The metadata parsers read through `Storage`.** Each converted parser has
|
||||
an `*_in<S: Storage + ?Sized>(&S, ..)` core (a `&dyn Storage` works too),
|
||||
and its `&[u8]` function is now a thin wrapper over it, so no caller
|
||||
changes. The wrappers compile to a `[u8]` instance of the same code, so a
|
||||
structure read in memory is a bounds check and a borrowed slice, with no
|
||||
indirect call and no copy. Converted: the superblock
|
||||
(`Superblock::parse_in`), its extension and cache image
|
||||
(`read_superblock_extension_in`, `cache_image_state_in`), object headers
|
||||
with their continuation chunks (`ObjectHeader::parse_in`), local and
|
||||
global heaps, symbol-table nodes and the group B-tree (v1), fractal heaps,
|
||||
fixed and extensible array chunk indexes, shared messages and the SOHM
|
||||
table (`message_data_in`, `message_data_with_sohm_in`,
|
||||
`load_sohm_table_in`, …), attributes (`extract_attributes_full_in`,
|
||||
`extract_attributes_tolerant_in`, `AttributeMessage::parse_in_storage`),
|
||||
fill values (`dataset_fill_value_from_storage`) and virtual-dataset
|
||||
mappings (`DataLayout::resolve_vds_mappings_in`); also
|
||||
`signature::find_signature_in`. Each structure is read with bounded reads
|
||||
(a prefix, then the structure) instead of slicing the whole file; the
|
||||
open-ended `&file_data[addr..]` slices in these modules are gone. Bounds
|
||||
errors keep their values (absolute position, file length).
|
||||
- Reads sized by untrusted fields are bounded by what the parser uses, so
|
||||
a crafted size cannot turn one structure into a read of the rest of the
|
||||
file on a range backend: local-heap names are read in growing pieces
|
||||
(64 bytes first) rather than to the end of the data segment; a fractal
|
||||
heap indirect block is read up to the entry covering the object (the
|
||||
whole block only when that entry is unallocated); paged fixed and
|
||||
extensible array data blocks over 1 MiB are read page by page, only the
|
||||
pages in use; and a block under one checksum whose claimed size runs
|
||||
past the end of the file fails its bounds check before any read (with
|
||||
the `checksum` feature). An object header's prefix is one read (was
|
||||
two).
|
||||
- Structures still indexed by a v2 B-tree — dense attribute storage, a SOHM
|
||||
B-tree index and huge fractal-heap objects found through their B-tree —
|
||||
are not converted yet (the
|
||||
v2 B-tree and dense groups come with milestone M3); over a backend without
|
||||
the whole file in memory they are the clean `ContiguousStorageRequired`
|
||||
error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL
|
||||
data are milestone M2.
|
||||
- **No behaviour change**, checked three ways (2026-09-26, tank): every
|
||||
existing test passes unchanged; the conformance sweep
|
||||
(`conformance/run.sh --no-fetch`) gives a byte-identical `results.json`
|
||||
at `f2ff2c4` and on this branch, and identical per-file probe output for
|
||||
696 of the 697 files — the exception, `cve-2025-2310.h5`, reports one of
|
||||
two errors depending on which parallel chunk decode fails first, at
|
||||
`f2ff2c4` as on this branch; and a transcript of every converted `&[u8]`
|
||||
function's result over the fixtures, the conformance corpus and the
|
||||
h5py-written files below (748 files, 7 603 object headers) is
|
||||
byte-identical between the two builds.
|
||||
- **Speed on local files** (provisional: tank was shared with other jobs;
|
||||
both builds linked into one binary and timed alternately, 200 rounds;
|
||||
new Criterion bench `clawhdf5/benches/local_metadata_bench.rs` over a
|
||||
400-group version-1 file, `clawhdf5-format/tests/fixtures/v1_groups_400.h5`):
|
||||
against `f2ff2c4`, listing the file through the facade is 2.7% faster,
|
||||
`ObjectHeader::parse` is within ±1%, symbol-table nodes and the group
|
||||
B-tree walk are about 19% faster (their entry loops were tightened),
|
||||
local-heap names and `resolve_group_children` 1.5–3% faster. The same
|
||||
harness run on two copies of the old code differs by up to 2%. The
|
||||
Criterion bench itself, old and new as separate binaries run alternately
|
||||
(3 rounds), agrees except for `ObjectHeader::parse`, which it puts about
|
||||
7% slower (25.9 vs 24.1 µs for 401 headers) while the listing that
|
||||
parses those headers is 5–8% faster; that one remains unexplained and is
|
||||
to be rechecked on an idle machine.
|
||||
- New equivalence harness `clawhdf5-format/tests/storage_equivalence.rs`:
|
||||
every converted parser runs over the file as a slice and over
|
||||
`storage::CountingStorage` — a `Storage` that serves an in-memory buffer
|
||||
through `read_at` only (`as_contiguous()` is `None`), copying what it
|
||||
serves and counting reads — and must give identical results. It walks
|
||||
the fixtures, files h5py writes for it (extensible arrays with super
|
||||
blocks and paged data blocks, paged fixed arrays, large v1 and dense
|
||||
groups, a user block, SOHM list and B-tree indexes, dense, shared and
|
||||
committed-type attributes, fixed and extensible array data blocks over
|
||||
1 MiB, whole and truncated), and with `CLAWHDF5_STORAGE_CORPUS=<dir>` a
|
||||
corpus (all 653 HDF5 files of the conformance corpus pass). Only the
|
||||
structures listed above as not converted may answer
|
||||
`ContiguousStorageRequired`, and only in the checks that reach them; a
|
||||
converted parser falling back to the whole file fails it. Milestones M2
|
||||
and M3 extend it.
|
||||
|
||||
### Chunked full reads (2026-09-26)
|
||||
- **Chunks are decoded straight into the output, into reused buffers.** A
|
||||
full read of a chunked dataset faulted in about three times its size in
|
||||
|
||||
Reference in New Issue
Block a user