docs: range-read milestone M2 — changelog, limits, design status
CHANGELOG (Unreleased): File::open_storage, raw data and v2 B-trees over
Storage, the tests and their corpus results (2026-09-26, tank; conformance
600 of 697, results.json identical to 8f59b2e). known-issues: what
open_storage does not do yet (no remote backend or block cache, read_at
counts of a one-pass read, v1 group lookups, whole-file VDS sources,
zero-copy methods, SWMR growth, hash-order error choice on damaged chunked
datasets). Design: M2 status and the choices made.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
+93
-1
@@ -2,6 +2,97 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Range reads, milestone M2: raw data and `File::open_storage` (2026-09-26)
|
||||
- **`clawhdf5::File::open_storage(Arc<dyn Storage + Send + Sync>)`** opens
|
||||
a file served by any `clawhdf5_format::storage::Storage` and gives the
|
||||
whole read API over it: groups and paths, datasets, attributes, the
|
||||
`read_*` methods, selections, variable-length strings and sequences, and
|
||||
virtual datasets. Every byte comes through `Storage::read_at` /
|
||||
`read_ranges`; a user block is found and skipped, nothing past the
|
||||
superblock's end of file is read, and a metadata cache image is laid over
|
||||
the reads it covers (new `CacheImage::entries`). External virtual-dataset
|
||||
sources are read through the new `File::set_vds_resolver` (any `File`;
|
||||
without one a storage-backed file cannot follow them). `File::open` and
|
||||
`File::from_bytes` keep their mmap and in-memory paths: the file's view
|
||||
is now a `Storage` whose `as_contiguous()` is that buffer, and every hot
|
||||
loop takes it. New exports: `clawhdf5::{Storage, SharedStorage,
|
||||
VdsResolver}`, `File::contiguous_bytes()`.
|
||||
- Over a storage without the whole file in memory the zero-copy methods
|
||||
(`read_raw_ref`, `read_as_slice`, `read_*_zerocopy`) answer
|
||||
`FormatError::ContiguousStorageRequired`, and `File::as_bytes` panics
|
||||
(documented; use `contiguous_bytes`). The typed readers keep their fast
|
||||
paths: a contiguous dataset is read in one piece and converted, and a
|
||||
contiguous selection of a native type reads only its runs
|
||||
(`data_read::read_selection_native_in`).
|
||||
- **Nothing in the format crate needs the whole file any more.** The
|
||||
structures M1 left to `ContiguousStorageRequired` read through `Storage`:
|
||||
v2 B-trees (`BTreeV2Header::parse_in`, `collect_btree_v2_records_in`,
|
||||
`find_btree_v2_records_in`; one bounded read per node, whose size is known
|
||||
before it is read), hence dense attributes, a SOHM B-tree index, huge
|
||||
fractal-heap objects, and dense groups; v1 and v2 group listings, lookups
|
||||
and paths (`group_v2::resolve_group_children_in`, `resolve_child_in`,
|
||||
`resolve_path_any_in`, `group_v1::*_in`).
|
||||
- **Raw data reads through `Storage`**, each with a generic `*_in` core and
|
||||
its `&[u8]` function as a thin wrapper (callers do not change):
|
||||
`data_read` (`read_raw_data*_in`, `read_raw_data_selection_in`,
|
||||
`read_chunked_native_in`), `chunked_read` (the v1 B-tree chunk index —
|
||||
one read of each node's header, one of its entries — `list_chunks_in`,
|
||||
and the full, cached, sweep and indexed reads), `parallel_read`,
|
||||
`partial_read`, `fill_value` (`read_full_with_fill_in`,
|
||||
`apply_to_unallocated_chunks_in`; `dataset_fill_value_from_storage` is
|
||||
now generic, so a `&dyn Storage` still works), `vds`
|
||||
(`read_virtual_dataset_in`, `virtual_dataset_extent_in`: the virtual
|
||||
file through `Storage`, external source files still loaded whole through
|
||||
the resolver), `vl_data` (`VlResolver<'a, S = [u8]>` with `new_in`;
|
||||
`read_vl_strings_in`, `read_vl_bytes_in`),
|
||||
`AttributeMessage::read_vl_strings_in`, `provenance::verify_dataset_in`.
|
||||
- A chunked read first lists the chunks it needs, then fetches all their
|
||||
stored bytes with **one `read_ranges` call** (per 64 MiB of stored
|
||||
data), so a remote backend can coalesce and parallelise them, then
|
||||
decodes as before (in parallel with the `parallel` feature). Chunks the
|
||||
file's chunk cache already holds are not fetched. A selection fetches
|
||||
only the chunks its bounding box overlaps; a contiguous selection only
|
||||
its runs (adjacent ones merged). A global-heap collection is read once
|
||||
per resolver and kept (within the resolver's 32 MiB budget).
|
||||
- Each extent's bounds error is the one the slice readers gave, reported
|
||||
when the read reaches that extent, so a damaged file fails with the
|
||||
same error, in the same order, through either path.
|
||||
- **No behaviour change for in-memory and mapped files:** with
|
||||
`as_contiguous()` every path slices the file as before (checked below).
|
||||
- Tests (2026-09-26, tank):
|
||||
- `clawhdf5-format/tests/storage_equivalence.rs` now also reads every
|
||||
dataset — whole, fill-aware, through a chunk cache (twice) and the
|
||||
indexed path, three selections, virtual datasets with their sibling
|
||||
sources, VL strings, sequences and bytes — through the read_at-only
|
||||
`CountingStorage` and requires the slice results, and fails on any
|
||||
`ContiguousStorageRequired`. With
|
||||
`CLAWHDF5_STORAGE_CORPUS=conformance/.cache/corpus`, all 653 HDF5
|
||||
files of the corpus agree (82 396 checks). The cached and indexed
|
||||
paths are compared on values only when a read fails: they order
|
||||
chunks by hash map, so which failing chunk a damaged dataset reports
|
||||
varies between two caches even for the same slice (seen on
|
||||
`cve-2025-2310.h5`; see `docs/known-issues.md`).
|
||||
- A misbehaving storage (fails its N-th read; serves short reads) over
|
||||
every fixture: each listing and dataset read is an error or exactly the
|
||||
in-memory result, never other data (1 137 runs).
|
||||
- A chunked read issues one `read_ranges` call with one range per chunk,
|
||||
and a one-chunk selection one call with one range.
|
||||
- `clawhdf5/tests/storage_equivalence.rs` reads every fixture (61 files)
|
||||
and, with `CLAWHDF5_STORAGE_CORPUS`, every corpus file (701 files, 621
|
||||
that open) through `File::open` and through `File::open_storage` over
|
||||
`CountingStorage`: the tree, every attribute (all, and each by name),
|
||||
every dataset's shape, types and values (all bytes, `f64`, `f32`,
|
||||
`i64`, a hyperslab, strings, VL sequences) must be identical, and are.
|
||||
It also counts what one pass — open, list, read every attribute and
|
||||
every dataset once — asks of a storage with no cache: 176 092 `read_at`
|
||||
calls and 208 MB for the 621 corpus files (254 MB of files); the most
|
||||
are `h5stat_newgrat.h5` (35 001 groups: 92 489 calls) and
|
||||
`ref_hdf5_compat1.nc` (16 062). A remote backend needs the block cache
|
||||
of milestone M3. Command: `CLAWHDF5_STORAGE_CORPUS=… cargo test
|
||||
--release -p clawhdf5 --test storage_equivalence -- --nocapture`.
|
||||
- Conformance sweep (`conformance/run.sh --no-fetch`): 600 of 697 files
|
||||
ok, `results.json` byte-identical to `8f59b2e`.
|
||||
|
||||
### Name lookups through the name index (2026-09-26)
|
||||
- **Finding one link or attribute by name reads the name index, not every
|
||||
entry.** In a dense group (links in a fractal heap) the v2 B-tree name
|
||||
@@ -108,7 +199,8 @@
|
||||
v2 B-tree and dense groups come with milestone M3); over a backend without
|
||||
the whole file in memory they are the clean `ContiguousStorageRequired`
|
||||
error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL
|
||||
data are milestone M2.
|
||||
data are milestone M2. (All of them read through `Storage` since M2,
|
||||
above.)
|
||||
- **No behaviour change**, checked three ways (2026-09-26, tank): every
|
||||
existing test passes unchanged; the conformance sweep
|
||||
(`conformance/run.sh --no-fetch`) gives a byte-identical `results.json`
|
||||
|
||||
@@ -1,9 +1,12 @@
|
||||
# Design: range reads (reading HDF5 without holding the whole file)
|
||||
|
||||
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
|
||||
change. Progress: M1, first part (the `Storage` trait and the metadata
|
||||
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
|
||||
group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was
|
||||
change. Progress: M0 and M1 are done, and so is M2 (branch
|
||||
`feat/p3-m2-raw-data`): every read path of the format crate works through
|
||||
`Storage`, v2 B-trees, dense groups and raw data included, and
|
||||
`File::open_storage` gives the facade's read API over any `Storage` (see
|
||||
`CHANGELOG.md`, "Range reads, milestone M2"). M3 (a remote backend with
|
||||
its block cache) is next. Every count in §1–§2 was
|
||||
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
|
||||
next to it. No timing numbers appear here on purpose: the machine was shared
|
||||
with other build jobs when this was written.
|
||||
@@ -421,6 +424,32 @@ fast path within benchmark noise.
|
||||
on other backends (they already return `Option`/`Result`).
|
||||
- Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
|
||||
keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
|
||||
- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data`. As planned,
|
||||
with these choices:
|
||||
- `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the
|
||||
file handle is shared by its datasets and may be sent across threads).
|
||||
The file's view (user block skipped, bounded by the recorded end of
|
||||
file, cache image laid over its reads) is itself a `Storage`, and the
|
||||
facade calls the generic cores with it; for a `Vec` or an mmap its
|
||||
`as_contiguous()` is the buffer, so the local paths are the slice code
|
||||
(checked: identical conformance results; the bench gate below still
|
||||
has to be run on an idle machine).
|
||||
- Chunked reads fetch in batches of at most 64 MiB of stored bytes, one
|
||||
`read_ranges` call each, so a remote read never holds more than that
|
||||
undecoded; chunks already in the chunk cache are not fetched.
|
||||
- The typed readers' zero-copy fast path became "read the contiguous
|
||||
bytes once": over a range storage a contiguous `read_f64` is one read,
|
||||
and a native contiguous selection reads only its runs.
|
||||
- External VDS source files stay whole-file, through a resolver that
|
||||
returns bytes (`File::set_vds_resolver`).
|
||||
- The v2 B-tree and dense groups were done here rather than in M3, so no
|
||||
format-crate path answers `ContiguousStorageRequired` any more; only the
|
||||
facade's zero-copy methods do.
|
||||
- Measured with the M2 harness (`crates/clawhdf5/tests/storage_equivalence.rs`,
|
||||
tank, 2026-09-26): one pass over the 621 corpus files that open — list,
|
||||
every attribute, every dataset once — is 176 092 `read_at` calls and
|
||||
208 MB through a storage with no cache (254 MB of files). The block
|
||||
cache of M3 is what turns that into requests (§2).
|
||||
|
||||
**M3 — HTTP/S3 backend (1–2 weeks).**
|
||||
- `clawhdf5-io`, feature `remote` (off by default, so the default tree stays
|
||||
|
||||
@@ -763,6 +763,44 @@ the same agent-store interop test.
|
||||
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
||||
which is what libhdf5 itself writes.
|
||||
|
||||
## Range reads (`File::open_storage`) limits
|
||||
|
||||
**Status:** open (added 2026-09-26, milestone M2 of
|
||||
`docs/design/range-reads.md`). `File::open_storage` reads any
|
||||
`clawhdf5_format::storage::Storage` through the whole read API, and every
|
||||
format-crate read path works through `Storage::read_at`/`read_ranges`, but:
|
||||
|
||||
- **No remote backend and no block cache yet** (milestone M3). A `Storage`
|
||||
is asked for each structure as the parsers need it, several times over
|
||||
for some (an object header is re-read by each lookup through it): one
|
||||
pass over the conformance corpus — open, list, every attribute, every
|
||||
dataset once — is 176 092 `read_at` calls for 621 files, 92 489 of them
|
||||
for the 35 001-group `h5stat_newgrat.h5` (2026-09-26, tank,
|
||||
`crates/clawhdf5/tests/storage_equivalence.rs` with
|
||||
`CLAWHDF5_STORAGE_CORPUS`). A backend over a network needs a cache in
|
||||
front of it. `Storage::read_ranges` defaults to one `read_at` per range;
|
||||
coalescing is the backend's job.
|
||||
- A group lookup by name in a version-1 (symbol-table) group lists the whole
|
||||
group (dense groups use their name index). Over a range backend that is
|
||||
one read per symbol-table node and name, per lookup.
|
||||
- External virtual-dataset source files are loaded whole through the
|
||||
resolver (`File::set_vds_resolver`), as bytes; they are not read through
|
||||
a `Storage`.
|
||||
- The zero-copy methods (`Dataset::read_raw_ref`, `read_as_slice`,
|
||||
`read_*_zerocopy`) need the file in memory and answer
|
||||
`FormatError::ContiguousStorageRequired` otherwise; `File::as_bytes()`
|
||||
panics for such a file (`File::contiguous_bytes()` is the fallible form).
|
||||
`LazyFile`, `MmapFile`, the Python and wasm bindings and `h5rs` still read
|
||||
a whole file.
|
||||
- The file's length is read once, at open: a growing file (SWMR) is not
|
||||
followed (milestone M5).
|
||||
- Not new, but visible through the equivalence tests: a full read through
|
||||
the file's chunk cache (`read_raw_data_cached`, `read_raw_data_indexed`,
|
||||
and so `Dataset::read_*`) lists a damaged dataset's chunks in hash-map
|
||||
order, so which failing chunk it reports can differ from one `File` to
|
||||
the next (`cve-2025-2310.h5`); the values of a dataset that reads are
|
||||
not affected.
|
||||
|
||||
## `clawhdf5-wasm` (browser) limits
|
||||
|
||||
**Status:** open (by design for now; added 2026-09-26).
|
||||
|
||||
Reference in New Issue
Block a user