docs: range-read milestone M2 — changelog, limits, design status
CHANGELOG (Unreleased): File::open_storage, raw data and v2 B-trees over
Storage, the tests and their corpus results (2026-09-26, tank; conformance
600 of 697, results.json identical to 8f59b2e). known-issues: what
open_storage does not do yet (no remote backend or block cache, read_at
counts of a one-pass read, v1 group lookups, whole-file VDS sources,
zero-copy methods, SWMR growth, hash-order error choice on damaged chunked
datasets). Design: M2 status and the choices made.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -763,6 +763,44 @@ the same agent-store interop test.
|
||||
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
||||
which is what libhdf5 itself writes.
|
||||
|
||||
## Range reads (`File::open_storage`) limits
|
||||
|
||||
**Status:** open (added 2026-09-26, milestone M2 of
|
||||
`docs/design/range-reads.md`). `File::open_storage` reads any
|
||||
`clawhdf5_format::storage::Storage` through the whole read API, and every
|
||||
format-crate read path works through `Storage::read_at`/`read_ranges`, but:
|
||||
|
||||
- **No remote backend and no block cache yet** (milestone M3). A `Storage`
|
||||
is asked for each structure as the parsers need it, several times over
|
||||
for some (an object header is re-read by each lookup through it): one
|
||||
pass over the conformance corpus — open, list, every attribute, every
|
||||
dataset once — is 176 092 `read_at` calls for 621 files, 92 489 of them
|
||||
for the 35 001-group `h5stat_newgrat.h5` (2026-09-26, tank,
|
||||
`crates/clawhdf5/tests/storage_equivalence.rs` with
|
||||
`CLAWHDF5_STORAGE_CORPUS`). A backend over a network needs a cache in
|
||||
front of it. `Storage::read_ranges` defaults to one `read_at` per range;
|
||||
coalescing is the backend's job.
|
||||
- A group lookup by name in a version-1 (symbol-table) group lists the whole
|
||||
group (dense groups use their name index). Over a range backend that is
|
||||
one read per symbol-table node and name, per lookup.
|
||||
- External virtual-dataset source files are loaded whole through the
|
||||
resolver (`File::set_vds_resolver`), as bytes; they are not read through
|
||||
a `Storage`.
|
||||
- The zero-copy methods (`Dataset::read_raw_ref`, `read_as_slice`,
|
||||
`read_*_zerocopy`) need the file in memory and answer
|
||||
`FormatError::ContiguousStorageRequired` otherwise; `File::as_bytes()`
|
||||
panics for such a file (`File::contiguous_bytes()` is the fallible form).
|
||||
`LazyFile`, `MmapFile`, the Python and wasm bindings and `h5rs` still read
|
||||
a whole file.
|
||||
- The file's length is read once, at open: a growing file (SWMR) is not
|
||||
followed (milestone M5).
|
||||
- Not new, but visible through the equivalence tests: a full read through
|
||||
the file's chunk cache (`read_raw_data_cached`, `read_raw_data_indexed`,
|
||||
and so `Dataset::read_*`) lists a damaged dataset's chunks in hash-map
|
||||
order, so which failing chunk it reports can differ from one `File` to
|
||||
the next (`cve-2025-2310.h5`); the values of a dataset that reads are
|
||||
not affected.
|
||||
|
||||
## `clawhdf5-wasm` (browser) limits
|
||||
|
||||
**Status:** open (by design for now; added 2026-09-26).
|
||||
|
||||
Reference in New Issue
Block a user