docs: range-read milestone M2 — changelog, limits, design status

CHANGELOG (Unreleased): File::open_storage, raw data and v2 B-trees over
Storage, the tests and their corpus results (2026-09-26, tank; conformance
600 of 697, results.json identical to 8f59b2e). known-issues: what
open_storage does not do yet (no remote backend or block cache, read_at
counts of a one-pass read, v1 group lookups, whole-file VDS sources,
zero-copy methods, SWMR growth, hash-order error choice on damaged chunked
datasets). Design: M2 status and the choices made.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs/design/range-reads.md

@@ -1,9 +1,12 @@
 # Design: range reads (reading HDF5 without holding the whole file)
 
 Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
-change. Progress: M1, first part (the `Storage` trait and the metadata
-parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
-group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was
+change. Progress: M0 and M1 are done, and so is M2 (branch
+`feat/p3-m2-raw-data`): every read path of the format crate works through
+`Storage`, v2 B-trees, dense groups and raw data included, and
+`File::open_storage` gives the facade's read API over any `Storage` (see
+`CHANGELOG.md`, "Range reads, milestone M2"). M3 (a remote backend with
+its block cache) is next. Every count in §1–§2 was
 taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
 next to it. No timing numbers appear here on purpose: the machine was shared
 with other build jobs when this was written.
@@ -421,6 +424,32 @@ fast path within benchmark noise.
   on other backends (they already return `Option`/`Result`).
 - Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
   keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
+- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data`. As planned,
+  with these choices:
+  - `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the
+    file handle is shared by its datasets and may be sent across threads).
+    The file's view (user block skipped, bounded by the recorded end of
+    file, cache image laid over its reads) is itself a `Storage`, and the
+    facade calls the generic cores with it; for a `Vec` or an mmap its
+    `as_contiguous()` is the buffer, so the local paths are the slice code
+    (checked: identical conformance results; the bench gate below still
+    has to be run on an idle machine).
+  - Chunked reads fetch in batches of at most 64 MiB of stored bytes, one
+    `read_ranges` call each, so a remote read never holds more than that
+    undecoded; chunks already in the chunk cache are not fetched.
+  - The typed readers' zero-copy fast path became "read the contiguous
+    bytes once": over a range storage a contiguous `read_f64` is one read,
+    and a native contiguous selection reads only its runs.
+  - External VDS source files stay whole-file, through a resolver that
+    returns bytes (`File::set_vds_resolver`).
+  - The v2 B-tree and dense groups were done here rather than in M3, so no
+    format-crate path answers `ContiguousStorageRequired` any more; only the
+    facade's zero-copy methods do.
+  - Measured with the M2 harness (`crates/clawhdf5/tests/storage_equivalence.rs`,
+    tank, 2026-09-26): one pass over the 621 corpus files that open — list,
+    every attribute, every dataset once — is 176 092 `read_at` calls and
+    208 MB through a storage with no cache (254 MB of files). The block
+    cache of M3 is what turns that into requests (§2).
 
 **M3 — HTTP/S3 backend (1–2 weeks).**
 - `clawhdf5-io`, feature `remote` (off by default, so the default tree stays

docs/known-issues.md

@@ -763,6 +763,44 @@ the same agent-store interop test.
 **Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
 which is what libhdf5 itself writes.
 
+## Range reads (`File::open_storage`) limits
+
+**Status:** open (added 2026-09-26, milestone M2 of
+`docs/design/range-reads.md`). `File::open_storage` reads any
+`clawhdf5_format::storage::Storage` through the whole read API, and every
+format-crate read path works through `Storage::read_at`/`read_ranges`, but:
+
+- **No remote backend and no block cache yet** (milestone M3). A `Storage`
+  is asked for each structure as the parsers need it, several times over
+  for some (an object header is re-read by each lookup through it): one
+  pass over the conformance corpus — open, list, every attribute, every
+  dataset once — is 176 092 `read_at` calls for 621 files, 92 489 of them
+  for the 35 001-group `h5stat_newgrat.h5` (2026-09-26, tank,
+  `crates/clawhdf5/tests/storage_equivalence.rs` with
+  `CLAWHDF5_STORAGE_CORPUS`). A backend over a network needs a cache in
+  front of it. `Storage::read_ranges` defaults to one `read_at` per range;
+  coalescing is the backend's job.
+- A group lookup by name in a version-1 (symbol-table) group lists the whole
+  group (dense groups use their name index). Over a range backend that is
+  one read per symbol-table node and name, per lookup.
+- External virtual-dataset source files are loaded whole through the
+  resolver (`File::set_vds_resolver`), as bytes; they are not read through
+  a `Storage`.
+- The zero-copy methods (`Dataset::read_raw_ref`, `read_as_slice`,
+  `read_*_zerocopy`) need the file in memory and answer
+  `FormatError::ContiguousStorageRequired` otherwise; `File::as_bytes()`
+  panics for such a file (`File::contiguous_bytes()` is the fallible form).
+  `LazyFile`, `MmapFile`, the Python and wasm bindings and `h5rs` still read
+  a whole file.
+- The file's length is read once, at open: a growing file (SWMR) is not
+  followed (milestone M5).
+- Not new, but visible through the equivalence tests: a full read through
+  the file's chunk cache (`read_raw_data_cached`, `read_raw_data_indexed`,
+  and so `Dataset::read_*`) lists a damaged dataset's chunks in hash-map
+  order, so which failing chunk it reports can differ from one `File` to
+  the next (`cve-2025-2310.h5`); the values of a dataset that reads are
+  not affected.
+
 ## `clawhdf5-wasm` (browser) limits
 
 **Status:** open (by design for now; added 2026-09-26).