docs: range-read milestone M2 — changelog, limits, design status

CHANGELOG (Unreleased): File::open_storage, raw data and v2 B-trees over
Storage, the tests and their corpus results (2026-09-26, tank; conformance
600 of 697, results.json identical to 8f59b2e). known-issues: what
open_storage does not do yet (no remote backend or block cache, read_at
counts of a one-pass read, v1 group lookups, whole-file VDS sources,
zero-copy methods, SWMR growth, hash-order error choice on damaged chunked
datasets). Design: M2 status and the choices made.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

CHANGELOG.md

@@ -2,6 +2,97 @@
 
 ## Unreleased
 
+### Range reads, milestone M2: raw data and `File::open_storage` (2026-09-26)
+- **`clawhdf5::File::open_storage(Arc<dyn Storage + Send + Sync>)`** opens
+  a file served by any `clawhdf5_format::storage::Storage` and gives the
+  whole read API over it: groups and paths, datasets, attributes, the
+  `read_*` methods, selections, variable-length strings and sequences, and
+  virtual datasets. Every byte comes through `Storage::read_at` /
+  `read_ranges`; a user block is found and skipped, nothing past the
+  superblock's end of file is read, and a metadata cache image is laid over
+  the reads it covers (new `CacheImage::entries`). External virtual-dataset
+  sources are read through the new `File::set_vds_resolver` (any `File`;
+  without one a storage-backed file cannot follow them). `File::open` and
+  `File::from_bytes` keep their mmap and in-memory paths: the file's view
+  is now a `Storage` whose `as_contiguous()` is that buffer, and every hot
+  loop takes it. New exports: `clawhdf5::{Storage, SharedStorage,
+  VdsResolver}`, `File::contiguous_bytes()`.
+  - Over a storage without the whole file in memory the zero-copy methods
+    (`read_raw_ref`, `read_as_slice`, `read_*_zerocopy`) answer
+    `FormatError::ContiguousStorageRequired`, and `File::as_bytes` panics
+    (documented; use `contiguous_bytes`). The typed readers keep their fast
+    paths: a contiguous dataset is read in one piece and converted, and a
+    contiguous selection of a native type reads only its runs
+    (`data_read::read_selection_native_in`).
+- **Nothing in the format crate needs the whole file any more.** The
+  structures M1 left to `ContiguousStorageRequired` read through `Storage`:
+  v2 B-trees (`BTreeV2Header::parse_in`, `collect_btree_v2_records_in`,
+  `find_btree_v2_records_in`; one bounded read per node, whose size is known
+  before it is read), hence dense attributes, a SOHM B-tree index, huge
+  fractal-heap objects, and dense groups; v1 and v2 group listings, lookups
+  and paths (`group_v2::resolve_group_children_in`, `resolve_child_in`,
+  `resolve_path_any_in`, `group_v1::*_in`).
+- **Raw data reads through `Storage`**, each with a generic `*_in` core and
+  its `&[u8]` function as a thin wrapper (callers do not change):
+  `data_read` (`read_raw_data*_in`, `read_raw_data_selection_in`,
+  `read_chunked_native_in`), `chunked_read` (the v1 B-tree chunk index —
+  one read of each node's header, one of its entries — `list_chunks_in`,
+  and the full, cached, sweep and indexed reads), `parallel_read`,
+  `partial_read`, `fill_value` (`read_full_with_fill_in`,
+  `apply_to_unallocated_chunks_in`; `dataset_fill_value_from_storage` is
+  now generic, so a `&dyn Storage` still works), `vds`
+  (`read_virtual_dataset_in`, `virtual_dataset_extent_in`: the virtual
+  file through `Storage`, external source files still loaded whole through
+  the resolver), `vl_data` (`VlResolver<'a, S = [u8]>` with `new_in`;
+  `read_vl_strings_in`, `read_vl_bytes_in`),
+  `AttributeMessage::read_vl_strings_in`, `provenance::verify_dataset_in`.
+  - A chunked read first lists the chunks it needs, then fetches all their
+    stored bytes with **one `read_ranges` call** (per 64 MiB of stored
+    data), so a remote backend can coalesce and parallelise them, then
+    decodes as before (in parallel with the `parallel` feature). Chunks the
+    file's chunk cache already holds are not fetched. A selection fetches
+    only the chunks its bounding box overlaps; a contiguous selection only
+    its runs (adjacent ones merged). A global-heap collection is read once
+    per resolver and kept (within the resolver's 32 MiB budget).
+  - Each extent's bounds error is the one the slice readers gave, reported
+    when the read reaches that extent, so a damaged file fails with the
+    same error, in the same order, through either path.
+- **No behaviour change for in-memory and mapped files:** with
+  `as_contiguous()` every path slices the file as before (checked below).
+- Tests (2026-09-26, tank):
+  - `clawhdf5-format/tests/storage_equivalence.rs` now also reads every
+    dataset — whole, fill-aware, through a chunk cache (twice) and the
+    indexed path, three selections, virtual datasets with their sibling
+    sources, VL strings, sequences and bytes — through the read_at-only
+    `CountingStorage` and requires the slice results, and fails on any
+    `ContiguousStorageRequired`. With
+    `CLAWHDF5_STORAGE_CORPUS=conformance/.cache/corpus`, all 653 HDF5
+    files of the corpus agree (82 396 checks). The cached and indexed
+    paths are compared on values only when a read fails: they order
+    chunks by hash map, so which failing chunk a damaged dataset reports
+    varies between two caches even for the same slice (seen on
+    `cve-2025-2310.h5`; see `docs/known-issues.md`).
+  - A misbehaving storage (fails its N-th read; serves short reads) over
+    every fixture: each listing and dataset read is an error or exactly the
+    in-memory result, never other data (1 137 runs).
+  - A chunked read issues one `read_ranges` call with one range per chunk,
+    and a one-chunk selection one call with one range.
+  - `clawhdf5/tests/storage_equivalence.rs` reads every fixture (61 files)
+    and, with `CLAWHDF5_STORAGE_CORPUS`, every corpus file (701 files, 621
+    that open) through `File::open` and through `File::open_storage` over
+    `CountingStorage`: the tree, every attribute (all, and each by name),
+    every dataset's shape, types and values (all bytes, `f64`, `f32`,
+    `i64`, a hyperslab, strings, VL sequences) must be identical, and are.
+    It also counts what one pass — open, list, read every attribute and
+    every dataset once — asks of a storage with no cache: 176 092 `read_at`
+    calls and 208 MB for the 621 corpus files (254 MB of files); the most
+    are `h5stat_newgrat.h5` (35 001 groups: 92 489 calls) and
+    `ref_hdf5_compat1.nc` (16 062). A remote backend needs the block cache
+    of milestone M3. Command: `CLAWHDF5_STORAGE_CORPUS=… cargo test
+    --release -p clawhdf5 --test storage_equivalence -- --nocapture`.
+  - Conformance sweep (`conformance/run.sh --no-fetch`): 600 of 697 files
+    ok, `results.json` byte-identical to `8f59b2e`.
+
 ### Name lookups through the name index (2026-09-26)
 - **Finding one link or attribute by name reads the name index, not every
   entry.** In a dense group (links in a fractal heap) the v2 B-tree name
@@ -108,7 +199,8 @@
   v2 B-tree and dense groups come with milestone M3); over a backend without
   the whole file in memory they are the clean `ContiguousStorageRequired`
   error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL
-  data are milestone M2.
+  data are milestone M2. (All of them read through `Storage` since M2,
+  above.)
 - **No behaviour change**, checked three ways (2026-09-26, tank): every
   existing test passes unchanged; the conformance sweep
   (`conformance/run.sh --no-fetch`) gives a byte-identical `results.json`