docs/design/range-reads.md

@@ -0,0 +1,499 @@
+# Design: range reads (reading HDF5 without holding the whole file)
+
+Status: proposal, 2026-09-26. No library code has changed; this document is
+the plan for Phase 3's largest architectural change. Every count below was
+taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
+next to it. No timing numbers appear here on purpose: the machine was shared
+with other build jobs when this was written.
+
+## The problem
+
+Every reader in clawhdf5 parses the file through one `&[u8]` that covers the
+whole file:
+
+- `clawhdf5_io::HDF5Read::as_bytes(&self) -> &[u8]` is the only read method of
+  the I/O trait (`crates/clawhdf5-io/src/lib.rs`). `FileReader` reads the
+  whole file into a `Vec`, `MmapReader` maps it.
+- The facade's `File` keeps a `Backing` (`Vec<u8>` or mmap) and hands
+  `FileData::as_bytes()` — the file from the superblock to the recorded end of
+  file — to every `clawhdf5-format` call (`crates/clawhdf5/src/reader.rs`).
+- The format crate's parsers take `file_data: &[u8]` plus a `usize` address and
+  slice into it.
+- `LazyFile` (facade) parses metadata lazily, but still over
+  `HDF5Read::as_bytes`. `AsyncHDF5Read` (`clawhdf5-io`, `async` feature)
+  already has `read_at(offset, len)`, but `AsyncHDF5File::open` calls
+  `read_all()` and parses the result.
+- The browser reader (`clawhdf5-wasm`) is `open(bytes: Vec<u8>)`: the page
+  must download the whole file before it can list a group.
+
+That rules out four things users of other HDF5 readers have:
+
+1. **Remote files by range request** — libhdf5's `ros3` driver, h5py over
+   `fsspec`/`s3fs`, pyfive over `fsspec`. Today a 30 GB file on S3 has to be
+   downloaded before its tree can be listed.
+2. **Lazy loading in the browser** — h5wasm can back a file with HTTP range
+   requests; our wasm reader holds the whole file.
+3. **SWMR readers of a growing file** — the slice is fixed at open; a reader
+   cannot see the file grow (and an mmap of a growing file is fragile).
+4. **Files larger than the address space on 32-bit targets** — including
+   `wasm32`, where `usize` is 32 bits and a file above 4 GiB cannot even be put
+   in a slice; browsers cap `ArrayBuffer`s lower than that.
+
+## 1. Inventory
+
+### 1.1 Functions that take the whole file
+
+`python3 docs/design/tools/inventory.py` scans non-test source (code before a
+file's `#[cfg(test)] mod`, no `tests/`) for functions with a
+`file_data: &[u8]` parameter (the repository's convention), and — as a
+heuristic — functions with a `data`/`buf`/`file`/`bytes` slice *and* an
+integer address/offset parameter. `--list` prints every hit.
+
+- **103 functions take `file_data: &[u8]`** (73 of them `pub`), in 25 files.
+- The heuristic adds 27, of which only **6 really take the whole file under
+  another name**: `ObjectHeader::parse` and its internal `parse_v1`,
+  `parse_v1_chunk`, `parse_v2`, `parse_v2_continuation` (`data: &[u8]`), and
+  `Superblock::parse`. The other 21 are helpers (`ensure_len`,
+  `read_length`, `read_uint`, …) that bound-check whatever slice they are
+  given, or writer-side `write_at`s in `clawhdf5-io`.
+
+So **109 functions** have to change signature. Grouped by what they read:
+
+| Read pattern | Modules (whole-file functions) | Count |
+|---|---|---:|
+| **Small metadata reads at addresses** — a structure whose size is usually only known after its prefix is read | `group_v2` 9, `shared_message` 9, `fractal_heap` 7, `attribute` 6, `object_header` 5, `group_v1` 5, `extensible_array` 4, `btree_v2` 4, `btree_v1` 3, `fixed_array` 3, `local_heap` 3, `fill_value` 3, `superblock` 2, `data_layout` 1, `symbol_table` 1 | 65 |
+| **Bulk raw data** — a contiguous extent, or a list of chunk extents known once the chunk index has been walked | `chunked_read` 12, `data_read` 9, `vds` 6, `parallel_read` 3, `partial_read` 1, `provenance` 1 | 32 |
+| **Heap-resident data** — VL strings and sequences: many small reads into global-heap collections | `vl_data` 3, `global_heap` 2, facade `vlen.rs` 3, facade `types.rs` 3 | 11 |
+| Bindings | `clawhdf5-py/src/convert.rs` | 1 |
+
+### 1.2 How they read
+
+`python3 docs/design/tools/inventory.py --patterns` (non-test code):
+
+| Pattern | Where | Count |
+|---|---|---:|
+| `file_data` passed on to another function (call sites) | format 262, facade 7, py 2 | 271 |
+| `file_data[..]` slicing | format | 137 |
+| of which open-ended `file_data[x..]` (slice to end of file) | format | 5 |
+| `file_data.len()` (end-of-file bound checks) | format | 38 |
+| address/offset `as usize` casts | format 120, facade 5, io 5, ann 3 | 133 |
+| `.as_bytes()` on a file or reader, feeding format calls | facade 26, py 8, tools 1, wasm 1 | 36 |
+
+Three things in this table shape the design:
+
+- **Open-ended slices and `len()` checks** assume the whole file is present.
+  With a range reader "the rest of the file" is a request for gigabytes; each
+  of these 43 sites must become a bounded read.
+- **The 133 `as usize` casts** are where a 64-bit HDF5 address is squeezed
+  into a pointer-sized index. On 32-bit targets they truncate or must fail.
+  A storage trait that takes `u64` offsets removes most of them.
+- **The bytes escape through the public API.** `File::as_bytes()`,
+  `LazyFile::as_bytes()`, `MmapFile::as_bytes()` return the whole file, and 15
+  facade methods return `&'f [u8]`/`&'f [T]` borrowed from it
+  (`Dataset::read_raw_ref`, `read_as_slice`, `read_*_zerocopy`). The Python
+  bindings (`clawhdf5-py/src/node.rs`, `attrs.rs`, …) call
+  `clawhdf5_format` directly with `file.as_bytes()`, and so do `h5rs` and the
+  wasm reader. These must keep working for local files (they are the
+  zero-copy fast path) and fail cleanly for remote ones.
+
+### 1.3 Pieces that already exist
+
+- `clawhdf5_format::metadata_cache::MetadataCache` — LRU keyed by file offset
+  with a byte budget (2 MiB default). Not currently in the read path.
+- `clawhdf5_io::prefetch::PrefetchReader` — ring buffer of `(offset, len)`
+  entries over an `HDF5Read`, plus a sweep detector for chunk access patterns.
+- `clawhdf5_io::async_read::AsyncHDF5Read::read_at` — the right shape of
+  trait, but async and only used to read everything.
+- `clawhdf5_io::hsds` — a client for the HSDS REST service. Different
+  protocol (the server parses HDF5), not a range reader.
+
+## 2. Measurement: how chatty is a naive range reader?
+
+`docs/design/tools/range-trace/` loads a file into a page-aligned buffer,
+opens it with `clawhdf5::File::from_bytes` (today's code, unchanged),
+`mprotect`s the buffer and single-steps every load that faults (SIGSEGV logs
+the exact address and unprotects the page; the x86 trap flag re-protects it
+after one instruction). So every load instruction that touches the file is
+recorded, with no change to the library. A page loaded more than 4096 times in
+a phase is left readable and counted as wholly read. Phases: **open**
+(superblock), **list** (walk every group; shape and dtype of every dataset —
+what a tree view or `h5ls -r -v` needs), **read** (one dataset, whole). With
+the dataset's chunk extents (from h5py, `libhdf5_reads.py --extents`) the read
+phase is split into the chunk index and the raw data.
+
+Columns: *uncached requests* — a new request every time the access stream
+leaves the neighbourhood (±64 B) of the current run, i.e. a reader that turns
+each parse into a read with no cache; *ranges* — distinct byte ranges after
+merging accesses closer than 64 B; *N KiB blocks* — distinct aligned blocks,
+i.e. the requests a reader with a block cache of that size would make. For
+comparison, `libhdf5_reads.py` opens the same file in h5py through a Python
+file object (the `fileobj` driver — exactly how h5py + fsspec read remote
+files) and counts libhdf5's `read` calls, default h5py settings.
+
+```bash
+CARGO_TARGET_DIR=$PWD/target cargo build --release \
+  --manifest-path docs/design/tools/range-trace/Cargo.toml
+PY=/home/osobh/projects/clawhdf5/.venv/bin/python
+$PY docs/design/tools/libhdf5_reads.py FILE DSET --extents > ext.txt
+target/release/range-trace FILE DSET ext.txt
+$PY docs/design/tools/libhdf5_reads.py FILE DSET
+```
+
+Files (from `conformance/.cache/corpus`), 2026-09-26 on tank:
+
+- **A** `xarray-data/imerghh_730.hdf5` — NASA IMERG, 7.7 MB, superblock v0,
+  21 objects; read `/Grid/precipitation` (25 deflated chunks).
+- **B** `NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc` — netCDF-4, 48 MB,
+  superblock v2, 9 objects; read `/tas` (780 deflated chunks, 43.7 MB stored).
+- **C** `hdf5/tools/test/testfiles/h5stat_newgrat.h5` — 6.4 MB, superblock
+  v3, 35 001 groups in one dense (fractal heap + v2 B-tree) root group; read
+  `/DATASET_NAME` (a scalar with no storage).
+- **B′** B repacked with paged aggregation:
+  `h5repack -S PAGE -G 65536` (55.5 MB).
+
+clawhdf5 (range-trace):
+
+| File | Phase | Uncached requests | Ranges | Bytes in ranges | 4 KiB blocks | 64 KiB blocks | 1 MiB blocks |
+|---|---|---:|---:|---:|---:|---:|---:|
+| A | open + list | 410 | 5 | 25 285 | 8 | 3 | 2 |
+| A | read: chunk index | 33 | 8 | 4 608 | 7 | 3 | 2 |
+| A | read: raw data | (25 chunks) | 2 | 1 496 533 | 368 | 25 | 3 |
+| A | all metadata | 443 | 6 | 26 549 | 10 | 3 | 2 |
+| B | open + list | 143 | 5 | 17 951 | 7 | 1 | 1 |
+| B | read: chunk index | 46 | 19 | 49 583 | 28 | 15 | 14 |
+| B | read: raw data | (780 chunks) | 15 | 43 656 557 | 10 671 | 668 | 43 |
+| B | all metadata | 189 | 20 | 57 023 | 30 | 15 | 14 |
+| B′ | all metadata | 413 | 24 | 71 083 | 25 | 3 | 2 |
+| B′ | read: raw data | (780 chunks) | 780 | 43 613 717 | 10 920 | 780 | 49 |
+| C | open + list | 455 779 | 3 | 6 357 627 | 1 553 | 98 | 7 |
+| C | read: resolve `/DATASET_NAME` | 38 525 | 936 | 1 137 116 | 948 | 98 | 7 |
+
+libhdf5 through h5py's `fileobj` driver (read calls, bytes):
+
+| File | open | list | read | total calls | total bytes |
+|---|---:|---:|---:|---:|---:|
+| A | 3 | 93 | 25 | 121 | 1 563 559 |
+| B | 2 | 78 | 780 | 860 | 43 769 952 |
+| C | 4 | 36 105 | 1 | 36 110 | 19 725 823 |
+
+What this says:
+
+1. **A reader with no cache is hopeless.** clawhdf5's parsers revisit the same
+   structures many times (A: 26 000 loads to list 21 objects:
+   `Group::dataset(name)` and `Group::group(name)` re-read the group's whole
+   link list for every lookup). Mapped
+   one-to-one onto requests that is 410 round trips to list 21 objects, and
+   455 779 to list C. **A cache is not an optimisation, it is the design.**
+2. **With a block cache, metadata is cheap on typical files.** Metadata of A
+   and B touches 5–20 ranges and 2–15 blocks of 1 MiB; a 1 MiB-block cache
+   lists A in 2 requests and B in 1, where libhdf5 issues 96 and 80 calls
+   (fsspec's block cache absorbs those for h5py).
+3. **"Prefetch the metadata region" alone does not work.** In B the chunk
+   index (v1 B-tree nodes) is interleaved with the raw data: the metadata
+   touches 14 distinct 1 MiB blocks spread over 48 MB. Only 4 of the 611
+   corpus files h5py can open use paged aggregation (all four are libhdf5
+   test files; 502 have a v0 superblock). Repacked as B′, the same metadata
+   fits in 3 blocks of 64 KiB — paging helps a lot when present, but a reader
+   cannot count on it.
+4. **Raw data is naturally a batch.** Once the chunk index is walked, all
+   chunk extents are known: B's 780 chunks are 15 byte ranges after merging
+   neighbours, so a coalescing `get_ranges` call reads the dataset in a
+   handful of parallel requests. libhdf5 issues one read per chunk (780).
+5. **An existing inefficiency becomes a blocker.** Resolving `/DATASET_NAME`
+   in C reads 1.1 MB (936 ranges) because
+   `group_v2::resolve_path_following_links` enumerates every link of the
+   group (`resolve_group_entries`) and compares names, instead of hashing
+   the name and descending the v2 B-tree name index. On an mmap this is
+   merely slow; over HTTP it is 98 requests of 64 KiB for one lookup.
+   libhdf5 reads one 512-byte block. Listing C is inherently whole-file (35 001
+   object headers spread over the file), and libhdf5 reads 19.7 MB — three
+   times the file — to do it.
+
+Caveats: loads are logged with their first byte and a nominal width of 8 B,
+so byte totals are approximate (±64 B per range); "uncached requests" is a
+model, not a measured reader. These are counts, not timings.
+
+## 3. Options
+
+### (a) A storage trait threaded through the format crate
+
+```rust
+// clawhdf5-format (no_std + alloc)
+pub trait Storage {
+    /// Bytes [offset, offset + len). Short only at end of file.
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError>;
+    /// Current length; may grow between calls (SWMR).
+    fn len(&self) -> u64;
+    /// Batch read; backends coalesce and parallelise. Default: loop.
+    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> { .. }
+    /// The whole file as one slice, when the backend has it (Vec, mmap).
+    fn as_contiguous(&self) -> Option<&[u8]> { None }
+}
+impl Storage for [u8] { /* Cow::Borrowed, as_contiguous = Some(self) */ }
+```
+
+Every `file_data: &[u8]` becomes `file: &dyn Storage` (or `&S` generic), and
+every `file_data[a..b]` becomes `file.read_at(a, b - a)?`.
+
+- **For:** explicit, sound, errors are values (a network error is an `Err`,
+  not a panic); works for any backend; `u64` offsets fix 32-bit; `len()` can
+  grow, which is what SWMR needs; `read_ranges` gives raw data its natural
+  batch shape; `impl Storage for [u8]` lets the migration go one module at a
+  time with `&[u8]` callers unchanged.
+- **Against:** touches all 109 functions and ~400 slice/len/call sites
+  (§1.2); every parse that learns a structure's size from its prefix needs two
+  reads (prefix, then body) — cheap against a cache, but it must be written
+  that way. `Cow::Owned` results cost a copy per read for remote backends.
+- **Sync vs async:** the format crate is `no_std` and synchronous and should
+  stay so (parsing is CPU work; an async parser would colour 109 functions and
+  their callers). Remote backends are async. Two bridges, both needed:
+  - native: a backend that blocks on its own runtime (`object_store` +
+    a private tokio runtime), used from ordinary threads;
+  - wasm (no blocking on the main thread): a **restartable** mode where the
+    cache returns `FormatError::NeedBytes { offset, len }` on a miss; an async
+    driver fetches the block and re-runs the (pure, idempotent) operation.
+    parquet-rs's async reader uses the same "fetch, then parse synchronously"
+    split. The measurement bounds the retries: one per missed block, i.e. 1–15
+    for A and B's metadata at 1 MiB blocks.
+- **`&dyn` vs generic:** generics monomorphise 109 functions per backend
+  (binary size matters for wasm); `&dyn Storage` costs one indirect call per
+  structure read, negligible next to parsing. Hot raw-data loops keep their
+  speed through `as_contiguous()`.
+
+### (b) A page-cache "virtual slice"
+
+A type that implements `Index<Range<usize>, Output = [u8]>` and faults pages
+in on demand from interior-mutable storage, so parsers keep their slicing
+syntax.
+
+- `Index::index` returns `&[u8]` borrowed from `&self`. A cache that evicts
+  pages invalidates references that are still alive — unsound — so pages can
+  never be evicted while the object lives (unbounded memory, which is the
+  problem we are solving).
+- A range that crosses a page boundary has to be copied into a contiguous
+  buffer that also must outlive the borrow: an append-only arena that only
+  grows.
+- `Index` cannot fail: a network error or truncated response becomes a
+  panic, which breaks "never return wrong data; unsupported is a clean
+  error".
+- Open-ended slices (`file_data[x..]`) and `file_data.len()` still mean "the
+  whole rest of the file".
+- It hides the cost model: a `for` loop over `file_data[i]` becomes thousands
+  of cache lookups, and nothing in the signature says the call can block.
+
+Rejected. It saves the signature churn of (a) but moves every failure into
+panics and unsoundness.
+
+### (c) Keep `&[u8]` for metadata by prefetching it; range-read raw data only
+
+Fetch "the metadata" up front into a buffer, parse it with today's code, and
+range-read only chunk data.
+
+- HDF5 has no pointer to a metadata region. libhdf5's paged aggregation
+  (`H5Pset_file_space_strategy(H5F_FSPACE_STRATEGY_PAGE)`,
+  `h5repack -S PAGE -G <size>`) keeps metadata and raw data in separate pages,
+  and "setting an appropriate page size can have all internal file metadata
+  in just one page" ([Cloud-Optimized HDF/NetCDF guide][cog]) — but only
+  when the writer chose it: 4 of 611 corpus files (§2).
+- Without paging, metadata is scattered: in B the chunk index is spread over
+  14 MiB-blocks of a 48 MB file. Prefetching the first N MiB (what earthaccess'
+  block cache effectively does for "shared metadata at the file beginning"
+  ([earthaccess][ea])) is a good *heuristic*, not a correctness basis.
+- A sparse buffer still presents as a `&[u8]` of the whole file's length, so
+  every metadata read outside the prefetched part must be caught — back to (b).
+
+Useful as a **policy** on top of (a) (prefetch the first block, use the page
+size of paged files as the block size), not as the architecture.
+
+### (d) mmap a userfaultfd- or FUSE-backed file
+
+Keep `&[u8]` everywhere and let the kernel fault in remote pages
+(userfaultfd handler, or a FUSE file system doing range GETs).
+
+- Linux-only (userfaultfd) or needs a FUSE mount and privileges; nothing on
+  macOS/Windows without a kernel extension; impossible in wasm.
+- Faults are synchronous and uninterruptible from the parser's view: a network
+  error becomes `SIGBUS`, a stalled request hangs the thread.
+- No batching: raw data arrives one fault (4 KiB, or the FUSE read size) at a
+  time unless the handler guesses read-ahead.
+- 32-bit address space is still exhausted by mapping a large file.
+
+Rejected. It is how one would retrofit a C library that cannot change; we can.
+
+### How other readers do it
+
+- **libhdf5 `ros3`** replaces each POSIX read with an S3 range GET
+  ([h5py docs][h5py-file], [HDF Group, cloud storage options][hdfg-cloud]).
+  libhdf5 has a real metadata cache above the driver, and HDF5 2.2.0 added an
+  I/O block cache to ros3 "to reduce the number of requests to S3 for files
+  not using paged allocation", while paged files use the page buffer
+  ([HDF5 2.2.0 release][hdf5-220], [ros3 issue #4700][ros3-4700]). The page
+  buffer (`page_buf_size`) is "only allowed for HDF5 files created with
+  fs_strategy='page'" ([h5py docs][h5py-file]). PyPI h5py wheels do not
+  include ros3.
+- **h5py + fsspec** hands libhdf5 a Python file object; each libhdf5 read
+  becomes a `read()` on an fsspec file, whose cache decides the requests.
+  fsspec's old default `readahead` cache requested 16x more data than
+  `blockcache` when opening HDF5 files; earthaccess now defaults to
+  `blockcache` with 4–16 MiB blocks by file size ([earthaccess][ea]); the
+  Cloud-Optimized guide recommends `blockcache` plus h5py's `page_buf_size`
+  and `rdcc_nbytes` ([guide][cog]). Our §2 comparison uses exactly this
+  path, without fsspec's cache.
+- **pyfive** (pure Python, like us) supports lazy loading "on both Posix and
+  S3 filesystems" through fsspec, reads a variable's attributes and chunk
+  B-tree when it is accessed, and can merge chunk range requests with
+  fsspec's `merge_range_requests` ([pyfive][pyfive], [pyfive #257][pyfive-257]).
+- **h5wasm** (libhdf5 compiled with Emscripten) backs a file with
+  `FS.createLazyFile`, which "can use range requests to incrementally access
+  the h5 file over the wire" ([h5wasm][h5wasm]); Emscripten's lazy files use
+  synchronous XHR, which browsers only allow in Web Workers
+  ([Emscripten FS API][emfs]). Files larger than memory remain an open issue
+  ([h5wasm #40][h5wasm-40]).
+- **jsfive** (pure JS port of pyfive) reads from an `ArrayBuffer` of the whole
+  file ([jsfive][jsfive]); no lazy loading.
+- **Rust `object_store`** (0.14): one `ObjectStore` trait over S3, GCS, Azure,
+  HTTP/WebDAV, local files and memory; `get_ranges` "will automatically
+  coalesce adjacent ranges into an appropriate number of parallel requests",
+  with `OBJECT_STORE_COALESCE_DEFAULT` as the gap below which ranges merge
+  ([docs.rs][os]). It builds for wasm32 except the local-filesystem and
+  some chunked-upload parts.
+
+## 4. Recommendation and migration plan
+
+Adopt **(a)**, with a block cache as a required part of every non-local
+backend, **(c)** as a cache policy, and the wasm path through the restartable
+`NeedBytes` mode. Every milestone keeps `main` green: `cargo test
+--workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap
+fast path within benchmark noise.
+
+**M0 — prerequisites (≈1 week).**
+- Dense-group name lookup through the v2 B-tree name index (Jenkins hash,
+  record type 5), instead of enumerating all links; same for creation-order
+  and attribute name lookups. §2 shows 936 ranges → a handful. Regression
+  test: count links decoded for one lookup in a 35 001-link group.
+- One checked helper for address → index conversion; replace the 133
+  `as usize` casts. On 32-bit an address past `usize::MAX` is a clean error
+  (today it truncates or panics).
+- `Group::dataset(name)`/`Group::group(name)` call `children()`, which
+  re-reads the group's whole link list on every lookup, so walking a group of
+  n children decodes its links O(n) times. Look names up through the index
+  (above) and let a listing hand out its entries, so the cache has less to
+  absorb.
+
+**M1 — metadata over the trait, in-memory impl identical to today (2–3 weeks).**
+- Add `Storage` (above) to `clawhdf5-format`, `no_std`-compatible, with
+  `impl Storage for [u8]`. Add a storage error variant — `FormatError` and the
+  facade `Error` are not `#[non_exhaustive]`, so this is a breaking change for
+  exhaustive matches: bundle it with the next major version, or mark both
+  enums `#[non_exhaustive]` first.
+- Convert modules bottom-up — superblock, object header, local/global heap,
+  B-tree v1/v2, fractal heap, fixed/extensible array, symbol table, group
+  v1/v2, shared messages, attributes, fill value, data layout — one commit
+  each. The old `&[u8]` signature stays as a thin wrapper over the new one
+  (`fn parse(data: &[u8], ..) { parse_in(data as &dyn Storage, ..) }`), so
+  callers and the other crates don't move yet.
+- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.
+
+**M2 — raw data over the trait (1–2 weeks).**
+- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`,
+  VL/global heap. Chunked reads first collect the chunk extents the selection
+  needs, then call `read_ranges` once and decompress in parallel as today.
+- Zero-copy stays: when `as_contiguous()` is `Some`, contiguous reads return
+  borrowed slices; `read_raw_ref`/`read_*_zerocopy`/`File::as_bytes` keep
+  their signatures and return a clear "not available for this storage" error
+  on other backends (they already return `Option`/`Result`).
+- Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
+  keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
+
+**M3 — HTTP/S3 backend (1–2 weeks).**
+- `clawhdf5-io`, feature `remote` (off by default, so the default tree stays
+  free of C and TLS stacks): `RangeStorage` over `object_store` (HTTP, S3, GCS,
+  Azure), with a `BlockCache` (LRU, block size configurable, default 1 MiB per
+  §2; the page size for paged files; the first block prefetched on open) and
+  a request counter exposed for tests and users.
+- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
+
+**M4 — wasm lazy loading (1–2 weeks).**
+- `clawhdf5-wasm`: `openUrl(url) -> Promise<H5File>` backed by `fetch` with a
+  `Range` header, on the main thread, via the restartable `NeedBytes` loop (no
+  Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
+  to a whole download when the server does not answer 206.
+- `examples/wasm-viewer`: open by URL.
+
+**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
+add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
+blocks past the old end. Needs libhdf5 SWMR semantics research first.
+
+Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured).
+
+### Keeping the local fast path
+
+- `impl Storage for [u8]` returns `Cow::Borrowed` — no copy, no allocation.
+- Hot loops (raw-data copies, contiguous typed reads, `read_selection_native`)
+  branch once on `as_contiguous()` and then run today's code.
+- `&dyn` dispatch is per structure, not per byte; parse code keeps working on
+  the returned slice.
+- Gate: `crates/clawhdf5/benches/mmap_bench.rs`, the concurrent-read benches in
+  `clawhdf5-bench`, and the conformance run time, before and after each M1/M2
+  commit, on an otherwise idle machine. Anything outside noise blocks the
+  commit.
+
+### Risks
+
+- **Silent regressions on local files** — mitigated by the bench gate above
+  and by M1 being a pure refactor (every conformance hash identical).
+- **Two reads per structure** (prefix, then body) could double requests on a
+  cold cache. Block-aligned caching makes the second read a hit; the measured
+  block counts already include this pattern.
+- **API break** — new error variant (see M1); `as_bytes()`-style APIs become
+  fallible for non-local storage. ClawBrainHub uses `File`, `FileBuilder`,
+  `AttrValue`, `Selection` on local files only, so it is unaffected by
+  behaviour, only by exhaustive matches on `Error`.
+- **Restartable parsing** assumes operations are pure over the storage. The
+  chunk cache and metadata cache must only be filled by completed reads.
+- **Cache memory** — the block cache needs a byte budget and eviction, which
+  (unlike option (b)) is sound because parsers hold `Cow`s, not borrows into
+  the cache.
+
+### Testing
+
+- **In-memory equivalence:** a `CountingStorage` wrapper over `[u8]` that
+  records every `read_at`; run the conformance probe through it and require
+  identical results. It also produces the request counts of §2 without the
+  trap-flag tracer.
+- **Adversarial storage:** a wrapper that returns short reads, errors on the
+  Nth request, or serves blocks of 1 byte, to prove every miss is an `Err`
+  and never wrong data.
+- **HTTP:** an in-process server on `127.0.0.1` (std `TcpListener`) that
+  honours `Range`, answers 206/416, can refuse ranges (200), and counts
+  requests. Tests assert both correctness against h5py and **request budgets**
+  (e.g. listing file A with a 1 MiB block cache takes ≤ 3 requests) so a
+  change that makes the reader chattier fails CI.
+- **S3:** `object_store`'s in-memory store in unit tests; a MinIO or real
+  bucket only in an opt-in job.
+- **wasm:** the existing Node/Chromium harness in `examples/wasm-viewer/test`
+  with a local range-capable server.
+- **32-bit:** a `wasm32` or `i686` build that opens a sparse > 4 GiB file
+  through the counting storage.
+
+## Tools
+
+- `docs/design/tools/inventory.py` — §1 tables (`--list`, `--patterns`).
+- `docs/design/tools/range-trace/` — §2 tracer (standalone crate, x86-64
+  Linux, not part of the workspace).
+- `docs/design/tools/libhdf5_reads.py` — §2 libhdf5 comparison and chunk
+  extents.
+
+[cog]: https://guide.cloudnativegeo.org/cloud-optimized-netcdf4-hdf5/
+[ea]: https://earthaccess.readthedocs.io/en/latest/user/explanation/fsspec/
+[h5py-file]: https://docs.h5py.org/en/stable/high/file.html
+[hdfg-cloud]: https://www.hdfgroup.org/2022/08/08/cloud-storage-options-for-hdf5/
+[hdf5-220]: https://www.hdfgroup.org/2026/07/30/release-of-hdf5-2-2-0-and-two-august-events-newsletter-210/
+[ros3-4700]: https://github.com/HDFGroup/hdf5/issues/4700
+[pyfive]: https://pyfive.readthedocs.io/en/latest/quickstart/usage.html
+[pyfive-257]: https://github.com/NCAS-CMS/pyfive/issues/257
+[h5wasm]: https://github.com/usnistgov/h5wasm
+[h5wasm-40]: https://github.com/usnistgov/h5wasm/issues/40
+[emfs]: https://emscripten.org/docs/api_reference/Filesystem-API.html
+[jsfive]: https://github.com/usnistgov/jsfive
+[os]: https://docs.rs/object_store/latest/object_store/