clawhdf5/docs/design/range-reads.md

# Design: range reads (reading HDF5 without holding the whole file)

Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
change. Progress: M0 and M1 are done, and so is M2 (branch
`feat/p3-m2-raw-data`): every read path of the format crate works through
`Storage`, v2 B-trees, dense groups and raw data included, and
`File::open_storage` gives the facade's read API over any `Storage` (see
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
next. Every count in §1–§2 was
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
next to it. No timing numbers appear here on purpose: the machine was shared
with other build jobs when this was written.

## The problem

Every reader in clawhdf5 parses the file through one `&[u8]` that covers the
whole file:

- `clawhdf5_io::HDF5Read::as_bytes(&self) -> &[u8]` is the only read method of
  the I/O trait (`crates/clawhdf5-io/src/lib.rs`). `FileReader` reads the
  whole file into a `Vec`, `MmapReader` maps it.
- The facade's `File` keeps a `Backing` (`Vec<u8>` or mmap) and hands
  `FileData::as_bytes()` — the file from the superblock to the recorded end of
  file — to every `clawhdf5-format` call (`crates/clawhdf5/src/reader.rs`).
- The format crate's parsers take `file_data: &[u8]` plus a `usize` address and
  slice into it.
- `LazyFile` (facade) parses metadata lazily, but still over
  `HDF5Read::as_bytes`. `AsyncHDF5Read` (`clawhdf5-io`, `async` feature)
  already has `read_at(offset, len)`, but `AsyncHDF5File::open` calls
  `read_all()` and parses the result.
- The browser reader (`clawhdf5-wasm`) is `open(bytes: Vec<u8>)`: the page
  must download the whole file before it can list a group.

That rules out four things users of other HDF5 readers have:

1. **Remote files by range request** — libhdf5's `ros3` driver, h5py over
   `fsspec`/`s3fs`, pyfive over `fsspec`. Today a 30 GB file on S3 has to be
   downloaded before its tree can be listed.
2. **Lazy loading in the browser** — h5wasm can back a file with HTTP range
   requests; our wasm reader holds the whole file.
3. **SWMR readers of a growing file** — the slice is fixed at open; a reader
   cannot see the file grow (and an mmap of a growing file is fragile).
4. **Files larger than the address space on 32-bit targets** — including
   `wasm32`, where `usize` is 32 bits and a file above 4 GiB cannot even be put
   in a slice; browsers cap `ArrayBuffer`s lower than that.

## 1. Inventory

### 1.1 Functions that take the whole file

`python3 docs/design/tools/inventory.py` scans non-test source (code before a
file's `#[cfg(test)] mod`, no `tests/`) for functions with a
`file_data: &[u8]` parameter (the repository's convention), and — as a
heuristic — functions with a `data`/`buf`/`file`/`bytes` slice *and* an
integer address/offset parameter. `--list` prints every hit.

- **103 functions take `file_data: &[u8]`** (73 of them `pub`), in 25 files.
- The heuristic adds 27, of which only **6 really take the whole file under
  another name**: `ObjectHeader::parse` and its internal `parse_v1`,
  `parse_v1_chunk`, `parse_v2`, `parse_v2_continuation` (`data: &[u8]`), and
  `Superblock::parse`. The other 21 are helpers (`ensure_len`,
  `read_length`, `read_uint`, …) that bound-check whatever slice they are
  given, or writer-side `write_at`s in `clawhdf5-io`.

So **109 functions** have to change signature. Grouped by what they read:

| Read pattern | Modules (whole-file functions) | Count |
|---|---|---:|
| **Small metadata reads at addresses** — a structure whose size is usually only known after its prefix is read | `group_v2` 9, `shared_message` 9, `fractal_heap` 7, `attribute` 6, `object_header` 5, `group_v1` 5, `extensible_array` 4, `btree_v2` 4, `btree_v1` 3, `fixed_array` 3, `local_heap` 3, `fill_value` 3, `superblock` 2, `data_layout` 1, `symbol_table` 1 | 65 |
| **Bulk raw data** — a contiguous extent, or a list of chunk extents known once the chunk index has been walked | `chunked_read` 12, `data_read` 9, `vds` 6, `parallel_read` 3, `partial_read` 1, `provenance` 1 | 32 |
| **Heap-resident data** — VL strings and sequences: many small reads into global-heap collections | `vl_data` 3, `global_heap` 2, facade `vlen.rs` 3, facade `types.rs` 3 | 11 |
| Bindings | `clawhdf5-py/src/convert.rs` | 1 |

### 1.2 How they read

`python3 docs/design/tools/inventory.py --patterns` (non-test code):

| Pattern | Where | Count |
|---|---|---:|
| `file_data` passed on to another function (call sites) | format 262, facade 7, py 2 | 271 |
| `file_data[..]` slicing | format | 137 |
| of which open-ended `file_data[x..]` (slice to end of file) | format | 5 |
| `file_data.len()` (end-of-file bound checks) | format | 38 |
| address/offset `as usize` casts | format 120, facade 5, io 5, ann 3 | 133 |
| `.as_bytes()` on a file or reader, feeding format calls | facade 26, py 8, tools 1, wasm 1 | 36 |

Three things in this table shape the design:

- **Open-ended slices and `len()` checks** assume the whole file is present.
  With a range reader "the rest of the file" is a request for gigabytes; each
  of these 43 sites must become a bounded read.
- **The 133 `as usize` casts** are where a 64-bit HDF5 address is squeezed
  into a pointer-sized index. On 32-bit targets they truncate or must fail.
  A storage trait that takes `u64` offsets removes most of them.
- **The bytes escape through the public API.** `File::as_bytes()`,
  `LazyFile::as_bytes()`, `MmapFile::as_bytes()` return the whole file, and 15
  facade methods return `&'f [u8]`/`&'f [T]` borrowed from it
  (`Dataset::read_raw_ref`, `read_as_slice`, `read_*_zerocopy`). The Python
  bindings (`clawhdf5-py/src/node.rs`, `attrs.rs`, …) call
  `clawhdf5_format` directly with `file.as_bytes()`, and so do `h5rs` and the
  wasm reader. These must keep working for local files (they are the
  zero-copy fast path) and fail cleanly for remote ones.

### 1.3 Pieces that already exist

- `clawhdf5_format::metadata_cache::MetadataCache` — LRU keyed by file offset
  with a byte budget (2 MiB default). Not currently in the read path.
- `clawhdf5_io::prefetch::PrefetchReader` — ring buffer of `(offset, len)`
  entries over an `HDF5Read`, plus a sweep detector for chunk access patterns.
- `clawhdf5_io::async_read::AsyncHDF5Read::read_at` — the right shape of
  trait, but async and only used to read everything.
- `clawhdf5_io::hsds` — a client for the HSDS REST service. Different
  protocol (the server parses HDF5), not a range reader.

## 2. Measurement: how chatty is a naive range reader?

`docs/design/tools/range-trace/` loads a file into a page-aligned buffer,
opens it with `clawhdf5::File::from_bytes` (today's code, unchanged),
`mprotect`s the buffer and single-steps every load that faults (SIGSEGV logs
the exact address and unprotects the page; the x86 trap flag re-protects it
after one instruction). So every load instruction that touches the file is
recorded, with no change to the library. A page loaded more than 4096 times in
a phase is left readable and counted as wholly read. Phases: **open**
(superblock), **list** (walk every group; shape and dtype of every dataset —
what a tree view or `h5ls -r -v` needs), **read** (one dataset, whole). With
the dataset's chunk extents (from h5py, `libhdf5_reads.py --extents`) the read
phase is split into the chunk index and the raw data.

Columns: *uncached requests* — a new request every time the access stream
leaves the neighbourhood (±64 B) of the current run, i.e. a reader that turns
each parse into a read with no cache; *ranges* — distinct byte ranges after
merging accesses closer than 64 B; *N KiB blocks* — distinct aligned blocks,
i.e. the requests a reader with a block cache of that size would make. For
comparison, `libhdf5_reads.py` opens the same file in h5py through a Python
file object (the `fileobj` driver — exactly how h5py + fsspec read remote
files) and counts libhdf5's `read` calls, default h5py settings.

```bash
CARGO_TARGET_DIR=$PWD/target cargo build --release \
  --manifest-path docs/design/tools/range-trace/Cargo.toml
PY=/home/osobh/projects/clawhdf5/.venv/bin/python
$PY docs/design/tools/libhdf5_reads.py FILE DSET --extents > ext.txt
target/release/range-trace FILE DSET ext.txt
$PY docs/design/tools/libhdf5_reads.py FILE DSET
```

Files (from `conformance/.cache/corpus`), 2026-09-26 on tank:

- **A** `xarray-data/imerghh_730.hdf5` — NASA IMERG, 7.7 MB, superblock v0,
  21 objects; read `/Grid/precipitation` (25 deflated chunks).
- **B** `NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc` — netCDF-4, 48 MB,
  superblock v2, 9 objects; read `/tas` (780 deflated chunks, 43.7 MB stored).
- **C** `hdf5/tools/test/testfiles/h5stat_newgrat.h5` — 6.4 MB, superblock
  v3, 35 001 groups in one dense (fractal heap + v2 B-tree) root group; read
  `/DATASET_NAME` (a scalar with no storage).
- **B′** B repacked with paged aggregation:
  `h5repack -S PAGE -G 65536` (55.5 MB).

clawhdf5 (range-trace):

| File | Phase | Uncached requests | Ranges | Bytes in ranges | 4 KiB blocks | 64 KiB blocks | 1 MiB blocks |
|---|---|---:|---:|---:|---:|---:|---:|
| A | open + list | 410 | 5 | 25 285 | 8 | 3 | 2 |
| A | read: chunk index | 33 | 8 | 4 608 | 7 | 3 | 2 |
| A | read: raw data | (25 chunks) | 2 | 1 496 533 | 368 | 25 | 3 |
| A | all metadata | 443 | 6 | 26 549 | 10 | 3 | 2 |
| B | open + list | 143 | 5 | 17 951 | 7 | 1 | 1 |
| B | read: chunk index | 46 | 19 | 49 583 | 28 | 15 | 14 |
| B | read: raw data | (780 chunks) | 15 | 43 656 557 | 10 671 | 668 | 43 |
| B | all metadata | 189 | 20 | 57 023 | 30 | 15 | 14 |
| B′ | all metadata | 413 | 24 | 71 083 | 25 | 3 | 2 |
| B′ | read: raw data | (780 chunks) | 780 | 43 613 717 | 10 920 | 780 | 49 |
| C | open + list | 455 779 | 3 | 6 357 627 | 1 553 | 98 | 7 |
| C | read: resolve `/DATASET_NAME` | 38 525 | 936 | 1 137 116 | 948 | 98 | 7 |

libhdf5 through h5py's `fileobj` driver (read calls, bytes):

| File | open | list | read | total calls | total bytes |
|---|---:|---:|---:|---:|---:|
| A | 3 | 93 | 25 | 121 | 1 563 559 |
| B | 2 | 78 | 780 | 860 | 43 769 952 |
| C | 4 | 36 105 | 1 | 36 110 | 19 725 823 |

What this says:

1. **A reader with no cache is hopeless.** clawhdf5's parsers revisit the same
   structures many times (A: 26 000 loads to list 21 objects:
   `Group::dataset(name)` and `Group::group(name)` re-read the group's whole
   link list for every lookup). Mapped
   one-to-one onto requests that is 410 round trips to list 21 objects, and
   455 779 to list C. **A cache is not an optimisation, it is the design.**
2. **With a block cache, metadata is cheap on typical files.** Metadata of A
   and B touches 5–20 ranges and 2–15 blocks of 1 MiB; a 1 MiB-block cache
   lists A in 2 requests and B in 1, where libhdf5 issues 96 and 80 calls
   (fsspec's block cache absorbs those for h5py).
3. **"Prefetch the metadata region" alone does not work.** In B the chunk
   index (v1 B-tree nodes) is interleaved with the raw data: the metadata
   touches 14 distinct 1 MiB blocks spread over 48 MB. Only 4 of the 611
   corpus files h5py can open use paged aggregation (all four are libhdf5
   test files; 502 have a v0 superblock). Repacked as B′, the same metadata
   fits in 3 blocks of 64 KiB — paging helps a lot when present, but a reader
   cannot count on it.
4. **Raw data is naturally a batch.** Once the chunk index is walked, all
   chunk extents are known: B's 780 chunks are 15 byte ranges after merging
   neighbours, so a coalescing `get_ranges` call reads the dataset in a
   handful of parallel requests. libhdf5 issues one read per chunk (780).
5. **An existing inefficiency becomes a blocker.** Resolving `/DATASET_NAME`
   in C reads 1.1 MB (936 ranges) because
   `group_v2::resolve_path_following_links` enumerates every link of the
   group (`resolve_group_entries`) and compares names, instead of hashing
   the name and descending the v2 B-tree name index. On an mmap this is
   merely slow; over HTTP it is 98 requests of 64 KiB for one lookup.
   libhdf5 reads one 512-byte block. Listing C is inherently whole-file (35 001
   object headers spread over the file), and libhdf5 reads 19.7 MB — three
   times the file — to do it.

Caveats: loads are logged with their first byte and a nominal width of 8 B,
so byte totals are approximate (±64 B per range); "uncached requests" is a
model, not a measured reader. These are counts, not timings.

## 3. Options

### (a) A storage trait threaded through the format crate

```rust
// clawhdf5-format (no_std + alloc)
pub trait Storage {
    /// Bytes [offset, offset + len). Short only at end of file.
    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError>;
    /// Current length; may grow between calls (SWMR).
    fn len(&self) -> u64;
    /// Batch read; backends coalesce and parallelise. Default: loop.
    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> { .. }
    /// The whole file as one slice, when the backend has it (Vec, mmap).
    fn as_contiguous(&self) -> Option<&[u8]> { None }
}
impl Storage for [u8] { /* Cow::Borrowed, as_contiguous = Some(self) */ }
```

Every `file_data: &[u8]` becomes `file: &dyn Storage` (or `&S` generic), and
every `file_data[a..b]` becomes `file.read_at(a, b - a)?`.

- **For:** explicit, sound, errors are values (a network error is an `Err`,
  not a panic); works for any backend; `u64` offsets fix 32-bit; `len()` can
  grow, which is what SWMR needs; `read_ranges` gives raw data its natural
  batch shape; `impl Storage for [u8]` lets the migration go one module at a
  time with `&[u8]` callers unchanged.
- **Against:** touches all 109 functions and ~400 slice/len/call sites
  (§1.2); every parse that learns a structure's size from its prefix needs two
  reads (prefix, then body) — cheap against a cache, but it must be written
  that way. `Cow::Owned` results cost a copy per read for remote backends.
- **Sync vs async:** the format crate is `no_std` and synchronous and should
  stay so (parsing is CPU work; an async parser would colour 109 functions and
  their callers). Remote backends are async. Two bridges, both needed:
  - native: a backend that blocks on its own runtime (`object_store` +
    a private tokio runtime), used from ordinary threads;
  - wasm (no blocking on the main thread): a **restartable** mode where the
    cache returns `FormatError::NeedBytes { offset, len }` on a miss; an async
    driver fetches the block and re-runs the (pure, idempotent) operation.
    parquet-rs's async reader uses the same "fetch, then parse synchronously"
    split. The measurement bounds the retries: one per missed block, i.e. 1–15
    for A and B's metadata at 1 MiB blocks.
- **`&dyn` vs generic:** generics monomorphise 109 functions per backend
  (binary size matters for wasm); `&dyn Storage` costs one indirect call per
  structure read, negligible next to parsing. Hot raw-data loops keep their
  speed through `as_contiguous()`.
  *M1 outcome:* `&dyn` was not negligible for small structures — with it,
  `ObjectHeader::parse` was ~25% and a 400-group facade listing ~14% slower
  than the slice code (provisional, shared machine). The cores are now
  generic (`S: Storage + ?Sized`), so the `&[u8]` wrappers get a `[u8]`
  instance and `dyn Storage` is one more instance, not one per backend.

### (b) A page-cache "virtual slice"

A type that implements `Index<Range<usize>, Output = [u8]>` and faults pages
in on demand from interior-mutable storage, so parsers keep their slicing
syntax.

- `Index::index` returns `&[u8]` borrowed from `&self`. A cache that evicts
  pages invalidates references that are still alive — unsound — so pages can
  never be evicted while the object lives (unbounded memory, which is the
  problem we are solving).
- A range that crosses a page boundary has to be copied into a contiguous
  buffer that also must outlive the borrow: an append-only arena that only
  grows.
- `Index` cannot fail: a network error or truncated response becomes a
  panic, which breaks "never return wrong data; unsupported is a clean
  error".
- Open-ended slices (`file_data[x..]`) and `file_data.len()` still mean "the
  whole rest of the file".
- It hides the cost model: a `for` loop over `file_data[i]` becomes thousands
  of cache lookups, and nothing in the signature says the call can block.

Rejected. It saves the signature churn of (a) but moves every failure into
panics and unsoundness.

### (c) Keep `&[u8]` for metadata by prefetching it; range-read raw data only

Fetch "the metadata" up front into a buffer, parse it with today's code, and
range-read only chunk data.

- HDF5 has no pointer to a metadata region. libhdf5's paged aggregation
  (`H5Pset_file_space_strategy(H5F_FSPACE_STRATEGY_PAGE)`,
  `h5repack -S PAGE -G <size>`) keeps metadata and raw data in separate pages,
  and "setting an appropriate page size can have all internal file metadata
  in just one page" ([Cloud-Optimized HDF/NetCDF guide][cog]) — but only
  when the writer chose it: 4 of 611 corpus files (§2).
- Without paging, metadata is scattered: in B the chunk index is spread over
  14 MiB-blocks of a 48 MB file. Prefetching the first N MiB (what earthaccess'
  block cache effectively does for "shared metadata at the file beginning"
  ([earthaccess][ea])) is a good *heuristic*, not a correctness basis.
- A sparse buffer still presents as a `&[u8]` of the whole file's length, so
  every metadata read outside the prefetched part must be caught — back to (b).

Useful as a **policy** on top of (a) (prefetch the first block, use the page
size of paged files as the block size), not as the architecture.

### (d) mmap a userfaultfd- or FUSE-backed file

Keep `&[u8]` everywhere and let the kernel fault in remote pages
(userfaultfd handler, or a FUSE file system doing range GETs).

- Linux-only (userfaultfd) or needs a FUSE mount and privileges; nothing on
  macOS/Windows without a kernel extension; impossible in wasm.
- Faults are synchronous and uninterruptible from the parser's view: a network
  error becomes `SIGBUS`, a stalled request hangs the thread.
- No batching: raw data arrives one fault (4 KiB, or the FUSE read size) at a
  time unless the handler guesses read-ahead.
- 32-bit address space is still exhausted by mapping a large file.

Rejected. It is how one would retrofit a C library that cannot change; we can.

### How other readers do it

- **libhdf5 `ros3`** replaces each POSIX read with an S3 range GET
  ([h5py docs][h5py-file], [HDF Group, cloud storage options][hdfg-cloud]).
  libhdf5 has a real metadata cache above the driver, and HDF5 2.2.0 added an
  I/O block cache to ros3 "to reduce the number of requests to S3 for files
  not using paged allocation", while paged files use the page buffer
  ([HDF5 2.2.0 release][hdf5-220], [ros3 issue #4700][ros3-4700]). The page
  buffer (`page_buf_size`) is "only allowed for HDF5 files created with
  fs_strategy='page'" ([h5py docs][h5py-file]). PyPI h5py wheels do not
  include ros3.
- **h5py + fsspec** hands libhdf5 a Python file object; each libhdf5 read
  becomes a `read()` on an fsspec file, whose cache decides the requests.
  fsspec's old default `readahead` cache requested 16x more data than
  `blockcache` when opening HDF5 files; earthaccess now defaults to
  `blockcache` with 4–16 MiB blocks by file size ([earthaccess][ea]); the
  Cloud-Optimized guide recommends `blockcache` plus h5py's `page_buf_size`
  and `rdcc_nbytes` ([guide][cog]). Our §2 comparison uses exactly this
  path, without fsspec's cache.
- **pyfive** (pure Python, like us) supports lazy loading "on both Posix and
  S3 filesystems" through fsspec, reads a variable's attributes and chunk
  B-tree when it is accessed, and can merge chunk range requests with
  fsspec's `merge_range_requests` ([pyfive][pyfive], [pyfive #257][pyfive-257]).
- **h5wasm** (libhdf5 compiled with Emscripten) backs a file with
  `FS.createLazyFile`, which "can use range requests to incrementally access
  the h5 file over the wire" ([h5wasm][h5wasm]); Emscripten's lazy files use
  synchronous XHR, which browsers only allow in Web Workers
  ([Emscripten FS API][emfs]). Files larger than memory remain an open issue
  ([h5wasm #40][h5wasm-40]).
- **jsfive** (pure JS port of pyfive) reads from an `ArrayBuffer` of the whole
  file ([jsfive][jsfive]); no lazy loading.
- **Rust `object_store`** (0.14): one `ObjectStore` trait over S3, GCS, Azure,
  HTTP/WebDAV, local files and memory; `get_ranges` "will automatically
  coalesce adjacent ranges into an appropriate number of parallel requests",
  with `OBJECT_STORE_COALESCE_DEFAULT` as the gap below which ranges merge
  ([docs.rs][os]). It builds for wasm32 except the local-filesystem and
  some chunked-upload parts.

## 4. Recommendation and migration plan

Adopt **(a)**, with a block cache as a required part of every non-local
backend, **(c)** as a cache policy, and the wasm path through the restartable
`NeedBytes` mode. Every milestone keeps `main` green: `cargo test
--workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap
fast path within benchmark noise.

**M0 — prerequisites (≈1 week).**
- Dense-group name lookup through the v2 B-tree name index (Jenkins hash,
  record type 5), instead of enumerating all links; same for creation-order
  and attribute name lookups. §2 shows 936 ranges → a handful. Regression
  test: count links decoded for one lookup in a 35 001-link group.
- One checked helper for address → index conversion; replace the 133
  `as usize` casts. On 32-bit an address past `usize::MAX` is a clean error
  (today it truncates or panics).
- `Group::dataset(name)`/`Group::group(name)` call `children()`, which
  re-reads the group's whole link list on every lookup, so walking a group of
  n children decodes its links O(n) times. Look names up through the index
  (above) and let a listing hand out its entries, so the cache has less to
  absorb.
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` — link and
  attribute names through the name indexes (`group_v2::resolve_child`,
  `attribute::find_attribute_in_file`; creation-order lookups by name do
  not exist in the API, so the creation-order index is still only listed),
  `addr::to_usize`/`saturating_usize` for all 119 truncating `u64 as usize`
  casts clippy finds in `clawhdf5-format` under any CI-built feature set but
  `szip` (the 133 above counted any `*addr*/*offset* as usize`, mostly
  widening `u8`/`u32` casts; `scripts/check-32bit-casts.sh` lints those
  feature sets for new ones), and `Group::entries`/`File::group_at`.
  The facade, io and ann casts are not converted.

**M1 — metadata over the trait, in-memory impl identical to today (2–3 weeks).**
- Add `Storage` (above) to `clawhdf5-format`, `no_std`-compatible, with
  `impl Storage for [u8]`. Add a storage error variant — `FormatError` and the
  facade `Error` are not `#[non_exhaustive]`, so this is a breaking change for
  exhaustive matches: bundle it with the next major version, or mark both
  enums `#[non_exhaustive]` first.
- Convert modules bottom-up — superblock, object header, local/global heap,
  B-tree v1/v2, fractal heap, fixed/extensible array, symbol table, group
  v1/v2, shared messages, attributes, fill value, data layout — one commit
  each. The old `&[u8]` signature stays as a thin wrapper over the new one
  (`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so
  callers and the other crates don't move yet.
- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.

**M2 — raw data over the trait (1–2 weeks).**
- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`,
  VL/global heap. Chunked reads first collect the chunk extents the selection
  needs, then call `read_ranges` once and decompress in parallel as today.
- Zero-copy stays: when `as_contiguous()` is `Some`, contiguous reads return
  borrowed slices; `read_raw_ref`/`read_*_zerocopy`/`File::as_bytes` keep
  their signatures and return a clear "not available for this storage" error
  on other backends (they already return `Option`/`Result`).
- Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
  keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data`. As planned,
  with these choices:
  - `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the
    file handle is shared by its datasets and may be sent across threads).
    The file's view (user block skipped, bounded by the recorded end of
    file, cache image laid over its reads) is itself a `Storage`, and the
    facade calls the generic cores with it; for a `Vec` or an mmap its
    `as_contiguous()` is the buffer, so the local paths are the slice code
    (checked: identical conformance results; the bench gate below still
    has to be run on an idle machine).
  - Chunked reads fetch in batches of at most 64 MiB of stored bytes, one
    `read_ranges` call each, so a remote read never holds more than that
    undecoded; chunks already in the chunk cache are not fetched.
  - The typed readers' zero-copy fast path became "read the contiguous
    bytes once": over a range storage a contiguous `read_f64` is one read,
    and a native contiguous selection reads only its runs.
  - External VDS source files stay whole-file, through a resolver that
    returns bytes (`File::set_vds_resolver`).
  - The v2 B-tree and dense groups were done here rather than in M3, so no
    format-crate path answers `ContiguousStorageRequired` any more; only the
    facade's zero-copy methods do.
  - Measured with the M2 harness (`crates/clawhdf5/tests/storage_equivalence.rs`,
    tank, 2026-09-26): one pass over the 621 corpus files that open — list,
    every attribute, every dataset once — is 176 092 `read_at` calls and
    208 MB through a storage with no cache (254 MB of files). The block
    cache of M3 is what turns that into requests (§2).

**M3 — HTTP/S3 backend (1–2 weeks).**
- `clawhdf5-io`, feature `remote` (off by default, so the default tree stays
  free of C and TLS stacks): `RangeStorage` over `object_store` (HTTP, S3, GCS,
  Azure), with a `BlockCache` (LRU, block size configurable, default 1 MiB per
  §2; the page size for paged files; the first block prefetched on open) and
  a request counter exposed for tests and users.
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the
  Python bindings, with these choices:
  - A new crate, `clawhdf5-remote`, instead of a `remote` feature of
    `clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
    sits below the facade.
  - HTTP(S) through `ureq` (`HttpStorage`), not object_store's HTTP store:
    that one pulls reqwest with aws-lc-rs (C), while plain HTTP through
    ureq builds no C, so it is the default feature; `https` adds rustls
    with ring. S3/GCS/Azure go through `object_store` (`ObjectStoreStorage`,
    features `s3`/`gcs`/`azure`, opt-in because of aws-lc-rs); the
    `object-store` feature alone (in-memory, local files, a store you
    build) is pure Rust. object_store is async: each read runs on a
    two-thread tokio runtime of the storage's own while the caller waits,
    so the caller's context (a plain thread, `spawn_blocking`, another
    runtime) does not matter.
  - `BlockCache` (any `Storage`): 1 MiB blocks and a 64 MiB LRU budget by
    default, the first block fetched at open (for HTTP by the request
    that learns the length), the missing blocks of one read fetched as
    runs of consecutive blocks in one parallel batch, per-block in-flight
    deduplication across threads, and reads that miss more than half the
    budget not kept. The page size of paged files is not used as the block
    size yet. `CacheStats` and `HttpStats` count requests and bytes.
  - The file is pinned at open by ETag (else Last-Modified, or the object's
    version) and length; a change is an error, not mixed data.
  - `h5rs` (feature `remote`) reads through the new `File::storage()`, the
    file's view as a `Storage`, so its parsing works on remote files; its
    `check` downloads the file whole.
  - Measured with `crates/clawhdf5-remote/tests/http.rs` (tank, 2026-09-26,
    `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus
    CLAWHDF5_REMOTE_REPORT=1 cargo test --release -p clawhdf5-remote --test
    http -- --nocapture corpus`), requests as the test server counted
    them: the 621 corpus files that open read over HTTP exactly as through
    `File::open`. Open + list (every group's entries, every dataset's shape
    and type) of all of them: 640 requests, 55.5 MB of 254 MB; then reading
    each file's largest dataset under 64 MiB: 96 more (171 MB in all). The
    same work without a cache: 141 936 requests. Per file: A lists in 2
    requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
    6.4 MB: 35 001 object headers spread over the file).

**M4 — wasm lazy loading (1–2 weeks).**
- `clawhdf5-wasm`: `openUrl(url) -> Promise<H5File>` backed by `fetch` with a
  `Range` header, on the main thread, via the restartable `NeedBytes` loop (no
  Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
  to a whole download when the server does not answer 206.
- `examples/wasm-viewer`: open by URL.

**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
blocks past the old end. Needs libhdf5 SWMR semantics research first.

Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured).

### Keeping the local fast path

- `impl Storage for [u8]` returns `Cow::Borrowed` — no copy, no allocation.
- Hot loops (raw-data copies, contiguous typed reads, `read_selection_native`)
  branch once on `as_contiguous()` and then run today's code.
- The parser cores are generic over `S: Storage + ?Sized`, so the in-memory
  instance has no dispatch at all; a remote backend behind `&dyn Storage`
  pays one indirect call per structure read. Parse code keeps working on
  the returned slice.
- Reads sized by untrusted fields cover what the parser uses (see
  `CHANGELOG.md`, M1), so a hostile size costs no read of the rest of the
  file.
- Gate: `crates/clawhdf5/benches/mmap_bench.rs`, the concurrent-read benches in
  `clawhdf5-bench`, and the conformance run time, before and after each M1/M2
  commit, on an otherwise idle machine. Anything outside noise blocks the
  commit.

### Risks

- **Silent regressions on local files** — mitigated by the bench gate above
  and by M1 being a pure refactor (every conformance hash identical).
- **Two reads per structure** (prefix, then body) could double requests on a
  cold cache. Block-aligned caching makes the second read a hit; the measured
  block counts already include this pattern.
- **API break** — new error variant (see M1); `as_bytes()`-style APIs become
  fallible for non-local storage. ClawBrainHub uses `File`, `FileBuilder`,
  `AttrValue`, `Selection` on local files only, so it is unaffected by
  behaviour, only by exhaustive matches on `Error`.
- **Restartable parsing** assumes operations are pure over the storage. The
  chunk cache and metadata cache must only be filled by completed reads.
- **Cache memory** — the block cache needs a byte budget and eviction, which
  (unlike option (b)) is sound because parsers hold `Cow`s, not borrows into
  the cache.

### Testing

- **In-memory equivalence:** a `CountingStorage` wrapper over `[u8]` that
  records every `read_at`; run the conformance probe through it and require
  identical results. It also produces the request counts of §2 without the
  trap-flag tracer.
- **Adversarial storage:** a wrapper that returns short reads, errors on the
  Nth request, or serves blocks of 1 byte, to prove every miss is an `Err`
  and never wrong data.
- **HTTP:** an in-process server on `127.0.0.1` (std `TcpListener`) that
  honours `Range`, answers 206/416, can refuse ranges (200), and counts
  requests. Tests assert both correctness against h5py and **request budgets**
  (e.g. listing file A with a 1 MiB block cache takes ≤ 3 requests) so a
  change that makes the reader chattier fails CI.
- **S3:** `object_store`'s in-memory store in unit tests; a MinIO or real
  bucket only in an opt-in job.
- **wasm:** the existing Node/Chromium harness in `examples/wasm-viewer/test`
  with a local range-capable server.
- **32-bit:** a `wasm32` or `i686` build that opens a sparse > 4 GiB file
  through the counting storage.

## Tools

- `docs/design/tools/inventory.py` — §1 tables (`--list`, `--patterns`).
- `docs/design/tools/range-trace/` — §2 tracer (standalone crate, x86-64
  Linux, not part of the workspace).
- `docs/design/tools/libhdf5_reads.py` — §2 libhdf5 comparison and chunk
  extents.

[cog]: https://guide.cloudnativegeo.org/cloud-optimized-netcdf4-hdf5/
[ea]: https://earthaccess.readthedocs.io/en/latest/user/explanation/fsspec/
[h5py-file]: https://docs.h5py.org/en/stable/high/file.html
[hdfg-cloud]: https://www.hdfgroup.org/2022/08/08/cloud-storage-options-for-hdf5/
[hdf5-220]: https://www.hdfgroup.org/2026/07/30/release-of-hdf5-2-2-0-and-two-august-events-newsletter-210/
[ros3-4700]: https://github.com/HDFGroup/hdf5/issues/4700
[pyfive]: https://pyfive.readthedocs.io/en/latest/quickstart/usage.html
[pyfive-257]: https://github.com/NCAS-CMS/pyfive/issues/257
[h5wasm]: https://github.com/usnistgov/h5wasm
[h5wasm-40]: https://github.com/usnistgov/h5wasm/issues/40
[emfs]: https://emscripten.org/docs/api_reference/Filesystem-API.html
[jsfive]: https://github.com/usnistgov/jsfive
[os]: https://docs.rs/object_store/latest/object_store/