From 1c1af460b6830851adf032ff064d56a621ac6798 Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 10:17:02 -0500 Subject: [PATCH] docs: design for range reads (S3/HTTP, wasm lazy loading, SWMR, 32-bit) Inventory of the 109 whole-file parser functions, a measurement of the metadata ranges clawhdf5 touches on three corpus files against libhdf5, options (storage trait, virtual slice, metadata prefetch, userfaultfd) with how ros3, h5py+fsspec, pyfive, h5wasm and object_store do it, and an incremental plan: M0 name-index lookups, M1 metadata over a Storage trait, M2 raw data with batched ranges, M3 object_store backend, M4 wasm fetch-driven lazy loading. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/range-reads.md | 499 +++++++++++++++++++++++++++++++++++++ 1 file changed, 499 insertions(+) create mode 100644 docs/design/range-reads.md diff --git a/docs/design/range-reads.md b/docs/design/range-reads.md new file mode 100644 index 0000000..2e82a6c --- /dev/null +++ b/docs/design/range-reads.md @@ -0,0 +1,499 @@ +# Design: range reads (reading HDF5 without holding the whole file) + +Status: proposal, 2026-09-26. No library code has changed; this document is +the plan for Phase 3's largest architectural change. Every count below was +taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given +next to it. No timing numbers appear here on purpose: the machine was shared +with other build jobs when this was written. + +## The problem + +Every reader in clawhdf5 parses the file through one `&[u8]` that covers the +whole file: + +- `clawhdf5_io::HDF5Read::as_bytes(&self) -> &[u8]` is the only read method of + the I/O trait (`crates/clawhdf5-io/src/lib.rs`). `FileReader` reads the + whole file into a `Vec`, `MmapReader` maps it. +- The facade's `File` keeps a `Backing` (`Vec` or mmap) and hands + `FileData::as_bytes()` — the file from the superblock to the recorded end of + file — to every `clawhdf5-format` call (`crates/clawhdf5/src/reader.rs`). +- The format crate's parsers take `file_data: &[u8]` plus a `usize` address and + slice into it. +- `LazyFile` (facade) parses metadata lazily, but still over + `HDF5Read::as_bytes`. `AsyncHDF5Read` (`clawhdf5-io`, `async` feature) + already has `read_at(offset, len)`, but `AsyncHDF5File::open` calls + `read_all()` and parses the result. +- The browser reader (`clawhdf5-wasm`) is `open(bytes: Vec)`: the page + must download the whole file before it can list a group. + +That rules out four things users of other HDF5 readers have: + +1. **Remote files by range request** — libhdf5's `ros3` driver, h5py over + `fsspec`/`s3fs`, pyfive over `fsspec`. Today a 30 GB file on S3 has to be + downloaded before its tree can be listed. +2. **Lazy loading in the browser** — h5wasm can back a file with HTTP range + requests; our wasm reader holds the whole file. +3. **SWMR readers of a growing file** — the slice is fixed at open; a reader + cannot see the file grow (and an mmap of a growing file is fragile). +4. **Files larger than the address space on 32-bit targets** — including + `wasm32`, where `usize` is 32 bits and a file above 4 GiB cannot even be put + in a slice; browsers cap `ArrayBuffer`s lower than that. + +## 1. Inventory + +### 1.1 Functions that take the whole file + +`python3 docs/design/tools/inventory.py` scans non-test source (code before a +file's `#[cfg(test)] mod`, no `tests/`) for functions with a +`file_data: &[u8]` parameter (the repository's convention), and — as a +heuristic — functions with a `data`/`buf`/`file`/`bytes` slice *and* an +integer address/offset parameter. `--list` prints every hit. + +- **103 functions take `file_data: &[u8]`** (73 of them `pub`), in 25 files. +- The heuristic adds 27, of which only **6 really take the whole file under + another name**: `ObjectHeader::parse` and its internal `parse_v1`, + `parse_v1_chunk`, `parse_v2`, `parse_v2_continuation` (`data: &[u8]`), and + `Superblock::parse`. The other 21 are helpers (`ensure_len`, + `read_length`, `read_uint`, …) that bound-check whatever slice they are + given, or writer-side `write_at`s in `clawhdf5-io`. + +So **109 functions** have to change signature. Grouped by what they read: + +| Read pattern | Modules (whole-file functions) | Count | +|---|---|---:| +| **Small metadata reads at addresses** — a structure whose size is usually only known after its prefix is read | `group_v2` 9, `shared_message` 9, `fractal_heap` 7, `attribute` 6, `object_header` 5, `group_v1` 5, `extensible_array` 4, `btree_v2` 4, `btree_v1` 3, `fixed_array` 3, `local_heap` 3, `fill_value` 3, `superblock` 2, `data_layout` 1, `symbol_table` 1 | 65 | +| **Bulk raw data** — a contiguous extent, or a list of chunk extents known once the chunk index has been walked | `chunked_read` 12, `data_read` 9, `vds` 6, `parallel_read` 3, `partial_read` 1, `provenance` 1 | 32 | +| **Heap-resident data** — VL strings and sequences: many small reads into global-heap collections | `vl_data` 3, `global_heap` 2, facade `vlen.rs` 3, facade `types.rs` 3 | 11 | +| Bindings | `clawhdf5-py/src/convert.rs` | 1 | + +### 1.2 How they read + +`python3 docs/design/tools/inventory.py --patterns` (non-test code): + +| Pattern | Where | Count | +|---|---|---:| +| `file_data` passed on to another function (call sites) | format 262, facade 7, py 2 | 271 | +| `file_data[..]` slicing | format | 137 | +| of which open-ended `file_data[x..]` (slice to end of file) | format | 5 | +| `file_data.len()` (end-of-file bound checks) | format | 38 | +| address/offset `as usize` casts | format 120, facade 5, io 5, ann 3 | 133 | +| `.as_bytes()` on a file or reader, feeding format calls | facade 26, py 8, tools 1, wasm 1 | 36 | + +Three things in this table shape the design: + +- **Open-ended slices and `len()` checks** assume the whole file is present. + With a range reader "the rest of the file" is a request for gigabytes; each + of these 43 sites must become a bounded read. +- **The 133 `as usize` casts** are where a 64-bit HDF5 address is squeezed + into a pointer-sized index. On 32-bit targets they truncate or must fail. + A storage trait that takes `u64` offsets removes most of them. +- **The bytes escape through the public API.** `File::as_bytes()`, + `LazyFile::as_bytes()`, `MmapFile::as_bytes()` return the whole file, and 15 + facade methods return `&'f [u8]`/`&'f [T]` borrowed from it + (`Dataset::read_raw_ref`, `read_as_slice`, `read_*_zerocopy`). The Python + bindings (`clawhdf5-py/src/node.rs`, `attrs.rs`, …) call + `clawhdf5_format` directly with `file.as_bytes()`, and so do `h5rs` and the + wasm reader. These must keep working for local files (they are the + zero-copy fast path) and fail cleanly for remote ones. + +### 1.3 Pieces that already exist + +- `clawhdf5_format::metadata_cache::MetadataCache` — LRU keyed by file offset + with a byte budget (2 MiB default). Not currently in the read path. +- `clawhdf5_io::prefetch::PrefetchReader` — ring buffer of `(offset, len)` + entries over an `HDF5Read`, plus a sweep detector for chunk access patterns. +- `clawhdf5_io::async_read::AsyncHDF5Read::read_at` — the right shape of + trait, but async and only used to read everything. +- `clawhdf5_io::hsds` — a client for the HSDS REST service. Different + protocol (the server parses HDF5), not a range reader. + +## 2. Measurement: how chatty is a naive range reader? + +`docs/design/tools/range-trace/` loads a file into a page-aligned buffer, +opens it with `clawhdf5::File::from_bytes` (today's code, unchanged), +`mprotect`s the buffer and single-steps every load that faults (SIGSEGV logs +the exact address and unprotects the page; the x86 trap flag re-protects it +after one instruction). So every load instruction that touches the file is +recorded, with no change to the library. A page loaded more than 4096 times in +a phase is left readable and counted as wholly read. Phases: **open** +(superblock), **list** (walk every group; shape and dtype of every dataset — +what a tree view or `h5ls -r -v` needs), **read** (one dataset, whole). With +the dataset's chunk extents (from h5py, `libhdf5_reads.py --extents`) the read +phase is split into the chunk index and the raw data. + +Columns: *uncached requests* — a new request every time the access stream +leaves the neighbourhood (±64 B) of the current run, i.e. a reader that turns +each parse into a read with no cache; *ranges* — distinct byte ranges after +merging accesses closer than 64 B; *N KiB blocks* — distinct aligned blocks, +i.e. the requests a reader with a block cache of that size would make. For +comparison, `libhdf5_reads.py` opens the same file in h5py through a Python +file object (the `fileobj` driver — exactly how h5py + fsspec read remote +files) and counts libhdf5's `read` calls, default h5py settings. + +```bash +CARGO_TARGET_DIR=$PWD/target cargo build --release \ + --manifest-path docs/design/tools/range-trace/Cargo.toml +PY=/home/osobh/projects/clawhdf5/.venv/bin/python +$PY docs/design/tools/libhdf5_reads.py FILE DSET --extents > ext.txt +target/release/range-trace FILE DSET ext.txt +$PY docs/design/tools/libhdf5_reads.py FILE DSET +``` + +Files (from `conformance/.cache/corpus`), 2026-09-26 on tank: + +- **A** `xarray-data/imerghh_730.hdf5` — NASA IMERG, 7.7 MB, superblock v0, + 21 objects; read `/Grid/precipitation` (25 deflated chunks). +- **B** `NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc` — netCDF-4, 48 MB, + superblock v2, 9 objects; read `/tas` (780 deflated chunks, 43.7 MB stored). +- **C** `hdf5/tools/test/testfiles/h5stat_newgrat.h5` — 6.4 MB, superblock + v3, 35 001 groups in one dense (fractal heap + v2 B-tree) root group; read + `/DATASET_NAME` (a scalar with no storage). +- **B′** B repacked with paged aggregation: + `h5repack -S PAGE -G 65536` (55.5 MB). + +clawhdf5 (range-trace): + +| File | Phase | Uncached requests | Ranges | Bytes in ranges | 4 KiB blocks | 64 KiB blocks | 1 MiB blocks | +|---|---|---:|---:|---:|---:|---:|---:| +| A | open + list | 410 | 5 | 25 285 | 8 | 3 | 2 | +| A | read: chunk index | 33 | 8 | 4 608 | 7 | 3 | 2 | +| A | read: raw data | (25 chunks) | 2 | 1 496 533 | 368 | 25 | 3 | +| A | all metadata | 443 | 6 | 26 549 | 10 | 3 | 2 | +| B | open + list | 143 | 5 | 17 951 | 7 | 1 | 1 | +| B | read: chunk index | 46 | 19 | 49 583 | 28 | 15 | 14 | +| B | read: raw data | (780 chunks) | 15 | 43 656 557 | 10 671 | 668 | 43 | +| B | all metadata | 189 | 20 | 57 023 | 30 | 15 | 14 | +| B′ | all metadata | 413 | 24 | 71 083 | 25 | 3 | 2 | +| B′ | read: raw data | (780 chunks) | 780 | 43 613 717 | 10 920 | 780 | 49 | +| C | open + list | 455 779 | 3 | 6 357 627 | 1 553 | 98 | 7 | +| C | read: resolve `/DATASET_NAME` | 38 525 | 936 | 1 137 116 | 948 | 98 | 7 | + +libhdf5 through h5py's `fileobj` driver (read calls, bytes): + +| File | open | list | read | total calls | total bytes | +|---|---:|---:|---:|---:|---:| +| A | 3 | 93 | 25 | 121 | 1 563 559 | +| B | 2 | 78 | 780 | 860 | 43 769 952 | +| C | 4 | 36 105 | 1 | 36 110 | 19 725 823 | + +What this says: + +1. **A reader with no cache is hopeless.** clawhdf5's parsers revisit the same + structures many times (A: 26 000 loads to list 21 objects: + `Group::dataset(name)` and `Group::group(name)` re-read the group's whole + link list for every lookup). Mapped + one-to-one onto requests that is 410 round trips to list 21 objects, and + 455 779 to list C. **A cache is not an optimisation, it is the design.** +2. **With a block cache, metadata is cheap on typical files.** Metadata of A + and B touches 5–20 ranges and 2–15 blocks of 1 MiB; a 1 MiB-block cache + lists A in 2 requests and B in 1, where libhdf5 issues 96 and 80 calls + (fsspec's block cache absorbs those for h5py). +3. **"Prefetch the metadata region" alone does not work.** In B the chunk + index (v1 B-tree nodes) is interleaved with the raw data: the metadata + touches 14 distinct 1 MiB blocks spread over 48 MB. Only 4 of the 611 + corpus files h5py can open use paged aggregation (all four are libhdf5 + test files; 502 have a v0 superblock). Repacked as B′, the same metadata + fits in 3 blocks of 64 KiB — paging helps a lot when present, but a reader + cannot count on it. +4. **Raw data is naturally a batch.** Once the chunk index is walked, all + chunk extents are known: B's 780 chunks are 15 byte ranges after merging + neighbours, so a coalescing `get_ranges` call reads the dataset in a + handful of parallel requests. libhdf5 issues one read per chunk (780). +5. **An existing inefficiency becomes a blocker.** Resolving `/DATASET_NAME` + in C reads 1.1 MB (936 ranges) because + `group_v2::resolve_path_following_links` enumerates every link of the + group (`resolve_group_entries`) and compares names, instead of hashing + the name and descending the v2 B-tree name index. On an mmap this is + merely slow; over HTTP it is 98 requests of 64 KiB for one lookup. + libhdf5 reads one 512-byte block. Listing C is inherently whole-file (35 001 + object headers spread over the file), and libhdf5 reads 19.7 MB — three + times the file — to do it. + +Caveats: loads are logged with their first byte and a nominal width of 8 B, +so byte totals are approximate (±64 B per range); "uncached requests" is a +model, not a measured reader. These are counts, not timings. + +## 3. Options + +### (a) A storage trait threaded through the format crate + +```rust +// clawhdf5-format (no_std + alloc) +pub trait Storage { + /// Bytes [offset, offset + len). Short only at end of file. + fn read_at(&self, offset: u64, len: usize) -> Result, FormatError>; + /// Current length; may grow between calls (SWMR). + fn len(&self) -> u64; + /// Batch read; backends coalesce and parallelise. Default: loop. + fn read_ranges(&self, ranges: &[Range]) -> Result>, FormatError> { .. } + /// The whole file as one slice, when the backend has it (Vec, mmap). + fn as_contiguous(&self) -> Option<&[u8]> { None } +} +impl Storage for [u8] { /* Cow::Borrowed, as_contiguous = Some(self) */ } +``` + +Every `file_data: &[u8]` becomes `file: &dyn Storage` (or `&S` generic), and +every `file_data[a..b]` becomes `file.read_at(a, b - a)?`. + +- **For:** explicit, sound, errors are values (a network error is an `Err`, + not a panic); works for any backend; `u64` offsets fix 32-bit; `len()` can + grow, which is what SWMR needs; `read_ranges` gives raw data its natural + batch shape; `impl Storage for [u8]` lets the migration go one module at a + time with `&[u8]` callers unchanged. +- **Against:** touches all 109 functions and ~400 slice/len/call sites + (§1.2); every parse that learns a structure's size from its prefix needs two + reads (prefix, then body) — cheap against a cache, but it must be written + that way. `Cow::Owned` results cost a copy per read for remote backends. +- **Sync vs async:** the format crate is `no_std` and synchronous and should + stay so (parsing is CPU work; an async parser would colour 109 functions and + their callers). Remote backends are async. Two bridges, both needed: + - native: a backend that blocks on its own runtime (`object_store` + + a private tokio runtime), used from ordinary threads; + - wasm (no blocking on the main thread): a **restartable** mode where the + cache returns `FormatError::NeedBytes { offset, len }` on a miss; an async + driver fetches the block and re-runs the (pure, idempotent) operation. + parquet-rs's async reader uses the same "fetch, then parse synchronously" + split. The measurement bounds the retries: one per missed block, i.e. 1–15 + for A and B's metadata at 1 MiB blocks. +- **`&dyn` vs generic:** generics monomorphise 109 functions per backend + (binary size matters for wasm); `&dyn Storage` costs one indirect call per + structure read, negligible next to parsing. Hot raw-data loops keep their + speed through `as_contiguous()`. + +### (b) A page-cache "virtual slice" + +A type that implements `Index, Output = [u8]>` and faults pages +in on demand from interior-mutable storage, so parsers keep their slicing +syntax. + +- `Index::index` returns `&[u8]` borrowed from `&self`. A cache that evicts + pages invalidates references that are still alive — unsound — so pages can + never be evicted while the object lives (unbounded memory, which is the + problem we are solving). +- A range that crosses a page boundary has to be copied into a contiguous + buffer that also must outlive the borrow: an append-only arena that only + grows. +- `Index` cannot fail: a network error or truncated response becomes a + panic, which breaks "never return wrong data; unsupported is a clean + error". +- Open-ended slices (`file_data[x..]`) and `file_data.len()` still mean "the + whole rest of the file". +- It hides the cost model: a `for` loop over `file_data[i]` becomes thousands + of cache lookups, and nothing in the signature says the call can block. + +Rejected. It saves the signature churn of (a) but moves every failure into +panics and unsoundness. + +### (c) Keep `&[u8]` for metadata by prefetching it; range-read raw data only + +Fetch "the metadata" up front into a buffer, parse it with today's code, and +range-read only chunk data. + +- HDF5 has no pointer to a metadata region. libhdf5's paged aggregation + (`H5Pset_file_space_strategy(H5F_FSPACE_STRATEGY_PAGE)`, + `h5repack -S PAGE -G `) keeps metadata and raw data in separate pages, + and "setting an appropriate page size can have all internal file metadata + in just one page" ([Cloud-Optimized HDF/NetCDF guide][cog]) — but only + when the writer chose it: 4 of 611 corpus files (§2). +- Without paging, metadata is scattered: in B the chunk index is spread over + 14 MiB-blocks of a 48 MB file. Prefetching the first N MiB (what earthaccess' + block cache effectively does for "shared metadata at the file beginning" + ([earthaccess][ea])) is a good *heuristic*, not a correctness basis. +- A sparse buffer still presents as a `&[u8]` of the whole file's length, so + every metadata read outside the prefetched part must be caught — back to (b). + +Useful as a **policy** on top of (a) (prefetch the first block, use the page +size of paged files as the block size), not as the architecture. + +### (d) mmap a userfaultfd- or FUSE-backed file + +Keep `&[u8]` everywhere and let the kernel fault in remote pages +(userfaultfd handler, or a FUSE file system doing range GETs). + +- Linux-only (userfaultfd) or needs a FUSE mount and privileges; nothing on + macOS/Windows without a kernel extension; impossible in wasm. +- Faults are synchronous and uninterruptible from the parser's view: a network + error becomes `SIGBUS`, a stalled request hangs the thread. +- No batching: raw data arrives one fault (4 KiB, or the FUSE read size) at a + time unless the handler guesses read-ahead. +- 32-bit address space is still exhausted by mapping a large file. + +Rejected. It is how one would retrofit a C library that cannot change; we can. + +### How other readers do it + +- **libhdf5 `ros3`** replaces each POSIX read with an S3 range GET + ([h5py docs][h5py-file], [HDF Group, cloud storage options][hdfg-cloud]). + libhdf5 has a real metadata cache above the driver, and HDF5 2.2.0 added an + I/O block cache to ros3 "to reduce the number of requests to S3 for files + not using paged allocation", while paged files use the page buffer + ([HDF5 2.2.0 release][hdf5-220], [ros3 issue #4700][ros3-4700]). The page + buffer (`page_buf_size`) is "only allowed for HDF5 files created with + fs_strategy='page'" ([h5py docs][h5py-file]). PyPI h5py wheels do not + include ros3. +- **h5py + fsspec** hands libhdf5 a Python file object; each libhdf5 read + becomes a `read()` on an fsspec file, whose cache decides the requests. + fsspec's old default `readahead` cache requested 16x more data than + `blockcache` when opening HDF5 files; earthaccess now defaults to + `blockcache` with 4–16 MiB blocks by file size ([earthaccess][ea]); the + Cloud-Optimized guide recommends `blockcache` plus h5py's `page_buf_size` + and `rdcc_nbytes` ([guide][cog]). Our §2 comparison uses exactly this + path, without fsspec's cache. +- **pyfive** (pure Python, like us) supports lazy loading "on both Posix and + S3 filesystems" through fsspec, reads a variable's attributes and chunk + B-tree when it is accessed, and can merge chunk range requests with + fsspec's `merge_range_requests` ([pyfive][pyfive], [pyfive #257][pyfive-257]). +- **h5wasm** (libhdf5 compiled with Emscripten) backs a file with + `FS.createLazyFile`, which "can use range requests to incrementally access + the h5 file over the wire" ([h5wasm][h5wasm]); Emscripten's lazy files use + synchronous XHR, which browsers only allow in Web Workers + ([Emscripten FS API][emfs]). Files larger than memory remain an open issue + ([h5wasm #40][h5wasm-40]). +- **jsfive** (pure JS port of pyfive) reads from an `ArrayBuffer` of the whole + file ([jsfive][jsfive]); no lazy loading. +- **Rust `object_store`** (0.14): one `ObjectStore` trait over S3, GCS, Azure, + HTTP/WebDAV, local files and memory; `get_ranges` "will automatically + coalesce adjacent ranges into an appropriate number of parallel requests", + with `OBJECT_STORE_COALESCE_DEFAULT` as the gap below which ranges merge + ([docs.rs][os]). It builds for wasm32 except the local-filesystem and + some chunked-upload parts. + +## 4. Recommendation and migration plan + +Adopt **(a)**, with a block cache as a required part of every non-local +backend, **(c)** as a cache policy, and the wasm path through the restartable +`NeedBytes` mode. Every milestone keeps `main` green: `cargo test +--workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap +fast path within benchmark noise. + +**M0 — prerequisites (≈1 week).** +- Dense-group name lookup through the v2 B-tree name index (Jenkins hash, + record type 5), instead of enumerating all links; same for creation-order + and attribute name lookups. §2 shows 936 ranges → a handful. Regression + test: count links decoded for one lookup in a 35 001-link group. +- One checked helper for address → index conversion; replace the 133 + `as usize` casts. On 32-bit an address past `usize::MAX` is a clean error + (today it truncates or panics). +- `Group::dataset(name)`/`Group::group(name)` call `children()`, which + re-reads the group's whole link list on every lookup, so walking a group of + n children decodes its links O(n) times. Look names up through the index + (above) and let a listing hand out its entries, so the cache has less to + absorb. + +**M1 — metadata over the trait, in-memory impl identical to today (2–3 weeks).** +- Add `Storage` (above) to `clawhdf5-format`, `no_std`-compatible, with + `impl Storage for [u8]`. Add a storage error variant — `FormatError` and the + facade `Error` are not `#[non_exhaustive]`, so this is a breaking change for + exhaustive matches: bundle it with the next major version, or mark both + enums `#[non_exhaustive]` first. +- Convert modules bottom-up — superblock, object header, local/global heap, + B-tree v1/v2, fractal heap, fixed/extensible array, symbol table, group + v1/v2, shared messages, attributes, fill value, data layout — one commit + each. The old `&[u8]` signature stays as a thin wrapper over the new one + (`fn parse(data: &[u8], ..) { parse_in(data as &dyn Storage, ..) }`), so + callers and the other crates don't move yet. +- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads. + +**M2 — raw data over the trait (1–2 weeks).** +- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`, + VL/global heap. Chunked reads first collect the chunk extents the selection + needs, then call `read_ranges` once and decompress in parallel as today. +- Zero-copy stays: when `as_contiguous()` is `Some`, contiguous reads return + borrowed slices; `read_raw_ref`/`read_*_zerocopy`/`File::as_bytes` keep + their signatures and return a clear "not available for this storage" error + on other backends (they already return `Option`/`Result`). +- Facade: `File::open_storage(Box)`; `File::open` + keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`. + +**M3 — HTTP/S3 backend (1–2 weeks).** +- `clawhdf5-io`, feature `remote` (off by default, so the default tree stays + free of C and TLS stacks): `RangeStorage` over `object_store` (HTTP, S3, GCS, + Azure), with a `BlockCache` (LRU, block size configurable, default 1 MiB per + §2; the page size for paged files; the first block prefetched on open) and + a request counter exposed for tests and users. +- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it. + +**M4 — wasm lazy loading (1–2 weeks).** +- `clawhdf5-wasm`: `openUrl(url) -> Promise` backed by `fetch` with a + `Range` header, on the main thread, via the restartable `NeedBytes` loop (no + Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back + to a whole download when the server does not answer 206. +- `examples/wasm-viewer`: open by URL. + +**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow; +add `File::refresh()` that re-reads the superblock/EOF and invalidates cached +blocks past the old end. Needs libhdf5 SWMR semantics research first. + +Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured). + +### Keeping the local fast path + +- `impl Storage for [u8]` returns `Cow::Borrowed` — no copy, no allocation. +- Hot loops (raw-data copies, contiguous typed reads, `read_selection_native`) + branch once on `as_contiguous()` and then run today's code. +- `&dyn` dispatch is per structure, not per byte; parse code keeps working on + the returned slice. +- Gate: `crates/clawhdf5/benches/mmap_bench.rs`, the concurrent-read benches in + `clawhdf5-bench`, and the conformance run time, before and after each M1/M2 + commit, on an otherwise idle machine. Anything outside noise blocks the + commit. + +### Risks + +- **Silent regressions on local files** — mitigated by the bench gate above + and by M1 being a pure refactor (every conformance hash identical). +- **Two reads per structure** (prefix, then body) could double requests on a + cold cache. Block-aligned caching makes the second read a hit; the measured + block counts already include this pattern. +- **API break** — new error variant (see M1); `as_bytes()`-style APIs become + fallible for non-local storage. ClawBrainHub uses `File`, `FileBuilder`, + `AttrValue`, `Selection` on local files only, so it is unaffected by + behaviour, only by exhaustive matches on `Error`. +- **Restartable parsing** assumes operations are pure over the storage. The + chunk cache and metadata cache must only be filled by completed reads. +- **Cache memory** — the block cache needs a byte budget and eviction, which + (unlike option (b)) is sound because parsers hold `Cow`s, not borrows into + the cache. + +### Testing + +- **In-memory equivalence:** a `CountingStorage` wrapper over `[u8]` that + records every `read_at`; run the conformance probe through it and require + identical results. It also produces the request counts of §2 without the + trap-flag tracer. +- **Adversarial storage:** a wrapper that returns short reads, errors on the + Nth request, or serves blocks of 1 byte, to prove every miss is an `Err` + and never wrong data. +- **HTTP:** an in-process server on `127.0.0.1` (std `TcpListener`) that + honours `Range`, answers 206/416, can refuse ranges (200), and counts + requests. Tests assert both correctness against h5py and **request budgets** + (e.g. listing file A with a 1 MiB block cache takes ≤ 3 requests) so a + change that makes the reader chattier fails CI. +- **S3:** `object_store`'s in-memory store in unit tests; a MinIO or real + bucket only in an opt-in job. +- **wasm:** the existing Node/Chromium harness in `examples/wasm-viewer/test` + with a local range-capable server. +- **32-bit:** a `wasm32` or `i686` build that opens a sparse > 4 GiB file + through the counting storage. + +## Tools + +- `docs/design/tools/inventory.py` — §1 tables (`--list`, `--patterns`). +- `docs/design/tools/range-trace/` — §2 tracer (standalone crate, x86-64 + Linux, not part of the workspace). +- `docs/design/tools/libhdf5_reads.py` — §2 libhdf5 comparison and chunk + extents. + +[cog]: https://guide.cloudnativegeo.org/cloud-optimized-netcdf4-hdf5/ +[ea]: https://earthaccess.readthedocs.io/en/latest/user/explanation/fsspec/ +[h5py-file]: https://docs.h5py.org/en/stable/high/file.html +[hdfg-cloud]: https://www.hdfgroup.org/2022/08/08/cloud-storage-options-for-hdf5/ +[hdf5-220]: https://www.hdfgroup.org/2026/07/30/release-of-hdf5-2-2-0-and-two-august-events-newsletter-210/ +[ros3-4700]: https://github.com/HDFGroup/hdf5/issues/4700 +[pyfive]: https://pyfive.readthedocs.io/en/latest/quickstart/usage.html +[pyfive-257]: https://github.com/NCAS-CMS/pyfive/issues/257 +[h5wasm]: https://github.com/usnistgov/h5wasm +[h5wasm-40]: https://github.com/usnistgov/h5wasm/issues/40 +[emfs]: https://emscripten.org/docs/api_reference/Filesystem-API.html +[jsfive]: https://github.com/usnistgov/jsfive +[os]: https://docs.rs/object_store/latest/object_store/