clawhdf5/docs/design/range-reads.md

Design: range reads (reading HDF5 without holding the whole file)

Status: proposal, 2026-09-26. No library code has changed; this document is the plan for Phase 3's largest architectural change. Every count below was taken on tank on 2026-09-26 at commit de2a53f, with the commands given next to it. No timing numbers appear here on purpose: the machine was shared with other build jobs when this was written.

The problem

Every reader in clawhdf5 parses the file through one &[u8] that covers the whole file:

• clawhdf5_io::HDF5Read::as_bytes(&self) -> &[u8] is the only read method of the I/O trait (crates/clawhdf5-io/src/lib.rs). FileReader reads the whole file into a Vec, MmapReader maps it.
• The facade's File keeps a Backing (Vec<u8> or mmap) and hands FileData::as_bytes() — the file from the superblock to the recorded end of file — to every clawhdf5-format call (crates/clawhdf5/src/reader.rs).
• The format crate's parsers take file_data: &[u8] plus a usize address and slice into it.
• LazyFile (facade) parses metadata lazily, but still over HDF5Read::as_bytes. AsyncHDF5Read (clawhdf5-io, async feature) already has read_at(offset, len), but AsyncHDF5File::open calls read_all() and parses the result.
• The browser reader (clawhdf5-wasm) is open(bytes: Vec<u8>): the page must download the whole file before it can list a group.

That rules out four things users of other HDF5 readers have:

1. Remote files by range request — libhdf5's ros3 driver, h5py over fsspec/s3fs, pyfive over fsspec. Today a 30 GB file on S3 has to be downloaded before its tree can be listed.
2. Lazy loading in the browser — h5wasm can back a file with HTTP range requests; our wasm reader holds the whole file.
3. SWMR readers of a growing file — the slice is fixed at open; a reader cannot see the file grow (and an mmap of a growing file is fragile).
4. Files larger than the address space on 32-bit targets — including wasm32, where usize is 32 bits and a file above 4 GiB cannot even be put in a slice; browsers cap ArrayBuffers lower than that.

1. Inventory

1.1 Functions that take the whole file

python3 docs/design/tools/inventory.py scans non-test source (code before a file's #[cfg(test)] mod, no tests/) for functions with a file_data: &[u8] parameter (the repository's convention), and — as a heuristic — functions with a data/buf/file/bytes slice and an integer address/offset parameter. --list prints every hit.

• 103 functions take file_data: &[u8] (73 of them pub), in 25 files.
• The heuristic adds 27, of which only 6 really take the whole file under another name: ObjectHeader::parse and its internal parse_v1, parse_v1_chunk, parse_v2, parse_v2_continuation (data: &[u8]), and Superblock::parse. The other 21 are helpers (ensure_len, read_length, read_uint, …) that bound-check whatever slice they are given, or writer-side write_ats in clawhdf5-io.

So 109 functions have to change signature. Grouped by what they read:

Read pattern	Modules (whole-file functions)	Count
Small metadata reads at addresses — a structure whose size is usually only known after its prefix is read	group_v2 9, shared_message 9, fractal_heap 7, attribute 6, object_header 5, group_v1 5, extensible_array 4, btree_v2 4, btree_v1 3, fixed_array 3, local_heap 3, fill_value 3, superblock 2, data_layout 1, symbol_table 1	65
Bulk raw data — a contiguous extent, or a list of chunk extents known once the chunk index has been walked	chunked_read 12, data_read 9, vds 6, parallel_read 3, partial_read 1, provenance 1	32
Heap-resident data — VL strings and sequences: many small reads into global-heap collections	vl_data 3, global_heap 2, facade vlen.rs 3, facade types.rs 3	11
Bindings	clawhdf5-py/src/convert.rs	1

1.2 How they read

python3 docs/design/tools/inventory.py --patterns (non-test code):

Pattern	Where	Count
file_data passed on to another function (call sites)	format 262, facade 7, py 2	271
file_data[..] slicing	format	137
of which open-ended file_data[x..] (slice to end of file)	format	5
file_data.len() (end-of-file bound checks)	format	38
address/offset as usize casts	format 120, facade 5, io 5, ann 3	133
.as_bytes() on a file or reader, feeding format calls	facade 26, py 8, tools 1, wasm 1	36

Three things in this table shape the design:

• Open-ended slices and len() checks assume the whole file is present. With a range reader "the rest of the file" is a request for gigabytes; each of these 43 sites must become a bounded read.
• The 133 as usize casts are where a 64-bit HDF5 address is squeezed into a pointer-sized index. On 32-bit targets they truncate or must fail. A storage trait that takes u64 offsets removes most of them.
• The bytes escape through the public API. File::as_bytes(), LazyFile::as_bytes(), MmapFile::as_bytes() return the whole file, and 15 facade methods return &'f [u8]/&'f [T] borrowed from it (Dataset::read_raw_ref, read_as_slice, read_*_zerocopy). The Python bindings (clawhdf5-py/src/node.rs, attrs.rs, …) call clawhdf5_format directly with file.as_bytes(), and so do h5rs and the wasm reader. These must keep working for local files (they are the zero-copy fast path) and fail cleanly for remote ones.

1.3 Pieces that already exist

• clawhdf5_format::metadata_cache::MetadataCache — LRU keyed by file offset with a byte budget (2 MiB default). Not currently in the read path.
• clawhdf5_io::prefetch::PrefetchReader — ring buffer of (offset, len) entries over an HDF5Read, plus a sweep detector for chunk access patterns.
• clawhdf5_io::async_read::AsyncHDF5Read::read_at — the right shape of trait, but async and only used to read everything.
• clawhdf5_io::hsds — a client for the HSDS REST service. Different protocol (the server parses HDF5), not a range reader.

2. Measurement: how chatty is a naive range reader?

docs/design/tools/range-trace/ loads a file into a page-aligned buffer, opens it with clawhdf5::File::from_bytes (today's code, unchanged), mprotects the buffer and single-steps every load that faults (SIGSEGV logs the exact address and unprotects the page; the x86 trap flag re-protects it after one instruction). So every load instruction that touches the file is recorded, with no change to the library. A page loaded more than 4096 times in a phase is left readable and counted as wholly read. Phases: open (superblock), list (walk every group; shape and dtype of every dataset — what a tree view or h5ls -r -v needs), read (one dataset, whole). With the dataset's chunk extents (from h5py, libhdf5_reads.py --extents) the read phase is split into the chunk index and the raw data.

Columns: uncached requests — a new request every time the access stream leaves the neighbourhood (±64 B) of the current run, i.e. a reader that turns each parse into a read with no cache; ranges — distinct byte ranges after merging accesses closer than 64 B; N KiB blocks — distinct aligned blocks, i.e. the requests a reader with a block cache of that size would make. For comparison, libhdf5_reads.py opens the same file in h5py through a Python file object (the fileobj driver — exactly how h5py + fsspec read remote files) and counts libhdf5's read calls, default h5py settings.

CARGO_TARGET_DIR=$PWD/target cargo build --release \
  --manifest-path docs/design/tools/range-trace/Cargo.toml
PY=/home/osobh/projects/clawhdf5/.venv/bin/python
$PY docs/design/tools/libhdf5_reads.py FILE DSET --extents > ext.txt
target/release/range-trace FILE DSET ext.txt
$PY docs/design/tools/libhdf5_reads.py FILE DSET

Files (from conformance/.cache/corpus), 2026-09-26 on tank:

• A xarray-data/imerghh_730.hdf5 — NASA IMERG, 7.7 MB, superblock v0, 21 objects; read /Grid/precipitation (25 deflated chunks).
• B NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc — netCDF-4, 48 MB, superblock v2, 9 objects; read /tas (780 deflated chunks, 43.7 MB stored).
• C hdf5/tools/test/testfiles/h5stat_newgrat.h5 — 6.4 MB, superblock v3, 35 001 groups in one dense (fractal heap + v2 B-tree) root group; read /DATASET_NAME (a scalar with no storage).
• B′ B repacked with paged aggregation: h5repack -S PAGE -G 65536 (55.5 MB).

clawhdf5 (range-trace):

File	Phase	Uncached requests	Ranges	Bytes in ranges	4 KiB blocks	64 KiB blocks	1 MiB blocks
A	open + list	410	5	25 285	8	3	2
A	read: chunk index	33	8	4 608	7	3	2
A	read: raw data	(25 chunks)	2	1 496 533	368	25	3
A	all metadata	443	6	26 549	10	3	2
B	open + list	143	5	17 951	7	1	1
B	read: chunk index	46	19	49 583	28	15	14
B	read: raw data	(780 chunks)	15	43 656 557	10 671	668	43
B	all metadata	189	20	57 023	30	15	14
B′	all metadata	413	24	71 083	25	3	2
B′	read: raw data	(780 chunks)	780	43 613 717	10 920	780	49
C	open + list	455 779	3	6 357 627	1 553	98	7
C	read: resolve /DATASET_NAME	38 525	936	1 137 116	948	98	7

libhdf5 through h5py's fileobj driver (read calls, bytes):

File	open	list	read	total calls	total bytes
A	3	93	25	121	1 563 559
B	2	78	780	860	43 769 952
C	4	36 105	1	36 110	19 725 823

What this says:

1. A reader with no cache is hopeless. clawhdf5's parsers revisit the same structures many times (A: 26 000 loads to list 21 objects: Group::dataset(name) and Group::group(name) re-read the group's whole link list for every lookup). Mapped one-to-one onto requests that is 410 round trips to list 21 objects, and 455 779 to list C. A cache is not an optimisation, it is the design.
2. With a block cache, metadata is cheap on typical files. Metadata of A and B touches 5–20 ranges and 2–15 blocks of 1 MiB; a 1 MiB-block cache lists A in 2 requests and B in 1, where libhdf5 issues 96 and 80 calls (fsspec's block cache absorbs those for h5py).
3. "Prefetch the metadata region" alone does not work. In B the chunk index (v1 B-tree nodes) is interleaved with the raw data: the metadata touches 14 distinct 1 MiB blocks spread over 48 MB. Only 4 of the 611 corpus files h5py can open use paged aggregation (all four are libhdf5 test files; 502 have a v0 superblock). Repacked as B′, the same metadata fits in 3 blocks of 64 KiB — paging helps a lot when present, but a reader cannot count on it.
4. Raw data is naturally a batch. Once the chunk index is walked, all chunk extents are known: B's 780 chunks are 15 byte ranges after merging neighbours, so a coalescing get_ranges call reads the dataset in a handful of parallel requests. libhdf5 issues one read per chunk (780).
5. An existing inefficiency becomes a blocker. Resolving /DATASET_NAME in C reads 1.1 MB (936 ranges) because group_v2::resolve_path_following_links enumerates every link of the group (resolve_group_entries) and compares names, instead of hashing the name and descending the v2 B-tree name index. On an mmap this is merely slow; over HTTP it is 98 requests of 64 KiB for one lookup. libhdf5 reads one 512-byte block. Listing C is inherently whole-file (35 001 object headers spread over the file), and libhdf5 reads 19.7 MB — three times the file — to do it.

Caveats: loads are logged with their first byte and a nominal width of 8 B, so byte totals are approximate (±64 B per range); "uncached requests" is a model, not a measured reader. These are counts, not timings.

3. Options

(a) A storage trait threaded through the format crate

// clawhdf5-format (no_std + alloc)
pub trait Storage {
    /// Bytes [offset, offset + len). Short only at end of file.
    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError>;
    /// Current length; may grow between calls (SWMR).
    fn len(&self) -> u64;
    /// Batch read; backends coalesce and parallelise. Default: loop.
    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> { .. }
    /// The whole file as one slice, when the backend has it (Vec, mmap).
    fn as_contiguous(&self) -> Option<&[u8]> { None }
}
impl Storage for [u8] { /* Cow::Borrowed, as_contiguous = Some(self) */ }

Every file_data: &[u8] becomes file: &dyn Storage (or &S generic), and every file_data[a..b] becomes file.read_at(a, b - a)?.

• For: explicit, sound, errors are values (a network error is an Err, not a panic); works for any backend; u64 offsets fix 32-bit; len() can grow, which is what SWMR needs; read_ranges gives raw data its natural batch shape; impl Storage for [u8] lets the migration go one module at a time with &[u8] callers unchanged.
• Against: touches all 109 functions and ~400 slice/len/call sites (§1.2); every parse that learns a structure's size from its prefix needs two reads (prefix, then body) — cheap against a cache, but it must be written that way. Cow::Owned results cost a copy per read for remote backends.
• Sync vs async: the format crate is no_std and synchronous and should stay so (parsing is CPU work; an async parser would colour 109 functions and their callers). Remote backends are async. Two bridges, both needed:
  • native: a backend that blocks on its own runtime (object_store + a private tokio runtime), used from ordinary threads;
  • wasm (no blocking on the main thread): a restartable mode where the cache returns FormatError::NeedBytes { offset, len } on a miss; an async driver fetches the block and re-runs the (pure, idempotent) operation. parquet-rs's async reader uses the same "fetch, then parse synchronously" split. The measurement bounds the retries: one per missed block, i.e. 1–15 for A and B's metadata at 1 MiB blocks.
• &dyn vs generic: generics monomorphise 109 functions per backend (binary size matters for wasm); &dyn Storage costs one indirect call per structure read, negligible next to parsing. Hot raw-data loops keep their speed through as_contiguous().

(b) A page-cache "virtual slice"

A type that implements Index<Range<usize>, Output = [u8]> and faults pages in on demand from interior-mutable storage, so parsers keep their slicing syntax.

• Index::index returns &[u8] borrowed from &self. A cache that evicts pages invalidates references that are still alive — unsound — so pages can never be evicted while the object lives (unbounded memory, which is the problem we are solving).
• A range that crosses a page boundary has to be copied into a contiguous buffer that also must outlive the borrow: an append-only arena that only grows.
• Index cannot fail: a network error or truncated response becomes a panic, which breaks "never return wrong data; unsupported is a clean error".
• Open-ended slices (file_data[x..]) and file_data.len() still mean "the whole rest of the file".
• It hides the cost model: a for loop over file_data[i] becomes thousands of cache lookups, and nothing in the signature says the call can block.

Rejected. It saves the signature churn of (a) but moves every failure into panics and unsoundness.

(c) Keep &[u8] for metadata by prefetching it; range-read raw data only

Fetch "the metadata" up front into a buffer, parse it with today's code, and range-read only chunk data.

• HDF5 has no pointer to a metadata region. libhdf5's paged aggregation (H5Pset_file_space_strategy(H5F_FSPACE_STRATEGY_PAGE), h5repack -S PAGE -G <size>) keeps metadata and raw data in separate pages, and "setting an appropriate page size can have all internal file metadata in just one page" (Cloud-Optimized HDF/NetCDF guide) — but only when the writer chose it: 4 of 611 corpus files (§2).
• Without paging, metadata is scattered: in B the chunk index is spread over 14 MiB-blocks of a 48 MB file. Prefetching the first N MiB (what earthaccess' block cache effectively does for "shared metadata at the file beginning" (earthaccess)) is a good heuristic, not a correctness basis.
• A sparse buffer still presents as a &[u8] of the whole file's length, so every metadata read outside the prefetched part must be caught — back to (b).

Useful as a policy on top of (a) (prefetch the first block, use the page size of paged files as the block size), not as the architecture.

(d) mmap a userfaultfd- or FUSE-backed file

Keep &[u8] everywhere and let the kernel fault in remote pages (userfaultfd handler, or a FUSE file system doing range GETs).

• Linux-only (userfaultfd) or needs a FUSE mount and privileges; nothing on macOS/Windows without a kernel extension; impossible in wasm.
• Faults are synchronous and uninterruptible from the parser's view: a network error becomes SIGBUS, a stalled request hangs the thread.
• No batching: raw data arrives one fault (4 KiB, or the FUSE read size) at a time unless the handler guesses read-ahead.
• 32-bit address space is still exhausted by mapping a large file.

Rejected. It is how one would retrofit a C library that cannot change; we can.

How other readers do it

• libhdf5 ros3 replaces each POSIX read with an S3 range GET (h5py docs, HDF Group, cloud storage options). libhdf5 has a real metadata cache above the driver, and HDF5 2.2.0 added an I/O block cache to ros3 "to reduce the number of requests to S3 for files not using paged allocation", while paged files use the page buffer (HDF5 2.2.0 release, ros3 issue #4700). The page buffer (page_buf_size) is "only allowed for HDF5 files created with fs_strategy='page'" (h5py docs). PyPI h5py wheels do not include ros3.
• h5py + fsspec hands libhdf5 a Python file object; each libhdf5 read becomes a read() on an fsspec file, whose cache decides the requests. fsspec's old default readahead cache requested 16x more data than blockcache when opening HDF5 files; earthaccess now defaults to blockcache with 4–16 MiB blocks by file size (earthaccess); the Cloud-Optimized guide recommends blockcache plus h5py's page_buf_size and rdcc_nbytes (guide). Our §2 comparison uses exactly this path, without fsspec's cache.
• pyfive (pure Python, like us) supports lazy loading "on both Posix and S3 filesystems" through fsspec, reads a variable's attributes and chunk B-tree when it is accessed, and can merge chunk range requests with fsspec's merge_range_requests (pyfive, pyfive #257).
• h5wasm (libhdf5 compiled with Emscripten) backs a file with FS.createLazyFile, which "can use range requests to incrementally access the h5 file over the wire" (h5wasm); Emscripten's lazy files use synchronous XHR, which browsers only allow in Web Workers (Emscripten FS API). Files larger than memory remain an open issue (h5wasm #40).
• jsfive (pure JS port of pyfive) reads from an ArrayBuffer of the whole file (jsfive); no lazy loading.
• Rust object_store (0.14): one ObjectStore trait over S3, GCS, Azure, HTTP/WebDAV, local files and memory; get_ranges "will automatically coalesce adjacent ranges into an appropriate number of parallel requests", with OBJECT_STORE_COALESCE_DEFAULT as the gap below which ranges merge (docs.rs). It builds for wasm32 except the local-filesystem and some chunked-upload parts.

4. Recommendation and migration plan

Adopt (a), with a block cache as a required part of every non-local backend, (c) as a cache policy, and the wasm path through the restartable NeedBytes mode. Every milestone keeps main green: cargo test --workspace, clippy, the conformance gate at 575/697 unchanged, and the mmap fast path within benchmark noise.

M0 — prerequisites (≈1 week).

• Dense-group name lookup through the v2 B-tree name index (Jenkins hash, record type 5), instead of enumerating all links; same for creation-order and attribute name lookups. §2 shows 936 ranges → a handful. Regression test: count links decoded for one lookup in a 35 001-link group.
• One checked helper for address → index conversion; replace the 133 as usize casts. On 32-bit an address past usize::MAX is a clean error (today it truncates or panics).
• Group::dataset(name)/Group::group(name) call children(), which re-reads the group's whole link list on every lookup, so walking a group of n children decodes its links O(n) times. Look names up through the index (above) and let a listing hand out its entries, so the cache has less to absorb.

M1 — metadata over the trait, in-memory impl identical to today (2–3 weeks).

• Add Storage (above) to clawhdf5-format, no_std-compatible, with impl Storage for [u8]. Add a storage error variant — FormatError and the facade Error are not #[non_exhaustive], so this is a breaking change for exhaustive matches: bundle it with the next major version, or mark both enums #[non_exhaustive] first.
• Convert modules bottom-up — superblock, object header, local/global heap, B-tree v1/v2, fractal heap, fixed/extensible array, symbol table, group v1/v2, shared messages, attributes, fill value, data layout — one commit each. The old &[u8] signature stays as a thin wrapper over the new one (fn parse(data: &[u8], ..) { parse_in(data as &dyn Storage, ..) }), so callers and the other crates don't move yet.
• Replace the 5 open-ended slices and 38 len() checks with bounded reads.

M2 — raw data over the trait (1–2 weeks).

• data_read, chunked_read, parallel_read, partial_read, vds, VL/global heap. Chunked reads first collect the chunk extents the selection needs, then call read_ranges once and decompress in parallel as today.
• Zero-copy stays: when as_contiguous() is Some, contiguous reads return borrowed slices; read_raw_ref/read_*_zerocopy/File::as_bytes keep their signatures and return a clear "not available for this storage" error on other backends (they already return Option/Result).
• Facade: File::open_storage(Box<dyn Storage + Send + Sync>); File::open keeps mmap and from_bytes keeps Vec, both through impl Storage for [u8].

M3 — HTTP/S3 backend (1–2 weeks).

• clawhdf5-io, feature remote (off by default, so the default tree stays free of C and TLS stacks): RangeStorage over object_store (HTTP, S3, GCS, Azure), with a BlockCache (LRU, block size configurable, default 1 MiB per §2; the page size for paged files; the first block prefetched on open) and a request counter exposed for tests and users.
• Python bindings: clawhdf5.File("s3://…") / https:// through it.

M4 — wasm lazy loading (1–2 weeks).

• clawhdf5-wasm: openUrl(url) -> Promise<H5File> backed by fetch with a Range header, on the main thread, via the restartable NeedBytes loop (no Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back to a whole download when the server does not answer 206.
• examples/wasm-viewer: open by URL.

M5 — SWMR and growth (later, separate design). Storage::len() may grow; add File::refresh() that re-reads the superblock/EOF and invalidates cached blocks past the old end. Needs libhdf5 SWMR semantics research first.

Total: roughly 6–10 engineer-weeks for M0–M4 (estimate, not measured).

Keeping the local fast path

• impl Storage for [u8] returns Cow::Borrowed — no copy, no allocation.
• Hot loops (raw-data copies, contiguous typed reads, read_selection_native) branch once on as_contiguous() and then run today's code.
• &dyn dispatch is per structure, not per byte; parse code keeps working on the returned slice.
• Gate: crates/clawhdf5/benches/mmap_bench.rs, the concurrent-read benches in clawhdf5-bench, and the conformance run time, before and after each M1/M2 commit, on an otherwise idle machine. Anything outside noise blocks the commit.

Risks

• Silent regressions on local files — mitigated by the bench gate above and by M1 being a pure refactor (every conformance hash identical).
• Two reads per structure (prefix, then body) could double requests on a cold cache. Block-aligned caching makes the second read a hit; the measured block counts already include this pattern.
• API break — new error variant (see M1); as_bytes()-style APIs become fallible for non-local storage. ClawBrainHub uses File, FileBuilder, AttrValue, Selection on local files only, so it is unaffected by behaviour, only by exhaustive matches on Error.
• Restartable parsing assumes operations are pure over the storage. The chunk cache and metadata cache must only be filled by completed reads.
• Cache memory — the block cache needs a byte budget and eviction, which (unlike option (b)) is sound because parsers hold Cows, not borrows into the cache.

Testing

• In-memory equivalence: a CountingStorage wrapper over [u8] that records every read_at; run the conformance probe through it and require identical results. It also produces the request counts of §2 without the trap-flag tracer.
• Adversarial storage: a wrapper that returns short reads, errors on the Nth request, or serves blocks of 1 byte, to prove every miss is an Err and never wrong data.
• HTTP: an in-process server on 127.0.0.1 (std TcpListener) that honours Range, answers 206/416, can refuse ranges (200), and counts requests. Tests assert both correctness against h5py and request budgets (e.g. listing file A with a 1 MiB block cache takes ≤ 3 requests) so a change that makes the reader chattier fails CI.
• S3: object_store's in-memory store in unit tests; a MinIO or real bucket only in an opt-in job.
• wasm: the existing Node/Chromium harness in examples/wasm-viewer/test with a local range-capable server.
• 32-bit: a wasm32 or i686 build that opens a sparse > 4 GiB file through the counting storage.

Tools

• docs/design/tools/inventory.py — §1 tables (--list, --patterns).
• docs/design/tools/range-trace/ — §2 tracer (standalone crate, x86-64 Linux, not part of the workspace).
• docs/design/tools/libhdf5_reads.py — §2 libhdf5 comparison and chunk extents.