format: slice entry points for the facade's hot *_in calls (local listing back to main's speed)

Since the M2 merge the facade handed in-memory files to the generic
`*_in` parsers as `&[u8]` (`with_bytes!`), which instantiates them in
the facade crate, where the format crate's private helpers do not
inline without LTO: listing a 400-group v1 file through `File::open`
was 7-10% slower than main. `ObjectHeader::parse_in`,
`group_v2::{resolve_child_in, resolve_group_children_in,
resolve_path_any_in}` and `attribute::{extract_attributes_tolerant_in,
find_attribute_in}` now pass a storage with `as_contiguous()` to their
non-generic slice entry point, compiled once in the format crate; other
storages reach the same generic core as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

CHANGELOG.md

@@ -2,6 +2,340 @@
 
 ## Unreleased
 
+### Range reads, milestone M3: remote files (2026-09-26)
+- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
+  a `clawhdf5::File` (through `File::open_storage`) that reads the file by
+  HTTP `Range` requests; `storage_for_url` returns the cached storage to
+  read its statistics. Features: `http` (default; `ureq` without TLS, no
+  C), `https` (rustls with ring, which compiles C), `object-store` (any
+  `object_store` store, pure Rust), `s3`/`gcs`/`azure` (`s3://`, `gs://`,
+  `az://` URLs, configured from the environment; object_store's cloud
+  clients use aws-lc-rs, C).
+  - **`BlockCache`** wraps any `Storage`: aligned 1 MiB blocks (the size
+    `docs/design/range-reads.md` §2 measured), LRU with a byte budget
+    (64 MiB), the blocks one read misses fetched with one backend
+    `read_ranges` call as runs of consecutive blocks (a one-block gap is
+    fetched to merge two runs; at most 8 MiB per request). Readers on
+    several threads share it without holding its lock across a request, and
+    a block being fetched is waited for, not fetched again. A read that
+    misses more than half the budget is not kept (a large dataset does not
+    evict the metadata). A failed fetch is an error for every reader waiting
+    on it and is not cached.
+  - **`HttpStorage`**: opening is one ranged `GET` of the first block,
+    whose `Content-Range` gives the length. The file is pinned by its strong
+    `ETag` (`If-Match`) or else `Last-Modified` (`If-Unmodified-Since`), and
+    its length, checked on every response: a file changed while open is
+    `RemoteError::FileChanged`, never mixed data. A server that ignores
+    `Range` is refused without reading the body unless
+    `HttpOptions::allow_full_download`. Connection failures, timeouts,
+    `408`/`429`/`5xx` and cut-off bodies are retried with exponential
+    backoff; bodies are asked for with `Accept-Encoding: identity` and an
+    encoded one is refused. The ranges of one call are fetched in parallel.
+    A `200` answer to the first request whose body fits the range asked
+    for is taken as the whole file (a server may answer so for a small
+    file). Timeouts scale with the request: `HttpOptions::timeout` (30 s)
+    to connect and to get the headers, plus the body's size at
+    `HttpOptions::min_speed` (16 KiB/s), so a slow link is not cut off.
+    Redirects: at most `HttpOptions::max_redirects` (5), never from `https`
+    to `http`, and `HttpOptions::headers` are not sent to another origin.
+    No error or `Debug` output contains a URL's userinfo or query values
+    (`redact_url`; presigned URLs carry their signature there).
+  - **Hostile lengths**: the length a server claims is never used to
+    allocate. The cache's arithmetic is checked (a length near `u64::MAX`
+    used to overflow), a read spanning more than the budget is fetched
+    piece by piece with its output growing as data arrives, and
+    `download(storage, max_bytes)` reads a whole file only up to a limit
+    (`RemoteError::TooLarge` before any request otherwise).
+  - **`ObjectStoreStorage`** reads one object of any `object_store` store,
+    pinned by ETag (else version or modification time) and size. Each read
+    runs on a small tokio runtime the storage owns while the caller waits,
+    so it works from any thread, `spawn_blocking` and other runtimes
+    included.
+    `open_object(store, path, options)` opens a file through a block cache.
+  - Counted on the conformance corpus (tank, 2026-09-26,
+    `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus CLAWHDF5_REMOTE_REPORT=1
+    cargo test --release -p clawhdf5-remote --test http -- --nocapture corpus`):
+    the 621 files that open (254 MB) read over HTTP exactly as through
+    `File::open`; opening and listing them all (every group's entries, every
+    dataset's shape and type) took 640 requests and 55.5 MB, and then
+    reading each file's largest dataset under 64 MiB 96 more requests
+    (171 MB in all). Without the cache the same work is 141 936 requests.
+    Listing the 7.7 MB IMERG file (file A of the design's §2) takes 2
+    requests; the tests hold it to at most 3.
+- **`h5rs` takes URLs** with the new `remote` feature (`remote-https` for
+  `https://`): `ls`, `dump`, `stat` and `diff` read by range requests;
+  `check` downloads the file whole, since it validates every byte, up to
+  `--max-download N` (1 GiB by default). URLs are printed without their
+  credentials. The
+  tools now read through `File::storage` and the format crate's `*_in`
+  functions; local output is unchanged.
+- **`File::storage()`** (facade) returns the file's bytes from the
+  superblock on (the `as_bytes` view, cache image laid over) as a
+  `&(dyn Storage + Send + Sync)` for every backend, so code that parses a
+  file itself works on remote files too.
+- **`VlResolver::element_in` / `string_element_in`** (format): the
+  `element`/`string_element` lookups over any `Storage`.
+
+### Range reads, milestone M2: raw data and `File::open_storage` (2026-09-26)
+- **`clawhdf5::File::open_storage(Arc<dyn Storage + Send + Sync>)`** opens
+  a file served by any `clawhdf5_format::storage::Storage` and gives the
+  whole read API over it: groups and paths, datasets, attributes, the
+  `read_*` methods, selections, variable-length strings and sequences, and
+  virtual datasets. Every byte comes through `Storage::read_at` /
+  `read_ranges`; a user block is found and skipped, nothing past the
+  superblock's end of file is read, and a metadata cache image is laid over
+  the reads it covers (new `CacheImage::entries`). External virtual-dataset
+  sources are read through the new `File::set_vds_resolver` (any `File`;
+  without one a storage-backed file cannot follow them). `File::open` and
+  `File::from_bytes` keep their mmap and in-memory paths: the file's view
+  is now a `Storage` whose `as_contiguous()` is that buffer, and every hot
+  loop takes it. New exports: `clawhdf5::{Storage, SharedStorage,
+  VdsResolver}`, `File::contiguous_bytes()`.
+  - Over a storage without the whole file in memory the zero-copy methods
+    (`read_raw_ref`, `read_as_slice`, `read_*_zerocopy`) answer
+    `FormatError::ContiguousStorageRequired`, and `File::as_bytes` panics
+    (documented; use `contiguous_bytes`). The typed readers keep their fast
+    paths: a contiguous dataset is read in one piece and converted, and a
+    contiguous selection of a native type reads only its runs
+    (`data_read::read_selection_native_in`).
+- **Nothing in the format crate needs the whole file any more.** The
+  structures M1 left to `ContiguousStorageRequired` read through `Storage`:
+  v2 B-trees (`BTreeV2Header::parse_in`, `collect_btree_v2_records_in`,
+  `find_btree_v2_records_in`; one bounded read per node, whose size is known
+  before it is read), hence dense attributes, a SOHM B-tree index, huge
+  fractal-heap objects, and dense groups; v1 and v2 group listings, lookups
+  and paths (`group_v2::resolve_group_children_in`, `resolve_child_in`,
+  `resolve_path_any_in`, `group_v1::*_in`).
+- **Raw data reads through `Storage`**, each with a generic `*_in` core and
+  its `&[u8]` function as a thin wrapper (callers do not change):
+  `data_read` (`read_raw_data*_in`, `read_raw_data_selection_in`,
+  `read_chunked_native_in`), `chunked_read` (the v1 B-tree chunk index —
+  one read of each node's header, one of its entries — `list_chunks_in`,
+  and the full, cached, sweep and indexed reads), `parallel_read`,
+  `partial_read`, `fill_value` (`read_full_with_fill_in`,
+  `apply_to_unallocated_chunks_in`; `dataset_fill_value_from_storage` is
+  now generic, so a `&dyn Storage` still works), `vds`
+  (`read_virtual_dataset_in`, `virtual_dataset_extent_in`: the virtual
+  file through `Storage`, external source files still loaded whole through
+  the resolver), `vl_data` (`VlResolver<'a, S = [u8]>` with `new_in`;
+  `read_vl_strings_in`, `read_vl_bytes_in`),
+  `AttributeMessage::read_vl_strings_in`, `provenance::verify_dataset_in`.
+  - A chunked read first lists the chunks it needs, then fetches all their
+    stored bytes with **one `read_ranges` call** per batch of at most 64 MiB
+    (`storage::RAW_BATCH_BYTES`), so a remote backend can coalesce and
+    parallelise them, then decodes each batch as before (in parallel with
+    the `parallel` feature) before fetching the next. Every path that reads
+    chunks — full, cached, indexed, sweep, selection and the
+    `parallel_read` decoders — goes through the same batching, and no chunk
+    fetches more of its stored bytes than its decoded size can need (the
+    chunk size if unfiltered; else each applied filter's worst-case growth,
+    generously: `n + n/4 + 4096` per codec, unbounded only for a codec the
+    application registered). A crafted chunk index that points every chunk
+    at one huge extent therefore costs a bounded fetch, not
+    `chunks x extent` bytes (`tests/raw_fetch_bounds.rs`). Chunks the
+    file's chunk cache already holds are not fetched. A selection fetches
+    only the chunks its bounding box overlaps; a contiguous selection only
+    its runs, merged into reads of up to 8 MiB across gaps of up to 4 KiB
+    (a stride-2 selection of 32M `f32` is 32 reads and 0.3 s over a
+    `CountingStorage`, where one read per element was 16.8M reads, 2.0 s
+    and 2.1 GB peak). A global-heap collection is read once
+    per resolver and kept (within the resolver's 32 MiB budget).
+  - Each extent's bounds error is the one the slice readers gave, reported
+    when the read reaches that extent, so a damaged file fails with the
+    same error, in the same order, through either path.
+  - A backend that answers a read with more bytes than asked (breaking
+    `read_at`'s contract) never has the extra bytes used: every read is
+    cut to the range asked for (`storage::exact_len`), and a short answer
+    inside the file is an error.
+- **No behaviour change for in-memory and mapped files:** with
+  `as_contiguous()` every path slices the file as before (checked below).
+- **Speed on local files** (provisional: tank was shared with other jobs,
+  load 4–16 during the runs; Criterion `local_metadata_bench`, `main`
+  `8f59b2e` and this branch as separate binaries, 8 alternating rounds,
+  best round of each). Listing the 400-group version-1 file through the
+  facade (`File::open`, mmap) had become 7–10% slower than `main` (8.95–9.04
+  vs 8.14–8.36 ms), from the M2 merge on (bisected over the merges: `main`
+  8.17–8.27, `2893b6c` 8.69–8.76, `93e2d5f`/`7447dce`/`4313917` the same).
+  The parsers were unchanged: the facade now called the generic `*_in`
+  entry points with the slice (`with_bytes!`), which instantiates each
+  parser in the facade crate, where the format crate's private helpers do
+  not inline (no LTO); `main` called the `&[u8]` wrappers, compiled in the
+  format crate. Routing just `resolve_child`/`resolve_group_children` to
+  the wrappers took the listing from 8.66–8.70 to 8.33–8.46 ms. Fixed in
+  the format crate: `ObjectHeader::parse_in`, `group_v2::{resolve_child_in,
+  resolve_group_children_in, resolve_path_any_in}` and
+  `attribute::{extract_attributes_tolerant_in, find_attribute_in}` hand a
+  storage with `as_contiguous()` to their non-generic slice entry point,
+  compiled once in the format crate; everything else goes to the same
+  generic core as before, so `File::open_storage` is unchanged. Best
+  rounds, `main` / before / after: facade listing 8.49 / 9.07 / 8.68 ms
+  (+2.2% on `main`, was +6.9%); `ObjectHeader::parse` ×401 24.91 / 25.90 /
+  25.51 µs; symbol-table nodes 1.99 / 1.94 / 1.94 µs; group B-tree walk
+  360.6 / 365.0 / 361.1 ns. The B-tree walk's `btree_v1.rs` is identical to
+  `main` and it calls only format-crate code; its earlier +5–10% (323–355
+  vs 353–363 ns) was run-to-run layout noise (at `2893b6c` it measured
+  357–374 ns against `main`'s 355–371 in the same rounds).
+- Tests (2026-09-26, tank):
+  - `clawhdf5-format/tests/storage_equivalence.rs` now also reads every
+    dataset — whole, fill-aware, through a chunk cache (twice) and the
+    indexed path, three selections, virtual datasets with their sibling
+    sources, VL strings, sequences and bytes — through the read_at-only
+    `CountingStorage` and requires the slice results, and fails on any
+    `ContiguousStorageRequired`. With
+    `CLAWHDF5_STORAGE_CORPUS=conformance/.cache/corpus`, all 653 HDF5
+    files of the corpus agree (82 396 checks). The cached and indexed
+    paths are compared on values only when a read fails: they order
+    chunks by hash map, so which failing chunk a damaged dataset reports
+    varies between two caches even for the same slice (seen on
+    `cve-2025-2310.h5`; see `docs/known-issues.md`).
+  - A misbehaving storage (fails its N-th read; serves short reads) over
+    every fixture: each listing and dataset read is an error or exactly the
+    in-memory result, never other data (1 137 runs).
+  - A chunked read issues one `read_ranges` call with one range per chunk,
+    and a one-chunk selection one call with one range.
+  - `clawhdf5/tests/storage_equivalence.rs` reads every fixture (61 files)
+    and, with `CLAWHDF5_STORAGE_CORPUS`, every corpus file (701 files, 621
+    that open) through `File::open` and through `File::open_storage` over
+    `CountingStorage`: the tree, every attribute (all, and each by name),
+    every dataset's shape, types and values (all bytes, `f64`, `f32`,
+    `i64`, a box hyperslab, a strided one, out-of-order points, strings, VL
+    sequences) must be identical, and are — errors included, in full (open
+    errors and every read's). The one allowance is a line on which
+    `File::open` itself varies between two opens (the chunk cache lists a
+    damaged dataset's chunks in hash-map order, so which failing chunk a
+    full read reports varies: `cve-2025-2310.h5`), and then only if a fresh
+    `File::open` reproduces the storage's error. No storage read may answer
+    `ContiguousStorageRequired`. A storage that returns more bytes than
+    asked reads every fixture identically too.
+    It also counts what one pass — open, list, read every attribute and
+    every dataset once — asks of a storage with no cache: 176 092 `read_at`
+    calls and 208 MB for the 621 corpus files (254 MB of files); the most
+    are `h5stat_newgrat.h5` (35 001 groups: 92 489 calls) and
+    `ref_hdf5_compat1.nc` (16 062). A remote backend needs the block cache
+    of milestone M3. Command: `CLAWHDF5_STORAGE_CORPUS=… cargo test
+    --release -p clawhdf5 --test storage_equivalence -- --nocapture`.
+  - Conformance sweep (`conformance/run.sh --no-fetch`): 600 of 697 files
+    ok, `results.json` byte-identical to `8f59b2e`.
+
+### Correctness: Fletcher-32 (2026-09-26)
+- **Fletcher-32 checksums disagreed with libhdf5's on about one chunk in
+  32768** (fixed 2026-09-26). **Every release is affected, v2.1.0 through
+  v2.7.0**, both directions: `FileBuilder`/`FileWriter` (`with_fletcher32`)
+  and, before release, `FileEditor` wrote chunks that h5py and libhdf5
+  refuse ("filter returned failure during read"), and every reader
+  rejected valid libhdf5-written chunks with `Fletcher32Mismatch`. Our
+  checksum reduced its sums with `% 65535`; libhdf5's
+  `H5_checksum_fletcher32` uses the ones'-complement fold
+  `(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves
+  0, so the two differ whenever a sum is a non-zero multiple of 65535.
+  `clawhdf5_format::checksum::fletcher32` (new, public) is a port of
+  `H5_checksum_fletcher32` and the only implementation; the filter writes
+  and verifies with it, and, as libhdf5 does, also accepts a stored
+  checksum with the bytes of each 16-bit half swapped (libhdf5 1.6.2 and
+  earlier) and the `% 65535` form v2.7.0 and earlier wrote, so their files
+  stay readable. Tests: `crates/clawhdf5/tests/fletcher32_interop.rs` compares
+  it with libhdf5's own function (through ctypes) on every 1- and 2-byte
+  input and 40 000 random and fold-heavy ones, and has h5py read
+  fold-case chunks written by `FileBuilder` and `FileEditor` and us read
+  h5py's. Files written by earlier releases read with a fixed build; to
+  make one readable by libhdf5, rewrite its Fletcher-32 datasets with a
+  fixed build (see `docs/known-issues.md`). `clawhdf5_accel::checksum_fletcher32` is a
+  different, textbook Fletcher-32 (sums start at 0xffff) and is not used
+  for HDF5.
+
+### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
+- **`FileEditor` adds, moves and resizes chunks of datasets with two or
+  more unlimited dimensions** (version-2 B-tree chunk index, record types
+  10/11), as libhdf5's `H5B2` code does: `H5B2_update`'s insert-or-modify,
+  the preemptive split/redistribute loop, `split1`/`split_root` (depth
+  growth), `redistribute2/3`, and removal with `merge2/3`, root collapse
+  and the internal-record swap; node pointer widths and cumulative record
+  counts per depth; a missing index is created from the layout message's
+  parameters. After the same growth libhdf5's and the editor's trees are
+  node for node the same (tested through a depth increase).
+- **`FileEditor::resize` shrinks** along any dimension (h5py's
+  `Dataset.resize` to a smaller shape), as `H5D__chunk_prune_by_extent`
+  does, visiting the same chunks in the same order: chunks wholly outside
+  the new extent leave the index (version-1 B-tree removal with libhdf5's
+  sibling key and link fix-ups and empty-root case, version-2 B-tree
+  removal, Fixed/Extensible Array elements reset; an implicit index keeps
+  its chunks, as in libhdf5) and their space is freed; the part of a
+  partial edge chunk outside the extent is overwritten with the fill value,
+  so it reads as fill after a later growth. Growth under early allocation
+  now allocates and fills the new chunks (`H5D__chunk_allocate`), which an
+  implicit index needs. Shrinking was `Error::Unsupported`. Only the
+  chunks that exist are visited (placed in libhdf5's order), so shrinking
+  a sparse dataset costs memory and time in its chunks, not in the
+  coordinates cut off (a 2 x 10^12-coordinate shrink takes 0.6 s).
+- **`FileEditor::set_attr` handles dense attribute storage and creation
+  order**: objects that track (and index) attribute creation order; the
+  move to dense storage when an object reaches its compact limit (or an
+  attribute is too large for a header message), as `H5O__attr_create`
+  does it (new fractal heap, name index, creation-order index when
+  indexed, compact attributes moved over in header order); objects
+  already in dense storage (h5py- or clawhdf5-written): insertion,
+  same-size rewrites in place, other replacements by removal and
+  insertion. The heap is changed as `H5HF` changes it — best-fit free
+  sections from its free-space manager (kept as libhdf5 keeps `FSHD`/
+  `FSSE`), new direct blocks through the root indirect block (created,
+  doubled), blocks too small for an attribute skipped as libhdf5 skips
+  them (`H5HF__hdr_skip_blocks`: an indirect free section with its row
+  sections, serialized as libhdf5 serializes them, merged with the range
+  skipped just before it, and later attributes given skipped blocks from
+  either end or the middle of a range, which splits it), huge objects
+  through the huge-object B-tree (deleted with the last huge object),
+  removed objects' space merged back — with libhdf5's statistics: after
+  the same attribute workload the heap, its free space and both index
+  B-trees equal libhdf5's (`dense_skipped_blocks_match_libhdf5` covers
+  every way of skipping, with libhdf5 doing one edit per session as the
+  editor does). In a random attribute workload (1-4 KiB attributes among
+  small ones) 24% of `set_attr` calls were refused before skipping was
+  implemented; 2.2% are now, all replacements of the last attribute in a
+  heap block. Attributes are encoded as libhdf5
+  encodes them for a file h5py opens `r+` (message version 1, 3 for
+  non-ASCII names; simple dataspaces with their maximum dimensions).
+  Still refused: see `docs/known-issues.md`.
+- **Freed space is reused within an editing session.** A `FileEditor`
+  reuses (best fit, zeroed) what its earlier edits freed — moved filtered
+  chunks, pruned chunks, merged B-tree nodes, replaced heap blocks — never
+  what the current edit frees, and writes reused blocks with the new space
+  before any existing byte changes. `FileEditor::reusable_bytes`. The
+  append workload of `measure_append_waste` leaks less (sizes in
+  `docs/known-issues.md`).
+- **Reader: implicit chunk indexes below their maximum shape.** libhdf5
+  places an implicit index's chunks by their position in the *maximum*
+  chunk grid; the reader used the current grid and returned other chunks'
+  values from the second chunk row on (h5py early allocation with a fixed
+  `maxshape` larger than the shape).
+  `chunked_read::generate_implicit_chunks_in_grid` takes the maximum.
+- **Reader: object headers with long continuation chains.** A version-1
+  header whose continuation chunks chain more than 32 deep (a header that
+  gains a chunk per attribute added when full, as libhdf5 and the editor
+  grow it) was refused with `NestingDepthExceeded`; version-2 headers
+  stopped at 256 chunks. Chunks are now read one at a time from a queue,
+  in the order their continuation messages are found (libhdf5's
+  `H5O_protect` order, which the editor already used; a version-1
+  chunk's messages used to be inserted at its continuation message), each
+  buffer released before the next is read; a chunk address seen twice (a
+  cycle), chunks adding up to more than the file (a crafted chain of
+  chunks nested in each other made storage with owned buffers read and
+  hold the square of the file's size), or more than 65 536 chunks are
+  refused, so a header's chunks read at most the file's size.
+  The first version allocated a queue and a set of chunk starts for every
+  header and made `ObjectHeader::parse` of 401 small headers 1.8x slower
+  (`local_metadata_bench`: 45.7 vs 24.8 µs on `main` `8f59b2e`). The first
+  8 chunks now live in an inline array (cycle check by scan; only a longer
+  header allocates), and the per-chunk message loop is its own function
+  instead of being inlined into the generic parser. Provisional (tank load
+  3–4; Criterion, separate binaries, 2 alternating rounds): 24.96–25.12 vs
+  24.85–25.15 µs on `main`.
+- Tests: `crates/clawhdf5-tools/tests/edit_coverage_interop.rs` (h5py
+  `earliest`/`v110`/`latest` and clawhdf5-written files; structure
+  comparisons with libhdf5 for version-2 B-trees, shrink on every index,
+  and dense attribute heaps); the random-operation property test in
+  `edit_interop.rs` now shrinks, grows two unlimited dimensions and moves
+  attributes to dense storage (`CLAWHDF5_EDIT_SEED` for other seeds).
+
 ### Name lookups through the name index (2026-09-26)
 - **Finding one link or attribute by name reads the name index, not every
   entry.** In a dense group (links in a fractal heap) the v2 B-tree name
@@ -108,7 +442,8 @@
   v2 B-tree and dense groups come with milestone M3); over a backend without
   the whole file in memory they are the clean `ContiguousStorageRequired`
   error, never a partial result. Raw data, chunk B-tree (v1) indexes and VL
-  data are milestone M2.
+  data are milestone M2. (All of them read through `Storage` since M2,
+  above.)
 - **No behaviour change**, checked three ways (2026-09-26, tank): every
   existing test passes unchanged; the conformance sweep
   (`conformance/run.sh --no-fetch`) gives a byte-identical `results.json`
@@ -237,7 +572,8 @@
     before any existing byte changes, then the metadata that links it in,
     then a second sync. There is no journal: a crash during the second
     phase can leave the file inconsistent (as with libhdf5 without SWMR).
-    Freed space is not reused (see `docs/known-issues.md`).
+    Freed space is not reused (see `docs/known-issues.md`; since reused
+    within an editing session, above).
   - Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
     `v114` and `latest` files and clawhdf5 files; after every round h5py
     reads the expected values, h5dump and `h5rs check --data` accept the

CLAUDE.md

@@ -5,7 +5,7 @@ Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persist
 
 ## Architecture
 
-Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
+Cargo workspace with 19 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
 
 | Crate | Role |
 |-------|------|
@@ -26,6 +26,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
 | `clawhdf5-napi` | Node.js native addon bindings |
 | `clawhdf5-py` | PyO3 Python bindings |
 | `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
+| `clawhdf5-remote` | Remote files: `open_url` over HTTP(S) range requests and object stores (`object_store`: S3, GCS, Azure) through a mandatory block cache (`BlockCache`) |
 | `clawhdf5-bench` | Benchmark suite |
 
 ## Key Features
@@ -151,12 +152,34 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
   `MemorySource` for this bookkeeping is inferred from the caller-supplied
   `source_channel` string (a heuristic, not an authenticated trust boundary).
 - In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`)
-  overwrites values, grows chunked datasets and sets attributes in existing
+  overwrites values, grows and shrinks chunked datasets (every chunk index,
-  files (h5py- or clawhdf5-written) without rewriting them; anything it
+  version-2 B-trees included) and sets attributes (compact and dense
-  cannot do safely is `Error::Unsupported` before any write (limits in
+  storage) in existing files (h5py- or clawhdf5-written) without rewriting
+  them, changing indexes and heaps as libhdf5 does (index shapes and heap
+  bookkeeping are compared with libhdf5's in the tests); space an edit
+  frees is reused by later edits of the same editor. Anything it cannot do
+  safely is `Error::Unsupported` before any write (limits in
   `docs/known-issues.md`). Test changes with
-  `cargo test -p clawhdf5-tools --test edit_interop` (h5py, h5dump,
+  `cargo test -p clawhdf5-tools --test edit_interop --test
-  `h5rs check`).
+  edit_coverage_interop` (h5py, h5dump, `h5rs check`, structure comparisons
+  with libhdf5; libhdf5 sources for the algorithms are at
+  github.com/HDFGroup/hdf5, tag `hdf5_1_14_6`).
+- Remote files (`clawhdf5-remote`, range-read milestone M3 of
+  `docs/design/range-reads.md`): `open_url("http://…")` gives a
+  `clawhdf5::File` over `File::open_storage`, read through `BlockCache`
+  (1 MiB blocks, LRU byte budget, per-block in-flight dedup across threads,
+  runs coalesced into parallel requests). `HttpStorage` pins the file by
+  ETag/Last-Modified and length (a change is `RemoteError::FileChanged`),
+  refuses servers that ignore `Range` unless a full download is allowed,
+  and retries transient failures. `ObjectStoreStorage` (feature
+  `object-store`, pure Rust) runs each read on a small owned tokio
+  runtime and waits on a channel, so it works from any thread, including
+  inside `spawn_blocking` or another runtime. Default build is plain HTTP with
+  no C; `https` (rustls + ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are
+  opt-in. Tests run a std-only HTTP server
+  (`tests/common/server.rs`, also the `range_server` example);
+  `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus
+  file over HTTP with `File::open`.
 - GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
 - Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
   no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.

CONFORMANCE.md

@@ -13,8 +13,8 @@ fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
 
 | | |
 |---|---|
-| date | 2026-09-26 20:06 UTC |
+| date | 2026-09-27 00:34 UTC |
-| clawhdf5 commit | `8fadb9f4242a35323262701328d380806d379140` |
+| clawhdf5 commit | `f37e7ae3263277319dba4bc39be5397194eb00c3` |
 | machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
 | command | `conformance/run.sh --no-fetch --update-baseline` |
 | rustc | rustc 1.98.1 (48a229cea 2026-09-01) |

Cargo.toml

@@ -18,6 +18,7 @@ members = [
     "crates/clawhdf5-bench",
     "crates/clawhdf5-tools",
     "crates/clawhdf5-wasm",
+    "crates/clawhdf5-remote",
     "crates/libaec-sys",
 ]
 resolver = "2"

README.md

@@ -96,6 +96,12 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
   filtered top-k, never slower than unfiltered), and opt-in re-ranking and
   confidence rejection, which used to be reachable only through `ClawhdfBackend`.
 
+**Remote files (unreleased)**
+- New crate `clawhdf5-remote`: `open_url("http://…")` reads a file on an
+  HTTP server (or in S3/GCS/Azure, opt-in) by range requests through a
+  block cache, without downloading it; `h5rs` takes URLs with its `remote`
+  feature. See [Reading remote files](#reading-remote-files).
+
 **Tooling**
 - CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
   silently) and runs an aarch64 job for the NEON kernels.
@@ -444,12 +450,60 @@ ed.resize("x", &[1100])?;                   // h5py: ds.resize((1100,))
 let sel = Selection::Hyperslab { start: vec![1000], stride: vec![1], count: vec![100], block: vec![1] };
 ed.write_values("x", &sel, &[0.5f64; 100])?; // ds[1000:1100] = 0.5
 ed.set_attr("x", "units", &AttrValue::String("m/s".into()))?;
+ed.resize("x", &[900])?;                    // shrinking prunes chunks, like h5py
 ```
 
-Each call changes the file in place (no rewrite) and syncs it. What it
+Each call changes the file in place (no rewrite) and syncs it. Any chunk
+index (version-2 B-trees for several unlimited dimensions included) and
+attributes in compact or dense storage are handled as libhdf5 handles
+them; space an edit frees is reused by later edits of the same editor. What it
 cannot change safely is refused before anything is written; see
 [known issues](docs/known-issues.md) for the limits.
 
+### Reading remote files
+
+[`clawhdf5-remote`](crates/clawhdf5-remote/README.md) opens a file on an
+HTTP server (or, with its `s3`/`gcs`/`azure` features, in an object store)
+without downloading it: the read API is the same `clawhdf5::File`, and
+only the bytes an operation needs are fetched, by `Range` requests through
+a block cache (1 MiB blocks; opening fetches the first one). A file that
+changes on the server while it is open is an error, never a mix of old and
+new bytes.
+
+```rust
+let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/tall.h5")?;
+let values = file.dataset("/g2/dset2.1")?.read_f64()?;
+```
+
+To try it without a server of your own, the crate's test server serves a
+directory with range support:
+
+```bash
+cargo run -p clawhdf5-remote --example range_server -- crates/clawhdf5/tests/fixtures 127.0.0.1:8000
+# in another shell: list the file, read one dataset, print what it cost
+cargo run -p clawhdf5-remote --example read_url -- http://127.0.0.1:8000/tall.h5 /g2/dset2.1
+```
+
+```text
+/g1  group
+/g2  group
+/g2/dset2.1  dataset [10] F32
+/g2/dset2.2  dataset [3, 5] F32
+/g1/g1.1  group
+/g1/g1.2  group
+/g1/g1.2/g1.2.1  group
+/g1/g1.1/dset1.1.1  dataset [10, 10] I32
+/g1/g1.1/dset1.1.2  dataset [20] I32
+/g2/dset2.1: 10 values, first [1.0, 1.100000023841858, 1.2000000476837158, ...]
+1 range requests (the one at open included), 9968 bytes fetched, 9968 bytes cached
+```
+
+(`tall.h5` is 9 968 bytes, so the first block holds all of it.) `h5rs`
+built with `--features remote` takes the same URLs:
+`h5rs ls -r http://127.0.0.1:8000/tall.h5`. Plain HTTP builds no C;
+`https://` is the `https` feature (rustls with ring, which compiles C).
+Limits are in [known issues](docs/known-issues.md).
+
 ### Python
 
 `crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
@@ -678,7 +732,7 @@ let exported = backend.export_markdown("MEMORY.md")?;
 ## Crate Map
 
 ```
-clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
+clawhdf5 workspace (19 crates, ~86K lines of Rust in src/, ~104K with tests
                      and benches; plus libaec-sys, an internal FFI bindings
                      crate for the optional szip feature)
 │
@@ -690,7 +744,8 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
 │   ├── clawhdf5             — High-level API
 │   ├── clawhdf5-netcdf4     — NetCDF-4 support
 │   ├── clawhdf5-accel       — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
-│   └── clawhdf5-gpu         — GPU compute (wgpu, hand-written WGSL compute shaders)
+│   ├── clawhdf5-gpu         — GPU compute (wgpu, hand-written WGSL compute shaders)
+│   └── clawhdf5-remote      — Remote files: HTTP(S) range requests, object stores, block cache
 │
 ├── Agent Memory
 │   ├── clawhdf5-agent       — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
@@ -705,6 +760,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
 │   └── clawhdf5-wasm        — Browser (WebAssembly, wasm-bindgen; read-only)
 │
 └── Tooling
+    ├── clawhdf5-tools       — h5rs: ls, dump, stat, diff, check
     └── clawhdf5-bench       — Benchmark suite
 ```
 

conformance/baseline.json

@@ -1,7 +1,7 @@
 {
  "comment": "conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
- "commit": "8fadb9f4242a35323262701328d380806d379140",
+ "commit": "f37e7ae3263277319dba4bc39be5397194eb00c3",
- "date": "2026-09-26 20:06 UTC",
+ "date": "2026-09-27 00:34 UTC",
  "reference": "h5py 3.16.0 / HDF5 2.0.0",
  "files": 697,
  "ok": 600,

crates/clawhdf5-accel/src/lib.rs

@@ -237,7 +237,10 @@ pub fn f16_to_f32_batch(input: &[u16], output: &mut [f32]) {
     convert::f16_to_f32_batch(input, output);
 }
 
-/// Compute Fletcher-32 checksum.
+/// Compute a textbook Fletcher-32 checksum (both sums start at 0xffff).
+///
+/// This is not HDF5's checksum; the Fletcher-32 I/O filter uses
+/// `clawhdf5_format::checksum::fletcher32`.
 pub fn checksum_fletcher32(data: &[u8]) -> u32 {
     checksum::checksum_fletcher32(data)
 }

crates/clawhdf5-format/src/addr.rs

@@ -23,6 +23,15 @@ pub fn to_usize(value: u64) -> Result<usize, FormatError> {
     to_index::<usize>(value)
 }
 
+/// A file address for a [`crate::storage::Storage`] read, checked as
+/// [`to_usize`] checks it: the parsers read through 64-bit offsets, but an
+/// address that could not index an in-memory file on this platform is the
+/// same [`FormatError::Overflow`] the slice parsers gave for it.
+#[inline]
+pub fn checked_addr(value: u64) -> Result<u64, FormatError> {
+    to_usize(value).map(|_| value)
+}
+
 /// [`to_usize`] for an index type of any width. `usize` is 64 bits wide on
 /// the hosts CI tests on, where the error path cannot be reached through
 /// `usize`; tests run the same code with `u32` in its place, as on a 32-bit

crates/clawhdf5-format/src/attribute.rs

@@ -7,7 +7,7 @@ use std::borrow::Cow;
 
 use crate::addr::to_usize;
 use crate::attribute_info::AttributeInfoMessage;
-use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records, find_btree_v2_records};
+use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records_in, find_btree_v2_records_in};
 use crate::checksum::jenkins_lookup3;
 use crate::data_read;
 use crate::dataspace::Dataspace;
@@ -17,7 +17,7 @@ use crate::fractal_heap::FractalHeapHeader;
 use crate::message_type::MessageType;
 use crate::object_header::ObjectHeader;
 use crate::shared_message;
-use crate::storage::{Storage, require_contiguous};
+use crate::storage::Storage;
 use crate::vl_data;
 
 /// A parsed HDF5 attribute message.
@@ -336,9 +336,19 @@ impl AttributeMessage {
         file_data: &[u8],
         offset_size: u8,
         length_size: u8,
+    ) -> Result<Vec<String>, FormatError> {
+        self.read_vl_strings_in(file_data, offset_size, length_size)
+    }
+
+    /// [`Self::read_vl_strings`] over any [`Storage`].
+    pub fn read_vl_strings_in<S: Storage + ?Sized>(
+        &self,
+        file_data: &S,
+        offset_size: u8,
+        length_size: u8,
     ) -> Result<Vec<String>, FormatError> {
         let num_elements = self.dataspace.num_elements();
-        vl_data::read_vl_strings(
+        vl_data::read_vl_strings_in(
             file_data,
             &self.raw_data,
             num_elements,
@@ -460,16 +470,31 @@ pub fn extract_attributes_tolerant(
     offset_size: u8,
     length_size: u8,
 ) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
-    extract_attributes_tolerant_in(file_data, header, offset_size, length_size)
+    extract_attributes_tolerant_core(file_data, header, offset_size, length_size)
 }
 
 /// [`extract_attributes_tolerant`] over any [`Storage`] (see
-/// [`extract_attributes_full_in`] for dense storage).
+/// [`extract_attributes_full_in`] for dense storage). One with the whole
+/// file in memory is read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
 pub fn extract_attributes_tolerant_in<S: Storage + ?Sized>(
     file_data: &S,
     header: &ObjectHeader,
     offset_size: u8,
     length_size: u8,
+) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => extract_attributes_tolerant(all, header, offset_size, length_size),
+        None => extract_attributes_tolerant_core(file_data, header, offset_size, length_size),
+    }
+}
+
+fn extract_attributes_tolerant_core<S: Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
     let mut errors = Vec::new();
     let attrs = extract_attributes_with(file_data, header, offset_size, length_size, &mut |e| {
@@ -551,18 +576,34 @@ pub fn find_attribute_in_file(
     offset_size: u8,
     length_size: u8,
 ) -> Result<Option<AttributeMessage>, FormatError> {
-    find_attribute_in(file_data, header, name, offset_size, length_size)
+    find_attribute_core(file_data, header, name, offset_size, length_size)
 }
 
 /// [`find_attribute_in_file`] over any [`Storage`] (see
 /// [`extract_attributes_full_in`] for dense storage, whose name index still
-/// needs the whole file in memory).
+/// needs the whole file in memory). One with the whole file in memory is
+/// read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
 pub fn find_attribute_in<S: Storage + ?Sized>(
     file_data: &S,
     header: &ObjectHeader,
     name: &str,
     offset_size: u8,
     length_size: u8,
+) -> Result<Option<AttributeMessage>, FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => find_attribute_in_file(all, header, name, offset_size, length_size),
+        None => find_attribute_core(file_data, header, name, offset_size, length_size),
+    }
+}
+
+fn find_attribute_core<S: Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Option<AttributeMessage>, FormatError> {
     let attr_info = find_attribute_info(header, offset_size)?;
     let dense = attr_info
@@ -578,9 +619,12 @@ pub fn find_attribute_in<S: Storage + ?Sized>(
                 .find(|a| a.name == name),
         );
     };
-    let contiguous = require_contiguous(file_data, "dense attribute storage (a v2 B-tree)")?;
+    let btree_hdr = BTreeV2Header::parse_in(
-    let btree_hdr =
+        file_data,
-        BTreeV2Header::parse(contiguous, to_usize(btree_addr)?, offset_size, length_size)?;
+        to_usize(btree_addr)? as u64,
+        offset_size,
+        length_size,
+    )?;
     let fh = FractalHeapHeader::parse_in(file_data, fh_addr, offset_size, length_size)?;
     if btree_hdr.tree_type != ATTRIBUTE_NAME_INDEX || btree_hdr.record_size < 4 {
         return Ok(
@@ -610,7 +654,7 @@ pub fn find_attribute_in<S: Storage + ?Sized>(
     // hash is the last field.
     let hash = jenkins_lookup3(name.as_bytes());
     let hash_at = usize::from(btree_hdr.record_size) - 4;
-    let records = find_btree_v2_records(contiguous, &btree_hdr, offset_size, &mut |r| match r
+    let records = find_btree_v2_records_in(file_data, &btree_hdr, offset_size, &mut |r| match r
         .get(hash_at..hash_at + 4)
     {
         Some(h) => u32::from_le_bytes([h[0], h[1], h[2], h[3]]).cmp(&hash),
@@ -722,10 +766,13 @@ fn extract_dense_attributes<S: Storage + ?Sized>(
             expected: 1,
             available: 0,
         })?;
-    let contiguous = require_contiguous(file_data, "dense attribute storage (a v2 B-tree)")?;
+    let btree_hdr = BTreeV2Header::parse_in(
-    let btree_hdr =
+        file_data,
-        BTreeV2Header::parse(contiguous, to_usize(btree_addr)?, offset_size, length_size)?;
+        to_usize(btree_addr)? as u64,
-    let records = collect_btree_v2_records(contiguous, &btree_hdr, offset_size, length_size)?;
+        offset_size,
+        length_size,
+    )?;
+    let records = collect_btree_v2_records_in(file_data, &btree_hdr, offset_size, length_size)?;
 
     for record in &records {
         // Per HDF5 spec, both type 8 and type 9 records start with heap_id:
@@ -1156,11 +1203,9 @@ mod tests {
     }
 
     /// Every object's attributes in h5py-written files read identically
-    /// through a read_at-only CountingStorage — compact ones, shared ones
+    /// through a read_at-only CountingStorage — compact ones, shared ones,
-    /// and those behind an Attribute Info message — except dense storage,
+    /// those behind an Attribute Info message and dense storage (its v2
-    /// whose v2 B-tree index is not read over Storage yet: that is the clean
+    /// B-tree name index included) — and through a slice as Storage.
-    /// ContiguousStorageRequired error, never a partial list. Through a
-    /// slice as Storage every object matches.
     #[test]
     fn storage_reads_match_slice_reads() {
         use crate::storage::CountingStorage;
@@ -1206,18 +1251,17 @@ mod tests {
                     .unwrap()
                     .is_some_and(|i| i.fractal_heap_address.is_some());
                 if is_dense {
-                    let e = FormatError::ContiguousStorageRequired(
-                        "dense attribute storage (a v2 B-tree)",
-                    );
-                    assert_eq!(got.unwrap_err(), e, "{name}");
-                    assert_eq!(got_t.unwrap_err(), e, "{name}");
                     dense += 1;
-                } else {
+                }
-                    attrs += want.as_ref().map_or(0, Vec::len);
+                attrs += want.as_ref().map_or(0, Vec::len);
-                    assert_eq!(format!("{got:?}"), format!("{want:?}"), "{name}");
+                assert_eq!(format!("{got:?}"), format!("{want:?}"), "{name}");
-                    let want_t = extract_attributes_tolerant(file, &header, os, ls);
+                let want_t = extract_attributes_tolerant(file, &header, os, ls);
-                    assert_eq!(format!("{got_t:?}"), format!("{want_t:?}"), "{name}");
+                assert_eq!(format!("{got_t:?}"), format!("{want_t:?}"), "{name}");
-                    same += 1;
+                same += 1;
+                for a in want.iter().flatten() {
+                    let one = find_attribute_in(&storage, &header, &a.name, os, ls);
+                    let want_one = find_attribute_in_file(file, &header, &a.name, os, ls);
+                    assert_eq!(format!("{one:?}"), format!("{want_one:?}"), "{name}");
                 }
             }
         }

crates/clawhdf5-format/src/btree_v2.rs

@@ -9,6 +9,7 @@ use byteorder::{ByteOrder, LittleEndian};
 
 use crate::addr::to_usize;
 use crate::error::FormatError;
+use crate::storage::{Storage, Window, len_usize};
 
 /// Parsed B-tree v2 header (signature "BTHD").
 #[derive(Debug, Clone)]
@@ -99,38 +100,52 @@ impl BTreeV2Header {
         offset_size: u8,
         length_size: u8,
     ) -> Result<BTreeV2Header, FormatError> {
-        ensure_len(file_data, offset, 4)?;
+        Self::parse_in(file_data, offset as u64, offset_size, length_size)
-        if &file_data[offset..offset + 4] != b"BTHD" {
+    }
+
+    /// [`Self::parse`] over any [`Storage`]: one bounded read of the
+    /// header.
+    pub fn parse_in<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        offset_size: u8,
+        length_size: u8,
+    ) -> Result<BTreeV2Header, FormatError> {
+        // Every field and the checksum; the window holds all of it or ends
+        // at the end of the file, so its bounds checks are the whole-file
+        // ones.
+        let full = 16 + usize::from(offset_size) + 2 + usize::from(length_size) + 4;
+        let w = Window::read(file, offset, full)?;
+        let d = &w.bytes;
+        w.ensure(0, 4)?;
+        if &d[..4] != b"BTHD" {
             return Err(FormatError::InvalidBTreeV2Signature);
         }
 
-        ensure_len(file_data, offset, 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1)?;
+        w.ensure(0, 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1)?;
-        let version = file_data[offset + 4];
+        let version = d[4];
         if version != 0 {
             return Err(FormatError::InvalidBTreeV2Version(version));
         }
 
-        let tree_type = file_data[offset + 5];
+        let tree_type = d[5];
-        let node_size = u32::from_le_bytes([
+        let node_size = u32::from_le_bytes([d[6], d[7], d[8], d[9]]);
-            file_data[offset + 6],
+        let record_size = u16::from_le_bytes([d[10], d[11]]);
-            file_data[offset + 7],
+        let depth = u16::from_le_bytes([d[12], d[13]]);
-            file_data[offset + 8],
+        let _split_percent = d[14];
-            file_data[offset + 9],
+        let _merge_percent = d[15];
-        ]);
-        let record_size = u16::from_le_bytes([file_data[offset + 10], file_data[offset + 11]]);
-        let depth = u16::from_le_bytes([file_data[offset + 12], file_data[offset + 13]]);
-        let _split_percent = file_data[offset + 14];
-        let _merge_percent = file_data[offset + 15];
 
-        let mut pos = offset + 16;
+        let mut pos = 16;
-        let root_node_address = read_offset(file_data, pos, offset_size)?;
+        w.ensure(pos, usize::from(offset_size))?;
+        let root_node_address = read_offset(d, pos, offset_size)?;
         pos += offset_size as usize;
 
-        ensure_len(file_data, pos, 2)?;
+        w.ensure(pos, 2)?;
-        let num_records_in_root = u16::from_le_bytes([file_data[pos], file_data[pos + 1]]);
+        let num_records_in_root = u16::from_le_bytes([d[pos], d[pos + 1]]);
         pos += 2;
 
-        let total_records = read_offset(file_data, pos, length_size)?;
+        w.ensure(pos, usize::from(length_size))?;
+        let total_records = read_offset(d, pos, length_size)?;
         #[allow(unused_assignments)]
         {
             pos += length_size as usize;
@@ -139,9 +154,9 @@ impl BTreeV2Header {
         // Validate header checksum
         #[cfg(feature = "checksum")]
         {
-            ensure_len(file_data, pos, 4)?;
+            w.ensure(pos, 4)?;
-            let stored = LittleEndian::read_u32(&file_data[pos..pos + 4]);
+            let stored = LittleEndian::read_u32(&d[pos..pos + 4]);
-            let computed = crate::checksum::jenkins_lookup3(&file_data[offset..pos]);
+            let computed = crate::checksum::jenkins_lookup3(&d[..pos]);
             if computed != stored {
                 return Err(FormatError::ChecksumMismatch {
                     expected: stored,
@@ -191,6 +206,17 @@ pub fn collect_btree_v2_records(
     header: &BTreeV2Header,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<BTreeV2Record>, FormatError> {
+    collect_btree_v2_records_in(file_data, header, offset_size, length_size)
+}
+
+/// [`collect_btree_v2_records`] over any [`Storage`]: one bounded read per
+/// node.
+pub fn collect_btree_v2_records_in<S: Storage + ?Sized>(
+    file: &S,
+    header: &BTreeV2Header,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<BTreeV2Record>, FormatError> {
     if header.total_records == 0 || header.num_records_in_root == 0 {
         return Ok(Vec::new());
@@ -210,23 +236,24 @@ pub fn collect_btree_v2_records(
     // millions of records from a few kilobytes. Counting against what the
     // file could physically contain bounds that without trusting the
     // header's own `total_records`.
-    let mut budget = file_data.len() / usize::from(header.record_size.max(1));
+    let mut budget = len_usize(file) / usize::from(header.record_size.max(1));
 
     let max_leaf_nrec = max_records_leaf(header.node_size, header.record_size);
 
     if header.depth == 0 {
         // Root is a leaf
         parse_leaf_records(
-            file_data,
+            file,
             to_usize(header.root_node_address)?,
             header.num_records_in_root,
             header.record_size,
+            header.node_size,
         )
     } else {
         // Root is internal; traverse recursively
         let mut records = Vec::new();
         collect_internal_records(
-            file_data,
+            file,
             to_usize(header.root_node_address)?,
             header.num_records_in_root,
             header.depth,
@@ -242,36 +269,72 @@ pub fn collect_btree_v2_records(
     }
 }
 
+/// A node's bytes: `want` bytes at `offset` (fewer only at the end of the
+/// file), after checking its 4-byte signature. A node is read in one piece
+/// when it fits in `node_size` (every valid node does); a larger claimed
+/// extent — record counts from a damaged parent — is first checked against
+/// the end of the file, so it costs a read only of bytes the file has.
+/// Bounds errors are the whole-file ones: the signature check needs the
+/// first 6 bytes, then `checks` — `(position, length)` pairs relative to
+/// the node, in the order the parser checks them — must lie in the file.
+fn read_node<'a, S: Storage + ?Sized>(
+    file: &'a S,
+    offset: usize,
+    want: usize,
+    node_size: u32,
+    signature: &[u8; 4],
+    checks: &[(usize, usize)],
+) -> Result<Window<'a>, FormatError> {
+    let one_read = usize::try_from(node_size).unwrap_or(usize::MAX).max(6);
+    let w = Window::read(file, offset as u64, want.min(one_read))?;
+    w.ensure(0, 6)?;
+    if &w.bytes[..4] != signature {
+        return Err(FormatError::InvalidBTreeV2Signature);
+    }
+    if want <= one_read {
+        return Ok(w);
+    }
+    for &(rel, len) in checks {
+        Window::check_extent(file, offset as u64, rel, len)?;
+    }
+    Window::read(file, offset as u64, want)
+}
+
 /// Parse records from a leaf node (signature "BTLF").
-fn parse_leaf_records(
+fn parse_leaf_records<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file: &S,
     offset: usize,
     num_records: u16,
     record_size: u16,
+    node_size: u32,
 ) -> Result<Vec<BTreeV2Record>, FormatError> {
     // signature(4) + version(1) + type(1) = 6 bytes header
-    ensure_len(file_data, offset, 6)?;
+    let pos = 6;
-    if &file_data[offset..offset + 4] != b"BTLF" {
-        return Err(FormatError::InvalidBTreeV2Signature);
-    }
-
-    let pos = offset + 6;
     let rs = record_size as usize;
     let total = (num_records as usize)
         .checked_mul(rs)
         .ok_or(FormatError::UnexpectedEof {
             expected: usize::MAX,
-            available: file_data.len(),
+            available: len_usize(file),
         })?;
-    ensure_len(file_data, pos, total)?;
+    let w = read_node(
+        file,
+        offset,
+        pos + total + 4,
+        node_size,
+        b"BTLF",
+        &[(pos, total)],
+    )?;
+    let d = &w.bytes;
+    w.ensure(pos, total)?;
 
     // Validate checksum: 4 bytes after records + padding
     #[cfg(feature = "checksum")]
     {
         let checksum_pos = pos + total;
-        if file_data.len() >= checksum_pos + 4 {
+        if d.len() >= checksum_pos + 4 {
-            let stored = LittleEndian::read_u32(&file_data[checksum_pos..checksum_pos + 4]);
+            let stored = LittleEndian::read_u32(&d[checksum_pos..checksum_pos + 4]);
-            let computed = crate::checksum::jenkins_lookup3(&file_data[offset..checksum_pos]);
+            let computed = crate::checksum::jenkins_lookup3(&d[..checksum_pos]);
             if computed != stored {
                 return Err(FormatError::ChecksumMismatch {
                     expected: stored,
@@ -285,17 +348,41 @@ fn parse_leaf_records(
     for i in 0..num_records as usize {
         let start = pos + i * rs;
         records.push(BTreeV2Record {
-            data: file_data[start..start + rs].to_vec(),
+            data: d[start..start + rs].to_vec(),
         });
     }
     Ok(records)
 }
 
+/// An internal node read from the file: its bytes (from the signature on),
+/// where its records start, and its children as `(address, record count)`.
+struct InternalNode<'a> {
+    node: Window<'a>,
+    records_start: usize,
+    children: Vec<(u64, u16)>,
+}
+
+impl InternalNode<'_> {
+    /// Record `i`, `rs` bytes long.
+    fn record(&self, i: usize, rs: usize) -> Result<&[u8], FormatError> {
+        let overflow = || FormatError::UnexpectedEof {
+            expected: usize::MAX,
+            available: usize::MAX,
+        };
+        let rec_start = i
+            .checked_mul(rs)
+            .and_then(|o| self.records_start.checked_add(o))
+            .ok_or_else(overflow)?;
+        self.node.ensure(rec_start, rs)?;
+        Ok(&self.node.bytes[rec_start..rec_start + rs])
+    }
+}
+
 /// An internal node's layout: where its records start, and its children as
 /// `(address, record count)`.
 #[allow(clippy::too_many_arguments)]
-fn read_internal_node(
+fn read_internal_node<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file: &S,
     offset: usize,
     num_records: u16,
     depth: u16,
@@ -303,25 +390,15 @@ fn read_internal_node(
     node_size: u32,
     offset_size: u8,
     max_leaf_nrec: u64,
-) -> Result<(usize, Vec<(u64, u16)>), FormatError> {
+) -> Result<InternalNode<'_>, FormatError> {
-    // signature(4) + version(1) + type(1) = 6
-    ensure_len(file_data, offset, 6)?;
-    if &file_data[offset..offset + 4] != b"BTIN" {
-        return Err(FormatError::InvalidBTreeV2Signature);
-    }
-
     let nr = num_records as usize;
     let rs = record_size as usize;
-    let mut pos = offset + 6;
 
     // Records first
     let records_total = nr.checked_mul(rs).ok_or(FormatError::UnexpectedEof {
         expected: usize::MAX,
-        available: file_data.len(),
+        available: len_usize(file),
     })?;
-    ensure_len(file_data, pos, records_total)?;
-    let records_start = pos;
-    pos += records_total;
 
     // Child pointer layout, as libhdf5 computes it (H5B2__hdr_init): the
     // child's record count is always encoded in the width needed for a
@@ -344,13 +421,30 @@ fn read_internal_node(
 
     let num_children = nr + 1;
     let child_ptr_size = offset_size as usize + nrec_width + total_nrec_width;
-    ensure_len(file_data, pos, num_children * child_ptr_size)?;
+    let pointers = num_children * child_ptr_size;
+
+    // signature(4) + version(1) + type(1) = 6, records, pointers, checksum.
+    let w = read_node(
+        file,
+        offset,
+        6 + records_total + pointers + 4,
+        node_size,
+        b"BTIN",
+        &[(6, records_total), (6 + records_total, pointers)],
+    )?;
+    let d = &w.bytes;
+    let mut pos = 6;
+    w.ensure(pos, records_total)?;
+    let records_start = pos;
+    pos += records_total;
+
+    w.ensure(pos, pointers)?;
 
     let mut children = Vec::with_capacity(num_children);
     for _ in 0..num_children {
-        let addr = read_offset(file_data, pos, offset_size)?;
+        let addr = read_offset(d, pos, offset_size)?;
         pos += offset_size as usize;
-        let child_nrec = read_var_uint(file_data, pos, nrec_width)? as u16;
+        let child_nrec = read_var_uint(d, pos, nrec_width)? as u16;
         pos += nrec_width;
         pos += total_nrec_width; // skip total records in subtree
         children.push((addr, child_nrec));
@@ -362,9 +456,9 @@ fn read_internal_node(
     // a mismatch here, and so does this.
     #[cfg(feature = "checksum")]
     {
-        ensure_len(file_data, pos, 4)?;
+        w.ensure(pos, 4)?;
-        let stored = LittleEndian::read_u32(&file_data[pos..pos + 4]);
+        let stored = LittleEndian::read_u32(&d[pos..pos + 4]);
-        let computed = crate::checksum::jenkins_lookup3(&file_data[offset..pos]);
+        let computed = crate::checksum::jenkins_lookup3(&d[..pos]);
         if computed != stored {
             return Err(FormatError::ChecksumMismatch {
                 expected: stored,
@@ -372,37 +466,17 @@ fn read_internal_node(
             });
         }
     }
-    Ok((records_start, children))
+    Ok(InternalNode {
-}
+        node: w,
-
+        records_start,
-/// Record `i` of an internal node whose records start at `records_start`.
+        children,
-fn internal_record(
+    })
-    file_data: &[u8],
-    records_start: usize,
-    i: usize,
-    rs: usize,
-) -> Result<&[u8], FormatError> {
-    let overflow = || FormatError::UnexpectedEof {
-        expected: usize::MAX,
-        available: file_data.len(),
-    };
-    let rec_start = i
-        .checked_mul(rs)
-        .and_then(|o| records_start.checked_add(o))
-        .ok_or_else(overflow)?;
-    let rec_end = rec_start.checked_add(rs).ok_or_else(overflow)?;
-    file_data
-        .get(rec_start..rec_end)
-        .ok_or(FormatError::UnexpectedEof {
-            expected: rec_end,
-            available: file_data.len(),
-        })
 }
 
 /// Recursively collect records from an internal node.
 #[allow(clippy::too_many_arguments, clippy::only_used_in_recursion)]
-fn collect_internal_records(
+fn collect_internal_records<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file: &S,
     offset: usize,
     num_records: u16,
     depth: u16,
@@ -416,8 +490,8 @@ fn collect_internal_records(
 ) -> Result<(), FormatError> {
     let nr = num_records as usize;
     let rs = record_size as usize;
-    let (records_start, children) = read_internal_node(
+    let node = read_internal_node(
-        file_data,
+        file,
         offset,
         num_records,
         depth,
@@ -430,16 +504,21 @@ fn collect_internal_records(
 
     // Interleave: child[0], record[0], child[1], record[1], ..., child[nr]
     // We collect child[0] records, then record[0], then child[1], etc.
-    for (i, &(child_addr, child_nrec)) in children.iter().enumerate() {
+    for (i, &(child_addr, child_nrec)) in node.children.iter().enumerate() {
         if child_depth == 0 {
             // Before parsing, so a refused tree is not also a large allocation.
             spend(budget, usize::from(child_nrec))?;
-            let leaf_recs =
+            let leaf_recs = parse_leaf_records(
-                parse_leaf_records(file_data, to_usize(child_addr)?, child_nrec, record_size)?;
+                file,
+                to_usize(child_addr)?,
+                child_nrec,
+                record_size,
+                node_size,
+            )?;
             out.extend(leaf_recs);
         } else {
             collect_internal_records(
-                file_data,
+                file,
                 to_usize(child_addr)?,
                 child_nrec,
                 child_depth,
@@ -455,7 +534,7 @@ fn collect_internal_records(
 
         // Add record[i] (except after the last child)
         if i < nr {
-            let data = internal_record(file_data, records_start, i, rs)?;
+            let data = node.record(i, rs)?;
             spend(budget, 1)?;
             out.push(BTreeV2Record {
                 data: data.to_vec(),
@@ -481,6 +560,17 @@ pub fn find_btree_v2_records(
     header: &BTreeV2Header,
     offset_size: u8,
     cmp: &mut dyn FnMut(&[u8]) -> Ordering,
+) -> Result<Vec<BTreeV2Record>, FormatError> {
+    find_btree_v2_records_in(file_data, header, offset_size, cmp)
+}
+
+/// [`find_btree_v2_records`] over any [`Storage`]: one bounded read per
+/// node visited.
+pub fn find_btree_v2_records_in<S: Storage + ?Sized>(
+    file: &S,
+    header: &BTreeV2Header,
+    offset_size: u8,
+    cmp: &mut dyn FnMut(&[u8]) -> Ordering,
 ) -> Result<Vec<BTreeV2Record>, FormatError> {
     if header.total_records == 0 || header.num_records_in_root == 0 {
         return Ok(Vec::new());
@@ -490,11 +580,11 @@ pub fn find_btree_v2_records(
     }
     // As in `collect_btree_v2_records`: a valid tree cannot hold more
     // records than the file has room for, however its children are shared.
-    let mut budget = file_data.len() / usize::from(header.record_size.max(1));
+    let mut budget = len_usize(file) / usize::from(header.record_size.max(1));
     let max_leaf_nrec = max_records_leaf(header.node_size, header.record_size);
     let mut out = Vec::new();
     find_in_node(
-        file_data,
+        file,
         header,
         to_usize(header.root_node_address)?,
         header.num_records_in_root,
@@ -509,8 +599,8 @@ pub fn find_btree_v2_records(
 }
 
 #[allow(clippy::too_many_arguments)]
-fn find_in_node(
+fn find_in_node<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file: &S,
     header: &BTreeV2Header,
     offset: usize,
     num_records: u16,
@@ -523,7 +613,13 @@ fn find_in_node(
 ) -> Result<(), FormatError> {
     spend(budget, usize::from(num_records))?;
     if depth == 0 {
-        let records = parse_leaf_records(file_data, offset, num_records, header.record_size)?;
+        let records = parse_leaf_records(
+            file,
+            offset,
+            num_records,
+            header.record_size,
+            header.node_size,
+        )?;
         out.extend(
             records
                 .into_iter()
@@ -532,8 +628,8 @@ fn find_in_node(
         return Ok(());
     }
     let rs = usize::from(header.record_size);
-    let (records_start, children) = read_internal_node(
+    let node = read_internal_node(
-        file_data,
+        file,
         offset,
         num_records,
         depth,
@@ -545,17 +641,17 @@ fn find_in_node(
     let nr = usize::from(num_records);
     let mut order = Vec::with_capacity(nr);
     for i in 0..nr {
-        order.push(cmp(internal_record(file_data, records_start, i, rs)?));
+        order.push(cmp(node.record(i, rs)?));
     }
     // Child `i` holds the keys between record `i - 1` and record `i`: it can
     // hold a match unless the record before it is already past the range or
     // the record after it is still before it.
-    for (i, &(child_addr, child_nrec)) in children.iter().enumerate() {
+    for (i, &(child_addr, child_nrec)) in node.children.iter().enumerate() {
         let after_left = i == 0 || order[i - 1] != Ordering::Greater;
         let before_right = i == nr || order[i] != Ordering::Less;
         if after_left && before_right {
             find_in_node(
-                file_data,
+                file,
                 header,
                 to_usize(child_addr)?,
                 child_nrec,
@@ -569,7 +665,7 @@ fn find_in_node(
         }
         if i < nr && order[i] == Ordering::Equal {
             out.push(BTreeV2Record {
-                data: internal_record(file_data, records_start, i, rs)?.to_vec(),
+                data: node.record(i, rs)?.to_vec(),
             });
         }
     }

crates/clawhdf5-format/src/btree_v2_write.rs

@@ -441,6 +441,57 @@ mod tests {
         }
     }
 
+    /// A two-level tree read through a `read_at`-only storage gives what
+    /// the slice gives — records, descents and errors — whole, truncated
+    /// at every length, and with each byte of its nodes flipped, and each
+    /// node costs one read.
+    #[test]
+    fn storage_reads_match_slice_reads() {
+        use crate::btree_v2::{
+            collect_btree_v2_records_in, find_btree_v2_records, find_btree_v2_records_in,
+        };
+        use crate::storage::CountingStorage;
+        let (rs, n, base) = (11usize, 120usize, 64usize);
+        let recs = records(n, rs);
+        let tree = build_btree_v2(params(128, 11), &recs, base as u64, 8, 8).unwrap();
+        let mut whole = vec![0u8; base];
+        whole.extend_from_slice(&tree);
+        let hdr = BTreeV2Header::parse(&whole, base, 8, 8).unwrap();
+        assert!(hdr.depth >= 1, "{hdr:?}");
+        let key = |r: &[u8]| u64::from_be_bytes(r[..8].try_into().unwrap());
+        let mut files = Vec::new();
+        for cut in base..=whole.len() {
+            files.push(whole[..cut].to_vec());
+        }
+        for at in base..whole.len() {
+            let mut bad = whole.clone();
+            bad[at] ^= 0x5a;
+            files.push(bad);
+        }
+        let mut ok = 0;
+        for f in &files {
+            let st = CountingStorage::new(f.clone());
+            let want_h = BTreeV2Header::parse(f, base, 8, 8);
+            let got_h = BTreeV2Header::parse_in(&st, base as u64, 8, 8);
+            assert_eq!(format!("{got_h:?}"), format!("{want_h:?}"));
+            // The nodes of the intact header, over each damaged file.
+            let want = collect_btree_v2_records(f, &hdr, 8, 8);
+            st.reset();
+            let got = collect_btree_v2_records_in(&st, &hdr, 8, 8);
+            assert_eq!(format!("{got:?}"), format!("{want:?}"));
+            if want.is_ok() {
+                ok += 1;
+                assert!(st.reads() <= 1 + n as u64 / 3, "{} reads", st.reads());
+            }
+            for k in [0u64, 7, 60, 119, 500] {
+                let want = find_btree_v2_records(f, &hdr, 8, &mut |r: &[u8]| key(r).cmp(&k));
+                let got = find_btree_v2_records_in(&st, &hdr, 8, &mut |r: &[u8]| key(r).cmp(&k));
+                assert_eq!(format!("{got:?}"), format!("{want:?}"));
+            }
+        }
+        assert!(ok > 1);
+    }
+
     #[test]
     fn a_node_too_small_or_too_big_is_an_error() {
         assert!(build_btree_v2(params(16, 11), &records(1, 11), 0, 8, 8).is_err());

crates/clawhdf5-format/src/checksum.rs

@@ -14,6 +14,45 @@ pub fn jenkins_lookup3(data: &[u8]) -> u32 {
     hashlittle(data, 0)
 }
 
+/// HDF5's Fletcher-32 checksum, as the Fletcher-32 I/O filter (filter id 3)
+/// stores it after each chunk.
+///
+/// A line-for-line port of `H5_checksum_fletcher32` (H5checksum.c, libhdf5
+/// 1.8 through 1.14): big-endian 16-bit words summed in blocks of 360, each
+/// sum reduced after a block by the ones'-complement fold
+/// `(s & 0xffff) + (s >> 16)` rather than `% 65535`, an odd trailing byte
+/// taken as the high byte of a last word, and a final fold of both sums.
+/// The fold and `% 65535` differ whenever a sum is a non-zero multiple of
+/// 65535: the fold leaves 0xffff where the modulo gives 0, so the two
+/// disagree on about one chunk in 32768 and libhdf5 rejects the other's
+/// checksum. This must stay the only implementation.
+pub fn fletcher32(data: &[u8]) -> u32 {
+    let mut sum1: u32 = 0;
+    let mut sum2: u32 = 0;
+    // 360 words keep both sums inside 32 bits between folds (the bound
+    // libhdf5 uses: after a fold sum1 < 0x10200, so sum2 stays below
+    // 360 * 361 / 2 * 0xffff + 360 * 0x10200 + 0x1fffe < 2^32). The adds wrap
+    // like the C unsigned arithmetic all the same.
+    let (words, odd) = data.as_chunks::<2>();
+    for block in words.chunks(360) {
+        for w in block {
+            sum1 = sum1.wrapping_add((u32::from(w[0]) << 8) | u32::from(w[1]));
+            sum2 = sum2.wrapping_add(sum1);
+        }
+        sum1 = (sum1 & 0xffff) + (sum1 >> 16);
+        sum2 = (sum2 & 0xffff) + (sum2 >> 16);
+    }
+    if let [last] = odd {
+        sum1 = sum1.wrapping_add(u32::from(*last) << 8);
+        sum2 = sum2.wrapping_add(sum1);
+        sum1 = (sum1 & 0xffff) + (sum1 >> 16);
+        sum2 = (sum2 & 0xffff) + (sum2 >> 16);
+    }
+    sum1 = (sum1 & 0xffff) + (sum1 >> 16);
+    sum2 = (sum2 & 0xffff) + (sum2 >> 16);
+    (sum2 << 16) | sum1
+}
+
 /// Compute CRC32 (IEEE / ISO 3309) over data.
 ///
 /// When the `fast-checksum` feature is enabled, this uses hardware CRC32
@@ -207,6 +246,19 @@ fn hashlittle(data: &[u8], initval: u32) -> u32 {
 mod tests {
     use super::*;
 
+    /// Values of libhdf5's `H5_checksum_fletcher32` (h5py 3.x's bundled
+    /// libhdf5, called through ctypes). The first three are sums that are
+    /// multiples of 65535, where `% 65535` gave 0 instead of 0xffff.
+    #[test]
+    fn fletcher32_matches_libhdf5() {
+        assert_eq!(fletcher32(&[0x00, 0x01, 0xff, 0xfe]), 0x0001_ffff);
+        assert_eq!(fletcher32(&[0xff; 720]), 0xffff_ffff);
+        assert_eq!(fletcher32(&[0xff; 721]), 0xff00_ff00);
+        assert_eq!(fletcher32(&[0xff; 1441]), 0xff00_ff00);
+        assert_eq!(fletcher32(&[]), 0);
+        assert_eq!(fletcher32(&[7]), 0x0700_0700);
+    }
+
     #[test]
     fn empty_input() {
         // Empty input should return the initial state after no mixing

crates/clawhdf5-format/src/chunked_read.rs

@@ -6,19 +6,20 @@ extern crate alloc;
 #[cfg(not(feature = "std"))]
 use alloc::{format, vec, vec::Vec};
 
-use crate::addr::to_usize;
+use crate::addr::{checked_addr, to_usize};
 #[cfg(feature = "std")]
 use crate::chunk_cache::{CacheAlignedBuffer, ChunkCache};
 use crate::data_layout::DataLayout;
 use crate::dataspace::Dataspace;
 use crate::datatype::Datatype;
 use crate::error::FormatError;
-use crate::extensible_array::{ExtensibleArrayHeader, read_extensible_array_chunks};
+use crate::extensible_array::{ExtensibleArrayHeader, read_extensible_array_chunks_in};
 use crate::filter_pipeline::FilterPipeline;
 use crate::filters::{DecodeScratch, decompress_chunk_exact_with};
 #[cfg(feature = "std")]
 use crate::filters::{all_filters_skipped, decompress_chunk_exact};
-use crate::fixed_array::{FixedArrayHeader, read_fixed_array_chunks};
+use crate::fixed_array::{FixedArrayHeader, read_fixed_array_chunks_in};
+use crate::storage::{ExtentBytes, ExtentReq, Storage, Window, for_each_extent_batch, read_extent};
 #[cfg(feature = "std")]
 use std::sync::Arc;
 
@@ -238,19 +239,128 @@ type CacheUse<'a> = Option<&'a core::convert::Infallible>;
 /// ([`parallel_read::run_with_helpers`]), so the caller never waits on a
 /// busy pool. The error returned is the first failing chunk's, in `chunks`
 /// order.
+///
+/// With the whole file in memory the chunks are sliced from it. Otherwise
+/// their stored bytes are fetched first, with one
+/// [`Storage::read_ranges`] call per batch of up to
+/// [`crate::storage::RAW_BATCH_BYTES`] (all of them, for most datasets),
+/// and then decoded as above; chunks the cache already holds are not
+/// fetched.
 #[allow(clippy::too_many_arguments)]
-fn fill_from_chunks(
+fn fill_from_chunks<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     chunks: &[ChunkInfo],
     pipeline: Option<&FilterPipeline>,
     placer: &ChunkPlacer,
     chunk_total_bytes: usize,
     cache: CacheUse<'_>,
     output: &mut [u8],
+) -> Result<(), FormatError> {
+    let out = OutBuf::new(output);
+    #[cfg(not(feature = "std"))]
+    let _ = cache;
+    #[cfg(feature = "std")]
+    let rank = placer.rank;
+
+    // Over a backend without the whole file in memory, the decoded chunks
+    // the cache holds are taken now (so a chunk evicted before it is placed
+    // is not left without bytes), and only the others are fetched.
+    #[cfg(feature = "std")]
+    let hits: Vec<CacheHit> = match cache {
+        Some((cache, key, true)) if file_data.as_contiguous().is_none() => chunks
+            .iter()
+            .map(|c| {
+                if c.offsets.len() >= rank && uses_cache(Some((cache, key, true)), pipeline, c) {
+                    cache.get_decompressed_in(key, &c.offsets[..rank])
+                } else {
+                    None
+                }
+            })
+            .collect(),
+        _ => Vec::new(),
+    };
+    #[cfg(not(feature = "std"))]
+    let hits: Vec<CacheHit> = Vec::new();
+    let reqs: Vec<ExtentReq> = chunks
+        .iter()
+        .enumerate()
+        .map(|(i, c)| {
+            let wanted = hits.get(i).is_none_or(Option::is_none);
+            chunk_req(c, pipeline, chunk_total_bytes, wanted)
+        })
+        .collect();
+    for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
+        fill_batch(
+            &chunks[batch.clone()],
+            &reqs[batch.clone()],
+            hits.get(batch.clone()).unwrap_or_default(),
+            batch.start,
+            raw_bytes,
+            pipeline,
+            placer,
+            chunk_total_bytes,
+            cache,
+            &out,
+        )
+    })
+}
+
+/// The extent of chunk `c`'s stored bytes to fetch (`wanted`) or only
+/// bounds-check: its whole stored size is checked against the file, and at
+/// most [`crate::filters::stored_chunk_limit`] of it is read and decoded.
+pub(crate) fn chunk_req(
+    c: &ChunkInfo,
+    pipeline: Option<&FilterPipeline>,
+    chunk_bytes: usize,
+    wanted: bool,
+) -> ExtentReq {
+    let len = c.chunk_size as usize;
+    ExtentReq {
+        addr: c.address,
+        len,
+        fetch: wanted.then(|| {
+            len.min(crate::filters::stored_chunk_limit(
+                pipeline,
+                c.filter_mask,
+                chunk_bytes,
+            ))
+        }),
+    }
+}
+
+/// A decoded chunk a full read took from the cache.
+#[cfg(feature = "std")]
+type CacheHit = Option<Arc<CacheAlignedBuffer>>;
+#[cfg(not(feature = "std"))]
+type CacheHit = Option<core::convert::Infallible>;
+
+/// Whether a full read looks chunk `c` up in the cache (see
+/// [`fill_from_chunks`]): a filtered chunk of a dataset the cache keeps.
+#[cfg(feature = "std")]
+fn uses_cache(cache: CacheUse<'_>, pipeline: Option<&FilterPipeline>, c: &ChunkInfo) -> bool {
+    matches!(cache, Some((_, _, true)))
+        && pipeline.is_some_and(|pl| !all_filters_skipped(pl, c.filter_mask))
+}
+
+/// [`fill_from_chunks`] for one batch of chunks: `chunks` (with their
+/// extents `reqs` and, over a backend without the file in memory, the cache
+/// hits taken for them) are chunks `first..` of the read, whose stored bytes
+/// `raw_bytes` holds.
+#[allow(clippy::too_many_arguments)]
+fn fill_batch(
+    chunks: &[ChunkInfo],
+    reqs: &[ExtentReq],
+    hits: &[CacheHit],
+    first: usize,
+    raw_bytes: &ExtentBytes<'_>,
+    pipeline: Option<&FilterPipeline>,
+    placer: &ChunkPlacer,
+    chunk_total_bytes: usize,
+    cache: CacheUse<'_>,
+    out: &OutBuf<'_>,
 ) -> Result<(), FormatError> {
     let rank = placer.rank;
     let elem_size = placer.elem_size as u32;
-    let out = OutBuf::new(output);
     #[cfg(not(feature = "std"))]
     let _ = cache;
 
@@ -266,13 +376,17 @@ fn fill_from_chunks(
             )));
         }
         let offsets = &c.offsets[..rank];
-        let c_addr = to_usize(c.address)?;
+        #[cfg(feature = "std")]
-        let size = c.chunk_size as usize;
+        if let Some(Some(hit)) = hits.get(i) {
-        ensure_len(file_data, c_addr, size)?;
+            raw_bytes.check(first + i, &reqs[i])?;
-        let raw = &file_data[c_addr..c_addr + size];
+            // SAFETY: see above.
+            unsafe { placer.place(hit, offsets, out) };
+            return Ok(());
+        }
+        let raw = raw_bytes.get(first + i, &reqs[i])?;
         let Some(pl) = pipeline else {
             // SAFETY: see above.
-            unsafe { placer.place(raw, offsets, &out) };
+            unsafe { placer.place(raw, offsets, out) };
             return Ok(());
         };
         // A chunk stored as-is (every filter skipped) is checked and placed
@@ -296,7 +410,7 @@ fn fill_from_chunks(
                 }
             };
             // SAFETY: see above.
-            unsafe { placer.place(&cached, offsets, &out) };
+            unsafe { placer.place(&cached, offsets, out) };
             return Ok(());
         }
         let data = decompress_chunk_exact_with(
@@ -309,7 +423,7 @@ fn fill_from_chunks(
             scratch,
         )?;
         // SAFETY: see above.
-        unsafe { placer.place(data, offsets, &out) };
+        unsafe { placer.place(data, offsets, out) };
         Ok(())
     };
 
@@ -370,7 +484,29 @@ pub fn decompress_all_chunks_with_stats(
     seed: u64,
     num_lanes: Option<usize>,
 ) -> Result<(Vec<Vec<u8>>, PartitionStats), FormatError> {
-    parallel_read::decompress_chunks_lane_partitioned(
+    decompress_all_chunks_with_stats_in(
+        file_data,
+        chunks,
+        pipeline,
+        chunk_total_bytes,
+        element_size,
+        seed,
+        num_lanes,
+    )
+}
+
+/// [`decompress_all_chunks_with_stats`] over any [`Storage`].
+#[cfg(feature = "parallel")]
+pub fn decompress_all_chunks_with_stats_in<S: Storage + ?Sized>(
+    file_data: &S,
+    chunks: &[ChunkInfo],
+    pipeline: &FilterPipeline,
+    chunk_total_bytes: usize,
+    element_size: u32,
+    seed: u64,
+    num_lanes: Option<usize>,
+) -> Result<(Vec<Vec<u8>>, PartitionStats), FormatError> {
+    parallel_read::decompress_chunks_lane_partitioned_in(
         file_data,
         chunks,
         pipeline,
@@ -394,21 +530,6 @@ pub struct ChunkInfo {
     pub address: u64,
 }
 
-/// Checks that `[offset, offset + needed)` fits within `data`, guarding the
-/// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
-fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
-    if offset
-        .checked_add(needed)
-        .is_none_or(|end| end > data.len())
-    {
-        return Err(FormatError::UnexpectedEof {
-            expected: offset.saturating_add(needed),
-            available: data.len(),
-        });
-    }
-    Ok(())
-}
-
 /// `elements * elem_size` for sizes that come from the file. Dataspace and
 /// chunk dimensions are untrusted 64-bit fields, so a crafted file can make
 /// the plain product wrap to a small number (or to something enormous).
@@ -588,6 +709,17 @@ pub fn collect_chunk_info(
     ndims: usize,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<ChunkInfo>, FormatError> {
+    collect_chunk_info_in(file_data, btree_address, ndims, offset_size, length_size)
+}
+
+/// [`collect_chunk_info`] over any [`Storage`].
+pub fn collect_chunk_info_in<S: Storage + ?Sized>(
+    file_data: &S,
+    btree_address: u64,
+    ndims: usize,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<ChunkInfo>, FormatError> {
     let _ = length_size;
     let mut chunks = Vec::new();
@@ -630,6 +762,23 @@ pub fn collect_chunk_info_checked(
     chunk_dimensions: &[u32],
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<ChunkInfo>, FormatError> {
+    collect_chunk_info_checked_in(
+        file_data,
+        btree_address,
+        chunk_dimensions,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`collect_chunk_info_checked`] over any [`Storage`].
+pub fn collect_chunk_info_checked_in<S: Storage + ?Sized>(
+    file_data: &S,
+    btree_address: u64,
+    chunk_dimensions: &[u32],
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<ChunkInfo>, FormatError> {
     let _ = length_size;
     let ndims = chunk_dimensions.len();
@@ -813,8 +962,8 @@ const MAX_CHUNK_BTREE_DEPTH: usize = 64;
 
 /// Parse the v1 B-tree chunk index node at `btree_address` and its
 /// subtree, appending its chunks to `stored` in tree order.
-fn parse_chunk_node(
+fn parse_chunk_node<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     btree_address: u64,
     ndims: usize,
     chunk_dimensions: Option<&[u32]>,
@@ -831,21 +980,24 @@ fn parse_chunk_node(
 
     // Parse B-tree v1 header
     let header_size = 8 + os * 2;
-    ensure_len(file_data, offset, header_size)?;
+    let head = Window::read(file_data, offset as u64, header_size)?;
+    head.ensure(0, header_size)?;
+    let h = &head.bytes;
 
-    if &file_data[offset..offset + 4] != b"TREE" {
+    if &h[..4] != b"TREE" {
         return Err(FormatError::InvalidBTreeSignature);
     }
 
-    let node_type = file_data[offset + 4];
+    let node_type = h[4];
     if node_type != 1 {
         return Err(FormatError::InvalidBTreeNodeType(node_type));
     }
 
-    let node_level = file_data[offset + 5];
+    let node_level = h[5];
-    let entries_used = u16::from_le_bytes([file_data[offset + 6], file_data[offset + 7]]) as usize;
+    let entries_used = u16::from_le_bytes([h[6], h[7]]) as usize;
 
-    let mut pos = offset + 8 + os * 2; // skip left/right sibling
+    // Positions below are relative to the node's start.
+    let mut pos = header_size; // skip left/right sibling
 
     // Key: chunk_size(4) + filter_mask(4) + one offset per dimension. The
     // offsets are always 8 bytes each — they are dataset coordinates, not file
@@ -858,28 +1010,20 @@ fn parse_chunk_node(
 
     // key[0], child[0], key[1], child[1], ..., key[N-1], child[N-1], key[N]
     let needed = entries_used * (key_size + os) + key_size;
-    ensure_len(file_data, pos, needed)?;
+    let node = Window::read(file_data, offset as u64, header_size + needed)?;
+    node.ensure(pos, needed)?;
+    let d = &node.bytes;
 
     let mut keys = Vec::with_capacity((entries_used + 1) * ndims);
     let mut chunks = Vec::new();
     let mut child_addrs = Vec::new();
     for _ in 0..entries_used {
-        let chunk_size = u32::from_le_bytes([
+        let chunk_size = u32::from_le_bytes([d[pos], d[pos + 1], d[pos + 2], d[pos + 3]]);
-            file_data[pos],
+        let filter_mask = u32::from_le_bytes([d[pos + 4], d[pos + 5], d[pos + 6], d[pos + 7]]);
-            file_data[pos + 1],
-            file_data[pos + 2],
-            file_data[pos + 3],
-        ]);
-        let filter_mask = u32::from_le_bytes([
-            file_data[pos + 4],
-            file_data[pos + 5],
-            file_data[pos + 6],
-            file_data[pos + 7],
-        ]);
         let k = keys.len();
-        read_key_offsets(file_data, pos, ndims, chunk_dimensions, &mut keys)?;
+        read_key_offsets(d, pos, ndims, chunk_dimensions, &mut keys)?;
         pos += key_size;
-        let address = read_offset(file_data, pos, offset_size)?;
+        let address = read_offset(d, pos, offset_size)?;
         pos += os;
         if node_level == 0 {
             chunks.push(stored.len());
@@ -894,7 +1038,7 @@ fn parse_chunk_node(
         }
     }
     // The final key only bounds the node; libhdf5 still checks it.
-    read_key_offsets(file_data, pos, ndims, chunk_dimensions, &mut keys)?;
+    read_key_offsets(d, pos, ndims, chunk_dimensions, &mut keys)?;
 
     let children = if node_level == 0 {
         ChunkChildren::Chunks(chunks)
@@ -933,16 +1077,40 @@ pub fn generate_implicit_chunks(
     dataset_dims: &[u64],
     chunk_dimensions: &[u32],
     element_size: u32,
+) -> Vec<ChunkInfo> {
+    generate_implicit_chunks_in_grid(
+        base_address,
+        dataset_dims,
+        dataset_dims,
+        chunk_dimensions,
+        element_size,
+    )
+}
+
+/// [`generate_implicit_chunks`] for a dataset whose maximum dimensions
+/// (`max_dims`) exceed its current ones: libhdf5 allocates the chunks of
+/// the whole maximum extent and places chunk `scaled` at its row-major
+/// position in the *maximum* chunk grid (`H5D__none_idx_get_addr`,
+/// `max_down_chunks`), so the current extent's chunks are not contiguous.
+/// Only the chunks of the current extent are listed.
+pub fn generate_implicit_chunks_in_grid(
+    base_address: u64,
+    dataset_dims: &[u64],
+    max_dims: &[u64],
+    chunk_dimensions: &[u32],
+    element_size: u32,
 ) -> Vec<ChunkInfo> {
     let rank = chunk_dimensions.len();
     let chunk_byte_size: u64 =
         chunk_dimensions.iter().map(|&d| d as u64).product::<u64>() * element_size as u64;
 
     let mut num_chunks_per_dim = Vec::with_capacity(rank);
+    let mut grid_per_dim = Vec::with_capacity(rank);
     for d in 0..rank {
-        let ds = dataset_dims[d];
         let ch = chunk_dimensions[d] as u64;
-        num_chunks_per_dim.push(ds.div_ceil(ch));
+        let n = dataset_dims[d].div_ceil(ch);
+        num_chunks_per_dim.push(n);
+        grid_per_dim.push(max_dims.get(d).map_or(n, |m| m.div_ceil(ch)).max(n));
     }
     let total_chunks: u64 = num_chunks_per_dim.iter().product();
 
@@ -951,18 +1119,22 @@ pub fn generate_implicit_chunks(
     for linear_idx in 0..total_chunks {
         let mut offsets = vec![0u64; rank];
         let mut remaining = linear_idx;
+        let mut grid_idx = 0u64;
+        let mut down = 1u64;
         for d in (0..rank).rev() {
             let nchunks = num_chunks_per_dim[d];
             let chunk_idx = remaining % nchunks;
             remaining /= nchunks;
             offsets[d] = chunk_idx * chunk_dimensions[d] as u64;
+            grid_idx = grid_idx.saturating_add(chunk_idx.saturating_mul(down));
+            down = down.saturating_mul(grid_per_dim[d]);
         }
 
         chunks.push(ChunkInfo {
             chunk_size: chunk_byte_size as u32,
             filter_mask: 0,
             offsets,
-            address: base_address + linear_idx * chunk_byte_size,
+            address: base_address.saturating_add(grid_idx.saturating_mul(chunk_byte_size)),
         });
     }
 
@@ -984,18 +1156,18 @@ const BT2_CHUNK_FILTERED: u8 = 11;
 /// The width of the stored-size field depends on the largest possible chunk;
 /// rather than re-derive the library's formula it is taken from the record
 /// size the tree header declares, which is what actually governs the bytes.
-fn read_btree_v2_chunks(
+fn read_btree_v2_chunks<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     addr: u64,
     chunk_dims: &[usize],
     elem_size: usize,
     offset_size: u8,
     length_size: u8,
 ) -> Result<Vec<ChunkInfo>, FormatError> {
-    use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+    use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records_in};
 
     let bad = |what: &str| FormatError::ChunkedReadError(format!("B-tree v2 chunk index: {what}"));
-    let header = BTreeV2Header::parse(file_data, to_usize(addr)?, offset_size, length_size)?;
+    let header = BTreeV2Header::parse_in(file_data, checked_addr(addr)?, offset_size, length_size)?;
     let rank = chunk_dims.len();
     let os = offset_size as usize;
     let record_size = header.record_size as usize;
@@ -1022,7 +1194,7 @@ fn read_btree_v2_chunks(
     let unfiltered_bytes =
         u32::try_from(unfiltered_bytes).map_err(|_| bad("chunk larger than 4 GiB"))?;
 
-    let records = collect_btree_v2_records(file_data, &header, offset_size, length_size)?;
+    let records = collect_btree_v2_records_in(file_data, &header, offset_size, length_size)?;
     let mut chunks = Vec::with_capacity(records.len());
     for record in &records {
         let data = record.data.as_slice();
@@ -1085,6 +1257,25 @@ pub fn list_chunks(
     elem_size: usize,
     offset_size: u8,
     length_size: u8,
+) -> Result<(Vec<ChunkInfo>, Vec<usize>), FormatError> {
+    list_chunks_in(
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`list_chunks`] over any [`Storage`].
+pub fn list_chunks_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<(Vec<ChunkInfo>, Vec<usize>), FormatError> {
     let (
         chunk_dimensions,
@@ -1132,9 +1323,13 @@ pub fn list_chunks(
 
     // Collect chunks based on version and index type
     let mut chunks = match (version, chunk_index_type) {
-        (3, _) => {
+        (3, _) => collect_chunk_info_checked_in(
-            collect_chunk_info_checked(file_data, addr, chunk_dimensions, offset_size, length_size)?
+            file_data,
-        }
+            addr,
+            chunk_dimensions,
+            offset_size,
+            length_size,
+        )?,
         (4, Some(1)) => {
             // Single chunk — one chunk covering the entire dataset
             let chunk_byte_size = checked_chunk_byte_len(&chunk_dims, elem_size)?;
@@ -1153,9 +1348,13 @@ pub fn list_chunks(
         (4, Some(2)) => {
             // Implicit index — use spatial chunk dims only
             let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-            generate_implicit_chunks(
+            generate_implicit_chunks_in_grid(
                 addr,
                 &dataspace.dimensions,
+                dataspace
+                    .max_dimensions
+                    .as_deref()
+                    .unwrap_or(&dataspace.dimensions),
                 spatial_chunk_dims,
                 elem_size as u32,
             )
@@ -1163,9 +1362,13 @@ pub fn list_chunks(
         (4, Some(3)) => {
             // Fixed Array — use spatial chunk dims only
             let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-            let header =
+            let header = FixedArrayHeader::parse_in(
-                FixedArrayHeader::parse(file_data, to_usize(addr)?, offset_size, length_size)?;
+                file_data,
-            read_fixed_array_chunks(
+                checked_addr(addr)?,
+                offset_size,
+                length_size,
+            )?;
+            read_fixed_array_chunks_in(
                 file_data,
                 &header,
                 &dataspace.dimensions,
@@ -1179,9 +1382,13 @@ pub fn list_chunks(
         (4, Some(4)) => {
             // Extensible Array — use spatial chunk dims only
             let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-            let header =
+            let header = ExtensibleArrayHeader::parse_in(
-                ExtensibleArrayHeader::parse(file_data, to_usize(addr)?, offset_size, length_size)?;
+                file_data,
-            read_extensible_array_chunks(
+                checked_addr(addr)?,
+                offset_size,
+                length_size,
+            )?;
+            read_extensible_array_chunks_in(
                 file_data,
                 &header,
                 &dataspace.dimensions,
@@ -1248,7 +1455,28 @@ pub fn list_chunks_for_read(
     offset_size: u8,
     length_size: u8,
 ) -> Result<(Vec<ChunkInfo>, Vec<usize>), FormatError> {
-    let (chunks, chunk_dims) = list_chunks(
+    list_chunks_for_read_in(
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        pipeline,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`list_chunks_for_read`] over any [`Storage`].
+pub fn list_chunks_for_read_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<(Vec<ChunkInfo>, Vec<usize>), FormatError> {
+    let (chunks, chunk_dims) = list_chunks_in(
         file_data,
         layout,
         dataspace,
@@ -1284,8 +1512,8 @@ pub(crate) type CacheRef<'a> = Option<&'a core::convert::Infallible>;
 /// output with `alloc` (zeroed, `total_bytes` long, as bytes through
 /// `bytes`), and decode every chunk straight into it.
 #[allow(clippy::too_many_arguments)]
-pub(crate) fn read_chunked_full<O>(
+pub(crate) fn read_chunked_full<O, S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     layout: &DataLayout,
     dataspace: &Dataspace,
     datatype: &Datatype,
@@ -1299,7 +1527,7 @@ pub(crate) fn read_chunked_full<O>(
     check_chunk_element_size(layout, datatype, offset_size)?;
     let elem_size = datatype.type_size() as usize;
     let list = || {
-        list_chunks_for_read(
+        list_chunks_for_read_in(
             file_data,
             layout,
             dataspace,
@@ -1390,6 +1618,27 @@ pub fn read_chunked_data(
     pipeline: Option<&FilterPipeline>,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<u8>, FormatError> {
+    read_chunked_data_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`read_chunked_data`] over any [`Storage`].
+pub fn read_chunked_data_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<u8>, FormatError> {
     read_chunked_full(
         file_data,
@@ -1422,6 +1671,31 @@ pub fn read_chunked_data_cached(
     offset_size: u8,
     length_size: u8,
     cache: &ChunkCache,
+) -> Result<Vec<u8>, FormatError> {
+    read_chunked_data_cached_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+        cache,
+    )
+}
+
+/// [`read_chunked_data_cached`] over any [`Storage`].
+#[cfg(feature = "std")]
+#[allow(clippy::too_many_arguments)]
+pub fn read_chunked_data_cached_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    cache: &ChunkCache,
 ) -> Result<Vec<u8>, FormatError> {
     read_chunked_full(
         file_data,
@@ -1592,6 +1866,33 @@ pub fn read_chunked_data_sweep(
     length_size: u8,
     cache: &ChunkCache,
     sweep: &mut SweepContext,
+) -> Result<Vec<u8>, FormatError> {
+    read_chunked_data_sweep_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+        cache,
+        sweep,
+    )
+}
+
+/// [`read_chunked_data_sweep`] over any [`Storage`].
+#[cfg(feature = "std")]
+#[allow(clippy::too_many_arguments)]
+pub fn read_chunked_data_sweep_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    cache: &ChunkCache,
+    sweep: &mut SweepContext,
 ) -> Result<Vec<u8>, FormatError> {
     let (chunk_dimensions, version, addr_opt) = match layout {
         DataLayout::Chunked {
@@ -1623,7 +1924,7 @@ pub fn read_chunked_data_sweep(
     // lookup is keyed by this dataset's chunk-index address, so another
     // dataset's index or chunks are never used for this read.
     let chunks = cache.chunks_for(addr, rank, || {
-        list_chunks_for_read(
+        list_chunks_for_read_in(
             file_data,
             layout,
             dataspace,
@@ -1673,10 +1974,10 @@ pub fn read_chunked_data_sweep(
             cached
         } else {
             // Decompress from file
-            let c_addr = to_usize(chunk_info.address)?;
+            to_usize(chunk_info.address)?;
-            let size = chunk_info.chunk_size as usize;
+            let req = chunk_req(chunk_info, pipeline, chunk_total_bytes, true);
-            ensure_len(file_data, c_addr, size)?;
+            let raw_chunk = read_extent(file_data, &req)?;
-            let raw_chunk = &file_data[c_addr..c_addr + size];
+            let raw_chunk = &*raw_chunk;
             let dec = if let Some(pl) = pipeline {
                 decompress_chunk_exact(
                     raw_chunk,
@@ -1736,6 +2037,31 @@ pub fn read_chunked_data_indexed(
     offset_size: u8,
     length_size: u8,
     cache: &ChunkCache,
+) -> Result<Vec<u8>, FormatError> {
+    read_chunked_data_indexed_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+        cache,
+    )
+}
+
+/// [`read_chunked_data_indexed`] over any [`Storage`].
+#[cfg(feature = "std")]
+#[allow(clippy::too_many_arguments)]
+pub fn read_chunked_data_indexed_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    cache: &ChunkCache,
 ) -> Result<Vec<u8>, FormatError> {
     let (chunk_dimensions, version, addr_opt) = match layout {
         DataLayout::Chunked {
@@ -1769,7 +2095,7 @@ pub fn read_chunked_data_indexed(
         addr,
         rank,
         || {
-            list_chunks_for_read(
+            list_chunks_for_read_in(
                 file_data,
                 layout,
                 dataspace,
@@ -1786,35 +2112,61 @@ pub fn read_chunked_data_indexed(
     )?;
     let chunk_total_bytes = plan.chunk_total_bytes;
 
+    // The decoded chunks the cache holds, and the stored bytes of the
+    // others: fetched batch by batch when the file is not in memory.
+    let hits: Vec<Option<Arc<CacheAlignedBuffer>>> = plan
+        .mappings
+        .iter()
+        .map(|m| cache.get_decompressed_in(addr, &m.coord))
+        .collect();
+    let reqs: Vec<ExtentReq> = plan
+        .mappings
+        .iter()
+        .zip(&hits)
+        .map(|(m, hit)| {
+            let len = m.file_size as usize;
+            ExtentReq {
+                addr: m.file_offset,
+                len,
+                fetch: hit.is_none().then(|| {
+                    len.min(crate::filters::stored_chunk_limit(
+                        pipeline,
+                        m.filter_mask,
+                        chunk_total_bytes,
+                    ))
+                }),
+            }
+        })
+        .collect();
+
     // Decompress chunks (using LRU cache where possible)
     let mut chunk_buffers: Vec<Arc<CacheAlignedBuffer>> = Vec::with_capacity(plan.mappings.len());
-    for m in &plan.mappings {
+    let mut hits = hits.into_iter();
-        let (coord, file_offset, file_size, filter_mask) =
+    for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
-            (&m.coord, &m.file_offset, &m.file_size, &m.filter_mask);
+        for i in batch {
-        if let Some(cached) = cache.get_decompressed_in(addr, coord) {
+            let m = &plan.mappings[i];
-            chunk_buffers.push(cached);
+            if let Some(cached) = hits.next().flatten() {
-        } else {
+                chunk_buffers.push(cached);
-            let c_addr = to_usize(*file_offset)?;
+                continue;
-            let size = *file_size as usize;
+            }
-            ensure_len(file_data, c_addr, size)?;
+            let raw_chunk = raw_bytes.get(i, &reqs[i])?;
-            let raw_chunk = &file_data[c_addr..c_addr + size];
             let decompressed = if let Some(pl) = pipeline {
                 decompress_chunk_exact(
                     raw_chunk,
                     pl,
                     chunk_total_bytes,
                     elem_size as u32,
-                    *filter_mask,
+                    m.filter_mask,
-                    coord,
+                    &m.coord,
                 )?
             } else {
                 raw_chunk.to_vec()
             };
             let aligned = CacheAlignedBuffer::from_vec(decompressed);
-            let arc = cache.put_decompressed_aligned_in(addr, coord.clone(), aligned);
+            chunk_buffers.push(cache.put_decompressed_aligned_in(addr, m.coord.clone(), aligned));
-            chunk_buffers.push(arc);
         }
-    }
+        Ok(())
+    })?;
 
     // Assemble using pre-computed layout
     let mut output = vec![0u8; plan.output_bytes];
@@ -3015,6 +3367,17 @@ mod tests {
         }
     }
 
+    /// A dataset below its maximum extent: libhdf5 lays chunks out over the
+    /// maximum chunk grid, so row 1 starts after a whole maximum row (here
+    /// 4 chunks), not after the current row of 3.
+    #[test]
+    fn implicit_chunks_use_the_maximum_grid() {
+        let chunks = generate_implicit_chunks_in_grid(0x100, &[2, 3], &[4, 4], &[1, 1], 4);
+        let addrs: Vec<u64> = chunks.iter().map(|c| (c.address - 0x100) / 4).collect();
+        assert_eq!(addrs, vec![0, 1, 2, 4, 5, 6]);
+        assert_eq!(chunks[3].offsets, vec![1, 0]);
+    }
+
     #[test]
     fn implicit_chunks_partial_last() {
         // 25 elements, chunk size 10 => 3 chunks (last partial)

crates/clawhdf5-format/src/data_read.rs

@@ -1,7 +1,9 @@
 //! Raw data reading and typed conversion for HDF5 datasets.
 
 #[cfg(not(feature = "std"))]
-use alloc::{collections::BTreeMap, format, string::String, vec, vec::Vec};
+use alloc::{borrow::Cow, collections::BTreeMap, format, string::String, vec, vec::Vec};
+#[cfg(feature = "std")]
+use std::borrow::Cow;
 
 #[cfg(feature = "std")]
 use std::collections::BTreeMap;
@@ -9,14 +11,15 @@ use std::collections::BTreeMap;
 use crate::addr::to_usize;
 #[cfg(feature = "std")]
 use crate::chunk_cache::ChunkCache;
-use crate::chunked_read::read_chunked_data;
+use crate::chunked_read::read_chunked_data_in;
 #[cfg(feature = "std")]
-use crate::chunked_read::{read_chunked_data_cached, read_chunked_data_indexed};
+use crate::chunked_read::{read_chunked_data_cached_in, read_chunked_data_indexed_in};
 use crate::data_layout::DataLayout;
 use crate::dataspace::Dataspace;
 use crate::datatype::{Datatype, DatatypeByteOrder};
 use crate::error::FormatError;
 use crate::filter_pipeline::FilterPipeline;
+use crate::storage::{Storage, read_exact_at};
 
 /// Checks that `[offset, offset + needed)` fits within `data`, guarding the
 /// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
@@ -149,7 +152,17 @@ pub fn read_raw_data(
     dataspace: &Dataspace,
     datatype: &Datatype,
 ) -> Result<Vec<u8>, FormatError> {
-    read_raw_data_full(file_data, layout, dataspace, datatype, None, 8, 8)
+    read_raw_data_in(file_data, layout, dataspace, datatype)
+}
+
+/// [`read_raw_data`] over any [`Storage`].
+pub fn read_raw_data_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+) -> Result<Vec<u8>, FormatError> {
+    read_raw_data_full_in(file_data, layout, dataspace, datatype, None, 8, 8)
 }
 
 /// Resolves a Virtual Dataset source **file name** (as stored in the mapping,
@@ -171,6 +184,27 @@ pub fn read_raw_data_full(
     pipeline: Option<&FilterPipeline>,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<u8>, FormatError> {
+    read_raw_data_full_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`read_raw_data_full`] over any [`Storage`].
+pub fn read_raw_data_full_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<u8>, FormatError> {
     read_raw_data_full_impl(
         file_data,
@@ -196,6 +230,30 @@ pub fn read_raw_data_full_with_resolver(
     offset_size: u8,
     length_size: u8,
     resolver: Option<&VdsSourceResolver>,
+) -> Result<Vec<u8>, FormatError> {
+    read_raw_data_full_with_resolver_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+        resolver,
+    )
+}
+
+/// [`read_raw_data_full_with_resolver`] over any [`Storage`].
+#[allow(clippy::too_many_arguments)]
+pub fn read_raw_data_full_with_resolver_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    resolver: Option<&VdsSourceResolver>,
 ) -> Result<Vec<u8>, FormatError> {
     read_raw_data_full_impl(
         file_data,
@@ -210,8 +268,8 @@ pub fn read_raw_data_full_with_resolver(
 }
 
 #[allow(clippy::too_many_arguments)]
-fn read_raw_data_full_impl(
+fn read_raw_data_full_impl<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     layout: &DataLayout,
     dataspace: &Dataspace,
     datatype: &Datatype,
@@ -242,12 +300,17 @@ fn read_raw_data_full_impl(
             let addr = address.ok_or(FormatError::NoDataAllocated)?;
             let addr = to_usize(addr)?;
             let sz = contiguous_read_len(*size, expected_size)?;
-            ensure_len(file_data, addr, sz)?;
+            match read_exact_at(file_data, addr as u64, sz)? {
-            let mut out = crate::bulk_alloc::vec_for_bulk(sz);
+                Cow::Borrowed(bytes) => {
-            out.extend_from_slice(&file_data[addr..addr + sz]);
+                    let mut out = crate::bulk_alloc::vec_for_bulk(sz);
-            Ok(out)
+                    out.extend_from_slice(bytes);
+                    Ok(out)
+                }
+                // Fetched for this read: already the caller's copy.
+                Cow::Owned(out) => Ok(out),
+            }
         }
-        DataLayout::Chunked { .. } => read_chunked_data(
+        DataLayout::Chunked { .. } => read_chunked_data_in(
             file_data,
             layout,
             dataspace,
@@ -284,9 +347,34 @@ pub fn read_raw_data_cached(
     offset_size: u8,
     length_size: u8,
     cache: &ChunkCache,
+) -> Result<Vec<u8>, FormatError> {
+    read_raw_data_cached_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+        cache,
+    )
+}
+
+/// [`read_raw_data_cached`] over any [`Storage`].
+#[cfg(feature = "std")]
+#[allow(clippy::too_many_arguments)]
+pub fn read_raw_data_cached_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    cache: &ChunkCache,
 ) -> Result<Vec<u8>, FormatError> {
     match layout {
-        DataLayout::Chunked { .. } => read_chunked_data_cached(
+        DataLayout::Chunked { .. } => read_chunked_data_cached_in(
             file_data,
             layout,
             dataspace,
@@ -296,7 +384,7 @@ pub fn read_raw_data_cached(
             length_size,
             cache,
         ),
-        _ => read_raw_data_full(
+        _ => read_raw_data_full_in(
             file_data,
             layout,
             dataspace,
@@ -325,9 +413,34 @@ pub fn read_raw_data_indexed(
     offset_size: u8,
     length_size: u8,
     cache: &ChunkCache,
+) -> Result<Vec<u8>, FormatError> {
+    read_raw_data_indexed_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+        cache,
+    )
+}
+
+/// [`read_raw_data_indexed`] over any [`Storage`].
+#[cfg(feature = "std")]
+#[allow(clippy::too_many_arguments)]
+pub fn read_raw_data_indexed_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    cache: &ChunkCache,
 ) -> Result<Vec<u8>, FormatError> {
     match layout {
-        DataLayout::Chunked { .. } => read_chunked_data_indexed(
+        DataLayout::Chunked { .. } => read_chunked_data_indexed_in(
             file_data,
             layout,
             dataspace,
@@ -337,7 +450,7 @@ pub fn read_raw_data_indexed(
             length_size,
             cache,
         ),
-        _ => read_raw_data_full(
+        _ => read_raw_data_full_in(
             file_data,
             layout,
             dataspace,
@@ -366,6 +479,30 @@ pub fn read_raw_data_selection(
     offset_size: u8,
     length_size: u8,
     selection: &crate::selection::Selection,
+) -> Result<Vec<u8>, FormatError> {
+    read_raw_data_selection_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+        selection,
+    )
+}
+
+/// [`read_raw_data_selection`] over any [`Storage`].
+#[allow(clippy::too_many_arguments)]
+pub fn read_raw_data_selection_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    selection: &crate::selection::Selection,
 ) -> Result<Vec<u8>, FormatError> {
     use crate::selection::Selection;
 
@@ -375,7 +512,7 @@ pub fn read_raw_data_selection(
     // Read only what the selection's bounding box touches when that is
     // possible; everything below is the decode-everything-then-pick path,
     // kept for the cases `partial_read` declines.
-    if let Some(selected) = crate::partial_read::read_selection(
+    if let Some(selected) = crate::partial_read::read_selection_in(
         file_data,
         layout,
         dataspace,
@@ -390,7 +527,7 @@ pub fn read_raw_data_selection(
 
     match selection {
         Selection::All => {
-            return read_raw_data_full(
+            return read_raw_data_full_in(
                 file_data,
                 layout,
                 dataspace,
@@ -410,7 +547,7 @@ pub fn read_raw_data_selection(
     match layout {
         DataLayout::Compact { .. } | DataLayout::Contiguous { .. } => {
             // Read all data, then extract the selection
-            let full_data = read_raw_data_full(
+            let full_data = read_raw_data_full_in(
                 file_data,
                 layout,
                 dataspace,
@@ -434,7 +571,7 @@ pub fn read_raw_data_selection(
             // implicit-index generator, which then indexed past the rank and
             // panicked — only to decode the full dataset anyway.
             crate::chunked_read::chunk_geometry(chunk_dimensions, *version, dataspace, elem_size)?;
-            let full_data = read_raw_data_full(
+            let full_data = read_raw_data_full_in(
                 file_data,
                 layout,
                 dataspace,
@@ -447,7 +584,7 @@ pub fn read_raw_data_selection(
         }
         DataLayout::Virtual { .. } => {
             // Assemble the full virtual dataset, then apply the read selection.
-            let full_data = read_raw_data_full(
+            let full_data = read_raw_data_full_in(
                 file_data,
                 layout,
                 dataspace,
@@ -471,8 +608,8 @@ pub fn read_raw_data_selection(
 /// would report differently from the stored dataspace (unlimited mappings).
 /// Use [`crate::vds::read_virtual_dataset`] to read those.
 #[allow(clippy::too_many_arguments)]
-fn read_virtual_data(
+fn read_virtual_data<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     layout: &DataLayout,
     dataspace: &Dataspace,
     datatype: &Datatype,
@@ -483,7 +620,7 @@ fn read_virtual_data(
     let wrapped =
         resolver.map(|r| move |name: &str| -> Result<Option<Vec<u8>>, FormatError> { Ok(r(name)) });
     let wrapped_ref = wrapped.as_ref().map(|w| w as &crate::vds::VdsFileResolver);
-    let v = crate::vds::read_virtual_dataset(
+    let v = crate::vds::read_virtual_dataset_in(
         file_data,
         layout,
         dataspace,
@@ -819,6 +956,78 @@ pub fn read_selection_native<T: NativeElement>(
     crate::gather::gather::<T>(raw, dims, elem_size, selection).map(Some)
 }
 
+/// [`read_selection_native`] of a contiguous dataset in any [`Storage`],
+/// reading only the selected elements' runs (adjacent ones merged, one
+/// [`Storage::read_ranges`] call) instead of the whole dataset.
+///
+/// `Ok(None)` wherever the in-memory fast path does not apply and the
+/// caller converts through the byte readers instead: `datatype` is not
+/// `T`'s native representation, the layout is not contiguous, or the
+/// dataset's bytes cannot be located in the file (no address, storage too
+/// small, past the end of file: the cases [`read_raw_data_zerocopy`]
+/// fails). Otherwise the result and errors are [`read_selection_native`]'s
+/// over those bytes.
+pub fn read_selection_native_in<T: NativeElement, S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    selection: &crate::selection::Selection,
+) -> Result<Option<Vec<T>>, FormatError> {
+    if !T::is_native(datatype) {
+        return Ok(None);
+    }
+    let DataLayout::Contiguous {
+        address: Some(address),
+        size,
+    } = layout
+    else {
+        return Ok(None);
+    };
+    // Where the dataset's bytes are, as `read_raw_data_zerocopy` finds them.
+    let located = to_usize(dataspace.num_elements())
+        .ok()
+        .and_then(|n| n.checked_mul(datatype.type_size() as usize))
+        .filter(|&len| contiguous_read_len(*size, len).is_ok())
+        .filter(|&len| {
+            address
+                .checked_add(len as u64)
+                .is_some_and(|end| end <= file_data.len())
+        });
+    let Some(len) = located else {
+        return Ok(None);
+    };
+    if let Some(all) = file_data.as_contiguous() {
+        let start = to_usize(*address)?;
+        return read_selection_native(
+            &all[start..start + len],
+            &dataspace.dimensions,
+            datatype,
+            selection,
+        );
+    }
+    let dims = &dataspace.dimensions;
+    let elem_size = core::mem::size_of::<T>();
+    let total = dims
+        .iter()
+        .try_fold(1u64, |acc, &d| acc.checked_mul(d))
+        .ok_or_else(|| FormatError::Overflow("dataset shape overflows".into()))?;
+    let expected = crate::chunked_read::checked_byte_len(total, elem_size)?;
+    if len != expected {
+        return Err(FormatError::DataSizeMismatch {
+            expected,
+            actual: len,
+        });
+    }
+    let bytes = if let crate::selection::Selection::All = selection {
+        read_exact_at(file_data, *address, len)?.into_owned()
+    } else {
+        crate::partial_read::validate(selection, dims)?;
+        crate::gather::gather_storage(file_data, *address, len, dims, elem_size, selection)?
+    };
+    Ok(Some(native_to_vec(&bytes, bytes.len() / elem_size)))
+}
+
 /// The bytes of a slice of [`NativeElement`]s.
 #[cfg(feature = "std")]
 fn bytes_of_mut<T: NativeElement>(values: &mut [T]) -> &mut [u8] {
@@ -885,6 +1094,33 @@ pub fn read_chunked_native<T: NativeElement>(
     offset_size: u8,
     length_size: u8,
     cache: Option<&ChunkCache>,
+) -> Result<Option<Vec<T>>, FormatError> {
+    read_chunked_native_in(
+        messages,
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+        cache,
+    )
+}
+
+/// [`read_chunked_native`] over any [`Storage`].
+#[cfg(feature = "std")]
+#[allow(clippy::too_many_arguments)]
+pub fn read_chunked_native_in<T: NativeElement, S: Storage + ?Sized>(
+    messages: &[crate::object_header::HeaderMessage],
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    cache: Option<&ChunkCache>,
 ) -> Result<Option<Vec<T>>, FormatError> {
     use crate::fill_value;
     use crate::message_type::MessageType;
@@ -919,8 +1155,9 @@ pub fn read_chunked_native<T: NativeElement>(
         },
         |values| bytes_of_mut(values),
     )?;
-    let fill = fill_value::dataset_fill_value_in(file_data, messages, offset_size, length_size)?;
+    let fill =
-    fill_value::apply_to_unallocated_chunks(
+        fill_value::dataset_fill_value_from_storage(file_data, messages, offset_size, length_size)?;
+    fill_value::apply_to_unallocated_chunks_in(
         bytes_of_mut(&mut values),
         file_data,
         layout,

crates/clawhdf5-format/src/fill_value.rs

@@ -13,7 +13,7 @@
 use alloc::{format, vec, vec::Vec};
 
 use crate::addr::to_usize;
-use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks};
+use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks_in};
 use crate::data_layout::DataLayout;
 use crate::dataspace::Dataspace;
 use crate::error::FormatError;
@@ -122,10 +122,11 @@ pub fn dataset_fill_value_in(
 }
 
 /// [`dataset_fill_value_in`] with the file behind any
-/// [`Storage`](crate::storage::Storage). (The trait is not imported here:
+/// [`Storage`](crate::storage::Storage) (a `&dyn Storage` too). (The trait
-/// its `len` would shadow the slice method in this module.)
+/// is not imported here: its `len` would shadow the slice method in this
-pub fn dataset_fill_value_from_storage(
+/// module.)
-    file: &dyn crate::storage::Storage,
+pub fn dataset_fill_value_from_storage<S: crate::storage::Storage + ?Sized>(
+    file: &S,
     messages: &[HeaderMessage],
     offset_size: u8,
     length_size: u8,
@@ -212,6 +213,30 @@ pub fn read_full_with_fill<E: From<FormatError>>(
     offset_size: u8,
     length_size: u8,
     read: impl FnOnce() -> Result<Vec<u8>, E>,
+) -> Result<Vec<u8>, E> {
+    read_full_with_fill_in(
+        messages,
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        offset_size,
+        length_size,
+        read,
+    )
+}
+
+/// [`read_full_with_fill`] over any [`Storage`](crate::storage::Storage).
+#[allow(clippy::too_many_arguments)]
+pub fn read_full_with_fill_in<E: From<FormatError>, S: crate::storage::Storage + ?Sized>(
+    messages: &[HeaderMessage],
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    offset_size: u8,
+    length_size: u8,
+    read: impl FnOnce() -> Result<Vec<u8>, E>,
 ) -> Result<Vec<u8>, E> {
     // A dataset with external raw data also has no data address in this
     // file. It is NOT unallocated — its values live elsewhere — so it must
@@ -222,12 +247,12 @@ pub fn read_full_with_fill<E: From<FormatError>>(
     {
         return Err(FormatError::ExternalDataFilesUnsupported.into());
     }
-    let fill = dataset_fill_value_in(file_data, messages, offset_size, length_size)?;
+    let fill = dataset_fill_value_from_storage(file_data, messages, offset_size, length_size)?;
     if !has_storage(layout) {
         return Ok(filled_dataset(dataspace, elem_size, fill.as_deref())?);
     }
     let mut output = read()?;
-    apply_to_unallocated_chunks(
+    apply_to_unallocated_chunks_in(
         &mut output,
         file_data,
         layout,
@@ -253,6 +278,30 @@ pub fn apply_to_unallocated_chunks(
     fill: Option<&[u8]>,
     offset_size: u8,
     length_size: u8,
+) -> Result<(), FormatError> {
+    apply_to_unallocated_chunks_in(
+        output,
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        fill,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`apply_to_unallocated_chunks`] over any [`Storage`](crate::storage::Storage).
+#[allow(clippy::too_many_arguments)]
+pub fn apply_to_unallocated_chunks_in<S: crate::storage::Storage + ?Sized>(
+    output: &mut [u8],
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    fill: Option<&[u8]>,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<(), FormatError> {
     let Some(fill) = fill.filter(|f| f.len() == elem_size && !is_default(Some(f))) else {
         return Ok(());
@@ -260,7 +309,7 @@ pub fn apply_to_unallocated_chunks(
     if !matches!(layout, DataLayout::Chunked { .. }) || elem_size == 0 {
         return Ok(());
     }
-    let (chunks, chunk_dims) = list_chunks(
+    let (chunks, chunk_dims) = list_chunks_in(
         file_data,
         layout,
         dataspace,

crates/clawhdf5-format/src/filter_registry.rs

@@ -188,6 +188,22 @@ pub fn is_filter_available(id: u16) -> bool {
     }
 }
 
+/// Whether chunks filtered with `id` may be decoded by a codec the
+/// application registered (whose stored sizes this crate cannot bound).
+pub(crate) fn may_be_registered(id: u16) -> bool {
+    if builtin_filter(id).is_some_and(|b| !b.is_shared()) {
+        return false;
+    }
+    #[cfg(feature = "std")]
+    {
+        registered(id).is_some()
+    }
+    #[cfg(not(feature = "std"))]
+    {
+        false
+    }
+}
+
 #[cfg(feature = "std")]
 mod custom {
     use super::FilterCodec;

crates/clawhdf5-format/src/filters.rs

@@ -63,6 +63,50 @@ fn filter_output_bound(filter_id: u16, input: usize) -> usize {
     }
 }
 
+/// Most stored bytes a chunk whose decoded size is `chunk_bytes` can need:
+/// what a raw-data read fetches of (and decodes from) a chunk, whatever size
+/// its index entry claims. A crafted index that points many chunks at huge
+/// extents then costs no more than legitimate chunks would.
+///
+/// An unfiltered chunk (no pipeline, or every filter skipped by
+/// `filter_mask`) is the chunk itself: `chunk_bytes`, and bytes past them
+/// were never used. A filtered chunk is bounded by each applied filter's
+/// worst-case growth in the write direction: shuffle keeps the size,
+/// Fletcher32 adds 4 bytes, ZFP gets `4n + 4096` (its fixed-rate mode stores
+/// up to 64 bits per value), and any other codec gets `n + n/4 + 4096` — a
+/// deliberately generous bound (bzip2 grows 1000 random bytes by 252, more
+/// than the decoders' own `n + n/8 + 64` output bound), since a legitimate
+/// chunk cut short here would fail to read. A filter handled by a codec the
+/// application registered is not ours to bound: such a chunk is limited
+/// only by the file.
+pub(crate) fn stored_chunk_limit(
+    pipeline: Option<&FilterPipeline>,
+    filter_mask: u32,
+    chunk_bytes: usize,
+) -> usize {
+    let Some(pipeline) = pipeline else {
+        return chunk_bytes;
+    };
+    let mut size = chunk_bytes;
+    for (i, filter) in pipeline.filters.iter().enumerate() {
+        if filter_skipped(filter_mask, i) {
+            continue;
+        }
+        size = match filter.filter_id {
+            FILTER_SHUFFLE => size,
+            FILTER_FLETCHER32 => size.saturating_add(4),
+            id if filter_registry::may_be_registered(id) => return usize::MAX,
+            // ZFP's fixed-rate mode stores up to 64 bits per value, so a
+            // 4-byte type doubles, and precision/accuracy modes add group
+            // test bits per bit plane on top: 4x plus a header is still a
+            // bound, where `n + n/4` cut rate-64 chunks short.
+            crate::filter_pipeline::FILTER_ZFP => size.saturating_mul(4).saturating_add(4096),
+            _ => size.saturating_add(size / 4).saturating_add(4096),
+        };
+    }
+    size
+}
+
 /// Whether bit `index` of a chunk's filter mask says filter `index` was
 /// skipped when the chunk was written.
 fn filter_skipped(filter_mask: u32, index: usize) -> bool {
@@ -1678,56 +1722,6 @@ fn shuffle_compress_general(data: &[u8], n: usize, element_size: usize, result:
     }
 }
 
-/// Compute HDF5 Fletcher32 checksum over data.
-/// HDF5 uses a modified Fletcher32 that operates on 16-bit words.
-///
-/// Optimized with wider accumulators: processes blocks of 360 words before
-/// taking the modulo, reducing the number of expensive modulo operations.
-/// (360 is the maximum block size that avoids u32 overflow for sum2.)
-fn fletcher32_compute(data: &[u8]) -> u32 {
-    let mut sum1: u32 = 0;
-    let mut sum2: u32 = 0;
-
-    // Process in blocks of 360 16-bit words (720 bytes) to delay modulo.
-    // Max sum1 before mod: 360 * 65535 = 23_592_600 < u32::MAX
-    // Max sum2 before mod: 360 * 23_592_600 ~ 8.5B > u32::MAX, but actual
-    // sum2 accumulates incrementally, so worst case is 360*360*65535/2 which
-    // fits in u64. We use u32 with block size 360 which is safe.
-    const BLOCK_WORDS: usize = 360;
-    const BLOCK_BYTES: usize = BLOCK_WORDS * 2;
-
-    let mut offset = 0;
-    let len = data.len();
-
-    while offset + BLOCK_BYTES <= len {
-        let end = offset + BLOCK_BYTES;
-        let mut i = offset;
-        while i < end {
-            let val = ((data[i] as u32) << 8) | (data[i + 1] as u32);
-            sum1 += val;
-            sum2 += sum1;
-            i += 2;
-        }
-        sum1 %= 65535;
-        sum2 %= 65535;
-        offset = end;
-    }
-
-    // Handle remaining bytes
-    while offset < len {
-        let val = if offset + 1 < len {
-            ((data[offset] as u32) << 8) | (data[offset + 1] as u32)
-        } else {
-            (data[offset] as u32) << 8
-        };
-        sum1 = (sum1 + val) % 65535;
-        sum2 = (sum2 + sum1) % 65535;
-        offset += 2;
-    }
-
-    (sum2 << 16) | sum1
-}
-
 /// Verify Fletcher32 checksum and strip it from the data.
 /// The last 4 bytes are the stored checksum.
 fn fletcher32_verify(data: &[u8]) -> Result<Vec<u8>, FormatError> {
@@ -1749,8 +1743,18 @@ fn fletcher32_payload(data: &[u8]) -> Result<usize, FormatError> {
         data[data.len() - 2],
         data[data.len() - 1],
     ]);
-    let computed = fletcher32_compute(payload);
+    let computed = crate::checksum::fletcher32(payload);
-    if stored != computed {
+    // libhdf5 also accepts the checksum with the bytes of each 16-bit half
+    // swapped, which is how 1.6.2 and earlier stored it
+    // (H5Z__filter_fletcher32's `reversed_fletcher`).
+    let reversed = ((computed & 0x00ff_00ff) << 8) | ((computed >> 8) & 0x00ff_00ff);
+    // clawhdf5 v2.7.0 and earlier reduced the sums `% 65535`, which gives 0
+    // where libhdf5's fold gives 0xffff; accept that form too, so that files
+    // those releases wrote can still be read (and rewritten for libhdf5).
+    // It differs from `computed` only in a half that is 0xffff.
+    let half = |h: u32| if h == 0xffff { 0 } else { h };
+    let legacy = (half(computed >> 16) << 16) | half(computed & 0xffff);
+    if stored != computed && stored != reversed && stored != legacy {
         return Err(FormatError::Fletcher32Mismatch {
             expected: stored,
             computed,
@@ -1761,7 +1765,7 @@ fn fletcher32_payload(data: &[u8]) -> Result<usize, FormatError> {
 
 /// Append Fletcher32 checksum to data.
 fn fletcher32_append(data: &[u8]) -> Result<Vec<u8>, FormatError> {
-    let checksum = fletcher32_compute(data);
+    let checksum = crate::checksum::fletcher32(data);
     let mut result = data.to_vec();
     result.extend_from_slice(&checksum.to_le_bytes());
     Ok(result)

crates/clawhdf5-format/src/fractal_heap.rs

@@ -7,10 +7,10 @@ use alloc::{format, vec::Vec};
 use byteorder::{ByteOrder, LittleEndian};
 
 use crate::addr::to_usize;
-use crate::btree_v2::{BTreeV2Header, find_btree_v2_records};
+use crate::btree_v2::{BTreeV2Header, find_btree_v2_records_in};
 use crate::error::FormatError;
 use crate::filter_pipeline::FilterPipeline;
-use crate::storage::{Storage, Window, len_usize, read_exact_at, require_contiguous};
+use crate::storage::{Storage, Window, len_usize, read_exact_at};
 
 /// Parsed fractal heap header (signature "FRHP").
 #[derive(Debug, Clone)]
@@ -386,9 +386,7 @@ impl FractalHeapHeader {
         self.read_managed_object_in(file_data, id_bytes, offset_size)
     }
 
-    /// [`Self::read_managed_object`] over any [`Storage`]. A huge object
+    /// [`Self::read_managed_object`] over any [`Storage`].
-    /// found through the huge-object v2 B-tree still needs the whole file
-    /// in memory ([`FormatError::ContiguousStorageRequired`] otherwise).
     pub fn read_managed_object_in<S: Storage + ?Sized>(
         &self,
         file_data: &S,
@@ -491,11 +489,9 @@ impl FractalHeapHeader {
                 "huge object ID but the heap has no huge-object index",
             ));
         }
-        // The v2 B-tree is read from a slice until it is converted.
+        let hdr = BTreeV2Header::parse_in(
-        let file_data = require_contiguous(file, "a huge fractal-heap object's B-tree")?;
+            file,
-        let hdr = BTreeV2Header::parse(
+            self.huge_btree_address,
-            file_data,
-            to_usize(self.huge_btree_address)?,
             self.offset_size,
             self.length_size,
         )?;
@@ -513,7 +509,7 @@ impl FractalHeapHeader {
         // Records are ordered by ID (the last field): descend to the ones
         // equal to `key` instead of reading the whole index.
         let id_at = rec_len - ls;
-        let records = find_btree_v2_records(file_data, &hdr, self.offset_size, &mut |r| {
+        let records = find_btree_v2_records_in(file, &hdr, self.offset_size, &mut |r| {
             le_uint(&r[id_at..id_at + ls]).cmp(&key)
         })?;
         for rec in &records {
@@ -1317,22 +1313,20 @@ mod tests {
         assert_eq!(hdr.read_managed_object_in(&storage, &id, 8), want);
     }
 
-    /// A huge object found through the huge-object B-tree needs the whole
+    /// A huge object found through the huge-object B-tree reads the
-    /// file in memory until the B-tree reader is converted: a clean error
+    /// B-tree through Storage: the same result (here an error, there is no
-    /// on other storage.
+    /// B-tree at that address) as from the slice.
     #[test]
-    fn huge_object_btree_needs_contiguous_storage() {
+    fn huge_object_btree_reads_through_storage() {
         use crate::storage::CountingStorage;
         let (file, _) = build_simple_heap(8, 8);
         let mut hdr = FractalHeapHeader::parse(&file, 0, 8, 8).unwrap();
         hdr.huge_btree_address = 700;
+        let id = [0x10, 1, 0, 0, 0, 0, 0];
+        let want = hdr.read_managed_object(&file, &id, 8);
+        assert!(want.is_err());
         let storage = CountingStorage::new(file);
-        assert_eq!(
+        assert_eq!(hdr.read_managed_object_in(&storage, &id, 8), want);
-            hdr.read_managed_object_in(&storage, &[0x10, 1, 0, 0, 0, 0, 0], 8),
-            Err(FormatError::ContiguousStorageRequired(
-                "a huge fractal-heap object's B-tree"
-            ))
-        );
     }
 
     /// A header with an I/O filter pipeline (read in a second, longer

crates/clawhdf5-format/src/gather.rs

@@ -13,6 +13,7 @@ use alloc::{vec, vec::Vec};
 use crate::data_read::NativeElement;
 use crate::error::FormatError;
 use crate::selection::Selection;
+use crate::storage::{ExtentBytes, ExtentReq, Storage, raw_batches};
 
 /// Row-major element strides of `dims` (the last dimension has stride 1).
 fn strides(dims: &[u64]) -> Vec<u64> {
@@ -261,6 +262,259 @@ pub(crate) fn gather<T: NativeElement>(
     Ok(out)
 }
 
+/// Largest gap between two of a selection's runs that [`gather_storage`]
+/// reads through rather than asking for the runs separately: skipping a
+/// few KiB costs a remote backend far less than another request (and a
+/// local one less than another call and allocation).
+pub(crate) const GATHER_GAP_BYTES: usize = 4 << 10;
+
+/// Largest single read [`gather_storage`] makes of a selection's runs: runs
+/// are merged into reads up to this size, and a longer run is split.
+pub(crate) const GATHER_SPAN_BYTES: usize = 8 << 20;
+
+/// Call `emit(first_element, element_count)` for each run of a validated
+/// hyperslab or point selection (in output order; see [`hyperslab_runs`]),
+/// or the error for a hyperslab of the wrong rank or a point outside `dims`
+/// (runs before that point have been emitted).
+fn selection_runs(
+    dims: &[u64],
+    selection: &Selection,
+    emit: &mut dyn FnMut(u64, u64),
+) -> Result<(), FormatError> {
+    match selection {
+        Selection::Hyperslab {
+            start,
+            stride,
+            count,
+            block,
+        } => {
+            let rank = dims.len();
+            if [start.len(), stride.len(), count.len(), block.len()] != [rank; 4] {
+                return Err(FormatError::SelectionOutOfBounds(
+                    "hyperslab rank does not match dataset rank".into(),
+                ));
+            }
+            hyperslab_runs(dims, start, stride, count, block, emit);
+        }
+        Selection::Points(points) => {
+            let strides = strides(dims);
+            let mut coalesce = Coalesce {
+                start: 0,
+                len: 0,
+                emit,
+            };
+            for p in points {
+                if p.len() != dims.len() || p.iter().zip(dims).any(|(c, n)| c >= n) {
+                    return Err(FormatError::SelectionOutOfBounds(
+                        "selection addresses elements outside the dataset".into(),
+                    ));
+                }
+                let at = p
+                    .iter()
+                    .zip(&strides)
+                    .fold(0u64, |acc, (c, s)| acc.wrapping_add(c.wrapping_mul(*s)));
+                coalesce.push(at, 1);
+            }
+            coalesce.flush();
+        }
+        Selection::None | Selection::All => {}
+    }
+    Ok(())
+}
+
+/// One read of [`gather_storage`]: bytes `[start, end)` of the dataset,
+/// which hold the output's bytes up to `out_end` (from where the previous
+/// span's end left off).
+#[derive(Clone, Copy)]
+struct Span {
+    start: usize,
+    end: usize,
+    out_end: usize,
+}
+
+/// [`gather`] of bytes (`T = u8`) from a dataset that is not in memory: the
+/// dataset's `src_len` bytes start at `base` in `file`, which must hold all
+/// of them (the caller checks). Same checks and errors as [`gather`].
+///
+/// The selection's runs are walked twice. The first walk checks them and
+/// plans the reads: runs in increasing order with at most
+/// [`GATHER_GAP_BYTES`] between them are read as one span (the gap is read
+/// and dropped), up to [`GATHER_SPAN_BYTES`] per span. So a strided
+/// selection is a few large reads, not one per element, and nothing is
+/// allocated per run. The spans are fetched batch by batch (one
+/// [`Storage::read_ranges`] call per [`crate::storage::RAW_BATCH_BYTES`])
+/// while the second walk copies each run out of its span.
+pub(crate) fn gather_storage<S: Storage + ?Sized>(
+    file: &S,
+    base: u64,
+    src_len: usize,
+    dims: &[u64],
+    elem_size: usize,
+    selection: &Selection,
+) -> Result<Vec<u8>, FormatError> {
+    if elem_size == 0 {
+        return Err(FormatError::DataSizeMismatch {
+            expected: 1,
+            actual: elem_size,
+        });
+    }
+    let n_elements = match selection {
+        Selection::None => 0,
+        Selection::Hyperslab { count, block, .. } => count
+            .iter()
+            .zip(block)
+            .try_fold(1u64, |acc, (&c, &b)| acc.checked_mul(c.checked_mul(b)?))
+            .ok_or_else(|| FormatError::Overflow("hyperslab count x block overflows".into()))?,
+        Selection::Points(points) => points.len() as u64,
+        Selection::All => {
+            return Err(FormatError::SelectionOutOfBounds(
+                "gather does not take Selection::All".into(),
+            ));
+        }
+    };
+    let out_bytes = crate::chunked_read::checked_byte_len(n_elements, elem_size)?;
+    let outside = || {
+        FormatError::SelectionOutOfBounds("selection addresses elements outside the dataset".into())
+    };
+
+    // First walk: check every run and plan the spans.
+    let mut spans: Vec<Span> = Vec::new();
+    let mut total = 0usize;
+    let mut failed = false;
+    selection_runs(dims, selection, &mut |first: u64, n: u64| {
+        if failed {
+            return;
+        }
+        let range = usize::try_from(first)
+            .ok()
+            .and_then(|f| f.checked_mul(elem_size))
+            .zip(
+                usize::try_from(n)
+                    .ok()
+                    .and_then(|n| n.checked_mul(elem_size)),
+            )
+            .and_then(|(at, len)| Some((at, len, at.checked_add(len)?)));
+        let Some((mut at, mut len)) = range
+            .filter(|&(_, len, end)| end <= src_len && len <= out_bytes - total)
+            .map(|(at, len, _)| (at, len))
+        else {
+            failed = true;
+            return;
+        };
+        while len > 0 {
+            let room = match spans.last_mut() {
+                Some(s)
+                    if at >= s.end
+                        && at - s.end <= GATHER_GAP_BYTES
+                        && at - s.start < GATHER_SPAN_BYTES =>
+                {
+                    let take = len.min(GATHER_SPAN_BYTES - (at - s.start));
+                    s.end = at + take;
+                    s.out_end += take;
+                    take
+                }
+                _ => {
+                    let take = len.min(GATHER_SPAN_BYTES);
+                    spans.push(Span {
+                        start: at,
+                        end: at + take,
+                        out_end: total + take,
+                    });
+                    take
+                }
+            };
+            total += room;
+            at += room;
+            len -= room;
+        }
+    })?;
+    if failed || total != out_bytes {
+        return Err(outside());
+    }
+
+    // The spans' reads, and the batches they are fetched in.
+    let reqs: Vec<ExtentReq> = spans
+        .iter()
+        .map(|s| ExtentReq {
+            addr: base + s.start as u64,
+            len: s.end - s.start,
+            fetch: Some(s.end - s.start),
+        })
+        .collect();
+    let batches = raw_batches(reqs.len(), false, |i| reqs[i].len);
+
+    // Second walk: copy each run out of its span, fetching each batch of
+    // spans when the walk reaches it (and dropping the previous one).
+    let mut out = crate::bulk_alloc::vec_for_bulk(out_bytes);
+    let mut span = 0usize;
+    let mut batch = 0usize;
+    let mut fetched: Option<ExtentBytes<'_>> = None;
+    let mut error: Option<FormatError> = None;
+    selection_runs(dims, selection, &mut |first: u64, n: u64| {
+        if error.is_some() {
+            return;
+        }
+        // Checked by the first walk (these cannot saturate or wrap).
+        let mut at = crate::addr::saturating_usize(first).wrapping_mul(elem_size);
+        let mut len = crate::addr::saturating_usize(n).wrapping_mul(elem_size);
+        while len > 0 {
+            while spans.get(span).is_some_and(|s| s.out_end <= out.len()) {
+                span += 1;
+            }
+            if fetched.is_none() || span >= batches[batch].end {
+                fetched = None;
+                while batches.get(batch).is_some_and(|b| span >= b.end) {
+                    batch += 1;
+                }
+                let (Some(b), Some(_)) = (batches.get(batch).cloned(), spans.get(span)) else {
+                    // The second walk emitted more than the first.
+                    error = Some(outside());
+                    return;
+                };
+                match ExtentBytes::fetch(file, &reqs[b.clone()], b.start) {
+                    Ok(f) => fetched = Some(f),
+                    Err(e) => {
+                        error = Some(e);
+                        return;
+                    }
+                }
+            }
+            let s = spans[span];
+            let take = len.min(s.out_end - out.len());
+            let bytes = match fetched
+                .as_ref()
+                .map(|f| f.get(span, &reqs[span]))
+                .unwrap_or_else(|| Err(outside()))
+            {
+                Ok(b) => b,
+                Err(e) => {
+                    error = Some(e);
+                    return;
+                }
+            };
+            match at
+                .checked_sub(s.start)
+                .and_then(|o| bytes.get(o..o.checked_add(take)?))
+            {
+                Some(b) => out.extend_from_slice(b),
+                None => {
+                    error = Some(outside());
+                    return;
+                }
+            }
+            at += take;
+            len -= take;
+        }
+    })?;
+    if let Some(e) = error {
+        return Err(e);
+    }
+    if out.len() != out_bytes {
+        return Err(outside());
+    }
+    Ok(out)
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;

crates/clawhdf5-format/src/global_heap.rs

@@ -152,7 +152,7 @@ impl GlobalHeapCollection {
     /// Read the collection at `offset` and index its objects: the
     /// collection's bytes, its offset as a `usize`, and the index (with
     /// file offsets).
-    fn read_collection<S: Storage + ?Sized>(
+    pub(crate) fn read_collection<S: Storage + ?Sized>(
         file: &S,
         offset: u64,
         length_size: u8,

crates/clawhdf5-format/src/group_v1.rs

@@ -3,12 +3,13 @@
 #[cfg(not(feature = "std"))]
 use alloc::{string::String, vec::Vec};
 
-use crate::addr::to_usize;
+use crate::addr::checked_addr;
-use crate::btree_v1::collect_symbol_table_nodes;
+use crate::btree_v1::collect_symbol_table_nodes_in;
 use crate::error::FormatError;
 use crate::local_heap::LocalHeap;
 use crate::message_type::MessageType;
 use crate::object_header::ObjectHeader;
+use crate::storage::Storage;
 use crate::symbol_table::{SymbolTableMessage, SymbolTableNode};
 
 /// A resolved group entry (child name + object header address).
@@ -36,6 +37,16 @@ pub fn resolve_v1_group_entries(
     sym_table_msg: &SymbolTableMessage,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    resolve_v1_group_entries_in(file_data, sym_table_msg, offset_size, length_size)
+}
+
+/// [`resolve_v1_group_entries`] over any [`Storage`].
+pub fn resolve_v1_group_entries_in<S: Storage + ?Sized>(
+    file_data: &S,
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<GroupEntry>, FormatError> {
     let entries = v1_group_entries(file_data, sym_table_msg, offset_size, length_size)?;
     if entries.iter().any(|e| e.name.is_empty()) {
@@ -46,22 +57,22 @@ pub fn resolve_v1_group_entries(
 
 /// Every entry of a v1 group, empty names included — for looking a name up,
 /// which never matches an empty name.
-pub(crate) fn v1_group_entries(
+pub(crate) fn v1_group_entries<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     sym_table_msg: &SymbolTableMessage,
     offset_size: u8,
     length_size: u8,
 ) -> Result<Vec<GroupEntry>, FormatError> {
     // Parse local heap
-    let heap = LocalHeap::parse(
+    let heap = LocalHeap::parse_in(
         file_data,
-        to_usize(sym_table_msg.local_heap_address)?,
+        checked_addr(sym_table_msg.local_heap_address)?,
         offset_size,
         length_size,
     )?;
 
     // Collect all SNOD addresses from B-tree
-    let snod_addrs = collect_symbol_table_nodes(
+    let snod_addrs = collect_symbol_table_nodes_in(
         file_data,
         sym_table_msg.btree_address,
         offset_size,
@@ -71,15 +82,15 @@ pub(crate) fn v1_group_entries(
     let mut entries = Vec::new();
     let mut heap_checked = false;
     for snod_addr in snod_addrs {
-        let snod = SymbolTableNode::parse(file_data, to_usize(snod_addr)?, offset_size)?;
+        let snod = SymbolTableNode::parse_in(file_data, checked_addr(snod_addr)?, offset_size)?;
         for entry in &snod.entries {
             // Like libhdf5, look at the heap's free list only once a name is
             // needed: an empty group with a damaged heap still lists.
             if !heap_checked {
-                heap.validate_free_list(file_data, length_size)?;
+                heap.validate_free_list_in(file_data, length_size)?;
                 heap_checked = true;
             }
-            let name = heap.read_string(file_data, entry.link_name_offset)?;
+            let name = heap.read_string_in(file_data, entry.link_name_offset)?;
             entries.push(GroupEntry {
                 name,
                 object_header_address: entry.object_header_address,
@@ -103,6 +114,17 @@ pub fn find_v1_soft_link(
     name: &str,
     offset_size: u8,
     length_size: u8,
+) -> Result<Option<String>, FormatError> {
+    find_v1_soft_link_in(file_data, sym_table_msg, name, offset_size, length_size)
+}
+
+/// [`find_v1_soft_link`] over any [`Storage`].
+pub fn find_v1_soft_link_in<S: Storage + ?Sized>(
+    file_data: &S,
+    sym_table_msg: &SymbolTableMessage,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Option<String>, FormatError> {
     let mut found = None;
     for_each_v1_soft_link(
@@ -125,6 +147,16 @@ pub fn v1_soft_links(
     sym_table_msg: &SymbolTableMessage,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<(String, String)>, FormatError> {
+    v1_soft_links_in(file_data, sym_table_msg, offset_size, length_size)
+}
+
+/// [`v1_soft_links`] over any [`Storage`].
+pub fn v1_soft_links_in<S: Storage + ?Sized>(
+    file_data: &S,
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<(String, String)>, FormatError> {
     let mut links = Vec::new();
     for_each_v1_soft_link(
@@ -143,21 +175,21 @@ pub fn v1_soft_links(
 
 /// Visit the soft links of a v1 group whose name passes `wanted`, with their
 /// target paths, until `visit` returns false.
-fn for_each_v1_soft_link(
+fn for_each_v1_soft_link<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     sym_table_msg: &SymbolTableMessage,
     offset_size: u8,
     length_size: u8,
     wanted: impl Fn(&str) -> bool,
     mut visit: impl FnMut(&str, String) -> bool,
 ) -> Result<(), FormatError> {
-    let heap = LocalHeap::parse(
+    let heap = LocalHeap::parse_in(
         file_data,
-        to_usize(sym_table_msg.local_heap_address)?,
+        checked_addr(sym_table_msg.local_heap_address)?,
         offset_size,
         length_size,
     )?;
-    let snod_addrs = collect_symbol_table_nodes(
+    let snod_addrs = collect_symbol_table_nodes_in(
         file_data,
         sym_table_msg.btree_address,
         offset_size,
@@ -165,16 +197,16 @@ fn for_each_v1_soft_link(
     )?;
     let mut heap_checked = false;
     for snod_addr in snod_addrs {
-        let snod = SymbolTableNode::parse(file_data, to_usize(snod_addr)?, offset_size)?;
+        let snod = SymbolTableNode::parse_in(file_data, checked_addr(snod_addr)?, offset_size)?;
         for entry in &snod.entries {
             if entry.cache_type != CACHE_TYPE_SOFT_LINK {
                 continue;
             }
             if !heap_checked {
-                heap.validate_free_list(file_data, length_size)?;
+                heap.validate_free_list_in(file_data, length_size)?;
                 heap_checked = true;
             }
-            let name = heap.read_string(file_data, entry.link_name_offset)?;
+            let name = heap.read_string_in(file_data, entry.link_name_offset)?;
             if !wanted(&name) {
                 continue;
             }
@@ -184,7 +216,7 @@ fn for_each_v1_soft_link(
                 entry.scratch_pad[2],
                 entry.scratch_pad[3],
             ]);
-            let target = heap.read_string(file_data, u64::from(value_offset))?;
+            let target = heap.read_string_in(file_data, u64::from(value_offset))?;
             if !visit(&name, target) {
                 return Ok(());
             }
@@ -222,6 +254,17 @@ pub fn resolve_path(
     path: &str,
     offset_size: u8,
     length_size: u8,
+) -> Result<u64, FormatError> {
+    resolve_path_in(file_data, root_sym_table, path, offset_size, length_size)
+}
+
+/// [`resolve_path`] over any [`Storage`].
+pub fn resolve_path_in<S: Storage + ?Sized>(
+    file_data: &S,
+    root_sym_table: &SymbolTableMessage,
+    path: &str,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<u64, FormatError> {
     let components: Vec<&str> = path.split('/').filter(|s| !s.is_empty()).collect();
     if components.is_empty() {
@@ -241,9 +284,9 @@ pub fn resolve_path(
                     return Ok(entry.object_header_address);
                 }
                 // Not last — must be a group, parse its object header to get symbol table
-                let obj_header = ObjectHeader::parse(
+                let obj_header = ObjectHeader::parse_in(
                     file_data,
-                    to_usize(entry.object_header_address)?,
+                    checked_addr(entry.object_header_address)?,
                     offset_size,
                     length_size,
                 )?;

crates/clawhdf5-format/src/group_v2.rs

@@ -11,8 +11,8 @@ use alloc::collections::BTreeSet;
 #[cfg(feature = "std")]
 use std::collections::BTreeSet;
 
-use crate::addr::to_usize;
+use crate::addr::checked_addr;
-use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records, find_btree_v2_records};
+use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records_in, find_btree_v2_records_in};
 use crate::checksum::jenkins_lookup3;
 use crate::error::FormatError;
 use crate::fractal_heap::FractalHeapHeader;
@@ -21,6 +21,7 @@ use crate::link_info::LinkInfoMessage;
 use crate::link_message::{LinkMessage, LinkTarget};
 use crate::message_type::MessageType;
 use crate::object_header::ObjectHeader;
+use crate::storage::Storage;
 use crate::superblock::Superblock;
 use crate::symbol_table::SymbolTableMessage;
 
@@ -32,6 +33,16 @@ pub fn resolve_v2_group_entries(
     object_header: &ObjectHeader,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    resolve_v2_group_entries_in(file_data, object_header, offset_size, length_size)
+}
+
+/// [`resolve_v2_group_entries`] over any [`Storage`].
+pub fn resolve_v2_group_entries_in<S: Storage + ?Sized>(
+    file_data: &S,
+    object_header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<GroupEntry>, FormatError> {
     // Look for Link Info message to determine storage type
     let link_info = find_link_info(object_header, offset_size)?;
@@ -91,8 +102,8 @@ fn resolve_compact_entries(
 }
 
 /// Visit every link in dense storage (fractal heap + B-tree v2 name index).
-fn for_each_dense_link(
+fn for_each_dense_link<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     link_info: &LinkInfoMessage,
     fh_addr: u64,
     offset_size: u8,
@@ -100,15 +111,20 @@ fn for_each_dense_link(
     mut visit: impl FnMut(LinkMessage),
 ) -> Result<(), FormatError> {
     // Parse fractal heap
-    let fh = FractalHeapHeader::parse(file_data, to_usize(fh_addr)?, offset_size, length_size)?;
+    let fh =
+        FractalHeapHeader::parse_in(file_data, checked_addr(fh_addr)?, offset_size, length_size)?;
 
     // Parse B-tree v2 for name index
     let btree_addr = link_info
         .btree_name_index_address
         .ok_or_else(|| FormatError::PathNotFound(String::from("no B-tree v2 name index")))?;
-    let btree_hdr =
+    let btree_hdr = BTreeV2Header::parse_in(
-        BTreeV2Header::parse(file_data, to_usize(btree_addr)?, offset_size, length_size)?;
+        file_data,
-    let records = collect_btree_v2_records(file_data, &btree_hdr, offset_size, length_size)?;
+        checked_addr(btree_addr)?,
+        offset_size,
+        length_size,
+    )?;
+    let records = collect_btree_v2_records_in(file_data, &btree_hdr, offset_size, length_size)?;
 
     for record in &records {
         // For type 5 (name index): hash(4) + heap_id(heap_id_length)
@@ -125,7 +141,7 @@ fn for_each_dense_link(
         let id_bytes = &record.data[id_offset..id_offset + fh.heap_id_length as usize];
 
         // Read managed object from fractal heap
-        let link_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
+        let link_data = fh.read_managed_object_in(file_data, id_bytes, offset_size)?;
         if let Some(link) = parse_link(&link_data, offset_size)? {
             visit(link);
         }
@@ -134,8 +150,8 @@ fn for_each_dense_link(
 }
 
 /// Resolve entries from dense storage (fractal heap + B-tree v2).
-fn resolve_dense_entries(
+fn resolve_dense_entries<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     link_info: &LinkInfoMessage,
     fh_addr: u64,
     offset_size: u8,
@@ -167,8 +183,8 @@ fn resolve_dense_entries(
 /// The soft link called `name` in a v1 (symbol table) group, if there is
 /// one. Hard links are what `resolve_group_entries` returns; this is
 /// consulted only when a path component isn't among them.
-fn find_v1_symbolic_link(
+fn find_v1_symbolic_link<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     object_header: &ObjectHeader,
     name: &str,
     offset_size: u8,
@@ -182,7 +198,7 @@ fn find_v1_symbolic_link(
         return Ok(None);
     };
     let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
-    group_v1::find_v1_soft_link(file_data, &stm, name, offset_size, length_size)
+    group_v1::find_v1_soft_link_in(file_data, &stm, name, offset_size, length_size)
         .map(|target| target.map(|target_path| LinkTarget::Soft { target_path }))
 }
 
@@ -199,8 +215,8 @@ const LINK_NAME_INDEX: u8 = 5;
 /// link. libhdf5 orders records with equal hashes by name; all of them are
 /// read and compared here, so that order does not matter. An index of
 /// another type is scanned in full.
-fn links_named(
+fn links_named<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     object_header: &ObjectHeader,
     name: &str,
     offset_size: u8,
@@ -220,12 +236,17 @@ fn links_named(
         return Ok(found);
     };
 
-    let fh = FractalHeapHeader::parse(file_data, to_usize(fh_addr)?, offset_size, length_size)?;
+    let fh =
+        FractalHeapHeader::parse_in(file_data, checked_addr(fh_addr)?, offset_size, length_size)?;
     let btree_addr = link_info
         .btree_name_index_address
         .ok_or_else(|| FormatError::PathNotFound(String::from("no B-tree v2 name index")))?;
-    let btree_hdr =
+    let btree_hdr = BTreeV2Header::parse_in(
-        BTreeV2Header::parse(file_data, to_usize(btree_addr)?, offset_size, length_size)?;
+        file_data,
+        checked_addr(btree_addr)?,
+        offset_size,
+        length_size,
+    )?;
     if btree_hdr.tree_type != LINK_NAME_INDEX {
         for_each_dense_link(
             file_data,
@@ -244,7 +265,7 @@ fn links_named(
 
     // Record: hash(4) + heap ID.
     let hash = jenkins_lookup3(name.as_bytes());
-    let records = find_btree_v2_records(file_data, &btree_hdr, offset_size, &mut |r| {
+    let records = find_btree_v2_records_in(file_data, &btree_hdr, offset_size, &mut |r| {
         match r.get(..4) {
             Some(h) => u32::from_le_bytes([h[0], h[1], h[2], h[3]]).cmp(&hash),
             // Too short to hold a hash (a corrupt record size): never a match.
@@ -256,7 +277,7 @@ fn links_named(
         let Some(id_bytes) = record.data.get(4..4 + id_len) else {
             continue;
         };
-        let link_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
+        let link_data = fh.read_managed_object_in(file_data, id_bytes, offset_size)?;
         if let Some(link) = parse_link(&link_data, offset_size)?
             && link.name == name
         {
@@ -278,8 +299,8 @@ fn links_named(
 /// and may land on another of several exact duplicates. The listing
 /// ([`resolve_group_children`]), [`resolve_child`] and path resolution all
 /// apply this rule, so they agree.
-fn first_link_named(
+fn first_link_named<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     object_header: &ObjectHeader,
     name: &str,
     offset_size: u8,
@@ -296,8 +317,8 @@ fn first_link_named(
 /// the group with header `object_header`: a hard link (as `Hard`), else a
 /// soft or external link of that name, else `None`. Fails with
 /// `PathNotFound` if the object is not a group.
-fn lookup_link(
+fn lookup_link<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     object_header: &ObjectHeader,
     name: &str,
     offset_size: u8,
@@ -346,13 +367,38 @@ pub fn resolve_child(
     superblock: &Superblock,
     group_address: u64,
     name: &str,
+) -> Result<u64, FormatError> {
+    resolve_child_core(file_data, superblock, group_address, name)
+}
+
+/// [`resolve_child`] over any [`Storage`]. One with the whole file in memory
+/// is read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
+pub fn resolve_child_in<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
+    name: &str,
+) -> Result<u64, FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => resolve_child(all, superblock, group_address, name),
+        None => resolve_child_core(file_data, superblock, group_address, name),
+    }
+}
+
+fn resolve_child_core<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
+    name: &str,
 ) -> Result<u64, FormatError> {
     let os = superblock.offset_size;
     let ls = superblock.length_size;
     let not_found = || FormatError::PathNotFound(String::from(name));
-    let header = ObjectHeader::parse(file_data, to_usize(group_address)?, os, ls)?;
+    let header = ObjectHeader::parse_in(file_data, checked_addr(group_address)?, os, ls)?;
     if !is_v2_group(&header) || is_v1_group(&header) {
-        return resolve_group_children(file_data, superblock, group_address)?
+        return resolve_group_children_in(file_data, superblock, group_address)?
             .into_iter()
             .find(|e| e.name == name)
             .map(|e| e.object_header_address)
@@ -365,7 +411,7 @@ pub fn resolve_child(
             object_header_address,
         }) => Ok(object_header_address),
         Some(LinkTarget::Soft { target_path }) => {
-            match resolve_path_from(file_data, superblock, group_address, &target_path) {
+            match resolve_path_from_in(file_data, superblock, group_address, &target_path) {
                 // Left out of the listing: dangling, cyclic, or in another file.
                 Err(
                     FormatError::PathNotFound(_)
@@ -421,6 +467,29 @@ pub fn resolve_path_any(
     file_data: &[u8],
     superblock: &Superblock,
     path: &str,
+) -> Result<u64, FormatError> {
+    resolve_path_any_core(file_data, superblock, path)
+}
+
+/// [`resolve_path_any`] over any [`Storage`]. One with the whole file in memory
+/// is read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
+pub fn resolve_path_any_in<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    path: &str,
+) -> Result<u64, FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => resolve_path_any(all, superblock, path),
+        None => resolve_path_any_core(file_data, superblock, path),
+    }
+}
+
+fn resolve_path_any_core<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    path: &str,
 ) -> Result<u64, FormatError> {
     resolve_path_following_links(
         file_data,
@@ -439,6 +508,16 @@ pub fn resolve_path_from(
     superblock: &Superblock,
     group_address: u64,
     path: &str,
+) -> Result<u64, FormatError> {
+    resolve_path_from_in(file_data, superblock, group_address, path)
+}
+
+/// [`resolve_path_from`] over any [`Storage`].
+pub fn resolve_path_from_in<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
+    path: &str,
 ) -> Result<u64, FormatError> {
     let start = if path.starts_with('/') {
         superblock.root_group_address
@@ -462,10 +541,33 @@ pub fn resolve_group_children(
     file_data: &[u8],
     superblock: &Superblock,
     group_address: u64,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    resolve_group_children_core(file_data, superblock, group_address)
+}
+
+/// [`resolve_group_children`] over any [`Storage`]. One with the whole file in memory
+/// is read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
+pub fn resolve_group_children_in<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => resolve_group_children(all, superblock, group_address),
+        None => resolve_group_children_core(file_data, superblock, group_address),
+    }
+}
+
+fn resolve_group_children_core<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
 ) -> Result<Vec<GroupEntry>, FormatError> {
     let os = superblock.offset_size;
     let ls = superblock.length_size;
-    let header = ObjectHeader::parse(file_data, to_usize(group_address)?, os, ls)?;
+    let header = ObjectHeader::parse_in(file_data, checked_addr(group_address)?, os, ls)?;
 
     let mut entries = Vec::new();
     let mut soft = Vec::new();
@@ -476,9 +578,9 @@ pub fn resolve_group_children(
             .find(|m| m.msg_type == MessageType::SymbolTable)
             .ok_or_else(|| FormatError::PathNotFound(String::from("no symbol table message")))?;
         let stm = SymbolTableMessage::parse(&sym_msg.data, os)?;
-        let all = group_v1::resolve_v1_group_entries(file_data, &stm, os, ls)?;
+        let all = group_v1::resolve_v1_group_entries_in(file_data, &stm, os, ls)?;
         if all.iter().any(group_v1::is_v1_soft_link) {
-            soft = group_v1::v1_soft_links(file_data, &stm, os, ls)?;
+            soft = group_v1::v1_soft_links_in(file_data, &stm, os, ls)?;
         }
         entries.extend(all.into_iter().filter(|e| !group_v1::is_v1_soft_link(e)));
     } else if is_v2_group(&header) {
@@ -515,7 +617,7 @@ pub fn resolve_group_children(
     }
 
     for (name, target) in soft {
-        match resolve_path_from(file_data, superblock, group_address, &target) {
+        match resolve_path_from_in(file_data, superblock, group_address, &target) {
             Ok(object_header_address) => entries.push(GroupEntry {
                 name,
                 object_header_address,
@@ -538,8 +640,8 @@ pub fn resolve_group_children(
 const MAX_SOFT_LINK_DEPTH: u8 = 16;
 
 /// Walk `path` from the group at `start`, following soft links.
-fn resolve_path_following_links(
+fn resolve_path_following_links<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     superblock: &Superblock,
     start: u64,
     path: &str,
@@ -557,7 +659,7 @@ fn resolve_path_following_links(
     let ls = superblock.length_size;
 
     let mut current_addr = start;
-    let mut current_header = ObjectHeader::parse(file_data, to_usize(start)?, os, ls)?;
+    let mut current_header = ObjectHeader::parse_in(file_data, checked_addr(start)?, os, ls)?;
 
     for (i, component) in components.iter().enumerate() {
         match lookup_link(file_data, &current_header, component, os, ls)? {
@@ -568,7 +670,8 @@ fn resolve_path_following_links(
                     return Ok(object_header_address);
                 }
                 current_addr = object_header_address;
-                current_header = ObjectHeader::parse(file_data, to_usize(current_addr)?, os, ls)?;
+                current_header =
+                    ObjectHeader::parse_in(file_data, checked_addr(current_addr)?, os, ls)?;
             }
             found => {
                 return match found {
@@ -607,8 +710,8 @@ fn resolve_path_following_links(
 }
 
 /// Resolve group entries from an object header, auto-detecting v1 vs v2.
-fn resolve_group_entries(
+fn resolve_group_entries<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     object_header: &ObjectHeader,
     offset_size: u8,
     length_size: u8,
@@ -625,7 +728,7 @@ fn resolve_group_entries(
         // skipped by the name comparison, as in libhdf5.
         group_v1::v1_group_entries(file_data, &stm, offset_size, length_size)
     } else if is_v2_group(object_header) {
-        resolve_v2_group_entries(file_data, object_header, offset_size, length_size)
+        resolve_v2_group_entries_in(file_data, object_header, offset_size, length_size)
     } else {
         Err(FormatError::PathNotFound(String::from(
             "object header is not a group",

crates/clawhdf5-format/src/object_header.rs

@@ -1,7 +1,9 @@
 //! HDF5 Object Header parsing (v1 and v2).
 
 #[cfg(not(feature = "std"))]
-use alloc::vec::Vec;
+use alloc::{boxed::Box, collections::BTreeSet, vec::Vec};
+#[cfg(feature = "std")]
+use std::collections::BTreeSet;
 
 use byteorder::{ByteOrder, LittleEndian};
 
@@ -114,25 +116,52 @@ impl ObjectHeader {
     /// Parse an object header at the given offset in the data buffer.
     ///
     /// `offset_size` and `length_size` come from the superblock.
+    #[inline]
     pub fn parse(
         data: &[u8],
         offset: usize,
         offset_size: u8,
         length_size: u8,
     ) -> Result<ObjectHeader, FormatError> {
-        Self::parse_in(data, offset as u64, offset_size, length_size)
+        Self::parse_slice(data, offset as u64, offset_size, length_size)
     }
 
     /// [`Self::parse`] over any [`Storage`].
     ///
     /// Reads the prefix (at most [`V2_PREFIX_MAX`] bytes, signature
     /// included), then each chunk as one bounded read, continuation chunks
-    /// included.
+    /// included. A storage with the whole file in memory is parsed as its
+    /// slice, by code compiled in this crate (see
+    /// [`crate::storage`], "Slice entry points").
+    #[inline]
     pub fn parse_in<S: Storage + ?Sized>(
         file: &S,
         offset: u64,
         offset_size: u8,
         length_size: u8,
+    ) -> Result<ObjectHeader, FormatError> {
+        match file.as_contiguous() {
+            Some(all) => Self::parse_slice(all, offset, offset_size, length_size),
+            None => Self::parse_storage(file, offset, offset_size, length_size),
+        }
+    }
+
+    /// [`Self::parse_storage`] for the slice, compiled in this crate: the
+    /// one copy [`Self::parse`] and [`Self::parse_in`] (in memory) call.
+    fn parse_slice(
+        data: &[u8],
+        offset: u64,
+        offset_size: u8,
+        length_size: u8,
+    ) -> Result<ObjectHeader, FormatError> {
+        Self::parse_storage(data, offset, offset_size, length_size)
+    }
+
+    fn parse_storage<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        offset_size: u8,
+        length_size: u8,
     ) -> Result<ObjectHeader, FormatError> {
         // The longest prefix of either version, in one read. It holds the
         // whole prefix or ends at the end of the file, so its bounds checks
@@ -196,7 +225,6 @@ impl ObjectHeader {
             header_data_size,
             offset_size,
             length_size,
-            MAX_V1_CONTINUATION_DEPTH,
             &mut messages,
         )?;
         // libhdf5 reads every message in the first chunk and refuses a header
@@ -230,25 +258,54 @@ impl ObjectHeader {
     /// 8; libhdf5 refuses a message that is not aligned, that runs past the
     /// end of the chunk, or leftover bytes too few for a message header (a
     /// "gap", which only version 2 allows).
-    #[allow(clippy::too_many_arguments)]
+    ///
+    /// Continuation chunks are read in the order their messages are found,
+    /// as `H5O_protect` loads them (so the messages keep libhdf5's order):
+    /// a queue of (address, length) pairs, each chunk read, parsed and
+    /// released before the next, so only one chunk buffer is alive at a
+    /// time whatever the storage. Every chunk must start at a new address
+    /// (else a cycle), and the chunks together may be no larger than the
+    /// file, so the bytes read stay within the file's size; a header of
+    /// more than [`MAX_V1_CHUNKS`] chunks is refused.
     fn parse_v1_chunk<S: Storage + ?Sized>(
         file: &S,
         offset: u64,
         length: usize,
         offset_size: u8,
         length_size: u8,
-        depth_remaining: u16,
         messages: &mut Vec<HeaderMessage>,
     ) -> Result<usize, FormatError> {
-        if depth_remaining == 0 {
+        // The chunks found so far are also the queue of chunks to read.
-            return Err(FormatError::NestingDepthExceeded);
+        let mut spans = ChunkSpans::new(file.len(), offset, length)?;
+        let mut chunk0_count = 0usize;
+        let mut next = 0usize;
+        while let Some((chunk_offset, chunk_length)) = spans.get(next) {
+            let chunk = read_exact_at(file, chunk_offset, chunk_length)?;
+            let count =
+                Self::parse_v1_messages(&chunk, offset_size, length_size, messages, &mut spans)?;
+            // Only the first chunk's messages are held to the prefix count.
+            if next == 0 {
+                chunk0_count = count;
+            }
+            next += 1;
         }
-        let chunk = read_exact_at(file, offset, length)?;
+        Ok(chunk0_count)
-        let data: &[u8] = &chunk;
+    }
-        let end = length;
+
+    /// The messages of one version-1 chunk: each checked and appended to
+    /// `messages` (NIL ones dropped), each continuation added to `spans`.
+    /// Returns how many messages (NIL ones included) the chunk holds.
+    #[inline(never)]
+    fn parse_v1_messages(
+        data: &[u8],
+        offset_size: u8,
+        length_size: u8,
+        messages: &mut Vec<HeaderMessage>,
+        spans: &mut ChunkSpans,
+    ) -> Result<usize, FormatError> {
+        let end = data.len();
         let mut pos = 0usize;
         let mut count = 0usize;
-
         while pos < end {
             if end - pos < V1_MSG_HEADER_SIZE {
                 return Err(FormatError::InvalidObjectHeader(
@@ -272,7 +329,6 @@ impl ObjectHeader {
             let body = &data[pos..pos + msg_data_size];
             check_message(1, msg_type_raw, msg_flags, body, offset_size, length_size)?;
             count += 1;
-
             let msg_type = MessageType::from_u16(msg_type_raw);
             if msg_type != MessageType::Nil {
                 messages.push(HeaderMessage {
@@ -283,25 +339,15 @@ impl ObjectHeader {
                     data: body.to_vec(),
                 });
             }
-            pos += msg_data_size;
+            // Queue continuations (v1 continuation chunks are just raw
-
-            // Follow continuations (v1 continuation chunks are just raw
             // messages, no signature); check_message has checked the body.
             if msg_type == MessageType::ObjectHeaderContinuation {
-                let cont_offset = to_usize(read_offset(body, 0, offset_size)?)?;
+                let cont_offset = read_offset(body, 0, offset_size)?;
                 let cont_length = to_usize(read_offset(body, offset_size as usize, length_size)?)?;
-                Self::parse_v1_chunk(
+                spans.add(cont_offset, cont_length)?;
-                    file,
-                    cont_offset as u64,
-                    cont_length,
-                    offset_size,
-                    length_size,
-                    depth_remaining - 1,
-                    messages,
-                )?;
             }
+            pos += msg_data_size;
         }
-
         Ok(count)
     }
 
@@ -425,13 +471,14 @@ impl ObjectHeader {
             &mut continuations,
         )?;
 
-        // Follow continuations (limit to prevent cycles in malformed data)
+        // Follow continuations, one chunk buffer at a time. A chunk address
-        let mut cont_remaining = 256u16;
+        // seen twice is a cycle in malformed data, and the chunks may add up
+        // to no more than the file; a valid header can have many chunks (libhdf5 adds one
+        // whenever a message no longer fits), up to the same bound as a
+        // version-1 header.
+        let mut spans = ChunkSpans::new(file.len(), base as u64, chunk0_msg_end.saturating_add(4))?;
         while let Some((cont_offset, cont_length)) = continuations.pop() {
-            if cont_remaining == 0 {
+            spans.add(cont_offset as u64, cont_length)?;
-                return Err(FormatError::NestingDepthExceeded);
-            }
-            cont_remaining -= 1;
             Self::parse_v2_continuation(
                 file,
                 cont_offset as u64,
@@ -594,8 +641,107 @@ const V2_PREFIX_MAX: usize = 34;
 /// Size of a version-1 message header: type(2) + size(2) + flags(1) + reserved(3).
 const V1_MSG_HEADER_SIZE: usize = 8;
 
-/// How deep version-1 continuation chunks may chain (malformed-data guard).
+/// The chunks of one object header read so far, in the order they were
-const MAX_V1_CONTINUATION_DEPTH: u16 = 32;
+/// found (which is the order version-1 chunks are read in). A chunk starting
+/// where another did is a cycle. Chunks of a valid header do not overlap, so
+/// together they are no larger than the file; a header whose chunks add up
+/// to more is refused, which bounds what its chunks can make a reader read
+/// (a crafted chain of chunks each nested in the last would otherwise read
+/// the file over and over). Overlap itself is not refused: libhdf5 reads
+/// such headers (`cve-2025-7067.h5` has one).
+///
+/// Almost every header has at most a few chunks, and this runs once per
+/// header, so the first [`INLINE_CHUNKS`] live in an inline array and are
+/// checked for cycles by a scan; only a longer header allocates (the rest
+/// of the list, and a set of starts). Allocating a queue and a set for
+/// every header made parsing 401 small headers 1.8x slower.
+struct ChunkSpans {
+    inline: [(u64, usize); INLINE_CHUNKS],
+    /// Chunks after the first [`INLINE_CHUNKS`], and every chunk start.
+    spill: Option<Box<SpilledSpans>>,
+    /// How many chunks there are.
+    len: usize,
+    /// Bytes of the chunks so far, and the most they may add up to.
+    total: u64,
+    budget: u64,
+}
+
+/// The chunks of a [`ChunkSpans`] beyond its inline ones.
+struct SpilledSpans {
+    chunks: Vec<(u64, usize)>,
+    starts: BTreeSet<u64>,
+}
+
+/// How many chunks [`ChunkSpans`] holds without allocating.
+const INLINE_CHUNKS: usize = 8;
+
+impl ChunkSpans {
+    #[inline]
+    fn new(file_len: u64, start: u64, len: usize) -> Result<Self, FormatError> {
+        let mut s = Self {
+            inline: [(0, 0); INLINE_CHUNKS],
+            spill: None,
+            len: 0,
+            total: 0,
+            budget: file_len,
+        };
+        s.add(start, len)?;
+        Ok(s)
+    }
+
+    /// Record the chunk `len` bytes at `start`.
+    #[inline]
+    fn add(&mut self, start: u64, len: usize) -> Result<(), FormatError> {
+        self.total = self.total.saturating_add(len as u64);
+        if self.len < INLINE_CHUNKS {
+            if self.inline[..self.len].iter().any(|&(s, _)| s == start) {
+                return Err(FormatError::NestingDepthExceeded);
+            }
+            self.inline[self.len] = (start, len);
+        } else {
+            self.add_spilled(start, len)?;
+        }
+        self.len += 1;
+        if self.total > self.budget {
+            return Err(FormatError::InvalidObjectHeader(
+                "object header chunks larger than the file",
+            ));
+        }
+        Ok(())
+    }
+
+    #[cold]
+    #[inline(never)]
+    fn add_spilled(&mut self, start: u64, len: usize) -> Result<(), FormatError> {
+        let inline = &self.inline;
+        let spill = self.spill.get_or_insert_with(|| {
+            Box::new(SpilledSpans {
+                chunks: Vec::new(),
+                starts: inline.iter().map(|&(s, _)| s).collect(),
+            })
+        });
+        if !spill.starts.insert(start) || self.len >= MAX_V1_CHUNKS {
+            return Err(FormatError::NestingDepthExceeded);
+        }
+        spill.chunks.push((start, len));
+        Ok(())
+    }
+
+    /// The `i`th chunk recorded.
+    #[inline]
+    fn get(&self, i: usize) -> Option<(u64, usize)> {
+        if i < INLINE_CHUNKS {
+            (i < self.len).then(|| self.inline[i])
+        } else {
+            self.spill.as_ref()?.chunks.get(i - INLINE_CHUNKS).copied()
+        }
+    }
+}
+
+/// Most chunks a version-1 object header may have (malformed-data guard;
+/// libhdf5 has no limit, and a header that gains one continuation chunk per
+/// attribute added can have many).
+const MAX_V1_CHUNKS: usize = 1 << 16;
 
 /// Every defined version-2 object header status flag (libhdf5
 /// `H5O_HDR_ALL_FLAGS`): chunk-0 size width (bits 0-1), attribute creation
@@ -901,6 +1047,176 @@ mod tests {
         assert_eq!(hdr.messages[1].data[..2], [5, 6]);
     }
 
+    /// A version-1 header whose continuation chunks form a chain: chunk k
+    /// holds a Dataspace message `[k]` and the continuation to chunk k + 1.
+    /// With `cycle`, the last chunk points back at the first continuation
+    /// chunk.
+    fn v1_chain(n: usize, cycle: bool) -> Vec<u8> {
+        // Each continuation chunk: dataspace (8 + 8) + continuation (8 + 16).
+        let chunk_len = 40u64;
+        let first = 64u64;
+        let cont = |addr: u64| {
+            let mut b = addr.to_le_bytes().to_vec();
+            b.extend_from_slice(&chunk_len.to_le_bytes());
+            b
+        };
+        let mut data = build_v1_header(&[(0x0010, &cont(first)[..], 0)], 8, 8);
+        data.resize(first as usize, 0);
+        for k in 0..n {
+            let mut c = Vec::new();
+            c.extend_from_slice(&1u16.to_le_bytes());
+            c.extend_from_slice(&8u16.to_le_bytes());
+            c.extend_from_slice(&[0; 4]);
+            c.extend_from_slice(&(k as u64).to_le_bytes());
+            let next = if k + 1 < n {
+                first + (k as u64 + 1) * chunk_len
+            } else if cycle {
+                first
+            } else {
+                // The last chunk ends in a NIL message instead.
+                c.extend_from_slice(&[0, 0, 16, 0, 0, 0, 0, 0]);
+                c.extend_from_slice(&[0; 16]);
+                data.extend_from_slice(&c);
+                continue;
+            };
+            c.extend_from_slice(&0x10u16.to_le_bytes());
+            c.extend_from_slice(&16u16.to_le_bytes());
+            c.extend_from_slice(&[0; 4]);
+            c.extend_from_slice(&cont(next));
+            data.extend_from_slice(&c);
+        }
+        data
+    }
+
+    /// libhdf5 reads any chain of continuation chunks (a header grows one
+    /// per attribute added when full); the reader used to stop at 32.
+    #[test]
+    fn long_v1_continuation_chains_are_read() {
+        let data = v1_chain(200, false);
+        let hdr = ObjectHeader::parse(&data, 0, 8, 8).unwrap();
+        let spaces: Vec<u8> = hdr
+            .messages
+            .iter()
+            .filter(|m| m.msg_type == MessageType::Dataspace)
+            .map(|m| m.data[0])
+            .collect();
+        assert_eq!(spaces, (0..200).map(|k| k as u8).collect::<Vec<_>>());
+    }
+
+    /// A crafted version-1 header whose continuation chunks nest: each
+    /// chunk's continuation message points at the rest of that chunk. Read
+    /// depth-first with every enclosing chunk kept alive, from storage that
+    /// hands out owned buffers, it read n^2 bytes and held them all at once
+    /// (a 192 KB file read 768 MB). Chunks adding up to more than the file
+    /// are refused, and the bytes read stay within the file's size.
+    #[test]
+    fn nested_v1_continuation_chunks_are_bounded() {
+        use crate::storage::CountingStorage;
+        let n = 2000u64;
+        let a = 64u64;
+        let cont = |addr: u64, len: u64| {
+            let mut m = vec![0x10, 0, 16, 0, 0, 0, 0, 0];
+            m.extend_from_slice(&addr.to_le_bytes());
+            m.extend_from_slice(&len.to_le_bytes());
+            m
+        };
+        // Prefix: version 1, one message, reference count 1, 24 bytes.
+        let mut buf = vec![1, 0, 1, 0, 1, 0, 0, 0, 24, 0, 0, 0, 0, 0, 0, 0];
+        buf.extend_from_slice(&cont(a, 24 * n));
+        buf.resize(a as usize, 0);
+        for k in 0..n {
+            if k + 1 < n {
+                buf.extend_from_slice(&cont(a + 24 * (k + 1), 24 * (n - k - 1)));
+            } else {
+                buf.extend_from_slice(&[0, 0, 16, 0, 0, 0, 0, 0]);
+                buf.extend_from_slice(&[0; 16]);
+            }
+        }
+        let len = buf.len() as u64;
+        let s = CountingStorage::new(buf);
+        assert!(matches!(
+            ObjectHeader::parse_in(&s, 0, 8, 8),
+            Err(FormatError::InvalidObjectHeader(
+                "object header chunks larger than the file"
+            ))
+        ));
+        assert!(
+            s.bytes_read() <= 2 * len,
+            "read {} of {len}",
+            s.bytes_read()
+        );
+    }
+
+    /// libhdf5 reads a continuation chunk that overlaps the chunk holding
+    /// its message (`cve-2025-7067.h5` has one), and so does this reader.
+    #[test]
+    fn overlapping_v1_continuation_chunk_is_read() {
+        // Chunk 0 (at 16): continuation (24 bytes), then a NIL message at
+        // 40; the continuation chunk is that NIL message's 8-byte header.
+        let mut cont = 40u64.to_le_bytes().to_vec();
+        cont.extend_from_slice(&8u64.to_le_bytes());
+        let data = build_v1_header(&[(0x0010, &cont[..], 0), (0x0000, &[][..], 0)], 8, 8);
+        let hdr = ObjectHeader::parse(&data, 0, 8, 8).unwrap();
+        assert_eq!(hdr.messages.len(), 1);
+    }
+
+    /// A valid chain over owned-buffer storage reads each chunk once.
+    #[test]
+    fn long_v1_chain_reads_each_chunk_once() {
+        use crate::storage::CountingStorage;
+        let data = v1_chain(3000, false);
+        let len = data.len() as u64;
+        let s = CountingStorage::new(data);
+        let hdr = ObjectHeader::parse_in(&s, 0, 8, 8).unwrap();
+        assert_eq!(
+            hdr.messages
+                .iter()
+                .filter(|m| m.msg_type == MessageType::Dataspace)
+                .count(),
+            3000
+        );
+        assert!(s.bytes_read() <= len, "read {} of {len}", s.bytes_read());
+    }
+
+    /// Continuation chunks are read in the order their messages are found
+    /// (libhdf5's `H5O_protect`), so a chunk's messages follow every
+    /// message of the chunk before, not the continuation message.
+    #[test]
+    fn v1_continuation_messages_keep_libhdf5_order() {
+        // Chunk 0: continuation to A, dataspace [1]; A: dataspace [2].
+        let a = 64u64;
+        let mut cont = a.to_le_bytes().to_vec();
+        cont.extend_from_slice(&16u64.to_le_bytes());
+        let mut data = build_v1_header(&[(0x0010, &cont[..], 0), (0x0001, &[1; 8][..], 0)], 8, 8);
+        data.resize(a as usize, 0);
+        data.extend_from_slice(&[1, 0, 8, 0, 0, 0, 0, 0]);
+        data.extend_from_slice(&[2; 8]);
+        let hdr = ObjectHeader::parse(&data, 0, 8, 8).unwrap();
+        let spaces: Vec<u8> = hdr
+            .messages
+            .iter()
+            .filter(|m| m.msg_type == MessageType::Dataspace)
+            .map(|m| m.data[0])
+            .collect();
+        assert_eq!(spaces, [1, 2]);
+    }
+
+    #[test]
+    fn v1_continuation_cycles_are_refused() {
+        // Within the inline chunk list, and past it (the cycle returns to
+        // an inline chunk once the list has spilled).
+        for n in [5, 7, 8, 9, 40] {
+            let data = v1_chain(n, true);
+            assert!(
+                matches!(
+                    ObjectHeader::parse(&data, 0, 8, 8),
+                    Err(FormatError::NestingDepthExceeded)
+                ),
+                "{n} chunks"
+            );
+        }
+    }
+
     #[test]
     fn parse_v1_unknown_message_ok() {
         let messages = [(0x00FFu16, &[0xAA, 0xBB][..], 0u8)];

crates/clawhdf5-format/src/parallel_read.rs

@@ -7,12 +7,27 @@
 //! The lane assignment is seeded by dataset metadata so repeated reads of
 //! the same region produce identical partitions (cache-friendly, reproducible).
 
-use crate::addr::to_usize;
 use crate::chunked_read::ChunkInfo;
 use crate::error::FormatError;
 use crate::filter_pipeline::FilterPipeline;
 use crate::filters::decompress_chunk_exact;
 use crate::lane_partition::{self, LaneStats, PartitionStats};
+use crate::storage::{ExtentReq, Storage, for_each_extent_batch};
+
+/// The extents of `chunks`' stored bytes (see
+/// [`crate::chunked_read::chunk_req`]), fetched batch by batch with
+/// [`for_each_extent_batch`] when the file is not in memory (each chunk's
+/// bounds error is reported when that chunk is decoded, as before).
+fn chunk_reqs(
+    chunks: &[ChunkInfo],
+    pipeline: Option<&FilterPipeline>,
+    chunk_total_bytes: usize,
+) -> Vec<ExtentReq> {
+    chunks
+        .iter()
+        .map(|c| crate::chunked_read::chunk_req(c, pipeline, chunk_total_bytes, true))
+        .collect()
+}
 
 /// Threshold: only use parallel decompression when chunk count exceeds this.
 const PARALLEL_THRESHOLD: usize = 4;
@@ -190,6 +205,27 @@ pub fn decompress_chunks_lane_partitioned(
     element_size: u32,
     seed: u64,
     num_lanes: Option<usize>,
+) -> Result<(Vec<Vec<u8>>, PartitionStats), FormatError> {
+    decompress_chunks_lane_partitioned_in(
+        file_data,
+        chunks,
+        pipeline,
+        chunk_total_bytes,
+        element_size,
+        seed,
+        num_lanes,
+    )
+}
+
+/// [`decompress_chunks_lane_partitioned`] over any [`Storage`].
+pub fn decompress_chunks_lane_partitioned_in<S: Storage + ?Sized>(
+    file_data: &S,
+    chunks: &[ChunkInfo],
+    pipeline: &FilterPipeline,
+    chunk_total_bytes: usize,
+    element_size: u32,
+    seed: u64,
+    num_lanes: Option<usize>,
 ) -> Result<(Vec<Vec<u8>>, PartitionStats), FormatError> {
     use rayon::prelude::*;
 
@@ -199,72 +235,75 @@ pub fn decompress_chunks_lane_partitioned(
             .unwrap_or(1)
     });
 
-    let assignments = lane_partition::partition_chunks(chunks.len(), lanes, seed);
+    let reqs = chunk_reqs(chunks, Some(pipeline), chunk_total_bytes);
-    let num_lanes = assignments.len();
+    let mut ordered: Vec<Vec<u8>> = Vec::with_capacity(chunks.len());
-
+    let mut partition_stats = PartitionStats::new(0);
-    // Each lane processes its assigned chunks and returns results + stats.
-    let lane_results: Result<Vec<(Vec<DecompressedChunk>, LaneStats)>, FormatError> = assignments
-        .into_par_iter()
-        .map(|indices| {
-            let mut results = Vec::with_capacity(indices.len());
-            let mut stats = LaneStats::default();
-
-            for &index in &indices {
-                let chunk_info = &chunks[index];
-                let c_addr = to_usize(chunk_info.address)?;
-                let size = chunk_info.chunk_size as usize;
-
-                if c_addr
-                    .checked_add(size)
-                    .is_none_or(|end| end > file_data.len())
-                {
-                    return Err(FormatError::UnexpectedEof {
-                        expected: c_addr.saturating_add(size),
-                        available: file_data.len(),
-                    });
-                }
-                let raw_chunk = &file_data[c_addr..c_addr + size];
-
-                let decompressed = decompress_chunk_exact(
-                    raw_chunk,
-                    pipeline,
-                    chunk_total_bytes,
-                    element_size,
-                    chunk_info.filter_mask,
-                    &chunk_info.offsets,
-                )?;
-
-                stats.chunks_processed += 1;
-                stats.compressed_bytes += size as u64;
-                stats.decompressed_bytes += decompressed.len() as u64;
-
-                results.push(DecompressedChunk {
-                    index,
-                    data: decompressed,
-                });
-            }
-
-            Ok((results, stats))
-        })
-        .collect();
-
-    let lane_results = lane_results?;
-
-    // Aggregate stats
-    let mut partition_stats = PartitionStats::new(num_lanes);
     partition_stats.total_chunks = chunks.len();
-    for (lane_idx, (_, stats)) in lane_results.iter().enumerate() {
+    // Each batch of fetched chunks is partitioned into lanes and decoded
-        partition_stats.per_lane[lane_idx] = stats.clone();
+    // before the next batch is fetched (with the file in memory there is
-    }
+    // one batch: all the chunks).
+    for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
+        let assignments = lane_partition::partition_chunks(batch.len(), lanes, seed);
+        // Each lane processes its assigned chunks and returns results + stats.
+        let lane_results: Result<Vec<(Vec<DecompressedChunk>, LaneStats)>, FormatError> =
+            assignments
+                .into_par_iter()
+                .map(|indices| {
+                    let mut results = Vec::with_capacity(indices.len());
+                    let mut stats = LaneStats::default();
 
-    // Flatten and sort by original index to restore order
+                    for &local in &indices {
-    let mut all_chunks: Vec<DecompressedChunk> = lane_results
+                        let index = batch.start + local;
-        .into_iter()
+                        let chunk_info = &chunks[index];
-        .flat_map(|(chunks, _)| chunks)
+                        let size = chunk_info.chunk_size as usize;
-        .collect();
+                        let raw_chunk = raw_bytes.get(index, &reqs[index])?;
-    all_chunks.sort_by_key(|dc| dc.index);
 
-    let ordered = all_chunks.into_iter().map(|dc| dc.data).collect();
+                        let decompressed = decompress_chunk_exact(
+                            raw_chunk,
+                            pipeline,
+                            chunk_total_bytes,
+                            element_size,
+                            chunk_info.filter_mask,
+                            &chunk_info.offsets,
+                        )?;
+
+                        stats.chunks_processed += 1;
+                        stats.compressed_bytes += size as u64;
+                        stats.decompressed_bytes += decompressed.len() as u64;
+
+                        results.push(DecompressedChunk {
+                            index,
+                            data: decompressed,
+                        });
+                    }
+
+                    Ok((results, stats))
+                })
+                .collect();
+        let lane_results = lane_results?;
+
+        // Aggregate stats
+        if partition_stats.per_lane.len() < lane_results.len() {
+            partition_stats
+                .per_lane
+                .resize_with(lane_results.len(), LaneStats::default);
+            partition_stats.num_lanes = lane_results.len();
+        }
+        for (lane, (_, stats)) in partition_stats.per_lane.iter_mut().zip(&lane_results) {
+            lane.chunks_processed += stats.chunks_processed;
+            lane.compressed_bytes += stats.compressed_bytes;
+            lane.decompressed_bytes += stats.decompressed_bytes;
+        }
+
+        // Flatten and sort by original index to restore order
+        let mut all_chunks: Vec<DecompressedChunk> = lane_results
+            .into_iter()
+            .flat_map(|(chunks, _)| chunks)
+            .collect();
+        all_chunks.sort_by_key(|dc| dc.index);
+        ordered.extend(all_chunks.into_iter().map(|dc| dc.data));
+        Ok(())
+    })?;
     Ok((ordered, partition_stats))
 }
 
@@ -282,45 +321,52 @@ pub fn decompress_chunks_parallel(
     pipeline: &FilterPipeline,
     chunk_total_bytes: usize,
     element_size: u32,
+) -> Result<Vec<Vec<u8>>, FormatError> {
+    decompress_chunks_parallel_in(file_data, chunks, pipeline, chunk_total_bytes, element_size)
+}
+
+/// [`decompress_chunks_parallel`] over any [`Storage`].
+pub fn decompress_chunks_parallel_in<S: Storage + ?Sized>(
+    file_data: &S,
+    chunks: &[ChunkInfo],
+    pipeline: &FilterPipeline,
+    chunk_total_bytes: usize,
+    element_size: u32,
 ) -> Result<Vec<Vec<u8>>, FormatError> {
     use rayon::prelude::*;
 
-    let results: Result<Vec<DecompressedChunk>, FormatError> = chunks
+    let reqs = chunk_reqs(chunks, Some(pipeline), chunk_total_bytes);
-        .par_iter()
+    let mut ordered: Vec<Vec<u8>> = Vec::with_capacity(chunks.len());
-        .enumerate()
+    for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
-        .map(|(index, chunk_info)| {
+        let results: Result<Vec<DecompressedChunk>, FormatError> = batch
-            let c_addr = to_usize(chunk_info.address)?;
+            .clone()
-            let size = chunk_info.chunk_size as usize;
+            .into_par_iter()
-            if c_addr
+            .map(|index| {
-                .checked_add(size)
+                let chunk_info = &chunks[index];
-                .is_none_or(|end| end > file_data.len())
+                let raw_chunk = raw_bytes.get(index, &reqs[index])?;
-            {
-                return Err(FormatError::UnexpectedEof {
-                    expected: c_addr.saturating_add(size),
-                    available: file_data.len(),
-                });
-            }
-            let raw_chunk = &file_data[c_addr..c_addr + size];
 
-            let decompressed = decompress_chunk_exact(
+                let decompressed = decompress_chunk_exact(
-                raw_chunk,
+                    raw_chunk,
-                pipeline,
+                    pipeline,
-                chunk_total_bytes,
+                    chunk_total_bytes,
-                element_size,
+                    element_size,
-                chunk_info.filter_mask,
+                    chunk_info.filter_mask,
-                &chunk_info.offsets,
+                    &chunk_info.offsets,
-            )?;
+                )?;
 
-            Ok(DecompressedChunk {
+                Ok(DecompressedChunk {
-                index,
+                    index,
-                data: decompressed,
+                    data: decompressed,
+                })
             })
-        })
+            .collect();
-        .collect();
 
-    let mut result_vec = results?;
+        let mut result_vec = results?;
-    result_vec.sort_by_key(|dc| dc.index);
+        result_vec.sort_by_key(|dc| dc.index);
-    Ok(result_vec.into_iter().map(|dc| dc.data).collect())
+        ordered.extend(result_vec.into_iter().map(|dc| dc.data));
+        Ok(())
+    })?;
+    Ok(ordered)
 }
 
 /// Decompress chunks sequentially (fallback when parallel is not warranted).
@@ -331,35 +377,40 @@ pub fn decompress_chunks_sequential(
     chunk_total_bytes: usize,
     element_size: u32,
 ) -> Result<Vec<Vec<u8>>, FormatError> {
-    let mut result = Vec::with_capacity(chunks.len());
+    decompress_chunks_sequential_in(file_data, chunks, pipeline, chunk_total_bytes, element_size)
-    for chunk_info in chunks {
+}
-        let c_addr = to_usize(chunk_info.address)?;
-        let size = chunk_info.chunk_size as usize;
-        if c_addr
-            .checked_add(size)
-            .is_none_or(|end| end > file_data.len())
-        {
-            return Err(FormatError::UnexpectedEof {
-                expected: c_addr.saturating_add(size),
-                available: file_data.len(),
-            });
-        }
-        let raw_chunk = &file_data[c_addr..c_addr + size];
 
-        let decompressed = if let Some(pl) = pipeline {
+/// [`decompress_chunks_sequential`] over any [`Storage`].
-            decompress_chunk_exact(
+pub fn decompress_chunks_sequential_in<S: Storage + ?Sized>(
-                raw_chunk,
+    file_data: &S,
-                pl,
+    chunks: &[ChunkInfo],
-                chunk_total_bytes,
+    pipeline: Option<&FilterPipeline>,
-                element_size,
+    chunk_total_bytes: usize,
-                chunk_info.filter_mask,
+    element_size: u32,
-                &chunk_info.offsets,
+) -> Result<Vec<Vec<u8>>, FormatError> {
-            )?
+    let reqs = chunk_reqs(chunks, pipeline, chunk_total_bytes);
-        } else {
+    let mut result = Vec::with_capacity(chunks.len());
-            raw_chunk.to_vec()
+    for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
-        };
+        for i in batch {
-        result.push(decompressed);
+            let chunk_info = &chunks[i];
-    }
+            let raw_chunk = raw_bytes.get(i, &reqs[i])?;
+
+            let decompressed = if let Some(pl) = pipeline {
+                decompress_chunk_exact(
+                    raw_chunk,
+                    pl,
+                    chunk_total_bytes,
+                    element_size,
+                    chunk_info.filter_mask,
+                    &chunk_info.offsets,
+                )?
+            } else {
+                raw_chunk.to_vec()
+            };
+            result.push(decompressed);
+        }
+        Ok(())
+    })?;
     Ok(result)
 }
 

crates/clawhdf5-format/src/partial_read.rs

@@ -18,7 +18,7 @@ use alloc::{format, vec, vec::Vec};
 #[cfg(feature = "std")]
 use std::string as alloc_or_std;
 
-use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks_for_read};
+use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks_for_read_in};
 use crate::data_layout::DataLayout;
 use crate::data_read::extract_selection_from_buffer;
 use crate::dataspace::Dataspace;
@@ -26,6 +26,7 @@ use crate::error::FormatError;
 use crate::filter_pipeline::FilterPipeline;
 use crate::filters::{all_filters_skipped, decompress_chunk_exact_with};
 use crate::selection::Selection;
+use crate::storage::{ExtentReq, Storage, for_each_extent_batch};
 
 /// The smallest axis-aligned box containing every selected element, as
 /// `(start, extent)` per dimension. `None` when there is nothing to gain or
@@ -256,6 +257,30 @@ pub fn read_selection(
     offset_size: u8,
     length_size: u8,
     selection: &Selection,
+) -> Result<Option<Vec<u8>>, FormatError> {
+    read_selection_in(
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        pipeline,
+        offset_size,
+        length_size,
+        selection,
+    )
+}
+
+/// [`read_selection`] over any [`Storage`].
+#[allow(clippy::too_many_arguments)]
+pub fn read_selection_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    selection: &Selection,
 ) -> Result<Option<Vec<u8>>, FormatError> {
     let dims = &dataspace.dimensions;
     if dims.is_empty() || elem_size == 0 {
@@ -276,14 +301,33 @@ pub fn read_selection(
         validate(selection, dims)?;
         let base = usize::try_from(*address)
             .map_err(|_| FormatError::Overflow("data address exceeds usize".into()))?;
-        let data = file_data
+        let file_len = crate::storage::len_usize(file_data);
-            .get(base..)
+        let eof = FormatError::UnexpectedEof {
-            .and_then(|d| d.get(..checked_byte_len(total, elem_size).ok()?))
+            expected: base,
-            .ok_or(FormatError::UnexpectedEof {
+            available: file_len,
-                expected: base,
+        };
-                available: file_data.len(),
+        if let Some(all) = file_data.as_contiguous() {
-            })?;
+            let data = all
-        return crate::gather::gather::<u8>(data, dims, elem_size, selection).map(Some);
+                .get(base..)
+                .and_then(|d| d.get(..checked_byte_len(total, elem_size).ok()?))
+                .ok_or(eof)?;
+            return crate::gather::gather::<u8>(data, dims, elem_size, selection).map(Some);
+        }
+        // Not in memory: the same bounds check, then only the selected runs
+        // are read.
+        let len = checked_byte_len(total, elem_size)
+            .ok()
+            .filter(|&len| base <= file_len && len <= file_len - base)
+            .ok_or(eof)?;
+        return crate::gather::gather_storage(
+            file_data,
+            base as u64,
+            len,
+            dims,
+            elem_size,
+            selection,
+        )
+        .map(Some);
     }
     let Some((box_start, box_extent)) = bounding_box(selection, dims) else {
         return Ok(None);
@@ -303,7 +347,7 @@ pub fn read_selection(
             btree_address: Some(_),
             ..
         } => {
-            let (chunks, chunk_dims) = list_chunks_for_read(
+            let (chunks, chunk_dims) = list_chunks_for_read_in(
                 file_data,
                 layout,
                 dataspace,
@@ -315,56 +359,64 @@ pub fn read_selection(
             let rank = dims.len();
             let chunk_shape: Vec<u64> = chunk_dims.iter().map(|&d| d as u64).collect();
             let chunk_bytes = crate::chunked_read::checked_chunk_byte_len(&chunk_dims, elem_size)?;
-            // Chunks are decoded into this thread's reusable buffers.
+            // The chunks overlapping the box, in index order.
-            crate::chunked_read::with_scratch(|scratch| -> Result<(), FormatError> {
+            let wanted: Vec<&crate::chunked_read::ChunkInfo> = chunks
-                for chunk in &chunks {
+                .iter()
+                .filter(|chunk| {
                     if chunk.offsets.len() < rank || chunk.address == u64::MAX {
-                        continue;
+                        return false;
                     }
                     let origin = &chunk.offsets[..rank];
-                    let overlaps = (0..rank).all(|d| {
+                    (0..rank).all(|d| {
                         origin[d] < box_start[d] + box_extent[d]
                             && origin[d].saturating_add(chunk_shape[d]) > box_start[d]
-                    });
+                    })
-                    if !overlaps {
+                })
-                        continue;
+                .collect();
-                    }
+            // Their stored bytes, batch by batch when the file is not in
-                    let at = usize::try_from(chunk.address)
+            // memory; each batch's chunks are decoded into this thread's
-                        .map_err(|_| FormatError::Overflow("chunk address exceeds usize".into()))?;
+            // reusable buffers before the next batch is fetched.
-                    let raw = at
+            let reqs: Vec<ExtentReq> = wanted
-                        .checked_add(chunk.chunk_size as usize)
+                .iter()
-                        .and_then(|end| file_data.get(at..end))
+                .map(|c| crate::chunked_read::chunk_req(c, pipeline, chunk_bytes, true))
-                        .ok_or(FormatError::UnexpectedEof {
+                .collect();
-                            expected: at.saturating_add(chunk.chunk_size as usize),
+            for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
-                            available: file_data.len(),
+                crate::chunked_read::with_scratch(|scratch| -> Result<(), FormatError> {
+                    for i in batch {
+                        let chunk = wanted[i];
+                        let origin = &chunk.offsets[..rank];
+                        usize::try_from(chunk.address).map_err(|_| {
+                            FormatError::Overflow("chunk address exceeds usize".into())
                         })?;
-                    // Mirrors the full-read path: filter-mask bit i set means
+                        let raw = raw_bytes.get(i, &reqs[i])?;
-                    // filter i was not applied to this chunk.
+                        // Mirrors the full-read path: filter-mask bit i set
-                    let data: &[u8] = match pipeline {
+                        // means filter i was not applied to this chunk.
-                        Some(pl) if !all_filters_skipped(pl, chunk.filter_mask) => {
+                        let data: &[u8] = match pipeline {
-                            decompress_chunk_exact_with(
+                            Some(pl) if !all_filters_skipped(pl, chunk.filter_mask) => {
-                                raw,
+                                decompress_chunk_exact_with(
-                                pl,
+                                    raw,
-                                chunk_bytes,
+                                    pl,
-                                elem_size as u32,
+                                    chunk_bytes,
-                                chunk.filter_mask,
+                                    elem_size as u32,
-                                &chunk.offsets[..rank],
+                                    chunk.filter_mask,
-                                scratch,
+                                    &chunk.offsets[..rank],
-                            )?
+                                    scratch,
-                        }
+                                )?
-                        _ => raw,
+                            }
-                    };
+                            _ => raw,
-                    copy_overlap(
+                        };
-                        data,
+                        copy_overlap(
-                        origin,
+                            data,
-                        &chunk_shape,
+                            origin,
-                        &mut boxed,
+                            &chunk_shape,
-                        &box_start,
+                            &mut boxed,
-                        &box_extent,
+                            &box_start,
-                        elem_size,
+                            &box_extent,
-                    );
+                            elem_size,
-                }
+                        );
-                Ok(())
+                    }
+                    Ok(())
+                })
             })?;
         }
         _ => return Ok(None),

crates/clawhdf5-format/src/provenance.rs

@@ -13,7 +13,6 @@ use sha2::{Digest, Sha256};
 
 use crate::attribute::AttributeMessage;
 use crate::data_layout::DataLayout;
-use crate::data_read::read_raw_data;
 use crate::dataspace::Dataspace;
 use crate::datatype::Datatype;
 use crate::error::FormatError;
@@ -128,10 +127,20 @@ pub fn verify_dataset(
     header: &ObjectHeader,
     offset_size: u8,
     length_size: u8,
+) -> Result<VerifyResult, FormatError> {
+    verify_dataset_in(file_data, header, offset_size, length_size)
+}
+
+/// [`verify_dataset`] over any [`Storage`](crate::storage::Storage).
+pub fn verify_dataset_in<S: crate::storage::Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<VerifyResult, FormatError> {
     // 1. Extract all attributes (compact + dense).
     let attrs =
-        crate::attribute::extract_attributes_full(file_data, header, offset_size, length_size)?;
+        crate::attribute::extract_attributes_full_in(file_data, header, offset_size, length_size)?;
 
     // 2. Find the stored hash.
     let stored_hash = attrs
@@ -174,7 +183,7 @@ pub fn verify_dataset(
         .transpose()?;
 
     let raw = match &dl {
-        DataLayout::Chunked { .. } => crate::chunked_read::read_chunked_data(
+        DataLayout::Chunked { .. } => crate::chunked_read::read_chunked_data_in(
             file_data,
             &dl,
             &ds,
@@ -183,7 +192,7 @@ pub fn verify_dataset(
             offset_size,
             length_size,
         )?,
-        _ => read_raw_data(file_data, &dl, &ds, &dt)?,
+        _ => crate::data_read::read_raw_data_in(file_data, &dl, &ds, &dt)?,
     };
 
     // 4. Compare.

crates/clawhdf5-format/src/shared_message.rs

@@ -23,12 +23,12 @@ use alloc::vec::Vec;
 #[cfg(feature = "std")]
 use std::borrow::Cow;
 
-use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records_in};
 use crate::error::FormatError;
 use crate::fractal_heap::FractalHeapHeader;
 use crate::message_type::MessageType;
 use crate::object_header::ObjectHeader;
-use crate::storage::{Storage, Window, read_exact_at, require_contiguous};
+use crate::storage::{Storage, Window, read_exact_at};
 
 /// Fractal heap ID length for SOHM entries (fixed at 8 bytes).
 const FHEAP_ID_LEN: usize = 8;
@@ -423,19 +423,15 @@ pub fn parse_sohm_btree_entries(
     parse_sohm_btree_entries_in(file_data, btree_addr as u64, offset_size, length_size)
 }
 
-/// [`parse_sohm_btree_entries`] over any [`Storage`]. The v2 B-tree is not
+/// [`parse_sohm_btree_entries`] over any [`Storage`].
-/// read over [`Storage`] yet, so this needs the whole file in memory
-/// ([`FormatError::ContiguousStorageRequired`] otherwise).
 pub fn parse_sohm_btree_entries_in<S: Storage + ?Sized>(
     file: &S,
     btree_addr: u64,
     offset_size: u8,
     length_size: u8,
 ) -> Result<Vec<SohmEntry>, FormatError> {
-    let file_data = require_contiguous(file, "a shared-message B-tree index")?;
+    let header = BTreeV2Header::parse_in(file, btree_addr, offset_size, length_size)?;
-    let btree_addr = usize::try_from(btree_addr).unwrap_or(usize::MAX);
+    let records = collect_btree_v2_records_in(file, &header, offset_size, length_size)?;
-    let header = BTreeV2Header::parse(file_data, btree_addr, offset_size, length_size)?;
-    let records = collect_btree_v2_records(file_data, &header, offset_size, length_size)?;
     let mut entries = Vec::with_capacity(records.len());
     for rec in &records {
         let entry = parse_sohm_entry(&rec.data, offset_size)?;
@@ -1243,11 +1239,12 @@ mod tests {
             }
         }
         assert!(compared > 200);
-        // The B-tree index is not read over Storage yet: a clean error.
+        // The B-tree index reads through Storage too, errors included.
-        let st = CountingStorage::new(vec![0u8; 64]);
+        let junk = vec![0u8; 64];
+        let st = CountingStorage::new(junk.clone());
         assert_eq!(
-            parse_sohm_btree_entries_in(&st, 0, 8, 8).unwrap_err(),
+            format!("{:?}", parse_sohm_btree_entries_in(&st, 0, 8, 8)),
-            FormatError::ContiguousStorageRequired("a shared-message B-tree index")
+            format!("{:?}", parse_sohm_btree_entries(&junk, 0, 8, 8))
         );
     }
 }

crates/clawhdf5-format/src/storage.rs

@@ -20,6 +20,18 @@
 //! still works (`S = dyn Storage`), and a remote backend pays one indirect
 //! call per structure read.
 //!
+//! # Slice entry points
+//!
+//! A generic core is instantiated in the crate that calls it, so a
+//! downstream crate calling `parse_in::<[u8]>` gets its own copy of the
+//! parser, compiled without this crate's private helpers inlined (there is
+//! no cross-crate inlining of non-`#[inline]` functions without LTO): a
+//! metadata walk through the facade ran about 6% slower that way than
+//! through the `&[u8]` wrappers. The `*_in` entry points on the facade's hot
+//! paths (object headers, group listing and lookup, attributes) therefore
+//! check [`Storage::as_contiguous`] first and hand an in-memory file to
+//! their non-generic `&[u8]` wrapper, compiled here; both run the one core.
+//!
 //! The trait is synchronous and `no_std`: parsing is CPU work, and a remote
 //! backend bridges to its own I/O.
 
@@ -43,6 +55,9 @@ pub trait Storage {
     /// end of the storage (and empty when `offset` is at or past the end);
     /// a backend that cannot serve a range returns an error instead of a
     /// short read.
+    /// It is never longer than `len`; the parsers cut a longer result to
+    /// `len` (see [`exact_len`]) rather than read bytes from outside the
+    /// range.
     fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError>;
 
     /// Current length of the storage in bytes.
@@ -218,13 +233,30 @@ pub fn read_exact_at<S: Storage + ?Sized>(
         Some(end) if end <= file.len() => {}
         _ => return Err(eof()),
     }
-    let bytes = file.read_at(offset, len)?;
+    // A short read (the storage shrank, or the backend served less inside
-    if bytes.len() < len {
+    // the file) is an error: never parse a partial structure.
-        // The storage shrank or the backend served a short read inside the
+    exact_len(file.read_at(offset, len)?, len)
-        // file: never parse a partial structure.
+}
-        return Err(short_read());
+
+/// `bytes`, the result of asking a [`Storage`] for `len` bytes, as exactly
+/// `len` bytes: a longer result (a backend that broke
+/// [`Storage::read_at`]'s contract) is cut to `len`, so bytes from outside
+/// the range asked for are never parsed or returned; a shorter one is an
+/// error (the storage shrank, or the backend failed), never a partial
+/// structure.
+#[inline]
+pub fn exact_len(bytes: Cow<'_, [u8]>, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+    match bytes.len().cmp(&len) {
+        core::cmp::Ordering::Equal => Ok(bytes),
+        core::cmp::Ordering::Less => Err(short_read()),
+        core::cmp::Ordering::Greater => Ok(match bytes {
+            Cow::Borrowed(b) => Cow::Borrowed(&b[..len]),
+            Cow::Owned(mut v) => {
+                v.truncate(len);
+                Cow::Owned(v)
+            }
+        }),
     }
-    Ok(bytes)
 }
 
 #[cold]
@@ -329,11 +361,236 @@ pub fn read_upto<S: Storage + ?Sized>(
     }
     let avail = file.len().saturating_sub(offset);
     let len = usize::try_from(avail).map_or(max, |a| a.min(max));
-    let bytes = file.read_at(offset, len)?;
+    exact_len(file.read_at(offset, len)?, len)
-    if bytes.len() < len {
+}
-        return Err(short_read());
+
+/// Most stored bytes fetched by one [`Storage::read_ranges`] call when a
+/// read gathers many extents (a chunked dataset's chunks, a selection's
+/// runs): a larger read is fetched and decoded batch by batch, so a backend
+/// without the file in memory never holds more than this much undecoded
+/// data per read (or one extent, when a single one is larger — and every
+/// chunk's extent is bounded by what the chunk can need, see
+/// [`crate::filters::stored_chunk_limit`]).
+pub const RAW_BATCH_BYTES: usize = 64 << 20;
+
+/// One extent of a raw-data read: `len` bytes stored at `addr`, whose
+/// bounds are checked against the file, of which the first `fetch` bytes
+/// are read (`None`: only checked, not read — its bytes are not needed).
+///
+/// `fetch` below `len` bounds what a crafted size field can make a read
+/// fetch: a chunk never needs more of its stored bytes than its decoded
+/// size allows, however large its index entry says it is.
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct ExtentReq {
+    pub addr: u64,
+    pub len: usize,
+    pub fetch: Option<usize>,
+}
+
+impl ExtentReq {
+    /// How many bytes are read for this extent.
+    #[inline]
+    fn fetch_len(&self) -> usize {
+        self.fetch.map_or(0, |f| f.min(self.len))
     }
-    Ok(bytes)
+}
+
+/// One extent's bytes on their own (see [`ExtentReq`]): the whole extent's
+/// bounds checked as [`read_exact_at`] checks them, and its first
+/// `req.fetch` bytes read (none when `fetch` is `None`).
+pub(crate) fn read_extent<'a, S: Storage + ?Sized>(
+    file: &'a S,
+    req: &ExtentReq,
+) -> Result<Cow<'a, [u8]>, FormatError> {
+    let start = usize::try_from(req.addr).unwrap_or(usize::MAX);
+    match start.checked_add(req.len) {
+        Some(end) if end <= len_usize(file) => read_exact_at(file, req.addr, req.fetch_len()),
+        _ => Err(FormatError::UnexpectedEof {
+            expected: start.saturating_add(req.len),
+            available: len_usize(file),
+        }),
+    }
+}
+
+/// The stored bytes of one batch of extents (chunks, contiguous runs),
+/// fetched together: [`Storage::read_ranges`] is called once for the batch,
+/// so a remote backend can coalesce and parallelise the requests. See
+/// [`for_each_extent_batch`], which is how every raw-data read gets them.
+///
+/// With the whole file in memory nothing is fetched: [`Self::get`] slices
+/// it, as the slice readers did. Either way an extent that does not lie in
+/// the file is the error the slice readers gave for it
+/// ([`FormatError::UnexpectedEof`] with its end and the file length, or
+/// [`FormatError::Overflow`] for an address past this platform's `usize`),
+/// reported when that extent is asked for — so a read reports the first
+/// failing extent in its own order, whatever fails after it.
+pub(crate) enum ExtentBytes<'a> {
+    /// The whole file.
+    Contiguous(&'a [u8]),
+    /// Each extent's bytes, or its bounds error; the first is extent
+    /// `base` of the read.
+    Fetched {
+        base: usize,
+        extents: Vec<Extent<'a>>,
+    },
+}
+
+/// One extent of [`ExtentBytes::Fetched`].
+pub(crate) enum Extent<'a> {
+    /// Its bytes.
+    Bytes(Cow<'a, [u8]>),
+    /// In the file, but not fetched (the caller did not want its bytes).
+    NotFetched,
+    /// The error reading it gives.
+    Err(FormatError),
+}
+
+impl<'a> ExtentBytes<'a> {
+    /// Fetch `reqs`, extents `base..base + reqs.len()` of the read: the
+    /// bytes of those wanted, and the bounds check of all of them.
+    pub(crate) fn fetch<S: Storage + ?Sized>(
+        file: &'a S,
+        reqs: &[ExtentReq],
+        base: usize,
+    ) -> Result<Self, FormatError> {
+        if let Some(all) = file.as_contiguous() {
+            return Ok(ExtentBytes::Contiguous(all));
+        }
+        let file_len = len_usize(file);
+        let mut ranges = Vec::new();
+        let mut out = Vec::with_capacity(reqs.len());
+        // Positions in `out` of the extents being read, in `ranges` order.
+        let mut slots = Vec::new();
+        for req in reqs {
+            let checked = crate::addr::to_usize(req.addr).and_then(|start| {
+                match start.checked_add(req.len) {
+                    Some(end) if end <= file_len => Ok(()),
+                    _ => Err(FormatError::UnexpectedEof {
+                        expected: start.saturating_add(req.len),
+                        available: file_len,
+                    }),
+                }
+            });
+            match checked {
+                Ok(()) if req.fetch.is_some() => {
+                    slots.push(out.len());
+                    ranges.push(req.addr..req.addr + req.fetch_len() as u64);
+                    out.push(Extent::NotFetched);
+                }
+                Ok(()) => out.push(Extent::NotFetched),
+                Err(e) => out.push(Extent::Err(e)),
+            }
+        }
+        if !ranges.is_empty() {
+            let got = file.read_ranges(&ranges)?;
+            if got.len() != ranges.len() {
+                return Err(FormatError::Storage(
+                    "read_ranges returned the wrong number of ranges".into(),
+                ));
+            }
+            for ((slot, bytes), r) in slots.into_iter().zip(got).zip(&ranges) {
+                let len = crate::addr::saturating_usize(r.end - r.start);
+                out[slot] = Extent::Bytes(exact_len(bytes, len)?);
+            }
+        }
+        Ok(ExtentBytes::Fetched { base, extents: out })
+    }
+
+    /// Whether extent `i` of the read (`req`) lies in the file: its bounds
+    /// error if not.
+    pub(crate) fn check(&self, i: usize, req: &ExtentReq) -> Result<(), FormatError> {
+        match self {
+            ExtentBytes::Contiguous(_) => self.get(i, req).map(|_| ()),
+            ExtentBytes::Fetched { base, extents } => {
+                match i.checked_sub(*base).and_then(|j| extents.get(j)) {
+                    Some(Extent::Err(e)) => Err(e.clone()),
+                    Some(_) => Ok(()),
+                    None => Err(not_fetched()),
+                }
+            }
+        }
+    }
+
+    /// Extent `i` of the read (`req`): its first `req.fetch` bytes, the
+    /// same whether the file is in memory or not.
+    pub(crate) fn get(&self, i: usize, req: &ExtentReq) -> Result<&[u8], FormatError> {
+        match self {
+            ExtentBytes::Contiguous(all) => {
+                let start = crate::addr::to_usize(req.addr)?;
+                start
+                    .checked_add(req.len)
+                    .and_then(|end| all.get(start..end))
+                    .map(|b| &b[..req.fetch_len()])
+                    .ok_or(FormatError::UnexpectedEof {
+                        expected: start.saturating_add(req.len),
+                        available: <[u8]>::len(all),
+                    })
+            }
+            ExtentBytes::Fetched { base, extents } => {
+                match i.checked_sub(*base).and_then(|j| extents.get(j)) {
+                    Some(Extent::Bytes(b)) => Ok(b),
+                    Some(Extent::Err(e)) => Err(e.clone()),
+                    _ => Err(not_fetched()),
+                }
+            }
+        }
+    }
+}
+
+#[cold]
+fn not_fetched() -> FormatError {
+    FormatError::Storage("an extent that was not fetched was asked for".into())
+}
+
+/// The one way raw-data reads fetch stored bytes: `reqs` are split into
+/// consecutive batches of at most [`RAW_BATCH_BYTES`] of fetched bytes (at
+/// least one extent each — and no extent fetches more than its
+/// [`ExtentReq::fetch`]), and for each batch in turn its bytes are fetched
+/// with one [`Storage::read_ranges`] call and `f(batch, &bytes)` is called,
+/// with `bytes` indexed by the extent's position in `reqs`. A batch's bytes
+/// are dropped before the next batch is fetched, and an error from `f`
+/// stops the read before anything more is fetched.
+///
+/// With the whole file in memory there is nothing to fetch: one call, over
+/// all of `reqs`, that slices the file.
+pub(crate) fn for_each_extent_batch<'a, S: Storage + ?Sized>(
+    file: &'a S,
+    reqs: &[ExtentReq],
+    mut f: impl FnMut(Range<usize>, &ExtentBytes<'a>) -> Result<(), FormatError>,
+) -> Result<(), FormatError> {
+    let contiguous = file.as_contiguous().is_some();
+    for batch in raw_batches(reqs.len(), contiguous, |i| reqs[i].fetch_len()) {
+        let bytes = ExtentBytes::fetch(file, &reqs[batch.clone()], batch.start)?;
+        f(batch, &bytes)?;
+    }
+    Ok(())
+}
+
+/// Split `n` extents, whose sizes `size(i)` gives, into consecutive batches
+/// of at most [`RAW_BATCH_BYTES`] (at least one extent each): the ranges of
+/// `0..n` to fetch together. With the whole file in memory (`contiguous`)
+/// there is nothing to fetch, and one batch.
+pub(crate) fn raw_batches(
+    n: usize,
+    contiguous: bool,
+    size: impl Fn(usize) -> usize,
+) -> Vec<Range<usize>> {
+    if contiguous || n == 0 {
+        return core::iter::once(0..n).collect();
+    }
+    let mut out = Vec::new();
+    let (mut start, mut bytes) = (0, 0usize);
+    for i in 0..n {
+        let s = size(i);
+        if i > start && bytes.saturating_add(s) > RAW_BATCH_BYTES {
+            out.push(start..i);
+            start = i;
+            bytes = 0;
+        }
+        bytes = bytes.saturating_add(s);
+    }
+    out.push(start..n);
+    out
 }
 
 /// Borrow the whole file for a code path that has not been converted to

crates/clawhdf5-format/src/superblock_ext.rs

@@ -490,6 +490,23 @@ impl CacheImage {
         image_block_in(file, self.location)
     }
 
+    /// Every entry as `(file address, its bytes)`, taken from `block` (the
+    /// image block, see [`Self::block_in`]), in the order [`Self::apply`]
+    /// writes them: for a reader that cannot write the image over the
+    /// file's bytes and lays the entries over each read instead.
+    pub fn entries<'b>(&self, block: &'b [u8]) -> Result<Vec<(u64, &'b [u8])>, FormatError> {
+        let short = || FormatError::InvalidCacheImage("image applied to the wrong file");
+        self.entries
+            .iter()
+            .map(|e| {
+                let src = block
+                    .get(e.image_offset..e.image_offset + e.len)
+                    .ok_or_else(short)?;
+                Ok((e.address, src))
+            })
+            .collect()
+    }
+
     /// Write every entry over `dst`, the file's bytes from the superblock
     /// on (as long as the `data` the image was decoded from), taking the
     /// entries from `block` (the image block, see [`Self::block`]). `block`

crates/clawhdf5-format/src/vds.rs

@@ -15,12 +15,13 @@
 #[cfg(not(feature = "std"))]
 use alloc::{format, string::String, vec, vec::Vec};
 
-use crate::addr::to_usize;
+use crate::addr::{checked_addr, to_usize};
 use crate::data_layout::{DataLayout, VdsMapping, parse_vds_mappings};
 use crate::dataspace::Dataspace;
 use crate::datatype::Datatype;
 use crate::error::FormatError;
 use crate::selection::{SerializedSelection, UNLIMITED};
+use crate::storage::Storage;
 
 /// Resolves the name of an external VDS source file, as stored in the
 /// mapping, to that file's bytes.
@@ -192,8 +193,8 @@ fn non_unlimited_elements(sel: &SerializedSelection, skip: usize) -> Option<u64>
 }
 
 /// Load and decode the mapping list of a virtual layout.
-fn load_mappings(
+fn load_mappings<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     layout: &DataLayout,
     length_size: u8,
 ) -> Result<Vec<Mapping>, FormatError> {
@@ -208,8 +209,11 @@ fn load_mappings(
     let Some(addr) = *global_heap_address else {
         return Ok(Vec::new());
     };
-    let coll =
+    let coll = crate::global_heap::GlobalHeapCollection::parse_in(
-        crate::global_heap::GlobalHeapCollection::parse(file_data, to_usize(addr)?, length_size)?;
+        file_data,
+        checked_addr(addr)?,
+        length_size,
+    )?;
     let index = u16::try_from(*global_heap_index)
         .map_err(|_| vds_err("VDS mapping heap index out of range"))?;
     let obj = coll
@@ -330,7 +334,11 @@ enum Step {
 /// Work out the extent libhdf5 gives the virtual dataset
 /// (`H5D__virtual_set_extent_unlim`, default view `H5D_VDS_LAST_AVAILABLE`
 /// with a printf gap of 0) and how much of each unlimited mapping is read.
-fn plan(mappings: &[Mapping], stored: &[u64], sources: &mut Sources) -> Result<Plan, FormatError> {
+fn plan<S: Storage + ?Sized>(
+    mappings: &[Mapping],
+    stored: &[u64],
+    sources: &mut Sources<'_, '_, S>,
+) -> Result<Plan, FormatError> {
     let overflow = || FormatError::Overflow("VDS extent overflow".into());
     let rank = stored.len();
     let mut new_dims: Vec<Option<u64>> = vec![None; rank];
@@ -472,6 +480,25 @@ pub fn virtual_dataset_extent(
     _offset_size: u8,
     length_size: u8,
     resolver: Option<&VdsFileResolver>,
+) -> Result<Vec<u64>, FormatError> {
+    virtual_dataset_extent_in(
+        file_data,
+        layout,
+        dataspace,
+        _offset_size,
+        length_size,
+        resolver,
+    )
+}
+
+/// [`virtual_dataset_extent`] over any [`Storage`].
+pub fn virtual_dataset_extent_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    _offset_size: u8,
+    length_size: u8,
+    resolver: Option<&VdsFileResolver>,
 ) -> Result<Vec<u64>, FormatError> {
     let mappings = load_mappings(file_data, layout, length_size)?;
     if mappings.iter().all(|m| m.kind == Kind::Fixed) {
@@ -498,6 +525,30 @@ pub fn read_virtual_dataset(
     _offset_size: u8,
     length_size: u8,
     resolver: Option<&VdsFileResolver>,
+) -> Result<VirtualData, FormatError> {
+    read_virtual_dataset_in(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        fill,
+        _offset_size,
+        length_size,
+        resolver,
+    )
+}
+
+/// [`read_virtual_dataset`] over any [`Storage`].
+#[allow(clippy::too_many_arguments)]
+pub fn read_virtual_dataset_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    fill: Option<&[u8]>,
+    _offset_size: u8,
+    length_size: u8,
+    resolver: Option<&VdsFileResolver>,
 ) -> Result<VirtualData, FormatError> {
     let mappings = load_mappings(file_data, layout, length_size)?;
     let mut sources = Sources::new(file_data, resolver);
@@ -511,7 +562,7 @@ pub fn read_virtual_dataset(
     let mut data = crate::chunked_read::alloc_output(crate::chunked_read::checked_byte_len(
         total, elem_size,
     )?)?;
-    if let Some(fill) = fill.filter(|f| f.len() == elem_size && f.iter().any(|&b| b != 0)) {
+    if let Some(fill) = fill.filter(|f| <[u8]>::len(f) == elem_size && f.iter().any(|&b| b != 0)) {
         for element in data.chunks_exact_mut(elem_size) {
             element.copy_from_slice(fill);
         }
@@ -780,14 +831,17 @@ struct SourceData {
 
 /// Source files and datasets, fetched on demand. The most recently used
 /// external file is kept, since consecutive mappings usually share one.
-struct Sources<'a, 'r> {
+///
-    file_data: &'a [u8],
+/// The virtual dataset's own file (`"."`) is read through its [`Storage`];
+/// an external source file is loaded whole, through the resolver.
+struct Sources<'a, 'r, S: Storage + ?Sized> {
+    file_data: &'a S,
     resolver: Option<&'r VdsFileResolver<'r>>,
     cached_file: Option<(String, Option<Vec<u8>>)>,
 }
 
-impl<'a, 'r> Sources<'a, 'r> {
+impl<'a, 'r, S: Storage + ?Sized> Sources<'a, 'r, S> {
-    fn new(file_data: &'a [u8], resolver: Option<&'r VdsFileResolver<'r>>) -> Self {
+    fn new(file_data: &'a S, resolver: Option<&'r VdsFileResolver<'r>>) -> Self {
         Sources {
             file_data,
             resolver,
@@ -795,11 +849,9 @@ impl<'a, 'r> Sources<'a, 'r> {
         }
     }
 
-    /// The bytes of source file `name`, or `None` if it does not exist.
+    /// The bytes of external source file `name` (not `"."`), or `None` if it
-    fn file(&mut self, name: &str) -> Result<Option<&[u8]>, FormatError> {
+    /// does not exist.
-        if name == "." {
+    fn external(&mut self, name: &str) -> Result<Option<&[u8]>, FormatError> {
-            return Ok(Some(self.file_data));
-        }
         if self.cached_file.as_ref().is_none_or(|(n, _)| n != name) {
             let resolver = self.resolver.ok_or_else(|| {
                 vds_err("external-file virtual dataset sources require a file resolver")
@@ -821,7 +873,10 @@ impl<'a, 'r> Sources<'a, 'r> {
     /// The extent of source dataset `path` in file `file`, or `None` when
     /// either does not exist.
     fn dims(&mut self, file: &str, path: &str) -> Result<Option<Vec<u64>>, FormatError> {
-        let Some(bytes) = self.file(file)? else {
+        if file == "." {
+            return Ok(open_source(self.file_data, path)?.map(|s| s.dataspace.dimensions));
+        }
+        let Some(bytes) = self.external(file)? else {
             return Ok(None);
         };
         Ok(open_source(bytes, path)?.map(|s| s.dataspace.dimensions))
@@ -846,7 +901,13 @@ impl<'a, 'r> Sources<'a, 'r> {
                  from another file is not supported"
             )));
         }
-        let Some(bytes) = self.file(file)? else {
+        if file == "." {
+            let Some(src) = open_source(self.file_data, path)? else {
+                return Ok(None);
+            };
+            return read_source(self.file_data, src, path, datatype).map(Some);
+        }
+        let Some(bytes) = self.external(file)? else {
             return Ok(None);
         };
         let Some(src) = open_source(bytes, path)? else {
@@ -910,19 +971,23 @@ fn source_message<'h>(
 
 /// Open source dataset `path` of the file in `file_data`, or `None` if there
 /// is no such object (libhdf5 reads a missing source as fill).
-fn open_source(file_data: &[u8], path: &str) -> Result<Option<OpenSource>, FormatError> {
+fn open_source<S: Storage + ?Sized>(
+    file_data: &S,
+    path: &str,
+) -> Result<Option<OpenSource>, FormatError> {
     use crate::message_type::MessageType;
-    use crate::shared_message::message_data_with_sohm;
+    use crate::shared_message::message_data_with_sohm_in as message_data_with_sohm;
 
-    // `file_data` starts at the superblock (see `Sources::file`).
+    // `file_data` starts at the superblock (see `Sources::external`).
-    let sb = crate::superblock::Superblock::parse(file_data, 0)?;
+    let sb = crate::superblock::Superblock::parse_in(file_data, 0)?;
     let (os, ls) = (sb.offset_size, sb.length_size);
-    let addr = match crate::group_v2::resolve_path_any(file_data, &sb, path) {
+    let addr = match crate::group_v2::resolve_path_any_in(file_data, &sb, path) {
         Ok(a) => a,
         Err(FormatError::PathNotFound(_)) => return Ok(None),
         Err(e) => return Err(e),
     };
-    let header = crate::object_header::ObjectHeader::parse(file_data, to_usize(addr)?, os, ls)?;
+    let header =
+        crate::object_header::ObjectHeader::parse_in(file_data, checked_addr(addr)?, os, ls)?;
     let mut src = OpenSource {
         offset_size: os,
         length_size: ls,
@@ -941,15 +1006,15 @@ fn open_source(file_data: &[u8], path: &str) -> Result<Option<OpenSource>, Forma
 
 /// Read an opened source dataset in full (its own fill value applied to
 /// unallocated chunks).
-fn read_source(
+fn read_source<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     src: OpenSource,
     path: &str,
     datatype: &Datatype,
 ) -> Result<SourceData, FormatError> {
     use crate::filter_pipeline::FilterPipeline;
     use crate::message_type::MessageType;
-    use crate::shared_message::message_data_with_sohm;
+    use crate::shared_message::message_data_with_sohm_in as message_data_with_sohm;
 
     let (os, ls) = (src.offset_size, src.length_size);
     let dt_msg = source_message(&src, path, MessageType::Datatype)?;
@@ -987,7 +1052,7 @@ fn read_source(
             message_data_with_sohm(file_data, m, os, ls).and_then(|d| FilterPipeline::parse(&d))
         })
         .transpose()?;
-    let raw = crate::fill_value::read_full_with_fill(
+    let raw = crate::fill_value::read_full_with_fill_in(
         &src.header.messages,
         file_data,
         &layout,
@@ -996,7 +1061,7 @@ fn read_source(
         os,
         ls,
         || {
-            crate::data_read::read_raw_data_full(
+            crate::data_read::read_raw_data_full_in(
                 file_data,
                 &layout,
                 &src.dataspace,

crates/clawhdf5-format/src/vl_data.rs

@@ -5,9 +5,9 @@
 //! `sequence_length(4 LE) + collection_address(offset_size LE) + object_index(4 LE)`.
 
 #[cfg(not(feature = "std"))]
-use alloc::{collections::BTreeMap, format, string::String, vec, vec::Vec};
+use alloc::{borrow::Cow, collections::BTreeMap, format, string::String, vec, vec::Vec};
 #[cfg(feature = "std")]
-use std::collections::BTreeMap;
+use std::{borrow::Cow, collections::BTreeMap};
 
 use crate::addr::to_usize;
 use crate::error::FormatError;
@@ -137,13 +137,15 @@ pub fn check_element_size(stored_size: u32, offset_size: u8) -> Result<(), Forma
 
 /// A collection's objects, located in the file data but not copied:
 /// `(index, offset, size)` of the first object with each index, sorted by
-/// index.
+/// index. Over a storage without the whole file in memory, also the
-struct CachedCollection {
+/// collection's bytes (`(offset, bytes)`), read once when it is indexed.
+struct CachedCollection<'a> {
     objects: Vec<(u16, usize, usize)>,
+    bytes: Option<(usize, Cow<'a, [u8]>)>,
 }
 
-impl CachedCollection {
+impl<'a> CachedCollection<'a> {
-    fn new(index: GlobalHeapIndex) -> Self {
+    fn new(index: GlobalHeapIndex, bytes: Option<(usize, Cow<'a, [u8]>)>) -> Self {
         let mut objects: Vec<(u16, usize, usize)> = index
             .objects
             .iter()
@@ -152,12 +154,16 @@ impl CachedCollection {
         // Stable, so the first object with a repeated index is kept.
         objects.sort_by_key(|o| o.0);
         objects.dedup_by_key(|o| o.0);
-        Self { objects }
+        Self { objects, bytes }
     }
 
     /// What this entry costs to keep, in bytes (roughly).
     fn cost(&self) -> usize {
-        64 + self.objects.len() * core::mem::size_of::<(u16, usize, usize)>()
+        let held = match &self.bytes {
+            Some((_, Cow::Owned(b))) => b.len(),
+            _ => 0,
+        };
+        64 + self.objects.len() * core::mem::size_of::<(u16, usize, usize)>() + held
     }
 
     fn get(&self, index: u32) -> Option<(usize, usize)> {
@@ -170,6 +176,8 @@ impl CachedCollection {
 /// How many bytes of collection indexes a [`VlResolver`] keeps before it
 /// drops them and starts again. Values are never copied into the cache, so
 /// this bounds what a read retains however many collections it visits.
+/// (Over a storage without the whole file in memory the collections' bytes
+/// are kept too, and count against this.)
 const CACHE_BUDGET: usize = 32 << 20;
 
 /// Resolves variable-length elements against a file's global heap, parsing
@@ -185,11 +193,18 @@ const CACHE_BUDGET: usize = 32 << 20;
 /// that overlap one another are refused (libhdf5 never writes them), so a
 /// file cannot make the resolver parse the same bytes as the objects of
 /// many collections.
-pub struct VlResolver<'a> {
+///
-    file_data: &'a [u8],
+/// The file is any [`Storage`](crate::storage::Storage) (`S`, a slice by default). Over one without
+/// the whole file in memory each collection is read once, when first used,
+/// and kept (within the budget above); [`Self::strings`],
+/// [`Self::string_bytes`] and [`Self::sequences`] work over any storage,
+/// [`Self::element`] and [`Self::string_element`], which borrow from the
+/// file, over a slice.
+pub struct VlResolver<'a, S: crate::storage::Storage + ?Sized = [u8]> {
+    file_data: &'a S,
     offset_size: u8,
     length_size: u8,
-    cache: BTreeMap<u64, CachedCollection>,
+    cache: BTreeMap<u64, CachedCollection<'a>>,
     cached_bytes: usize,
     budget: usize,
     /// Start → end of every collection parsed so far (kept when the cache
@@ -201,6 +216,56 @@ impl<'a> VlResolver<'a> {
     /// A resolver over `file_data` (the file from its superblock on), with
     /// the superblock's offset and length sizes.
     pub fn new(file_data: &'a [u8], offset_size: u8, length_size: u8) -> Self {
+        Self::new_in(file_data, offset_size, length_size)
+    }
+
+    /// One element (the first [`element_size`](Self::element_size) bytes of
+    /// `elem`) of a variable-length sequence whose base type is `base_size`
+    /// bytes: its `length × base_size` bytes, or `None` for a null element
+    /// (heap address 0).
+    pub fn element(
+        &mut self,
+        elem: &[u8],
+        base_size: usize,
+    ) -> Result<Option<&'a [u8]>, FormatError> {
+        let vl = parse_vl_references(elem, 1, self.offset_size)?;
+        let vl = &vl[0];
+        if vl.collection_address == 0 {
+            return Ok(None);
+        }
+        let (start, size) = self.locate(vl)?;
+        let data = &self.file_data[start..start + size];
+        check_object_size(vl, data.len(), base_size)?;
+        Ok(Some(data))
+    }
+
+    /// One variable-length string element: its bytes up to the first NUL,
+    /// or `None` for a null element (h5dump prints it as `NULL`, h5py
+    /// returns it as empty).
+    pub fn string_element(&mut self, elem: &[u8]) -> Result<Option<&'a [u8]>, FormatError> {
+        Ok(self.element(elem, 1)?.map(cut_at_nul))
+    }
+}
+
+/// `data_len`, the size of `vl`'s heap object, against the `length ×
+/// base_size` bytes the element says it holds.
+fn check_object_size(vl: &VlElement, data_len: usize, base_size: usize) -> Result<(), FormatError> {
+    let expected = (vl.length as usize)
+        .checked_mul(base_size)
+        .ok_or_else(|| FormatError::Overflow("variable-length element size".into()))?;
+    if data_len != expected {
+        return Err(FormatError::VlDataError(format!(
+            "global heap object {} in the collection at {} holds {data_len} bytes; the element \
+             says {} × {base_size}",
+            vl.object_index, vl.collection_address, vl.length
+        )));
+    }
+    Ok(())
+}
+
+impl<'a, S: crate::storage::Storage + ?Sized> VlResolver<'a, S> {
+    /// [`VlResolver::new`] over any [`Storage`](crate::storage::Storage).
+    pub fn new_in(file_data: &'a S, offset_size: u8, length_size: u8) -> Self {
         Self {
             file_data,
             offset_size,
@@ -231,49 +296,31 @@ impl<'a> VlResolver<'a> {
 
     /// The bytes of one element: `length × base_size` bytes from the heap,
     /// or `None` for a null element.
-    fn resolve(
+    fn resolve(&mut self, vl: &VlElement, base_size: usize) -> Result<Option<&[u8]>, FormatError> {
-        &mut self,
+        if vl.collection_address == 0 {
-        vl: &VlElement,
-        base_size: usize,
-    ) -> Result<Option<&'a [u8]>, FormatError> {
-        let addr = vl.collection_address;
-        if addr == 0 {
             return Ok(None);
         }
         let data = self.object(vl)?;
-        let expected = (vl.length as usize)
+        check_object_size(vl, data.len(), base_size)?;
-            .checked_mul(base_size)
-            .ok_or_else(|| FormatError::Overflow("variable-length element size".into()))?;
-        if data.len() != expected {
-            return Err(FormatError::VlDataError(format!(
-                "global heap object {} in the collection at {addr} holds {} bytes; the element \
-                 says {} × {base_size}",
-                vl.object_index,
-                data.len(),
-                vl.length
-            )));
-        }
         Ok(Some(data))
     }
 
-    /// One element (the first [`element_size`](Self::element_size) bytes of
+    /// [`VlResolver::element`] over any storage: the element's bytes
-    /// `elem`) of a variable-length sequence whose base type is `base_size`
+    /// (borrowed from the resolver's cache of heap collections, so they
-    /// bytes: its `length × base_size` bytes, or `None` for a null element
+    /// live until the next call), or `None` for a null element.
-    /// (heap address 0).
+    pub fn element_in(
-    pub fn element(
         &mut self,
         elem: &[u8],
         base_size: usize,
-    ) -> Result<Option<&'a [u8]>, FormatError> {
+    ) -> Result<Option<&[u8]>, FormatError> {
         let vl = parse_vl_references(elem, 1, self.offset_size)?;
         self.resolve(&vl[0], base_size)
     }
 
-    /// One variable-length string element: its bytes up to the first NUL,
+    /// [`VlResolver::string_element`] over any storage (see
-    /// or `None` for a null element (h5dump prints it as `NULL`, h5py
+    /// [`element_in`](Self::element_in)).
-    /// returns it as empty).
+    pub fn string_element_in(&mut self, elem: &[u8]) -> Result<Option<&[u8]>, FormatError> {
-    pub fn string_element(&mut self, elem: &[u8]) -> Result<Option<&'a [u8]>, FormatError> {
+        Ok(self.element_in(elem, 1)?.map(cut_at_nul))
-        Ok(self.element(elem, 1)?.map(cut_at_nul))
     }
 
     /// The strings of the variable-length string elements in `raw`, as
@@ -330,9 +377,20 @@ pub fn read_vl_strings(
     num_elements: u64,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<String>, FormatError> {
+    read_vl_strings_in(file_data, raw_data, num_elements, offset_size, length_size)
+}
+
+/// [`read_vl_strings`] over any [`Storage`](crate::storage::Storage).
+pub fn read_vl_strings_in<S: crate::storage::Storage + ?Sized>(
+    file_data: &S,
+    raw_data: &[u8],
+    num_elements: u64,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<String>, FormatError> {
     let raw = first_elements(raw_data, num_elements, offset_size)?;
-    VlResolver::new(file_data, offset_size, length_size).strings(raw)
+    VlResolver::new_in(file_data, offset_size, length_size).strings(raw)
 }
 
 /// The first `num_elements` elements of `raw`, or an error if it is shorter.
@@ -364,9 +422,20 @@ pub fn read_vl_bytes(
     num_elements: u64,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<Vec<u8>>, FormatError> {
+    read_vl_bytes_in(file_data, raw_data, num_elements, offset_size, length_size)
+}
+
+/// [`read_vl_bytes`] over any [`Storage`](crate::storage::Storage).
+pub fn read_vl_bytes_in<S: crate::storage::Storage + ?Sized>(
+    file_data: &S,
+    raw_data: &[u8],
+    num_elements: u64,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<Vec<u8>>, FormatError> {
     let refs = parse_vl_references(raw_data, num_elements, offset_size)?;
-    let mut resolver = VlResolver::new(file_data, offset_size, length_size);
+    let mut resolver = VlResolver::new_in(file_data, offset_size, length_size);
     let mut result = Vec::with_capacity(refs.len());
 
     for vl in &refs {
@@ -385,10 +454,10 @@ pub fn read_vl_bytes(
     Ok(result)
 }
 
-impl<'a> VlResolver<'a> {
+impl<'a, S: crate::storage::Storage + ?Sized> VlResolver<'a, S> {
-    /// The heap object `vl` points to, whatever its size; its collection is
+    /// Where the heap object `vl` points to lies in the file, whatever its
-    /// parsed on first use.
+    /// size (`(offset, size)`); its collection is parsed on first use.
-    fn object(&mut self, vl: &VlElement) -> Result<&'a [u8], FormatError> {
+    fn locate(&mut self, vl: &VlElement) -> Result<(usize, usize), FormatError> {
         let addr = vl.collection_address;
         // libhdf5 writes a null element with address 0, never the undefined
         // address, and fails to read one ("addr undefined") even when its
@@ -402,14 +471,20 @@ impl<'a> VlResolver<'a> {
         if !self.cache.contains_key(&addr) {
             let offset = usize::try_from(addr).map_err(|_| FormatError::UnexpectedEof {
                 expected: usize::MAX,
-                available: self.file_data.len(),
+                available: crate::storage::len_usize(self.file_data),
             })?;
-            let index =
+            let (bytes, base, index) =
-                GlobalHeapCollection::parse_index(self.file_data, offset, self.length_size)?;
+                GlobalHeapCollection::read_collection(self.file_data, addr, self.length_size)?;
-            // parse_index checked that the collection lies in the file.
+            // read_collection checked that the collection lies in the file.
             let end = offset + to_usize(index.collection_size)?;
             self.check_overlap(offset, end)?;
-            let coll = CachedCollection::new(index);
+            // With the whole file in memory the objects are sliced from it;
+            // otherwise the collection's bytes are kept.
+            let bytes = match self.file_data.as_contiguous() {
+                Some(_) => None,
+                None => Some((base, bytes)),
+            };
+            let coll = CachedCollection::new(index, bytes);
             if self.cached_bytes.saturating_add(coll.cost()) > self.budget {
                 self.cache.clear();
                 self.cached_bytes = 0;
@@ -417,13 +492,27 @@ impl<'a> VlResolver<'a> {
             self.cached_bytes += coll.cost();
             self.cache.insert(addr, coll);
         }
-        let (start, size) = self.cache[&addr].get(vl.object_index).ok_or(
+        self.cache[&addr]
-            FormatError::GlobalHeapObjectNotFound {
+            .get(vl.object_index)
+            .ok_or(FormatError::GlobalHeapObjectNotFound {
                 collection_address: addr,
                 index: vl.object_index as u16,
-            },
+            })
-        )?;
+    }
-        Ok(&self.file_data[start..start + size])
+
+    /// The heap object `vl` points to, whatever its size; its collection is
+    /// parsed on first use.
+    fn object(&mut self, vl: &VlElement) -> Result<&[u8], FormatError> {
+        let (start, size) = self.locate(vl)?;
+        if let Some(all) = self.file_data.as_contiguous() {
+            return Ok(&all[start..start + size]);
+        }
+        match &self.cache[&vl.collection_address].bytes {
+            Some((base, bytes)) => Ok(&bytes[start - base..start - base + size]),
+            None => Err(FormatError::Storage(
+                "global heap collection bytes were not kept".into(),
+            )),
+        }
     }
 
     /// Record the collection at `start..end`, refusing one that overlaps a
@@ -574,6 +663,32 @@ mod tests {
         }
     }
 
+    #[test]
+    fn element_in_over_a_storage_matches_element_over_a_slice() {
+        let mut file_data = vec![0u8; 512];
+        build_gcol_at(&mut file_data, 256, &[(1, b"Alice\0x"), (2, b"Bob")]);
+        let mut raw = build_vl_refs(&["Alice\0x", "Bob"], 256, 1, 8);
+        raw.extend(element(0, 0, 0, 8)); // null
+        raw.extend(element(9, 256, 1, 8)); // wrong length: an error
+        let storage = crate::storage::CountingStorage::new(file_data.clone());
+        let dynamic: &dyn crate::storage::Storage = &storage;
+        let mut slice = VlResolver::new(&file_data, 8, 8);
+        let mut any = VlResolver::new_in(dynamic, 8, 8);
+        for e in raw.chunks(16) {
+            let want = slice.element(e, 1).map(|o| o.map(<[u8]>::to_vec));
+            let got = any.element_in(e, 1).map(|o| o.map(<[u8]>::to_vec));
+            assert_eq!(format!("{want:?}"), format!("{got:?}"));
+            let want = slice.string_element(e).map(|o| o.map(<[u8]>::to_vec));
+            let got = any.string_element_in(e).map(|o| o.map(<[u8]>::to_vec));
+            assert_eq!(format!("{want:?}"), format!("{got:?}"));
+        }
+        assert_eq!(
+            any.string_element_in(&raw[..16]).unwrap(),
+            Some(&b"Alice"[..])
+        );
+        assert!(storage.reads() > 0);
+    }
+
     #[test]
     fn null_vl_element_zero_address() {
         let mut raw = Vec::new();
@@ -709,6 +824,7 @@ mod tests {
         let mut r = VlResolver::new(&file_data, 8, 8);
         let one = CachedCollection {
             objects: vec![(0, 0, 0); 3],
+            bytes: None,
         }
         .cost();
         r.budget = 2 * one + 1;

crates/clawhdf5-format/tests/raw_fetch_bounds.rs

@@ -0,0 +1,332 @@
+//! What a raw-data read fetches from a [`Storage`] without the file in
+//! memory is bounded: by batch (at most `RAW_BATCH_BYTES` per
+//! `read_ranges` call) and by chunk (never more of a chunk's stored bytes
+//! than its decoded size can need), on every path that reads chunks — full,
+//! cached, indexed, sweep, selection and the `parallel_read` decoders.
+//!
+//! A crafted chunk index can point every chunk at one huge extent. Slicing
+//! an in-memory file costs nothing there, but a backend that fetches would
+//! hold `chunks x extent` bytes before the first chunk failed to decode.
+
+use std::borrow::Cow;
+use std::ops::Range;
+use std::path::Path;
+use std::sync::atomic::{AtomicU64, Ordering::Relaxed};
+
+use clawhdf5_format::chunk_cache::ChunkCache;
+use clawhdf5_format::chunked_read::{
+    ChunkInfo, SweepContext, list_chunks, read_chunked_data_sweep_in,
+};
+use clawhdf5_format::data_layout::DataLayout;
+use clawhdf5_format::data_read::{
+    read_raw_data_cached_in, read_raw_data_full_in, read_raw_data_indexed_in,
+    read_raw_data_selection_in,
+};
+use clawhdf5_format::dataspace::Dataspace;
+use clawhdf5_format::datatype::Datatype;
+use clawhdf5_format::error::FormatError;
+use clawhdf5_format::filter_pipeline::FilterPipeline;
+use clawhdf5_format::group_v2;
+use clawhdf5_format::message_type::MessageType;
+use clawhdf5_format::object_header::ObjectHeader;
+use clawhdf5_format::selection::Selection;
+use clawhdf5_format::storage::{RAW_BATCH_BYTES, Storage};
+use clawhdf5_format::superblock::Superblock;
+
+/// A read_at-only storage that records the most bytes one call fetched
+/// (a `read_ranges` call counts all its ranges together) and the total.
+struct PeakStorage {
+    data: Vec<u8>,
+    peak: AtomicU64,
+    total: AtomicU64,
+}
+
+impl PeakStorage {
+    fn new(data: Vec<u8>) -> Self {
+        PeakStorage {
+            data,
+            peak: AtomicU64::new(0),
+            total: AtomicU64::new(0),
+        }
+    }
+
+    fn reset(&self) {
+        self.peak.store(0, Relaxed);
+        self.total.store(0, Relaxed);
+    }
+
+    fn served(&self, offset: u64, len: usize) -> Vec<u8> {
+        self.data
+            .as_slice()
+            .read_at(offset, len)
+            .unwrap()
+            .into_owned()
+    }
+}
+
+impl Storage for PeakStorage {
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        let got = self.served(offset, len);
+        self.peak.fetch_max(got.len() as u64, Relaxed);
+        self.total.fetch_add(got.len() as u64, Relaxed);
+        Ok(Cow::Owned(got))
+    }
+
+    fn len(&self) -> u64 {
+        self.data.len() as u64
+    }
+
+    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
+        let got: Vec<Vec<u8>> = ranges
+            .iter()
+            .map(|r| self.served(r.start, (r.end - r.start) as usize))
+            .collect();
+        let bytes: u64 = got.iter().map(|g| g.len() as u64).sum();
+        self.peak.fetch_max(bytes, Relaxed);
+        self.total.fetch_add(bytes, Relaxed);
+        Ok(got.into_iter().map(Cow::Owned).collect())
+    }
+}
+
+struct Chunked {
+    layout: DataLayout,
+    dataspace: Dataspace,
+    datatype: Datatype,
+    pipeline: Option<FilterPipeline>,
+    os: u8,
+    ls: u8,
+}
+
+/// The one chunked dataset of fixture `name`.
+fn chunked(bytes: &[u8]) -> Chunked {
+    let sb = Superblock::parse(bytes, 0).unwrap();
+    let (os, ls) = (sb.offset_size, sb.length_size);
+    for child in group_v2::resolve_group_children(bytes, &sb, sb.root_group_address).unwrap() {
+        let header =
+            ObjectHeader::parse(bytes, child.object_header_address as usize, os, ls).unwrap();
+        let msg = |t: MessageType| {
+            header
+                .messages
+                .iter()
+                .find(|m| m.msg_type == t)
+                .map(|m| m.data.clone())
+        };
+        let Some(dl) = msg(MessageType::DataLayout) else {
+            continue;
+        };
+        let layout = DataLayout::parse(&dl, os, ls).unwrap();
+        if !matches!(layout, DataLayout::Chunked { .. }) {
+            continue;
+        }
+        return Chunked {
+            layout,
+            datatype: Datatype::parse(&msg(MessageType::Datatype).unwrap())
+                .unwrap()
+                .0,
+            dataspace: Dataspace::parse(&msg(MessageType::Dataspace).unwrap(), ls).unwrap(),
+            pipeline: msg(MessageType::FilterPipeline).map(|p| FilterPipeline::parse(&p).unwrap()),
+            os,
+            ls,
+        };
+    }
+    panic!("no chunked dataset");
+}
+
+/// Claimed stored size of every chunk in the crafted file.
+const HUGE: u32 = 20 << 20;
+
+/// `chunked_large.h5` (1000 `i32` in ten gzip chunks, a v1 B-tree index)
+/// with `HUGE` bytes of padding appended and every chunk's index entry
+/// rewritten to claim `HUGE` stored bytes at the padding: ten chunks, 200
+/// MiB of extents, in a 20 MiB file.
+fn crafted() -> (Vec<u8>, Chunked, Vec<ChunkInfo>) {
+    let dir = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures");
+    let mut bytes = std::fs::read(dir.join("chunked_large.h5")).unwrap();
+    let ds = chunked(&bytes);
+    let es = ds.datatype.type_size() as usize;
+    let (chunks, _) = list_chunks(&bytes, &ds.layout, &ds.dataspace, es, ds.os, ds.ls).unwrap();
+    assert_eq!(chunks.len(), 10);
+    let blob = bytes.len() as u64;
+    for c in &chunks {
+        // v1 B-tree key (size, filter mask, offsets + 0) then the child
+        // address.
+        let mut pat = Vec::new();
+        pat.extend_from_slice(&c.chunk_size.to_le_bytes());
+        pat.extend_from_slice(&c.filter_mask.to_le_bytes());
+        // The key holds one offset per dimension plus the element offset
+        // (0); `offsets` may or may not list that last one.
+        for d in 0..=ds.dataspace.dimensions.len() {
+            pat.extend_from_slice(&c.offsets.get(d).copied().unwrap_or(0).to_le_bytes());
+        }
+        pat.extend_from_slice(&c.address.to_le_bytes());
+        let at = bytes
+            .windows(pat.len())
+            .position(|w| w == pat.as_slice())
+            .expect("chunk key");
+        bytes[at..at + 4].copy_from_slice(&HUGE.to_le_bytes());
+        let a = at + pat.len() - 8;
+        bytes[a..a + 8].copy_from_slice(&blob.to_le_bytes());
+    }
+    bytes.resize(bytes.len() + HUGE as usize, 0x5a);
+    let ds = chunked(&bytes);
+    let (chunks, _) = list_chunks(&bytes, &ds.layout, &ds.dataspace, es, ds.os, ds.ls).unwrap();
+    assert!(
+        chunks
+            .iter()
+            .all(|c| c.chunk_size == HUGE && c.address == blob)
+    );
+    (bytes, ds, chunks)
+}
+
+/// Most a crafted chunk of the fixture may fetch: its decoded size (400
+/// bytes) grown by one codec, generously.
+const CHUNK_LIMIT: u64 = 400 + 100 + 4096;
+
+#[test]
+fn crafted_chunk_index_cannot_amplify_fetches() {
+    let (bytes, ds, chunks) = crafted();
+    let st = PeakStorage::new(bytes.clone());
+    let pl = ds.pipeline.as_ref();
+    let (dl, sp, dt, os, ls) = (&ds.layout, &ds.dataspace, &ds.datatype, ds.os, ds.ls);
+    let check =
+        |what: &str, got: Result<Vec<u8>, FormatError>, want: Result<Vec<u8>, FormatError>| {
+            // Same outcome as slicing the whole file.
+            assert_eq!(got, want, "{what}");
+            let (peak, total) = (st.peak.load(Relaxed), st.total.load(Relaxed));
+            assert!(
+                peak <= RAW_BATCH_BYTES as u64,
+                "{what}: one fetch of {peak} bytes"
+            );
+            // Every chunk's fetch is bounded by what it can need, whatever its
+            // index entry claims (plus the index and header reads).
+            assert!(
+                total <= chunks.len() as u64 * CHUNK_LIMIT + 64 * 1024,
+                "{what}: fetched {total} bytes"
+            );
+            st.reset();
+        };
+    let slice: &[u8] = &bytes;
+
+    let sel = Selection::Hyperslab {
+        start: vec![100],
+        stride: vec![1],
+        count: vec![400],
+        block: vec![1],
+    };
+    check(
+        "selection",
+        read_raw_data_selection_in(&st, dl, sp, dt, pl, os, ls, &sel),
+        read_raw_data_selection_in(slice, dl, sp, dt, pl, os, ls, &sel),
+    );
+    check(
+        "full",
+        read_raw_data_full_in(&st, dl, sp, dt, pl, os, ls),
+        read_raw_data_full_in(slice, dl, sp, dt, pl, os, ls),
+    );
+    check(
+        "cached",
+        read_raw_data_cached_in(&st, dl, sp, dt, pl, os, ls, &ChunkCache::new()),
+        read_raw_data_cached_in(slice, dl, sp, dt, pl, os, ls, &ChunkCache::new()),
+    );
+    check(
+        "indexed",
+        read_raw_data_indexed_in(&st, dl, sp, dt, pl, os, ls, &ChunkCache::new()),
+        read_raw_data_indexed_in(slice, dl, sp, dt, pl, os, ls, &ChunkCache::new()),
+    );
+    check(
+        "sweep",
+        read_chunked_data_sweep_in(
+            &st,
+            dl,
+            sp,
+            dt,
+            pl,
+            os,
+            ls,
+            &ChunkCache::new(),
+            &mut SweepContext::new(4, 2),
+        ),
+        read_chunked_data_sweep_in(
+            slice,
+            dl,
+            sp,
+            dt,
+            pl,
+            os,
+            ls,
+            &ChunkCache::new(),
+            &mut SweepContext::new(4, 2),
+        ),
+    );
+    #[cfg(feature = "parallel")]
+    {
+        use clawhdf5_format::parallel_read::{
+            decompress_chunks_lane_partitioned_in, decompress_chunks_parallel_in,
+            decompress_chunks_sequential_in,
+        };
+        let pl = pl.unwrap();
+        let flat = |r: Result<Vec<Vec<u8>>, FormatError>| r.map(|v| v.concat());
+        check(
+            "parallel",
+            flat(decompress_chunks_parallel_in(&st, &chunks, pl, 400, 4)),
+            flat(decompress_chunks_parallel_in(slice, &chunks, pl, 400, 4)),
+        );
+        check(
+            "sequential",
+            flat(decompress_chunks_sequential_in(
+                &st,
+                &chunks,
+                Some(pl),
+                400,
+                4,
+            )),
+            flat(decompress_chunks_sequential_in(
+                slice,
+                &chunks,
+                Some(pl),
+                400,
+                4,
+            )),
+        );
+        check(
+            "lane partitioned",
+            flat(
+                decompress_chunks_lane_partitioned_in(&st, &chunks, pl, 400, 4, 7, Some(3))
+                    .map(|(v, _)| v),
+            ),
+            flat(
+                decompress_chunks_lane_partitioned_in(slice, &chunks, pl, 400, 4, 7, Some(3))
+                    .map(|(v, _)| v),
+            ),
+        );
+    }
+}
+
+/// Legitimately large chunks (unfiltered, 4 MiB each, 160 MiB in all) are
+/// fetched batch by batch: no call holds more than the batch budget, and
+/// the data is right.
+#[cfg(feature = "parallel")]
+#[test]
+fn large_reads_are_fetched_in_batches() {
+    use clawhdf5_format::parallel_read::decompress_chunks_sequential_in;
+    const CHUNK: usize = 4 << 20;
+    let data: Vec<u8> = (0..2 * CHUNK).map(|i| (i % 251) as u8).collect();
+    let chunks: Vec<ChunkInfo> = (0..40u64)
+        .map(|i| ChunkInfo {
+            chunk_size: CHUNK as u32,
+            filter_mask: 0,
+            offsets: vec![i * CHUNK as u64],
+            address: (i % 2) * CHUNK as u64,
+        })
+        .collect();
+    let st = PeakStorage::new(data.clone());
+    let got = decompress_chunks_sequential_in(&st, &chunks, None, CHUNK, 1).unwrap();
+    assert_eq!(got.len(), 40);
+    for (i, c) in got.iter().enumerate() {
+        let at = (i % 2) * CHUNK;
+        assert!(c == &data[at..at + CHUNK], "chunk {i}");
+    }
+    let peak = st.peak.load(Relaxed);
+    assert!(peak <= RAW_BATCH_BYTES as u64, "one fetch of {peak} bytes");
+    assert_eq!(st.total.load(Relaxed), 40 * CHUNK as u64);
+}

crates/clawhdf5-format/tests/storage_equivalence.rs

@@ -1,24 +1,19 @@
 //! Equivalence harness for the range-read migration
-//! (`docs/design/range-reads.md`, milestone M1).
+//! (`docs/design/range-reads.md`, milestones M1 and M2).
 //!
-//! Every metadata parser converted to [`Storage`] must give exactly what its
+//! Every parser converted to [`Storage`] must give exactly what its `&[u8]`
-//! `&[u8]` form gives. This walks real files — the fixtures, files h5py
+//! form gives. This walks real files — the fixtures, files h5py writes to
-//! writes to exercise the less common structures, and optionally the
+//! exercise the less common structures, and optionally the conformance
-//! conformance corpus — and, for every object, runs each converted parser
+//! corpus — and, for every object, runs each converted parser twice: over
-//! twice: over the file as a slice, and over a [`CountingStorage`] that
+//! the file as a slice, and over a [`CountingStorage`] that serves the same
-//! serves the same bytes through `read_at` only (`as_contiguous()` is
+//! bytes through `read_at` only (`as_contiguous()` is `None`, so no parser
-//! `None`, so no parser can fall back to the whole slice). The results must
+//! can fall back to the whole slice). The results must be identical, value
-//! be identical, value for value and error for error.
+//! for value and error for error.
 //!
-//! The one allowed difference is [`FormatError::ContiguousStorageRequired`]
+//! Nothing may answer [`FormatError::ContiguousStorageRequired`] any more:
-//! from the storage path, and only from the structures still indexed by a v2
+//! since milestone M2 every read path works over `read_at` alone, the v2
-//! B-tree (dense attributes, a SOHM B-tree index, huge fractal-heap objects;
+//! B-tree structures (dense groups and attributes, a SOHM B-tree index,
-//! see `CONTIGUOUS_REQUIRED`), which fail cleanly instead of reading the
+//! huge fractal-heap objects) included.
-//! whole file. Those are counted; the error from any other site or check
-//! fails the harness.
-//!
-//! Milestones M2/M3 extend `check_object` with the raw-data and group
-//! parsers as they are converted.
 //!
 //! - `CLAWHDF5_STORAGE_CORPUS=dir[:dir...]` adds every `.h5`/`.hdf5`/`.he5`/
 //!   `.nc`/`.h5ad` file under those directories (the conformance corpus is
@@ -38,23 +33,39 @@ use clawhdf5_format::attribute::{
 };
 use clawhdf5_format::attribute_info::AttributeInfoMessage;
 use clawhdf5_format::btree_v1::{collect_symbol_table_nodes, collect_symbol_table_nodes_in};
-use clawhdf5_format::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+use clawhdf5_format::btree_v2::{
+    BTreeV2Header, collect_btree_v2_records, collect_btree_v2_records_in, find_btree_v2_records,
+    find_btree_v2_records_in,
+};
+use clawhdf5_format::chunk_cache::ChunkCache;
+use clawhdf5_format::chunked_read::{list_chunks, list_chunks_in};
 use clawhdf5_format::data_layout::DataLayout;
+use clawhdf5_format::data_read::{
+    read_raw_data_cached, read_raw_data_cached_in, read_raw_data_full, read_raw_data_full_in,
+    read_raw_data_indexed, read_raw_data_indexed_in, read_raw_data_selection,
+    read_raw_data_selection_in,
+};
 use clawhdf5_format::dataspace::Dataspace;
 use clawhdf5_format::datatype::Datatype;
 use clawhdf5_format::error::FormatError;
 use clawhdf5_format::extensible_array::{
     ExtensibleArrayHeader, read_extensible_array_chunks, read_extensible_array_chunks_in,
 };
-use clawhdf5_format::fill_value::{dataset_fill_value_from_storage, dataset_fill_value_in};
+use clawhdf5_format::fill_value::{
+    dataset_fill_value_from_storage, dataset_fill_value_in, read_full_with_fill,
+    read_full_with_fill_in,
+};
+use clawhdf5_format::filter_pipeline::FilterPipeline;
 use clawhdf5_format::fixed_array::{
     FixedArrayHeader, read_fixed_array_chunks, read_fixed_array_chunks_in,
 };
 use clawhdf5_format::fractal_heap::FractalHeapHeader;
+use clawhdf5_format::group_v2;
 use clawhdf5_format::link_info::LinkInfoMessage;
 use clawhdf5_format::local_heap::LocalHeap;
 use clawhdf5_format::message_type::MessageType;
 use clawhdf5_format::object_header::ObjectHeader;
+use clawhdf5_format::selection::Selection;
 use clawhdf5_format::shared_message::{
     self, load_sohm_table, load_sohm_table_in, message_data_with_sohm, message_data_with_sohm_in,
     parse_sohm_btree_entries, parse_sohm_btree_entries_in, parse_sohm_list, parse_sohm_list_in,
@@ -67,50 +78,24 @@ use clawhdf5_format::superblock_ext::{
     read_superblock_extension_in,
 };
 use clawhdf5_format::symbol_table::{SymbolTableMessage, SymbolTableNode};
+use clawhdf5_format::vds::{
+    read_virtual_dataset, read_virtual_dataset_in, virtual_dataset_extent,
+    virtual_dataset_extent_in,
+};
+use clawhdf5_format::vl_data::{VlResolver, read_vl_bytes, read_vl_bytes_in};
 
 /// Objects visited per file, heap objects read per heap: enough to cover
 /// every structure kind while keeping a 35 000-group file fast.
 const MAX_OBJECTS: usize = 1500;
 const MAX_HEAP_IDS: usize = 200;
-
+/// Datasets larger than this are not read (their chunk indexes still are).
-/// The structures that still need the whole file in memory, because they
+const MAX_DATA_BYTES: u64 = 16 << 20;
-/// are found through a version-2 B-tree (not converted yet), and the checks
-/// that can reach each of them. Anything else answering
-/// [`FormatError::ContiguousStorageRequired`] is a converted parser falling
-/// back to the whole file, and fails the harness.
-const CONTIGUOUS_REQUIRED: &[(&str, &[&str])] = &[
-    (
-        "dense attribute storage (a v2 B-tree)",
-        &["attributes", "attributes (tolerant)"],
-    ),
-    (
-        "a shared-message B-tree index",
-        &[
-            "SOHM B-tree",
-            "shared message",
-            "fill value",
-            "attributes",
-            "attributes (tolerant)",
-        ],
-    ),
-    (
-        "a huge fractal-heap object's B-tree",
-        &["heap object", "attributes", "attributes (tolerant)"],
-    ),
-];
-
-fn may_require_contiguous(check: &str, site: &str) -> bool {
-    CONTIGUOUS_REQUIRED
-        .iter()
-        .any(|(s, checks)| *s == site && checks.contains(&check))
-}
 
 #[derive(Default, Debug)]
 struct Tally {
     files: usize,
     objects: usize,
     checks: usize,
-    contiguous_required: usize,
     reads: u64,
     bytes: u64,
     /// Chunk indexes (fixed and extensible arrays) read, and the most bytes
@@ -123,12 +108,14 @@ struct Walk<'a> {
     slice: &'a [u8],
     storage: &'a CountingStorage,
     name: String,
+    /// The file's directory, for external VDS sources.
+    dir: Option<PathBuf>,
     tally: &'a mut Tally,
 }
 
 impl Walk<'_> {
-    /// The storage result must equal the slice result, or be the clean
+    /// The storage result must equal the slice result; no parser may ask
-    /// "needs the whole file" error.
+    /// for the whole file.
     fn same<T: Debug>(
         &mut self,
         what: &str,
@@ -137,14 +124,11 @@ impl Walk<'_> {
     ) {
         self.tally.checks += 1;
         if let Err(FormatError::ContiguousStorageRequired(site)) = got {
-            assert!(
+            panic!(
-                may_require_contiguous(what, site),
+                "{}: {what} fell back to the whole file ({site}); every read path \
-                "{}: {what} fell back to the whole file ({site}), which only the \
+                 must work through read_at",
-                 v2-B-tree-indexed structures may do",
                 self.name
             );
-            self.tally.contiguous_required += 1;
-            return;
         }
         let (w, g) = (format!("{want:?}"), format!("{got:?}"));
         assert!(
@@ -206,13 +190,34 @@ impl Walk<'_> {
             }
             self.tally.objects += 1;
             self.check_object(&sb, addr);
-            // Traversal only (group lookups are milestone M0/M3 work).
+            let children = group_v2::resolve_group_children(slice, &sb, addr);
-            if let Ok(children) =
+            let got = group_v2::resolve_group_children_in(self.st(), &sb, addr);
-                clawhdf5_format::group_v2::resolve_group_children(slice, &sb, addr)
+            self.same("group listing", &children, &got);
-            {
+            if let Ok(children) = children {
+                for c in children.iter().take(MAX_HEAP_IDS) {
+                    let want = group_v2::resolve_child(slice, &sb, addr, &c.name);
+                    let got = group_v2::resolve_child_in(self.st(), &sb, addr, &c.name);
+                    self.same("child lookup", &want, &got);
+                }
+                // A name no group has: the lookup's not-found path.
+                let want = group_v2::resolve_child(slice, &sb, addr, "no such child");
+                let got = group_v2::resolve_child_in(self.st(), &sb, addr, "no such child");
+                self.same("child lookup (missing)", &want, &got);
                 queue.extend(children.iter().map(|c| c.object_header_address));
             }
         }
+        // Paths: every listed name from the root, and one that is missing.
+        if let Ok(children) = group_v2::resolve_group_children(slice, &sb, sb.root_group_address) {
+            for c in children.iter().take(MAX_HEAP_IDS) {
+                let path = format!("/{}", c.name);
+                let want = group_v2::resolve_path_any(slice, &sb, &path);
+                let got = group_v2::resolve_path_any_in(self.st(), &sb, &path);
+                self.same("path", &want, &got);
+            }
+        }
+        let want = group_v2::resolve_path_any(slice, &sb, "/no/such/path");
+        let got = group_v2::resolve_path_any_in(self.st(), &sb, "/no/such/path");
+        self.same("path (missing)", &want, &got);
     }
 
     fn check_object(&mut self, sb: &Superblock, addr: u64) {
@@ -280,6 +285,7 @@ impl Walk<'_> {
             }
         }
         self.check_layout(&header, os, ls);
+        self.check_data(&header, os, ls);
     }
 
     /// A symbol-table group: its local heap, B-tree, nodes and names.
@@ -336,12 +342,24 @@ impl Walk<'_> {
         let (Ok(fh), Some(index)) = (fh, index) else {
             return;
         };
-        let Ok(bt) = BTreeV2Header::parse(slice, index as usize, os, ls) else {
+        let bt = BTreeV2Header::parse(slice, index as usize, os, ls);
-            return;
+        self.same(
-        };
+            "v2 B-tree header",
-        let Ok(records) = collect_btree_v2_records(slice, &bt, os, ls) else {
+            &bt,
-            return;
+            &BTreeV2Header::parse_in(self.st(), index, os, ls),
-        };
+        );
+        let Ok(bt) = bt else { return };
+        let records = collect_btree_v2_records(slice, &bt, os, ls);
+        let got = collect_btree_v2_records_in(self.st(), &bt, os, ls);
+        self.same("v2 B-tree records", &records, &got);
+        let Ok(records) = records else { return };
+        // Descents to single records (by their bytes), as name lookups do.
+        for rec in records.iter().take(8) {
+            let key = rec.data.clone();
+            let want = find_btree_v2_records(slice, &bt, os, &mut |r| r.cmp(&key[..]));
+            let got = find_btree_v2_records_in(self.st(), &bt, os, &mut |r| r.cmp(&key[..]));
+            self.same("v2 B-tree descent", &want, &got);
+        }
         let id_len = fh.heap_id_length as usize;
         for rec in records.iter().take(MAX_HEAP_IDS) {
             let Some(id) = rec.data.get(id_at..id_at + id_len) else {
@@ -356,6 +374,179 @@ impl Walk<'_> {
         }
     }
 
+    /// A dataset's values through every raw-data path: whole reads (plain,
+    /// cached, indexed, fill-aware, virtual), chunk listings, selections
+    /// (a box, a strided hyperslab, points), VL strings and sequences.
+    fn check_data(&mut self, header: &ObjectHeader, os: u8, ls: u8) {
+        let slice = self.slice;
+        let find = |t: MessageType| {
+            header
+                .messages
+                .iter()
+                .find(|m| m.msg_type == t)
+                .and_then(|m| shared_message::message_data_with_sohm(slice, m, os, ls).ok())
+        };
+        let (Some(dt), Some(ds), Some(dl)) = (
+            find(MessageType::Datatype),
+            find(MessageType::Dataspace),
+            find(MessageType::DataLayout),
+        ) else {
+            return;
+        };
+        let (Ok((dt, _)), Ok(ds), Ok(dl)) = (
+            Datatype::parse(&dt),
+            Dataspace::parse(&ds, ls),
+            DataLayout::parse(&dl, os, ls),
+        ) else {
+            return;
+        };
+        let pipeline = match find(MessageType::FilterPipeline).map(|p| FilterPipeline::parse(&p)) {
+            Some(Ok(p)) => Some(p),
+            Some(Err(_)) => return,
+            None => None,
+        };
+        let pl = pipeline.as_ref();
+        let elem = dt.type_size() as u64;
+        let bytes = ds
+            .dimensions
+            .iter()
+            .try_fold(elem, |a, &d| a.checked_mul(d));
+        if bytes.is_none_or(|b| b > MAX_DATA_BYTES) {
+            return;
+        }
+
+        if matches!(dl, DataLayout::Virtual { .. }) {
+            let resolver = self.resolver();
+            let r: &clawhdf5_format::vds::VdsFileResolver = &resolver;
+            let want = virtual_dataset_extent(slice, &dl, &ds, os, ls, Some(r));
+            let got = virtual_dataset_extent_in(self.st(), &dl, &ds, os, ls, Some(r));
+            self.same("VDS extent", &want, &got);
+            let want = read_virtual_dataset(slice, &dl, &ds, &dt, None, os, ls, Some(r));
+            let got = read_virtual_dataset_in(self.st(), &dl, &ds, &dt, None, os, ls, Some(r));
+            self.same("VDS read", &want, &got);
+            return;
+        }
+
+        let want = read_raw_data_full(slice, &dl, &ds, &dt, pl, os, ls);
+        let got = read_raw_data_full_in(self.st(), &dl, &ds, &dt, pl, os, ls);
+        self.same("raw data", &want, &got);
+        let want_fill = read_full_with_fill(
+            &header.messages,
+            slice,
+            &dl,
+            &ds,
+            elem as usize,
+            os,
+            ls,
+            || read_raw_data_full(slice, &dl, &ds, &dt, pl, os, ls),
+        );
+        let got_fill = read_full_with_fill_in(
+            &header.messages,
+            self.st(),
+            &dl,
+            &ds,
+            elem as usize,
+            os,
+            ls,
+            || read_raw_data_full_in(self.st(), &dl, &ds, &dt, pl, os, ls),
+        );
+        self.same("raw data with fill", &want_fill, &got_fill);
+
+        if matches!(dl, DataLayout::Chunked { .. }) {
+            let want = list_chunks(slice, &dl, &ds, elem as usize, os, ls);
+            let got = list_chunks_in(self.st(), &dl, &ds, elem as usize, os, ls);
+            self.same("chunk list", &want, &got);
+            // Through a chunk cache, twice (the second read is served from
+            // it), and through the indexed path.
+            let (c1, c2) = (ChunkCache::new(), ChunkCache::new());
+            for _ in 0..2 {
+                let want = read_raw_data_cached(slice, &dl, &ds, &dt, pl, os, ls, &c1);
+                let got = read_raw_data_cached_in(self.st(), &dl, &ds, &dt, pl, os, ls, &c2);
+                // A cache lists the chunks in hash-map order (see below).
+                if want.is_err() && got.is_err() {
+                    self.tally.checks += 1;
+                } else {
+                    self.same("raw data (cached)", &want, &got);
+                }
+            }
+            let (c1, c2) = (ChunkCache::new(), ChunkCache::new());
+            let want = read_raw_data_indexed(slice, &dl, &ds, &dt, pl, os, ls, &c1);
+            let got = read_raw_data_indexed_in(self.st(), &dl, &ds, &dt, pl, os, ls, &c2);
+            // The indexed path decodes chunks in hash-map order, so which
+            // failing chunk it reports varies between two caches (with the
+            // slice alone, too): only whether it fails must agree.
+            if want.is_err() && got.is_err() {
+                self.tally.checks += 1;
+            } else {
+                self.same("raw data (indexed)", &want, &got);
+            }
+        }
+
+        let dims = &ds.dimensions;
+        if !dims.is_empty() && dims.iter().all(|&d| d > 0) {
+            let rank = dims.len();
+            let ones = vec![1u64; rank];
+            let quarter = Selection::Hyperslab {
+                start: dims.iter().map(|&d| d / 4).collect(),
+                stride: ones.clone(),
+                count: dims.iter().map(|&d| (d / 3).max(1)).collect(),
+                block: ones.clone(),
+            };
+            let mut stride = ones.clone();
+            stride[rank - 1] = 2;
+            let mut count = dims.clone();
+            count[rank - 1] = dims[rank - 1].div_ceil(2);
+            let strided = Selection::Hyperslab {
+                start: vec![0; rank],
+                stride,
+                count,
+                block: ones.clone(),
+            };
+            let points = Selection::Points(vec![
+                dims.iter().map(|&d| d - 1).collect(),
+                vec![0; rank],
+                dims.iter().map(|&d| d / 2).collect(),
+            ]);
+            for (what, sel) in [
+                ("selection (box)", &quarter),
+                ("selection (strided)", &strided),
+                ("selection (points)", &points),
+            ] {
+                let want = read_raw_data_selection(slice, &dl, &ds, &dt, pl, os, ls, sel);
+                let got = read_raw_data_selection_in(self.st(), &dl, &ds, &dt, pl, os, ls, sel);
+                self.same(what, &want, &got);
+            }
+        }
+
+        // Variable-length strings and sequences, resolved in the global heap.
+        if let (Datatype::VariableLength { base_type, .. }, Ok(raw)) = (&dt, &want) {
+            let n = raw.len() / clawhdf5_format::vl_data::element_size(os).max(1);
+            let raw = &raw[..n * clawhdf5_format::vl_data::element_size(os)];
+            let want = VlResolver::new(slice, os, ls).string_bytes(raw);
+            let got = VlResolver::new_in(self.st(), os, ls).string_bytes(raw);
+            self.same("VL strings", &want, &got);
+            let base = base_type.type_size() as usize;
+            let want = VlResolver::new(slice, os, ls).sequences(raw, base);
+            let got = VlResolver::new_in(self.st(), os, ls).sequences(raw, base);
+            self.same("VL sequences", &want, &got);
+            let want = read_vl_bytes(slice, raw, n as u64, os, ls);
+            let got = read_vl_bytes_in(self.st(), raw, n as u64, os, ls);
+            self.same("VL bytes", &want, &got);
+        }
+    }
+
+    /// External VDS source files: siblings of the file being walked.
+    fn resolver(&self) -> impl Fn(&str) -> Result<Option<Vec<u8>>, FormatError> + use<> {
+        let dir = self.dir.clone();
+        move |name: &str| {
+            let (Some(dir), false) = (dir.as_ref(), name.contains("..") || name.starts_with('/'))
+            else {
+                return Ok(None);
+            };
+            Ok(std::fs::read(dir.join(name)).ok())
+        }
+    }
+
     /// A dataset's layout: VDS mappings, and fixed/extensible array chunk
     /// indexes.
     fn check_layout(&mut self, header: &ObjectHeader, os: u8, ls: u8) {
@@ -465,10 +656,19 @@ fn check_file(path: &Path, tally: &mut Tally) {
     let Ok(bytes) = std::fs::read(path) else {
         return;
     };
-    check_bytes(&path.display().to_string(), &bytes, tally);
+    check_bytes_in(
+        &path.display().to_string(),
+        &bytes,
+        path.parent().map(Path::to_path_buf),
+        tally,
+    );
 }
 
 fn check_bytes(name: &str, bytes: &[u8], tally: &mut Tally) {
+    check_bytes_in(name, bytes, None, tally);
+}
+
+fn check_bytes_in(name: &str, bytes: &[u8], dir: Option<PathBuf>, tally: &mut Tally) {
     let Ok((_, hdf5)) = split_user_block(bytes) else {
         return;
     };
@@ -478,6 +678,7 @@ fn check_bytes(name: &str, bytes: &[u8], tally: &mut Tally) {
         slice: hdf5,
         storage: &storage,
         name: name.to_string(),
+        dir,
         tally,
     };
     walk.run();
@@ -552,6 +753,362 @@ fn corpus_parses_identically_through_storage() {
     assert!(tally.files > 0);
 }
 
+/// The whole read of every object reachable from the root group of `sb`'s
+/// file — each dataset whole (fill-aware) and half of it through a
+/// selection, and each group's listing — as one result to compare.
+fn read_everything(file: &dyn Storage, sb: &Superblock) -> Result<String, FormatError> {
+    let (os, ls) = (sb.offset_size, sb.length_size);
+    let mut out = String::new();
+    let mut queue = VecDeque::from([sb.root_group_address]);
+    let mut seen = HashSet::new();
+    while let Some(addr) = queue.pop_front() {
+        if seen.len() > 200 || !seen.insert(addr) {
+            continue;
+        }
+        let header = ObjectHeader::parse_in(file, addr, os, ls)?;
+        let find = |t: MessageType| {
+            header
+                .messages
+                .iter()
+                .find(|m| m.msg_type == t)
+                .map(|m| message_data_with_sohm_in(file, m, os, ls))
+                .transpose()
+        };
+        if let (Some(dt), Some(ds), Some(dl)) = (
+            find(MessageType::Datatype)?,
+            find(MessageType::Dataspace)?,
+            find(MessageType::DataLayout)?,
+        ) {
+            let dt = Datatype::parse(&dt)?.0;
+            let ds = Dataspace::parse(&ds, ls)?;
+            let dl = DataLayout::parse(&dl, os, ls)?;
+            let pl = find(MessageType::FilterPipeline)?
+                .map(|p| FilterPipeline::parse(&p))
+                .transpose()?;
+            let data = read_full_with_fill_in(
+                &header.messages,
+                file,
+                &dl,
+                &ds,
+                dt.type_size() as usize,
+                os,
+                ls,
+                || read_raw_data_full_in(file, &dl, &ds, &dt, pl.as_ref(), os, ls),
+            );
+            out.push_str(&format!("{addr}: {data:?}\n"));
+            if let Some(&d0) = ds.dimensions.first() {
+                let rank = ds.dimensions.len();
+                let sel = Selection::Hyperslab {
+                    start: vec![0; rank],
+                    stride: vec![1; rank],
+                    count: std::iter::once(d0.div_ceil(2))
+                        .chain(ds.dimensions[1..].iter().copied())
+                        .collect(),
+                    block: vec![1; rank],
+                };
+                let part =
+                    read_raw_data_selection_in(file, &dl, &ds, &dt, pl.as_ref(), os, ls, &sel);
+                out.push_str(&format!("{addr} half: {part:?}\n"));
+            }
+        }
+        let children = group_v2::resolve_group_children_in(file, sb, addr);
+        out.push_str(&format!("{addr} children: {children:?}\n"));
+        if let Ok(c) = children {
+            queue.extend(c.iter().map(|c| c.object_header_address));
+        }
+    }
+    Ok(out)
+}
+
+/// A storage that misbehaves: fails its `fail_at`-th read (1-based, `0`
+/// never), and, with `short`, serves one byte less than asked for inside
+/// the file (a truncated response).
+struct Adversary {
+    data: Vec<u8>,
+    reads: std::sync::atomic::AtomicUsize,
+    fail_at: usize,
+    short: bool,
+}
+
+impl Storage for Adversary {
+    fn read_at(&self, offset: u64, len: usize) -> Result<std::borrow::Cow<'_, [u8]>, FormatError> {
+        let n = self
+            .reads
+            .fetch_add(1, std::sync::atomic::Ordering::Relaxed)
+            + 1;
+        if n == self.fail_at {
+            return Err(FormatError::Storage(format!(
+                "injected failure of read {n}"
+            )));
+        }
+        let got = self.data.as_slice().read_at(offset, len)?;
+        let mut v = got.into_owned();
+        if self.short && v.len() > 1 {
+            v.pop();
+        }
+        Ok(std::borrow::Cow::Owned(v))
+    }
+
+    fn len(&self) -> u64 {
+        self.data.len() as u64
+    }
+}
+
+/// Every group listing and every dataset's values (whole, fill-aware and
+/// through a selection) read through a storage that fails or serves short
+/// reads: each result is an error or exactly the in-memory result, never
+/// other data; and a failing read is reported as that failure.
+#[test]
+fn misbehaving_storage_never_returns_wrong_data() {
+    let dir = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures");
+    let mut files = Vec::new();
+    hdf5_files(&dir, &mut files);
+    files.sort();
+    let (mut compared, mut failures_seen) = (0usize, 0usize);
+    for path in &files {
+        let Ok(bytes) = std::fs::read(path) else {
+            continue;
+        };
+        let Ok((_, hdf5)) = split_user_block(&bytes) else {
+            continue;
+        };
+        let Ok(sb) = Superblock::parse(hdf5, 0) else {
+            continue;
+        };
+        let everything = |file: &dyn Storage| read_everything(file, &sb);
+        let want = everything(&hdf5);
+        let counting = CountingStorage::new(hdf5.to_vec());
+        assert_eq!(
+            format!("{:?}", everything(&counting)),
+            format!("{want:?}"),
+            "{}",
+            path.display()
+        );
+        let total = counting.reads() as usize;
+        let step = (total / 25).max(1);
+        for fail_at in (1..=total).step_by(step) {
+            for short in [false, true] {
+                if short && fail_at != 1 {
+                    continue;
+                }
+                let adv = Adversary {
+                    data: hdf5.to_vec(),
+                    reads: Default::default(),
+                    fail_at: if short { 0 } else { fail_at },
+                    short,
+                };
+                let got = everything(&adv);
+                compared += 1;
+                match (&got, &want) {
+                    (Ok(g), Ok(w)) => {
+                        // Per-object results inside may be errors; values
+                        // that were read must be the right ones.
+                        for (gl, wl) in g.lines().zip(w.lines()) {
+                            if gl != wl {
+                                assert!(
+                                    gl.contains("Err("),
+                                    "{}: fail_at {fail_at} short {short}:\n got  {gl}\n want {wl}",
+                                    path.display()
+                                );
+                                failures_seen += 1;
+                                // A listing that failed ends the walk
+                                // differently from here on.
+                                if gl.contains("children: Err(") {
+                                    break;
+                                }
+                            }
+                        }
+                    }
+                    (Err(FormatError::Storage(_)), _) => failures_seen += 1,
+                    (Err(e), Ok(_)) => panic!(
+                        "{}: fail_at {fail_at} short {short}: {e:?} instead of a storage error",
+                        path.display()
+                    ),
+                    (Err(_), Err(_)) => {}
+                    // A listing failed, so the walk never reached the
+                    // object that fails in memory.
+                    (Ok(g), Err(e)) => assert!(
+                        g.contains("children: Err(Storage"),
+                        "{}: fail_at {fail_at} short {short}: read where memory fails ({e:?})",
+                        path.display()
+                    ),
+                }
+            }
+        }
+    }
+    eprintln!("misbehaving storage: {compared} runs, {failures_seen} failures reported");
+    assert!(
+        compared > 500 && failures_seen > 100,
+        "{compared} {failures_seen}"
+    );
+}
+
+/// A storage that breaks `read_at`'s contract the other way: every read
+/// comes back with 37 bytes more than asked for (junk past the range).
+struct Overlong {
+    data: Vec<u8>,
+}
+
+impl Storage for Overlong {
+    fn read_at(&self, offset: u64, len: usize) -> Result<std::borrow::Cow<'_, [u8]>, FormatError> {
+        let mut v = self.data.as_slice().read_at(offset, len)?.into_owned();
+        v.extend(std::iter::repeat_n(0xa5, 37));
+        Ok(std::borrow::Cow::Owned(v))
+    }
+
+    fn len(&self) -> u64 {
+        self.data.len() as u64
+    }
+}
+
+/// Bytes past the range asked for are never used: every read through a
+/// storage that returns more than asked gives exactly the in-memory result.
+#[test]
+fn overlong_reads_are_cut_to_the_range_asked_for() {
+    let dir = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures");
+    let mut files = Vec::new();
+    hdf5_files(&dir, &mut files);
+    files.sort();
+    let mut compared = 0;
+    for path in &files {
+        let Ok(bytes) = std::fs::read(path) else {
+            continue;
+        };
+        let Ok((_, hdf5)) = split_user_block(&bytes) else {
+            continue;
+        };
+        let Ok(sb) = Superblock::parse(hdf5, 0) else {
+            continue;
+        };
+        let want = read_everything(&hdf5, &sb);
+        let got = read_everything(
+            &Overlong {
+                data: hdf5.to_vec(),
+            },
+            &sb,
+        );
+        assert_eq!(
+            format!("{got:?}"),
+            format!("{want:?}"),
+            "{}",
+            path.display()
+        );
+        compared += 1;
+    }
+    assert!(compared > 40, "{compared}");
+}
+
+/// A read_at-only storage that also counts `read_ranges` calls and ranges.
+struct BatchCounting {
+    inner: CountingStorage,
+    batches: std::sync::atomic::AtomicUsize,
+    ranges: std::sync::atomic::AtomicUsize,
+}
+
+impl Storage for BatchCounting {
+    fn read_at(&self, offset: u64, len: usize) -> Result<std::borrow::Cow<'_, [u8]>, FormatError> {
+        self.inner.read_at(offset, len)
+    }
+
+    fn len(&self) -> u64 {
+        self.inner.len()
+    }
+
+    fn read_ranges(
+        &self,
+        ranges: &[std::ops::Range<u64>],
+    ) -> Result<Vec<std::borrow::Cow<'_, [u8]>>, FormatError> {
+        use std::sync::atomic::Ordering::Relaxed;
+        self.batches.fetch_add(1, Relaxed);
+        self.ranges.fetch_add(ranges.len(), Relaxed);
+        ranges
+            .iter()
+            .map(|r| self.inner.read_at(r.start, (r.end - r.start) as usize))
+            .collect()
+    }
+}
+
+/// A chunked read lists its chunks, then fetches all their bytes with one
+/// `read_ranges` call (a remote backend coalesces and parallelises it), and
+/// a selection fetches only the chunks it overlaps, in one call too.
+#[test]
+fn chunked_reads_fetch_their_chunks_in_one_batch() {
+    use std::sync::atomic::Ordering::Relaxed;
+    let dir = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures");
+    let mut datasets = 0;
+    for name in [
+        "chunked_large.h5",
+        "chunked_deflate.h5",
+        "chunked_2d.h5",
+        "v4_fixed_array.h5",
+    ] {
+        let bytes = std::fs::read(dir.join(name)).unwrap();
+        let sb = Superblock::parse(&bytes, 0).unwrap();
+        let (os, ls) = (sb.offset_size, sb.length_size);
+        let st = BatchCounting {
+            inner: CountingStorage::new(bytes.clone()),
+            batches: Default::default(),
+            ranges: Default::default(),
+        };
+        for child in group_v2::resolve_group_children(&bytes, &sb, sb.root_group_address).unwrap() {
+            let header =
+                ObjectHeader::parse(&bytes, child.object_header_address as usize, os, ls).unwrap();
+            let msg = |t: MessageType| {
+                header
+                    .messages
+                    .iter()
+                    .find(|m| m.msg_type == t)
+                    .map(|m| m.data.clone())
+            };
+            let Some(dl) = msg(MessageType::DataLayout) else {
+                continue;
+            };
+            let dl = DataLayout::parse(&dl, os, ls).unwrap();
+            if !matches!(dl, DataLayout::Chunked { .. }) {
+                continue;
+            }
+            let dt = Datatype::parse(&msg(MessageType::Datatype).unwrap())
+                .unwrap()
+                .0;
+            let ds = Dataspace::parse(&msg(MessageType::Dataspace).unwrap(), ls).unwrap();
+            let pl = msg(MessageType::FilterPipeline).map(|p| FilterPipeline::parse(&p).unwrap());
+            let es = dt.type_size() as usize;
+            let (chunks, _) = list_chunks(&bytes, &dl, &ds, es, os, ls).unwrap();
+            let want = read_raw_data_full(&bytes, &dl, &ds, &dt, pl.as_ref(), os, ls).unwrap();
+            st.batches.store(0, Relaxed);
+            st.ranges.store(0, Relaxed);
+            let got = read_raw_data_full_in(&st, &dl, &ds, &dt, pl.as_ref(), os, ls).unwrap();
+            assert_eq!(got, want, "{name} {}", child.name);
+            assert_eq!(st.batches.load(Relaxed), 1, "{name} {}", child.name);
+            assert_eq!(
+                st.ranges.load(Relaxed),
+                chunks.len(),
+                "{name} {}",
+                child.name
+            );
+            // The first chunk only.
+            let rank = ds.dimensions.len();
+            let sel = Selection::Hyperslab {
+                start: vec![0; rank],
+                stride: vec![1; rank],
+                count: vec![1; rank],
+                block: vec![1; rank],
+            };
+            let want = read_raw_data_selection(&bytes, &dl, &ds, &dt, pl.as_ref(), os, ls, &sel);
+            st.batches.store(0, Relaxed);
+            st.ranges.store(0, Relaxed);
+            let got = read_raw_data_selection_in(&st, &dl, &ds, &dt, pl.as_ref(), os, ls, &sel);
+            assert_eq!(got, want, "{name} {}", child.name);
+            if chunks.len() > 2 {
+                assert_eq!(st.batches.load(Relaxed), 1, "{name} {}", child.name);
+                assert_eq!(st.ranges.load(Relaxed), 1, "{name} {}", child.name);
+            }
+            datasets += 1;
+        }
+    }
+    assert!(datasets >= 4, "{datasets}");
+}
+
 fn python() -> String {
     std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
 }
@@ -696,6 +1253,4 @@ fn h5py_files_parse_identically_through_storage() {
     }
     eprintln!("h5py files: {tally:?}");
     assert!(tally.objects >= 700, "{tally:?}");
-    // Dense attributes and the SOHM B-tree are the known clean errors.
-    assert!(tally.contiguous_required > 0, "{tally:?}");
 }

crates/clawhdf5-remote/Cargo.toml

@@ -0,0 +1,39 @@
+[package]
+name = "clawhdf5-remote"
+version = "2.7.0"
+edition = "2024"
+rust-version.workspace = true
+description = "Read HDF5 files over HTTP(S) range requests and object stores (S3, GCS, Azure) with clawhdf5, through a block cache"
+license = "MIT"
+repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
+readme = "README.md"
+keywords = ["hdf5", "http", "s3", "range-requests", "science"]
+categories = ["science", "network-programming"]
+
+[features]
+# Plain-HTTP range reads and the block cache: pure Rust, no TLS stack, no C.
+default = ["http"]
+http = ["dep:ureq"]
+# HTTPS through rustls (ring provider, Mozilla roots). ring compiles C and
+# assembly, so this is not part of the default build.
+https = ["http", "ureq/rustls"]
+# Any object_store backend (in-memory, local files, or one you configure),
+# driven by a small tokio runtime the storage owns. Pure Rust.
+object-store = ["dep:object_store", "dep:tokio", "dep:futures-util"]
+# s3:// gs:// az:// URLs in open_url, credentials from the environment.
+# object_store's cloud clients use aws-lc-rs (C), hence opt-in.
+s3 = ["object-store", "object_store/aws"]
+gcs = ["object-store", "object_store/gcp"]
+azure = ["object-store", "object_store/azure"]
+
+[dependencies]
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+ureq = { version = "3.4", optional = true, default-features = false }
+object_store = { version = "0.14", optional = true, default-features = false, features = ["fs"] }
+tokio = { version = "1", optional = true, default-features = false, features = ["rt-multi-thread"] }
+futures-util = { version = "0.3", optional = true, default-features = false, features = ["std"] }
+
+[dev-dependencies]
+tempfile = { workspace = true }
+tokio = { version = "1", default-features = false, features = ["rt"] }

crates/clawhdf5-remote/README.md

@@ -0,0 +1,116 @@
+# clawhdf5-remote
+
+Read HDF5 files where they live — on an HTTP(S) server or in an object
+store (S3, GCS, Azure) — with
+[clawhdf5](../../README.md), without downloading them first.
+
+```rust
+let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/data.h5")?;
+let temps = file.dataset("/grid/temperature")?.read_f64()?;
+```
+
+The result is an ordinary `clawhdf5::File`: groups, datasets, attributes,
+selections, variable-length data. Only the bytes an operation needs are
+fetched, by `Range` requests, through a block cache.
+
+## How it reads
+
+- **Block cache** (`BlockCache`, mandatory for remote files; see
+  `docs/design/range-reads.md` §2): aligned blocks of 1 MiB by default, LRU
+  with a byte budget (64 MiB by default), the missing blocks of one read
+  fetched as runs of consecutive blocks (a gap of one block is fetched to
+  merge two runs), each request at most 8 MiB, the requests of one read in
+  parallel. A read that misses more than half the budget is not kept, so a
+  big dataset read does not evict the metadata. Readers on several threads
+  share one cache; a block is never fetched twice at once — a second reader
+  waits for the first one's request.
+- **Opening costs one request**: a `GET` of the first block, whose
+  `Content-Range` gives the file's length. HDF5 files keep the superblock
+  and usually the root group's metadata there, so listing a small file
+  often needs nothing more.
+- **Pinned to one version of the file**: a strong `ETag` is sent back as
+  `If-Match` (else `Last-Modified` as `If-Unmodified-Since`) and checked on
+  every response, as is the length. A file replaced while open is an error
+  (`RemoteError::FileChanged`), never a mix of old and new bytes. A server
+  with neither validator can only be checked by length;
+  `HttpOptions::require_validator` refuses it.
+- **Servers that ignore `Range`** (answer `200` with the whole file) are
+  refused (`RemoteError::RangeNotSupported`) without reading the body,
+  unless `HttpOptions::allow_full_download` is set; then the file is
+  downloaded once and read from memory.
+  A `200` whose body is no longer than the range asked for is the whole
+  (small) file, and is accepted.
+- **Retries**: connection failures, timeouts, `408`/`429`/`5xx` and bodies
+  that end early are retried with exponential backoff (3 retries, from
+  200 ms). Bodies are requested with `Accept-Encoding: identity`; an encoded
+  body is refused.
+- **Timeouts** scale with the request: `HttpOptions::timeout` (30 s) to
+  connect and to receive the headers, and for the body that plus its size
+  at `HttpOptions::min_speed` (16 KiB/s) — a slow link is not cut off
+  mid-block, a stalled connection still fails.
+- **Redirects** are followed up to `HttpOptions::max_redirects` (5; 0
+  refuses them), never from `https` to `http`. Once a redirect leaves the
+  URL's origin (scheme, host, port), `HttpOptions::headers` (API keys,
+  `Authorization`, cookies) are no longer sent.
+- **Credentials stay out of messages**: every error and `Debug` output
+  shows URLs through `redact_url` — no `user:password@`, query values
+  replaced by `REDACTED` (a presigned S3/GCS URL's signature lives there).
+- **Claimed lengths are not trusted**: nothing is allocated for the length
+  a server reports; a read spanning more than the cache budget is fetched
+  in pieces as data arrives, and `download(&storage, max_bytes)` reads a
+  whole file only up to a limit (`DEFAULT_MAX_DOWNLOAD`, 1 GiB).
+
+The zero-copy methods of `clawhdf5` (`read_raw_ref`, `read_*_zerocopy`,
+`File::as_bytes`) borrow the whole file from memory, so they are errors
+(`as_bytes` a panic; use `File::contiguous_bytes`) on a remote file.
+
+## Features
+
+| Feature | What | C code |
+|---|---|---|
+| `http` (default) | `http://` through `ureq`, no TLS | none |
+| `https` | `https://` through rustls, ring provider, Mozilla roots | ring (C and assembly) |
+| `object-store` | `ObjectStoreStorage` and `open_object` over any [`object_store`](https://docs.rs/object_store) store (in-memory, local files, or one you configure) | none |
+| `s3`, `gcs`, `azure` | `s3://bucket/key`, `gs://bucket/key`, `az://container/key` in `open_url`, configured from the environment (`AWS_*`, `GOOGLE_*`, `AZURE_*`) as object_store's `from_env` builders read it | aws-lc-rs (object_store's cloud clients) |
+
+The default build and `object-store` compile no C (`scripts/ci-test.sh`
+checks both).
+
+## Object stores
+
+`object_store` is async; `Storage` is synchronous (parsing is CPU work).
+`ObjectStoreStorage` owns a small tokio runtime (two worker threads): each
+read runs there while the calling thread waits, so it works from any
+thread, several at once — including `tokio::task::spawn_blocking` and code
+inside another runtime (where `spawn_blocking` is still the better place,
+since a read blocks the thread it is called on). The object is pinned by its ETag
+(`If-Match`, and compared on every response), else its version or
+modification time, and its size. The ranges of one read are fetched
+concurrently (up to 8).
+
+```rust
+use std::sync::Arc;
+use clawhdf5_remote::object_store::{memory::InMemory, ObjectStore};
+let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+// ... put a file at "data.h5" ...
+let (file, cache) = clawhdf5_remote::open_object(store, "data.h5", &Default::default())?;
+```
+
+The tests use object_store's in-memory and local-file stores; no cloud
+account is needed. The cloud schemes are only built (and unit-tested for
+URL parsing) in CI, not run against a real bucket.
+
+## Counting requests
+
+```rust
+use clawhdf5_remote::{storage_for_url, Options};
+let storage = storage_for_url(url, &Options::default())?;
+let file = clawhdf5::File::open_storage(storage.clone())?;
+// ... read ...
+let s = storage.stats(); // requests, bytes_fetched, hits, misses, cached_bytes, ...
+```
+
+`cargo run -p clawhdf5-remote --example range_server -- DIR` serves a
+directory with range support (the server the tests use), and
+`cargo run -p clawhdf5-remote --example read_url -- URL [DATASET]` lists a
+file and prints what it cost.

crates/clawhdf5-remote/examples/range_server.rs

@@ -0,0 +1,56 @@
+//! Serve the HDF5 files of a directory over HTTP with range support — the
+//! server the tests use — to try `clawhdf5_remote` and `h5rs` on URLs.
+//!
+//! ```text
+//! cargo run -p clawhdf5-remote --example range_server -- DIR [127.0.0.1:8000]
+//! ```
+//!
+//! Every file under `DIR` is served at its path relative to `DIR`. The
+//! files are read into memory at start. Requests are logged to stderr.
+
+#[path = "../tests/common/server.rs"]
+mod server;
+
+fn main() {
+    let mut args = std::env::args().skip(1);
+    let Some(dir) = args.next() else {
+        eprintln!("usage: range_server DIR [ADDR]");
+        std::process::exit(2);
+    };
+    let addr = args.next().unwrap_or_else(|| "127.0.0.1:8000".into());
+    let root = std::path::PathBuf::from(&dir);
+    let mut files = Vec::new();
+    let mut stack = vec![root.clone()];
+    while let Some(d) = stack.pop() {
+        let Ok(entries) = std::fs::read_dir(&d) else {
+            continue;
+        };
+        for e in entries.flatten() {
+            let p = e.path();
+            if p.is_dir() {
+                stack.push(p);
+            } else if let (Ok(rel), Ok(bytes)) = (p.strip_prefix(&root), std::fs::read(&p)) {
+                let url = format!("/{}", rel.to_string_lossy().replace('\\', "/"));
+                files.push((url, bytes));
+            }
+        }
+    }
+    files.sort();
+    let server = server::Server::bind(&addr, files.clone());
+    for (path, bytes) in &files {
+        eprintln!("{} ({} bytes)", server.url(path), bytes.len());
+    }
+    eprintln!("serving {} files on http://{}", files.len(), server.addr);
+    let mut logged = 0;
+    loop {
+        std::thread::sleep(std::time::Duration::from_millis(200));
+        let log = server.log();
+        for (path, range) in &log[logged.min(log.len())..] {
+            match range {
+                Some((a, b)) => eprintln!("GET {path} bytes={a}-{b}"),
+                None => eprintln!("GET {path} (whole file)"),
+            }
+        }
+        logged = log.len();
+    }
+}

crates/clawhdf5-remote/examples/read_url.rs

@@ -0,0 +1,45 @@
+//! List a remote HDF5 file and read one dataset, then print what it cost.
+//!
+//! ```text
+//! cargo run -p clawhdf5-remote --example read_url -- URL [DATASET]
+//! ```
+
+use clawhdf5::File;
+use clawhdf5_remote::{Options, storage_for_url};
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let mut args = std::env::args().skip(1);
+    let url = args.next().ok_or("usage: read_url URL [DATASET]")?;
+    let dataset = args.next();
+
+    let storage = storage_for_url(&url, &Options::default())?;
+    let file = File::open_storage(storage.clone())?;
+
+    // Walk the tree.
+    let mut stack = vec![("/".to_string(), file.root())];
+    while let Some((path, group)) = stack.pop() {
+        for (name, addr) in group.entries()? {
+            let child = format!("{}/{name}", path.trim_end_matches('/'));
+            match file.dataset_at(addr) {
+                Ok(ds) => println!("{child}  dataset {:?} {:?}", ds.shape()?, ds.dtype()?),
+                Err(_) => {
+                    println!("{child}  group");
+                    stack.push((child, file.group_at(addr)));
+                }
+            }
+        }
+    }
+
+    if let Some(name) = dataset {
+        let values = file.dataset(&name)?.read_f64()?;
+        let shown: Vec<_> = values.iter().take(8).collect();
+        println!("{name}: {} values, first {shown:?}", values.len());
+    }
+
+    let s = storage.stats();
+    println!(
+        "{} range requests (the one at open included), {} bytes fetched, {} bytes cached",
+        s.requests, s.bytes_fetched, s.cached_bytes
+    );
+    Ok(())
+}

crates/clawhdf5-remote/src/error.rs

@@ -0,0 +1,297 @@
+//! Errors of the remote backends.
+
+use clawhdf5_format::error::FormatError;
+
+/// `url` as it may be shown in an error, a `Debug` output or a log: no
+/// userinfo (`user:password@`), no fragment, and the query string's values
+/// replaced by `REDACTED` (a presigned S3/GCS/Azure URL carries its
+/// signature or token there). Keys are kept when they look like plain
+/// names, so a message still says which kind of URL it was.
+///
+/// ```
+/// assert_eq!(
+///     clawhdf5_remote::redact_url("https://me:pw@host/f.h5?X-Amz-Signature=abc&a=1#x"),
+///     "https://host/f.h5?X-Amz-Signature=REDACTED&a=REDACTED"
+/// );
+/// ```
+pub fn redact_url(url: &str) -> String {
+    let (scheme, rest) = match url.split_once("://") {
+        Some((s, r)) => (Some(s), r),
+        None => (None, url),
+    };
+    let rest = rest.split('#').next().unwrap_or("");
+    let (before_query, query) = match rest.split_once('?') {
+        Some((a, q)) => (a, Some(q)),
+        None => (rest, None),
+    };
+    let mut out = String::with_capacity(url.len());
+    if let Some(s) = scheme {
+        out.push_str(s);
+        out.push_str("://");
+        let auth_end = before_query.find('/').unwrap_or(before_query.len());
+        let (authority, path) = before_query.split_at(auth_end);
+        out.push_str(
+            authority
+                .rsplit_once('@')
+                .map_or(authority, |(_, host)| host),
+        );
+        out.push_str(path);
+    } else {
+        out.push_str(before_query);
+    }
+    if let Some(q) = query {
+        out.push('?');
+        let plain = |k: &str| {
+            !k.is_empty()
+                && k.len() <= 64
+                && k.bytes()
+                    .all(|b| b.is_ascii_alphanumeric() || matches!(b, b'-' | b'_' | b'.'))
+        };
+        let parts: Vec<String> = q
+            .split('&')
+            .map(|kv| {
+                let k = kv.split('=').next().unwrap_or("");
+                if plain(k) {
+                    format!("{k}=REDACTED")
+                } else {
+                    "REDACTED".to_string()
+                }
+            })
+            .collect();
+        out.push_str(&parts.join("&"));
+    }
+    out
+}
+
+/// Replaces the secret parts of one URL (its userinfo and query string,
+/// and the URL itself) wherever they appear in a message — such as the
+/// text of an error from the HTTP client.
+#[cfg_attr(
+    not(any(feature = "http", feature = "s3", feature = "gcs", feature = "azure")),
+    allow(dead_code)
+)]
+#[derive(Debug, Clone)]
+pub(crate) struct Redactor {
+    shown: String,
+    secrets: Vec<(String, String)>,
+}
+
+#[cfg_attr(
+    not(any(feature = "http", feature = "s3", feature = "gcs", feature = "azure")),
+    allow(dead_code)
+)]
+impl Redactor {
+    pub(crate) fn new(url: &str) -> Redactor {
+        let shown = redact_url(url);
+        let mut secrets = vec![(url.to_string(), shown.clone())];
+        let rest = url.split_once("://").map_or(url, |(_, r)| r);
+        let authority = rest.split(['/', '?', '#']).next().unwrap_or("");
+        if let Some((userinfo, _)) = authority.rsplit_once('@')
+            && !userinfo.is_empty()
+        {
+            secrets.push((format!("{userinfo}@"), String::new()));
+            secrets.push((userinfo.to_string(), "REDACTED".into()));
+        }
+        if let Some((_, q)) = rest.split('#').next().unwrap_or("").split_once('?')
+            && !q.is_empty()
+        {
+            let shown_q = shown.split_once('?').map_or("", |(_, q)| q).to_string();
+            secrets.push((q.to_string(), shown_q));
+            for kv in q.split('&') {
+                if let Some((_, v)) = kv.split_once('=')
+                    && v.len() >= 4
+                {
+                    secrets.push((v.to_string(), "REDACTED".into()));
+                }
+            }
+        }
+        Redactor { shown, secrets }
+    }
+
+    /// The URL, redacted.
+    pub(crate) fn shown(&self) -> &str {
+        &self.shown
+    }
+
+    /// `msg` with every secret part of the URL replaced.
+    pub(crate) fn scrub(&self, msg: &str) -> String {
+        let mut m = msg.to_string();
+        for (secret, with) in &self.secrets {
+            if m.contains(secret.as_str()) {
+                m = m.replace(secret.as_str(), with);
+            }
+        }
+        m
+    }
+}
+
+/// Why a remote file could not be opened or read.
+///
+/// No message carries a URL's credentials: URLs appear as
+/// [`redact_url`] shows them.
+///
+/// Inside a [`clawhdf5::File`] read these arrive as
+/// `clawhdf5::Error::Format(FormatError::Storage(message))`, the message
+/// being this error's `Display`.
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub enum RemoteError {
+    /// The URL is malformed.
+    InvalidUrl(String),
+    /// The URL's scheme is not supported by this build (for example
+    /// `s3://` without the `s3` feature, or `https://` without `https`).
+    UnsupportedScheme(String),
+    /// The server answered a range request with the whole file (status
+    /// 200), i.e. it does not support ranges, and a full download was not
+    /// allowed ([`HttpOptions::allow_full_download`](crate::HttpOptions)).
+    RangeNotSupported(String),
+    /// The file changed since it was opened (a different ETag,
+    /// Last-Modified or length, or a failed `If-Match` precondition).
+    /// Nothing read after the change is returned.
+    FileChanged(String),
+    /// The server answered with an unexpected status.
+    Status {
+        /// The HTTP status code.
+        code: u16,
+        /// What was being requested.
+        what: String,
+    },
+    /// A response did not carry what was asked for (a wrong
+    /// `Content-Range`, a body shorter or longer than announced), after
+    /// every retry.
+    BadResponse(String),
+    /// A network failure (connection, timeout, reset) after every retry.
+    Transport(String),
+    /// An error from the object store.
+    ObjectStore(String),
+    /// Called in a way the backend cannot serve.
+    Usage(String),
+    /// A redirect that is not followed: from `https` to `http`, to
+    /// another scheme, or beyond
+    /// [`HttpOptions::max_redirects`](crate::HttpOptions).
+    Redirect(String),
+    /// The file is larger than a download was allowed to be
+    /// ([`download`](crate::download)).
+    TooLarge {
+        /// The file's length, as the server reports it.
+        len: u64,
+        /// The limit.
+        limit: u64,
+    },
+    /// A read through a backend failed; its error, as text (the form a
+    /// [`clawhdf5::File`] read reports it in).
+    Backend(String),
+}
+
+impl RemoteError {
+    /// The error with every secret part of `r`'s URL scrubbed from its text.
+    #[cfg_attr(
+        not(any(feature = "http", feature = "s3", feature = "gcs", feature = "azure")),
+        allow(dead_code)
+    )]
+    pub(crate) fn scrubbed(self, r: &Redactor) -> RemoteError {
+        let f = |s: String| r.scrub(&s);
+        match self {
+            RemoteError::InvalidUrl(s) => RemoteError::InvalidUrl(f(s)),
+            RemoteError::UnsupportedScheme(s) => RemoteError::UnsupportedScheme(f(s)),
+            RemoteError::RangeNotSupported(s) => RemoteError::RangeNotSupported(f(s)),
+            RemoteError::FileChanged(s) => RemoteError::FileChanged(f(s)),
+            RemoteError::Status { code, what } => RemoteError::Status {
+                code,
+                what: f(what),
+            },
+            RemoteError::BadResponse(s) => RemoteError::BadResponse(f(s)),
+            RemoteError::Transport(s) => RemoteError::Transport(f(s)),
+            RemoteError::ObjectStore(s) => RemoteError::ObjectStore(f(s)),
+            RemoteError::Usage(s) => RemoteError::Usage(f(s)),
+            RemoteError::Redirect(s) => RemoteError::Redirect(f(s)),
+            e @ RemoteError::TooLarge { .. } => e,
+            RemoteError::Backend(s) => RemoteError::Backend(f(s)),
+        }
+    }
+
+    /// Whether retrying the same request may succeed.
+    #[cfg_attr(not(feature = "http"), allow(dead_code))]
+    pub(crate) fn is_transient(&self) -> bool {
+        match self {
+            RemoteError::Transport(_) | RemoteError::BadResponse(_) => true,
+            RemoteError::Status { code, .. } => {
+                matches!(code, 408 | 429 | 500 | 502 | 503 | 504)
+            }
+            _ => false,
+        }
+    }
+}
+
+impl std::fmt::Display for RemoteError {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            RemoteError::InvalidUrl(s) => write!(f, "invalid URL: {s}"),
+            RemoteError::UnsupportedScheme(s) => write!(f, "unsupported URL: {s}"),
+            RemoteError::RangeNotSupported(s) => {
+                write!(f, "the server does not support range requests: {s}")
+            }
+            RemoteError::FileChanged(s) => write!(f, "the remote file changed while open: {s}"),
+            RemoteError::Status { code, what } => write!(f, "HTTP status {code} for {what}"),
+            RemoteError::BadResponse(s) => write!(f, "bad response: {s}"),
+            RemoteError::Transport(s) => write!(f, "network error: {s}"),
+            RemoteError::ObjectStore(s) => write!(f, "object store: {s}"),
+            RemoteError::Usage(s) => write!(f, "{s}"),
+            RemoteError::Redirect(s) => write!(f, "redirect refused: {s}"),
+            RemoteError::TooLarge { len, limit } => write!(
+                f,
+                "the remote file is {len} bytes, more than the download limit of {limit} bytes"
+            ),
+            RemoteError::Backend(s) => write!(f, "{s}"),
+        }
+    }
+}
+
+impl std::error::Error for RemoteError {}
+
+impl From<RemoteError> for FormatError {
+    fn from(e: RemoteError) -> Self {
+        FormatError::Storage(e.to_string())
+    }
+}
+
+/// An error of [`open_url`](crate::open_url): the remote side, or the file
+/// itself.
+#[derive(Debug)]
+#[non_exhaustive]
+pub enum Error {
+    /// Reaching or reading the remote file failed.
+    Remote(RemoteError),
+    /// The bytes were read, but they are not an HDF5 file clawhdf5 can open.
+    Hdf5(clawhdf5::Error),
+}
+
+impl std::fmt::Display for Error {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Error::Remote(e) => e.fmt(f),
+            Error::Hdf5(e) => e.fmt(f),
+        }
+    }
+}
+
+impl std::error::Error for Error {
+    fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
+        match self {
+            Error::Remote(e) => Some(e),
+            Error::Hdf5(e) => Some(e),
+        }
+    }
+}
+
+impl From<RemoteError> for Error {
+    fn from(e: RemoteError) -> Self {
+        Error::Remote(e)
+    }
+}
+
+impl From<clawhdf5::Error> for Error {
+    fn from(e: clawhdf5::Error) -> Self {
+        Error::Hdf5(e)
+    }
+}

crates/clawhdf5-remote/src/http.rs

@@ -0,0 +1,794 @@
+//! HTTP(S) range requests: [`HttpStorage`].
+//!
+//! Every read is a `GET` with a `Range: bytes=a-b` header, answered `206
+//! Partial Content`. The file is pinned when it is opened:
+//!
+//! - its length comes from the `Content-Range` of the first request (which
+//!   also fetches the first block, so opening costs one request);
+//! - a strong `ETag` is sent back as `If-Match` on every later request, and
+//!   compared with the `ETag` of every response; without one, `Last-Modified`
+//!   is sent as `If-Unmodified-Since` and compared; the length in every
+//!   `Content-Range` must stay the same. A file that changes while it is
+//!   open is [`RemoteError::FileChanged`], never a mix of old and new bytes.
+//!   (A server that sends neither validator cannot be checked beyond the
+//!   length; [`HttpOptions::require_validator`] refuses such servers.)
+//! - a `200` answer to the first request whose body is no longer than the
+//!   range asked for is the whole file (a server may answer so when the
+//!   range covers it): it is kept and read from memory.
+//! - a server that ignores `Range` and answers `200` with the whole file is
+//!   refused with [`RemoteError::RangeNotSupported`], unless
+//!   [`HttpOptions::allow_full_download`] is set: then the file is
+//!   downloaded once, at open, and read from memory.
+//!
+//! Redirects are followed (at most [`HttpOptions::max_redirects`]), but
+//! never from `https` to `http`, and the custom
+//! [`HttpOptions::headers`] are not sent to another origin.
+//!
+//! Timeouts scale with the request: [`HttpOptions::timeout`] to connect
+//! and to get the response headers, and for the body that plus its size at
+//! [`HttpOptions::min_speed`] — a slow link is not cut off mid-block, a
+//! stalled connection still is.
+//!
+//! Transient failures — connection errors, timeouts, `408`/`429`/`5xx`, and
+//! a body shorter or longer than its `Content-Range` — are retried with
+//! exponential backoff. Responses are requested with
+//! `Accept-Encoding: identity`, since a compressed body cannot be a byte
+//! range of the file.
+//!
+//! `HttpStorage` itself does not cache: each `read_at` is one request. Read
+//! it through [`BlockCache`](crate::BlockCache) (which [`open_url`](crate::open_url)
+//! does); its `read_ranges` fetches the ranges of one call in parallel.
+
+use std::borrow::Cow;
+use std::io::Read;
+use std::ops::Range;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+use std::time::Duration;
+
+use clawhdf5_format::error::FormatError;
+use clawhdf5_format::storage::Storage;
+
+use crate::error::{Redactor, RemoteError, redact_url};
+
+/// Settings of an [`HttpStorage`].
+#[derive(Debug, Clone)]
+pub struct HttpOptions {
+    /// Retries of a request that failed transiently (so up to `retries + 1`
+    /// attempts).
+    pub retries: u32,
+    /// Delay before the first retry; doubled for each further one.
+    pub backoff: Duration,
+    /// Time allowed to connect, and then to receive the response headers.
+    /// The body gets this plus the time it takes at
+    /// [`min_speed`](Self::min_speed), so a request's budget grows with its
+    /// size: a slow but moving link is not cut off, a stalled one is.
+    pub timeout: Duration,
+    /// Slowest transfer rate tolerated, in bytes per second: receiving a
+    /// body of `n` bytes may take `timeout + n / min_speed` (16 KiB/s by
+    /// default: 94 s for a 1 MiB block, 9 min for an 8 MiB request).
+    pub min_speed: u64,
+    /// Requests of one `read_ranges` call in flight at once.
+    pub max_parallel: usize,
+    /// Bytes fetched by the first request, from offset 0 (the superblock and
+    /// usually the root group's metadata); at least 1.
+    pub first_request: u64,
+    /// When the server ignores `Range` (answers `200`), download the whole
+    /// file once and read it from memory, instead of failing.
+    pub allow_full_download: bool,
+    /// Largest file [`allow_full_download`](Self::allow_full_download) will
+    /// download.
+    pub max_full_download: u64,
+    /// Refuse a server that sends neither a strong `ETag` nor
+    /// `Last-Modified`, since a change of the file could then go unnoticed
+    /// (only its length is checked).
+    pub require_validator: bool,
+    /// Extra headers sent with every request to the URL's own origin (for
+    /// example `Authorization` or `X-Api-Key`). They are never sent to
+    /// another origin a redirect leads to.
+    pub headers: Vec<(String, String)>,
+    /// Redirects followed per request (0: none, a redirect is an error).
+    /// A redirect from `https` to plain `http` is always refused; once a
+    /// redirect leaves the URL's origin (scheme, host and port), the
+    /// [`headers`](Self::headers) are no longer sent. Every request of a
+    /// file follows the redirects again (the target is not remembered, as
+    /// a presigned target may expire).
+    pub max_redirects: u32,
+}
+
+impl Default for HttpOptions {
+    fn default() -> Self {
+        HttpOptions {
+            retries: 3,
+            backoff: Duration::from_millis(200),
+            timeout: Duration::from_secs(30),
+            min_speed: 16 << 10,
+            max_parallel: 8,
+            first_request: crate::cache::DEFAULT_BLOCK_SIZE,
+            allow_full_download: false,
+            max_full_download: 1 << 30,
+            require_validator: false,
+            headers: Vec::new(),
+            max_redirects: 5,
+        }
+    }
+}
+
+/// Requests and bytes an [`HttpStorage`] has used.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct HttpStats {
+    /// HTTP requests sent (retries included).
+    pub requests: u64,
+    /// Requests that were retries.
+    pub retries: u64,
+    /// Response body bytes received.
+    pub bytes: u64,
+}
+
+/// How the file is pinned.
+#[derive(Debug, Clone, PartialEq, Eq)]
+enum Validator {
+    ETag(String),
+    LastModified(String),
+    None,
+}
+
+/// An HTTP(S) file read by range requests. See the [module
+/// documentation](self).
+pub struct HttpStorage {
+    agent: ureq::Agent,
+    /// The URL as given, credentials and all: only ever sent to the server.
+    url: String,
+    /// Shows the URL without its credentials, in every message.
+    redactor: Redactor,
+    len: u64,
+    validator: Validator,
+    options: HttpOptions,
+    /// The whole file, when the server ignores ranges and a full download
+    /// was allowed.
+    full: Option<Vec<u8>>,
+    requests: AtomicU64,
+    retries: AtomicU64,
+    bytes: AtomicU64,
+}
+
+impl std::fmt::Debug for HttpStorage {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("HttpStorage")
+            .field("url", &self.redactor.shown())
+            .field("len", &self.len)
+            .field("validator", &self.validator)
+            .field("full_download", &self.full.is_some())
+            .finish()
+    }
+}
+
+/// A parsed `Content-Range: bytes a-b/total`.
+fn content_range(v: &str) -> Option<(u64, u64, Option<u64>)> {
+    let rest = v.trim().strip_prefix("bytes")?.trim_start();
+    let (span, total) = rest.split_once('/')?;
+    let (a, b) = span.trim().split_once('-')?;
+    let a: u64 = a.trim().parse().ok()?;
+    let b: u64 = b.trim().parse().ok()?;
+    if b < a {
+        return None;
+    }
+    let total = match total.trim() {
+        "*" => None,
+        t => Some(t.parse().ok()?),
+    };
+    Some((a, b, total))
+}
+
+fn header<'a>(resp: &'a ureq::http::Response<ureq::Body>, name: &str) -> Option<&'a str> {
+    resp.headers().get(name).and_then(|v| v.to_str().ok())
+}
+
+/// A body with a `Content-Encoding` is not a byte range of the file.
+fn check_identity(url: &str, resp: &ureq::http::Response<ureq::Body>) -> Result<(), RemoteError> {
+    match header(resp, "content-encoding") {
+        Some(enc) if !enc.trim().eq_ignore_ascii_case("identity") => {
+            Err(RemoteError::Usage(format!(
+                "{url}: the server sent a {enc}-encoded body despite Accept-Encoding: identity"
+            )))
+        }
+        _ => Ok(()),
+    }
+}
+
+/// Scheme, host (lowercase, no userinfo) and port of an absolute URL.
+fn origin(url: &str) -> Option<(String, String, u16)> {
+    let (scheme, rest) = url.split_once("://")?;
+    let scheme = scheme.to_ascii_lowercase();
+    let authority = rest.split(['/', '?', '#']).next().unwrap_or("");
+    let host_port = authority.rsplit_once('@').map_or(authority, |(_, h)| h);
+    let default = match scheme.as_str() {
+        "http" => 80,
+        "https" => 443,
+        _ => return None,
+    };
+    // "[v6]:port", "host:port", or either without a port.
+    let (host, port) = match host_port.rsplit_once(':') {
+        Some((h, p)) if !p.contains(']') => (h, p.parse().ok()?),
+        _ => (host_port, default),
+    };
+    Some((scheme, host.to_ascii_lowercase(), port))
+}
+
+fn same_origin(a: &str, b: &str) -> bool {
+    matches!((origin(a), origin(b)), (Some(x), Some(y)) if x == y)
+}
+
+/// The URL a redirect from `base` to `location` goes to, if it may be
+/// followed: `http`/`https` only, and never from `https` to `http`.
+fn redirect_target(base: &str, location: &str) -> Result<String, RemoteError> {
+    let location = location.trim();
+    let (scheme, rest) = base.split_once("://").unwrap_or(("http", base));
+    let authority_end = rest.find(['/', '?', '#']).unwrap_or(rest.len());
+    let (authority, path) = rest.split_at(authority_end);
+    let target = if location.contains("://") {
+        location.to_string()
+    } else if let Some(r) = location.strip_prefix("//") {
+        format!("{scheme}://{r}")
+    } else if location.starts_with('/') {
+        format!("{scheme}://{authority}{location}")
+    } else {
+        let path = path.split(['?', '#']).next().unwrap_or("");
+        let dir = path.rsplit_once('/').map_or("", |(d, _)| d);
+        format!("{scheme}://{authority}{dir}/{location}")
+    };
+    let Some((to_scheme, _, _)) = origin(&target) else {
+        return Err(RemoteError::Redirect(format!(
+            "{} redirects to {}, which is not an http(s) URL",
+            redact_url(base),
+            redact_url(&target)
+        )));
+    };
+    if scheme.eq_ignore_ascii_case("https") && to_scheme != "https" {
+        return Err(RemoteError::Redirect(format!(
+            "{} redirects to {}: a downgrade from https is refused",
+            redact_url(base),
+            redact_url(&target)
+        )));
+    }
+    Ok(target)
+}
+
+fn transport(e: ureq::Error) -> RemoteError {
+    match e {
+        ureq::Error::StatusCode(code) => RemoteError::Status {
+            code,
+            what: "request".into(),
+        },
+        ureq::Error::BadUri(s) => RemoteError::InvalidUrl(s),
+        other => RemoteError::Transport(other.to_string()),
+    }
+}
+
+impl HttpStorage {
+    /// Open `url` (`http://`, or `https://` with the `https` feature):
+    /// one ranged `GET` of the first [`HttpOptions::first_request`] bytes,
+    /// which gives the file's length and validators. Returns the storage
+    /// and the bytes that request fetched (the file's start), for a
+    /// [`BlockCache`](crate::BlockCache) to keep.
+    pub fn open(url: &str, options: HttpOptions) -> Result<(HttpStorage, Vec<u8>), RemoteError> {
+        let redactor = Redactor::new(url);
+        HttpStorage::open_inner(url, options, &redactor).map_err(|e| e.scrubbed(&redactor))
+    }
+
+    fn open_inner(
+        url: &str,
+        options: HttpOptions,
+        redactor: &Redactor,
+    ) -> Result<(HttpStorage, Vec<u8>), RemoteError> {
+        let shown = redactor.shown();
+        let lower = url.to_ascii_lowercase();
+        if lower.starts_with("https://") {
+            if !cfg!(feature = "https") {
+                return Err(RemoteError::UnsupportedScheme(format!(
+                    "{shown}: https:// needs the `https` feature of clawhdf5-remote"
+                )));
+            }
+        } else if !lower.starts_with("http://") {
+            return Err(RemoteError::UnsupportedScheme(shown.to_string()));
+        }
+        // Redirects are followed by `call`, which applies our rules.
+        let config = ureq::Agent::config_builder()
+            .http_status_as_error(false)
+            .max_redirects(0)
+            .timeout_connect(Some(options.timeout))
+            .timeout_recv_response(Some(options.timeout))
+            .build();
+        let mut storage = HttpStorage {
+            agent: ureq::Agent::new_with_config(config),
+            url: url.to_string(),
+            redactor: redactor.clone(),
+            len: 0,
+            validator: Validator::None,
+            options,
+            full: None,
+            requests: AtomicU64::new(0),
+            retries: AtomicU64::new(0),
+            bytes: AtomicU64::new(0),
+        };
+        let first = storage.with_retries(|| storage.probe())?;
+        let (len, validator, bytes, full) = first;
+        storage.len = len;
+        storage.validator = validator;
+        if full {
+            storage.full = Some(bytes);
+            return Ok((storage, Vec::new()));
+        }
+        if storage.options.require_validator && storage.validator == Validator::None {
+            return Err(RemoteError::Usage(format!(
+                "{shown}: the server sends neither a strong ETag nor Last-Modified, so a change \
+                 of the file could not be detected (HttpOptions::require_validator)"
+            )));
+        }
+        Ok((storage, bytes))
+    }
+
+    /// The URL as given — with any credentials it carries, so do not log
+    /// it; [`redact_url`](crate::redact_url) gives a form that can be.
+    pub fn url(&self) -> &str {
+        &self.url
+    }
+
+    /// Whether the file was downloaded whole, because the server does not
+    /// support ranges and [`HttpOptions::allow_full_download`] was set.
+    pub fn is_full_download(&self) -> bool {
+        self.full.is_some()
+    }
+
+    /// The `ETag` the file is pinned to, if the server sent a strong one.
+    pub fn etag(&self) -> Option<&str> {
+        match &self.validator {
+            Validator::ETag(e) => Some(e),
+            _ => None,
+        }
+    }
+
+    /// Requests and bytes so far.
+    pub fn stats(&self) -> HttpStats {
+        HttpStats {
+            requests: self.requests.load(Ordering::Relaxed),
+            retries: self.retries.load(Ordering::Relaxed),
+            bytes: self.bytes.load(Ordering::Relaxed),
+        }
+    }
+
+    fn with_retries<T>(
+        &self,
+        mut attempt: impl FnMut() -> Result<T, RemoteError>,
+    ) -> Result<T, RemoteError> {
+        let mut delay = self.options.backoff;
+        let mut n = 0;
+        loop {
+            match attempt() {
+                Ok(v) => return Ok(v),
+                Err(e) if e.is_transient() && n < self.options.retries => {
+                    n += 1;
+                    self.retries.fetch_add(1, Ordering::Relaxed);
+                    std::thread::sleep(delay);
+                    delay = delay.saturating_mul(2);
+                }
+                Err(e) => return Err(e),
+            }
+        }
+    }
+
+    /// One `GET` of `url`. The options' custom headers are sent only to
+    /// the URL's own origin (`trusted`).
+    fn request(
+        &self,
+        url: &str,
+        range: Option<(u64, u64)>,
+        trusted: bool,
+    ) -> ureq::RequestBuilder<ureq::typestate::WithoutBody> {
+        let mut req = self.agent.get(url).header("Accept-Encoding", "identity");
+        if let Some((a, b)) = range {
+            req = req.header("Range", format!("bytes={a}-{b}"));
+        }
+        // The body's budget scales with what it may carry: the range, or
+        // a whole file the server may send instead.
+        let mut body = range.map_or(0, |(a, b)| b.saturating_sub(a).saturating_add(1));
+        if self.options.allow_full_download {
+            body = body.max(self.options.max_full_download);
+        }
+        let secs = body as f64 / self.options.min_speed.max(1) as f64;
+        let body_timeout = self
+            .options
+            .timeout
+            .saturating_add(Duration::try_from_secs_f64(secs).unwrap_or(Duration::MAX));
+        let mut req = req.config().timeout_recv_body(Some(body_timeout)).build();
+        match &self.validator {
+            Validator::ETag(e) => req = req.header("If-Match", e),
+            Validator::LastModified(t) => req = req.header("If-Unmodified-Since", t),
+            Validator::None => {}
+        }
+        if trusted {
+            for (k, v) in &self.options.headers {
+                req = req.header(k, v);
+            }
+        }
+        req
+    }
+
+    /// Send a ranged `GET`, following redirects by the rules of
+    /// [`HttpOptions::max_redirects`]: at most that many, never from
+    /// `https` to anything else, and without the custom headers once the
+    /// chain has left the URL's origin. Each hop counts as a request.
+    fn call(
+        &self,
+        range: Option<(u64, u64)>,
+    ) -> Result<ureq::http::Response<ureq::Body>, RemoteError> {
+        let mut url = self.url.clone();
+        let mut trusted = true;
+        let mut hops = 0u32;
+        loop {
+            if hops > 0 {
+                self.requests.fetch_add(1, Ordering::Relaxed);
+            }
+            let resp = self
+                .request(&url, range, trusted)
+                .call()
+                .map_err(|e| transport(e).scrubbed(&Redactor::new(&url)))?;
+            if !matches!(resp.status().as_u16(), 301 | 302 | 303 | 307 | 308) {
+                return Ok(resp);
+            }
+            let status = resp.status().as_u16();
+            let Some(location) = header(&resp, "location") else {
+                return Err(RemoteError::BadResponse(format!(
+                    "{}: status {status} without a Location",
+                    redact_url(&url)
+                )));
+            };
+            let next = redirect_target(&url, location)?;
+            if hops >= self.options.max_redirects {
+                return Err(RemoteError::Redirect(format!(
+                    "{} redirects to {}: more than HttpOptions::max_redirects ({})",
+                    redact_url(&url),
+                    redact_url(&next),
+                    self.options.max_redirects
+                )));
+            }
+            if !same_origin(&next, &self.url) {
+                trusted = false;
+            }
+            url = next;
+            hops += 1;
+        }
+    }
+
+    /// Read a body of exactly `want` bytes (or up to `limit` when `want` is
+    /// unknown).
+    fn body(
+        &self,
+        resp: ureq::http::Response<ureq::Body>,
+        want: Option<u64>,
+        limit: u64,
+    ) -> Result<Vec<u8>, RemoteError> {
+        let cap = want.unwrap_or(limit);
+        let mut buf = Vec::with_capacity(usize::try_from(cap.min(64 << 20)).unwrap_or(0));
+        let reader = resp.into_body().into_reader();
+        let got = reader
+            .take(cap.saturating_add(1))
+            .read_to_end(&mut buf)
+            .map_err(|e| {
+                RemoteError::Transport(format!("{}: reading the body: {e}", self.redactor.shown()))
+            });
+        self.bytes.fetch_add(buf.len() as u64, Ordering::Relaxed);
+        got?;
+        match want {
+            Some(n) if buf.len() as u64 != n => Err(RemoteError::BadResponse(format!(
+                "{}: body of {} bytes, expected {n}",
+                self.redactor.shown(),
+                buf.len()
+            ))),
+            None if buf.len() as u64 > limit => Err(RemoteError::Usage(format!(
+                "{}: the file is larger than HttpOptions::max_full_download ({limit} bytes)",
+                self.redactor.shown()
+            ))),
+            _ => Ok(buf),
+        }
+    }
+
+    /// The first request: length, validators and the file's first bytes.
+    /// The last field is true when the server ignored the range and sent
+    /// the whole file (only kept when a full download is allowed).
+    fn probe(&self) -> Result<(u64, Validator, Vec<u8>, bool), RemoteError> {
+        let n = self.options.first_request.max(1);
+        self.requests.fetch_add(1, Ordering::Relaxed);
+        let resp = self.call(Some((0, n - 1)))?;
+        let status = resp.status().as_u16();
+        check_identity(self.redactor.shown(), &resp)?;
+        let validator = match (header(&resp, "etag"), header(&resp, "last-modified")) {
+            (Some(e), _) if !e.starts_with("W/") => Validator::ETag(e.to_string()),
+            (_, Some(t)) => Validator::LastModified(t.to_string()),
+            _ => Validator::None,
+        };
+        match status {
+            206 => {
+                let cr = header(&resp, "content-range").ok_or_else(|| {
+                    RemoteError::BadResponse(format!(
+                        "{}: 206 without Content-Range",
+                        self.redactor.shown()
+                    ))
+                })?;
+                let (a, b, total) = content_range(cr).ok_or_else(|| {
+                    RemoteError::BadResponse(format!(
+                        "{}: bad Content-Range {cr:?}",
+                        self.redactor.shown()
+                    ))
+                })?;
+                let total = total.ok_or_else(|| {
+                    RemoteError::BadResponse(format!(
+                        "{}: the server does not report the file's length (Content-Range {cr:?})",
+                        self.redactor.shown()
+                    ))
+                })?;
+                if a != 0 || b >= total || b > n - 1 {
+                    return Err(RemoteError::BadResponse(format!(
+                        "{}: asked for bytes 0-{}, got Content-Range {cr:?}",
+                        self.redactor.shown(),
+                        n - 1
+                    )));
+                }
+                let bytes = self.body(resp, Some(b - a + 1), 0)?;
+                Ok((total, validator, bytes, false))
+            }
+            200 => {
+                // RFC 9110 lets a server answer 200 when the range covers
+                // the whole file: a body no longer than the range asked
+                // for is the whole file, ranges supported or not.
+                let want: Option<u64> =
+                    header(&resp, "content-length").and_then(|v| v.trim().parse().ok());
+                let refused = || {
+                    RemoteError::RangeNotSupported(format!(
+                        "{} answered a range request with the whole file (status 200); set \
+                         HttpOptions::allow_full_download to download it",
+                        self.redactor.shown()
+                    ))
+                };
+                if !self.options.allow_full_download {
+                    return match want {
+                        Some(w) if w <= n => {
+                            let bytes = self.body(resp, Some(w), 0)?;
+                            Ok((bytes.len() as u64, validator, bytes, true))
+                        }
+                        Some(_) => Err(refused()),
+                        // No length: read at most the range asked for.
+                        None => match self.body(resp, None, n) {
+                            Ok(bytes) => Ok((bytes.len() as u64, validator, bytes, true)),
+                            Err(RemoteError::Usage(_)) => Err(refused()),
+                            Err(e) => Err(e),
+                        },
+                    };
+                }
+                if want.is_some_and(|w| w > self.options.max_full_download) {
+                    return Err(RemoteError::Usage(format!(
+                        "{}: the file is larger than HttpOptions::max_full_download ({} bytes)",
+                        self.redactor.shown(),
+                        self.options.max_full_download
+                    )));
+                }
+                let bytes = self.body(resp, want, self.options.max_full_download)?;
+                Ok((bytes.len() as u64, validator, bytes, true))
+            }
+            416 => Err(RemoteError::Usage(format!(
+                "{}: status 416 for the first bytes (an empty file?)",
+                self.redactor.shown()
+            ))),
+            code => Err(RemoteError::Status {
+                code,
+                what: self.redactor.shown().to_string(),
+            }),
+        }
+    }
+
+    /// One request for `[start, end)` (inside the file, non-empty).
+    fn fetch_once(&self, start: u64, end: u64) -> Result<Vec<u8>, RemoteError> {
+        self.requests.fetch_add(1, Ordering::Relaxed);
+        let resp = self.call(Some((start, end - 1)))?;
+        let changed =
+            |why: String| RemoteError::FileChanged(format!("{}: {why}", self.redactor.shown()));
+        check_identity(self.redactor.shown(), &resp)?;
+        match resp.status().as_u16() {
+            206 => {}
+            200 => {
+                return Err(RemoteError::RangeNotSupported(format!(
+                    "{} answered a range request with the whole file (status 200)",
+                    self.redactor.shown()
+                )));
+            }
+            412 => {
+                return Err(changed(
+                    "If-Match/If-Unmodified-Since failed (status 412)".into(),
+                ));
+            }
+            416 => return Err(changed("range no longer satisfiable (status 416)".into())),
+            code => {
+                return Err(RemoteError::Status {
+                    code,
+                    what: format!("{} bytes {start}-{}", self.redactor.shown(), end - 1),
+                });
+            }
+        }
+        match &self.validator {
+            Validator::ETag(e) => {
+                if let Some(got) = header(&resp, "etag")
+                    && got != e
+                {
+                    return Err(changed(format!("ETag {got} instead of {e}")));
+                }
+            }
+            Validator::LastModified(t) => {
+                if let Some(got) = header(&resp, "last-modified")
+                    && got != t
+                {
+                    return Err(changed(format!("Last-Modified {got} instead of {t}")));
+                }
+            }
+            Validator::None => {}
+        }
+        let cr = header(&resp, "content-range").ok_or_else(|| {
+            RemoteError::BadResponse(format!(
+                "{}: 206 without Content-Range",
+                self.redactor.shown()
+            ))
+        })?;
+        let (a, b, total) = content_range(cr).ok_or_else(|| {
+            RemoteError::BadResponse(format!(
+                "{}: bad Content-Range {cr:?}",
+                self.redactor.shown()
+            ))
+        })?;
+        if let Some(total) = total
+            && total != self.len
+        {
+            return Err(changed(format!("length {total} instead of {}", self.len)));
+        }
+        if a != start || b != end - 1 {
+            return Err(RemoteError::BadResponse(format!(
+                "{}: asked for bytes {start}-{}, got Content-Range {cr:?}",
+                self.redactor.shown(),
+                end - 1
+            )));
+        }
+        self.body(resp, Some(end - start), 0)
+    }
+
+    fn fetch(&self, start: u64, end: u64) -> Result<Vec<u8>, RemoteError> {
+        self.with_retries(|| self.fetch_once(start, end))
+            .map_err(|e| e.scrubbed(&self.redactor))
+    }
+
+    /// `[offset, offset + len)` clamped to the file, or `None` if empty.
+    fn clamp(&self, offset: u64, len: u64) -> Option<(u64, u64)> {
+        let end = offset.saturating_add(len).min(self.len);
+        (offset < end).then_some((offset, end))
+    }
+}
+
+impl Storage for HttpStorage {
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        if let Some(all) = &self.full {
+            return all.as_slice().read_at(offset, len);
+        }
+        match self.clamp(offset, len as u64) {
+            None => Ok(Cow::Owned(Vec::new())),
+            Some((a, b)) => Ok(Cow::Owned(self.fetch(a, b)?)),
+        }
+    }
+
+    fn len(&self) -> u64 {
+        self.len
+    }
+
+    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
+        if let Some(all) = &self.full {
+            return all.as_slice().read_ranges(ranges);
+        }
+        let parallel = self.options.max_parallel.clamp(1, ranges.len().max(1));
+        if parallel <= 1 {
+            return ranges
+                .iter()
+                .map(|r| self.read_at(r.start, (r.end.saturating_sub(r.start)) as usize))
+                .collect();
+        }
+        let next = AtomicUsize::new(0);
+        let failed = std::sync::atomic::AtomicBool::new(false);
+        type Slot = Option<Result<Vec<u8>, RemoteError>>;
+        let results: std::sync::Mutex<Vec<Slot>> =
+            std::sync::Mutex::new((0..ranges.len()).map(|_| None).collect());
+        std::thread::scope(|s| {
+            for _ in 0..parallel {
+                s.spawn(|| {
+                    loop {
+                        let i = next.fetch_add(1, Ordering::Relaxed);
+                        if i >= ranges.len() || failed.load(Ordering::Relaxed) {
+                            break;
+                        }
+                        let r = &ranges[i];
+                        let got = match self.clamp(r.start, r.end.saturating_sub(r.start)) {
+                            None => Ok(Vec::new()),
+                            Some((a, b)) => self.fetch(a, b),
+                        };
+                        if got.is_err() {
+                            failed.store(true, Ordering::Relaxed);
+                        }
+                        results
+                            .lock()
+                            .unwrap_or_else(std::sync::PoisonError::into_inner)[i] = Some(got);
+                    }
+                });
+            }
+        });
+        let results = results
+            .into_inner()
+            .unwrap_or_else(std::sync::PoisonError::into_inner);
+        let mut out = Vec::with_capacity(ranges.len());
+        for r in results {
+            match r {
+                Some(Ok(v)) => out.push(Cow::Owned(v)),
+                Some(Err(e)) => return Err(e.into()),
+                // Not fetched because another range failed first.
+                None => continue,
+            }
+        }
+        if out.len() != ranges.len() {
+            return Err(FormatError::Storage(format!(
+                "{}: a parallel range read failed",
+                self.redactor.shown()
+            )));
+        }
+        Ok(out)
+    }
+
+    fn as_contiguous(&self) -> Option<&[u8]> {
+        self.full.as_deref()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::content_range;
+
+    #[test]
+    fn content_range_parses() {
+        assert_eq!(content_range("bytes 0-99/1000"), Some((0, 99, Some(1000))));
+        assert_eq!(content_range("bytes 5-5/*"), Some((5, 5, None)));
+        assert_eq!(content_range("bytes 9-5/10"), None);
+        assert_eq!(content_range("items 0-1/2"), None);
+        assert_eq!(content_range("bytes */1000"), None);
+    }
+
+    #[test]
+    fn redirect_targets_and_origins() {
+        use super::{redirect_target, same_origin};
+        let t = |b: &str, l: &str| redirect_target(b, l).map_err(|e| e.to_string());
+        assert_eq!(
+            t("http://a:8/d/f.h5?x=1", "g.h5").unwrap(),
+            "http://a:8/d/g.h5"
+        );
+        assert_eq!(t("http://a/d/f.h5", "/g.h5").unwrap(), "http://a/g.h5");
+        assert_eq!(t("https://a/d/f.h5", "//b/g.h5").unwrap(), "https://b/g.h5");
+        assert_eq!(t("http://a/f", "https://b/g").unwrap(), "https://b/g");
+        let e = t("https://a/f.h5", "http://a/f.h5").unwrap_err();
+        assert!(e.contains("downgrade"), "{e}");
+        assert!(t("HTTPS://a/f.h5", "http://b/f.h5").is_err());
+        assert!(t("https://a/f", "//b/g").is_ok());
+        assert!(t("http://a/f", "ftp://b/g").is_err());
+        let e = t("https://u:pw@a/f?sig=SECRET", "http://b/g?sig=OTHER").unwrap_err();
+        assert!(
+            !e.contains("SECRET") && !e.contains("OTHER") && !e.contains("pw"),
+            "{e}"
+        );
+        assert!(same_origin("http://a/x", "http://A:80/y"));
+        assert!(same_origin("https://u:p@a:443/x", "https://a/y"));
+        assert!(!same_origin("http://a/x", "https://a/x"));
+        assert!(!same_origin("http://a:1/x", "http://a:2/x"));
+        assert!(!same_origin("http://a/x", "http://b/x"));
+        assert!(same_origin("http://[::1]:8/x", "http://[::1]:8/y"));
+        assert!(!same_origin("http://[::1]/x", "http://[::1]:8/y"));
+    }
+}

crates/clawhdf5-remote/src/lib.rs

@@ -0,0 +1,228 @@
+//! Read HDF5 files where they are — on an HTTP(S) server or in an object
+//! store — without downloading them first.
+//!
+//! This is milestone M3 of `docs/design/range-reads.md`: remote backends for
+//! [`clawhdf5::File::open_storage`], each read through a mandatory
+//! [`BlockCache`].
+//!
+//! ```no_run
+//! let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/data.h5")?;
+//! let temperature = file.dataset("/grid/temperature")?.read_f64()?;
+//! # Ok::<(), Box<dyn std::error::Error>>(())
+//! ```
+//!
+//! - [`open_url`] / [`open_url_with`]: `http://` (default feature `http`),
+//!   `https://` (feature `https`), `s3://`, `gs://`, `az://` (features `s3`,
+//!   `gcs`, `azure`) → a [`clawhdf5::File`] with the whole read API.
+//! - [`storage_for_url`] gives the cached storage itself, to open with
+//!   [`clawhdf5::File::open_storage`] and to read its [`CacheStats`].
+//! - [`download`] reads a whole remote file into memory, up to a limit.
+//! - [`HttpStorage`] (range `GET`s, pinned by ETag/Last-Modified, retried
+//!   with backoff), [`ObjectStoreStorage`] (any `object_store` store,
+//!   feature `object-store`), and [`BlockCache`] over any
+//!   [`Storage`](clawhdf5_format::storage::Storage).
+//!
+//! What costs what: opening costs one request (it also fetches the first
+//! block, 1 MiB by default); listing a file whose metadata sits in its
+//! first blocks costs nothing more; reading a chunked dataset costs one
+//! parallel batch of requests for the blocks holding its chunk index, then
+//! one for its chunks. The zero-copy methods of `clawhdf5`
+//! (`read_raw_ref`, `read_*_zerocopy`, `File::as_bytes`) need the file in
+//! memory and are errors (`as_bytes` a panic) on a remote file.
+
+#![warn(missing_docs)]
+
+pub mod cache;
+pub mod error;
+#[cfg(feature = "http")]
+pub mod http;
+#[cfg(feature = "object-store")]
+pub mod object;
+
+use std::sync::Arc;
+
+use clawhdf5::File;
+use clawhdf5_format::storage::Storage;
+
+pub use cache::{BlockCache, CacheConfig, CacheStats};
+pub use error::{Error, RemoteError, redact_url};
+#[cfg(feature = "http")]
+pub use http::{HttpOptions, HttpStats, HttpStorage};
+#[cfg(feature = "object-store")]
+pub use object::ObjectStoreStorage;
+#[cfg(feature = "object-store")]
+pub use object_store;
+
+/// A backend a [`BlockCache`] can read through.
+pub type Backend = Box<dyn Storage + Send + Sync>;
+
+/// The storage [`storage_for_url`] returns: a block cache over the URL's
+/// backend.
+pub type RemoteStorage = BlockCache<Backend>;
+
+/// Settings of [`open_url_with`] and [`storage_for_url`].
+#[derive(Debug, Clone, Default)]
+pub struct Options {
+    /// The block cache.
+    pub cache: CacheConfig,
+    /// HTTP(S) requests.
+    #[cfg(feature = "http")]
+    pub http: HttpOptions,
+}
+
+/// Open the HDF5 file at `url` with default [`Options`].
+///
+/// `http://…` needs the (default) `http` feature, `https://…` the `https`
+/// feature, `s3://bucket/key`, `gs://bucket/key` and `az://container/key`
+/// the `s3`, `gcs` and `azure` features (credentials and region from the
+/// environment, as `object_store`'s `from_env` builders read them).
+pub fn open_url(url: &str) -> Result<File, Error> {
+    open_url_with(url, &Options::default())
+}
+
+/// [`open_url`] with explicit [`Options`].
+pub fn open_url_with(url: &str, options: &Options) -> Result<File, Error> {
+    let storage = storage_for_url(url, options)?;
+    Ok(File::open_storage(storage)?)
+}
+
+/// The cached storage for `url`, with the first block already fetched:
+/// open it with [`clawhdf5::File::open_storage`] (a clone of the `Arc`),
+/// and read its [`BlockCache::stats`] as you go.
+pub fn storage_for_url(url: &str, options: &Options) -> Result<Arc<RemoteStorage>, Error> {
+    let scheme = url
+        .split_once("://")
+        .map(|(s, _)| s.to_ascii_lowercase())
+        .ok_or_else(|| RemoteError::InvalidUrl(format!("{}: no scheme", redact_url(url))))?;
+    match scheme.as_str() {
+        "http" | "https" => http_storage(url, options),
+        "s3" | "s3a" | "gs" | "az" | "azure" | "abfs" | "abfss" | "adl" => {
+            cloud_storage(url, &scheme, options)
+        }
+        _ => Err(RemoteError::UnsupportedScheme(redact_url(url)).into()),
+    }
+}
+
+#[cfg(feature = "http")]
+fn http_storage(url: &str, options: &Options) -> Result<Arc<RemoteStorage>, Error> {
+    let mut http = options.http.clone();
+    http.first_request = http.first_request.max(options.cache.block_size.max(1));
+    let (storage, first) = HttpStorage::open(url, http)?;
+    let cache = BlockCache::new(Box::new(storage) as Backend, options.cache.clone());
+    cache.insert(0, &first);
+    Ok(Arc::new(cache))
+}
+
+#[cfg(not(feature = "http"))]
+fn http_storage(url: &str, _options: &Options) -> Result<Arc<RemoteStorage>, Error> {
+    Err(RemoteError::UnsupportedScheme(format!(
+        "{}: http(s):// needs the `http` feature of clawhdf5-remote",
+        redact_url(url)
+    ))
+    .into())
+}
+
+#[cfg(any(feature = "s3", feature = "gcs", feature = "azure"))]
+fn cloud_storage(url: &str, _scheme: &str, options: &Options) -> Result<Arc<RemoteStorage>, Error> {
+    let (store, path) = object::store_for_url(url)?;
+    let storage = ObjectStoreStorage::new(store, path)?;
+    Ok(Arc::new(object_cached(storage, options)?))
+}
+
+#[cfg(not(any(feature = "s3", feature = "gcs", feature = "azure")))]
+fn cloud_storage(url: &str, scheme: &str, _options: &Options) -> Result<Arc<RemoteStorage>, Error> {
+    let feature = match scheme {
+        "s3" | "s3a" => "s3",
+        "gs" => "gcs",
+        _ => "azure",
+    };
+    Err(RemoteError::UnsupportedScheme(format!(
+        "{}: {scheme}:// needs the `{feature}` feature of clawhdf5-remote",
+        redact_url(url)
+    ))
+    .into())
+}
+
+/// A [`BlockCache`] over `backend` with its first block fetched (readahead
+/// of the superblock and the metadata usually written next to it). A
+/// failure of that fetch is [`Error::Remote`] ([`RemoteError::Backend`],
+/// with the backend's message).
+pub fn cached(backend: Backend, options: &Options) -> Result<RemoteStorage, Error> {
+    let cache = BlockCache::new(backend, options.cache.clone());
+    let first = cache.config().block_size;
+    cache
+        .prefetch(0, first)
+        .map_err(|e| RemoteError::Backend(e.to_string()))?;
+    Ok(cache)
+}
+
+/// [`cached`] for an [`ObjectStoreStorage`]: its first block is fetched
+/// directly, so a failure keeps its kind (`FileChanged`, `ObjectStore`).
+#[cfg(feature = "object-store")]
+fn object_cached(storage: ObjectStoreStorage, options: &Options) -> Result<RemoteStorage, Error> {
+    // The block size BlockCache::new will use.
+    let block = options.cache.block_size.max(512);
+    let first = storage.fetch_first(block)?;
+    let cache = BlockCache::new(Box::new(storage) as Backend, options.cache.clone());
+    cache.insert(0, &first);
+    Ok(cache)
+}
+
+/// Default limit of [`download`]: 1 GiB.
+pub const DEFAULT_MAX_DOWNLOAD: u64 = 1 << 30;
+
+/// The whole file behind `storage`, read into memory — at most
+/// `max_bytes` of it (for example [`DEFAULT_MAX_DOWNLOAD`]).
+///
+/// The length is only what the server claims, so it is never used to
+/// allocate: a file longer than `max_bytes` is refused with
+/// [`RemoteError::TooLarge`] before anything is read, and the buffer grows
+/// only as bytes arrive (64 MiB per step, fetched as parallel requests by a
+/// [`BlockCache`]). A read that comes back short is an error.
+pub fn download(storage: &dyn Storage, max_bytes: u64) -> Result<Vec<u8>, Error> {
+    let len = storage.len();
+    if len > max_bytes {
+        return Err(RemoteError::TooLarge {
+            len,
+            limit: max_bytes,
+        }
+        .into());
+    }
+    const STEP: u64 = 64 << 20;
+    let mut out = Vec::new();
+    let mut pos = 0u64;
+    while pos < len {
+        let want = (len - pos).min(STEP);
+        let got = storage
+            .read_at(pos, want as usize)
+            .map_err(|e| RemoteError::Backend(e.to_string()))?;
+        if got.len() as u64 != want {
+            return Err(RemoteError::BadResponse(format!(
+                "{} bytes at offset {pos} instead of {want}",
+                got.len()
+            ))
+            .into());
+        }
+        out.extend_from_slice(&got);
+        pos += want;
+    }
+    Ok(out)
+}
+
+/// Open the object at `path` of any `object_store` store (in memory, local
+/// files, or a cloud store you configured) through a block cache.
+#[cfg(feature = "object-store")]
+pub fn open_object(
+    store: Arc<dyn object_store::ObjectStore>,
+    path: &str,
+    options: &Options,
+) -> Result<(File, Arc<RemoteStorage>), Error> {
+    let path = object_store::path::Path::parse(path)
+        .map_err(|e| RemoteError::InvalidUrl(format!("{path}: {e}")))?;
+    let storage = Arc::new(object_cached(
+        ObjectStoreStorage::new(store, path)?,
+        options,
+    )?);
+    let file = File::open_storage(storage.clone())?;
+    Ok((file, storage))
+}

crates/clawhdf5-remote/src/object.rs

@@ -0,0 +1,297 @@
+//! Object stores (S3, GCS, Azure, local files, memory) through the
+//! [`object_store`] crate: [`ObjectStoreStorage`].
+//!
+//! `object_store` is async and [`Storage`] is synchronous (parsing is CPU
+//! work; `docs/design/range-reads.md` §3 (a)). The storage owns a small
+//! multi-threaded tokio runtime (two worker threads): each read is spawned
+//! on it and the calling thread waits for the result, so it can be used
+//! from any thread — several at once, and from async code too. From async
+//! code prefer `tokio::task::spawn_blocking` (a read blocks the thread it
+//! is called on, which inside a runtime is one of its workers).
+//!
+//! The object is pinned when the storage is made: its size, and its ETag
+//! (sent as `If-Match` with every read, and compared with every response)
+//! or, without one, its version or modification time. A change while it is
+//! open is [`RemoteError::FileChanged`]. The ranges of one `read_ranges`
+//! call are fetched concurrently (at most 8 at a time).
+
+use std::borrow::Cow;
+use std::ops::Range;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, Ordering};
+
+use clawhdf5_format::error::FormatError;
+use clawhdf5_format::storage::Storage;
+use futures_util::{StreamExt, TryStreamExt};
+use object_store::path::Path;
+use object_store::{GetOptions, GetRange, ObjectMeta, ObjectStore, ObjectStoreExt};
+
+use crate::error::RemoteError;
+
+/// Concurrent requests of one `read_ranges` call.
+const MAX_CONCURRENT: usize = 8;
+
+/// One object of an [`ObjectStore`], read by ranged `get`s. See the
+/// [module documentation](self).
+pub struct ObjectStoreStorage {
+    store: Arc<dyn ObjectStore>,
+    path: Path,
+    meta: ObjectMeta,
+    runtime: Option<tokio::runtime::Runtime>,
+    requests: AtomicU64,
+    bytes: AtomicU64,
+}
+
+impl std::fmt::Debug for ObjectStoreStorage {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("ObjectStoreStorage")
+            .field("store", &self.store.to_string())
+            .field("path", &self.path)
+            .field("size", &self.meta.size)
+            .field("e_tag", &self.meta.e_tag)
+            .finish()
+    }
+}
+
+fn os_error(e: object_store::Error) -> RemoteError {
+    match e {
+        object_store::Error::Precondition { .. } | object_store::Error::NotModified { .. } => {
+            RemoteError::FileChanged(e.to_string())
+        }
+        other => RemoteError::ObjectStore(other.to_string()),
+    }
+}
+
+impl ObjectStoreStorage {
+    /// Open the object at `path` of `store`: one `head` request for its
+    /// size and validators.
+    pub fn new(store: Arc<dyn ObjectStore>, path: Path) -> Result<Self, RemoteError> {
+        let runtime = tokio::runtime::Builder::new_multi_thread()
+            .worker_threads(2)
+            .thread_name("clawhdf5-remote")
+            .enable_all()
+            .build()
+            .map_err(|e| RemoteError::Usage(format!("cannot start a tokio runtime: {e}")))?;
+        let mut s = ObjectStoreStorage {
+            store,
+            path,
+            meta: ObjectMeta {
+                location: Path::default(),
+                last_modified: Default::default(),
+                size: 0,
+                e_tag: None,
+                version: None,
+            },
+            runtime: Some(runtime),
+            requests: AtomicU64::new(1),
+            bytes: AtomicU64::new(0),
+        };
+        let (store, path) = (s.store.clone(), s.path.clone());
+        s.meta = s.block_on(async move { store.head(&path).await.map_err(os_error) })?;
+        Ok(s)
+    }
+
+    /// The object's metadata as pinned at open.
+    pub fn meta(&self) -> &ObjectMeta {
+        &self.meta
+    }
+
+    /// Requests made (the `head` included) and bytes received.
+    pub fn stats(&self) -> (u64, u64) {
+        (
+            self.requests.load(Ordering::Relaxed),
+            self.bytes.load(Ordering::Relaxed),
+        )
+    }
+
+    /// Run `fut` on the storage's own runtime and wait for it. The future
+    /// never runs on the caller's thread, so the caller's context does not
+    /// matter: a plain thread, `spawn_blocking`, or even inside another
+    /// runtime (whose thread is then blocked for the duration of the read,
+    /// as by any blocking call, but nothing deadlocks or panics).
+    fn block_on<T: Send + 'static>(
+        &self,
+        fut: impl std::future::Future<Output = Result<T, RemoteError>> + Send + 'static,
+    ) -> Result<T, RemoteError> {
+        let rt = self.runtime.as_ref().expect("runtime lives until drop");
+        let (tx, rx) = std::sync::mpsc::sync_channel(1);
+        rt.spawn(async move {
+            let _ = tx.send(fut.await);
+        });
+        rx.recv().map_err(|_| {
+            RemoteError::ObjectStore("the object store task ended without a result".into())
+        })?
+    }
+
+    fn options(&self, range: Range<u64>) -> GetOptions {
+        let mut o = GetOptions {
+            range: Some(GetRange::Bounded(range)),
+            ..GetOptions::default()
+        };
+        if let Some(e) = &self.meta.e_tag {
+            o.if_match = Some(e.clone());
+        } else if let Some(v) = &self.meta.version {
+            o.version = Some(v.clone());
+        } else {
+            o.if_unmodified_since = Some(self.meta.last_modified);
+        }
+        o
+    }
+
+    /// The object's first `n` bytes (fewer if it is shorter).
+    pub(crate) fn fetch_first(&self, n: u64) -> Result<Vec<u8>, RemoteError> {
+        Ok(self
+            .fetch_all(std::slice::from_ref(&(0..n)))?
+            .pop()
+            .unwrap_or_default())
+    }
+
+    fn fetch_all(&self, ranges: &[Range<u64>]) -> Result<Vec<Vec<u8>>, RemoteError> {
+        let len = self.meta.size;
+        let jobs: Vec<(usize, Range<u64>)> = ranges
+            .iter()
+            .enumerate()
+            .filter_map(|(i, r)| {
+                let end = r.end.min(len);
+                (r.start < end).then_some((i, r.start..end))
+            })
+            .collect();
+        let store = self.store.clone();
+        let path = self.path.clone();
+        let pinned = self.meta.e_tag.clone();
+        let reqs: Vec<(usize, Range<u64>, GetOptions)> = jobs
+            .into_iter()
+            .map(|(i, r)| {
+                let o = self.options(r.clone());
+                (i, r, o)
+            })
+            .collect();
+        self.requests
+            .fetch_add(reqs.len() as u64, Ordering::Relaxed);
+        let fetched: Vec<(usize, Vec<u8>)> = self.block_on(async move {
+            futures_util::stream::iter(reqs)
+                .map(|(i, r, o)| {
+                    let (store, path, pinned) = (store.clone(), path.clone(), pinned.clone());
+                    async move {
+                        let got = store.get_opts(&path, o).await.map_err(os_error)?;
+                        if let (Some(want), Some(have)) = (&pinned, &got.meta.e_tag)
+                            && want != have
+                        {
+                            return Err(RemoteError::FileChanged(format!(
+                                "{path}: ETag {have} instead of {want}"
+                            )));
+                        }
+                        if got.meta.size != len {
+                            return Err(RemoteError::FileChanged(format!(
+                                "{path}: size {} instead of {len}",
+                                got.meta.size
+                            )));
+                        }
+                        let bytes = got.bytes().await.map_err(os_error)?;
+                        if bytes.len() as u64 != r.end - r.start {
+                            return Err(RemoteError::BadResponse(format!(
+                                "{path}: {} bytes for range {r:?}",
+                                bytes.len()
+                            )));
+                        }
+                        Ok::<_, RemoteError>((i, bytes.to_vec()))
+                    }
+                })
+                .buffer_unordered(MAX_CONCURRENT)
+                .try_collect()
+                .await
+        })?;
+        let mut out = vec![Vec::new(); ranges.len()];
+        for (i, b) in fetched {
+            self.bytes.fetch_add(b.len() as u64, Ordering::Relaxed);
+            out[i] = b;
+        }
+        Ok(out)
+    }
+}
+
+impl Drop for ObjectStoreStorage {
+    fn drop(&mut self) {
+        // Dropping a runtime blocks, which panics inside an async context.
+        if let Some(rt) = self.runtime.take() {
+            rt.shutdown_background();
+        }
+    }
+}
+
+impl Storage for ObjectStoreStorage {
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        let range = offset..offset.saturating_add(len as u64);
+        let mut v = self.fetch_all(std::slice::from_ref(&range))?;
+        Ok(Cow::Owned(v.pop().unwrap_or_default()))
+    }
+
+    fn len(&self) -> u64 {
+        self.meta.size
+    }
+
+    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
+        if ranges.iter().any(|r| r.end < r.start) {
+            return Err(FormatError::Storage(
+                "read range ends before it starts".into(),
+            ));
+        }
+        Ok(self
+            .fetch_all(ranges)?
+            .into_iter()
+            .map(Cow::Owned)
+            .collect())
+    }
+}
+
+/// The store and object path for a cloud URL (`s3://bucket/key`,
+/// `gs://bucket/key`, `az://container/key`, ...), configured from the
+/// environment as `object_store`'s `from_env` builders do (`AWS_*`,
+/// `GOOGLE_*`, `AZURE_*`).
+#[cfg(any(feature = "s3", feature = "gcs", feature = "azure"))]
+pub(crate) fn store_for_url(url: &str) -> Result<(Arc<dyn ObjectStore>, Path), RemoteError> {
+    let redactor = crate::error::Redactor::new(url);
+    let parsed = object_store::path::Path::parse(
+        url.split_once("://")
+            .and_then(|(_, rest)| rest.split_once('/'))
+            .map(|(_, key)| key)
+            .unwrap_or(""),
+    )
+    .map_err(|e| RemoteError::InvalidUrl(format!("{}: {e}", crate::redact_url(url))))?;
+    let scheme = url.split_once("://").map(|(s, _)| s.to_ascii_lowercase());
+    let store: Arc<dyn ObjectStore> = match scheme.as_deref() {
+        #[cfg(feature = "s3")]
+        Some("s3" | "s3a") => Arc::new(
+            object_store::aws::AmazonS3Builder::from_env()
+                .with_url(url)
+                .build()
+                .map_err(|e| os_error(e).scrubbed(&redactor))?,
+        ),
+        #[cfg(feature = "gcs")]
+        Some("gs") => Arc::new(
+            object_store::gcp::GoogleCloudStorageBuilder::from_env()
+                .with_url(url)
+                .build()
+                .map_err(|e| os_error(e).scrubbed(&redactor))?,
+        ),
+        #[cfg(feature = "azure")]
+        Some("az" | "azure" | "abfs" | "abfss" | "adl") => Arc::new(
+            object_store::azure::MicrosoftAzureBuilder::from_env()
+                .with_url(url)
+                .build()
+                .map_err(|e| os_error(e).scrubbed(&redactor))?,
+        ),
+        _ => return Err(RemoteError::UnsupportedScheme(crate::redact_url(url))),
+    };
+    Ok((store, parsed))
+}
+
+#[cfg(all(test, feature = "s3"))]
+mod tests {
+    #[test]
+    fn s3_urls_name_the_bucket_and_key() {
+        let (store, path) = super::store_for_url("s3://my-bucket/dir/file.h5").unwrap();
+        assert_eq!(path.as_ref(), "dir/file.h5");
+        assert!(store.to_string().contains("my-bucket"), "{store}");
+    }
+}

crates/clawhdf5-remote/tests/common/mod.rs

@@ -0,0 +1,318 @@
+//! Shared by the integration tests: the test server, a transcript of a
+//! file (tree, attributes, values) to compare two ways of reading it, and
+//! the test files.
+
+#![allow(dead_code)]
+
+pub mod server;
+
+use std::collections::{BTreeMap, HashSet, VecDeque};
+use std::fmt::Write as _;
+use std::path::{Path, PathBuf};
+use std::process::Command;
+use std::sync::Arc;
+
+use clawhdf5::{DType, File, Selection};
+use clawhdf5_format::error::FormatError;
+
+/// Objects visited per file.
+const MAX_OBJECTS: usize = 2000;
+/// Datasets with more bytes than this are not read (their metadata is).
+pub const MAX_DATA_BYTES: u64 = 64 << 20;
+
+/// A short, stable digest of a value's `Debug` form.
+fn digest<T: std::fmt::Debug>(v: &T) -> String {
+    let s = format!("{v:?}");
+    if s.len() <= 200 {
+        return s;
+    }
+    let mut h = 0xcbf2_9ce4_8422_2325u64;
+    for b in s.bytes() {
+        h = (h ^ u64::from(b)).wrapping_mul(0x100_0000_01b3);
+    }
+    format!("{}…[{} bytes, fnv {h:016x}]", &s[..80], s.len())
+}
+
+/// A data read's value, or `Err` (which chunk a damaged dataset reports
+/// can vary between two `File`s: the chunk cache lists in hash order).
+fn value<T: std::fmt::Debug, E>(r: &Result<T, E>) -> String {
+    match r {
+        Ok(v) => digest(v),
+        Err(_) => "Err".into(),
+    }
+}
+
+fn sorted<V: std::fmt::Debug>(m: std::collections::HashMap<String, V>) -> BTreeMap<String, V> {
+    m.into_iter().collect()
+}
+
+/// Everything a reader sees in `file`: every group's entries, every
+/// object's attributes, and every dataset's shape, types and values.
+pub fn transcript(file: &File) -> String {
+    let mut out = String::new();
+    let mut seen = HashSet::new();
+    let mut queue = VecDeque::from([(String::from("/"), file.superblock().root_group_address)]);
+    while let Some((path, addr)) = queue.pop_front() {
+        if seen.len() >= MAX_OBJECTS || !seen.insert(addr) {
+            continue;
+        }
+        let group = file.group_at(addr);
+        let entries = group.entries();
+        writeln!(out, "{path} @{addr} entries {}", digest(&entries)).unwrap();
+        if let Ok(ds) = file.dataset_at(addr) {
+            dataset(&mut out, &path, &ds);
+        }
+        let attrs = group.attrs_with_errors().map(|(a, e)| (sorted(a), e));
+        writeln!(out, "{path} attrs {}", digest(&attrs)).unwrap();
+        if let Ok(entries) = entries {
+            for (name, child) in entries {
+                queue.push_back((format!("{}/{name}", path.trim_end_matches('/')), child));
+            }
+        }
+    }
+    out
+}
+
+fn dataset(out: &mut String, path: &str, ds: &clawhdf5::Dataset<'_>) {
+    let shape = ds.shape();
+    let dtype = ds.dtype();
+    writeln!(
+        out,
+        "{path} shape {} dtype {} raw {}",
+        digest(&shape),
+        digest(&dtype),
+        digest(&ds.raw_datatype())
+    )
+    .unwrap();
+    let (Ok(shape), Ok(dtype), Ok(raw_dt)) = (shape, dtype, ds.raw_datatype()) else {
+        return;
+    };
+    let elements = shape.iter().try_fold(1u64, |a, &d| a.checked_mul(d));
+    let bytes = elements.and_then(|n| n.checked_mul(u64::from(raw_dt.type_size())));
+    if bytes.is_none_or(|b| b > MAX_DATA_BYTES) {
+        writeln!(out, "{path} too large to read").unwrap();
+        return;
+    }
+    writeln!(
+        out,
+        "{path} all {}",
+        value(&ds.read_selection(&Selection::All))
+    )
+    .unwrap();
+    if matches!(
+        dtype,
+        DType::F32
+            | DType::F64
+            | DType::I8
+            | DType::I16
+            | DType::I32
+            | DType::I64
+            | DType::U8
+            | DType::U16
+            | DType::U32
+            | DType::U64
+    ) {
+        writeln!(out, "{path} f64 {}", value(&ds.read_f64())).unwrap();
+        if let Some(&d0) = shape.first() {
+            let rank = shape.len();
+            let sel = Selection::Hyperslab {
+                start: std::iter::once(d0 / 3)
+                    .chain(std::iter::repeat_n(0, rank - 1))
+                    .collect(),
+                stride: vec![1; rank],
+                count: std::iter::once(d0.div_ceil(3))
+                    .chain(shape[1..].iter().copied())
+                    .collect(),
+                block: vec![1; rank],
+            };
+            writeln!(
+                out,
+                "{path} f64 third {}",
+                value(&ds.read_f64_selection(&sel))
+            )
+            .unwrap();
+        }
+    }
+    match &raw_dt {
+        clawhdf5_format::datatype::Datatype::String { .. }
+        | clawhdf5_format::datatype::Datatype::VariableLength {
+            is_string: true, ..
+        } => {
+            writeln!(out, "{path} strings {}", value(&ds.read_string_bytes())).unwrap();
+        }
+        clawhdf5_format::datatype::Datatype::VariableLength { .. } => {
+            writeln!(out, "{path} vlen {}", value(&ds.read_vlen::<f64>())).unwrap();
+        }
+        _ => {}
+    }
+}
+
+/// List the file as a tree view does — every group's entries, every
+/// dataset's shape and type — and return the largest dataset whose data
+/// is at most `MAX_DATA_BYTES` (address, bytes), the one a viewer would
+/// plot.
+pub fn list(file: &File) -> Option<(u64, u64)> {
+    let mut seen = HashSet::new();
+    let mut largest: Option<(u64, u64)> = None;
+    let mut queue = VecDeque::from([file.superblock().root_group_address]);
+    while let Some(addr) = queue.pop_front() {
+        if seen.len() >= MAX_OBJECTS || !seen.insert(addr) {
+            continue;
+        }
+        let group = file.group_at(addr);
+        if let Ok(ds) = file.dataset_at(addr) {
+            let _ = (ds.shape(), ds.dtype());
+            let bytes = ds.shape().ok().and_then(|s| {
+                let n = s.iter().try_fold(1u64, |a, &d| a.checked_mul(d))?;
+                let size = u64::from(ds.raw_datatype().ok()?.type_size());
+                n.checked_mul(size).filter(|&b| b <= MAX_DATA_BYTES)
+            });
+            if let Some(b) = bytes
+                && largest.is_none_or(|(_, l)| b > l)
+            {
+                largest = Some((addr, b));
+            }
+        }
+        if let Ok(entries) = group.entries() {
+            queue.extend(entries.into_iter().map(|(_, a)| a));
+        }
+    }
+    largest
+}
+
+/// Read the dataset at `addr` whole.
+pub fn read_one(file: &File, addr: u64) {
+    if let Ok(ds) = file.dataset_at(addr) {
+        let _ = ds.read_selection(&Selection::All);
+    }
+}
+
+/// [`list`], then [`read_one`] of the dataset it picks.
+pub fn list_and_read_one(file: &File) {
+    if let Some((addr, _)) = list(file) {
+        read_one(file, addr);
+    }
+}
+
+/// External virtual-dataset sources read from `dir`, as `File::open` finds
+/// them next to the file.
+pub fn sibling_resolver(dir: PathBuf) -> clawhdf5::VdsResolver {
+    Arc::new(move |name: &str| {
+        let p = Path::new(name);
+        if name.is_empty()
+            || !p
+                .components()
+                .all(|c| matches!(c, std::path::Component::Normal(_)))
+        {
+            return Err(FormatError::Storage(format!("{name:?} not followed")));
+        }
+        match std::fs::read(dir.join(p)) {
+            Ok(bytes) => Ok(Some(bytes)),
+            Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
+            Err(e) => Err(FormatError::Storage(e.to_string())),
+        }
+    })
+}
+
+/// HDF5 files under `dir`, recursively.
+pub fn hdf5_files(dir: &Path, out: &mut Vec<PathBuf>) {
+    let Ok(entries) = std::fs::read_dir(dir) else {
+        return;
+    };
+    for e in entries.flatten() {
+        let p = e.path();
+        if p.is_dir() {
+            hdf5_files(&p, out);
+        } else if p
+            .extension()
+            .and_then(|x| x.to_str())
+            .is_some_and(|x| matches!(x, "h5" | "hdf5" | "he5" | "nc" | "h5ad" | "hdf"))
+        {
+            out.push(p);
+        }
+    }
+}
+
+/// The repository's HDF5 test fixtures.
+pub fn fixtures() -> Vec<PathBuf> {
+    let root = Path::new(env!("CARGO_MANIFEST_DIR"));
+    let mut files = Vec::new();
+    hdf5_files(&root.join("../clawhdf5/tests/fixtures"), &mut files);
+    hdf5_files(&root.join("../clawhdf5-format/tests/fixtures"), &mut files);
+    files.sort();
+    files
+}
+
+/// Files of `CLAWHDF5_REMOTE_CORPUS` (directories separated like `PATH`).
+pub fn corpus() -> Option<Vec<PathBuf>> {
+    let dirs = std::env::var("CLAWHDF5_REMOTE_CORPUS").ok()?;
+    let mut files = Vec::new();
+    for d in std::env::split_paths(&dirs) {
+        hdf5_files(&d, &mut files);
+    }
+    files.sort();
+    Some(files)
+}
+
+pub fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+pub fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+/// Whether python3 with h5py and numpy runs; panics when interop is
+/// required and it does not.
+pub fn have_h5py() -> bool {
+    let ok = Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false);
+    assert!(
+        ok || !interop_required(),
+        "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+    );
+    if !ok {
+        eprintln!("SKIP: python3 with h5py not available");
+    }
+    ok
+}
+
+/// Run a Python script; its stdout.
+pub fn run_python(script: &str, args: &[&str]) -> String {
+    let out = Command::new(python())
+        .arg("-c")
+        .arg(script)
+        .args(args)
+        .output()
+        .expect("failed to run python");
+    assert!(
+        out.status.success(),
+        "python failed:\n{}",
+        String::from_utf8_lossy(&out.stderr)
+    );
+    String::from_utf8(out.stdout).unwrap()
+}
+
+/// A clawhdf5-written file with a multi-block chunked dataset (`/big`,
+/// 1 000 000 f64 in chunks of 10 000, deflated), a contiguous one and a
+/// group — several blocks of 1 MiB, with no Python needed.
+pub fn multi_block_file() -> Vec<u8> {
+    let mut b = clawhdf5::FileBuilder::new();
+    b.set_attr("title", clawhdf5::AttrValue::String("remote test".into()));
+    let big: Vec<f64> = (0..1_000_000u64)
+        .map(|i| ((i * 2_654_435_761) % 1_000_003) as f64 * 0.5)
+        .collect();
+    b.create_dataset("big")
+        .with_f64_data(&big)
+        .with_chunks(&[10_000])
+        .with_deflate(1);
+    let flat: Vec<f64> = (0..300_000u64).map(|i| i as f64).collect();
+    b.create_dataset("flat").with_f64_data(&flat);
+    let mut g = b.create_group("grp");
+    g.create_dataset("small").with_f64_data(&[1.0, 2.0, 3.0]);
+    b.add_group(g.finish());
+    b.finish().unwrap()
+}

crates/clawhdf5-remote/tests/common/server.rs

@@ -0,0 +1,438 @@
+//! A small HTTP/1.1 file server on 127.0.0.1 for tests and the example:
+//! `Range: bytes=a-b` requests (206, 416), `HEAD`, keep-alive, strong ETags
+//! and Last-Modified with `If-Match` / `If-Unmodified-Since` (412), and
+//! switches to misbehave — ignore ranges (200 with the whole file), cut
+//! bodies short, answer 503, respond slowly, send no validators. It counts
+//! requests and body bytes, and logs every range asked for — only for the
+//! paths it serves: a request for any other path (a local port scanner's
+//! `GET /`, say) is answered 404 and not counted, so request budgets in
+//! tests stay exact.
+
+#![allow(dead_code)]
+
+use std::collections::HashMap;
+use std::io::{BufRead, BufReader, Write};
+use std::net::{TcpListener, TcpStream};
+use std::sync::atomic::{AtomicBool, AtomicU32, AtomicU64, Ordering};
+use std::sync::{Arc, Mutex, RwLock};
+use std::time::Duration;
+
+/// A logged GET: the path and the range asked for (`None`: whole file).
+pub type LogEntry = (String, Option<(u64, u64)>);
+
+struct Resource {
+    data: Arc<Vec<u8>>,
+    etag: String,
+    last_modified: String,
+}
+
+/// Switches and counters shared with the connection threads.
+#[derive(Default)]
+pub struct Shared {
+    files: RwLock<HashMap<String, Resource>>,
+    version: AtomicU64,
+    /// Answer every request with 200 and the whole file.
+    pub ignore_range: AtomicBool,
+    /// Send no ETag and no Last-Modified.
+    pub no_validators: AtomicBool,
+    /// Send a weak ETag only (no Last-Modified).
+    pub weak_etag: AtomicBool,
+    /// Send Last-Modified but no ETag.
+    pub no_etag: AtomicBool,
+    /// Label bodies `Content-Encoding: gzip` (they are not).
+    pub gzip_label: AtomicBool,
+    /// Cut the body of the next N ranged responses in half (then close the
+    /// connection).
+    pub truncate_next: AtomicU32,
+    /// Answer the next N requests with 503.
+    pub fail_next: AtomicU32,
+    /// Sleep this long before answering each request.
+    pub delay_ms: AtomicU64,
+    /// When non-zero, claim the file is this long (in `Content-Range` and
+    /// when checking ranges) and serve zeros past its real end: a hostile
+    /// server lying about the length.
+    pub fake_total: AtomicU64,
+    /// When non-zero, answer every request for a served file with this
+    /// status (and an empty body).
+    pub force_status: AtomicU32,
+    /// When non-zero, send bodies at about this many bytes per second.
+    pub throttle_bps: AtomicU64,
+    /// When non-zero, stop this many milliseconds after the headers and
+    /// half the body (a stalled connection).
+    pub stall_ms: AtomicU64,
+    /// Answer ranges with a `Content-Range` one byte off.
+    pub wrong_range: AtomicBool,
+    /// Path → `Location`: answered `302 Found` (counted as a request).
+    pub redirects: RwLock<HashMap<String, String>>,
+    /// (path, headers) of every counted request, header names lowercase.
+    pub seen: Mutex<Vec<(String, HashMap<String, String>)>>,
+    /// Requests for a served path (every status); requests for other
+    /// paths are not counted.
+    pub requests: AtomicU64,
+    /// Body bytes sent.
+    pub bytes: AtomicU64,
+    /// (path, range) of every GET.
+    pub log: Mutex<Vec<LogEntry>>,
+    stop: AtomicBool,
+}
+
+/// A running server; stops accepting when dropped.
+pub struct Server {
+    pub addr: std::net::SocketAddr,
+    pub shared: Arc<Shared>,
+}
+
+impl Server {
+    /// Serve `files` (URL path such as `/a.h5` → bytes) on `127.0.0.1`, on
+    /// a free port.
+    pub fn start(files: Vec<(String, Vec<u8>)>) -> Server {
+        Server::bind("127.0.0.1:0", files)
+    }
+
+    /// Serve `files` on `addr`.
+    pub fn bind(addr: &str, files: Vec<(String, Vec<u8>)>) -> Server {
+        let listener = TcpListener::bind(addr).expect("bind");
+        let addr = listener.local_addr().unwrap();
+        let shared = Arc::new(Shared::default());
+        for (path, data) in files {
+            shared.put(&path, data);
+        }
+        let s = shared.clone();
+        std::thread::spawn(move || {
+            for conn in listener.incoming() {
+                if s.stop.load(Ordering::SeqCst) {
+                    break;
+                }
+                let Ok(conn) = conn else { continue };
+                let s = s.clone();
+                std::thread::spawn(move || {
+                    let _ = serve(conn, &s);
+                });
+            }
+        });
+        Server { addr, shared }
+    }
+
+    /// The URL of `path` (which starts with `/`).
+    pub fn url(&self, path: &str) -> String {
+        format!("http://{}{path}", self.addr)
+    }
+
+    /// Requests served so far.
+    pub fn requests(&self) -> u64 {
+        self.shared.requests.load(Ordering::SeqCst)
+    }
+
+    /// Body bytes sent so far.
+    pub fn bytes(&self) -> u64 {
+        self.shared.bytes.load(Ordering::SeqCst)
+    }
+
+    /// Zero the counters and the logs.
+    pub fn reset(&self) {
+        self.shared.requests.store(0, Ordering::SeqCst);
+        self.shared.bytes.store(0, Ordering::SeqCst);
+        self.shared.log.lock().unwrap().clear();
+        self.shared.seen.lock().unwrap().clear();
+    }
+
+    /// Answer `path` with a redirect to `location`.
+    pub fn redirect(&self, path: &str, location: &str) {
+        self.shared
+            .redirects
+            .write()
+            .unwrap()
+            .insert(path.to_string(), location.to_string());
+    }
+
+    /// Whether any request the server counted carried header `name`.
+    pub fn saw_header(&self, name: &str) -> bool {
+        self.shared
+            .seen
+            .lock()
+            .unwrap()
+            .iter()
+            .any(|(_, h)| h.contains_key(name))
+    }
+
+    /// The ranges asked for so far.
+    pub fn log(&self) -> Vec<LogEntry> {
+        self.shared.log.lock().unwrap().clone()
+    }
+}
+
+impl Drop for Server {
+    fn drop(&mut self) {
+        self.shared.stop.store(true, Ordering::SeqCst);
+        let _ = TcpStream::connect(self.addr);
+    }
+}
+
+impl Shared {
+    /// Add or replace a file (a replacement gets a new ETag and
+    /// Last-Modified).
+    pub fn put(&self, path: &str, data: Vec<u8>) {
+        let v = self.version.fetch_add(1, Ordering::SeqCst) + 1;
+        let secs = 1_700_000_000 + v;
+        self.files.write().unwrap().insert(
+            path.to_string(),
+            Resource {
+                data: Arc::new(data),
+                etag: format!("\"v{v}-{path}\""),
+                last_modified: http_date(secs),
+            },
+        );
+    }
+}
+
+/// An IMF-fixdate for `secs` since the epoch (enough for distinct values).
+fn http_date(secs: u64) -> String {
+    const DAYS: [&str; 7] = ["Thu", "Fri", "Sat", "Sun", "Mon", "Tue", "Wed"];
+    const MONTHS: [&str; 12] = [
+        "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec",
+    ];
+    let days = secs / 86_400;
+    let rem = secs % 86_400;
+    // Civil-from-days (Howard Hinnant).
+    let z = days as i64 + 719_468;
+    let era = z.div_euclid(146_097);
+    let doe = z - era * 146_097;
+    let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365;
+    let y = yoe + era * 400;
+    let doy = doe - (365 * yoe + yoe / 4 - yoe / 100);
+    let mp = (5 * doy + 2) / 153;
+    let d = doy - (153 * mp + 2) / 5 + 1;
+    let m = if mp < 10 { mp + 3 } else { mp - 9 };
+    let y = if m <= 2 { y + 1 } else { y };
+    format!(
+        "{}, {:02} {} {} {:02}:{:02}:{:02} GMT",
+        DAYS[(days % 7) as usize],
+        d,
+        MONTHS[(m - 1) as usize],
+        y,
+        rem / 3600,
+        rem % 3600 / 60,
+        rem % 60
+    )
+}
+
+fn parse_range(v: &str, len: u64) -> Option<Result<(u64, u64), ()>> {
+    let spec = v.trim().strip_prefix("bytes=")?;
+    if spec.contains(',') {
+        return None; // multiple ranges: serve the whole file
+    }
+    let (a, b) = spec.split_once('-')?;
+    let a: u64 = a.trim().parse().ok()?;
+    let b: u64 = match b.trim() {
+        "" => u64::MAX,
+        b => b.parse().ok()?,
+    };
+    if b < a {
+        return None;
+    }
+    if a >= len {
+        return Some(Err(()));
+    }
+    Some(Ok((a, b.min(len - 1))))
+}
+
+fn serve(conn: TcpStream, s: &Shared) -> std::io::Result<()> {
+    conn.set_read_timeout(Some(Duration::from_secs(30)))?;
+    // One write per response and no Nagle delay: otherwise every request
+    // waits for a delayed ACK (~40 ms).
+    conn.set_nodelay(true)?;
+    let mut reader = BufReader::new(conn.try_clone()?);
+    let mut out = conn;
+    loop {
+        let mut line = String::new();
+        if reader.read_line(&mut line)? == 0 {
+            return Ok(());
+        }
+        let mut parts = line.split_whitespace();
+        let method = parts.next().unwrap_or("").to_string();
+        let path = parts.next().unwrap_or("").to_string();
+        let mut headers = HashMap::new();
+        loop {
+            let mut h = String::new();
+            if reader.read_line(&mut h)? == 0 {
+                return Ok(());
+            }
+            let h = h.trim_end();
+            if h.is_empty() {
+                break;
+            }
+            if let Some((k, v)) = h.split_once(':') {
+                headers.insert(k.trim().to_ascii_lowercase(), v.trim().to_string());
+            }
+        }
+        // The query string (a presigned URL's signature, say) is not part
+        // of the file's name.
+        let path = path.split('?').next().unwrap_or("").to_string();
+        let close = headers
+            .get("connection")
+            .is_some_and(|v| v.eq_ignore_ascii_case("close"));
+        let redirect = s.redirects.read().unwrap().get(&path).cloned();
+        if let Some(location) = redirect {
+            s.requests.fetch_add(1, Ordering::SeqCst);
+            s.seen.lock().unwrap().push((path.clone(), headers.clone()));
+            write!(
+                out,
+                "HTTP/1.1 302 Found\r\nLocation: {location}\r\nContent-Length: 0\r\n\r\n"
+            )?;
+            continue;
+        }
+        let res = {
+            let files = s.files.read().unwrap();
+            files
+                .get(&path)
+                .map(|r| (r.data.clone(), r.etag.clone(), r.last_modified.clone()))
+        };
+        let Some((data, etag, lm)) = res else {
+            // Not ours: not counted, not delayed, no failure injected.
+            write!(out, "HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n")?;
+            if close {
+                return Ok(());
+            }
+            continue;
+        };
+        s.requests.fetch_add(1, Ordering::SeqCst);
+        s.seen.lock().unwrap().push((path.clone(), headers.clone()));
+        let delay = s.delay_ms.load(Ordering::SeqCst);
+        if delay > 0 {
+            std::thread::sleep(Duration::from_millis(delay));
+        }
+        let forced = s.force_status.load(Ordering::SeqCst);
+        if forced != 0 {
+            write!(out, "HTTP/1.1 {forced} Forced\r\nContent-Length: 0\r\n\r\n")?;
+            continue;
+        }
+        if s.fail_next
+            .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |n| n.checked_sub(1))
+            .is_ok()
+        {
+            write!(
+                out,
+                "HTTP/1.1 503 Service Unavailable\r\nContent-Length: 0\r\n\r\n"
+            )?;
+            continue;
+        }
+        let fake = s.fake_total.load(Ordering::SeqCst);
+        let len = if fake > 0 { fake } else { data.len() as u64 };
+        let mut validators = String::new();
+        if s.weak_etag.load(Ordering::SeqCst) {
+            validators = format!("ETag: W/{etag}\r\n");
+        } else if s.no_etag.load(Ordering::SeqCst) {
+            validators = format!("Last-Modified: {lm}\r\n");
+        } else if !s.no_validators.load(Ordering::SeqCst) {
+            validators = format!("ETag: {etag}\r\nLast-Modified: {lm}\r\n");
+        }
+        let precondition_failed = headers.get("if-match").is_some_and(|v| v != &etag)
+            || headers.get("if-unmodified-since").is_some_and(|v| v != &lm);
+        if precondition_failed {
+            write!(
+                out,
+                "HTTP/1.1 412 Precondition Failed\r\n{validators}Content-Length: 0\r\n\r\n"
+            )?;
+            continue;
+        }
+        let range = if s.ignore_range.load(Ordering::SeqCst) {
+            None
+        } else {
+            headers.get("range").and_then(|v| parse_range(v, len))
+        };
+        if method == "GET" {
+            s.log
+                .lock()
+                .unwrap()
+                .push((path.clone(), range.and_then(Result::ok)));
+        }
+        let mut padded = Vec::new();
+        let (status, body, extra) = match range {
+            Some(Err(())) => {
+                write!(
+                    out,
+                    "HTTP/1.1 416 Range Not Satisfiable\r\nContent-Range: bytes */{len}\r\n\
+                     {validators}Content-Length: 0\r\n\r\n"
+                )?;
+                continue;
+            }
+            Some(Ok((a, b))) => (
+                "206 Partial Content",
+                slice_or_zeros(&data, a, b, &mut padded),
+                if s.wrong_range.load(Ordering::SeqCst) {
+                    format!("Content-Range: bytes {}-{}/{len}\r\n", a + 1, b + 1)
+                } else {
+                    format!("Content-Range: bytes {a}-{b}/{len}\r\n")
+                },
+            ),
+            None => ("200 OK", &data[..], String::new()),
+        };
+        let extra = if s.gzip_label.load(Ordering::SeqCst) {
+            format!("{extra}Content-Encoding: gzip\r\n")
+        } else {
+            extra
+        };
+        let head = format!(
+            "HTTP/1.1 {status}\r\nContent-Length: {}\r\nAccept-Ranges: bytes\r\n{extra}{validators}\r\n",
+            body.len()
+        );
+        if method == "HEAD" {
+            out.write_all(head.as_bytes())?;
+            continue;
+        }
+        let truncate = status.starts_with("206")
+            && s.truncate_next
+                .fetch_update(Ordering::SeqCst, Ordering::SeqCst, |n| n.checked_sub(1))
+                .is_ok();
+        let body = if truncate {
+            &body[..body.len() / 2]
+        } else {
+            body
+        };
+        // Counted before the client can have the bytes, so a test that
+        // resets the counters after a read never sees them arrive late.
+        s.bytes.fetch_add(body.len() as u64, Ordering::SeqCst);
+        let bps = s.throttle_bps.load(Ordering::SeqCst);
+        let stall = s.stall_ms.load(Ordering::SeqCst);
+        if bps > 0 || stall > 0 {
+            out.write_all(head.as_bytes())?;
+            let (first, rest) = body.split_at(if stall > 0 { body.len() / 2 } else { 0 });
+            out.write_all(first)?;
+            out.flush()?;
+            if stall > 0 {
+                std::thread::sleep(Duration::from_millis(stall));
+            }
+            for piece in rest.chunks(4096) {
+                out.write_all(piece)?;
+                out.flush()?;
+                if let Some(us) = (4096 * 1_000_000u64).checked_div(bps) {
+                    std::thread::sleep(Duration::from_micros(us));
+                }
+            }
+        } else {
+            let mut response = head.into_bytes();
+            response.extend_from_slice(body);
+            out.write_all(&response)?;
+            out.flush()?;
+        }
+        if truncate || close {
+            let _ = out.shutdown(std::net::Shutdown::Both);
+            return Ok(());
+        }
+    }
+}
+
+/// `data[a..=b]`, padded with zeros past its end (into `padded`).
+fn slice_or_zeros<'a>(data: &'a [u8], a: u64, b: u64, padded: &'a mut Vec<u8>) -> &'a [u8] {
+    let real = data.len() as u64;
+    if b < real {
+        return &data[a as usize..=b as usize];
+    }
+    let n = usize::try_from(b - a + 1).expect("range fits in memory");
+    padded.resize(n, 0);
+    if a < real {
+        let have = (real - a) as usize;
+        padded[..have].copy_from_slice(&data[a as usize..]);
+    }
+    padded
+}

crates/clawhdf5-remote/tests/http.rs

@@ -0,0 +1,860 @@
+//! HTTP range reads against a local server (no internet): every value read
+//! over HTTP equals `File::open`'s, and the server's misbehaviour — no
+//! range support, a file replaced mid-read, cut-off bodies, errors, slow
+//! answers under concurrent readers — is an error or the right data, never
+//! wrong data.
+//!
+//! `CLAWHDF5_REMOTE_CORPUS=dir[:dir...]` adds every HDF5 file under those
+//! directories (the conformance corpus is `conformance/.cache/corpus`), and
+//! `CLAWHDF5_REMOTE_REPORT=1` prints the per-file request counts.
+
+#![cfg(feature = "http")]
+
+mod common;
+
+use std::sync::Arc;
+use std::sync::atomic::Ordering;
+use std::time::Duration;
+
+use clawhdf5::File;
+use clawhdf5_remote::{
+    CacheConfig, Error, HttpOptions, HttpStorage, Options, RemoteError, open_url, open_url_with,
+    storage_for_url,
+};
+use common::server::Server;
+use common::{list_and_read_one, multi_block_file, transcript};
+
+/// Options for tests: fast retries.
+fn quick() -> Options {
+    Options {
+        http: HttpOptions {
+            backoff: Duration::from_millis(5),
+            ..HttpOptions::default()
+        },
+        ..Options::default()
+    }
+}
+
+fn changed(e: &str) -> bool {
+    e.contains("changed while open")
+}
+
+/// Serve `files`, open each by URL and through `File::open`, and require
+/// identical transcripts. Returns (files compared, files both refused).
+fn compare(files: &[std::path::PathBuf], report: bool) -> (usize, usize) {
+    let served: Vec<(String, Vec<u8>)> = files
+        .iter()
+        .enumerate()
+        .filter_map(|(i, p)| {
+            let bytes = std::fs::read(p).ok()?;
+            let name = p.file_name()?.to_str()?.replace(' ', "_");
+            Some((format!("/f{i}/{name}"), bytes))
+        })
+        .collect();
+    let server = Server::start(served.clone());
+    let (mut same, mut refused) = (0, 0);
+    let mut totals = [0u64; 7];
+    for (i, p) in files.iter().enumerate() {
+        let Some((url_path, bytes)) = served
+            .iter()
+            .find(|(u, _)| u.starts_with(&format!("/f{i}/")))
+        else {
+            continue;
+        };
+        let url = server.url(url_path);
+        let local = File::open(p);
+        let remote = storage_for_url(&url, &quick())
+            .map_err(|e| e.to_string())
+            .and_then(|s| {
+                File::open_storage(s.clone())
+                    .map(|mut f| {
+                        f.set_vds_resolver(common::sibling_resolver(p.parent().unwrap().into()));
+                        (f, s)
+                    })
+                    .map_err(|e| e.to_string())
+            });
+        let (local, (remote, storage)) = match (local, remote) {
+            (Ok(l), Ok(r)) => (l, r),
+            (Err(l), Err(r)) => {
+                assert!(
+                    !r.contains("HTTP") && !r.contains("network"),
+                    "{url}: {r} ({l})"
+                );
+                refused += 1;
+                continue;
+            }
+            (l, r) => panic!(
+                "{}: File::open {:?}, open_url {:?}",
+                p.display(),
+                l.map(|_| ()),
+                r.map(|_| ())
+            ),
+        };
+        let want = transcript(&local);
+        let got = transcript(&remote);
+        if want != got {
+            let first = want
+                .lines()
+                .zip(got.lines())
+                .find(|(w, g)| w != g)
+                .map(|(w, g)| format!("\n local:  {w}\n remote: {g}"))
+                .unwrap_or_default();
+            panic!("{}: open_url differs from File::open{first}", p.display());
+        }
+        assert!(storage.stats().reads > 0, "read through the cache");
+        same += 1;
+
+        // Cost of "open + list" (a tree view) and of then reading the
+        // largest dataset (a plot): with the block cache, and with none
+        // (every read a request). Requests and bytes as the server saw
+        // them, the open included.
+        server.reset();
+        let with = storage_for_url(&url, &quick()).unwrap();
+        let f = File::open_storage(with.clone()).unwrap();
+        let pick = common::list(&f);
+        let (list_requests, list_bytes) = (server.requests(), server.bytes());
+        if let Some((addr, _)) = pick {
+            common::read_one(&f, addr);
+        }
+        let (cached_requests, cached_bytes) = (server.requests(), server.bytes());
+        server.reset();
+        let (bare, _) = HttpStorage::open(&url, quick().http).unwrap();
+        if let Ok(f) = File::open_storage(Arc::new(bare)) {
+            list_and_read_one(&f);
+        }
+        let uncached_requests = server.requests();
+        for (t, v) in totals.iter_mut().zip([
+            list_requests,
+            list_bytes,
+            cached_requests,
+            cached_bytes,
+            uncached_requests,
+            bytes.len() as u64,
+            1,
+        ]) {
+            *t += v;
+        }
+        if report {
+            eprintln!(
+                "open+list {list_requests:>4} req {list_bytes:>10} B | +read {:>5} req \
+                 {cached_bytes:>10} B | uncached {uncached_requests:>7} req | file {:>10} B  {}",
+                cached_requests,
+                bytes.len(),
+                p.display()
+            );
+        }
+        // Files within one block: opening fetched everything.
+        if bytes.len() as u64 <= with.config().block_size {
+            assert_eq!(cached_requests, 1, "{}", p.display());
+        }
+        // The budget of docs/design/range-reads.md (Testing): listing the
+        // IMERG file (file A of section 2) takes at most 3 requests.
+        if p.ends_with("xarray-data/imerghh_730.hdf5") {
+            assert!(list_requests <= 3, "{}: {list_requests}", p.display());
+        }
+    }
+    eprintln!(
+        "{} files ({} bytes): open + list {} requests, {} bytes; + read the largest \
+         dataset {} requests, {} bytes (1 MiB block cache); without a cache {} requests",
+        totals[6], totals[5], totals[0], totals[1], totals[2], totals[3], totals[4]
+    );
+    (same, refused)
+}
+
+#[test]
+fn fixtures_read_identically_over_http() {
+    let files = common::fixtures();
+    assert!(files.len() >= 45, "{} fixtures", files.len());
+    let report = std::env::var("CLAWHDF5_REMOTE_REPORT").is_ok_and(|v| v == "1");
+    let (same, _) = compare(&files, report);
+    assert!(same >= 40, "{same}");
+}
+
+#[test]
+fn corpus_reads_identically_over_http() {
+    let Some(files) = common::corpus() else {
+        eprintln!("CLAWHDF5_REMOTE_CORPUS not set; skipping the corpus");
+        return;
+    };
+    let report = std::env::var("CLAWHDF5_REMOTE_REPORT").is_ok_and(|v| v == "1");
+    let (same, refused) = compare(&files, report);
+    eprintln!("corpus: {same} files identical, {refused} refused by both");
+    assert!(same > 0);
+}
+
+#[test]
+fn multi_block_file_values_and_request_budget() {
+    let bytes = multi_block_file();
+    assert!(bytes.len() > 3 << 20, "{}", bytes.len());
+    let server = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    let url = server.url("/m.h5");
+    let local = File::from_bytes(bytes.clone()).unwrap();
+    let storage = storage_for_url(&url, &quick()).unwrap();
+    assert_eq!(server.requests(), 1, "opening is one request");
+    let remote = File::open_storage(storage.clone()).unwrap();
+    assert!(remote.contiguous_bytes().is_none());
+    let mut names = remote.root().datasets().unwrap();
+    names.sort();
+    assert_eq!(names, ["big", "flat"]);
+    assert_eq!(remote.root().groups().unwrap(), ["grp"]);
+    assert_eq!(
+        remote.dataset("grp/small").unwrap().read_f64().unwrap(),
+        [1.0, 2.0, 3.0]
+    );
+    let big = remote.dataset("big").unwrap().read_f64().unwrap();
+    assert_eq!(big, local.dataset("big").unwrap().read_f64().unwrap());
+    let flat = remote.dataset("flat").unwrap().read_f64().unwrap();
+    assert_eq!(flat.len(), 300_000);
+    assert_eq!(flat[299_999], 299_999.0);
+    // Every byte was fetched at most once, in whole 1 MiB blocks.
+    let log = server.log();
+    let mut blocks: Vec<u64> = Vec::new();
+    for (_, r) in &log {
+        let (a, b) = r.expect("every request is ranged");
+        assert_eq!(a % (1 << 20), 0, "block-aligned");
+        blocks.extend(a >> 20..=b >> 20);
+    }
+    let n = blocks.len();
+    blocks.sort_unstable();
+    blocks.dedup();
+    assert_eq!(n, blocks.len(), "a block was fetched twice: {log:?}");
+    assert!(
+        server.requests() <= (bytes.len() as u64 >> 20) + 2,
+        "{} requests for a {} byte file",
+        server.requests(),
+        bytes.len()
+    );
+    // Reading again costs nothing.
+    let before = server.requests();
+    assert_eq!(transcript(&remote), transcript(&local));
+    assert_eq!(server.requests(), before);
+}
+
+#[test]
+fn h5py_written_file_over_http() {
+    if !common::have_h5py() {
+        return;
+    }
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("py.h5");
+    common::run_python(
+        r#"
+import sys, h5py, numpy as np
+with h5py.File(sys.argv[1], "w", libver="latest") as f:
+    f.attrs["title"] = "h5py remote test"
+    g = f.create_group("sensors")
+    for i in range(40):
+        d = g.create_dataset(f"s{i:02d}", data=np.arange(1000, dtype="<i4") * (i + 1))
+        d.attrs["index"] = i
+    t = f.create_dataset("temps", data=np.sin(np.arange(2_000_000) / 1000.0),
+                         chunks=(50_000,), compression="gzip", shuffle=True)
+    t.attrs["units"] = "K"
+    f.create_dataset("names", data=[b"alpha", b"beta", b"gamma"])
+    f.create_dataset("vl", data=["one", "two", "three"], dtype=h5py.string_dtype())
+    f["link"] = h5py.SoftLink("/sensors/s03")
+"#,
+        &[path.to_str().unwrap()],
+    );
+    // What libhdf5 reads, for the values the test checks.
+    let sums = common::run_python(
+        r#"
+import sys, h5py, numpy as np
+with h5py.File(sys.argv[1], "r") as f:
+    print(repr(float(f["temps"][:].sum())), int(f["sensors/s39"][:].sum()), f["vl"].asstr()[1])
+"#,
+        &[path.to_str().unwrap()],
+    );
+    let bytes = std::fs::read(&path).unwrap();
+    let server = Server::start(vec![("/py.h5".into(), bytes.clone())]);
+    let remote = open_url_with(&server.url("/py.h5"), &quick()).unwrap();
+    let local = File::open(&path).unwrap();
+    assert_eq!(transcript(&remote), transcript(&local));
+    let temps = remote.dataset("temps").unwrap().read_f64().unwrap();
+    let s39: i64 = remote
+        .dataset("sensors/s39")
+        .unwrap()
+        .read_i64()
+        .unwrap()
+        .iter()
+        .sum();
+    let vl = remote.dataset("vl").unwrap().read_string().unwrap();
+    let mut it = sums.split_whitespace();
+    let want_sum: f64 = it.next().unwrap().parse().unwrap();
+    let got_sum: f64 = temps.iter().sum();
+    assert!((got_sum - want_sum).abs() <= 1e-6 * want_sum.abs().max(1.0));
+    assert_eq!(s39, it.next().unwrap().parse::<i64>().unwrap());
+    assert_eq!(vl[1], it.next().unwrap());
+    assert_eq!(
+        remote.dataset("link").unwrap().read_i32().unwrap(),
+        local.dataset("sensors/s03").unwrap().read_i32().unwrap()
+    );
+}
+
+#[test]
+fn a_server_that_ignores_range_is_refused_or_downloaded_when_allowed() {
+    let bytes = multi_block_file();
+    let server = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    server.shared.ignore_range.store(true, Ordering::SeqCst);
+    let url = server.url("/m.h5");
+    let err = open_url_with(&url, &quick()).unwrap_err();
+    assert!(
+        matches!(err, Error::Remote(RemoteError::RangeNotSupported(_))),
+        "{err}"
+    );
+    assert_eq!(server.requests(), 1, "refused at the first response");
+    let mut opts = quick();
+    opts.http.allow_full_download = true;
+    let storage = storage_for_url(&url, &opts).unwrap();
+    let f = File::open_storage(storage.clone()).unwrap();
+    assert_eq!(f.contiguous_bytes(), Some(&bytes[..]), "read from memory");
+    assert_eq!(server.requests(), 2);
+    let local = File::from_bytes(bytes).unwrap();
+    assert_eq!(transcript(&f), transcript(&local));
+    assert_eq!(server.requests(), 2, "no further requests");
+    // Too large for the download limit.
+    opts.http.max_full_download = 1000;
+    assert!(open_url_with(&url, &opts).is_err());
+}
+
+#[test]
+fn a_file_replaced_mid_read_is_an_error_not_mixed_data() {
+    for mode in ["etag", "last-modified", "none"] {
+        let bytes = multi_block_file();
+        let server = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+        match mode {
+            "last-modified" => server.shared.no_etag.store(true, Ordering::SeqCst),
+            "none" => server.shared.no_validators.store(true, Ordering::SeqCst),
+            _ => {}
+        }
+        let url = server.url("/m.h5");
+        let f = open_url_with(&url, &quick()).unwrap();
+        assert_eq!(f.root().groups().unwrap(), ["grp"], "{mode}");
+        // Replace the file: same layout, other values (and, for "none",
+        // one byte longer, which is all that can be checked).
+        let mut other = bytes.clone();
+        let n = other.len();
+        for b in &mut other[n / 2..n / 2 + 1000] {
+            *b ^= 0xff;
+        }
+        if mode == "none" {
+            other.push(0);
+        }
+        server.shared.put("/m.h5", other);
+        let err = f
+            .dataset("big")
+            .unwrap()
+            .read_f64()
+            .unwrap_err()
+            .to_string();
+        assert!(changed(&err), "{mode}: {err}");
+    }
+}
+
+#[test]
+fn a_weak_etag_or_no_validator_is_refused_when_required() {
+    let server = Server::start(vec![("/m.h5".into(), multi_block_file())]);
+    server.shared.weak_etag.store(true, Ordering::SeqCst);
+    let mut opts = quick();
+    opts.http.require_validator = true;
+    assert!(open_url_with(&server.url("/m.h5"), &opts).is_err());
+    opts.http.require_validator = false;
+    assert!(open_url_with(&server.url("/m.h5"), &opts).is_ok());
+}
+
+#[test]
+fn truncated_bodies_are_retried_then_an_error() {
+    let bytes = multi_block_file();
+    let server = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    let url = server.url("/m.h5");
+    let local = File::from_bytes(bytes).unwrap();
+    let want = local.dataset("big").unwrap().read_f64().unwrap();
+
+    // One cut-off body: retried, right values.
+    let (http, first) = HttpStorage::open(&url, quick().http).unwrap();
+    assert_eq!(first.len(), 1 << 20);
+    let storage = Arc::new(clawhdf5_remote::BlockCache::new(
+        http,
+        CacheConfig::default(),
+    ));
+    let f = File::open_storage(storage.clone()).unwrap();
+    server.shared.truncate_next.store(1, Ordering::SeqCst);
+    assert_eq!(f.dataset("big").unwrap().read_f64().unwrap(), want);
+    assert_eq!(storage.inner().stats().retries, 1);
+
+    // Every body cut off: an error, never partial data.
+    let f = open_url_with(&url, &quick()).unwrap();
+    server.shared.truncate_next.store(1000, Ordering::SeqCst);
+    let err = f
+        .dataset("big")
+        .unwrap()
+        .read_f64()
+        .unwrap_err()
+        .to_string();
+    assert!(
+        err.contains("network error") || err.contains("bad response"),
+        "{err}"
+    );
+    server.shared.truncate_next.store(0, Ordering::SeqCst);
+    // The failure was not cached: the next read succeeds.
+    assert_eq!(f.dataset("big").unwrap().read_f64().unwrap(), want);
+}
+
+#[test]
+fn server_errors_are_retried_with_backoff() {
+    let bytes = multi_block_file();
+    let server = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    server.shared.fail_next.store(2, Ordering::SeqCst);
+    let url = server.url("/m.h5");
+    let f = open_url_with(&url, &quick()).unwrap();
+    let local = File::from_bytes(bytes).unwrap();
+    server.shared.fail_next.store(3, Ordering::SeqCst);
+    assert_eq!(
+        f.dataset("flat").unwrap().read_f64().unwrap(),
+        local.dataset("flat").unwrap().read_f64().unwrap()
+    );
+    // More failures than retries: an error.
+    let f = open_url_with(&url, &quick()).unwrap();
+    server.shared.fail_next.store(100, Ordering::SeqCst);
+    let err = f
+        .dataset("big")
+        .unwrap()
+        .read_f64()
+        .unwrap_err()
+        .to_string();
+    assert!(err.contains("503"), "{err}");
+}
+
+#[test]
+fn slow_server_concurrent_readers_fetch_each_block_once() {
+    let bytes = multi_block_file();
+    let server = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    let url = server.url("/m.h5");
+    let opts = Options {
+        cache: CacheConfig {
+            block_size: 256 << 10,
+            max_request: 256 << 10,
+            ..CacheConfig::default()
+        },
+        ..quick()
+    };
+    let storage = storage_for_url(&url, &opts).unwrap();
+    let file = File::open_storage(storage.clone()).unwrap();
+    let local = File::from_bytes(bytes).unwrap();
+    let want_big = local.dataset("big").unwrap().read_f64().unwrap();
+    let want_flat = local.dataset("flat").unwrap().read_f64().unwrap();
+    server.shared.delay_ms.store(40, Ordering::SeqCst);
+    server.reset();
+    std::thread::scope(|s| {
+        for t in 0..8 {
+            let (file, want_big, want_flat) = (&file, &want_big, &want_flat);
+            s.spawn(move || {
+                if t % 2 == 0 {
+                    assert_eq!(&file.dataset("big").unwrap().read_f64().unwrap(), want_big);
+                } else {
+                    assert_eq!(
+                        &file.dataset("flat").unwrap().read_f64().unwrap(),
+                        want_flat
+                    );
+                }
+            });
+        }
+    });
+    let log = server.log();
+    let mut starts: Vec<u64> = log.iter().map(|(_, r)| r.unwrap().0).collect();
+    let n = starts.len();
+    starts.sort_unstable();
+    starts.dedup();
+    assert_eq!(n, starts.len(), "a block was fetched twice");
+    assert!(storage.stats().waits > 0, "readers shared fetches");
+}
+
+#[test]
+fn bad_urls_and_statuses_are_clean_errors() {
+    let server = Server::start(vec![("/m.h5".into(), multi_block_file())]);
+    let err = open_url_with(&server.url("/missing.h5"), &quick()).unwrap_err();
+    assert!(
+        matches!(err, Error::Remote(RemoteError::Status { code: 404, .. })),
+        "{err}"
+    );
+    assert!(matches!(
+        open_url("ftp://example.com/a.h5"),
+        Err(Error::Remote(RemoteError::UnsupportedScheme(_)))
+    ));
+    assert!(matches!(
+        open_url("no-scheme"),
+        Err(Error::Remote(RemoteError::InvalidUrl(_)))
+    ));
+    #[cfg(not(feature = "s3"))]
+    {
+        let e = open_url("s3://bucket/key.h5").unwrap_err().to_string();
+        assert!(e.contains("`s3` feature"), "{e}");
+    }
+    #[cfg(not(feature = "https"))]
+    {
+        let e = open_url("https://example.com/a.h5")
+            .unwrap_err()
+            .to_string();
+        assert!(e.contains("`https` feature"), "{e}");
+    }
+    // A content-encoded body is not a byte range.
+    let server = Server::start(vec![("/m.h5".into(), multi_block_file())]);
+    server.shared.gzip_label.store(true, Ordering::SeqCst);
+    let e = open_url(&server.url("/m.h5")).unwrap_err().to_string();
+    assert!(e.contains("gzip-encoded"), "{e}");
+    // Not HDF5.
+    let server = Server::start(vec![("/x.h5".into(), vec![7u8; 5000])]);
+    assert!(matches!(
+        open_url(&server.url("/x.h5")),
+        Err(Error::Hdf5(_))
+    ));
+    // A server that closes every connection unanswered: a network error
+    // after the retries. (Not a closed port: another test's server could
+    // take it meanwhile.)
+    let port = {
+        let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
+        let port = l.local_addr().unwrap().port();
+        std::thread::spawn(move || {
+            for c in l.incoming() {
+                drop(c);
+            }
+        });
+        port
+    };
+    let err = open_url_with(&format!("http://127.0.0.1:{port}/a.h5"), &quick()).unwrap_err();
+    assert!(
+        matches!(err, Error::Remote(RemoteError::Transport(_))),
+        "{err}"
+    );
+}
+
+/// A request for a path the server does not serve (a local port scanner's
+/// `GET /`) does not count against a test's request budget.
+#[test]
+fn requests_for_other_paths_are_not_counted() {
+    use std::io::{Read, Write};
+    let server = Server::start(vec![("/m.h5".into(), multi_block_file())]);
+    let mut probe = std::net::TcpStream::connect(server.addr).unwrap();
+    probe
+        .write_all(b"GET / HTTP/1.1\r\nHost: x\r\nConnection: close\r\n\r\n")
+        .unwrap();
+    let mut answer = String::new();
+    probe.read_to_string(&mut answer).unwrap();
+    assert!(answer.starts_with("HTTP/1.1 404"), "{answer}");
+    storage_for_url(&server.url("/m.h5"), &quick()).unwrap();
+    assert_eq!(server.requests(), 1, "only the open counts");
+    assert_eq!(server.log().len(), 1);
+}
+
+/// A server claiming a length near `u64::MAX` (and serving zeros past the
+/// real data), with a file whose addresses point at the end of that range:
+/// clean errors or zeros, never an arithmetic overflow (a panic in debug).
+#[test]
+fn a_server_claiming_a_huge_length_does_not_overflow() {
+    let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR"));
+    let mut bytes =
+        std::fs::read(root.join("../clawhdf5-format/tests/fixtures/legacy/h5ex_g_iterate.h5"))
+            .unwrap();
+    // Superblock v0: end-of-file address, then the root symbol table
+    // entry's object header, B-tree and heap addresses.
+    bytes[0x28..0x30].copy_from_slice(&(u64::MAX - 1).to_le_bytes());
+    for at in [0x40, 0x50, 0x58] {
+        bytes[at..at + 8].copy_from_slice(&0xFFFF_FFFF_FFFF_F000u64.to_le_bytes());
+    }
+    let server = Server::start(vec![("/h.h5".into(), bytes)]);
+    for total in [u64::MAX, u64::MAX - 1, 1 << 62] {
+        server.shared.fake_total.store(total, Ordering::SeqCst);
+        let url = server.url("/h.h5");
+        let storage = storage_for_url(&url, &quick()).unwrap();
+        assert_eq!(clawhdf5_format::storage::Storage::len(&*storage), total);
+        for (off, n) in [(total - 100, 50), (total - 10, 100), (total - 1, 1)] {
+            let got = clawhdf5_format::storage::Storage::read_at(&*storage, off, n).unwrap();
+            assert_eq!(got.len() as u64, (total - off).min(n as u64));
+            assert!(got.iter().all(|&b| b == 0));
+        }
+        if let Ok(f) = File::open_storage(storage) {
+            let _ = transcript(&f);
+        }
+        let _ = open_url_with(&url, &quick()).map(|f| transcript(&f));
+    }
+}
+
+/// `download` reads a whole file in bounded steps, and refuses a claimed
+/// length beyond its limit before reading anything.
+#[test]
+fn download_is_bounded_by_its_limit_not_the_claimed_length() {
+    let bytes = multi_block_file();
+    let server = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    let url = server.url("/m.h5");
+    let storage = storage_for_url(&url, &quick()).unwrap();
+    let got = clawhdf5_remote::download(&*storage, clawhdf5_remote::DEFAULT_MAX_DOWNLOAD).unwrap();
+    assert_eq!(got, bytes);
+    let e = clawhdf5_remote::download(&*storage, 1000).unwrap_err();
+    assert!(
+        matches!(e, Error::Remote(RemoteError::TooLarge { limit: 1000, .. })),
+        "{e}"
+    );
+    server.shared.fake_total.store(1 << 62, Ordering::SeqCst);
+    let storage = storage_for_url(&url, &quick()).unwrap();
+    server.reset();
+    let e =
+        clawhdf5_remote::download(&*storage, clawhdf5_remote::DEFAULT_MAX_DOWNLOAD).unwrap_err();
+    assert!(
+        matches!(e, Error::Remote(RemoteError::TooLarge { len, .. }) if len == 1 << 62),
+        "{e}"
+    );
+    assert_eq!(server.requests(), 0, "refused before reading");
+}
+
+/// A URL's credentials — userinfo, and the query string of a presigned
+/// URL — never appear in an error message or a `Debug` output, whatever
+/// failed.
+#[test]
+fn credentials_never_appear_in_errors_or_debug() {
+    const SECRETS: [&str; 4] = ["hunter2", "SECRETSIG", "AKIDSECRET", "user:"];
+    fn clean(what: &str, text: &str) {
+        for s in SECRETS {
+            assert!(!text.contains(s), "{what}: {s} leaked in {text}");
+        }
+    }
+    fn check_err<T>(what: &str, r: Result<T, Error>) {
+        let Err(e) = r else {
+            panic!("{what}: expected an error")
+        };
+        clean(what, &format!("{e}"));
+        clean(what, &format!("{e:?}"));
+    }
+    let bytes = multi_block_file();
+    let server = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    let secret_url = |path: &str| {
+        format!(
+            "http://user:hunter2@{}{path}?X-Amz-Credential=AKIDSECRET&X-Amz-Signature=SECRETSIG",
+            server.addr
+        )
+    };
+    let url = secret_url("/m.h5");
+    let mut opts = quick();
+    opts.http.retries = 0;
+
+    // Opening works through such a URL, and Debug shows it redacted.
+    let (http, _) = HttpStorage::open(&url, opts.http.clone()).unwrap();
+    let debug = format!("{http:?}");
+    clean("Debug", &debug);
+    assert!(debug.contains("X-Amz-Signature=REDACTED"), "{debug}");
+    assert_eq!(http.url(), url, "the URL itself is kept for requests");
+
+    check_err("404", open_url_with(&secret_url("/missing.h5"), &opts));
+    server.shared.force_status.store(403, Ordering::SeqCst);
+    check_err("403", open_url_with(&url, &opts));
+    server.shared.force_status.store(0, Ordering::SeqCst);
+    server.shared.wrong_range.store(true, Ordering::SeqCst);
+    check_err("bad range at open", open_url_with(&url, &opts));
+    server.shared.wrong_range.store(false, Ordering::SeqCst);
+    server.shared.ignore_range.store(true, Ordering::SeqCst);
+    check_err("no range support", open_url_with(&url, &opts));
+    server.shared.ignore_range.store(false, Ordering::SeqCst);
+    server.shared.gzip_label.store(true, Ordering::SeqCst);
+    check_err("encoded body", open_url_with(&url, &opts));
+    server.shared.gzip_label.store(false, Ordering::SeqCst);
+
+    // Errors of reads after the open, through a File.
+    let f = open_url_with(&url, &opts).unwrap();
+    server.shared.wrong_range.store(true, Ordering::SeqCst);
+    check_err(
+        "bad range",
+        f.dataset("big")
+            .map(|d| d.read_f64())
+            .and_then(|r| r)
+            .map_err(Error::Hdf5),
+    );
+    server.shared.wrong_range.store(false, Ordering::SeqCst);
+    let f = open_url_with(&url, &opts).unwrap();
+    server.shared.force_status.store(403, Ordering::SeqCst);
+    check_err(
+        "403 on a read",
+        f.dataset("big")
+            .map(|d| d.read_f64())
+            .and_then(|r| r)
+            .map_err(Error::Hdf5),
+    );
+    server.shared.force_status.store(0, Ordering::SeqCst);
+    let f = open_url_with(&url, &opts).unwrap();
+    let mut other = bytes.clone();
+    let n = other.len();
+    other[n / 2] ^= 0xff;
+    server.shared.put("/m.h5", other);
+    check_err(
+        "ETag change",
+        f.dataset("big")
+            .map(|d| d.read_f64())
+            .and_then(|r| r)
+            .map_err(Error::Hdf5),
+    );
+    // A timeout.
+    let mut slow = opts.clone();
+    slow.http.timeout = Duration::from_millis(100);
+    server.shared.delay_ms.store(1000, Ordering::SeqCst);
+    check_err("timeout", open_url_with(&url, &slow));
+    server.shared.delay_ms.store(0, Ordering::SeqCst);
+    // A connection closed unanswered, a bad scheme.
+    let port = {
+        let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
+        let port = l.local_addr().unwrap().port();
+        std::thread::spawn(move || {
+            for c in l.incoming() {
+                drop(c);
+            }
+        });
+        port
+    };
+    check_err(
+        "connection closed",
+        open_url_with(
+            &format!("http://user:[email protected]:{port}/a.h5?X-Amz-Signature=SECRETSIG"),
+            &opts,
+        ),
+    );
+    check_err(
+        "scheme",
+        open_url("ftp://user:[email protected]/a.h5?X-Amz-Signature=SECRETSIG"),
+    );
+    assert_eq!(
+        clawhdf5_remote::redact_url("https://me:pw@host:8/d/f.h5?X-Amz-Signature=abc&a=1#frag"),
+        "https://host:8/d/f.h5?X-Amz-Signature=REDACTED&a=REDACTED"
+    );
+}
+
+/// Redirects are followed within limits: custom credential headers reach
+/// only the URL's own origin (not another port a redirect leads to), a
+/// same-origin redirect keeps them, loops end at `max_redirects`, and
+/// `max_redirects = 0` refuses any redirect. (The https→http downgrade
+/// refusal is a unit test of `redirect_target`: no TLS server here.)
+#[test]
+fn redirects_are_followed_safely() {
+    let bytes = multi_block_file();
+    let local = File::from_bytes(bytes.clone()).unwrap();
+    let want = local.dataset("big").unwrap().read_f64().unwrap();
+    let target = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    let front = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    front.redirect("/r.h5", &target.url("/m.h5"));
+    front.redirect("/s.h5", "/m.h5");
+    front.redirect("/loop.h5", "/loop.h5");
+    let mut opts = quick();
+    opts.http.headers = vec![
+        ("X-Api-Key".into(), "sekrit".into()),
+        ("Authorization".into(), "Bearer tok".into()),
+    ];
+
+    // Cross-origin (another port): followed, the headers stay behind.
+    let f = open_url_with(&front.url("/r.h5"), &opts).unwrap();
+    assert_eq!(f.dataset("big").unwrap().read_f64().unwrap(), want);
+    assert!(front.saw_header("x-api-key"), "sent to the URL's origin");
+    assert!(target.requests() > 1);
+    assert!(
+        !target.saw_header("x-api-key") && !target.saw_header("authorization"),
+        "credential headers forwarded across origins: {:?}",
+        target.shared.seen.lock().unwrap()
+    );
+
+    // Same origin: followed with the headers.
+    front.reset();
+    let f = open_url_with(&front.url("/s.h5"), &opts).unwrap();
+    assert_eq!(f.dataset("big").unwrap().read_f64().unwrap(), want);
+    let seen = front.shared.seen.lock().unwrap().clone();
+    assert!(
+        seen.iter()
+            .filter(|(p, _)| p == "/m.h5")
+            .all(|(_, h)| h.get("x-api-key").map(String::as_str) == Some("sekrit"))
+    );
+
+    // A loop ends after max_redirects (5 by default): 6 requests.
+    front.reset();
+    let e = open_url_with(&front.url("/loop.h5"), &opts).unwrap_err();
+    assert!(matches!(e, Error::Remote(RemoteError::Redirect(_))), "{e}");
+    assert_eq!(front.requests(), 6);
+
+    // No redirects allowed.
+    opts.http.max_redirects = 0;
+    let e = open_url_with(&front.url("/s.h5"), &opts).unwrap_err();
+    assert!(matches!(e, Error::Remote(RemoteError::Redirect(_))), "{e}");
+}
+
+/// A server may answer the first request (bytes 0 to 1 MiB - 1) with 200
+/// when that covers the whole file: a body no longer than the range asked
+/// for is accepted as the whole file, in one request.
+#[test]
+fn a_200_covering_the_requested_range_is_the_whole_file() {
+    let root = std::path::Path::new(env!("CARGO_MANIFEST_DIR"));
+    let tall = root.join("../clawhdf5/tests/fixtures/tall.h5");
+    let bytes = std::fs::read(&tall).unwrap();
+    assert!(bytes.len() < 1 << 20);
+    let server = Server::start(vec![("/t.h5".into(), bytes.clone())]);
+    server.shared.ignore_range.store(true, Ordering::SeqCst);
+    let storage = storage_for_url(&server.url("/t.h5"), &quick()).unwrap();
+    let f = File::open_storage(storage).unwrap();
+    assert_eq!(f.contiguous_bytes(), Some(&bytes[..]));
+    assert_eq!(transcript(&f), transcript(&File::open(&tall).unwrap()));
+    assert_eq!(server.requests(), 1);
+    // Larger than the range asked for: still refused.
+    let mut opts = quick().http;
+    opts.first_request = 4096;
+    let e = HttpStorage::open(&server.url("/t.h5"), opts).unwrap_err();
+    assert!(matches!(e, RemoteError::RangeNotSupported(_)), "{e}");
+}
+
+/// A slow link is not cut off: the body's time budget grows with its size
+/// (`timeout` + size at `min_speed`), so a 256 KiB block at 256 KiB/s
+/// (1 s) reads with a 300 ms `timeout`. A stalled body still fails, soon.
+#[test]
+fn slow_links_read_and_stalled_ones_fail() {
+    let bytes = multi_block_file();
+    let server = Server::start(vec![("/m.h5".into(), bytes.clone())]);
+    let url = server.url("/m.h5");
+    let mut opts = quick();
+    opts.cache.block_size = 256 << 10;
+    opts.cache.max_request = 256 << 10;
+    opts.http.timeout = Duration::from_millis(300);
+    server
+        .shared
+        .throttle_bps
+        .store(256 << 10, Ordering::SeqCst);
+    let f = open_url_with(&url, &opts).unwrap();
+    assert_eq!(f.root().groups().unwrap(), ["grp"]);
+    assert_eq!(
+        f.dataset("grp/small").unwrap().read_f64().unwrap(),
+        [1.0, 2.0, 3.0]
+    );
+    server.shared.throttle_bps.store(0, Ordering::SeqCst);
+
+    // Stalled mid-body for 20 s: a timeout well before that.
+    server.shared.stall_ms.store(20_000, Ordering::SeqCst);
+    opts.http.retries = 0;
+    opts.http.min_speed = 64 << 20;
+    let t = std::time::Instant::now();
+    let e = open_url_with(&url, &opts).unwrap_err();
+    assert!(matches!(e, Error::Remote(RemoteError::Transport(_))), "{e}");
+    assert!(t.elapsed() < Duration::from_secs(5), "{:?}", t.elapsed());
+}
+
+/// `cached` reports a failed first fetch as a remote error, not as an
+/// HDF5 format error.
+#[test]
+fn cached_reports_a_failed_prefetch_as_remote() {
+    use clawhdf5_format::error::FormatError;
+    use std::borrow::Cow;
+    struct Down;
+    impl clawhdf5_format::storage::Storage for Down {
+        fn read_at(&self, _: u64, _: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+            Err(RemoteError::Transport("connection refused".into()).into())
+        }
+        fn len(&self) -> u64 {
+            1 << 20
+        }
+    }
+    let e = clawhdf5_remote::cached(Box::new(Down), &Options::default())
+        .map(|_| ())
+        .unwrap_err();
+    assert!(
+        matches!(&e, Error::Remote(RemoteError::Backend(m)) if m.contains("connection refused")),
+        "{e:?}"
+    );
+}

crates/clawhdf5-remote/tests/object_store.rs

@@ -0,0 +1,159 @@
+//! `ObjectStoreStorage` against object_store's in-memory and local-file
+//! backends (no cloud needed): values equal `File::open`'s, the block cache
+//! coalesces, and an object replaced while open is an error.
+
+#![cfg(feature = "object-store")]
+
+mod common;
+
+use std::sync::Arc;
+
+use clawhdf5::File;
+use clawhdf5_format::storage::Storage;
+use clawhdf5_remote::object_store::memory::InMemory;
+use clawhdf5_remote::object_store::path::Path as ObjectPath;
+use clawhdf5_remote::object_store::{ObjectStore, ObjectStoreExt, PutPayload};
+use clawhdf5_remote::{ObjectStoreStorage, Options, open_object};
+use common::{multi_block_file, transcript};
+
+fn put(store: &dyn ObjectStore, path: &str, bytes: Vec<u8>) {
+    let rt = tokio_rt();
+    rt.block_on(store.put(&ObjectPath::from(path), PutPayload::from(bytes)))
+        .unwrap();
+}
+
+fn tokio_rt() -> tokio::runtime::Runtime {
+    tokio::runtime::Builder::new_current_thread()
+        .enable_all()
+        .build()
+        .unwrap()
+}
+
+#[test]
+fn in_memory_store_reads_like_file_open() {
+    let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+    let mut compared = 0;
+    for (i, p) in common::fixtures().iter().enumerate() {
+        let bytes = std::fs::read(p).unwrap();
+        let key = format!("f{i}.h5");
+        put(store.as_ref(), &key, bytes);
+        let (Ok(local), Ok((mut remote, cache))) = (
+            File::open(p),
+            open_object(store.clone(), &key, &Options::default()),
+        ) else {
+            continue;
+        };
+        remote.set_vds_resolver(common::sibling_resolver(p.parent().unwrap().into()));
+        assert!(remote.contiguous_bytes().is_none());
+        assert_eq!(transcript(&remote), transcript(&local), "{}", p.display());
+        assert!(cache.stats().reads > 0);
+        compared += 1;
+    }
+    assert!(compared >= 40, "{compared}");
+}
+
+#[test]
+fn multi_block_object_is_fetched_in_coalesced_blocks() {
+    let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+    let bytes = multi_block_file();
+    put(store.as_ref(), "m.h5", bytes.clone());
+    let (remote, cache) = open_object(store, "m.h5", &Options::default()).unwrap();
+    let local = File::from_bytes(bytes.clone()).unwrap();
+    assert_eq!(
+        remote.dataset("big").unwrap().read_f64().unwrap(),
+        local.dataset("big").unwrap().read_f64().unwrap()
+    );
+    let s = cache.stats();
+    let blocks = (bytes.len() as u64).div_ceil(1 << 20);
+    assert!(s.bytes_fetched <= bytes.len() as u64);
+    assert!(s.requests <= blocks, "{s:?}");
+    assert_eq!(cache.inner().len(), bytes.len() as u64);
+}
+
+#[test]
+fn local_file_store_reads_like_file_open() {
+    let dir = tempfile::tempdir().unwrap();
+    let bytes = multi_block_file();
+    std::fs::write(dir.path().join("m.h5"), &bytes).unwrap();
+    let store: Arc<dyn ObjectStore> = Arc::new(
+        clawhdf5_remote::object_store::local::LocalFileSystem::new_with_prefix(dir.path()).unwrap(),
+    );
+    let (remote, _) = open_object(store, "m.h5", &Options::default()).unwrap();
+    let local = File::from_bytes(bytes).unwrap();
+    assert_eq!(transcript(&remote), transcript(&local));
+}
+
+#[test]
+fn an_object_replaced_while_open_is_an_error() {
+    let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+    let bytes = multi_block_file();
+    put(store.as_ref(), "m.h5", bytes.clone());
+    let (remote, _) = open_object(store.clone(), "m.h5", &Options::default()).unwrap();
+    assert_eq!(remote.root().groups().unwrap(), ["grp"]);
+    let mut other = bytes;
+    let n = other.len();
+    other[n / 2] ^= 0xff;
+    put(store.as_ref(), "m.h5", other);
+    let err = remote
+        .dataset("big")
+        .unwrap()
+        .read_f64()
+        .unwrap_err()
+        .to_string();
+    assert!(err.contains("changed while open"), "{err}");
+}
+
+#[test]
+fn missing_objects_and_async_callers_are_clean_errors() {
+    let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+    assert!(ObjectStoreStorage::new(store.clone(), ObjectPath::from("nope.h5")).is_err());
+    put(store.as_ref(), "m.h5", multi_block_file());
+    let storage = ObjectStoreStorage::new(store, ObjectPath::from("m.h5")).unwrap();
+    // From inside a current-thread runtime: works (the read runs on the
+    // storage's own runtime), no panic, no deadlock.
+    let rt = tokio_rt();
+    let r = rt.block_on(async { storage.read_at(0, 10).map(|b| b.len()) });
+    assert_eq!(r.unwrap(), 10);
+    // Dropping the storage inside a runtime does not panic.
+    rt.block_on(async move { drop(storage) });
+}
+
+/// The advice for async callers works: a read in `spawn_blocking` of a
+/// multi-threaded runtime (where `Handle::try_current` is Ok), and a read
+/// straight inside a current-thread runtime's task.
+#[test]
+fn object_stores_read_from_spawn_blocking_and_inside_runtimes() {
+    let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
+    let bytes = multi_block_file();
+    put(store.as_ref(), "m.h5", bytes.clone());
+    let want = File::from_bytes(bytes)
+        .unwrap()
+        .dataset("big")
+        .unwrap()
+        .read_f64()
+        .unwrap();
+    let rt = tokio::runtime::Builder::new_multi_thread()
+        .worker_threads(2)
+        .build()
+        .unwrap();
+    let (s, w) = (store.clone(), want.clone());
+    let got = rt
+        .block_on(async move {
+            tokio::task::spawn_blocking(move || {
+                assert!(tokio::runtime::Handle::try_current().is_ok());
+                let (f, _) =
+                    open_object(s, "m.h5", &Options::default()).map_err(|e| e.to_string())?;
+                let v = f.dataset("big").unwrap().read_f64().unwrap();
+                Ok::<_, String>(v == w)
+            })
+            .await
+        })
+        .unwrap();
+    assert_eq!(got, Ok(true));
+    let rt = tokio_rt();
+    let got = rt.block_on(async {
+        let (f, _) = open_object(store, "m.h5", &Options::default()).unwrap();
+        f.dataset("big").unwrap().read_f64().unwrap()
+    });
+    assert_eq!(got, want);
+}

crates/clawhdf5-tools/Cargo.toml

@@ -14,9 +14,16 @@ readme = "README.md"
 name = "h5rs"
 path = "src/main.rs"
 
+[features]
+# FILE arguments may be URLs: http:// with `remote` (no C), https:// with
+# `remote-https` (rustls + ring, which compiles C).
+remote = ["dep:clawhdf5-remote"]
+remote-https = ["remote", "clawhdf5-remote/https"]
+
 [dependencies]
 clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
 clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5-remote = { path = "../clawhdf5-remote", version = "2.7.0", optional = true }
 serde_json = "1"
 
 [dev-dependencies]

crates/clawhdf5-tools/README.md

@@ -24,6 +24,29 @@ Every command takes `--max-bytes N` where it reads values (default 1 GiB): a
 dataset whose dataspace claims more than that is reported instead of read, so
 a corrupt size cannot exhaust memory.
 
+### Remote files
+
+Built with the `remote` feature, every FILE argument may be an `http://`
+URL (`remote-https` adds `https://`, through rustls and ring, which compiles
+C; the default build has neither). The file is read by range requests
+through [clawhdf5-remote](../clawhdf5-remote/README.md)'s block cache, so
+`ls` of a large file fetches its metadata blocks, not the file:
+
+```bash
+cargo install --path crates/clawhdf5-tools --features remote
+h5rs ls -r -v http://127.0.0.1:8000/file.h5
+h5rs dump http://127.0.0.1:8000/file.h5
+h5rs diff local.h5 http://127.0.0.1:8000/file.h5
+```
+
+A URL names the whole file (`FILE/OBJECT` suffixes are for local paths).
+`check` validates every byte, so it downloads a remote file whole first —
+up to `--max-download N` bytes (default 1 GiB), refusing a longer file
+before reading any of it. URLs are printed without their credentials
+(userinfo, query string values).
+The output is the local file's (`tests/remote.rs` compares every
+subcommand).
+
 ## `h5rs ls`
 
 ```console

crates/clawhdf5-tools/src/check.rs

@@ -28,7 +28,7 @@ use crate::h5::{Error, ErrorKind, H5, Kind};
 use crate::info::{self, DsInfo};
 
 pub const USAGE: &str = "\
-usage: h5rs check [--data] [-q] [--max-bytes N] FILE
+usage: h5rs check [--data] [-q] [--max-bytes N] [--max-download N] FILE
 
 Validate FILE's structure: walk every object from the root group, parse
 every header message, verify the checksums of version 2+ structures
@@ -45,6 +45,9 @@ problem is printed with the address of the structure involved.
                   datasets and attributes into its global heap collection
   -q, --quiet     print only the problems, not the summary
   --max-bytes N   largest dataset read by --data (default 1 GiB)
+  --max-download N
+                  largest remote (URL) file downloaded to check it
+                  (default 1 GiB)
 
 Exit status: 0 no problems, 1 problems found, 2 usage error or file not
 found, 3 internal error.";
@@ -124,6 +127,7 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
     let mut read_data = false;
     let mut quiet = false;
     let mut max_bytes = None;
+    let mut max_download = 1u64 << 30;
     let mut file = None;
     while let Some(a) = args.next() {
         match a.as_str() {
@@ -133,6 +137,10 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
                 Some(n) => max_bytes = Some(n),
                 None => return args.usage_error(out, "--max-bytes needs a number", USAGE),
             },
+            "--max-download" => match args.number() {
+                Some(n) => max_download = n,
+                None => return args.usage_error(out, "--max-download needs a number", USAGE),
+            },
             "-h" | "--help" => {
                 writeln!(out.o, "{USAGE}")?;
                 return Ok(0);
@@ -148,13 +156,24 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
         return args.usage_error(out, "missing FILE", USAGE);
     };
     let path = std::path::Path::new(&file);
-    if !path.is_file() {
+    let mut h5 = if crate::h5::is_url(&file) {
-        writeln!(out.e, "h5rs check: {file}: no such file")?;
+        // check validates every byte, so a remote file is downloaded whole.
-        return Ok(2);
+        match H5::open_arg_whole(&file, max_download) {
-    }
+            Ok(h) => h,
-    let mut h5 = match H5::open(path) {
+            Err(e) => {
-        Ok(h) => h,
+                writeln!(out.e, "h5rs check: {e}")?;
-        Err(_) => return unopenable(path, out),
+                return Ok(2);
+            }
+        }
+    } else {
+        if !path.is_file() {
+            writeln!(out.e, "h5rs check: {file}: no such file")?;
+            return Ok(2);
+        }
+        match H5::open(path) {
+            Ok(h) => h,
+            Err(_) => return unopenable(path, out),
+        }
     };
     if let Some(m) = max_bytes {
         h5.max_bytes = m;
@@ -185,7 +204,7 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
         }
     }
     if !quiet {
-        c.summary(&file, out)?;
+        c.summary(&crate::h5::shown(&file), out)?;
     }
     Ok(if c.panicked {
         3

crates/clawhdf5-tools/src/diff.rs

@@ -174,7 +174,7 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
     }
     let mut files = Vec::new();
     for f in &pos[..2] {
-        match H5::open(std::path::Path::new(f)) {
+        match H5::open_arg(f) {
             Ok(mut h) => {
                 if let Some(m) = max_bytes {
                     h.max_bytes = m;
@@ -196,7 +196,8 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
             Err(_) => {
                 writeln!(
                     out.e,
-                    "h5rs diff: object <{obj}> could not be found in <{f}>"
+                    "h5rs diff: object <{obj}> could not be found in <{}>",
+                    crate::h5::shown(f)
                 )?;
                 return Ok(2);
             }
@@ -214,7 +215,7 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
         let entries = match collected {
             Ok(e) => e,
             Err(e) => {
-                writeln!(out.e, "h5rs diff: {f}: {e}")?;
+                writeln!(out.e, "h5rs diff: {}: {e}", crate::h5::shown(f))?;
                 return Ok(2);
             }
         };

crates/clawhdf5-tools/src/dump.rs

@@ -82,7 +82,7 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
     let Some(file) = file else {
         return args.usage_error(out, "missing FILE", USAGE);
     };
-    let mut h5 = match H5::open(std::path::Path::new(&file)) {
+    let mut h5 = match H5::open_arg(&file) {
         Ok(h) => h,
         Err(e) => {
             writeln!(out.e, "h5rs dump: {e}")?;
@@ -98,6 +98,7 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
         problems: 0,
         paths: OnceCell::new(),
     };
+    let file = crate::h5::shown(&file);
     let fname = std::path::Path::new(&file)
         .file_name()
         .map(|s| s.to_string_lossy().into_owned())

crates/clawhdf5-tools/src/h5.rs

@@ -10,9 +10,9 @@ use std::collections::HashMap;
 use std::path::{Path, PathBuf};
 
 use clawhdf5::File;
-use clawhdf5_format::attribute::{AttributeMessage, extract_attributes_tolerant};
+use clawhdf5_format::attribute::{AttributeMessage, extract_attributes_tolerant_in};
 use clawhdf5_format::attribute_info::AttributeInfoMessage;
-use clawhdf5_format::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+use clawhdf5_format::btree_v2::{BTreeV2Header, collect_btree_v2_records_in};
 use clawhdf5_format::data_layout::DataLayout;
 use clawhdf5_format::dataspace::{Dataspace, DataspaceType};
 use clawhdf5_format::datatype::Datatype;
@@ -24,6 +24,7 @@ use clawhdf5_format::link_info::LinkInfoMessage;
 use clawhdf5_format::link_message::{LinkMessage, LinkTarget};
 use clawhdf5_format::message_type::MessageType;
 use clawhdf5_format::object_header::ObjectHeader;
+use clawhdf5_format::storage::Storage;
 use clawhdf5_format::superblock::Superblock;
 use clawhdf5_format::symbol_table::SymbolTableMessage;
 
@@ -186,8 +187,11 @@ pub struct Link {
 
 /// An open file.
 pub struct H5 {
+    /// The file's path, or its URL for a remote file.
     pub path: PathBuf,
     pub file: File,
+    /// Size of the whole file in bytes (user block included).
+    pub size: u64,
     pub max_bytes: u64,
     /// Fractal heaps whose blocks were verified: `None` = sound.
     verified_heaps: RefCell<HashMap<u64, Option<Error>>>,
@@ -204,19 +208,96 @@ impl H5 {
                 path.display()
             ))
         })?;
-        Ok(H5 {
+        let size = std::fs::metadata(path).map(|m| m.len()).unwrap_or(0);
-            path: path.to_path_buf(),
+        Ok(H5::new(path.to_path_buf(), file, size))
+    }
+
+    fn new(path: PathBuf, file: File, size: u64) -> H5 {
+        H5 {
+            path,
             file,
+            size,
             max_bytes: DEFAULT_MAX_BYTES,
             verified_heaps: RefCell::new(HashMap::new()),
-        })
+        }
+    }
+
+    /// Open a command-line FILE argument: a path, or with the `remote`
+    /// feature an `http(s)://` (or `s3://`, `gs://`, `az://`) URL, read by
+    /// range requests through a block cache.
+    pub fn open_arg(arg: &str) -> Result<H5> {
+        if !is_url(arg) {
+            return H5::open(Path::new(arg));
+        }
+        let name = shown(arg);
+        #[cfg(feature = "remote")]
+        {
+            let storage =
+                clawhdf5_remote::storage_for_url(arg, &clawhdf5_remote::Options::default())
+                    .map_err(|e| Error::new(format!("{name}: {e}")))?;
+            let size = storage.len();
+            let file = File::open_storage(storage).map_err(|e| {
+                Error::new(format!("{name}: not an HDF5 file this tool can open: {e}"))
+            })?;
+            Ok(H5::new(PathBuf::from(&name), file, size))
+        }
+        #[cfg(not(feature = "remote"))]
+        Err(Error::new(format!(
+            "{name}: URLs need h5rs built with the `remote` feature"
+        )))
+    }
+
+    /// [`H5::open_arg`], with a remote file downloaded whole into memory
+    /// first — for `check`, which validates every byte of the file anyway
+    /// and parses it as one slice ([`H5::data`]). A remote file longer
+    /// than `max_download` bytes is refused before anything is read: its
+    /// length is only what the server claims.
+    pub fn open_arg_whole(arg: &str, max_download: u64) -> Result<H5> {
+        if !is_url(arg) {
+            return H5::open_arg(arg);
+        }
+        let name = shown(arg);
+        #[cfg(feature = "remote")]
+        {
+            // One open (one probe of the server); the download then reads
+            // through the same cache, the first block already in it.
+            let storage =
+                clawhdf5_remote::storage_for_url(arg, &clawhdf5_remote::Options::default())
+                    .map_err(|e| Error::new(format!("{name}: {e}")))?;
+            let bytes = clawhdf5_remote::download(&*storage, max_download)
+                .map_err(|e| Error::new(format!("{name}: {e}")))?;
+            let size = bytes.len() as u64;
+            let file = File::from_bytes(bytes).map_err(|e| {
+                Error::new(format!("{name}: not an HDF5 file this tool can open: {e}"))
+            })?;
+            Ok(H5::new(PathBuf::from(&name), file, size))
+        }
+        #[cfg(not(feature = "remote"))]
+        {
+            let _ = max_download;
+            Err(Error::new(format!(
+                "{name}: URLs need h5rs built with the `remote` feature"
+            )))
+        }
     }
 
     /// The file's bytes from the superblock on: what every address indexes.
+    ///
+    /// # Panics
+    ///
+    /// For a remote file opened by [`H5::open_arg`]; read it through
+    /// [`H5::store`], or open it with [`H5::open_arg_whole`].
     pub fn data(&self) -> &[u8] {
         self.file.as_bytes()
     }
 
+    /// The same bytes as a [`Storage`], for local and remote files alike:
+    /// in memory a read is a slice of [`H5::data`], remote it is served by
+    /// the block cache.
+    pub fn store(&self) -> &dyn Storage {
+        self.file.storage()
+    }
+
     pub fn sb(&self) -> &Superblock {
         self.file.superblock()
     }
@@ -239,8 +320,8 @@ impl H5 {
         if let Some(e) = self.file.cache_image_error() {
             return Err(Error::at(addr, format!("metadata cache image: {e}")));
         }
-        let off = usize::try_from(addr).map_err(|_| Error::at(addr, "address out of range"))?;
+        to_usize(addr)?;
-        ObjectHeader::parse(self.data(), off, self.os(), self.ls())
+        ObjectHeader::parse_in(self.store(), addr, self.os(), self.ls())
             .map_err(|e| Error::at(addr, format!("object header: {e}")))
     }
 
@@ -249,11 +330,14 @@ impl H5 {
     pub fn payload(&self, h: &ObjectHeader, t: MessageType) -> Result<Option<Vec<u8>>> {
         match h.messages.iter().find(|m| m.msg_type == t) {
             None => Ok(None),
-            Some(m) => {
+            Some(m) => clawhdf5_format::shared_message::message_data_in(
-                clawhdf5_format::shared_message::message_data(self.data(), m, self.os(), self.ls())
+                self.store(),
-                    .map(|c| Some(c.into_owned()))
+                m,
-                    .map_err(|e| Error::new(format!("{t:?} message: {e}")))
+                self.os(),
-            }
+                self.ls(),
+            )
+            .map(|c| Some(c.into_owned()))
+            .map_err(|e| Error::new(format!("{t:?} message: {e}"))),
         }
     }
 
@@ -305,8 +389,9 @@ impl H5 {
             self.verified_heap(fh)
                 .map_err(|e| e.context("dense attribute storage"))?;
         }
-        let (mut attrs, errs) = extract_attributes_tolerant(self.data(), h, self.os(), self.ls())
+        let (mut attrs, errs) =
-            .map_err(|e| Error::new(format!("attributes: {e}")))?;
+            extract_attributes_tolerant_in(self.store(), h, self.os(), self.ls())
+                .map_err(|e| Error::new(format!("attributes: {e}")))?;
         attrs.sort_by(|a, b| a.name.cmp(&b.name));
         Ok((attrs, errs.iter().map(|e| e.to_string()).collect()))
     }
@@ -316,7 +401,7 @@ impl H5 {
     pub fn links(&self, h: &ObjectHeader) -> Result<Vec<Link>> {
         let os = self.os();
         let ls = self.ls();
-        let data = self.data();
+        let data = self.store();
         let mut out = Vec::new();
         if let Some(m) = h
             .messages
@@ -325,7 +410,7 @@ impl H5 {
         {
             let stm = SymbolTableMessage::parse(&m.data, os)
                 .map_err(|e| Error::new(format!("symbol table message: {e}")))?;
-            let entries = group_v1::resolve_v1_group_entries(data, &stm, os, ls)
+            let entries = group_v1::resolve_v1_group_entries_in(data, &stm, os, ls)
                 .map_err(|e| Error::at(stm.btree_address, format!("symbol table: {e}")))?;
             let has_soft = entries.iter().any(group_v1::is_v1_soft_link);
             for e in entries {
@@ -337,7 +422,7 @@ impl H5 {
                 }
             }
             if has_soft {
-                let soft = group_v1::v1_soft_links(data, &stm, os, ls)
+                let soft = group_v1::v1_soft_links_in(data, &stm, os, ls)
                     .map_err(|e| Error::at(stm.btree_address, format!("soft links: {e}")))?;
                 for (name, target) in soft {
                     out.push(Link {
@@ -387,15 +472,15 @@ impl H5 {
         name_type: u8,
     ) -> Result<Vec<Vec<u8>>> {
         self.verified_heap(heap)?;
-        let data = self.data();
+        let data = self.store();
         let os = self.os();
         let ls = self.ls();
-        let fh = FractalHeapHeader::parse(data, to_usize(heap)?, os, ls)
+        let fh = FractalHeapHeader::parse_in(data, to_usize(heap).map(|_| heap)?, os, ls)
             .map_err(|e| Error::at(heap, format!("fractal heap header: {e}")))?;
         let bt = btree.ok_or_else(|| Error::at(heap, "dense storage without a name index"))?;
-        let hdr = BTreeV2Header::parse(data, to_usize(bt)?, os, ls)
+        let hdr = BTreeV2Header::parse_in(data, to_usize(bt).map(|_| bt)?, os, ls)
             .map_err(|e| Error::at(bt, format!("v2 B-tree header: {e}")))?;
-        let recs = collect_btree_v2_records(data, &hdr, os, ls)
+        let recs = collect_btree_v2_records_in(data, &hdr, os, ls)
             .map_err(|e| Error::at(bt, format!("v2 B-tree: {e}")))?;
         // Name-index records: hash(4) + heap ID; creation-order ones: order(8) + heap ID.
         let skip = if hdr.tree_type == name_type { 4 } else { 8 };
@@ -407,7 +492,7 @@ impl H5 {
                 .get(skip..skip + idlen)
                 .ok_or_else(|| Error::at(bt, "v2 B-tree record shorter than a heap ID"))?;
             let obj = fh
-                .read_managed_object(data, id, os)
+                .read_managed_object_in(data, id, os)
                 .map_err(|e| Error::at(heap, format!("fractal heap object: {e}")))?;
             out.push(obj);
         }
@@ -561,7 +646,7 @@ impl H5 {
         if p.is_empty() {
             return Ok(self.root());
         }
-        clawhdf5_format::group_v2::resolve_path_any(self.data(), self.sb(), p)
+        clawhdf5_format::group_v2::resolve_path_any_in(self.store(), self.sb(), p)
             .map_err(|e| Error::new(format!("{path}: {e}")))
     }
 }
@@ -681,7 +766,9 @@ pub fn byte_len(ds: &Dataspace, dt: &Datatype) -> Result<u64> {
 /// Split a `FILE[/object/path]` argument the way h5ls does: the longest
 /// prefix that is an existing file is the file.
 pub fn split_file_arg(arg: &str) -> (String, Option<String>) {
-    if Path::new(arg).is_file() {
+    // A URL names the file only (its path cannot be split against the
+    // local file system).
+    if is_url(arg) || Path::new(arg).is_file() {
         return (arg.to_string(), None);
     }
     let mut idx: Vec<usize> = arg.match_indices('/').map(|(i, _)| i).collect();
@@ -694,3 +781,35 @@ pub fn split_file_arg(arg: &str) -> (String, Option<String>) {
     }
     (arg.to_string(), None)
 }
+
+/// A FILE argument as it may be printed: a URL without its credentials
+/// (userinfo, query string values — a presigned URL's signature), a path
+/// as given.
+pub fn shown(arg: &str) -> String {
+    if !is_url(arg) {
+        return arg.to_string();
+    }
+    #[cfg(feature = "remote")]
+    {
+        clawhdf5_remote::redact_url(arg)
+    }
+    #[cfg(not(feature = "remote"))]
+    {
+        let (scheme, rest) = arg.split_once("://").unwrap_or(("", arg));
+        let rest = rest.split(['?', '#']).next().unwrap_or("");
+        let host_end = rest.find('/').unwrap_or(rest.len());
+        let (authority, path) = rest.split_at(host_end);
+        let host = authority.rsplit_once('@').map_or(authority, |(_, h)| h);
+        format!("{scheme}://{host}{path}")
+    }
+}
+
+/// Whether a FILE argument is a URL (`scheme://...`) rather than a path.
+pub fn is_url(arg: &str) -> bool {
+    arg.split_once("://").is_some_and(|(scheme, _)| {
+        !scheme.is_empty()
+            && scheme
+                .chars()
+                .all(|c| c.is_ascii_alphanumeric() || matches!(c, '+' | '-' | '.'))
+    })
+}

crates/clawhdf5-tools/src/heap_blocks.rs

@@ -4,10 +4,12 @@
 //! heap header's flag says so). The library reads only the blocks an object
 //! lives in and does not verify block checksums, so `check` does it here.
 
+use std::borrow::Cow;
 use std::collections::HashSet;
 
 use clawhdf5_format::checksum::jenkins_lookup3;
 use clawhdf5_format::fractal_heap::FractalHeapHeader;
+use clawhdf5_format::storage::{Storage, read_exact_at};
 
 use crate::h5::{Error, H5};
 
@@ -26,7 +28,7 @@ pub struct HeapReport {
 }
 
 struct Walk<'a> {
-    data: &'a [u8],
+    data: &'a dyn Storage,
     heap: u64,
     fh: FractalHeapHeader,
     checksum_dblocks: bool,
@@ -60,14 +62,14 @@ fn log2(v: u64) -> u32 {
 /// parsed (and its checksum verified) by the library; an error there is
 /// returned as the only problem.
 pub fn verify(h5: &H5, heap: u64) -> HeapReport {
-    let data = h5.data();
+    let data = h5.store();
     let Ok(off) = usize::try_from(heap) else {
         return HeapReport {
             problems: vec![Error::at(heap, "fractal heap address out of range")],
             ..Default::default()
         };
     };
-    let fh = match FractalHeapHeader::parse(data, off, h5.os(), h5.ls()) {
+    let fh = match FractalHeapHeader::parse_in(data, heap, h5.os(), h5.ls()) {
         Ok(f) => f,
         Err(e) => {
             return HeapReport {
@@ -77,7 +79,15 @@ pub fn verify(h5: &H5, heap: u64) -> HeapReport {
         }
     };
     // Flags: signature(4) version(1) heap ID length(2) filter length(2) flags(1).
-    let flags = data.get(off + 9).copied().unwrap_or(0);
+    let flags = match data.read_at(off as u64 + 9, 1) {
+        Ok(b) => b.first().copied().unwrap_or(0),
+        Err(e) => {
+            return HeapReport {
+                problems: vec![Error::at(heap, format!("fractal heap header: {e}"))],
+                ..Default::default()
+            };
+        }
+    };
     let mut w = Walk {
         data,
         heap,
@@ -107,11 +117,42 @@ pub fn verify(h5: &H5, heap: u64) -> HeapReport {
     w.r
 }
 
-impl Walk<'_> {
+impl<'a> Walk<'a> {
     fn problem(&mut self, addr: u64, msg: impl Into<String>) {
         self.r.problems.push(Error::at(addr, msg));
     }
 
+    /// Bytes `[start, end)` of the file: `Ok(None)` when they run past its
+    /// end (what a slice `get` of the whole file answered), `Err` when the
+    /// storage fails to read them (a remote file).
+    fn get(&self, start: usize, end: usize) -> Result<Option<Cow<'a, [u8]>>, String> {
+        let Some(len) = end.checked_sub(start) else {
+            return Ok(None);
+        };
+        if end as u64 > self.data.len() {
+            return Ok(None);
+        }
+        read_exact_at(self.data, start as u64, len)
+            .map(Some)
+            .map_err(|e| e.to_string())
+    }
+
+    /// [`Walk::get`], recording a read failure as a problem at `addr`.
+    fn get_or_note(
+        &mut self,
+        addr: u64,
+        start: usize,
+        end: usize,
+    ) -> Option<Option<Cow<'a, [u8]>>> {
+        match self.get(start, end) {
+            Ok(b) => Some(b),
+            Err(e) => {
+                self.problem(addr, format!("fractal heap block: {e}"));
+                None
+            }
+        }
+    }
+
     fn row_size(&self, row: usize) -> Option<u64> {
         let s = self.fh.starting_block_size;
         if row <= 1 {
@@ -155,10 +196,11 @@ impl Walk<'_> {
             return None;
         };
         let hdr_len = 5 + self.os + self.boff_bytes;
-        let Some(b) = start
+        let b = match start.checked_add(hdr_len) {
-            .checked_add(hdr_len)
+            Some(e) => self.get_or_note(addr, start, e)?,
-            .and_then(|e| self.data.get(start..e))
+            None => None,
-        else {
+        };
+        let Some(b) = b else {
             self.problem(
                 addr,
                 format!("fractal heap {what} block lies past the end of the file"),
@@ -209,7 +251,10 @@ impl Walk<'_> {
             self.problem(addr, "fractal heap direct block size out of range");
             return;
         };
-        let Some(block) = self.data.get(start..end) else {
+        let Some(block) = self.get_or_note(addr, start, end) else {
+            return;
+        };
+        let Some(block) = block else {
             self.problem(
                 addr,
                 "fractal heap direct block extends past the end of the file",
@@ -259,14 +304,17 @@ impl Walk<'_> {
             };
             let direct = row < direct_rows;
             for _ in 0..width {
-                let Some(b) = self.data.get(pos..pos + self.os) else {
+                let Some(b) = self.get_or_note(addr, pos, pos + self.os) else {
+                    return;
+                };
+                let Some(b) = b else {
                     self.problem(
                         addr,
                         "fractal heap indirect block extends past the end of the file",
                     );
                     return;
                 };
-                let child = le(b);
+                let child = le(&b);
                 pos += self.os;
                 if direct && filtered {
                     pos += self.ls + 4;
@@ -277,7 +325,10 @@ impl Walk<'_> {
                 off = off.saturating_add(rs);
             }
         }
-        let Some(stored) = self.data.get(pos..pos + 4) else {
+        let Some(stored) = self.get_or_note(addr, pos, pos + 4) else {
+            return;
+        };
+        let Some(stored) = stored else {
             self.problem(
                 addr,
                 "fractal heap indirect block extends past the end of the file",
@@ -285,7 +336,10 @@ impl Walk<'_> {
             return;
         };
         let stored = u32::from_le_bytes([stored[0], stored[1], stored[2], stored[3]]);
-        let computed = jenkins_lookup3(&self.data[start..pos]);
+        let Some(Some(body)) = self.get_or_note(addr, start, pos) else {
+            return;
+        };
+        let computed = jenkins_lookup3(&body);
         self.r.checksums += 1;
         if computed != stored {
             self.problem(

crates/clawhdf5-tools/src/info.rs

@@ -1,7 +1,7 @@
 //! Dataset facts shared by `ls`, `dump`, `stat` and `check`: shape text,
 //! layout, filters and storage.
 
-use clawhdf5_format::chunked_read::{ChunkInfo, list_chunks};
+use clawhdf5_format::chunked_read::{ChunkInfo, list_chunks_in};
 use clawhdf5_format::data_layout::DataLayout;
 use clawhdf5_format::dataspace::{Dataspace, DataspaceType};
 use clawhdf5_format::datatype::Datatype;
@@ -189,8 +189,8 @@ pub fn chunks(
     let Some(addr) = *btree_address else {
         return Ok(Vec::new());
     };
-    list_chunks(
+    list_chunks_in(
-        h5.data(),
+        h5.store(),
         layout,
         ds,
         dt.type_size() as usize,

crates/clawhdf5-tools/src/ls.rs

@@ -63,7 +63,7 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
         return args.usage_error(out, "missing FILE", USAGE);
     };
     let (file, obj) = split_file_arg(&target);
-    let mut h5 = match H5::open(std::path::Path::new(&file)) {
+    let mut h5 = match H5::open_arg(&file) {
         Ok(h) => h,
         Err(e) => {
             writeln!(out.e, "h5rs ls: {e}")?;

crates/clawhdf5-tools/src/stat.rs

@@ -86,7 +86,7 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
     let Some(file) = file else {
         return args.usage_error(out, "missing FILE", USAGE);
     };
-    let h5 = match H5::open(std::path::Path::new(&file)) {
+    let h5 = match H5::open_arg(&file) {
         Ok(h) => h,
         Err(e) => {
             writeln!(out.e, "h5rs stat: {e}")?;
@@ -206,7 +206,7 @@ pub fn run(args: &mut Args, out: &mut Out) -> std::io::Result<i32> {
     if let Err(e) = walk {
         errors.push(e.to_string());
     }
-    report(&h5, &file, &s, out)?;
+    report(&h5, &crate::h5::shown(&file), &s, out)?;
     for e in &errors {
         writeln!(out.e, "h5rs stat: {e}")?;
     }
@@ -291,7 +291,7 @@ fn report(h5: &H5, file: &str, s: &Stats, out: &mut Out) -> std::io::Result<()>
         s.attr_objects
     )?;
     writeln!(o, "\tMax. # of attributes to objects: {}", s.max_attrs)?;
-    let total = std::fs::metadata(&h5.path).map(|m| m.len()).unwrap_or(0);
+    let total = h5.size;
     let ub = h5.file.user_block_size();
     writeln!(o, "Summary of file space information:")?;
     writeln!(o, "  User block: {ub} bytes")?;

crates/clawhdf5-tools/src/value.rs

@@ -151,14 +151,14 @@ pub struct Decoder<'a> {
     pub h5: &'a H5,
     /// Variable-length elements are resolved as the library resolves them
     /// (so as libhdf5 does), not by a decoder of our own.
-    vl: RefCell<VlResolver<'a>>,
+    vl: RefCell<VlResolver<'a, dyn clawhdf5_format::storage::Storage + 'a>>,
 }
 
 impl<'a> Decoder<'a> {
     pub fn new(h5: &'a H5) -> Self {
         Self {
             h5,
-            vl: RefCell::new(VlResolver::new(h5.data(), h5.os(), h5.ls())),
+            vl: RefCell::new(VlResolver::new_in(h5.store(), h5.os(), h5.ls())),
         }
     }
 
@@ -268,7 +268,7 @@ impl<'a> Decoder<'a> {
     /// has it.
     fn decode_vlen(&self, is_string: bool, base: &Datatype, b: &[u8], depth: u32) -> Value {
         if is_string {
-            return match self.vl.borrow_mut().string_element(b) {
+            return match self.vl.borrow_mut().string_element_in(b) {
                 Ok(Some(s)) => Value::Str(String::from_utf8_lossy(s).into_owned()),
                 Ok(None) => Value::NullStr,
                 Err(e) => Value::Error(e.to_string()),
@@ -278,8 +278,9 @@ impl<'a> Decoder<'a> {
         if bs == 0 {
             return Value::Error("VL base type of size 0".into());
         }
-        let obj = match self.vl.borrow_mut().element(b, bs) {
+        // Copied out: decoding an element may resolve nested ones.
-            Ok(o) => o.unwrap_or(&[]),
+        let obj = match self.vl.borrow_mut().element_in(b, bs) {
+            Ok(o) => o.unwrap_or(&[]).to_vec(),
             Err(e) => return Value::Error(e.to_string()),
         };
         Value::Seq(

crates/clawhdf5-tools/tests/edit_interop.rs

@@ -113,7 +113,8 @@ impl Model {
             .fold(0u64, |a, (&x, &d)| a * d + x) as usize
     }
 
-    /// Grow to `shape`, new elements `fill`.
+    /// Change the extent to `shape`: elements inside both keep their
+    /// values, new ones are `fill`.
     fn resize(&mut self, shape: &[u64], fill: i32) {
         let old = self.clone();
         *self = Self::new(shape, |_| fill);
@@ -125,8 +126,10 @@ impl Model {
                 c[d] = r % old.shape[d];
                 r /= old.shape[d];
             }
-            let i = self.index(&c);
+            if c.iter().zip(shape).all(|(x, s)| x < s) {
-            self.data[i] = old.data[flat as usize];
+                let i = self.index(&c);
+                self.data[i] = old.data[flat as usize];
+            }
         }
     }
 
@@ -293,9 +296,24 @@ fn append_many_gzip() {
     }
 }
 
-/// Random operations — grow, hyperslab writes, point writes, attributes —
+/// A random attribute value: a scalar, an int64 array, a short string or
-/// on a 2-D dataset with one unlimited dimension, checked against a model
+/// one larger than a heap's managed-object limit (a huge heap object once
-/// after every few operations.
+/// the attributes are in dense storage).
+fn random_attr(rng: &mut Rng) -> AttrValue {
+    match rng.below(8) {
+        0..=2 => AttrValue::I64(rng.next() as i64 >> 3),
+        3..=4 => AttrValue::I64Array((0..1 + rng.below(40)).map(|k| k as i64 * 7).collect()),
+        5..=6 => AttrValue::String("s".repeat(1 + rng.below(200) as usize)),
+        _ => AttrValue::String("h".repeat(5000 + rng.below(100) as usize)),
+    }
+}
+
+/// Random operations — growth and shrinking along any dimension, hyperslab
+/// and point writes, attributes (enough names to move them to dense storage
+/// on version-2 object headers, replaced with values of any size) — on a
+/// 2-D dataset with one unlimited dimension and one with two (a version-2
+/// B-tree chunk index under `v114`/`latest`), checked against a model (and
+/// through h5py, numpy) after every few operations.
 fn random_ops(libver: &str, h5dump: bool, extra: &str, tag: &str, seed: u64) {
     let dir = tmpdir();
     let path = dir.path().join(format!("rand_{tag}.h5"));
@@ -304,106 +322,159 @@ fn random_ops(libver: &str, h5dump: bool, extra: &str, tag: &str, seed: u64) {
          with h5py.File({p:?}, 'w', libver={libver}) as f:\n\
          \x20   f.create_dataset('m', shape=(4, 7), maxshape=(None, 7), chunks=(3, 4), \
          dtype='<i4', fillvalue=-9{extra})\n\
-         \x20   f['m'][1:3, 2:6] = 5\n",
+         \x20   f['m'][1:3, 2:6] = 5\n\
+         \x20   f.create_dataset('b', shape=(5, 6), maxshape=(None, None), chunks=(2, 4), \
+         dtype='<i4', fillvalue=3{extra})\n\
+         \x20   f['b'][0:4, 1:5] = 8\n",
         p = path.to_str().unwrap()
     ));
-    let mut m = Model::new(&[4, 7], |_| -9);
+    let mut models = [Model::new(&[4, 7], |_| -9), Model::new(&[5, 6], |_| 3)];
-    m.write_block(&[1, 2], &[2, 4], &[5; 8]);
+    models[0].write_block(&[1, 2], &[2, 4], &[5; 8]);
-    let mut attrs: Vec<(String, i64)> = Vec::new();
+    models[1].write_block(&[0, 1], &[4, 4], &[8; 16]);
+    let fills = [-9, 3];
+    let names = ["m", "b"];
+    let mut attrs: Vec<(String, AttrValue)> = Vec::new();
     let mut rng = Rng(seed);
     let mut ed = FileEditor::open(&path).unwrap();
-    for step in 0..120 {
+    for step in 0..160 {
-        match rng.below(10) {
+        let d = rng.below(2) as usize;
-            0..=1 => {
+        let name = names[d];
-                let rows = m.shape[0] + 1 + rng.below(5);
+        let m = &mut models[d];
-                ed.resize("m", &[rows, 7]).unwrap();
+        match rng.below(12) {
-                m.resize(&[rows, 7], -9);
+            0..=2 => {
+                // Grow or shrink: dimension 1 of "m" is fixed at 7.
+                let rows = rng.below(m.shape[0] + 6);
+                let cols = if d == 0 { 7 } else { rng.below(m.shape[1] + 5) };
+                ed.resize(name, &[rows, cols]).unwrap();
+                m.resize(&[rows, cols], fills[d]);
             }
-            2..=6 => {
+            3..=7 if m.shape.iter().all(|&s| s > 0) => {
                 let r0 = rng.below(m.shape[0]);
-                let c0 = rng.below(7);
+                let c0 = rng.below(m.shape[1]);
                 let cnt = [
                     1 + rng.below((m.shape[0] - r0).min(6)),
-                    1 + rng.below(7 - c0),
+                    1 + rng.below((m.shape[1] - c0).min(6)),
                 ];
                 let n = cnt[0] * cnt[1];
                 let vals: Vec<i32> = (0..n).map(|_| (rng.next() % 100_000) as i32).collect();
-                ed.write_values("m", &block(&[r0, c0], &cnt), &vals)
+                ed.write_values(name, &block(&[r0, c0], &cnt), &vals)
                     .unwrap();
                 m.write_block(&[r0, c0], &cnt, &vals);
             }
-            7 => {
+            8 if m.shape.iter().all(|&s| s > 0) => {
                 let pts: Vec<Vec<u64>> = (0..1 + rng.below(4))
-                    .map(|_| vec![rng.below(m.shape[0]), rng.below(7)])
+                    .map(|_| vec![rng.below(m.shape[0]), rng.below(m.shape[1])])
                     .collect();
                 let vals: Vec<i32> = pts.iter().map(|_| rng.next() as i32).collect();
-                ed.write_values("m", &Selection::Points(pts.clone()), &vals)
+                ed.write_values(name, &Selection::Points(pts.clone()), &vals)
                     .unwrap();
                 for (p, v) in pts.iter().zip(&vals) {
                     let i = m.index(p);
                     m.data[i] = *v;
                 }
             }
-            _ => {
+            9..=11 => {
-                let k = rng.below(6);
+                let k = rng.below(20);
-                let name = format!("a{k}");
+                let aname = format!("a{k}");
-                let v = rng.next() as i64;
+                let v = random_attr(&mut rng);
-                match ed.set_attr("m", &name, &AttrValue::I64(v)) {
+                match ed.set_attr("m", &aname, &v) {
                     Ok(()) => {
-                        attrs.retain(|(n, _)| *n != name);
+                        attrs.retain(|(n, _)| *n != aname);
-                        attrs.push((name, v));
+                        attrs.push((aname, v));
                     }
-                    Err(e) => panic!("set_attr {name}: {e}"),
+                    // Replacing the only attribute in a heap block with one
+                    // of another size would have libhdf5 free the block.
+                    Err(Error::Unsupported(msg)) if msg.contains("last object") => {}
+                    Err(e) => panic!("set_attr {aname}: {e}"),
                 }
             }
+            _ => {}
         }
-        if step % 30 == 29 {
+        if step % 40 == 39 {
             drop(ed);
-            verify(&path, "m", &m);
+            for (n, m) in names.iter().zip(&models) {
+                verify(&path, n, m);
+            }
             check_tools(&path, h5dump);
             check_attrs(&path, "m", &attrs);
             ed = FileEditor::open(&path).unwrap();
         }
     }
     drop(ed);
-    verify(&path, "m", &m);
+    for (n, m) in names.iter().zip(&models) {
+        verify(&path, n, m);
+    }
     check_attrs(&path, "m", &attrs);
     py(&format!(
         "import h5py, numpy as np\n\
          with h5py.File({p:?}, 'r+') as f:\n\
-         \x20   d = f['m']\n\
+         \x20   for name, cols in (('m', 7), ('b', None)):\n\
-         \x20   n = d.shape[0]\n\
+         \x20       d = f[name]\n\
-         \x20   d.resize((n + 3, 7))\n\
+         \x20       n, c = d.shape\n\
-         \x20   d[n:, :] = 42\n\
+         \x20       d.resize((n + 3, cols or c + 2))\n\
-         \x20   d.attrs['from_h5py'] = 1.5\n",
+         \x20       d[n:, :] = 42\n\
+         \x20   f['m'].attrs['from_h5py'] = 1.5\n",
         p = path.to_str().unwrap()
     ));
-    let n = m.shape[0];
+    for (d, m) in models.iter_mut().enumerate() {
-    m.resize(&[n + 3, 7], -9);
+        let (n, c) = (m.shape[0], m.shape[1]);
-    m.write_block(&[n, 0], &[3, 7], &[42; 21]);
+        let c2 = if d == 0 { 7 } else { c + 2 };
-    verify(&path, "m", &m);
+        m.resize(&[n + 3, c2], fills[d]);
+        m.write_block(&[n, 0], &[3, c2], &vec![42; (3 * c2) as usize]);
+    }
+    for (n, m) in names.iter().zip(&models) {
+        verify(&path, n, m);
+    }
     check_tools(&path, h5dump);
+    check_attrs(&path, "m", &attrs);
 }
 
-fn check_attrs(path: &Path, obj: &str, attrs: &[(String, i64)]) {
+/// Our reader and h5py see `attrs` on dataset `obj` (and h5py's count of
+/// its attributes agrees with libhdf5's object info).
+fn check_attrs(path: &Path, obj: &str, attrs: &[(String, AttrValue)]) {
     let f = File::open(path).unwrap();
     let got = f.dataset(obj).unwrap().attrs().unwrap();
     for (n, v) in attrs {
-        match got.get(n) {
+        let g = got
-            Some(AttrValue::I64(g)) => assert_eq!(g, v, "attribute {n}"),
+            .get(n)
-            other => panic!("attribute {n}: {other:?}"),
+            .unwrap_or_else(|| panic!("attribute {n} missing"));
-        }
+        // Our reader reports a one-element array as a scalar.
+        let v = match v {
+            AttrValue::I64Array(a) if a.len() == 1 => &AttrValue::I64(a[0]),
+            v => v,
+        };
+        assert_eq!(format!("{g:?}"), format!("{v:?}"), "attribute {n}");
     }
-    let want: Vec<String> = attrs.iter().map(|(n, v)| format!("{n:?}: {v}")).collect();
+    let want: Vec<String> = attrs
-    py(&format!(
+        .iter()
+        .map(|(n, v)| {
+            let pv = match v {
+                AttrValue::I64(x) => format!("{x}"),
+                AttrValue::I64Array(a) => format!("{a:?}"),
+                AttrValue::String(s) => format!("{s:?}"),
+                other => panic!("{other:?}"),
+            };
+            format!("{n:?}: {pv}")
+        })
+        .collect();
+    let script = format!(
         "import h5py\n\
          f = h5py.File({p:?}, 'r')\n\
          want = {{{w}}}\n\
-         got = {{k: int(v) for k, v in f[{obj:?}].attrs.items() if k in want}}\n\
+         a = f[{obj:?}].attrs\n\
-         assert got == want, (got, want)\n",
+         def norm(v):\n\
+         \x20   v = v.decode() if isinstance(v, bytes) else v\n\
+         \x20   return v.tolist() if hasattr(v, 'tolist') else v\n\
+         got = {{k: norm(v) for k, v in a.items() if k in want}}\n\
+         assert got == want, sorted(set(want) ^ set(got))\n\
+         assert len(a) == h5py.h5o.get_info(f[{obj:?}].id).num_attrs == len(list(a))\n",
         p = path.to_str().unwrap(),
         w = want.join(", ")
-    ));
+    );
+    let sp = path.with_extension("attrs.py");
+    std::fs::write(&sp, script).unwrap();
+    let o = Command::new(python()).arg(&sp).output().unwrap();
+    assert!(o.status.success(), "attribute check failed:\n{}", text(&o));
 }
 
 #[test]
@@ -411,7 +482,11 @@ fn random_operations_match_a_model() {
     if !tools_ok() {
         return;
     }
-    let mut seed = 1;
+    // CLAWHDF5_EDIT_SEED runs the same workloads with other random choices.
+    let mut seed = std::env::var("CLAWHDF5_EDIT_SEED")
+        .ok()
+        .and_then(|s| s.parse::<u64>().ok())
+        .unwrap_or(1);
     for (i, (lv, dump)) in LIBVERS.iter().enumerate() {
         // h5dump has no LZF decoder (h5py's own filter).
         for (j, (extra, lzf)) in [
@@ -752,19 +827,20 @@ fn overwrite_every_layout() {
             }
             check_tools(&path, *dump);
         }
-        // A version-2 B-tree index can take new chunks only from libhdf5 for
+        // A version-2 B-tree index takes new chunks too.
-        // now: growing works, writing the new chunks is refused and changes
-        // nothing.
         if *lv != "'earliest'" {
             let mut ed = FileEditor::open(&path).unwrap();
-            ed.resize("bt2", &[8, 6]).unwrap();
+            ed.resize("bt2", &[8, 7]).unwrap();
-            let before = std::fs::read(&path).unwrap();
+            models[12].resize(&[8, 7], 0);
-            unsupported(ed.write_values("bt2", &block(&[6, 0], &[2, 6]), &[5; 12]));
+            let vals: Vec<i32> = (0..23).collect();
-            assert!(
+            ed.write_values("bt2", &block(&[6, 0], &[2, 7]), &vals[..14])
-                std::fs::read(&path).unwrap() == before,
+                .unwrap();
-                "a refused edit changed the file"
+            models[12].write_block(&[6, 0], &[2, 7], &vals[..14]);
-            );
+            ed.write_values("bt2", &block(&[0, 6], &[6, 1]), &vals[14..20])
-            models[12].resize(&[8, 6], 0);
+                .unwrap();
+            models[12].write_block(&[0, 6], &[6, 1], &vals[14..20]);
+            drop(ed);
+            verify(&path, "bt2", &models[12]);
         }
         // libhdf5 goes on modifying what we wrote.
         py(&format!(
@@ -835,19 +911,16 @@ fn attributes_in_place() {
             .unwrap();
         want.push(("d", "units".into(), AttrValue::String("km".into())));
         if *lv != "'earliest'" {
-            // Up to the compact limit (8) and no further; attributes in
+            // Up to the compact limit (8), then into dense storage; objects
-            // dense storage and tracked creation order are refused, and a
+            // already in dense storage and ones tracking creation order.
-            // refused edit writes nothing.
             ed.set_attr("g", "eighth", &AttrValue::I64(8)).unwrap();
             want.push(("g", "eighth".into(), AttrValue::I64(8)));
-            let before = std::fs::read(&path).unwrap();
+            ed.set_attr("g", "ninth", &AttrValue::I64(9)).unwrap();
-            unsupported(ed.set_attr("g", "ninth", &AttrValue::I64(9)));
+            want.push(("g", "ninth".into(), AttrValue::I64(9)));
-            unsupported(ed.set_attr("dense", "k0", &AttrValue::I64(1)));
+            ed.set_attr("dense", "k0", &AttrValue::I64(1)).unwrap();
-            unsupported(ed.set_attr("tracked", "b", &AttrValue::I64(1)));
+            want.push(("dense", "k0".into(), AttrValue::I64(1)));
-            assert!(
+            ed.set_attr("tracked", "b", &AttrValue::I64(1)).unwrap();
-                std::fs::read(&path).unwrap() == before,
+            want.push(("tracked", "b".into(), AttrValue::I64(1)));
-                "a refused edit changed the file"
-            );
         }
         drop(ed);
         check_tools(&path, *dump);
@@ -1027,7 +1100,7 @@ fn refused_edits_change_nothing() {
     let before = std::fs::read(&path).unwrap();
     let mut ed = FileEditor::open(&path).unwrap();
     unsupported(ed.write_all("s", &[0u8; 32]));
-    unsupported(ed.resize("x", &[4]));
+    unsupported(ed.resize("c", &[4]));
     unsupported(
         ed.resize("c", &[6, 1])
             .map_err(|_| Error::Unsupported(String::new())),
@@ -1124,9 +1197,10 @@ fn out_of_order_chunk_creation_matches_libhdf5() {
     }
 }
 
-/// Not a check: prints how much space an append workload leaks (the editor
+/// Not a check: prints how much space an append workload leaks (one
-/// never reuses space), against libhdf5 doing the same appends and against
+/// editor for the whole workload, which reuses the space it frees but not
-/// `h5repack` of each. Run with `--ignored --nocapture`.
+/// space it cannot fit a grown chunk into), against libhdf5 doing the same
+/// appends and against `h5repack` of each. Run with `--ignored --nocapture`.
 #[test]
 #[ignore]
 fn measure_append_waste() {

crates/clawhdf5-tools/tests/remote.rs

@@ -0,0 +1,171 @@
+//! `h5rs` on URLs (feature `remote`): every subcommand prints for
+//! `http://…/file.h5` what it prints for the local file (the name aside).
+//! The files are served by the range-request test server of
+//! clawhdf5-remote on 127.0.0.1.
+
+#![cfg(feature = "remote")]
+
+#[path = "../../clawhdf5-remote/tests/common/server.rs"]
+mod server;
+
+use std::path::{Path, PathBuf};
+use std::process::Command;
+
+fn h5rs(args: &[&str]) -> (String, i32) {
+    let out = Command::new(env!("CARGO_BIN_EXE_h5rs"))
+        .args(args)
+        .output()
+        .expect("run h5rs");
+    let text = format!(
+        "{}{}",
+        String::from_utf8_lossy(&out.stdout),
+        String::from_utf8_lossy(&out.stderr)
+    );
+    (text, out.status.code().unwrap_or(-1))
+}
+
+fn fixtures() -> Vec<PathBuf> {
+    let root = Path::new(env!("CARGO_MANIFEST_DIR"));
+    [
+        "../clawhdf5/tests/fixtures/tall.h5",
+        "../clawhdf5/tests/fixtures/written_by_v2_7_0.h5",
+        "../clawhdf5/tests/fixtures/written_by_v2_7_0_paged.h5",
+        "../clawhdf5/tests/fixtures/h5clear_mdc_image.h5",
+        "../clawhdf5-format/tests/fixtures/fractal_heap_multiblock.h5",
+        "../clawhdf5-format/tests/fixtures/legacy/tcompound.h5",
+        "../clawhdf5-format/tests/fixtures/legacy/h5ex_g_iterate.h5",
+    ]
+    .iter()
+    .map(|p| root.join(p))
+    .collect()
+}
+
+#[test]
+fn every_subcommand_reads_a_url_like_the_local_file() {
+    let files = fixtures();
+    let served: Vec<(String, Vec<u8>)> = files
+        .iter()
+        .enumerate()
+        .map(|(i, p)| {
+            let name = p.file_name().unwrap().to_str().unwrap();
+            (format!("/{i}/{name}"), std::fs::read(p).unwrap())
+        })
+        .collect();
+    let server = server::Server::start(served.clone());
+    for (p, (url_path, _)) in files.iter().zip(&served) {
+        let url = server.url(url_path);
+        let local = p.to_str().unwrap();
+        for cmd in [
+            &["ls", "-r", "-v"][..],
+            &["dump"],
+            &["dump", "--json"],
+            &["stat"],
+            &["check", "--data"],
+        ] {
+            fn args<'a>(cmd: &[&'a str], f: &'a str) -> Vec<&'a str> {
+                cmd.iter().copied().chain([f]).collect()
+            }
+            let (want, want_rc) = h5rs(&args(cmd, local));
+            let (got, got_rc) = h5rs(&args(cmd, &url));
+            assert_eq!(
+                got.replace(&url, local),
+                want,
+                "h5rs {} {url}",
+                cmd.join(" ")
+            );
+            assert_eq!(got_rc, want_rc, "h5rs {} {url}", cmd.join(" "));
+        }
+        let (a, rc) = h5rs(&["diff", local, &url]);
+        assert_eq!(rc, 0, "h5rs diff {local} {url}: {a}");
+    }
+}
+
+#[test]
+fn url_errors_are_clean() {
+    let server = server::Server::start(vec![("/x.h5".into(), vec![1u8; 100])]);
+    let (out, rc) = h5rs(&["ls", &server.url("/missing.h5")]);
+    assert_eq!(rc, 2, "{out}");
+    assert!(out.contains("404"), "{out}");
+    let (out, rc) = h5rs(&["ls", &server.url("/x.h5")]);
+    assert_eq!(rc, 2, "{out}");
+    assert!(out.contains("not an HDF5 file"), "{out}");
+    let (out, rc) = h5rs(&["check", &server.url("/missing.h5")]);
+    assert_eq!(rc, 2, "{out}");
+    #[cfg(not(feature = "remote-https"))]
+    {
+        let (out, rc) = h5rs(&["ls", "https://example.com/a.h5"]);
+        assert_eq!(rc, 2, "{out}");
+        assert!(out.contains("`https` feature"), "{out}");
+    }
+}
+
+/// `check` downloads a remote file whole, but never trusts the length the
+/// server claims: a server claiming 2^62 bytes for a small file is refused
+/// before anything is allocated (it used to abort the process), and
+/// `--max-download` caps real files too.
+#[test]
+fn check_refuses_a_remote_file_beyond_the_download_limit() {
+    use std::sync::atomic::Ordering;
+    let tall = Path::new(env!("CARGO_MANIFEST_DIR")).join("../clawhdf5/tests/fixtures/tall.h5");
+    let server = server::Server::start(vec![("/t.h5".into(), std::fs::read(&tall).unwrap())]);
+    let url = server.url("/t.h5");
+    let (out, rc) = h5rs(&["check", &url]);
+    assert_eq!(rc, 0, "{out}");
+    let (out, rc) = h5rs(&["check", "--max-download", "1000", &url]);
+    assert_eq!(rc, 2, "{out}");
+    assert!(out.contains("download limit of 1000 bytes"), "{out}");
+    server.shared.fake_total.store(1 << 62, Ordering::SeqCst);
+    let (out, rc) = h5rs(&["check", &url]);
+    assert_eq!(rc, 2, "{out}");
+    assert!(out.contains("more than the download limit"), "{out}");
+}
+
+/// A URL's credentials (userinfo, a presigned URL's query string) are not
+/// printed: not in errors, not in the file name of the output.
+#[test]
+fn credentials_in_urls_are_not_printed() {
+    let tall = Path::new(env!("CARGO_MANIFEST_DIR")).join("../clawhdf5/tests/fixtures/tall.h5");
+    let server = server::Server::start(vec![("/t.h5".into(), std::fs::read(&tall).unwrap())]);
+    let url = |path: &str| {
+        format!(
+            "http://user:hunter2@{}{path}?X-Amz-Signature=SECRETSIG",
+            server.addr
+        )
+    };
+    for args in [
+        vec!["ls", "-r"],
+        vec!["dump"],
+        vec!["stat"],
+        vec!["check"],
+        vec!["check", "--max-download", "10"],
+    ] {
+        for path in ["/t.h5", "/missing.h5"] {
+            let u = url(path);
+            let mut a = args.clone();
+            a.push(&u);
+            let (out, _) = h5rs(&a);
+            assert!(
+                !out.contains("hunter2") && !out.contains("SECRETSIG"),
+                "h5rs {}: {out}",
+                a.join(" ")
+            );
+        }
+    }
+    let (out, rc) = h5rs(&["diff", tall.to_str().unwrap(), &url("/t.h5"), "/nope"]);
+    assert_eq!(rc, 2, "{out}");
+    assert!(
+        !out.contains("hunter2") && !out.contains("SECRETSIG"),
+        "{out}"
+    );
+}
+
+/// `check URL` opens the file once: for a file within the first block,
+/// one request in all (it probed the server twice before).
+#[test]
+fn check_url_probes_the_server_once() {
+    let tall = Path::new(env!("CARGO_MANIFEST_DIR")).join("../clawhdf5/tests/fixtures/tall.h5");
+    let server = server::Server::start(vec![("/t.h5".into(), std::fs::read(&tall).unwrap())]);
+    let (out, rc) = h5rs(&["check", "--data", &server.url("/t.h5")]);
+    assert_eq!(rc, 0, "{out}");
+    assert_eq!(server.requests(), 1, "{:?}", server.log());
+}

crates/clawhdf5/src/edit/attrs.rs

@@ -0,0 +1,513 @@
+//! Setting attributes, as `H5O__attr_create` / `H5A__dense_insert` do:
+//! compact attributes are object header messages (with their creation
+//! index in the message header when the object tracks creation order);
+//! when an object reaches its compact limit (or an attribute is too large
+//! for a header message) its attributes move to dense storage — a fractal
+//! heap for the encoded messages, a version-2 B-tree indexing them by name
+//! hash (record type 8) and, when creation order is indexed, a second one
+//! by creation index (type 9) — and the Attribute Info message points at
+//! them.
+
+use std::cmp::Ordering;
+
+use clawhdf5_format::attribute::AttributeMessage;
+use clawhdf5_format::dataspace::DataspaceType;
+
+use crate::edit::btree2::Bt2;
+use crate::edit::fheap::Heap;
+use crate::edit::image::{Image, get_uint, put_uint, undef};
+use crate::edit::ohdr::{Header, MSG_ATTRIBUTE};
+use crate::edit::{MSG_ATTR_INFO, MSG_FLAG_DONTSHARE, MSG_FLAG_SHARED, check_plain};
+use crate::error::Error;
+use crate::reader::File;
+use crate::types::AttrValue;
+
+/// `H5O_MESG_MAX_SIZE`: a larger attribute goes to dense storage.
+const MESG_MAX_SIZE: usize = 65536;
+/// `H5O_MAX_CRT_ORDER_IDX`: the creation index of an attribute of an object
+/// that does not track creation order.
+const NO_CRT_IDX: u16 = u16::MAX;
+/// Name and creation-order index B-trees (`H5A_NAME_BT2_*`,
+/// `H5A_CORDER_BT2_*`).
+const NAME_BT2_TYPE: u8 = 8;
+const CORDER_BT2_TYPE: u8 = 9;
+const ATTR_BT2_NODE: u32 = 512;
+/// Heap IDs in attribute records.
+const ID_LEN: usize = 8;
+
+/// An object's Attribute Info message.
+#[derive(Debug, Clone)]
+struct AInfo {
+    /// Its message index in the header.
+    idx: usize,
+    track: bool,
+    index: bool,
+    max_crt: u16,
+    fheap: u64,
+    name_bt2: u64,
+    corder_bt2: u64,
+}
+
+impl AInfo {
+    fn load(img: &Image<'_>, hdr: &Header) -> Result<Option<Self>, Error> {
+        let Some(idx) = hdr.find(MSG_ATTR_INFO) else {
+            return Ok(None);
+        };
+        if hdr.msgs[idx].flags & MSG_FLAG_SHARED != 0 {
+            return Err(Error::Unsupported("shared attribute info message".into()));
+        }
+        let d = hdr.data(img, idx)?;
+        let os = img.os as usize;
+        let short = || Error::Unsupported("short attribute info message".into());
+        if d.first() != Some(&0) {
+            return Err(Error::Unsupported("attribute info message version".into()));
+        }
+        let flags = *d.get(1).ok_or_else(short)?;
+        let track = flags & 0x01 != 0;
+        let index = flags & 0x02 != 0;
+        let mut p = 2;
+        let mut max_crt = 0;
+        if track {
+            let b = d.get(p..p + 2).ok_or_else(short)?;
+            max_crt = u16::from_le_bytes([b[0], b[1]]);
+            p += 2;
+        }
+        let n = if index { 3 } else { 2 };
+        if d.len() < p + n * os {
+            return Err(short());
+        }
+        Ok(Some(Self {
+            idx,
+            track,
+            index,
+            max_crt,
+            fheap: get_uint(&d[p..], img.os),
+            name_bt2: get_uint(&d[p + os..], img.os),
+            corder_bt2: if index {
+                get_uint(&d[p + 2 * os..], img.os)
+            } else {
+                undef(img.os)
+            },
+        }))
+    }
+
+    fn dense(&self, os: u8) -> bool {
+        self.fheap != undef(os)
+    }
+
+    /// Store the changeable fields back into the message.
+    fn store(&self, img: &mut Image<'_>, hdr: &mut Header) -> Result<(), Error> {
+        let os = img.os as usize;
+        let mut p = 2;
+        if self.track {
+            hdr.patch(img, self.idx, p, &self.max_crt.to_le_bytes())?;
+            p += 2;
+        }
+        let mut a = vec![0u8; os];
+        for (k, v) in [self.fheap, self.name_bt2, self.corder_bt2]
+            .into_iter()
+            .enumerate()
+            .take(if self.index { 3 } else { 2 })
+        {
+            put_uint(&mut a, v, img.os);
+            hdr.patch(img, self.idx, p + k * os, &a)?;
+        }
+        Ok(())
+    }
+
+    /// The next creation index (`H5O__attr_create`), or libhdf5's "none".
+    fn next_crt(&mut self) -> Result<u16, Error> {
+        if !self.track {
+            return Ok(NO_CRT_IDX);
+        }
+        if self.max_crt == NO_CRT_IDX {
+            return Err(Error::Unsupported(
+                "object's attribute creation index is exhausted".into(),
+            ));
+        }
+        self.max_crt += 1;
+        Ok(self.max_crt - 1)
+    }
+}
+
+/// The name bytes of an attribute message body (without the NUL).
+pub(super) fn attr_name(d: &[u8]) -> Result<&[u8], Error> {
+    let bad = || Error::Unsupported("malformed attribute message".into());
+    let (len, at) = match d.first() {
+        Some(1) | Some(2) if d.len() >= 8 => (usize::from(u16::from_le_bytes([d[2], d[3]])), 8),
+        Some(3) if d.len() >= 9 => (usize::from(u16::from_le_bytes([d[2], d[3]])), 9),
+        _ => return Err(bad()),
+    };
+    let name = d.get(at..at + len).ok_or_else(bad)?;
+    Ok(name.split(|&b| b == 0).next().unwrap_or(name))
+}
+
+/// A new Attribute Info message for a version-2 header with flags
+/// `hdr_flags`, as `H5O__attr_create` makes it: version 0, creation order
+/// tracked / indexed as the header's flags say, the maximum creation index,
+/// and no dense storage (undefined fractal heap and B-tree addresses).
+fn attr_info_message(hdr_flags: u8, max_crt: u16, os: u8) -> Vec<u8> {
+    let track = hdr_flags & 0x04 != 0;
+    let index = hdr_flags & 0x08 != 0;
+    let mut b = vec![0u8, u8::from(track) | (u8::from(index) << 1)];
+    if track {
+        b.extend_from_slice(&max_crt.to_le_bytes());
+    }
+    let undef_addr = vec![0xffu8; os as usize];
+    b.extend_from_slice(&undef_addr);
+    b.extend_from_slice(&undef_addr);
+    if index {
+        b.extend_from_slice(&undef_addr);
+    }
+    b
+}
+
+/// A version-2 header's limit on compact attributes: stored when its flags
+/// say so, else libhdf5's default of 8.
+fn max_compact_attrs(img: &Image<'_>, hdr: &Header) -> Result<u16, Error> {
+    if hdr.flags & 0x10 == 0 {
+        return Ok(8);
+    }
+    let mut p = hdr.addr + 6;
+    if hdr.flags & 0x20 != 0 {
+        p += 16;
+    }
+    let b = img.read(p, 2)?;
+    Ok(u16::from_le_bytes([b[0], b[1]]))
+}
+
+/// A version-1 attribute message (what libhdf5 writes in a version-1 object
+/// header): name, datatype and dataspace each padded to 8 bytes, the
+/// dataspace as a version-1 dataspace message.
+fn encode_attr_v1(a: &AttributeMessage, ls: u8) -> Vec<u8> {
+    let mut name = a.name.as_bytes().to_vec();
+    name.push(0);
+    let dt = a.datatype.serialize();
+    let mut ds = vec![1u8, a.dataspace.rank, 0, 0, 0, 0, 0, 0];
+    if a.dataspace.space_type == DataspaceType::Simple {
+        let mut b = vec![0u8; ls as usize];
+        for &d in &a.dataspace.dimensions {
+            put_uint(&mut b, d, ls);
+            ds.extend_from_slice(&b);
+        }
+        if let Some(max) = &a.dataspace.max_dimensions {
+            ds[2] = 0x01;
+            for &d in max {
+                put_uint(&mut b, d, ls);
+                ds.extend_from_slice(&b);
+            }
+        }
+    } else {
+        ds[1] = 0;
+    }
+    let mut out = vec![1u8, 0];
+    out.extend_from_slice(&(name.len() as u16).to_le_bytes());
+    out.extend_from_slice(&(dt.len() as u16).to_le_bytes());
+    out.extend_from_slice(&(ds.len() as u16).to_le_bytes());
+    for part in [&name, &dt, &ds] {
+        out.extend_from_slice(part);
+        out.resize(out.len().next_multiple_of(8), 0);
+    }
+    out.extend_from_slice(&a.raw_data);
+    out
+}
+
+/// Dense storage opened for changes.
+struct Dense {
+    heap: Heap,
+    names: Bt2,
+    order: Option<Bt2>,
+}
+
+/// `H5_checksum_lookup3` of a name, as the name index keys it.
+fn name_hash(name: &[u8]) -> u32 {
+    clawhdf5_format::checksum::jenkins_lookup3(name)
+}
+
+/// Compare attribute `name` (hash `hash`) with a name-index record
+/// (`H5A__dense_btree2_name_compare`: the hash, then the stored name).
+fn cmp_name(
+    heap: &Heap,
+    img: &Image<'_>,
+    hash: u32,
+    name: &[u8],
+    rec: &[u8],
+) -> Result<Ordering, Error> {
+    let theirs = u32::from_le_bytes([rec[13], rec[14], rec[15], rec[16]]);
+    match hash.cmp(&theirs) {
+        Ordering::Equal => {
+            if rec[ID_LEN] & MSG_FLAG_SHARED != 0 {
+                return Err(Error::Unsupported(
+                    "shared attribute in dense storage".into(),
+                ));
+            }
+            let obj = heap.read(img, &rec[..ID_LEN])?;
+            Ok(name.cmp(attr_name(&obj)?))
+        }
+        o => Ok(o),
+    }
+}
+
+fn corder_of(rec: &[u8]) -> u32 {
+    u32::from_le_bytes([rec[9], rec[10], rec[11], rec[12]])
+}
+
+impl Dense {
+    fn open(img: &Image<'_>, ai: &AInfo) -> Result<Self, Error> {
+        let heap = Heap::open(img, ai.fheap)?;
+        let names = Bt2::open(img, ai.name_bt2)?;
+        if names.tree_type() != NAME_BT2_TYPE || names.record_size() != ID_LEN + 9 {
+            return Err(Error::Unsupported("attribute name index layout".into()));
+        }
+        let order = if ai.index {
+            let t = Bt2::open(img, ai.corder_bt2)?;
+            if t.tree_type() != CORDER_BT2_TYPE || t.record_size() != ID_LEN + 5 {
+                return Err(Error::Unsupported(
+                    "attribute creation-order index layout".into(),
+                ));
+            }
+            Some(t)
+        } else {
+            None
+        };
+        Ok(Self { heap, names, order })
+    }
+
+    /// `H5A__dense_create`: heap, name index, [creation-order index].
+    fn create(img: &mut Image<'_>, index: bool) -> Result<Self, Error> {
+        let heap = Heap::create_attribute_heap(img)?;
+        let names = Bt2::create(img, NAME_BT2_TYPE, ATTR_BT2_NODE, ID_LEN + 9, 100, 40)?;
+        let order = if index {
+            Some(Bt2::create(
+                img,
+                CORDER_BT2_TYPE,
+                ATTR_BT2_NODE,
+                ID_LEN + 5,
+                100,
+                40,
+            )?)
+        } else {
+            None
+        };
+        Ok(Self { heap, names, order })
+    }
+
+    /// `H5A__dense_insert` of an encoded attribute message.
+    fn insert(&mut self, img: &mut Image<'_>, body: &[u8], crt: u16) -> Result<(), Error> {
+        let name = attr_name(body)?.to_vec();
+        let id = self.heap.insert(img, body)?;
+        if id.len() != ID_LEN {
+            return Err(Error::Unsupported("attribute heap ID length".into()));
+        }
+        let hash = name_hash(&name);
+        let mut rec = id.clone();
+        rec.push(0);
+        rec.extend_from_slice(&u32::from(crt).to_le_bytes());
+        rec.extend_from_slice(&hash.to_le_bytes());
+        let heap = &self.heap;
+        self.names
+            .insert(img, &mut |im, r| cmp_name(heap, im, hash, &name, r), &rec)?;
+        if let Some(t) = &mut self.order {
+            let key = u32::from(crt);
+            t.insert(
+                img,
+                &mut |_, r| Ok(key.cmp(&corder_of(r))),
+                &rec[..ID_LEN + 5],
+            )?;
+        }
+        Ok(())
+    }
+
+    fn finish(&mut self, img: &mut Image<'_>) -> Result<(), Error> {
+        self.heap.finish(img)?;
+        self.names.finish(img)?;
+        if let Some(t) = &mut self.order {
+            t.finish(img)?;
+        }
+        Ok(())
+    }
+}
+
+/// Set attribute `name` of the object at `path` to `value`.
+pub(super) fn set_attr(
+    f: &File,
+    img: &mut Image<'_>,
+    path: &str,
+    name: &str,
+    value: &AttrValue,
+) -> Result<(), Error> {
+    let addr = clawhdf5_format::group_v2::resolve_path_any(f.as_bytes(), f.superblock(), path)?;
+    let mut hdr = Header::load(img, addr)?;
+    let mut msg = clawhdf5_format::type_builders::build_attr_message(name, value);
+    check_plain(&msg.datatype)?;
+    // libhdf5 encodes a simple dataspace with its maximum dimensions (the
+    // current ones when none were given), so an attribute takes the same
+    // space in a header or heap as when libhdf5 writes it.
+    if msg.dataspace.space_type == DataspaceType::Simple && msg.dataspace.max_dimensions.is_none() {
+        msg.dataspace.max_dimensions = Some(msg.dataspace.dimensions.clone());
+    }
+    // H5A__set_version: version 1 unless the name is not ASCII (then 3),
+    // raised to the file's low bound — which is the earliest for a file
+    // libhdf5 opens without a libver setting (h5py's `r+`).
+    let body = if hdr.version == 1 || name.is_ascii() {
+        encode_attr_v1(&msg, img.ls)
+    } else {
+        let mut b = msg.serialize_v3(img.ls);
+        if !name.is_ascii() {
+            b[8] = 1; // UTF-8 name
+        }
+        b
+    };
+    let mut ainfo = if hdr.version == 2 {
+        AInfo::load(img, &hdr)?
+    } else {
+        None
+    };
+    if let Some(ai) = ainfo.as_mut().filter(|a| a.dense(img.os)) {
+        let mut ai = ai.clone();
+        set_dense(img, &mut hdr, &mut ai, name.as_bytes(), &body)?;
+        return hdr.finish(img);
+    }
+
+    let mut existing = None;
+    let mut count = 0usize;
+    for i in 0..hdr.msgs.len() {
+        if hdr.msgs[i].mtype != MSG_ATTRIBUTE {
+            continue;
+        }
+        if hdr.msgs[i].flags & MSG_FLAG_SHARED != 0 {
+            return Err(Error::Unsupported("shared attribute message".into()));
+        }
+        count += 1;
+        if attr_name(&hdr.data(img, i)?)? == name.as_bytes() {
+            existing = Some(i);
+        }
+    }
+    if let Some(i) = existing {
+        hdr.delete(img, i)?;
+        count -= 1;
+    }
+    if hdr.version == 1 {
+        hdr.insert(img, MSG_ATTRIBUTE, 0, &body, None)?;
+        return hdr.finish(img);
+    }
+    let tracked = hdr.flags & 0x04 != 0;
+    // H5O__attr_create: a missing Attribute Info message starts from
+    // nothing (and is added below, holding the new maximum creation index).
+    let new_ainfo = ainfo.is_none();
+    let mut ai = ainfo.take().unwrap_or(AInfo {
+        idx: usize::MAX,
+        track: tracked,
+        index: hdr.flags & 0x08 != 0,
+        max_crt: 0,
+        fheap: undef(img.os),
+        name_bt2: undef(img.os),
+        corder_bt2: undef(img.os),
+    });
+    let max_compact = usize::from(max_compact_attrs(img, &hdr)?);
+    if count == max_compact || body.len() >= MESG_MAX_SIZE {
+        if new_ainfo {
+            return Err(Error::Unsupported(
+                "dense attribute storage for an object without an Attribute Info message".into(),
+            ));
+        }
+        to_dense(img, &mut hdr, &mut ai)?;
+        set_dense(img, &mut hdr, &mut ai, name.as_bytes(), &body)?;
+        return hdr.finish(img);
+    }
+    let crt = ai.next_crt()?;
+    let corder = tracked.then_some(crt);
+    if new_ainfo {
+        // libhdf5 appends the Attribute Info message before the attribute
+        // when free space holds both, else after it, so that a new
+        // continuation chunk made for the attribute has room for it too.
+        let a = attr_info_message(hdr.flags, ai.max_crt, img.os);
+        let first = hdr.has_free(a.len() + hdr.hsize() + body.len());
+        if first {
+            hdr.insert(img, MSG_ATTR_INFO, MSG_FLAG_DONTSHARE, &a, Some(0))?;
+        }
+        hdr.insert(img, MSG_ATTRIBUTE, 0, &body, corder)?;
+        if !first {
+            hdr.insert(img, MSG_ATTR_INFO, MSG_FLAG_DONTSHARE, &a, Some(0))?;
+        }
+    } else {
+        hdr.insert(img, MSG_ATTRIBUTE, 0, &body, corder)?;
+        ai.store(img, &mut hdr)?;
+    }
+    hdr.finish(img)
+}
+
+/// Move every compact attribute of the object into new dense storage, in
+/// header message order (`H5O__attr_to_dense_cb`), leaving free space where
+/// the messages were.
+fn to_dense(img: &mut Image<'_>, hdr: &mut Header, ai: &mut AInfo) -> Result<(), Error> {
+    let mut dense = Dense::create(img, ai.index)?;
+    for i in 0..hdr.msgs.len() {
+        if hdr.msgs[i].mtype != MSG_ATTRIBUTE {
+            continue;
+        }
+        if hdr.msgs[i].flags & MSG_FLAG_SHARED != 0 {
+            return Err(Error::Unsupported("shared attribute message".into()));
+        }
+        let body = hdr.data(img, i)?;
+        let crt = if ai.track {
+            hdr.msgs[i].corder.unwrap_or(0)
+        } else {
+            NO_CRT_IDX
+        };
+        dense.insert(img, &body, crt)?;
+        hdr.delete(img, i)?;
+    }
+    dense.finish(img)?;
+    ai.fheap = dense.heap.address();
+    ai.name_bt2 = dense.names.address();
+    if let Some(t) = &dense.order {
+        ai.corder_bt2 = t.address();
+    }
+    ai.store(img, hdr)
+}
+
+/// Set an attribute of an object whose attributes are in dense storage: an
+/// attribute of that name whose new encoding has the old one's size is
+/// rewritten in its heap object (`H5A__dense_write`); otherwise the old one
+/// is removed (`H5A__dense_remove`: name index, creation-order index, heap
+/// object) and the new one inserted with the next creation index.
+fn set_dense(
+    img: &mut Image<'_>,
+    hdr: &mut Header,
+    ai: &mut AInfo,
+    name: &[u8],
+    body: &[u8],
+) -> Result<(), Error> {
+    let mut dense = Dense::open(img, ai)?;
+    let hash = name_hash(name);
+    let found = {
+        let heap = &dense.heap;
+        dense
+            .names
+            .find(img, &mut |im, r| cmp_name(heap, im, hash, name, r))?
+    };
+    if let Some(rec) = found {
+        if dense.heap.write_in_place(img, &rec[..ID_LEN], body)? {
+            return dense.finish(img);
+        }
+        {
+            let heap = &dense.heap;
+            dense
+                .names
+                .remove(img, &mut |im, r| cmp_name(heap, im, hash, name, r))?;
+        }
+        if let Some(t) = &mut dense.order {
+            let key = corder_of(&rec);
+            t.remove(img, &mut |_, r| Ok(key.cmp(&corder_of(r))))?
+                .ok_or_else(|| {
+                    Error::Unsupported("attribute missing from its creation-order index".into())
+                })?;
+        }
+        dense.heap.remove(img, &rec[..ID_LEN])?;
+    }
+    let crt = ai.next_crt()?;
+    dense.insert(img, body, crt)?;
+    dense.finish(img)?;
+    ai.store(img, hdr)
+}

crates/clawhdf5/src/edit/btree1.rs

@@ -56,6 +56,15 @@ fn bad(why: &str) -> Error {
     ))
 }
 
+/// What a removal did below a node (`H5B_ins_t`), with the removed chunk's
+/// address and size.
+enum Rm {
+    NotFound,
+    Noop((u64, u32)),
+    /// The child is gone: the parent must drop it.
+    Remove((u64, u32)),
+}
+
 enum Ins {
     Done,
     /// The node split; the new right sibling and its first key.
@@ -144,7 +153,14 @@ impl BTree1 {
         put_uint(&mut d[8 + osz..], node.right, os);
         let ks = self.key_size();
         let mut p = 8 + 2 * osz;
-        for (i, k) in node.keys.iter().enumerate() {
+        // An empty node (a root whose last chunk was removed) stores no
+        // keys, as libhdf5 writes it.
+        let nkeys = if node.children.is_empty() {
+            0
+        } else {
+            node.keys.len()
+        };
+        for (i, k) in node.keys.iter().take(nkeys).enumerate() {
             d[p..p + 4].copy_from_slice(&k.size.to_le_bytes());
             d[p + 4..p + 8].copy_from_slice(&k.mask.to_le_bytes());
             for (j, o) in k.offs.iter().enumerate() {
@@ -210,6 +226,18 @@ impl BTree1 {
             return Err(bad("bad chunk key"));
         }
         let root = self.read(img, self.root)?;
+        if root.children.is_empty() {
+            // Every chunk was removed (H5B__insert_helper's first
+            // insertion): the root, a leaf again, takes it.
+            let right = self.right_key_after(&key);
+            let node = Node {
+                level: 0,
+                keys: vec![key, right],
+                children: vec![addr],
+                ..root
+            };
+            return self.write(img, &node);
+        }
         if let Ins::Split(mid, right_addr) = self.insert_at(img, root, &key, addr, 64)? {
             // The root split: move its (left) half to a new node so the root
             // keeps its address, then make the root the parent of both.
@@ -372,6 +400,143 @@ impl BTree1 {
         cmp(&key.offs, &right.keys[0].offs) != Ordering::Less
     }
 
+    /// Remove the chunk at offsets `offs` (element-size coordinate 0), as
+    /// `H5B_remove` does for the chunk index (whose critical key is the
+    /// left one): no rebalancing; a node left without children is deleted
+    /// and its siblings relinked (the left one takes over its right key),
+    /// a root left empty becomes an empty leaf. Returns the chunk's address
+    /// and stored size, or `None` when the tree has no such chunk (nothing
+    /// changes then). Deleted nodes are freed in `img`.
+    pub(crate) fn remove(
+        &mut self,
+        img: &mut Image<'_>,
+        offs: &[u64],
+    ) -> Result<Option<(u64, u32)>, Error> {
+        if offs.len() != self.ndims {
+            return Err(bad("bad chunk key"));
+        }
+        let mut lt = None;
+        match self.remove_at(img, self.root, 0, offs, &mut lt, 64)? {
+            Rm::NotFound => Ok(None),
+            Rm::Noop(c) | Rm::Remove(c) => Ok(Some(c)),
+        }
+    }
+
+    fn remove_at(
+        &self,
+        img: &mut Image<'_>,
+        addr: u64,
+        level: usize,
+        offs: &[u64],
+        lt_out: &mut Option<Key>,
+        depth: u8,
+    ) -> Result<Rm, Error> {
+        if depth == 0 {
+            return Err(bad("tree too deep"));
+        }
+        let mut node = self.read(img, addr)?;
+        let n = node.children.len();
+        // H5D__btree_cmp3 over (keys[i], keys[i + 1]), binary search.
+        let (mut lo, mut hi, mut idx) = (0usize, n, 0usize);
+        let mut c = 1i32;
+        while lo < hi && c != 0 {
+            idx = (lo + hi) / 2;
+            c = if cmp(offs, &node.keys[idx + 1].offs) != Ordering::Less {
+                1
+            } else if cmp(offs, &node.keys[idx].offs) == Ordering::Less {
+                -1
+            } else {
+                0
+            };
+            if c < 0 {
+                hi = idx;
+            } else {
+                lo = idx + 1;
+            }
+        }
+        if c != 0 {
+            return Ok(Rm::NotFound);
+        }
+        let mut lt_changed = None;
+        let res = if node.level > 0 {
+            let child = self.read(img, node.children[idx])?;
+            if usize::from(child.level) + 1 != usize::from(node.level) {
+                return Err(bad("inconsistent node levels"));
+            }
+            self.remove_at(
+                img,
+                node.children[idx],
+                level + 1,
+                offs,
+                &mut lt_changed,
+                depth - 1,
+            )?
+        } else {
+            if node.keys[idx].offs != offs {
+                return Ok(Rm::NotFound);
+            }
+            Rm::Remove((node.children[idx], node.keys[idx].size))
+        };
+        let chunk = match res {
+            Rm::NotFound => return Ok(Rm::NotFound),
+            Rm::Noop(c) | Rm::Remove(c) => c,
+        };
+        let mut dirty = false;
+        if let Some(k) = lt_changed {
+            node.keys[idx] = k;
+            dirty = true;
+            if idx == 0 {
+                *lt_out = Some(node.keys[0].clone());
+            }
+        }
+        let out = Rm::Noop(chunk);
+        if let Rm::Remove(_) = res {
+            let undefined = undef(img.os);
+            if n == 1 {
+                if level > 0 {
+                    if node.left != undefined {
+                        let mut sib = self.read(img, node.left)?;
+                        let last = sib.children.len();
+                        sib.keys[last] = node.keys[1].clone();
+                        sib.right = node.right;
+                        self.write(img, &sib)?;
+                    }
+                    if node.right != undefined {
+                        let mut sib = self.read(img, node.right)?;
+                        sib.left = node.left;
+                        self.write(img, &sib)?;
+                    }
+                    img.free(addr, self.node_size(img.os) as u64);
+                    return Ok(Rm::Remove(chunk));
+                }
+                node.children.clear();
+                node.keys.truncate(1);
+                node.level = 0;
+            } else if idx == 0 {
+                node.keys.remove(0);
+                node.children.remove(0);
+                *lt_out = Some(node.keys[0].clone());
+            } else {
+                // Right-most or middle child: its left key goes, the next
+                // key becomes the following child's left key.
+                node.keys.remove(idx);
+                node.children.remove(idx);
+            }
+            dirty = true;
+        }
+        if dirty {
+            self.write(img, &node)?;
+        }
+        // The left sibling's right key follows a changed left key.
+        if lt_out.is_some() && node.left != undef(img.os) && level > 0 {
+            let mut sib = self.read(img, node.left)?;
+            let last = sib.children.len();
+            sib.keys[last] = node.keys[0].clone();
+            self.write(img, &sib)?;
+        }
+        Ok(out)
+    }
+
     fn insert_child(&self, node: &mut Node, pos: usize, key: Key, addr: u64) {
         let n = node.children.len();
         if node.level == 0 {

crates/clawhdf5/src/edit/earray.rs

@@ -383,12 +383,23 @@ impl Ea {
     }
 
     /// Set element `idx` to `e`.
-    pub(crate) fn set(&mut self, img: &mut Image<'_>, idx: u64, e: Elem) -> Result<(), Error> {
+    /// Set element `idx` to `e`, or back to the fill element (`None`: a
+    /// removed chunk, `H5D__earray_idx_remove`), which creates no block.
+    pub(crate) fn set(
+        &mut self,
+        img: &mut Image<'_>,
+        idx: u64,
+        e: Option<Elem>,
+    ) -> Result<(), Error> {
         let os = img.os;
         let osz = u64::from(os);
         let es = self.slot_size(os) as u64;
-        let enc = encode_elem(Some(e), self.filtered, self.elem_size, os)?;
+        let enc = encode_elem(e, self.filtered, self.elem_size, os)?;
+        let clear = e.is_none();
         if self.iblock == undef(os) {
+            if clear {
+                return Ok(());
+            }
             self.create_iblock(img)?;
         }
         let ib = self.iblock;
@@ -420,6 +431,9 @@ impl Ea {
                 let dblk_idx = l.start_dblk + local;
                 let slot = dblks_at + dblk_idx * osz;
                 let mut addr = get_uint(&img.read(slot, os as usize)?, os);
+                if addr == undef(os) && clear {
+                    return Ok(());
+                }
                 if addr == undef(os) {
                     // libhdf5 records start_idx + (global data block index)
                     // * nelmts here (H5EA__lookup_elmt), not the block's
@@ -449,6 +463,9 @@ impl Ea {
                 let sb_prefix = self.dblk_prefix_len(os);
                 let sb_len = sb_prefix + bitmap_len + l.ndblks * osz;
                 let mut sb = get_uint(&img.read(sslot, os as usize)?, os);
+                if sb == undef(os) && clear {
+                    return Ok(());
+                }
                 if sb == undef(os) {
                     let mut d = self.block_prefix(b"EASB", l.start_idx, os);
                     d.resize(d.len() + bitmap_len as usize, 0);
@@ -471,6 +488,9 @@ impl Ea {
                 let local = (rel - l.start_idx) / l.dblk_nelmts;
                 let dslot = sb + sb_prefix + bitmap_len + local * osz;
                 let mut addr = get_uint(&img.read(dslot, os as usize)?, os);
+                if addr == undef(os) && clear {
+                    return Ok(());
+                }
                 if addr == undef(os) {
                     let off = l.start_idx + local * l.dblk_nelmts;
                     addr = self.create_dblock(img, l.dblk_nelmts, off)?;
@@ -491,6 +511,9 @@ impl Ea {
                     let bpos = sb + sb_prefix + bit / 8;
                     let mut byte = img.read(bpos, 1)?[0];
                     let mask = 0x80u8 >> (bit % 8);
+                    if byte & mask == 0 && clear {
+                        return Ok(());
+                    }
                     if byte & mask == 0 {
                         let fill = self.fill_elems(page, os)?;
                         img.write(page_at, &fill)?;
@@ -503,7 +526,7 @@ impl Ea {
                 }
             }
         }
-        if idx + 1 > self.stats[4] {
+        if !clear && idx + 1 > self.stats[4] {
             self.stats[4] = idx + 1;
             self.dirty_hdr = true;
         }

crates/clawhdf5/src/edit/farray.rs

@@ -137,13 +137,19 @@ impl Fa {
         Ok((fa, hdr))
     }
 
-    /// Set element `idx` to `e`.
+    /// Set element `idx` to `e`, or back to the fill element (`None`: a
-    pub(crate) fn set(&mut self, img: &mut Image<'_>, idx: u64, e: Elem) -> Result<(), Error> {
+    /// removed chunk, `H5D__farray_idx_remove`), which creates no page.
+    pub(crate) fn set(
+        &mut self,
+        img: &mut Image<'_>,
+        idx: u64,
+        e: Option<Elem>,
+    ) -> Result<(), Error> {
         let os = img.os;
         if idx >= self.nelmts {
             return Err(bad("index beyond the array"));
         }
-        let enc = encode_elem(Some(e), self.filtered, self.elem_size, os)?;
+        let enc = encode_elem(e, self.filtered, self.elem_size, os)?;
         let es = self.slot(os);
         let prefix = 6 + u64::from(os);
         let page = self.page();
@@ -162,6 +168,9 @@ impl Fa {
         let bpos = self.dblk + prefix + p / 8;
         let mut byte = img.read(bpos, 1)?[0];
         let mask = 0x80u8 >> (p % 8);
+        if byte & mask == 0 && e.is_none() {
+            return Ok(());
+        }
         if byte & mask == 0 {
             let fill = encode_elem(None, self.filtered, self.elem_size, os)?;
             img.write(page_at, &fill.repeat(count as usize))?;

crates/clawhdf5/src/edit/image.rs

@@ -35,6 +35,61 @@ pub(crate) struct Image<'a> {
     /// Width of addresses and lengths in the file.
     pub(crate) os: u8,
     pub(crate) ls: u8,
+    /// Space the edit stopped using. Not reused by this edit: until the
+    /// edit is committed, the file's metadata still points at it.
+    freed: Vec<(u64, u64)>,
+    /// Space earlier edits of the session freed, available to this one.
+    reusable: FreeList,
+    /// Blocks this edit took from `reusable`: nothing on disk refers to
+    /// them, so they are written with the new space, before the changes
+    /// that link them in (see [`Plan::commit`]).
+    fresh: Vec<(u64, u64)>,
+}
+
+/// Free space, address -> length, adjacent blocks merged.
+#[derive(Debug, Clone, Default)]
+pub(crate) struct FreeList(BTreeMap<u64, u64>);
+
+impl FreeList {
+    /// Add `[addr, addr + len)`, merged with neighbours it touches.
+    pub(crate) fn add(&mut self, addr: u64, len: u64) {
+        if len == 0 {
+            return;
+        }
+        let (mut lo, mut hi) = (addr, addr.saturating_add(len));
+        if let Some((&a, &l)) = self.0.range(..=lo).next_back()
+            && a + l >= lo
+        {
+            lo = a;
+            hi = hi.max(a + l);
+            self.0.remove(&a);
+        }
+        while let Some((&a, &l)) = self.0.range(lo..=hi).next() {
+            hi = hi.max(a + l);
+            self.0.remove(&a);
+        }
+        self.0.insert(lo, hi - lo);
+    }
+
+    /// Take `size` bytes from the smallest block that holds them (the
+    /// lowest address among equals), from its start.
+    fn take(&mut self, size: u64) -> Option<u64> {
+        let (&a, &l) = self
+            .0
+            .iter()
+            .filter(|&(_, &l)| l >= size)
+            .min_by_key(|&(&a, &l)| (l, a))?;
+        self.0.remove(&a);
+        if l > size {
+            self.0.insert(a + size, l - size);
+        }
+        Some(a)
+    }
+
+    /// Total bytes.
+    pub(crate) fn total(&self) -> u64 {
+        self.0.values().sum()
+    }
 }
 
 impl<'a> Image<'a> {
@@ -47,9 +102,24 @@ impl<'a> Image<'a> {
             old_eoa: eoa,
             os,
             ls,
+            freed: Vec::new(),
+            reusable: FreeList::default(),
+            fresh: Vec::new(),
         }
     }
 
+    /// Let the edit allocate from `free` (space earlier edits freed).
+    pub(crate) fn with_reusable(mut self, free: FreeList) -> Self {
+        // Only space inside the file as it is now.
+        self.reusable = FreeList(
+            free.0
+                .into_iter()
+                .filter(|&(a, l)| a.saturating_add(l) <= self.old_eoa)
+                .collect(),
+        );
+        self
+    }
+
     pub(crate) fn eoa(&self) -> u64 {
         self.eoa
     }
@@ -63,11 +133,24 @@ impl<'a> Image<'a> {
         !self.patches.is_empty() || self.eoa != self.old_eoa
     }
 
-    /// Allocate `size` bytes at the end of the file. The space reads as
+    /// Allocate `size` bytes: from space an earlier edit of this session
-    /// zeros until written. Nothing is ever freed: space an edit stops
+    /// freed when a block holds them (best fit), else at the end of the
-    /// using (a relocated chunk, say) is leaked, as there is no free-space
+    /// file. The space reads as zeros until written.
-    /// manager.
     pub(crate) fn alloc(&mut self, size: u64) -> Result<u64, Error> {
+        if size > 0
+            && let Some(a) = self.reusable.take(size)
+        {
+            self.fresh.push((a, size));
+            let n = usize::try_from(size)
+                .map_err(|_| Error::Unsupported("allocation too large".into()))?;
+            self.write(a, &vec![0u8; n])?;
+            return Ok(a);
+        }
+        self.alloc_end(size)
+    }
+
+    /// Allocate `size` bytes at the end of the file.
+    fn alloc_end(&mut self, size: u64) -> Result<u64, Error> {
         let addr = self.eoa;
         let end = addr
             .checked_add(size)
@@ -77,6 +160,14 @@ impl<'a> Image<'a> {
         Ok(addr)
     }
 
+    /// Note that the edit no longer uses `[addr, addr + len)`; later edits
+    /// of the session may reuse it.
+    pub(crate) fn free(&mut self, addr: u64, len: u64) {
+        if len > 0 {
+            self.freed.push((addr, len));
+        }
+    }
+
     /// If `[addr, addr + old_len)` is the last allocated space, grow it to
     /// `new_len` bytes (a structure at the end of the file can grow where
     /// it is) and return true.
@@ -91,7 +182,7 @@ impl<'a> Image<'a> {
         }
         let old_end = self.eoa;
         self.eoa = addr;
-        if let Err(e) = self.alloc(new_len) {
+        if let Err(e) = self.alloc_end(new_len) {
             self.eoa = old_end;
             return Err(e);
         }
@@ -189,12 +280,24 @@ impl<'a> Image<'a> {
     /// The edit's writes, detached from the base bytes (see the module's
     /// invariant: the reader that owns them can then be dropped before
     /// anything is written).
-    pub(crate) fn into_plan(self) -> Plan {
+    pub(crate) fn into_plan(self) -> (Plan, FreeList) {
-        Plan {
+        // What the session may reuse once this edit is committed: what it
-            patches: self.patches,
+        // did not take, and what it freed.
-            eoa: self.eoa,
+        let mut free = self.reusable;
-            old_eoa: self.old_eoa,
+        for (a, l) in self.freed {
+            free.add(a, l);
         }
+        let mut fresh = self.fresh;
+        fresh.sort_unstable();
+        (
+            Plan {
+                patches: self.patches,
+                eoa: self.eoa,
+                old_eoa: self.old_eoa,
+                fresh,
+            },
+            free,
+        )
     }
 }
 
@@ -203,33 +306,57 @@ pub(crate) struct Plan {
     patches: BTreeMap<u64, Vec<u8>>,
     eoa: u64,
     old_eoa: u64,
+    /// Reused blocks (sorted): written with the new space.
+    fresh: Vec<(u64, u64)>,
 }
 
 impl Plan {
+    /// Whether `addr` is in space nothing on disk refers to yet (past the
+    /// old end of file, or in a reused block), and up to where (before
+    /// `end`) that stays so.
+    fn new_space(&self, addr: u64, end: u64) -> (bool, u64) {
+        if addr >= self.old_eoa {
+            return (true, end);
+        }
+        let limit = end.min(self.old_eoa);
+        // The reused block holding `addr`, or the next one after it.
+        let i = self.fresh.partition_point(|&(a, l)| a + l <= addr);
+        match self.fresh.get(i) {
+            Some(&(a, l)) if a <= addr => (true, limit.min(a + l)),
+            Some(&(a, _)) => (false, limit.min(a)),
+            None => (false, limit),
+        }
+    }
+
     /// Write the edit to `file`, whose superblock is at `user_block`.
     ///
     /// Order: first everything in newly allocated space (new chunks, new
-    /// index blocks, relocated structures), which nothing on disk refers to
+    /// index blocks, relocated structures — past the old end of file, or in
-    /// yet, then a sync; then the changes to existing bytes — raw data
+    /// space an earlier edit of the session freed), which nothing on disk
-    /// overwritten in place and the metadata that links the new space in
+    /// refers to yet, then a sync; then the changes to existing bytes — raw
-    /// (superblock end of file, chunk index entries, object header
+    /// data overwritten in place and the metadata that links the new space
+    /// in (superblock end of file, chunk index entries, object header
     /// messages) — then a sync. A crash during the first phase leaves the
-    /// file as it was (plus unreferenced bytes past its end of file); a
+    /// file as it was (plus unreferenced bytes); a crash during the second
-    /// crash during the second can leave it inconsistent, as with libhdf5
+    /// can leave it inconsistent, as with libhdf5 without SWMR: there is no
-    /// without SWMR: there is no journal.
+    /// journal.
     pub(crate) fn commit(self, file: &mut std::fs::File, user_block: u64) -> Result<(), Error> {
         let old_eoa = self.old_eoa;
         let mut in_place: Vec<(u64, &[u8])> = Vec::new();
         for (&addr, bytes) in &self.patches {
-            // A patch may run from existing bytes into new space (writes
+            // A patch may run across new and existing space (writes
-            // merge); its new part goes with the new space.
+            // merge): split it where that changes.
-            let split = old_eoa.saturating_sub(addr).min(bytes.len() as u64) as usize;
+            let end = addr + bytes.len() as u64;
-            let (old, new) = bytes.split_at(split);
+            let mut at = addr;
-            if !new.is_empty() {
+            while at < end {
-                write_at(file, user_block + addr + split as u64, new)?;
+                let (new, upto) = self.new_space(at, end);
-            }
+                let part = &bytes[(at - addr) as usize..(upto - addr) as usize];
-            if !old.is_empty() {
+                if new {
-                in_place.push((addr, old));
+                    write_at(file, user_block + at, part)?;
+                } else {
+                    in_place.push((at, part));
+                }
+                at = upto;
             }
         }
         if self.eoa > old_eoa {
@@ -305,6 +432,52 @@ mod tests {
         assert!(img.write(42, &[1]).is_err());
     }
 
+    #[test]
+    fn free_list_merges_and_takes_best_fit() {
+        let mut f = FreeList::default();
+        f.add(100, 10);
+        f.add(120, 5);
+        f.add(110, 10); // joins both neighbours
+        assert_eq!(
+            f.0.iter().map(|(&a, &l)| (a, l)).collect::<Vec<_>>(),
+            [(100, 25)]
+        );
+        f.add(300, 8);
+        f.add(200, 40);
+        // Best fit: the 8-byte block for 6 bytes, from its start.
+        assert_eq!(f.take(6), Some(300));
+        assert_eq!(f.take(30), Some(200));
+        assert_eq!(f.take(26), None);
+        assert_eq!(f.total(), 25 + 2 + 10);
+    }
+
+    /// An edit allocates from space earlier edits freed (zeroed), never
+    /// from what it frees itself; the plan writes reused blocks with the
+    /// new space.
+    #[test]
+    fn reuse_across_edits_only() {
+        let base = vec![7u8; 64];
+        let mut free = FreeList::default();
+        free.add(8, 16);
+        let mut img = Image::new(&base, 8, 8).with_reusable(free);
+        img.free(32, 16); // freed by this edit: not reusable yet
+        let a = img.alloc(16).unwrap();
+        assert_eq!(a, 8);
+        assert_eq!(img.read(8, 16).unwrap(), vec![0u8; 16]);
+        let b = img.alloc(8).unwrap();
+        assert_eq!(b, 64, "the edit's own freed space is not reused");
+        img.write(4, &[1; 8]).unwrap(); // existing bytes 4..8, reused 8..12
+        let (plan, next) = img.into_plan();
+        assert_eq!(plan.new_space(4, 12), (false, 8));
+        assert_eq!(plan.new_space(8, 12), (true, 12));
+        assert_eq!(plan.new_space(30, 40), (false, 40));
+        assert_eq!(plan.new_space(64, 72), (true, 72));
+        assert_eq!(
+            next.0.iter().map(|(&a, &l)| (a, l)).collect::<Vec<_>>(),
+            [(32, 16)]
+        );
+    }
+
     /// Random reads and writes against a flat copy of the bytes.
     #[test]
     fn matches_a_flat_model() {

crates/clawhdf5/src/edit/ohdr.rs

@@ -59,7 +59,7 @@ pub(crate) struct Header {
     added: usize,
 }
 
-const MAX_CHUNKS: usize = 1024;
+const MAX_CHUNKS: usize = 1 << 16;
 
 fn corrupt(why: &'static str) -> Error {
     Error::Format(FormatError::InvalidObjectHeader(why))
@@ -118,7 +118,11 @@ impl Header {
             });
             h.scan(img, 0, addr + 16, addr + 16 + size, &mut pending)?;
         }
-        while let Some((caddr, clen)) = pending.pop() {
+        // Continuation chunks in the order their messages are found, as
+        // H5O_protect loads them (so messages keep libhdf5's order).
+        let mut next = 0;
+        while let Some(&(caddr, clen)) = pending.get(next) {
+            next += 1;
             if h.chunks.len() >= MAX_CHUNKS {
                 return Err(corrupt("too many object header chunks"));
             }

crates/clawhdf5/src/lib.rs

@@ -56,7 +56,7 @@ pub use error::Error;
 pub use lazy::{LazyDataset, LazyFile, LazyGroup};
 #[cfg(feature = "mmap")]
 pub use mmap_file::{MmapDataset, MmapFile, MmapGroup};
-pub use reader::{Dataset, File, Group};
+pub use reader::{Dataset, File, Group, SharedStorage, VdsResolver};
 pub use types::{AttrValue, DType};
 pub use vlen::VlenValue;
 pub use writer::FileBuilder;
@@ -72,6 +72,7 @@ pub use clawhdf5_format::property_list::{
 #[cfg(feature = "provenance")]
 pub use clawhdf5_format::provenance;
 pub use clawhdf5_format::selection::Selection;
+pub use clawhdf5_format::storage::Storage;
 pub use clawhdf5_format::superblock::swmr_flags;
 pub use clawhdf5_format::type_builders::{CompoundTypeBuilder, EnumTypeBuilder, FillTime};
 

crates/clawhdf5/src/reader.rs

@@ -5,7 +5,10 @@
 //! the traditional read-into-`Vec<u8>` fallback.  [`File::from_bytes`] remains
 //! available for in-memory usage (tests, etc.).
 
+use std::borrow::Cow;
 use std::collections::HashMap;
+use std::ops::Range;
+use std::sync::Arc;
 
 use clawhdf5_format::chunk_cache::ChunkCache;
 use clawhdf5_format::data_layout::DataLayout;
@@ -19,43 +22,79 @@ use clawhdf5_format::group_v2;
 use clawhdf5_format::message_type::MessageType;
 use clawhdf5_format::object_header::ObjectHeader;
 use clawhdf5_format::signature;
+use clawhdf5_format::storage::Storage;
 use clawhdf5_format::superblock::Superblock;
+use clawhdf5_format::superblock_ext::{self, CacheImageState};
 
 use crate::cache_image::{self, ImageView};
 use crate::error::Error;
 use crate::types::{AttrValue, DType, classify_datatype, read_attr, read_attrs};
 
 // ---------------------------------------------------------------------------
-// FileData — internal storage for either owned bytes or an mmap
+// FileData — internal storage for owned bytes, an mmap, or any Storage
 // ---------------------------------------------------------------------------
 
-/// Internal storage: either an owned `Vec<u8>` or a memory-mapped region.
+/// A [`Storage`] a [`File`] can be opened over: any backend that can be
+/// shared between threads (see [`File::open_storage`]).
+pub type SharedStorage = Arc<dyn Storage + Send + Sync>;
+
+/// Resolves an external Virtual Dataset source file name to that file's
+/// bytes (`Ok(None)`: the file does not exist, its mappings read as the
+/// fill value). See [`File::set_vds_resolver`].
+pub type VdsResolver = Arc<dyn Fn(&str) -> Result<Option<Vec<u8>>, FormatError> + Send + Sync>;
+
+/// Internal storage: an owned `Vec<u8>`, a memory-mapped region, or a
+/// [`Storage`] backend.
 enum Backing {
     Owned(Vec<u8>),
     #[cfg(feature = "mmap")]
     Mmap(clawhdf5_io::MmapReader),
+    Storage(SharedStorage),
 }
 
 impl Backing {
-    fn whole_file(&self) -> &[u8] {
+    /// The whole file, when it is in memory.
+    fn whole_file(&self) -> Option<&[u8]> {
         match self {
-            Backing::Owned(v) => v,
+            Backing::Owned(v) => Some(v),
             #[cfg(feature = "mmap")]
-            Backing::Mmap(r) => r.as_bytes(),
+            Backing::Mmap(r) => Some(r.as_bytes()),
+            Backing::Storage(s) => s.as_contiguous(),
         }
     }
 }
 
+/// Evaluate `$body` with `$d` bound to the bytes the `clawhdf5_format`
+/// parsers read: the in-memory slice when the file has one (a `Vec`, an
+/// mmap), so local files run the parsers monomorphised for `[u8]` — the
+/// slice code, as before the range-read migration — and otherwise the
+/// [`FileData`] itself, whose reads go to the storage.
+macro_rules! with_bytes {
+    ($data:expr, |$d:ident| $body:expr) => {{
+        let data: &FileData = $data;
+        match data.contiguous() {
+            Some($d) => $body,
+            None => {
+                let $d = data;
+                $body
+            }
+        }
+    }};
+}
+
 /// The file's bytes, viewed from the superblock on and up to the end of
 /// file the superblock records. A file may start with a user block (the
 /// superblock at 512, 1024, …); every HDF5 address is relative to the
-/// superblock, so all parsing goes through [`Self::as_bytes`].
+/// superblock, so all parsing goes through this view, which is a
+/// [`Storage`]: in memory (a `Vec`, an mmap) its reads are slices of the
+/// file, as before; over any other storage they are reads of that storage,
+/// shifted by the user block and bounded by the end of file.
 struct FileData {
     backing: Backing,
     /// Offset of the superblock in the file (the user-block size).
-    base: usize,
+    base: u64,
     /// End of the HDF5 data in the file (`Superblock::data_end`, absolute).
-    end: usize,
+    end: u64,
     /// A mapped file that holds a metadata cache image, with the image
     /// written in: a private copy-on-write mapping of the whole file, so
     /// only the pages the image's entries land on are copied (see
@@ -63,18 +102,45 @@ struct FileData {
     /// an image is read straight from the mapping, and an owned buffer has
     /// the image written into it in place.
     patched: Option<clawhdf5_io::PrivateCopy>,
+    /// A [`Storage`]-backed file's metadata cache image: its entries
+    /// (address relative to the superblock, bytes), laid over every read in
+    /// order, as libhdf5 reads them instead of the file's own bytes.
+    overlay: Vec<(u64, Vec<u8>)>,
     /// The file has a metadata cache image libhdf5 cannot load. libhdf5
     /// opens such a file and fails its first metadata read (the image loads
     /// then); every object lookup here fails with this error, and no
     /// metadata is read from the file's own, possibly stale, bytes.
     image_error: Option<FormatError>,
+    /// [`Self::contiguous`], worked out once at open: every structure a
+    /// parser reads asks for it, and the patched/overlay checks and range
+    /// conversions behind it cost a local metadata walk a few percent.
+    contiguous: Option<WholeView>,
 }
 
+/// A borrow of the HDF5 data held by a [`FileData`]'s own `backing` or
+/// `patched` buffer (see [`FileData::contiguous`]).
+#[derive(Clone, Copy)]
+struct WholeView {
+    ptr: *const u8,
+    len: usize,
+}
+
+// SAFETY: a `WholeView` is only a borrow of bytes owned (through `backing`
+// or `patched`) by the `FileData` holding it, which is `Send + Sync`: the
+// bytes are never written after open, so sharing the pointer across
+// threads is sharing a `&[u8]`.
+unsafe impl Send for WholeView {}
+// SAFETY: as above.
+unsafe impl Sync for WholeView {}
+
 impl FileData {
     /// Locate the superblock and parse it. A truncated file is refused, and
     /// bytes past the recorded end of file are not read, as in libhdf5.
     fn new(mut backing: Backing) -> Result<(Self, Superblock), Error> {
-        let whole = backing.whole_file();
+        if let Backing::Storage(storage) = backing {
+            return Self::new_storage(storage);
+        }
+        let whole = backing.whole_file().unwrap_or_default();
         let (user_block, hdf5) = signature::split_user_block(whole)?;
         let base = user_block.len();
         let superblock = Superblock::parse(hdf5, 0)?;
@@ -92,41 +158,198 @@ impl FileData {
                     clawhdf5_io::HDF5Read::private_copy(r)
                 })?
             }
+            Backing::Storage(_) => unreachable!("handled above"),
         };
         let (patched, image_error) = match view {
             ImageView::Plain => (None, None),
             ImageView::Patched(p) => (Some(p), None),
             ImageView::Unloadable(e) => (None, Some(e)),
         };
-        Ok((
+        let mut data = Self {
-            Self {
+            backing,
-                backing,
+            base: base as u64,
-                base,
+            end: end as u64,
-                end,
+            patched,
-                patched,
+            overlay: Vec::new(),
-                image_error,
+            image_error,
-            },
+            contiguous: None,
-            superblock,
+        };
-        ))
+        data.contiguous = data.find_contiguous();
+        Ok((data, superblock))
     }
 
-    fn as_bytes(&self) -> &[u8] {
+    /// [`Self::new`] for a [`Storage`] backend: the same checks, through
-        match &self.patched {
+    /// reads of the storage.
-            Some(p) => &p[self.base..self.end],
+    fn new_storage(storage: SharedStorage) -> Result<(Self, Superblock), Error> {
-            None => &self.backing.whole_file()[self.base..self.end],
+        let file_len = storage.len();
+        let base = signature::find_signature_in(&*storage)?;
+        let mut data = Self {
+            backing: Backing::Storage(storage),
+            base,
+            end: file_len,
+            patched: None,
+            overlay: Vec::new(),
+            image_error: None,
+            contiguous: None,
+        };
+        // Worked out again below, once the end of file and any cache image
+        // are known.
+        data.contiguous = data.find_contiguous();
+        let superblock = Superblock::parse_in(&data, 0)?;
+        data.end = base + superblock.data_end(base, file_len)?;
+        match superblock_ext::cache_image_state_in(&data, &superblock)? {
+            CacheImageState::Absent => {}
+            CacheImageState::Unloadable(e) => data.image_error = Some(e),
+            CacheImageState::Loaded(image) => {
+                let block = image.block_in(&data)?.into_owned();
+                data.overlay = image
+                    .entries(&block)?
+                    .into_iter()
+                    .map(|(addr, bytes)| (addr, bytes.to_vec()))
+                    .collect();
+            }
         }
+        data.contiguous = data.find_contiguous();
+        Ok((data, superblock))
     }
 
-    fn len(&self) -> usize {
+    /// The HDF5 data as one slice, when the file is in memory (a `Vec`, an
-        self.as_bytes().len()
+    /// mmap, or a storage that holds it all and has no cache image to lay
+    /// over it).
+    #[inline]
+    fn contiguous(&self) -> Option<&[u8]> {
+        // SAFETY: `find_contiguous` borrowed these bytes from `backing` or
+        // `patched`, which this `FileData` owns and never changes after
+        // open. They live on the heap or in a mapping (a `Vec`'s buffer, an
+        // mmap, a private copy, or a buffer inside the `Arc`'d storage), so
+        // they stay put when the `FileData` moves, and they live as long as
+        // `self`.
+        self.contiguous
+            .map(|v| unsafe { core::slice::from_raw_parts(v.ptr, v.len) })
+    }
+
+    /// [`Self::contiguous`], worked out from `backing` and `patched`.
+    fn find_contiguous(&self) -> Option<WholeView> {
+        let bytes = self.compute_contiguous()?;
+        Some(WholeView {
+            ptr: bytes.as_ptr(),
+            len: bytes.len(),
+        })
+    }
+
+    /// The HDF5 data as one slice, from `backing` and `patched` (see
+    /// [`Self::contiguous`]).
+    fn compute_contiguous(&self) -> Option<&[u8]> {
+        if let Some(p) = &self.patched {
+            return p.get(usize::try_from(self.base).ok()?..usize::try_from(self.end).ok()?);
+        }
+        if !self.overlay.is_empty() {
+            return None;
+        }
+        self.backing
+            .whole_file()?
+            .get(usize::try_from(self.base).ok()?..usize::try_from(self.end).ok()?)
     }
 
     /// The bytes to read metadata from; fails for a file whose cache image
     /// cannot be loaded (see [`Self::image_error`]).
-    fn meta(&self) -> Result<&[u8], FormatError> {
+    fn meta(&self) -> Result<&Self, FormatError> {
         match &self.image_error {
             Some(e) => Err(e.clone()),
-            None => Ok(self.as_bytes()),
+            None => Ok(self),
+        }
+    }
+
+    /// The backend of a file that is not in memory.
+    fn remote(&self) -> Result<&SharedStorage, FormatError> {
+        match &self.backing {
+            Backing::Storage(s) => Ok(s),
+            _ => Err(FormatError::Storage(
+                "in-memory file has no contiguous view".into(),
+            )),
+        }
+    }
+
+    /// `bytes`, read at `offset`, with the cache image entries they overlap
+    /// written over them.
+    fn with_overlay<'a>(&self, offset: u64, mut bytes: Cow<'a, [u8]>) -> Cow<'a, [u8]> {
+        let end = offset + bytes.len() as u64;
+        for (addr, entry) in &self.overlay {
+            let entry_end = addr + entry.len() as u64;
+            if *addr >= end || entry_end <= offset {
+                continue;
+            }
+            let from = (*addr).max(offset);
+            let to = entry_end.min(end);
+            let dst = bytes.to_mut();
+            dst[(from - offset) as usize..(to - offset) as usize]
+                .copy_from_slice(&entry[(from - addr) as usize..(to - addr) as usize]);
+        }
+        bytes
+    }
+}
+
+impl Storage for FileData {
+    #[inline]
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        if let Some(all) = self.contiguous() {
+            return all.read_at(offset, len);
+        }
+        let size = self.end - self.base;
+        let len = usize::try_from(size.saturating_sub(offset)).map_or(len, |avail| avail.min(len));
+        if len == 0 {
+            return Ok(Cow::Borrowed(&[]));
+        }
+        let bytes = cut_to(self.remote()?.read_at(self.base + offset, len)?, len);
+        Ok(self.with_overlay(offset, bytes))
+    }
+
+    fn len(&self) -> u64 {
+        self.end - self.base
+    }
+
+    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
+        if let Some(all) = self.contiguous() {
+            return all.read_ranges(ranges);
+        }
+        let size = self.end - self.base;
+        let shifted: Vec<Range<u64>> = ranges
+            .iter()
+            .map(|r| {
+                let (s, e) = (r.start.min(size), r.end.min(size));
+                self.base + s..self.base + e.max(s)
+            })
+            .collect();
+        let got = self.remote()?.read_ranges(&shifted)?;
+        Ok(got
+            .into_iter()
+            .zip(ranges.iter().zip(&shifted))
+            .map(|(bytes, (r, asked))| {
+                let bytes = cut_to(bytes, (asked.end - asked.start) as usize);
+                self.with_overlay(r.start, bytes)
+            })
+            .collect())
+    }
+
+    #[inline]
+    fn as_contiguous(&self) -> Option<&[u8]> {
+        self.contiguous()
+    }
+}
+
+/// `bytes`, a backend's answer to a read of `len` bytes, without anything
+/// past those `len` (a backend that returns more than asked breaks
+/// [`Storage::read_at`]'s contract; the extra bytes are not the file's). A
+/// short answer is passed on: the parsers' own checks refuse it.
+fn cut_to(bytes: Cow<'_, [u8]>, len: usize) -> Cow<'_, [u8]> {
+    if bytes.len() <= len {
+        return bytes;
+    }
+    match bytes {
+        Cow::Borrowed(b) => Cow::Borrowed(&b[..len]),
+        Cow::Owned(mut v) => {
+            v.truncate(len);
+            Cow::Owned(v)
         }
     }
 }
@@ -149,6 +372,9 @@ pub struct File {
     /// Directory the file was opened from, used to resolve external Virtual
     /// Dataset source files relative to this file. `None` for in-memory files.
     base_dir: Option<std::path::PathBuf>,
+    /// Resolves external Virtual Dataset source files instead of
+    /// `base_dir` (see [`File::set_vds_resolver`]).
+    vds_resolver: Option<VdsResolver>,
 }
 
 impl File {
@@ -167,6 +393,7 @@ impl File {
                 superblock,
                 chunk_cache: ChunkCache::new(),
                 base_dir,
+                vds_resolver: None,
             })
         }
         #[cfg(not(feature = "mmap"))]
@@ -200,9 +427,55 @@ impl File {
             superblock,
             chunk_cache: ChunkCache::new(),
             base_dir: None,
+            vds_resolver: None,
         })
     }
 
+    /// Open an HDF5 file served by any [`Storage`]: a range-reading remote
+    /// backend, a block cache, or an in-memory buffer — through the same
+    /// read API as [`File::open`] (groups, datasets, attributes, `read_*`,
+    /// selections, variable-length data, virtual datasets).
+    ///
+    /// Every read goes through the storage's [`Storage::read_at`] and
+    /// [`Storage::read_ranges`] (a chunked read fetches the chunks it needs
+    /// with one `read_ranges` call per batch of at most 64 MiB, decoding
+    /// each batch before the next, and never more of a chunk than its
+    /// decoded size can need), so nothing is read that the operation does
+    /// not need. A storage that has the whole file in
+    /// memory ([`Storage::as_contiguous`]) is read as [`File::from_bytes`]
+    /// reads its buffer. The storage holds the whole file: a user block is
+    /// found and skipped, and bytes past the end of file the superblock
+    /// records are not read. A metadata cache image is laid over the reads
+    /// it covers, as libhdf5 loads it.
+    ///
+    /// The zero-copy methods ([`Dataset::read_raw_ref`],
+    /// [`Dataset::read_as_slice`], `read_*_zerocopy`) borrow the file's bytes
+    /// and return [`FormatError::ContiguousStorageRequired`] over a storage
+    /// that does not hold them; [`File::as_bytes`] panics there (use
+    /// [`File::contiguous_bytes`]). External virtual-dataset source files
+    /// are read through [`File::set_vds_resolver`]; without one they cannot
+    /// be resolved.
+    pub fn open_storage(storage: SharedStorage) -> Result<Self, Error> {
+        let (data, superblock) = FileData::new(Backing::Storage(storage))?;
+        Ok(Self {
+            data,
+            superblock,
+            chunk_cache: ChunkCache::new(),
+            base_dir: None,
+            vds_resolver: None,
+        })
+    }
+
+    /// Resolve external Virtual Dataset source files (their names as the
+    /// mappings store them) with `resolver`, instead of reading them from
+    /// the directory of the file (for [`File::open`]) or refusing them (for
+    /// in-memory and [`Storage`]-backed files). `Ok(None)` means the source
+    /// file does not exist, and its mappings read as the fill value, as in
+    /// libhdf5; `Err` fails the read.
+    pub fn set_vds_resolver(&mut self, resolver: VdsResolver) {
+        self.vds_resolver = Some(resolver);
+    }
+
     /// Returns a handle to the root group.
     pub fn root(&self) -> Group<'_> {
         Group {
@@ -215,8 +488,11 @@ impl File {
     ///
     /// The path uses `/` separators (e.g., `"group1/values"`).
     pub fn dataset(&self, path: &str) -> Result<Dataset<'_>, Error> {
-        let data = self.data.meta()?;
+        let addr = with_bytes!(self.data.meta()?, |d| group_v2::resolve_path_any_in(
-        let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
+            d,
+            &self.superblock,
+            path
+        ))?;
         let hdr = self.parse_header(addr)?;
         if !has_message(&hdr, MessageType::DataLayout) {
             return Err(Error::NotADataset(path.to_string()));
@@ -262,8 +538,11 @@ impl File {
     /// The path uses `/` separators (e.g., `"sensors"`).
     /// Use `"/"` or `""` for the root group.
     pub fn group(&self, path: &str) -> Result<Group<'_>, Error> {
-        let data = self.data.meta()?;
+        let addr = with_bytes!(self.data.meta()?, |d| group_v2::resolve_path_any_in(
-        let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
+            d,
+            &self.superblock,
+            path
+        ))?;
         Ok(Group {
             file: self,
             address: addr,
@@ -322,8 +601,33 @@ impl File {
     /// with a metadata cache image these are the bytes with the image
     /// applied; when the image cannot be loaded they are the file's own
     /// bytes, whose metadata may be stale (every object lookup fails then).
+    ///
+    /// # Panics
+    ///
+    /// For a file opened with [`File::open_storage`] over a storage that
+    /// does not hold the whole file in memory; use
+    /// [`contiguous_bytes`](Self::contiguous_bytes) there.
     pub fn as_bytes(&self) -> &[u8] {
-        self.data.as_bytes()
+        self.data
+            .contiguous()
+            .expect("File::as_bytes: the file is not in memory (see File::contiguous_bytes)")
+    }
+
+    /// [`as_bytes`](Self::as_bytes), or `None` for a file whose bytes are
+    /// not all in memory ([`File::open_storage`]).
+    pub fn contiguous_bytes(&self) -> Option<&[u8]> {
+        self.data.contiguous()
+    }
+
+    /// The bytes [`as_bytes`](Self::as_bytes) returns, as a [`Storage`],
+    /// for every backend: from the superblock on, bounded by the recorded
+    /// end of file, with a metadata cache image laid over them. Code that
+    /// parses the file itself with the `clawhdf5_format` `*_in` functions
+    /// reads through this, so it works on remote files too; for a file in
+    /// memory its [`Storage::as_contiguous`] is [`as_bytes`](Self::as_bytes)
+    /// (and every read a slice of it).
+    pub fn storage(&self) -> &(dyn Storage + Send + Sync) {
+        &self.data
     }
 
     /// The error of a metadata cache image libhdf5 cannot load, when the
@@ -338,7 +642,7 @@ impl File {
     /// Size of the user block before the superblock (0 for most files).
     /// Matches h5py's `File.userblock_size`.
     pub fn user_block_size(&self) -> u64 {
-        self.data.base as u64
+        self.data.base
     }
 
     /// Returns a reference to the parsed superblock.
@@ -349,7 +653,7 @@ impl File {
     /// Returns `true` when the file is backed by memory-mapped I/O.
     pub fn is_mmap(&self) -> bool {
         match &self.data.backing {
-            Backing::Owned(_) => false,
+            Backing::Owned(_) | Backing::Storage(_) => false,
             #[cfg(feature = "mmap")]
             Backing::Mmap(_) => true,
         }
@@ -361,13 +665,13 @@ impl File {
     /// [`AttrValue::Raw`] attribute. Variable-length strings are resolved in
     /// this file's global heap; see [`Dataset::read_string`] for the values.
     pub fn decode_strings(&self, datatype: &Datatype, raw: &[u8]) -> Result<Vec<String>, Error> {
-        crate::vlen::decode_strings(
+        with_bytes!(&self.data, |d| crate::vlen::decode_strings(
-            self.as_bytes(),
+            d,
             datatype,
             raw,
             self.offset_size(),
             self.length_size(),
-        )
+        ))
     }
 
     /// Like [`decode_strings`](Self::decode_strings) for variable-length
@@ -378,13 +682,13 @@ impl File {
         datatype: &Datatype,
         raw: &[u8],
     ) -> Result<Vec<Vec<u8>>, Error> {
-        crate::vlen::decode_string_bytes(
+        with_bytes!(self.data.meta()?, |d| crate::vlen::decode_string_bytes(
-            self.data.meta()?,
+            d,
             datatype,
             raw,
             self.offset_size(),
             self.length_size(),
-        )
+        ))
     }
 
     /// Decode the variable-length sequences in `raw`, a buffer of elements
@@ -396,22 +700,22 @@ impl File {
         datatype: &Datatype,
         raw: &[u8],
     ) -> Result<Vec<Vec<T>>, Error> {
-        crate::vlen::decode_vlen(
+        with_bytes!(self.data.meta()?, |d| crate::vlen::decode_vlen(
-            self.data.meta()?,
+            d,
             datatype,
             raw,
             self.offset_size(),
             self.length_size(),
-        )
+        ))
     }
 
     fn parse_header(&self, address: u64) -> Result<ObjectHeader, FormatError> {
-        ObjectHeader::parse(
+        with_bytes!(self.data.meta()?, |d| ObjectHeader::parse_in(
-            self.data.meta()?,
+            d,
-            address as usize,
+            address,
             self.superblock.offset_size,
             self.superblock.length_size,
-        )
+        ))
     }
 
     fn offset_size(&self) -> u8 {
@@ -426,7 +730,7 @@ impl File {
 impl std::fmt::Debug for File {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         f.debug_struct("File")
-            .field("size", &self.data.len())
+            .field("size", &Storage::len(&self.data))
             .field("superblock_version", &self.superblock.version)
             .field("mmap", &self.is_mmap())
             .finish()
@@ -489,8 +793,12 @@ impl<'f> Group<'f> {
         &self,
     ) -> Result<(HashMap<String, AttrValue>, Vec<FormatError>), Error> {
         let hdr = self.file.parse_header(self.address)?;
-        let data = self.file.data.as_bytes();
+        with_bytes!(&self.file.data, |d| read_attrs(
-        read_attrs(data, &hdr, self.file.offset_size(), self.file.length_size())
+            d,
+            &hdr,
+            self.file.offset_size(),
+            self.file.length_size()
+        ))
     }
 
     /// Get a dataset within this group by name.
@@ -520,14 +828,13 @@ impl<'f> Group<'f> {
     /// stored densely.
     pub fn attr(&self, name: &str) -> Result<Option<AttrValue>, Error> {
         let hdr = self.file.parse_header(self.address)?;
-        let data = self.file.data.as_bytes();
+        with_bytes!(&self.file.data, |d| read_attr(
-        read_attr(
+            d,
-            data,
             &hdr,
             name,
             self.file.offset_size(),
             self.file.length_size(),
-        )
+        ))
     }
 
     /// The object header address of the child called `name`: the entry of
@@ -535,9 +842,13 @@ impl<'f> Group<'f> {
     /// name index rather than by listing the group (see
     /// [`group_v2::resolve_child`]).
     fn child_address(&self, name: &str) -> Result<u64, Error> {
-        let data = self.file.data.meta()?;
+        with_bytes!(self.file.data.meta()?, |d| group_v2::resolve_child_in(
-        group_v2::resolve_child(data, &self.file.superblock, self.address, name)
+            d,
-            .map_err(Error::Format)
+            &self.file.superblock,
+            self.address,
+            name
+        ))
+        .map_err(Error::Format)
     }
 
     /// This group's children that can be opened, as `(name, object header
@@ -558,9 +869,10 @@ impl<'f> Group<'f> {
     /// [`group_v2::resolve_group_children`]); dangling, external and
     /// user-defined links are left out.
     fn children(&self) -> Result<Vec<GroupEntry>, Error> {
-        let data = self.file.data.meta()?;
+        with_bytes!(self.file.data.meta()?, |d| {
-        group_v2::resolve_group_children(data, &self.file.superblock, self.address)
+            group_v2::resolve_group_children_in(d, &self.file.superblock, self.address)
-            .map_err(Error::Format)
+        })
+        .map_err(Error::Format)
     }
 }
 
@@ -589,7 +901,7 @@ impl<'f> Dataset<'f> {
             Ok((self.datatype()?, ds, self.data_layout()?))
         })();
         if let Ok((dt, ds, dl)) = decoded {
-            data_read::check_dataset_storage(&dl, &ds, &dt, self.file.data.len() as u64)?;
+            data_read::check_dataset_storage(&dl, &ds, &dt, Storage::len(&self.file.data))?;
         }
         Ok(self)
     }
@@ -629,8 +941,8 @@ impl<'f> Dataset<'f> {
         let dt = self.datatype()?;
         // A contiguous dataset is converted straight from the file bytes; going
         // through `read_raw` first copied the whole dataset an extra time.
-        if let Ok(Some(bytes)) = self.read_raw_ref() {
+        if let Ok(Some(bytes)) = self.contiguous_raw() {
-            return Ok(data_read::read_as_f64(bytes, &dt)?);
+            return Ok(data_read::read_as_f64(&bytes, &dt)?);
         }
         if let Some(values) = self.read_chunked_native::<f64>()? {
             return Ok(values);
@@ -649,8 +961,8 @@ impl<'f> Dataset<'f> {
         let dt = self.datatype()?;
         // A contiguous dataset is converted straight from the file bytes; going
         // through `read_raw` first copied the whole dataset an extra time.
-        if let Ok(Some(bytes)) = self.read_raw_ref() {
+        if let Ok(Some(bytes)) = self.contiguous_raw() {
-            return Ok(data_read::read_as_f32(bytes, &dt)?);
+            return Ok(data_read::read_as_f32(&bytes, &dt)?);
         }
         if let Some(values) = self.read_chunked_native::<f32>()? {
             return Ok(values);
@@ -664,8 +976,8 @@ impl<'f> Dataset<'f> {
         let dt = self.datatype()?;
         // A contiguous dataset is converted straight from the file bytes; going
         // through `read_raw` first copied the whole dataset an extra time.
-        if let Ok(Some(bytes)) = self.read_raw_ref() {
+        if let Ok(Some(bytes)) = self.contiguous_raw() {
-            return Ok(data_read::read_as_i32(bytes, &dt)?);
+            return Ok(data_read::read_as_i32(&bytes, &dt)?);
         }
         if let Some(values) = self.read_chunked_native::<i32>()? {
             return Ok(values);
@@ -679,8 +991,8 @@ impl<'f> Dataset<'f> {
         let dt = self.datatype()?;
         // A contiguous dataset is converted straight from the file bytes; going
         // through `read_raw` first copied the whole dataset an extra time.
-        if let Ok(Some(bytes)) = self.read_raw_ref() {
+        if let Ok(Some(bytes)) = self.contiguous_raw() {
-            return Ok(data_read::read_as_i64(bytes, &dt)?);
+            return Ok(data_read::read_as_i64(&bytes, &dt)?);
         }
         if let Some(values) = self.read_chunked_native::<i64>()? {
             return Ok(values);
@@ -694,8 +1006,8 @@ impl<'f> Dataset<'f> {
         let dt = self.datatype()?;
         // A contiguous dataset is converted straight from the file bytes; going
         // through `read_raw` first copied the whole dataset an extra time.
-        if let Ok(Some(bytes)) = self.read_raw_ref() {
+        if let Ok(Some(bytes)) = self.contiguous_raw() {
-            return Ok(data_read::read_as_u64(bytes, &dt)?);
+            return Ok(data_read::read_as_u64(&bytes, &dt)?);
         }
         if let Some(values) = self.read_chunked_native::<u64>()? {
             return Ok(values);
@@ -793,8 +1105,8 @@ impl<'f> Dataset<'f> {
         // sparse) dataset — select from a fill-aware full read instead. (The
         // selection reader currently decodes the full dataset too, so this
         // costs nothing extra.)
-        let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
+        let fill = clawhdf5_format::fill_value::dataset_fill_value_from_storage(
-            self.file.data.as_bytes(),
+            &self.file.data,
             &self.header.messages,
             self.file.offset_size(),
             self.file.length_size(),
@@ -813,8 +1125,8 @@ impl<'f> Dataset<'f> {
                 selection,
             )?);
         }
-        Ok(data_read::read_raw_data_selection(
+        Ok(data_read::read_raw_data_selection_in(
-            self.file.data.as_bytes(),
+            &self.file.data,
             &dl,
             &ds,
             &dt,
@@ -871,12 +1183,26 @@ impl<'f> Dataset<'f> {
             return full();
         }
         let dt = self.datatype()?;
-        if T::is_native(&dt)
+        if T::is_native(&dt) && self.file.data.contiguous().is_some() {
-            && let Ok(Some(raw)) = self.read_raw_ref()
+            if let Ok(Some(raw)) = self.read_raw_ref() {
+                let dims = self.dataspace()?.dimensions;
+                if let Some(values) =
+                    data_read::read_selection_native::<T>(raw, &dims, &dt, selection)?
+                {
+                    return Ok(values);
+                }
+            }
+        } else if T::is_native(&dt)
+            && let (Ok(dl), Ok(ds)) = (self.data_layout(), self.dataspace())
         {
-            let dims = self.dataspace()?.dimensions;
+            // Not in memory: only the selected runs are read.
-            if let Some(values) = data_read::read_selection_native::<T>(raw, &dims, &dt, selection)?
+            if let Some(values) = data_read::read_selection_native_in::<T, _>(
-            {
+                &self.file.data,
+                &dl,
+                &ds,
+                &dt,
+                selection,
+            )? {
                 return Ok(values);
             }
         }
@@ -893,10 +1219,46 @@ impl<'f> Dataset<'f> {
         let dl = self.data_layout()?;
         let ds = self.dataspace()?;
         let dt = self.datatype()?;
-        let slice = data_read::read_raw_data_zerocopy(self.file.data.as_bytes(), &dl, &ds, &dt)?;
+        let Some(bytes) = self.file.data.contiguous() else {
+            // The bytes are not in memory to borrow.
+            return match dl {
+                DataLayout::Contiguous { .. } => {
+                    Err(Error::Format(FormatError::ContiguousStorageRequired(
+                        "a zero-copy read (the file is not in memory)",
+                    )))
+                }
+                _ => Ok(None),
+            };
+        };
+        let slice = data_read::read_raw_data_zerocopy(bytes, &dl, &ds, &dt)?;
         Ok(slice)
     }
 
+    /// A contiguous dataset's stored bytes, for the typed readers' fast
+    /// path: borrowed from the file when it is in memory
+    /// ([`read_raw_ref`](Self::read_raw_ref)), read in one piece otherwise.
+    /// `Ok(None)` for other layouts; an error where `read_raw_ref` fails.
+    fn contiguous_raw(&self) -> Result<Option<Cow<'f, [u8]>>, Error> {
+        if self.file.data.contiguous().is_some() {
+            return Ok(self.read_raw_ref()?.map(Cow::Borrowed));
+        }
+        let dl = self.data_layout()?;
+        if !matches!(dl, DataLayout::Contiguous { .. }) {
+            return Ok(None);
+        }
+        let ds = self.dataspace()?;
+        let dt = self.datatype()?;
+        Ok(Some(Cow::Owned(data_read::read_raw_data_full_in(
+            &self.file.data,
+            &dl,
+            &ds,
+            &dt,
+            None,
+            self.file.offset_size(),
+            self.file.length_size(),
+        )?)))
+    }
+
     /// Zero-copy typed read of contiguous data as `&[T]`.
     ///
     /// Returns a borrowed slice of `T` directly from the file buffer with
@@ -1083,13 +1445,12 @@ impl<'f> Dataset<'f> {
     pub fn attrs_with_errors(
         &self,
     ) -> Result<(HashMap<String, AttrValue>, Vec<FormatError>), Error> {
-        let data = self.file.data.as_bytes();
+        with_bytes!(&self.file.data, |d| read_attrs(
-        read_attrs(
+            d,
-            data,
             &self.header,
             self.file.offset_size(),
             self.file.length_size(),
-        )
+        ))
     }
 
     /// The attribute called `name`, or `None` if it has none by that name
@@ -1097,14 +1458,13 @@ impl<'f> Dataset<'f> {
     /// that name, found without reading the other attributes when they are
     /// stored densely.
     pub fn attr(&self, name: &str) -> Result<Option<AttrValue>, Error> {
-        let data = self.file.data.as_bytes();
+        with_bytes!(&self.file.data, |d| read_attr(
-        read_attr(
+            d,
-            data,
             &self.header,
             name,
             self.file.offset_size(),
             self.file.length_size(),
-        )
+        ))
     }
 
     /// Verify this dataset's content against its stored provenance hash
@@ -1124,8 +1484,8 @@ impl<'f> Dataset<'f> {
     /// result is not a tamper-evidence or authenticity guarantee.
     #[cfg(feature = "provenance")]
     pub fn verify_provenance(&self) -> Result<clawhdf5_format::provenance::VerifyResult, Error> {
-        Ok(clawhdf5_format::provenance::verify_dataset(
+        Ok(clawhdf5_format::provenance::verify_dataset_in(
-            self.file.as_bytes(),
+            &self.file.data,
             &self.header,
             self.file.offset_size(),
             self.file.length_size(),
@@ -1144,12 +1504,14 @@ impl<'f> Dataset<'f> {
             .iter()
             .find(|m| m.msg_type == msg_type)
             .map(|msg| {
-                clawhdf5_format::shared_message::message_data(
+                with_bytes!(&self.file.data, |d| {
-                    self.file.as_bytes(),
+                    clawhdf5_format::shared_message::message_data_in(
-                    msg,
+                        d,
-                    self.file.offset_size(),
+                        msg,
-                    self.file.length_size(),
+                        self.file.offset_size(),
-                )
+                        self.file.length_size(),
+                    )
+                })
                 .map_err(Error::Format)
             })
             .transpose()
@@ -1179,8 +1541,8 @@ impl<'f> Dataset<'f> {
         // one (`H5Dget_space`).
         if let Ok(dl @ DataLayout::Virtual { .. }) = self.data_layout() {
             let resolver = self.vds_resolver();
-            ds.dimensions = clawhdf5_format::vds::virtual_dataset_extent(
+            ds.dimensions = clawhdf5_format::vds::virtual_dataset_extent_in(
-                self.file.data.as_bytes(),
+                &self.file.data,
                 &dl,
                 &ds,
                 self.file.offset_size(),
@@ -1225,9 +1587,9 @@ impl<'f> Dataset<'f> {
         }
         let ds = self.dataspace()?;
         let pipeline = self.filter_pipeline()?;
-        Ok(data_read::read_chunked_native::<T>(
+        Ok(data_read::read_chunked_native_in::<T, _>(
             &self.header.messages,
-            self.file.data.as_bytes(),
+            &self.file.data,
             &dl,
             &ds,
             &dt,
@@ -1251,17 +1613,17 @@ impl<'f> Dataset<'f> {
         }
 
         // Unallocated storage reads as the dataset's fill value.
-        clawhdf5_format::fill_value::read_full_with_fill(
+        clawhdf5_format::fill_value::read_full_with_fill_in(
             &self.header.messages,
-            self.file.data.as_bytes(),
+            &self.file.data,
             &dl,
             &ds,
             dt.type_size() as usize,
             self.file.offset_size(),
             self.file.length_size(),
             || {
-                Ok(data_read::read_raw_data_cached(
+                Ok(data_read::read_raw_data_cached_in(
-                    self.file.data.as_bytes(),
+                    &self.file.data,
                     &dl,
                     &ds,
                     &dt,
@@ -1281,7 +1643,11 @@ impl<'f> Dataset<'f> {
     /// refused with an error rather than read as fill.
     fn vds_resolver(&self) -> impl Fn(&str) -> Result<Option<Vec<u8>>, FormatError> + use<> {
         let base_dir = self.file.base_dir.clone();
+        let custom = self.file.vds_resolver.clone();
         move |name: &str| {
+            if let Some(resolve) = &custom {
+                return resolve(name);
+            }
             let Some(dir) = base_dir.as_ref() else {
                 return Err(FormatError::ChunkedReadError(format!(
                     "virtual dataset source file {name:?} cannot be resolved for an in-memory file"
@@ -1310,15 +1676,15 @@ impl<'f> Dataset<'f> {
         ds: &Dataspace,
         dt: &Datatype,
     ) -> Result<Vec<u8>, Error> {
-        let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
+        let fill = clawhdf5_format::fill_value::dataset_fill_value_from_storage(
-            self.file.data.as_bytes(),
+            &self.file.data,
             &self.header.messages,
             self.file.offset_size(),
             self.file.length_size(),
         )?;
         let resolver = self.vds_resolver();
-        let v = clawhdf5_format::vds::read_virtual_dataset(
+        let v = clawhdf5_format::vds::read_virtual_dataset_in(
-            self.file.data.as_bytes(),
+            &self.file.data,
             dl,
             ds,
             dt,
@@ -1439,7 +1805,7 @@ mod zero_copy_tests {
         let Backing::Mmap(r) = &f.data.backing else {
             return None;
         };
-        let mapped = r.as_bytes()[f.data.base..].as_ptr();
+        let mapped = r.as_bytes()[f.data.base as usize..].as_ptr();
         match &f.data.patched {
             None => Some(std::ptr::eq(f.as_bytes().as_ptr(), mapped)),
             Some(p) => {

crates/clawhdf5/src/types.rs

@@ -158,8 +158,8 @@ pub(crate) fn classify_datatype(dt: &clawhdf5_format::datatype::Datatype) -> DTy
 /// The attributes of the object with header `header` that could be read,
 /// and one error for each that could not (see
 /// [`extract_attributes_tolerant`](clawhdf5_format::attribute::extract_attributes_tolerant)).
-pub(crate) fn read_attrs(
+pub(crate) fn read_attrs<S: clawhdf5_format::storage::Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     header: &clawhdf5_format::object_header::ObjectHeader,
     offset_size: u8,
     length_size: u8,
@@ -170,7 +170,7 @@ pub(crate) fn read_attrs(
     ),
     crate::Error,
 > {
-    let (msgs, errors) = clawhdf5_format::attribute::extract_attributes_tolerant(
+    let (msgs, errors) = clawhdf5_format::attribute::extract_attributes_tolerant_in(
         file_data,
         header,
         offset_size,
@@ -185,14 +185,14 @@ pub(crate) fn read_attrs(
 /// The attribute called `name` on the object with header `header`, decoded
 /// as [`read_attrs`] decodes it, or `None` (see
 /// [`find_attribute_in_file`](clawhdf5_format::attribute::find_attribute_in_file)).
-pub(crate) fn read_attr(
+pub(crate) fn read_attr<S: clawhdf5_format::storage::Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     header: &clawhdf5_format::object_header::ObjectHeader,
     name: &str,
     offset_size: u8,
     length_size: u8,
 ) -> Result<Option<AttrValue>, crate::Error> {
-    let Some(msg) = clawhdf5_format::attribute::find_attribute_in_file(
+    let Some(msg) = clawhdf5_format::attribute::find_attribute_in(
         file_data,
         header,
         name,
@@ -211,9 +211,9 @@ pub(crate) fn read_attr(
     .remove(name))
 }
 
-pub(crate) fn attrs_to_map(
+pub(crate) fn attrs_to_map<S: clawhdf5_format::storage::Storage + ?Sized>(
     attrs: &[clawhdf5_format::attribute::AttributeMessage],
-    file_data: &[u8],
+    file_data: &S,
     offset_size: u8,
     length_size: u8,
 ) -> HashMap<String, AttrValue> {
@@ -266,9 +266,9 @@ fn decode_bool_enum(attr: &clawhdf5_format::attribute::AttributeMessage) -> Opti
     values.iter().all(|v| *v == 0 || *v == 1).then_some(values)
 }
 
-fn decode_attr_value(
+fn decode_attr_value<S: clawhdf5_format::storage::Storage + ?Sized>(
     attr: &clawhdf5_format::attribute::AttributeMessage,
-    file_data: &[u8],
+    file_data: &S,
     offset_size: u8,
     length_size: u8,
 ) -> Option<AttrValue> {
@@ -311,7 +311,7 @@ fn decode_attr_value(
             is_string: true, ..
         } => {
             let strings = attr
-                .read_vl_strings(file_data, offset_size, length_size)
+                .read_vl_strings_in(file_data, offset_size, length_size)
                 .ok()?;
             if strings.len() == 1 {
                 Some(AttrValue::String(strings[0].clone()))

crates/clawhdf5/src/vlen.rs

@@ -14,6 +14,7 @@
 use clawhdf5_format::data_read;
 use clawhdf5_format::datatype::Datatype;
 use clawhdf5_format::error::FormatError;
+use clawhdf5_format::storage::Storage;
 use clawhdf5_format::vl_data::{VlResolver, check_element_size};
 
 use crate::error::Error;
@@ -68,8 +69,8 @@ fn class_name(dt: &Datatype) -> &'static str {
 
 /// The strings in `raw`, elements of `dt`: fixed-length strings decoded as
 /// `read_string` always has, variable-length strings resolved in the heap.
-pub(crate) fn decode_strings(
+pub(crate) fn decode_strings<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     dt: &Datatype,
     raw: &[u8],
     offset_size: u8,
@@ -82,15 +83,15 @@ pub(crate) fn decode_strings(
             ..
         } => {
             check_element_size(*size, offset_size)?;
-            Ok(VlResolver::new(file_data, offset_size, length_size).strings(raw)?)
+            Ok(VlResolver::new_in(file_data, offset_size, length_size).strings(raw)?)
         }
         _ => Ok(data_read::read_as_strings(raw, dt)?),
     }
 }
 
 /// The exact bytes of the variable-length strings in `raw`.
-pub(crate) fn decode_string_bytes(
+pub(crate) fn decode_string_bytes<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     dt: &Datatype,
     raw: &[u8],
     offset_size: u8,
@@ -103,7 +104,7 @@ pub(crate) fn decode_string_bytes(
             ..
         } => {
             check_element_size(*size, offset_size)?;
-            Ok(VlResolver::new(file_data, offset_size, length_size).string_bytes(raw)?)
+            Ok(VlResolver::new_in(file_data, offset_size, length_size).string_bytes(raw)?)
         }
         other => Err(Error::Format(FormatError::TypeMismatch {
             expected: "variable-length string",
@@ -114,8 +115,8 @@ pub(crate) fn decode_string_bytes(
 
 /// The sequences in `raw`, elements of the variable-length sequence type
 /// `dt`, converted to `T`.
-pub(crate) fn decode_vlen<T: VlenValue>(
+pub(crate) fn decode_vlen<T: VlenValue, S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     dt: &Datatype,
     raw: &[u8],
     offset_size: u8,
@@ -135,7 +136,7 @@ pub(crate) fn decode_vlen<T: VlenValue>(
     };
     check_element_size(*size, offset_size)?;
     let base_size = base_type.type_size() as usize;
-    VlResolver::new(file_data, offset_size, length_size)
+    VlResolver::new_in(file_data, offset_size, length_size)
         .sequences(raw, base_size)?
         .iter()
         .map(|bytes| Ok(T::decode(bytes, base_type)?))

crates/clawhdf5/tests/contiguous_read_interop.rs

@@ -12,9 +12,11 @@
 
 use std::path::Path;
 use std::process::Command;
+use std::sync::Arc;
 
 use clawhdf5::File;
 use clawhdf5_format::selection::Selection;
+use clawhdf5_format::storage::CountingStorage;
 
 fn python() -> String {
     std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
@@ -364,9 +366,31 @@ with h5py.File("{path}", "r") as f:
     ));
 
     let file = File::open(&path).unwrap();
+    // The same file over a storage that serves only range reads: the
+    // selections read only their runs (see `strided_selections_over_storage_
+    // read_few_ranges`) and must give the same bytes.
+    let remote = File::open_storage(Arc::new(CountingStorage::new(
+        std::fs::read(&path).unwrap(),
+    )))
+    .unwrap();
     for (k, (name, code, sel)) in cases.iter().enumerate() {
         let ds = file.dataset(name).unwrap();
         let want_bytes = std::fs::read(dir.path().join(format!("sel_{k}.bin"))).unwrap();
+        let rds = remote.dataset(name).unwrap();
+        assert!(
+            rds.read_selection(sel).unwrap() == want_bytes,
+            "{name} {sel:?}: bytes over a range storage differ from libhdf5's"
+        );
+        assert_eq!(
+            rds.read_f64_selection(sel).unwrap(),
+            ds.read_f64_selection(sel).unwrap(),
+            "{name} {sel:?}: f64 over a range storage"
+        );
+        assert_eq!(
+            rds.read_i32_selection(sel).unwrap(),
+            ds.read_i32_selection(sel).unwrap(),
+            "{name} {sel:?}: i32 over a range storage"
+        );
         let got_bytes = ds.read_selection(sel).unwrap();
         assert!(
             got_bytes == want_bytes,
@@ -401,3 +425,64 @@ with h5py.File("{path}", "r") as f:
         );
     }
 }
+
+/// Over a storage without the file in memory, a selection of a contiguous
+/// dataset reads its runs merged across small gaps: a strided selection is
+/// a few large reads, not one per element (563 200 for the stride-2 case
+/// before), and the values are libhdf5's.
+#[test]
+fn strided_selections_over_storage_read_few_ranges() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("contig.h5");
+    write_file(&path);
+    let storage = Arc::new(CountingStorage::new(std::fs::read(&path).unwrap()));
+    let remote = File::open_storage(storage.clone()).unwrap();
+    let local = File::open(&path).unwrap();
+    // f4le_big is 1100 x 1024 f32: 4 KiB rows, 4.4 MB in all.
+    let name = "f4le_big";
+    let every_100th: Vec<Vec<u64>> = (0..1100u64 * 1024)
+        .step_by(100)
+        .map(|i| vec![i / 1024, i % 1024])
+        .collect();
+    let backwards: Vec<Vec<u64>> = every_100th.iter().rev().take(50).cloned().collect();
+    // (selection, most range reads it may take)
+    let cases: Vec<(Selection, u64)> = vec![
+        // Every other element of every row: one read of the whole dataset.
+        (slab(&[(0, 1, 1100, 1), (0, 2, 512, 1)]), 1),
+        // Blocks of 3 every 7 on every other row: rows are 4 KiB apart, so
+        // one read per selected row at most.
+        (slab(&[(0, 2, 550, 1), (1, 7, 146, 3)]), 550),
+        // Every 100th element, in order: 400-byte gaps, one read.
+        (Selection::Points(every_100th), 1),
+        // Points going backwards are not merged.
+        (Selection::Points(backwards), 50),
+        // A column: 4 KiB apart, merged.
+        (slab(&[(0, 1, 1100, 1), (5, 1, 1, 1)]), 1),
+    ];
+    let ds = remote.dataset(name).unwrap();
+    for (sel, most) in cases {
+        storage.reset();
+        let got = ds.read_f32_selection(&sel).unwrap();
+        let reads = storage.reads();
+        assert_eq!(
+            got,
+            local
+                .dataset(name)
+                .unwrap()
+                .read_f32_selection(&sel)
+                .unwrap(),
+            "{sel:?}"
+        );
+        // A few reads of metadata besides the data.
+        assert!(reads <= most + 8, "{sel:?}: {reads} range reads");
+        storage.reset();
+        let bytes = ds.read_selection(&sel).unwrap();
+        assert!(
+            storage.reads() <= most + 8,
+            "{sel:?}: {} reads",
+            storage.reads()
+        );
+        assert_eq!(bytes.len(), got.len() * 4);
+    }
+}

crates/clawhdf5/tests/edit_tests.rs

@@ -98,8 +98,8 @@ fn errors_leave_the_file_untouched() {
     let mut ed = FileEditor::open(&path).unwrap();
     assert!(matches!(FileEditor::open(&path), Err(Error::Locked(_))));
     assert!(ed.write_all("missing", &[0; 4]).is_err());
-    // Wrong length, wrong type, outside the extent, beyond maxshape,
+    // Wrong length, wrong type, outside the extent, beyond maxshape, a
-    // shrinking, a rank change.
+    // rank change, resizing a dataset that is not chunked.
     assert!(matches!(
         ed.write_all("flat", &[0; 7]),
         Err(Error::InvalidArgument(_))
@@ -116,7 +116,10 @@ fn errors_leave_the_file_untouched() {
         ed.resize("raw", &[3, 5]),
         Err(Error::InvalidArgument(_))
     ));
-    assert!(matches!(ed.resize("ext", &[4]), Err(Error::Unsupported(_))));
+    assert!(matches!(
+        ed.resize("flat", &[2]),
+        Err(Error::Unsupported(_))
+    ));
     assert!(matches!(
         ed.resize("ext", &[4, 1]),
         Err(Error::InvalidArgument(_))
@@ -136,3 +139,26 @@ fn errors_leave_the_file_untouched() {
     drop(ed);
     assert!(std::fs::read(&path).unwrap() == before);
 }
+
+/// Shrinking and growing again on a file clawhdf5 wrote: elements that come
+/// back read as the fill value, the ones kept keep their values.
+#[test]
+fn shrink_then_grow_reads_fill() {
+    let dir = tempfile::tempdir().unwrap();
+    let path = sample(dir.path());
+    {
+        let mut ed = FileEditor::open(&path).unwrap();
+        ed.resize("ext", &[2]).unwrap();
+        ed.resize("ext", &[9]).unwrap();
+        ed.resize("raw", &[1, 4]).unwrap();
+        ed.resize("raw", &[3, 4]).unwrap();
+    }
+    let f = File::open(&path).unwrap();
+    assert_eq!(
+        f.dataset("ext").unwrap().read_i32().unwrap(),
+        [0, 1, 0, 0, 0, 0, 0, 0, 0]
+    );
+    let mut raw = vec![0.0f64; 12];
+    raw[..4].fill(0.5);
+    assert_eq!(f.dataset("raw").unwrap().read_f64().unwrap(), raw);
+}

crates/clawhdf5/tests/fletcher32_interop.rs

@@ -0,0 +1,321 @@
+//! Fletcher-32 against libhdf5.
+//!
+//! libhdf5's `H5_checksum_fletcher32` reduces its sums with the
+//! ones'-complement fold `(s & 0xffff) + (s >> 16)`, which leaves 0xffff
+//! where `% 65535` leaves 0. Our checksum once used `% 65535`, so on about
+//! one chunk in 32768 (a sum that is a non-zero multiple of 65535) libhdf5
+//! rejected the chunks we wrote and we rejected the chunks it wrote.
+//!
+//! - The checksum is compared with libhdf5's own `H5_checksum_fletcher32`,
+//!   called through ctypes from the library h5py loads, over every one-byte
+//!   and two-byte input and a large corpus of random and fold-heavy inputs.
+//! - Chunks engineered to hit the fold are written by `FileBuilder` and by
+//!   `FileEditor` and read by h5py, and written by h5py and read by us.
+//!
+//! Skipped when python3 with h5py is unavailable, unless
+//! `CLAWHDF5_REQUIRE_INTEROP=1`.
+
+use std::process::Command;
+
+use clawhdf5::{File, FileBuilder, FileEditor};
+use clawhdf5_format::checksum::fletcher32;
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn have_h5py() -> bool {
+    let ok = Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .is_ok_and(|o| o.status.success());
+    if !ok {
+        assert!(
+            std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
+            "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+        );
+        eprintln!("SKIP: python3 with h5py not available");
+    }
+    ok
+}
+
+fn run_python(script: &str, args: &[&str]) -> String {
+    let out = Command::new(python())
+        .arg("-c")
+        .arg(script)
+        .args(args)
+        .output()
+        .expect("failed to run python");
+    assert!(
+        out.status.success(),
+        "python failed:\nSTDOUT: {}\nSTDERR: {}",
+        String::from_utf8_lossy(&out.stdout),
+        String::from_utf8_lossy(&out.stderr)
+    );
+    String::from_utf8_lossy(&out.stdout).trim().to_string()
+}
+
+fn tmp(name: &str) -> std::path::PathBuf {
+    let dir = std::env::temp_dir().join(format!("clawhdf5_fletcher32_{}", std::process::id()));
+    std::fs::create_dir_all(&dir).unwrap();
+    dir.join(name)
+}
+
+/// The checksum our code computed before it was fixed: each sum reduced
+/// `% 65535`. Only used to show that the test data hits the disagreement.
+fn fletcher32_mod(data: &[u8]) -> u32 {
+    let (mut s1, mut s2) = (0u64, 0u64);
+    for w in data.chunks(2) {
+        let v = (u64::from(w[0]) << 8) | w.get(1).map_or(0, |&b| u64::from(b));
+        s1 = (s1 + v) % 65535;
+        s2 = (s2 + s1) % 65535;
+    }
+    ((s2 as u32) << 16) | s1 as u32
+}
+
+/// Splitmix64, so the data is the same on every run.
+struct Rng(u64);
+impl Rng {
+    fn next(&mut self) -> u64 {
+        self.0 = self.0.wrapping_add(0x9e37_79b9_7f4a_7c15);
+        let mut z = self.0;
+        z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
+        z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
+        z ^ (z >> 31)
+    }
+}
+
+/// Writes every case of the file `argv[1]` (u32 LE length + bytes) back to
+/// `argv[2]` as libhdf5's checksum of each, u32 LE.
+const LIBHDF5_CHECKSUMS: &str = r#"
+import ctypes, glob, os, struct, sys
+import h5py
+cands = glob.glob(os.path.join(os.path.dirname(h5py.__file__), os.pardir, 'h5py.libs', 'libhdf5-*.so*'))
+cands += glob.glob(os.path.join(os.path.dirname(h5py.__file__), '.dylibs', 'libhdf5*.dylib'))
+if cands:
+    lib = ctypes.CDLL(cands[0])
+else:
+    # A system h5py links the system libhdf5, already loaded.
+    import h5py.h5
+    lib = ctypes.CDLL(h5py.h5.__file__)
+f = lib.H5_checksum_fletcher32
+f.restype = ctypes.c_uint32
+f.argtypes = [ctypes.c_char_p, ctypes.c_size_t]
+data = open(sys.argv[1], 'rb').read()
+out = bytearray()
+i = 0
+while i < len(data):
+    (n,) = struct.unpack_from('<I', data, i)
+    i += 4
+    b = data[i:i + n]
+    i += n
+    out += struct.pack('<I', f(b, n))
+open(sys.argv[2], 'wb').write(out)
+"#;
+
+#[test]
+fn checksum_matches_libhdf5() {
+    if !have_h5py() {
+        return;
+    }
+    let mut cases: Vec<Vec<u8>> = Vec::new();
+    // Every one-byte input (the odd-length path alone) and every one-word
+    // input (65535 = 0xffff is the smallest fold).
+    cases.extend((0..=255u8).map(|b| vec![b]));
+    cases.extend((0..=u16::MAX).map(|w| w.to_be_bytes().to_vec()));
+    let mut rng = Rng(0x5eed_f1e7);
+    // Words drawn from values that make multiples of 65535 frequent, at
+    // lengths around the 360-word block boundaries, odd and even.
+    const FOLDY: [u16; 6] = [0, 1, 0xfffe, 0xffff, 0x8000, 0x7fff];
+    for _ in 0..40_000 {
+        let len = match rng.next() % 4 {
+            0 => (rng.next() % 16) as usize,
+            1 => 718 + (rng.next() % 6) as usize,
+            2 => 1438 + (rng.next() % 6) as usize,
+            _ => (rng.next() % 3000) as usize,
+        };
+        let foldy = rng.next().is_multiple_of(2);
+        let mut v = Vec::with_capacity(len + 1);
+        while v.len() < len {
+            let w = if foldy {
+                FOLDY[(rng.next() % 6) as usize]
+            } else {
+                rng.next() as u16
+            };
+            v.extend_from_slice(&w.to_be_bytes());
+        }
+        v.truncate(len);
+        cases.push(v);
+    }
+    // Long runs of 0xff: sums are multiples of 65535 at every block.
+    for len in [720, 721, 1440, 1441, 7200, 65536, 65537] {
+        cases.push(vec![0xff; len]);
+    }
+    let mut blob = Vec::new();
+    for c in &cases {
+        blob.extend_from_slice(&(c.len() as u32).to_le_bytes());
+        blob.extend_from_slice(c);
+    }
+    let input = tmp("cases.bin");
+    let output = tmp("sums.bin");
+    std::fs::write(&input, &blob).unwrap();
+    run_python(
+        LIBHDF5_CHECKSUMS,
+        &[input.to_str().unwrap(), output.to_str().unwrap()],
+    );
+    let sums = std::fs::read(&output).unwrap();
+    assert_eq!(sums.len(), cases.len() * 4);
+    let mut folds = 0;
+    for (c, s) in cases.iter().zip(sums.as_chunks::<4>().0) {
+        let want = u32::from_le_bytes(*s);
+        assert_eq!(
+            fletcher32(c),
+            want,
+            "checksum of {} bytes {:02x?}...",
+            c.len(),
+            &c[..c.len().min(16)]
+        );
+        if fletcher32_mod(c) != want {
+            folds += 1;
+        }
+    }
+    // The corpus must exercise the case `% 65535` got wrong.
+    assert!(folds > 500, "only {folds} fold cases");
+}
+
+const CHUNK: usize = 8;
+
+/// `n` chunks of `CHUNK` bytes, each one a chunk on which the old
+/// `% 65535` checksum and libhdf5's differ (sum1, sum2 or both a non-zero
+/// multiple of 65535), with an ordinary chunk between them.
+fn fold_chunks(n: usize) -> Vec<u8> {
+    let mut rng = Rng(42);
+    let mut out = Vec::new();
+    let mut found = 0;
+    while found < n {
+        // Build a chunk whose sum1 is a multiple of 65535 half the time,
+        // otherwise search at random for a sum2 fold.
+        let mut c: Vec<u8> = (0..CHUNK).map(|_| rng.next() as u8).collect();
+        if found % 2 == 0 {
+            let words: u64 = c[..CHUNK - 2]
+                .chunks(2)
+                .map(|w| (u64::from(w[0]) << 8) | u64::from(w[1]))
+                .sum();
+            let last = ((65535 - words % 65535) % 65535) as u16;
+            c[CHUNK - 2..].copy_from_slice(&last.to_be_bytes());
+        }
+        if fletcher32(&c) != fletcher32_mod(&c) {
+            out.extend_from_slice(&c);
+            out.extend((0..CHUNK).map(|i| i as u8 + 1));
+            found += 1;
+        }
+    }
+    out
+}
+
+#[test]
+fn h5py_reads_fold_case_chunks_we_write() {
+    if !have_h5py() {
+        return;
+    }
+    let data = fold_chunks(32);
+    // FileBuilder.
+    let built = tmp("built.h5");
+    let mut b = FileBuilder::new();
+    b.create_dataset("d")
+        .with_u8_data(&data)
+        .with_chunks(&[CHUNK as u64])
+        .with_fletcher32();
+    b.write(&built).unwrap();
+    // FileEditor, into a dataset h5py created.
+    let edited = tmp("edited.h5");
+    run_python(
+        "import sys, h5py, numpy as np\n\
+         with h5py.File(sys.argv[1], 'w') as f:\n\
+         \x20   f.create_dataset('d', data=np.zeros(int(sys.argv[2]), 'u1'), chunks=(8,), fletcher32=True)",
+        &[edited.to_str().unwrap(), &data.len().to_string()],
+    );
+    FileEditor::open(&edited)
+        .unwrap()
+        .write_all("d", &data)
+        .unwrap();
+    for path in [&built, &edited] {
+        let got = run_python(
+            "import sys, h5py\n\
+             with h5py.File(sys.argv[1], 'r') as f:\n\
+             \x20   assert f['d'].fletcher32\n\
+             \x20   print(f['d'][:].tobytes().hex())",
+            &[path.to_str().unwrap()],
+        );
+        assert_eq!(got, hex(&data), "{}", path.display());
+    }
+}
+
+#[test]
+fn we_read_fold_case_chunks_h5py_writes() {
+    if !have_h5py() {
+        return;
+    }
+    let data = fold_chunks(32);
+    let path = tmp("h5py.h5");
+    run_python(
+        "import sys, h5py, numpy as np\n\
+         with h5py.File(sys.argv[1], 'w') as f:\n\
+         \x20   f.create_dataset('d', data=np.frombuffer(bytes.fromhex(sys.argv[2]), 'u1'), chunks=(8,), fletcher32=True)",
+        &[path.to_str().unwrap(), &hex(&data)],
+    );
+    let file = File::open(&path).unwrap();
+    let ds = file.dataset("d").unwrap();
+    assert_eq!(
+        ds.read_selection(&clawhdf5_format::selection::Selection::All)
+            .unwrap(),
+        data
+    );
+}
+
+/// A checksum stored with the bytes of each 16-bit half swapped, as
+/// libhdf5 1.6.2 and earlier wrote it, is accepted as libhdf5 accepts it;
+/// so is the `% 65535` form clawhdf5 v2.7.0 and earlier wrote, so that
+/// their files stay readable.
+#[test]
+fn legacy_checksums_are_accepted() {
+    use clawhdf5_format::filter_pipeline::{FILTER_FLETCHER32, FilterDescription, FilterPipeline};
+    let payload = [1u8, 2, 3, 4, 5];
+    let sum = fletcher32(&payload);
+    let swapped = ((sum & 0x00ff_00ff) << 8) | ((sum >> 8) & 0x00ff_00ff);
+    assert_ne!(sum, swapped);
+    let pipeline = FilterPipeline {
+        version: 2,
+        filters: vec![FilterDescription {
+            filter_id: FILTER_FLETCHER32,
+            name: None,
+            client_data: vec![],
+            flags: 0,
+        }],
+    };
+    for stored in [sum, swapped] {
+        let mut chunk = payload.to_vec();
+        chunk.extend_from_slice(&stored.to_le_bytes());
+        let out = clawhdf5_format::filters::decompress_chunk(&chunk, &pipeline, payload.len(), 1)
+            .unwrap();
+        assert_eq!(out, payload);
+    }
+    // Our old checksum of a fold-case chunk.
+    let fold = fold_chunks(1);
+    let fold = &fold[..CHUNK];
+    let old = fletcher32_mod(fold);
+    assert_ne!(old, fletcher32(fold));
+    let mut chunk = fold.to_vec();
+    chunk.extend_from_slice(&old.to_le_bytes());
+    let out = clawhdf5_format::filters::decompress_chunk(&chunk, &pipeline, CHUNK, 1).unwrap();
+    assert_eq!(out, fold);
+    let mut chunk = payload.to_vec();
+    chunk.extend_from_slice(&(sum ^ 1).to_le_bytes());
+    assert!(
+        clawhdf5_format::filters::decompress_chunk(&chunk, &pipeline, payload.len(), 1).is_err()
+    );
+}
+
+fn hex(b: &[u8]) -> String {
+    b.iter().map(|x| format!("{x:02x}")).collect()
+}

crates/clawhdf5/tests/storage_equivalence.rs

@@ -0,0 +1,609 @@
+//! `File::open_storage` over a read_at-only storage reads every file as
+//! `File::open` does (range reads, milestone M2 in
+//! `docs/design/range-reads.md`).
+//!
+//! Each file is read end to end twice — through `File::open` (the mmap
+//! fast path) and through `File::open_storage` over a
+//! [`CountingStorage`], which serves the file through `read_at` only
+//! (`as_contiguous()` is `None`, so no reader can fall back to a slice of
+//! the whole file) — and the two transcripts must be identical: the tree
+//! (every group's entries, followed by address), every object's attributes
+//! (the whole map and each one by name), and every dataset's shape, types
+//! and values (all bytes, as `f64`, a hyperslab of them, strings and
+//! variable-length sequences).
+//!
+//! The storage also counts its `read_at` calls and bytes: what a remote
+//! backend without a cache would be asked for. The totals and the files
+//! that cost most are printed.
+//!
+//! - `CLAWHDF5_STORAGE_CORPUS=dir[:dir...]` adds every HDF5 file under those
+//!   directories (the conformance corpus is `conformance/.cache/corpus`);
+//!   `CLAWHDF5_STORAGE_REPORT=1` prints every file's counts.
+
+use std::collections::{BTreeMap, HashSet, VecDeque};
+use std::fmt::Write as _;
+use std::path::{Path, PathBuf};
+use std::sync::Arc;
+
+use clawhdf5::{DType, File, Selection};
+use clawhdf5_format::error::FormatError;
+use clawhdf5_format::storage::CountingStorage;
+
+/// Objects visited per file.
+const MAX_OBJECTS: usize = 2000;
+/// Datasets with more bytes than this are not read (their metadata is).
+const MAX_DATA_BYTES: u64 = 64 << 20;
+
+/// A short, stable digest of a value's `Debug` form.
+fn digest<T: std::fmt::Debug>(v: &T) -> String {
+    let s = format!("{v:?}");
+    if s.len() <= 200 {
+        return s;
+    }
+    let mut h = 0xcbf2_9ce4_8422_2325u64;
+    for b in s.bytes() {
+        h = (h ^ u64::from(b)).wrapping_mul(0x100_0000_01b3);
+    }
+    format!("{}…[{} bytes, fnv {h:016x}]", &s[..80], s.len())
+}
+
+/// A data read's result: its value, or its error in full (the two paths
+/// must fail the same way, not just both fail). One case is known to vary
+/// between two `File`s and is allowed for in [`check`]: a full read goes
+/// through the file's chunk cache, which lists a damaged dataset's chunks
+/// in hash-map order, so which failing chunk it reports varies.
+fn value<T: std::fmt::Debug, E: std::fmt::Debug>(r: &Result<T, E>) -> String {
+    match r {
+        Ok(v) => digest(v),
+        Err(e) => format!("Err({e:?})"),
+    }
+}
+
+fn transcript(file: &File) -> String {
+    let mut out = String::new();
+    let mut seen = HashSet::new();
+    let mut queue = VecDeque::from([(String::from("/"), file.superblock().root_group_address)]);
+    while let Some((path, addr)) = queue.pop_front() {
+        if seen.len() >= MAX_OBJECTS || !seen.insert(addr) {
+            continue;
+        }
+        let group = file.group_at(addr);
+        let entries = group.entries();
+        writeln!(out, "{path} @{addr} entries {}", digest(&entries)).unwrap();
+        match file.dataset_at(addr) {
+            Ok(ds) => dataset(&mut out, &path, &ds),
+            Err(e) => writeln!(out, "{path} dataset_at {e:?}").unwrap(),
+        }
+        let attrs = group.attrs_with_errors().map(|(a, e)| (sorted(a), e));
+        writeln!(out, "{path} attrs {}", digest(&attrs)).unwrap();
+        if let Ok((attrs, _)) = &attrs {
+            for name in attrs.keys().take(50) {
+                writeln!(out, "{path} attr {name:?} {}", digest(&group.attr(name))).unwrap();
+            }
+        }
+        if let Ok(entries) = entries {
+            for (name, child) in entries {
+                queue.push_back((format!("{}/{name}", path.trim_end_matches('/')), child));
+                // Name lookups (through the name index of a dense group).
+                if queue.len() < 64 {
+                    writeln!(
+                        out,
+                        "{path} group({name:?}) {}",
+                        digest(&group.group(&name).map(|_| ()))
+                    )
+                    .unwrap();
+                }
+            }
+        }
+    }
+    out
+}
+
+fn sorted<V: std::fmt::Debug>(m: std::collections::HashMap<String, V>) -> BTreeMap<String, V> {
+    m.into_iter().collect()
+}
+
+fn dataset(out: &mut String, path: &str, ds: &clawhdf5::Dataset<'_>) {
+    let shape = ds.shape();
+    let dtype = ds.dtype();
+    writeln!(
+        out,
+        "{path} shape {} max {} dtype {} raw {}",
+        digest(&shape),
+        digest(&ds.max_dimensions()),
+        digest(&dtype),
+        digest(&ds.raw_datatype())
+    )
+    .unwrap();
+    let attrs = ds.attrs_with_errors().map(|(a, e)| (sorted(a), e));
+    writeln!(out, "{path} dataset attrs {}", digest(&attrs)).unwrap();
+    let (Ok(shape), Ok(dtype), Ok(raw_dt)) = (shape, dtype, ds.raw_datatype()) else {
+        return;
+    };
+    let elements = shape.iter().try_fold(1u64, |a, &d| a.checked_mul(d));
+    let bytes = elements.and_then(|n| n.checked_mul(u64::from(raw_dt.type_size())));
+    if bytes.is_none_or(|b| b > MAX_DATA_BYTES) {
+        writeln!(out, "{path} too large to read").unwrap();
+        return;
+    }
+    writeln!(
+        out,
+        "{path} all {}",
+        value(&ds.read_selection(&Selection::All))
+    )
+    .unwrap();
+    let numeric = matches!(
+        dtype,
+        DType::F32
+            | DType::F64
+            | DType::I8
+            | DType::I16
+            | DType::I32
+            | DType::I64
+            | DType::U8
+            | DType::U16
+            | DType::U32
+            | DType::U64
+    );
+    if numeric {
+        writeln!(out, "{path} f64 {}", value(&ds.read_f64())).unwrap();
+        writeln!(out, "{path} f32 {}", value(&ds.read_f32())).unwrap();
+        writeln!(out, "{path} i64 {}", value(&ds.read_i64())).unwrap();
+        if let Some(&d0) = shape.first() {
+            let rank = shape.len();
+            let sel = Selection::Hyperslab {
+                start: std::iter::once(d0 / 3)
+                    .chain(std::iter::repeat_n(0, rank - 1))
+                    .collect(),
+                stride: vec![1; rank],
+                count: std::iter::once(d0.div_ceil(3))
+                    .chain(shape[1..].iter().copied())
+                    .collect(),
+                block: vec![1; rank],
+            };
+            writeln!(
+                out,
+                "{path} f64 third {}",
+                value(&ds.read_f64_selection(&sel))
+            )
+            .unwrap();
+            writeln!(
+                out,
+                "{path} bytes third {}",
+                value(&ds.read_selection(&sel))
+            )
+            .unwrap();
+            // Every third row (a strided hyperslab), and a few points out
+            // of order: the last element, the first, one in the middle.
+            let strided = Selection::Hyperslab {
+                start: vec![0; rank],
+                stride: std::iter::once(3)
+                    .chain(std::iter::repeat_n(1, rank - 1))
+                    .collect(),
+                count: std::iter::once(d0.div_ceil(3))
+                    .chain(shape[1..].iter().copied())
+                    .collect(),
+                block: vec![1; rank],
+            };
+            writeln!(
+                out,
+                "{path} f64 strided {}",
+                value(&ds.read_f64_selection(&strided))
+            )
+            .unwrap();
+            if shape.iter().all(|&d| d > 0) {
+                let points = Selection::Points(vec![
+                    shape.iter().map(|&d| d - 1).collect(),
+                    vec![0; rank],
+                    shape.iter().map(|&d| d / 2).collect(),
+                ]);
+                writeln!(
+                    out,
+                    "{path} bytes points {}",
+                    value(&ds.read_selection(&points))
+                )
+                .unwrap();
+                writeln!(
+                    out,
+                    "{path} i64 points {}",
+                    value(&ds.read_i64_selection(&points))
+                )
+                .unwrap();
+            }
+        }
+    }
+    match &raw_dt {
+        clawhdf5_format::datatype::Datatype::String { .. }
+        | clawhdf5_format::datatype::Datatype::VariableLength {
+            is_string: true, ..
+        } => {
+            writeln!(out, "{path} strings {}", value(&ds.read_string_bytes())).unwrap();
+            writeln!(out, "{path} string {}", value(&ds.read_string())).unwrap();
+        }
+        clawhdf5_format::datatype::Datatype::VariableLength { .. } => {
+            writeln!(out, "{path} vlen {}", value(&ds.read_vlen::<f64>())).unwrap();
+        }
+        _ => {}
+    }
+}
+
+/// External virtual-dataset sources as `File::open` finds them: files in
+/// the same directory.
+fn sibling_resolver(dir: Option<PathBuf>) -> clawhdf5::VdsResolver {
+    Arc::new(move |name: &str| {
+        let Some(dir) = dir.as_ref() else {
+            return Err(FormatError::ChunkedReadError(format!(
+                "virtual dataset source file {name:?} cannot be resolved for an in-memory file"
+            )));
+        };
+        let p = Path::new(name);
+        if name.is_empty()
+            || !p.components().all(|c| {
+                matches!(
+                    c,
+                    std::path::Component::Normal(_) | std::path::Component::CurDir
+                )
+            })
+        {
+            return Err(FormatError::ChunkedReadError(format!(
+                "virtual dataset source file {name:?} is outside the virtual file's \
+                 directory and is not followed"
+            )));
+        }
+        match std::fs::read(dir.join(p)) {
+            Ok(bytes) => Ok(Some(bytes)),
+            Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
+            Err(e) => Err(FormatError::ChunkedReadError(format!(
+                "cannot read virtual dataset source file {name:?}: {e}"
+            ))),
+        }
+    })
+}
+
+#[derive(Default)]
+struct Totals {
+    files: usize,
+    opened: usize,
+    /// The comparison's reads (each dataset read several ways).
+    reads: u64,
+    bytes: u64,
+    /// One pass: open, list every group, read every attribute and every
+    /// dataset once (`read_selection(All)`).
+    pass_reads: u64,
+    pass_bytes: u64,
+    file_bytes: u64,
+    /// (one-pass reads, one-pass bytes, file size, name) per file.
+    per_file: Vec<(u64, u64, u64, String)>,
+}
+
+/// Open the file and read everything once, as a tree viewer that then
+/// shows every value would.
+fn one_pass(file: &File) {
+    let mut seen = HashSet::new();
+    let mut queue = VecDeque::from([file.superblock().root_group_address]);
+    while let Some(addr) = queue.pop_front() {
+        if seen.len() >= MAX_OBJECTS || !seen.insert(addr) {
+            continue;
+        }
+        let group = file.group_at(addr);
+        let _ = group.attrs();
+        if let Ok(ds) = file.dataset_at(addr) {
+            let small = ds.shape().ok().and_then(|s| {
+                let n = s.iter().try_fold(1u64, |a, &d| a.checked_mul(d))?;
+                let size = u64::from(ds.raw_datatype().ok()?.type_size());
+                n.checked_mul(size).filter(|&b| b <= MAX_DATA_BYTES)
+            });
+            if small.is_some() {
+                let _ = ds.read_selection(&Selection::All);
+            }
+        }
+        if let Ok(entries) = group.entries() {
+            queue.extend(entries.into_iter().map(|(_, a)| a));
+        }
+    }
+}
+
+fn check(path: &Path, totals: &mut Totals) {
+    let Ok(bytes) = std::fs::read(path) else {
+        return;
+    };
+    let name = path.display().to_string();
+    let local = File::open(path);
+    let storage = Arc::new(CountingStorage::new(bytes.clone()));
+    let resolver = sibling_resolver(path.parent().map(Path::to_path_buf));
+    let remote = File::open_storage(storage.clone()).map(|mut f| {
+        f.set_vds_resolver(resolver.clone());
+        f
+    });
+    totals.files += 1;
+    let (local, remote) = match (local, remote) {
+        (Ok(l), Ok(r)) => (l, r),
+        (l, r) => {
+            // Both refuse the file, with the same error.
+            assert_eq!(
+                format!("{:?}", l.map(|_| ())),
+                format!("{:?}", r.map(|_| ())),
+                "{name}: open"
+            );
+            return;
+        }
+    };
+    totals.opened += 1;
+    assert!(remote.contiguous_bytes().is_none(), "{name}");
+    assert_eq!(local.user_block_size(), remote.user_block_size(), "{name}");
+    let want = transcript(&local);
+    let got = transcript(&remote);
+    // Every read path works over a storage without the file in memory.
+    assert!(
+        !got.contains("ContiguousStorageRequired"),
+        "{name}: a read needed the file in memory"
+    );
+    if want != got {
+        let first = unexplained_difference(path, &want, &got);
+        if let Some(first) = first {
+            panic!("{name}: File::open_storage differs from File::open{first}");
+        }
+    }
+    totals.reads += storage.reads();
+    totals.bytes += storage.bytes_read();
+    totals.file_bytes += bytes.len() as u64;
+
+    let pass = Arc::new(CountingStorage::new(bytes.clone()));
+    if let Ok(mut f) = File::open_storage(pass.clone()) {
+        f.set_vds_resolver(resolver);
+        one_pass(&f);
+    }
+    let (reads, read_bytes) = (pass.reads(), pass.bytes_read());
+    totals.pass_reads += reads;
+    totals.pass_bytes += read_bytes;
+    if std::env::var("CLAWHDF5_STORAGE_REPORT").is_ok_and(|v| v == "1") {
+        eprintln!(
+            "{reads:>9} reads {read_bytes:>12} bytes {:>12} file  {name}",
+            bytes.len()
+        );
+    }
+    totals
+        .per_file
+        .push((reads, read_bytes, bytes.len() as u64, name));
+}
+
+/// Why the storage transcript `got` differs from the `File::open` one
+/// `want`, or `None` when every line that differs is one `File::open` can
+/// give too: a line that varies between two `File`s (the chunk cache's
+/// hash-map order picks which failing chunk a damaged dataset's full read
+/// reports) and whose storage value some fresh `File::open` reproduces.
+/// Nothing else is allowed to differ.
+fn unexplained_difference(path: &Path, want: &str, got: &str) -> Option<String> {
+    let (want, got): (Vec<&str>, Vec<&str>) = (want.lines().collect(), got.lines().collect());
+    if want.len() != got.len() {
+        return Some(format!("\n {} vs {} lines", want.len(), got.len()));
+    }
+    let mut open: Vec<usize> = (0..want.len()).filter(|&i| want[i] != got[i]).collect();
+    // Only reads that fail on both sides may vary.
+    if let Some(&i) = open
+        .iter()
+        .find(|&&i| !(want[i].contains(" Err(") && got[i].contains(" Err(")))
+    {
+        return Some(format!("\n local:   {}\n storage: {}", want[i], got[i]));
+    }
+    for _ in 0..64 {
+        let again = transcript(&File::open(path).unwrap());
+        let again: Vec<&str> = again.lines().collect();
+        open.retain(|&i| again.get(i) != Some(&got[i]));
+        if open.is_empty() {
+            return None;
+        }
+    }
+    let i = open[0];
+    Some(format!(
+        "\n local:   {}\n storage: {}\n (no File::open of 64 gave the storage's result)",
+        want[i], got[i]
+    ))
+}
+
+fn report(what: &str, totals: &mut Totals) {
+    eprintln!(
+        "{what}: {} files ({} open, {} bytes); comparison: {} read_at calls, {} bytes; \
+         one pass (list, attributes, every dataset once): {} read_at calls, {} bytes",
+        totals.files,
+        totals.opened,
+        totals.file_bytes,
+        totals.reads,
+        totals.bytes,
+        totals.pass_reads,
+        totals.pass_bytes
+    );
+    totals.per_file.sort_by_key(|a| std::cmp::Reverse(a.0));
+    for (reads, bytes, size, name) in totals.per_file.iter().take(10) {
+        eprintln!("  {reads:>9} reads {bytes:>12} bytes (file {size:>11})  {name}");
+    }
+}
+fn hdf5_files(dir: &Path, out: &mut Vec<PathBuf>) {
+    let Ok(entries) = std::fs::read_dir(dir) else {
+        return;
+    };
+    for e in entries.flatten() {
+        let p = e.path();
+        if p.is_dir() {
+            hdf5_files(&p, out);
+        } else if p
+            .extension()
+            .and_then(|x| x.to_str())
+            .is_some_and(|x| matches!(x, "h5" | "hdf5" | "he5" | "nc" | "h5ad" | "hdf"))
+        {
+            out.push(p);
+        }
+    }
+}
+
+#[test]
+fn fixtures_read_identically_through_open_storage() {
+    let root = Path::new(env!("CARGO_MANIFEST_DIR"));
+    let mut files = Vec::new();
+    hdf5_files(&root.join("tests/fixtures"), &mut files);
+    hdf5_files(&root.join("../clawhdf5-format/tests/fixtures"), &mut files);
+    files.sort();
+    assert!(files.len() >= 45, "{} fixtures", files.len());
+    let mut totals = Totals::default();
+    for f in &files {
+        check(f, &mut totals);
+    }
+    report("fixtures", &mut totals);
+    assert!(totals.opened >= 40, "{}", totals.opened);
+    assert!(totals.reads > 0);
+}
+
+#[test]
+fn corpus_reads_identically_through_open_storage() {
+    let Ok(dirs) = std::env::var("CLAWHDF5_STORAGE_CORPUS") else {
+        eprintln!("CLAWHDF5_STORAGE_CORPUS not set; skipping the corpus");
+        return;
+    };
+    let mut files = Vec::new();
+    for d in std::env::split_paths(&dirs) {
+        hdf5_files(&d, &mut files);
+    }
+    files.sort();
+    let mut totals = Totals::default();
+    for f in &files {
+        check(f, &mut totals);
+    }
+    report("corpus", &mut totals);
+    assert!(totals.files > 0);
+}
+
+/// A user block, a metadata cache image and a Storage that is itself in
+/// memory: the in-memory view of a storage that has one is used as is.
+#[test]
+fn storage_backed_files_keep_their_zero_copy_views_only_in_memory() {
+    let root = Path::new(env!("CARGO_MANIFEST_DIR"));
+    let path = root.join("tests/fixtures/h5clear_mdc_image.h5");
+    let bytes = std::fs::read(&path).unwrap();
+    let local = File::open(&path).unwrap();
+    // A Vec<u8> is a Storage with a contiguous view; the image still has
+    // to be laid over it, so the view is not used.
+    let in_memory = File::open_storage(Arc::new(bytes.clone())).unwrap();
+    assert!(in_memory.contiguous_bytes().is_none());
+    assert_eq!(transcript(&local), transcript(&in_memory));
+
+    let plain = root.join("../clawhdf5-format/tests/fixtures/chunked_2d.h5");
+    let bytes = std::fs::read(&plain).unwrap();
+    let in_memory = File::open_storage(Arc::new(bytes.clone())).unwrap();
+    assert_eq!(in_memory.contiguous_bytes(), Some(&bytes[..]));
+    let counting = File::open_storage(Arc::new(CountingStorage::new(bytes))).unwrap();
+    assert!(counting.contiguous_bytes().is_none());
+    let result =
+        std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| counting.as_bytes().len()));
+    assert!(
+        result.is_err(),
+        "as_bytes over a range storage must not answer"
+    );
+}
+
+/// A storage that returns 37 junk bytes more than every read asked for.
+struct Overlong(Vec<u8>);
+
+impl clawhdf5::Storage for Overlong {
+    fn read_at(&self, offset: u64, len: usize) -> Result<std::borrow::Cow<'_, [u8]>, FormatError> {
+        let mut v = clawhdf5::Storage::read_at(self.0.as_slice(), offset, len)?.into_owned();
+        v.extend(std::iter::repeat_n(0xa5, 37));
+        Ok(std::borrow::Cow::Owned(v))
+    }
+
+    fn len(&self) -> u64 {
+        self.0.len() as u64
+    }
+}
+
+/// Bytes a misbehaving storage returns past the range asked for are never
+/// read as the file's: every fixture reads through it as through
+/// `File::open`.
+#[test]
+fn overlong_storage_reads_identically() {
+    let root = Path::new(env!("CARGO_MANIFEST_DIR"));
+    let mut files = Vec::new();
+    hdf5_files(&root.join("tests/fixtures"), &mut files);
+    hdf5_files(&root.join("../clawhdf5-format/tests/fixtures"), &mut files);
+    files.sort();
+    let mut compared = 0;
+    for path in &files {
+        let Ok(bytes) = std::fs::read(path) else {
+            continue;
+        };
+        let Ok(local) = File::open(path) else {
+            continue;
+        };
+        let mut remote = File::open_storage(Arc::new(Overlong(bytes)))
+            .unwrap_or_else(|e| panic!("{}: {e}", path.display()));
+        remote.set_vds_resolver(sibling_resolver(path.parent().map(Path::to_path_buf)));
+        assert_eq!(
+            transcript(&local),
+            transcript(&remote),
+            "{}",
+            path.display()
+        );
+        compared += 1;
+    }
+    assert!(compared >= 40, "{compared}");
+}
+
+/// The harness tells failures apart: two different errors are two
+/// different transcripts, and only a difference `File::open` itself
+/// produces between two opens is let through.
+#[test]
+fn harness_compares_errors_not_just_failures() {
+    let a: Result<(), FormatError> = Err(FormatError::ContiguousStorageRequired("x"));
+    let b: Result<(), FormatError> = Err(FormatError::Storage("x".into()));
+    assert_ne!(value(&a), value(&b));
+    let path = Path::new(env!("CARGO_MANIFEST_DIR"))
+        .join("../clawhdf5-format/tests/fixtures/chunked_2d.h5");
+    let want = transcript(&File::open(&path).unwrap());
+    assert_eq!(unexplained_difference(&path, &want, &want), None);
+    // A read that fails differently through the storage.
+    let line = want.lines().position(|l| l.contains(" all ")).unwrap();
+    let (mut w, mut g): (Vec<String>, Vec<String>) = (
+        want.lines().map(String::from).collect(),
+        want.lines().map(String::from).collect(),
+    );
+    w[line] = format!(
+        "{} Err(Format(DataSizeMismatch))",
+        &w[line][..w[line].find(" all ").unwrap() + 4]
+    );
+    g[line] = format!(
+        "{} Err(Format(Storage(\"injected\")))",
+        &g[line][..g[line].find(" all ").unwrap() + 4]
+    );
+    assert!(unexplained_difference(&path, &w.join("\n"), &g.join("\n")).is_some());
+    // A value against an error is never let through.
+    assert!(unexplained_difference(&path, &want, &g.join("\n")).is_some());
+}
+
+/// `File::storage` is the view `as_bytes` gives, for every backend: the
+/// user block skipped, bounded by the end of file, a cache image laid over.
+#[test]
+fn file_storage_is_the_as_bytes_view_for_every_backend() {
+    let root = Path::new(env!("CARGO_MANIFEST_DIR"));
+    let mut files = Vec::new();
+    hdf5_files(&root.join("tests/fixtures"), &mut files);
+    hdf5_files(&root.join("../clawhdf5-format/tests/fixtures"), &mut files);
+    let mut compared = 0;
+    for p in files {
+        let Ok(local) = File::open(&p) else { continue };
+        let bytes = std::fs::read(&p).unwrap();
+        let remote = File::open_storage(Arc::new(CountingStorage::new(bytes))).unwrap();
+        let want = local.as_bytes();
+        let view = local.storage();
+        assert_eq!(view.as_contiguous(), Some(want), "{}", p.display());
+        let got = remote.storage();
+        assert!(got.as_contiguous().is_none());
+        assert_eq!(got.len(), want.len() as u64, "{}", p.display());
+        assert_eq!(
+            &*got.read_at(0, want.len()).unwrap(),
+            want,
+            "{}",
+            p.display()
+        );
+        compared += 1;
+    }
+    assert!(compared >= 40, "{compared}");
+}

docs/design/range-reads.md

@@ -1,9 +1,24 @@
 # Design: range reads (reading HDF5 without holding the whole file)
 
 Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
+change. Progress: M0 and M1 are done, and so is M2 (branch
+`feat/p3-m2-raw-data`): every read path of the format crate works through
+`Storage`, v2 B-trees, dense groups and raw data included, and
+`File::open_storage` gives the facade's read API over any `Storage` (see
+`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
+`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
+object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
+next. Every count in §1–§2 was
+
 change. Progress: M1, first part (the `Storage` trait and the metadata
 parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
-group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was
+group B-tree v2 lookups, dense groups and the facade are not converted yet.
+Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched
+converted code without changing the plan: object-header continuation chunks
+are followed without recursion (still one bounded `read_at` per chunk), and
+implicit chunk indexes are addressed over the maximum chunk grid (in
+`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps
+working on the whole file in memory; it is not part of this design. Every count below was
 taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
 next to it. No timing numbers appear here on purpose: the machine was shared
 with other build jobs when this was written.
@@ -421,6 +436,36 @@ fast path within benchmark noise.
   on other backends (they already return `Option`/`Result`).
 - Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
   keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
+- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data`. As planned,
+  with these choices:
+  - `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the
+    file handle is shared by its datasets and may be sent across threads).
+    The file's view (user block skipped, bounded by the recorded end of
+    file, cache image laid over its reads) is itself a `Storage`, and the
+    facade calls the generic cores with it; for a `Vec` or an mmap its
+    `as_contiguous()` is the buffer, so the local paths are the slice code
+    (checked: identical conformance results; the bench gate below still
+    has to be run on an idle machine).
+  - Chunked reads fetch in batches of at most 64 MiB of stored bytes, one
+    `read_ranges` call each, decoding each batch before fetching the next,
+    on every chunk-reading path (one helper,
+    `storage::for_each_extent_batch`); and no chunk fetches more than its
+    decoded size can need (`filters::stored_chunk_limit`), so a remote read
+    holds at most one batch undecoded even over a crafted chunk index;
+    chunks already in the chunk cache are not fetched.
+  - The typed readers' zero-copy fast path became "read the contiguous
+    bytes once": over a range storage a contiguous `read_f64` is one read,
+    and a native contiguous selection reads only its runs.
+  - External VDS source files stay whole-file, through a resolver that
+    returns bytes (`File::set_vds_resolver`).
+  - The v2 B-tree and dense groups were done here rather than in M3, so no
+    format-crate path answers `ContiguousStorageRequired` any more; only the
+    facade's zero-copy methods do.
+  - Measured with the M2 harness (`crates/clawhdf5/tests/storage_equivalence.rs`,
+    tank, 2026-09-26): one pass over the 621 corpus files that open — list,
+    every attribute, every dataset once — is 176 092 `read_at` calls and
+    208 MB through a storage with no cache (254 MB of files). The block
+    cache of M3 is what turns that into requests (§2).
 
 **M3 — HTTP/S3 backend (1–2 weeks).**
 - `clawhdf5-io`, feature `remote` (off by default, so the default tree stays
@@ -429,6 +474,44 @@ fast path within benchmark noise.
   §2; the page size for paged files; the first block prefetched on open) and
   a request counter exposed for tests and users.
 - Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
+- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the
+  Python bindings, with these choices:
+  - A new crate, `clawhdf5-remote`, instead of a `remote` feature of
+    `clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
+    sits below the facade.
+  - HTTP(S) through `ureq` (`HttpStorage`), not object_store's HTTP store:
+    that one pulls reqwest with aws-lc-rs (C), while plain HTTP through
+    ureq builds no C, so it is the default feature; `https` adds rustls
+    with ring. S3/GCS/Azure go through `object_store` (`ObjectStoreStorage`,
+    features `s3`/`gcs`/`azure`, opt-in because of aws-lc-rs); the
+    `object-store` feature alone (in-memory, local files, a store you
+    build) is pure Rust. object_store is async: each read runs on a
+    two-thread tokio runtime of the storage's own while the caller waits,
+    so the caller's context (a plain thread, `spawn_blocking`, another
+    runtime) does not matter.
+  - `BlockCache` (any `Storage`): 1 MiB blocks and a 64 MiB LRU budget by
+    default, the first block fetched at open (for HTTP by the request
+    that learns the length), the missing blocks of one read fetched as
+    runs of consecutive blocks in one parallel batch, per-block in-flight
+    deduplication across threads, and reads that miss more than half the
+    budget not kept. The page size of paged files is not used as the block
+    size yet. `CacheStats` and `HttpStats` count requests and bytes.
+  - The file is pinned at open by ETag (else Last-Modified, or the object's
+    version) and length; a change is an error, not mixed data.
+  - `h5rs` (feature `remote`) reads through the new `File::storage()`, the
+    file's view as a `Storage`, so its parsing works on remote files; its
+    `check` downloads the file whole.
+  - Measured with `crates/clawhdf5-remote/tests/http.rs` (tank, 2026-09-26,
+    `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus
+    CLAWHDF5_REMOTE_REPORT=1 cargo test --release -p clawhdf5-remote --test
+    http -- --nocapture corpus`), requests as the test server counted
+    them: the 621 corpus files that open read over HTTP exactly as through
+    `File::open`. Open + list (every group's entries, every dataset's shape
+    and type) of all of them: 640 requests, 55.5 MB of 254 MB; then reading
+    each file's largest dataset under 64 MiB: 96 more (171 MB in all). The
+    same work without a cache: 141 936 requests. Per file: A lists in 2
+    requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
+    6.4 MB: 35 001 object headers spread over the file).
 
 **M4 — wasm lazy loading (1–2 weeks).**
 - `clawhdf5-wasm`: `openUrl(url) -> Promise<H5File>` backed by `fetch` with a

docs/known-issues.md

@@ -7,6 +7,41 @@ deleting it.
 
 ---
 
+## Fletcher-32 checksums disagreed with libhdf5 on about 1 chunk in 32768
+
+**Status:** fixed 2026-09-26, after v2.7.0. **Every release (v2.1.0 to
+v2.7.0) is affected**, in both directions.
+
+Our Fletcher-32 reduced its two running sums with `% 65535`; libhdf5's
+`H5_checksum_fletcher32` (H5checksum.c) folds them with
+`(s & 0xffff) + (s >> 16)`. Both are arithmetic mod 65535, but where a sum
+is a non-zero multiple of 65535 the fold leaves 0xffff and the modulo 0, so
+the checksums differ — for random data about one chunk in 32768 (each of
+the two sums hits it with probability about 1/65535). Found by the review
+of the editor work: a random-edit fuzzer with gzip + Fletcher-32 hit it on
+13 of about 100 seeds.
+
+- Chunks we wrote (`FileBuilder`/`FileWriter` `with_fletcher32`, and the
+  unreleased `FileEditor`) with such a sum are refused by h5py and libhdf5:
+  "filter returned failure during read". h5py writing `[1, 0xfffe]` as
+  big-endian `u2` stores checksum `0x0001ffff`; we computed `0x00010000`.
+- Chunks libhdf5 wrote with such a sum were refused by every reader here
+  with `Fletcher32Mismatch`; the data itself was never wrong.
+
+**Fix:** `clawhdf5_format::checksum::fletcher32`, a port of
+`H5_checksum_fletcher32`, used by the filter for writing and verifying. It
+also accepts a checksum whose 16-bit halves are byte-swapped, as libhdf5
+does for files from 1.6.2 and earlier, and the `% 65535` form clawhdf5
+v2.7.0 and earlier wrote (the two differ only in a half that is 0xffff).
+**Test:**
+`crates/clawhdf5/tests/fletcher32_interop.rs` (libhdf5's own function
+through ctypes on every 1- and 2-byte input plus 40 000 random and
+fold-heavy inputs; h5py reads fold-case chunks from `FileBuilder` and
+`FileEditor`; we read h5py's). **Existing data:** a Fletcher-32 dataset
+written by v2.7.0 or earlier may hold chunks libhdf5 cannot read; a fixed
+build reads them. Rewrite such datasets with a fixed build (read, then
+write them again) before handing the file to libhdf5 or h5py.
+
 ## LZF/Blosc chunks written with a stale filter mask
 
 **Status:** fixed 2026-09-26, before any release (the LZF and Blosc writers
@@ -26,40 +61,57 @@ libhdf5 modify them.
 
 ## In-place modification (`FileEditor`) limits
 
-**Status:** open (documented 2026-09-26). `clawhdf5::FileEditor` refuses,
+**Status:** open (documented 2026-09-26, updated the same day when
-with `Error::Unsupported` and without writing anything:
+version-2 B-tree chunk indexes, shrinking, dense attributes and space
-- new, moved or resized chunks in a **version-2 B-tree** chunk index (what
+reuse were added). `clawhdf5::FileEditor` refuses, with
-  libhdf5 uses for two or more unlimited dimensions) — existing unfiltered
+`Error::Unsupported` and without writing anything:
-  chunks, and filtered ones that re-encode to the same size and filter
+- new chunks in an **implicit** index (it has all of its chunks from the
-  mask, are
+  start; they are written in place, and allocated/filled on growth under
-  overwritten in place; `resize` works — and new chunks in an **implicit**
+  early allocation as libhdf5 does);
-  index (it has all of its chunks from the start);
-- **shrinking** a dataset;
 - variable-length and reference data;
 - chunks through a filter this build cannot encode (scale-offset, N-Bit,
   SZIP, or a plugin filter it lacks), even an optional one: libhdf5 skips
   an optional filter only when its own build lacks it, which none does for
   these;
-- attributes of an object in **dense storage**, past its compact limit (8
+- attributes in dense storage when the heap cannot take them the way
-  by default) or with tracked **creation order**;
+  libhdf5 would: replacing the last attribute left in a heap block by one
+  of another size (libhdf5 frees the block), a heap with I/O filters or
+  child indirect blocks (more than about 512 KiB of attributes), free
+  space in child indirect blocks, directly addressed huge objects; and
+  shared attribute messages. Measured 2026-09-26 on tank with the review's
+  random-edit harness (120 runs of 150 random edits, `earliest`/`v110`/
+  `latest`, about 5600 `set_attr` calls of 8 bytes to 6 KiB): 2.2% of
+  `set_attr` calls are refused, every one the last-attribute-in-a-block
+  replacement; before blocks could be skipped (an attribute needing a heap
+  block larger than the next one — any attribute of about 1 KiB or more
+  once a heap has started, or at the move to dense storage), 24% were,
+  since an object whose move to dense storage was refused kept refusing
+  every new attribute;
+- version-1 object headers asked for an attribute larger than a header
+  message (they have no dense storage);
 - partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
   raw data files, virtual datasets;
 - files with a metadata cache image, paged or persistent free-space
   management, a driver info block, or version-3 consistency flags set.
 
-**Space is never reused.** There is no free-space manager: the old bytes of
+**Space is reused only within one editor.** Space an edit frees (a filtered
-a filtered chunk that grows and has to move, and of an attribute that is
+chunk that moves, chunks a shrink removes, B-tree nodes merged away, a
-replaced by a larger one, are leaked (`h5repack` reclaims them). A chunk
+heap's replaced blocks) is reused by later edits of the same `FileEditor`;
-that is the last thing in the file grows in place instead, which covers the
+what is left when it is dropped is leaked, as libhdf5 leaks it without a
-usual append. Measured 2026-09-26 on tank with
+persistent free-space manager (`h5repack` reclaims it). A chunk that is the
-`cargo test --release -p clawhdf5-tools --test edit_interop -- --ignored
+last thing in the file grows in place, which covers the usual append.
---nocapture measure_append_waste` (file sizes are deterministic): 1000
+Measured 2026-09-26 on tank with `cargo test -p clawhdf5-tools --test
-appends of 100 `f8` values to a 1-D dataset with 1024-element chunks give
+edit_interop -- --ignored --nocapture measure_append_waste` (one editor for
-810 504 bytes unfiltered, as libhdf5's file, and 307 210 bytes with gzip
+the whole workload; file sizes are deterministic): 1000 appends of 100 `f8`
-(libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values with
+values to a 1-D dataset with 1024-element chunks give 810 504 bytes
-4096-element gzip chunks give 119 684 bytes against libhdf5's 50 292
+unfiltered, as libhdf5's file (`h5repack` of either: 810 360), and 306 780
-(`h5repack`: 49 930), because the chunk being appended to is followed by
+bytes with gzip (307 210 before reuse; libhdf5's file: 306 058; `h5repack`
-new index blocks and moves each time it grows.
+of the editor's file: 306 104, of libhdf5's: 305 954); 2000 appends of 10
+values with 4096-element gzip chunks give 79 829 bytes (119 684 before
+reuse) against libhdf5's 50 292 (`h5repack` of the editor's file: 49 930,
+of libhdf5's: 50 188): the chunk being appended to
+is followed by new index blocks and moves each time it grows, and the
+space it leaves is too small for its next, larger version.
 
 **No journal.** A crash while an edit patches existing structures can leave
 the file inconsistent; see the `FileEditor` documentation.
@@ -763,6 +815,92 @@ the same agent-store interop test.
 **Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
 which is what libhdf5 itself writes.
 
+## Range reads (`File::open_storage`) limits
+
+**Status:** open (added 2026-09-26, milestone M2 of
+`docs/design/range-reads.md`; remote backends added by M3). `File::open_storage`
+reads any `clawhdf5_format::storage::Storage` through the whole read API,
+every format-crate read path works through `Storage::read_at`/`read_ranges`,
+and `clawhdf5-remote` serves HTTP(S) and object-store files through a block
+cache, but:
+
+- **A `Storage` without a cache is asked for each structure as the parsers
+  need it**, several times over for some (an object header is re-read by
+  each lookup through it): one pass over the conformance corpus — open,
+  list, every attribute, every dataset once — is 176 092 `read_at` calls
+  for 621 files, 92 489 of them for the 35 001-group `h5stat_newgrat.h5`
+  (2026-09-26, tank, `crates/clawhdf5/tests/storage_equivalence.rs` with
+  `CLAWHDF5_STORAGE_CORPUS`). A backend of your own over a network needs a
+  cache in front of it: wrap it in `clawhdf5_remote::BlockCache`, as
+  `open_url` does. `Storage::read_ranges` defaults to one `read_at` per
+  range; coalescing is the backend's (or the cache's) job.
+- A group lookup by name in a version-1 (symbol-table) group lists the whole
+  group (dense groups use their name index). Over a range backend that is
+  one read per symbol-table node and name, per lookup.
+- External virtual-dataset source files are loaded whole through the
+  resolver (`File::set_vds_resolver`), as bytes; they are not read through
+  a `Storage`.
+- The zero-copy methods (`Dataset::read_raw_ref`, `read_as_slice`,
+  `read_*_zerocopy`) need the file in memory and answer
+  `FormatError::ContiguousStorageRequired` otherwise; `File::as_bytes()`
+  panics for such a file (`File::contiguous_bytes()` is the fallible form).
+  `LazyFile`, `MmapFile` and the Python and wasm bindings still read a
+  whole file (`h5rs` reads through `File::storage`, and takes URLs with its
+  `remote` feature).
+- The file's length is read once, at open: a growing file (SWMR) is not
+  followed (milestone M5). A remote file is pinned at open, so one that
+  grows is `RemoteError::FileChanged`.
+- Not new, but visible through the equivalence tests: a full read through
+  the file's chunk cache (`read_raw_data_cached`, `read_raw_data_indexed`,
+  and so `Dataset::read_*`) lists a damaged dataset's chunks in hash-map
+  order, so which failing chunk it reports can differ from one `File` to
+  the next (`cve-2025-2310.h5`); the values of a dataset that reads are
+  not affected.
+
+## Remote files (`clawhdf5-remote`) limits
+
+**Status:** open (added 2026-09-26, milestone M3 of
+`docs/design/range-reads.md`).
+
+- **Python and the browser cannot open URLs yet.** `clawhdf5.File` (PyO3)
+  parses through `File::as_bytes`, which a remote file does not have; the
+  wasm reader's `openUrl` is milestone M4.
+- **The block size is fixed** (1 MiB unless `CacheConfig` says otherwise).
+  The design's policy of using a paged file's page size as the block size
+  is not implemented, and only the first block is read ahead.
+- **Checked against local servers and one public one.** The tests use an
+  in-process HTTP/1.1 server and object_store's in-memory and local-file
+  stores. HTTPS was checked by hand against `raw.githubusercontent.com`
+  (2026-09-26, tank: `h5rs dump` of h5py's `vlen_string_dset.h5` by URL
+  equals the downloaded file's). The `s3`, `gcs` and `azure` backends are
+  built and their URL parsing tested, but they have not been run against a
+  real bucket.
+- **A server with neither a strong ETag nor Last-Modified** can only be
+  checked by length, so a same-length replacement mid-read would go
+  unnoticed; `HttpOptions::require_validator` refuses such servers. A weak
+  ETag (`W/"…"`) cannot be sent as `If-Match`, so it counts as none.
+- **Credentials:** HTTP takes extra headers (`HttpOptions::headers`, e.g.
+  `Authorization`); `h5rs` has no option for them. The cloud stores read
+  credentials from the environment only. Messages and `Debug` output show
+  URLs through `redact_url` (no userinfo, query values `REDACTED`);
+  `HttpStorage::url()` returns the URL as given and must not be logged.
+- **Redirects:** at most `HttpOptions::max_redirects` (5) per request,
+  never from `https` to `http`, and the custom headers are not sent once a
+  redirect leaves the URL's origin. The redirect target is not remembered:
+  every request of a redirected file costs its hops again.
+- **Timeouts:** ureq has no idle timeout, only a total one for the body,
+  so the body's budget is `timeout + size / min_speed` (30 s + 16 KiB/s
+  by default). A connection that stalls mid-body is detected only when
+  that budget runs out (94 s for a 1 MiB block, 9 min for an 8 MiB run).
+- Each `ObjectStoreStorage` owns a tokio runtime with two worker threads.
+  A read called from inside another runtime blocks that runtime's thread
+  for its duration (it works, but `spawn_blocking` is the better place).
+- `h5rs check` downloads a remote file whole (it validates every byte), up
+  to `--max-download` (1 GiB by default), and a URL cannot carry a
+  `FILE/OBJECT` suffix; `h5rs` uses the default cache settings.
+- The zero-copy methods and `File::as_bytes` are unavailable on a remote
+  file (see the range-read limits above).
+
 ## `clawhdf5-wasm` (browser) limits
 
 **Status:** open (by design for now; added 2026-09-26).

scripts/ci-test.sh

@@ -109,23 +109,45 @@ run_step "cargo clippy (fast-deflate / zlib-ng)" cargo clippy \
     --features clawhdf5-format/fast-deflate,clawhdf5-filters/fast-deflate \
     -- -D warnings
 
+# clawhdf5-remote's optional backends: object_store (in-memory and local
+# stores in the tests), HTTPS through rustls, and the cloud stores.
+run_step "cargo clippy (remote, all backends)" cargo clippy \
+    -p clawhdf5-remote \
+    --all-targets \
+    --features object-store,https,s3,gcs,azure \
+    -- -D warnings
+
+# h5rs with URL arguments.
+run_step "cargo clippy (h5rs remote)" cargo clippy \
+    -p clawhdf5-tools \
+    --all-targets \
+    --features remote-https \
+    -- -D warnings
+
 # The README promises that the core crates build no C by default. Hold it to
 # that: fail if a crate that compiles C (a *-sys crate, cc or cmake) enters the
 # default dependency tree of any of them. clawhdf5-migrate (bundled SQLite),
 # clawhdf5-napi (Node) and clawhdf5-gpu (graphics drivers) are exempt.
 # js-sys (clawhdf5-wasm's bindings to JavaScript) builds no C.
+# clawhdf5-remote is checked by default (plain HTTP) and with its
+# object-store feature, and h5rs with URL support (remote); the https
+# (ring) and s3/gcs/azure (aws-lc-rs) features build C and are opt-in.
 no_c_in_default_build() {
-    local crate found=0
+    local entry crate features found=0
-    for crate in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \
+    for entry in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \
         clawhdf5-agent clawhdf5-ann clawhdf5-accel clawhdf5-netcdf4 clawhdf5-cli \
         clawhdf5-tools \
-        clawhdf5-wasm; do
+        clawhdf5-wasm \
+        clawhdf5-remote clawhdf5-remote:object-store clawhdf5-tools:remote; do
+        crate=${entry%%:*}
+        features=()
+        [ "$entry" != "$crate" ] && features=(--features "${entry#*:}")
         local c_deps
-        c_deps=$(cargo tree -q -p "$crate" -e normal,build --prefix none \
+        c_deps=$(cargo tree -q -p "$crate" "${features[@]}" -e normal,build --prefix none \
             | grep -E '^([a-z0-9_-]+-sys|cc|cmake) v' \
             | grep -v '^js-sys v' | sort -u)
         if [ -n "$c_deps" ]; then
-            echo "$crate pulls in C by default:"
+            echo "$entry pulls in C by default:"
             echo "$c_deps" | sed 's/^/    /'
             found=1
         fi
@@ -191,6 +213,15 @@ run_step "cargo test (facade parallel)" cargo test \
     -p clawhdf5 \
     --features parallel
 
+run_step "cargo test (remote, object_store backend, s3 URLs)" cargo test \
+    -p clawhdf5-remote \
+    --features object-store,s3
+
+run_step "cargo test (h5rs on URLs)" cargo test \
+    -p clawhdf5-tools \
+    --features remote \
+    --test remote
+
 run_step "cargo test (ann parallel)" cargo test \
     -p clawhdf5-ann \
     --features parallel