format: slice entry points for the facade's hot *_in calls (local listing back to main's speed)

Since the M2 merge the facade handed in-memory files to the generic
`*_in` parsers as `&[u8]` (`with_bytes!`), which instantiates them in
the facade crate, where the format crate's private helpers do not
inline without LTO: listing a 400-group v1 file through `File::open`
was 7-10% slower than main. `ObjectHeader::parse_in`,
`group_v2::{resolve_child_in, resolve_group_children_in,
resolve_path_any_in}` and `attribute::{extract_attributes_tolerant_in,
find_attribute_in}` now pass a storage with `as_contiguous()` to their
non-generic slice entry point, compiled once in the format crate; other
storages reach the same generic core as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

.gitea/workflows/ci.yml

@@ -22,6 +22,9 @@ jobs:
         run: rustup component add rustfmt clippy
       - name: Install thumbv7em-none-eabihf target
         run: rustup target add thumbv7em-none-eabihf
+      - name: Install wasm32-unknown-unknown target
+        # ci-test.sh builds the reader and clawhdf5-wasm for the browser.
+        run: rustup target add wasm32-unknown-unknown
       - name: Install Python interop dependencies
         # The interop suites used to skip silently when python3/h5py were
         # missing, so they never ran in CI. Install them and make a missing
@@ -31,12 +34,20 @@ jobs:
           # cmake builds libz-ng-sys for the opt-in `fast-deflate` (zlib-ng)
           # steps in ci-test.sh; rust:latest does not ship it. The default
           # build (pure-Rust zlib-rs) does not need it.
-          apt-get install -y --no-install-recommends python3 python3-venv cmake
+          # hdf5-tools: h5ls/h5stat/h5dump/h5diff, which the h5rs
+          # (clawhdf5-tools) interop tests compare against.
+          apt-get install -y --no-install-recommends python3 python3-venv cmake hdf5-tools
           python3 -m venv /opt/interop
-          /opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray hdf5plugin
+          # maturin + pytest: ci-test.sh builds the Python package
+          # (crates/clawhdf5-py) and runs its tests against h5py.
+          /opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray hdf5plugin maturin pytest
           echo "/opt/interop/bin" >> "$GITHUB_PATH"
       - name: Show interop library versions
-        run: /opt/interop/bin/python -c "import h5py, netCDF4, hdf5plugin; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'netCDF4', netCDF4.__version__, 'hdf5plugin', hdf5plugin.version)"
+        # h5dump's version too: the h5rs dump test requires its exact output
+        # (checked against Debian's 1.14.5 in rust:latest and 1.14.6).
+        run: |
+          /opt/interop/bin/python -c "import h5py, netCDF4, hdf5plugin; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'netCDF4', netCDF4.__version__, 'hdf5plugin', hdf5plugin.version)"
+          h5dump --version
       - name: Run CI script
         env:
           # Name the interpreter outright rather than relying on $GITHUB_PATH

.gitignore

@@ -5,3 +5,5 @@ benchmarks/longmemeval/*.json
 # Local model weights (MiniLM etc.) — large, not committed
 weights/
 .venv
+__pycache__/
+.pytest_cache/

BENCHMARKS.md

@@ -482,6 +482,219 @@ The rows and columns of the uncompressed layouts are within 20% (chunked
 column 0.45 -> 0.49 ms, contiguous column 2.55 -> 2.61 ms). This run does not
 explain the slower windows.
 
+## Concurrent reads
+
+### Results after in-place chunk decoding (2026-09-26, tank, `c5334b1`)
+
+Same machine, files and commands, re-run after chunked reads started
+decoding into reusable per-thread buffers straight into the (typed) output,
+with the calling thread decoding alongside the pool. Load average 1.78 at
+the start; it rose to 6-9 during the runs (the clawhdf5 runs' own threads,
+and it stayed around 5-6 through the h5py runs, so something else was
+active). **This run was noisier than the previous one: h5py's own contiguous
+figures are about 40% lower than in the run below, and ours dropped
+similarly, so compare ratios within a run rather than MB/s across runs.**
+h5py was re-run in the same session.
+
+Each read decoding on its calling thread (`--decode-threads 1`, like h5py):
+
+| layout | mode | threads | clawhdf5 MB/s (eff) | h5py threads MB/s (eff) | h5py processes MB/s (eff) | vs h5py processes |
+|---|---|---:|---:|---:|---:|---:|
+| deflate | distinct | 1 | 670 (1.00) | 410 (1.00) | 397 (1.00) | 1.69x |
+| deflate | distinct | 4 | 2434 (0.91) | 406 (0.25) | 1470 (0.93) | 1.66x |
+| deflate | distinct | 8 | 3749 (0.70) | 406 (0.12) | 2398 (0.76) | 1.56x |
+| deflate | distinct | 16 | 4944 (0.46) | 390 (0.06) | 3135 (0.49) | 1.58x |
+| deflate | same | 1 | 211 (1.00) | 125 (1.00) | 124 (1.00) | 1.70x |
+| deflate | same | 16 | 1835 (0.54) | 122 (0.06) | 961 (0.48) | 1.91x |
+| contiguous | distinct | 1 | 6718 (1.00) | 5545 (1.00) | 5200 (1.00) | 1.29x |
+| contiguous | distinct | 16 | 11035 (0.10) | 4950 (0.06) | 10558 (0.13) | 1.05x |
+| contiguous | same | 1 | 14483 (1.00) | 2593 (1.00) | 2737 (1.00) | 5.29x |
+| contiguous | same | 16 | 132175 (0.57) | 2224 (0.05) | 14809 (0.34) | 8.93x |
+
+With the default rayon pool, deflate `distinct` reads 6143 MB/s from a single
+thread (15x h5py's 410 on one call) and 4556 MB/s at 16 threads (1.45x h5py
+processes); the other rows are within the noise of the table above.
+
+What changed: full reads of chunked datasets were 0.69x-0.76x of h5py
+processes at 16 threads in the run below, and are 1.58x here; with one
+thread they were 1.44x and are 1.69x. Minor page faults for the 16-thread
+run fell from about 4.6M to 0.2M (`/usr/bin/time -v`, provisional, loaded
+machine). clawhdf5 now reads faster than 16 h5py processes in every row of
+this benchmark except contiguous full reads at 16 threads, where both
+saturate memory bandwidth (1.05x).
+
+### Results after the read fixes (2026-09-26, tank, `408f69e`)
+
+Same machine, files and commands as the first run below, re-run on an idle
+tank (load average 1.60 at the start; the 1-minute figure rose to about 5
+during the clawhdf5 runs, mostly their own threads) after two fixes:
+contiguous reads back their output with transparent huge pages and copy
+hyperslabs run by run, and full chunked reads no longer queue behind a
+one-thread rayon pool. h5py was re-run in the same session.
+
+Each read decoding on its calling thread (`--decode-threads 1`, like h5py):
+
+| layout | mode | threads | clawhdf5 MB/s (eff) | h5py threads MB/s (eff) | h5py processes MB/s (eff) |
+|---|---|---:|---:|---:|---:|
+| deflate | distinct | 1 | 606 (1.00) | 432 (1.00) | 421 (1.00) |
+| deflate | distinct | 4 | 1816 (0.75) | 428 (0.25) | 1654 (0.98) |
+| deflate | distinct | 8 | 2943 (0.61) | 428 (0.12) | 3042 (0.90) |
+| deflate | distinct | 16 | 2142 (0.22) | 375 (0.05) | 3083 (0.46) |
+| deflate | same | 1 | 154 (1.00) | 130 (1.00) | 129 (1.00) |
+| deflate | same | 4 | 599 (0.98) | 129 (0.25) | 499 (0.97) |
+| deflate | same | 16 | 1592 (0.65) | 128 (0.06) | 1399 (0.68) |
+| contiguous | distinct | 1 | 13665 (1.00) | 9490 (1.00) | 8781 (1.00) |
+| contiguous | distinct | 16 | 12674 (0.06) | 2285 (0.02) | 6942 (0.05) |
+| contiguous | same | 1 | 31991 (1.00) | 5087 (1.00) | 5078 (1.00) |
+| contiguous | same | 16 | 237151 (0.46) | 4304 (0.05) | 35772 (0.44) |
+
+With the default rayon pool: deflate `distinct` 2117 MB/s at 1 thread (4.9x
+h5py), 3163 at 4, 2341 at 16 (0.76x h5py processes); deflate `same` 1439 MB/s
+at 16; contiguous as above within a few percent.
+
+Before -> after for clawhdf5 (`--decode-threads 1` unless noted):
+contiguous full read at 1 thread 2495 -> 13665 MB/s (0.25x -> 1.44x h5py);
+contiguous 256 x 256 hyperslabs at 1 thread 624 -> 31991 MB/s (0.12x ->
+6.3x); deflate full reads at 8 threads 887 -> 2943 MB/s; deflate
+hyperslabs at 16 threads 1244 -> 1592 MB/s.
+
+Read with care:
+- `contiguous same` reads 1024 slabs of one 64 MiB dataset over and over, so
+  it mostly measures copies out of the CPU's caches (the 7800X3D has 96 MiB
+  of L3); the per-call overhead is what differs (h5py's is about 50 us).
+- At 16 threads every tool dropped in this run (h5py threads on contiguous
+  data from 8002 to 2285 MB/s, processes from 12846 to 6942), so the
+  16-thread rows are noisier than the others.
+- Still behind: full reads of chunked data at 16 threads (0.69x-0.76x h5py
+  processes). See `docs/known-issues.md`.
+
+### First run, before the read fixes (2026-09-26, tank, `91644d8`)
+
+Measured on tank (AMD Ryzen 7 7800X3D, 8 cores / 16 threads, 61 GiB, Linux
+7.0) at commit `91644d8`, load average 1.84 when the run started (the
+1-minute figure rose to 3.7 during the runs; that is mostly the benchmark's
+own threads). Warm page cache. clawhdf5 2.7.0 (workspace), h5py 3.16.0 on
+HDF5 2.0.0. Commands exactly as in the **Run** box below; files at their
+defaults (64 datasets of 16384 x 1024 `f32`, 64 MiB each; deflate chunks
+256 x 256, level 4). MB/s is decoded data, the median of the repetitions;
+eff is scaling efficiency against the same tool's 1-thread row.
+
+Each read decoding on its calling thread (`--decode-threads 1`, like h5py):
+
+| layout | mode | threads | clawhdf5 MB/s (eff) | h5py threads MB/s (eff) | h5py processes MB/s (eff) |
+|---|---|---:|---:|---:|---:|
+| deflate | distinct | 1 | 421 (1.00) | 433 (1.00) | 421 (1.00) |
+| deflate | distinct | 4 | 890 (0.53) | 428 (0.25) | 1651 (0.98) |
+| deflate | distinct | 16 | 880 (0.13) | 427 (0.06) | 4424 (0.66) |
+| deflate | same | 1 | 151 (1.00) | 130 (1.00) | 129 (1.00) |
+| deflate | same | 4 | 490 (0.81) | 129 (0.25) | 497 (0.96) |
+| deflate | same | 16 | 1244 (0.52) | 128 (0.06) | 1402 (0.68) |
+| contiguous | distinct | 1 | 2495 (1.00) | 9789 (1.00) | 9169 (1.00) |
+| contiguous | distinct | 16 | 8083 (0.20) | 8096 (0.05) | 12272 (0.08) |
+| contiguous | same | 1 | 624 (1.00) | 5022 (1.00) | 5172 (1.00) |
+| contiguous | same | 16 | 4778 (0.48) | 4411 (0.05) | 37138 (0.45) |
+
+With the default rayon pool decoding inside each read, deflate `distinct`
+is 912 MB/s at 1 thread (2.1x h5py) and 2824 MB/s at 16 (6.6x h5py threads,
+0.64x h5py processes); the other rows are within a few percent of the table
+above. Full tables (2, 4, 8 threads, both decode modes) come from
+`compare_concurrent_read.py` on the JSON files.
+
+What this shows:
+- **h5py threads do not scale** (flat at about 430 MB/s on deflate, every
+  thread count): libhdf5's global lock.
+- **clawhdf5 threads on one `File` do, for hyperslab reads of compressed
+  data:** 1244 MB/s at 16 threads, 9.7x h5py threads and 0.89x h5py
+  processes, without a process pool.
+- **Where clawhdf5 is behind** (open performance bugs, see
+  `docs/known-issues.md`):
+  - *Full reads of chunked datasets stop scaling at about 4 threads*
+    (about 880 MB/s) while h5py processes reach 4424 MB/s. Hyperslab
+    reads, which bypass the `File`'s chunk cache, keep scaling, so the
+    cache (one mutex and one 16 MiB budget per `File`, thrashed by 64 MiB
+    datasets) is the suspect. The cause of the `--decode-threads 1`
+    ceiling was not the cache: every full read queued its chunks for the
+    pool's single rayon worker. That case was fixed after these
+    measurements (2026-09-26, not yet re-measured here). With the default
+    pool the gap to h5py processes remains (see `docs/known-issues.md`).
+  - *Contiguous reads are slow*: 2.5 GB/s for a single-threaded full read
+    against h5py's 9.8 GB/s (0.25x), and 0.12x for 256 x 256 hyperslabs.
+    Threads close the gap (about 1.0x h5py at 16), but single-thread
+    contiguous I/O is a real deficit.
+
+The question: libhdf5's threadsafe build serialises every API call under one
+global mutex, and h5py holds a global lock around every call too, so threads
+reading through h5py cannot decode in parallel; h5py users scale with
+processes. A clawhdf5 `File` is `Send + Sync`, and nothing on the read paths
+this harness uses (`read_f32`, `read_f32_selection`) takes a library-wide
+lock: the one mutex is the `File`'s chunk cache (keyed per dataset), taken by
+full reads of chunked datasets for each chunk's O(1) lookup and insert, never
+across a decode; hyperslab reads do not use the cache. How does
+decoded throughput scale with threads on one open file, against h5py threads
+and h5py processes on the same files?
+
+Workload (`crates/clawhdf5-bench/src/bin/concurrent_read.rs`; the h5py script
+mirrors it): `<dir>/deflate.h5` and `<dir>/contiguous.h5`, each with 64 `f32`
+datasets of 64 MiB decoded (`[16384, 1024]`; the deflate file chunked
+`256 x 256`, level 4), written by clawhdf5 on first use and reused while
+`manifest.json` matches. The data is a slowly varying ramp plus 8 bits of
+noise per element, every value exact in `f32`, so both harnesses check what
+they read; it deflates about 3.1x (128 MiB -> 40.7 MiB for two 64 MiB
+datasets). For each layout and thread count
+(1, 2, 4, 8, 16; fixed total work per repetition, split among the threads):
+
+- `distinct`: every dataset read in full once, thread `t` taking datasets
+  `t, t + T, ...`;
+- `same`: 1024 random `256 x 256` hyperslabs of `d00` in total, from a seeded
+  splitmix64 stream that both harnesses generate identically.
+
+Reported per row: MB/s of decoded (selected) data from the median of the
+repetitions, and scaling efficiency `MB/s(T) / (T x MB/s(1))`. Each worker
+times itself from a start barrier; a repetition spans the earliest start to
+the latest finish. Page cache: warm by default (each file is read once before
+timing); `--cold` evicts the files with `posix_fadvise(POSIX_FADV_DONTNEED)`
+before every repetition (no root needed; best effort). clawhdf5 opens one
+`File` per repetition, shared by all threads; h5py threads share one
+`h5py.File`; h5py processes (spawned before timing) each open the file inside
+the timed region.
+
+Decode inside a single clawhdf5 read is itself parallel in this binary
+(clawhdf5-format's `parallel` feature, enabled here through clawhdf5-agent;
+it is off in the facade's default features), so a 1-thread clawhdf5 full read
+of the deflate file already uses the whole rayon pool. Run both
+`--decode-threads 1` (each read decodes on its calling thread, like h5py —
+this isolates the API's own scaling) and the default pool.
+
+> **Run** (from the repository root). The default files take about 5.4 GiB
+> of disk (4 GiB contiguous + about 1.3 GiB deflate). Generating them is
+> memory-hungry because `FileBuilder` holds a whole file in memory: peak RSS
+> was 676 MB for `--datasets 2 --mib 64` (2026-09-25, tank,
+> `/usr/bin/time -f %M`), about 5x one file's decoded size, so expect about
+> 21 GB at the defaults (once; later runs reuse the files). Put `--dir` on a
+> real disk, not tmpfs, if `--cold` is to mean anything.
+>
+> ```bash
+> DIR=/path/on/disk/concurrent-read
+> BENCH=crates/clawhdf5-bench/scripts
+> PY=.venv/bin/python   # h5py 3.16 / HDF5 2.0 in this repo
+> cargo build --release -p clawhdf5-bench --bin concurrent_read
+> B=target/release/concurrent_read
+> $B --dir $DIR --json claw-pool.json                    # generates on first run
+> $B --dir $DIR --decode-threads 1 --json claw-1.json
+> $PY $BENCH/concurrent_read_h5py.py --dir $DIR --executor threads   --json h5py-threads.json
+> $PY $BENCH/concurrent_read_h5py.py --dir $DIR --executor processes --json h5py-procs.json
+> $PY $BENCH/compare_concurrent_read.py claw-1.json    h5py-threads.json h5py-procs.json
+> $PY $BENCH/compare_concurrent_read.py claw-pool.json h5py-threads.json h5py-procs.json
+> ```
+>
+> Cold page cache: add `--cold` to every harness command. Smoke test (seconds):
+> `$B --dir /tmp/cr --datasets 4 --mib 1 --threads 1,2,4 --slabs 16 --reps 1`
+> and the same `--threads/--slabs/--reps` to the h5py script.
+
+Other flags (both harnesses): `--threads`, `--reps`, `--slab`, `--slabs`,
+`--seed`, `--modes distinct,same`, `--layouts deflate,contiguous`; sizes
+(`--datasets`, `--mib`) only on the Rust harness, which writes the files.
+
 ## Search harness baseline (v2.3.0)
 
 Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`

CLAUDE.md

@@ -5,13 +5,13 @@ Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persist
 
 ## Architecture
 
-Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
+Cargo workspace with 19 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
 
 | Crate | Role |
 |-------|------|
 | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
 | `clawhdf5-io` | Read/write implementation |
-| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec) live in `clawhdf5-format`. No Blosc. |
+| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. |
 | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
 | `clawhdf5` | Main facade crate |
 | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
@@ -21,9 +21,12 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
 | `clawhdf5-accel` | CPU SIMD acceleration path |
 | `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
 | `clawhdf5-android` | Android JNI bindings |
-| `clawhdf5-cli` | Command-line interface |
+| `clawhdf5-cli` | Command-line interface (agent memory) |
+| `clawhdf5-tools` | `h5rs`: pure-Rust HDF5 tools — `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` (structural + checksum validator) |
 | `clawhdf5-napi` | Node.js native addon bindings |
 | `clawhdf5-py` | PyO3 Python bindings |
+| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
+| `clawhdf5-remote` | Remote files: `open_url` over HTTP(S) range requests and object stores (`object_store`: S3, GCS, Azure) through a mandatory block cache (`BlockCache`) |
 | `clawhdf5-bench` | Benchmark suite |
 
 ## Key Features
@@ -148,7 +151,44 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
   Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
   `MemorySource` for this bookkeeping is inferred from the caller-supplied
   `source_channel` string (a heuristic, not an authenticated trust boundary).
+- In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`)
+  overwrites values, grows and shrinks chunked datasets (every chunk index,
+  version-2 B-trees included) and sets attributes (compact and dense
+  storage) in existing files (h5py- or clawhdf5-written) without rewriting
+  them, changing indexes and heaps as libhdf5 does (index shapes and heap
+  bookkeeping are compared with libhdf5's in the tests); space an edit
+  frees is reused by later edits of the same editor. Anything it cannot do
+  safely is `Error::Unsupported` before any write (limits in
+  `docs/known-issues.md`). Test changes with
+  `cargo test -p clawhdf5-tools --test edit_interop --test
+  edit_coverage_interop` (h5py, h5dump, `h5rs check`, structure comparisons
+  with libhdf5; libhdf5 sources for the algorithms are at
+  github.com/HDFGroup/hdf5, tag `hdf5_1_14_6`).
+- Remote files (`clawhdf5-remote`, range-read milestone M3 of
+  `docs/design/range-reads.md`): `open_url("http://…")` gives a
+  `clawhdf5::File` over `File::open_storage`, read through `BlockCache`
+  (1 MiB blocks, LRU byte budget, per-block in-flight dedup across threads,
+  runs coalesced into parallel requests). `HttpStorage` pins the file by
+  ETag/Last-Modified and length (a change is `RemoteError::FileChanged`),
+  refuses servers that ignore `Range` unless a full download is allowed,
+  and retries transient failures. `ObjectStoreStorage` (feature
+  `object-store`, pure Rust) runs each read on a small owned tokio
+  runtime and waits on a channel, so it works from any thread, including
+  inside `spawn_blocking` or another runtime. Default build is plain HTTP with
+  no C; `https` (rustls + ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are
+  opt-in. Tests run a std-only HTTP server
+  (`tests/common/server.rs`, also the `range_server` example);
+  `CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus
+  file over HTTP with `File::open`.
 - GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
+- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
+  no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.
+  `examples/wasm-viewer/test/run.sh` builds the package (needs the
+  `wasm-bindgen` CLI at the crate's exact version) and tests it under Node
+  and headless Chromium (a Playwright download in `~/.cache/ms-playwright`
+  on tank); the CI container has neither, so CI runs the native
+  `clawhdf5-wasm` `h5py_interop` test on the same fixture. Size numbers are
+  in the example's README.
 - Python and Node.js bindings for cross-language use
 - NetCDF-4 compatibility for scientific data interop
 
@@ -188,6 +228,16 @@ cargo run -p clawhdf5-cli -- --help
 # create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands
 ```
 
+### HDF5 tools (`h5rs`, crate `clawhdf5-tools`)
+```bash
+cargo run -p clawhdf5-tools -- ls -r file.h5    # also dump [--json], stat, diff, check
+bash scripts/h5rs-fuzz.sh                       # every subcommand over the CVE corpus: no panic/crash/hang
+bash scripts/h5rs-check-ok-files.sh --data      # check passes every fully-read conformance file
+```
+Its interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian
+`hdf5-tools`, installed in CI); `dump` must stay byte-identical to h5dump on
+the test files.
+
 ### Python bindings
 ```bash
 cd crates/clawhdf5-py

CONFORMANCE.md

@@ -13,15 +13,15 @@ fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
 
 | | |
 |---|---|
-| date | 2026-09-26 03:05 UTC |
+| date | 2026-09-27 00:34 UTC |
-| clawhdf5 commit | `42b81d9f1c3d9bef6050ad8a1326ac8c97f641d3` |
+| clawhdf5 commit | `f37e7ae3263277319dba4bc39be5397194eb00c3` |
 | machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
-| command | `conformance/run.sh --update-baseline` |
+| command | `conformance/run.sh --no-fetch --update-baseline` |
 | rustc | rustc 1.98.1 (48a229cea 2026-09-01) |
 | reference | h5py 3.16.0, HDF5 2.0.0, numpy 2.5.3, hdf5plugin 7.1.0, Python 3.14.4 |
 | h5dump | Version 1.14.6 (CVE corpus only) |
 | limits | 20 s timeout (SIGKILL), 4096 MiB address space, per process; 16 files in parallel |
-| runtime | 25 s probing + comparing (0 s fetch/build before it) |
+| runtime | 21 s probing + comparing (0 s fetch/build before it) |
 
 ## Results
 
@@ -35,17 +35,19 @@ A file's class is the first that applies:
 
 | corpus | files | ok | our-error | mismatch | h5py-cannot-read | panic | hang | crash | oom |
 |---|---|---|---|---|---|---|---|---|---|
-| NCAS-CMS_pyfive | 33 | 31 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
+| NCAS-CMS_pyfive | 33 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
-| cve_hdf5 | 147 | 87 | 17 | 11 | 32 | 0 | 0 | 0 | 0 |
+| cve_hdf5 | 147 | 113 | 2 | 0 | 32 | 0 | 0 | 0 | 0 |
 | h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| hdf5 | 466 | 300 | 103 | 3 | 60 | 0 | 0 | 0 | 0 |
+| hdf5 | 466 | 404 | 1 | 1 | 60 | 0 | 0 | 0 | 0 |
 | netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| netcdf4-python | 18 | 16 | 2 | 0 | 0 | 0 | 0 | 0 | 0 |
+| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
 | usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
 | xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| **all** | **697** | **467** | **123** | **15** | **92** | **0** | **0** | **0** | **0** |
+| **all** | **697** | **600** | **3** | **2** | **92** | **0** | **0** | **0** | **0** |
 
-2 of the 15 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
+2 of the 2 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
+
+3 of the 3 our-errors are corrupt data that HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).
 
 Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):
 
@@ -70,29 +72,13 @@ Grouped by normalised error message. *files* counts files whose class this cause
 
 | files | objects | error | examples |
 |---:|---:|---|---|
-| 84 | 205 | `InvalidLayoutVersion(N)` | `cve_hdf5/cvefiles/cve-2016-4330.h5`, `cve_hdf5/cvefiles/cve-2016-4333.h5`, `cve_hdf5/cvefiles/cve-2018-11206-old.h5` (+81 more) |
+| 3 | 3 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `cve_hdf5/cvefiles/cve-2025-44904.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
-| 10 | 10 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5`, `hdf5/tools/test/testfiles/vds/1_vds.h5` (+7 more) |
-| 9 | 17 | `InvalidObjectHeaderVersion(N)` | `cve_hdf5/cvefiles/cve-2021-36977.h5`, `cve_hdf5/cvefiles/unknown-1.h5`, `hdf5/tools/test/testfiles/h5clear_fsm_persist_user_equal.h5` (+6 more) |
-| 6 | 6 | `UnsupportedFilter(N)` | `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc2.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bshuf.h5` (+3 more) |
-| 6 | 6 | `InvalidLinkType(N)` | `hdf5/tools/test/testfiles/bigendian/tall.h5`, `hdf5/tools/test/testfiles/h5diff_types.h5`, `hdf5/tools/test/testfiles/tall.h5` (+3 more) |
-| 5 | 5 | `UnexpectedEof { expected: N, available: N }` | `NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc`, `cve_hdf5/cvefiles/cve-2019-9151.h5`, `hdf5/tools/test/testfiles/h5stat_newgrat.h5` (+2 more) |
-| 3 | 3 | `DataSizeMismatch { expected: N, actual: N }` | `cve_hdf5/cvefiles/cve-2020-18494.h5`, `cve_hdf5/cvefiles/cve-2024-32623.h5`, `cve_hdf5/cvefiles/cve-2025-2309.h5` |
-| 1 | 1 | `MissingMessage(Dataspace)` | `cve_hdf5/cvefiles/cve-2024-33874.h5` |
-| 1 | 2 | `InvalidSharedMessageVersion(N)` | `hdf5/tools/test/testfiles/h5stat_tsohm.h5` |
 
 ## Mismatch root causes
 
 | files | objects | cause | examples |
 |---:|---:|---|---|
-| 9 | 27 | `missing-object` | `cve_hdf5/cvefiles/cve-2019-8397.h5`, `cve_hdf5/cvefiles/cve-2019-8398.h5`, `cve_hdf5/cvefiles/cve-2021-46243.h5` (+6 more) |
-| 5 | 11 | `extra-object` | `cve_hdf5/cvefiles/cve-2021-46244.h5`, `cve_hdf5/cvefiles/cve-2024-32613.h5`, `cve_hdf5/cvefiles/cve-2024-32616.h5` (+2 more) |
-| 3 | 7 | `extra-attr` | `cve_hdf5/cvefiles/cve-2018-17438`, `cve_hdf5/cvefiles/cve-2018-17439`, `cve_hdf5/cvefiles/cve-2024-33874.h5` |
-| 2 | 4 | `missing-attr` | `hdf5/tools/test/testfiles/twithub.h5`, `hdf5/tools/test/testfiles/twithub513.h5` |
 | 1 | 1 | `attr-values: ours=vlen(>u8) h5py=object layout=- filters=-` | `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` |
-| 1 | 1 | `values: ours=<f4 h5py=float32 layout=chunked filters=-` | `cve_hdf5/cvefiles/cve-2025-44904.h5` |
-| 1 | 1 | `values: ours=>i2 h5py=>i2 layout=chunked filters=[6]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
-| 1 | 1 | `values: ours=>f4 h5py=>f4 layout=chunked filters=[2]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
-| 1 | 1 | `values: ours=<f4 h5py=float32 layout=chunked filters=[2]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
 | 1 | 1 | `values: ours=vlen({r:>f4,i:>f4}8) h5py=object layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tcomplex_be.h5` |
 
 ## CVE corpus: clawhdf5 vs h5dump vs h5py
@@ -105,7 +91,7 @@ columns are.
 
 | tool | read | error | panic | crash | hang | oom |
 |---|---:|---:|---:|---:|---:|---:|
-| clawhdf5 | 142 | 5 | 0 | 0 | 0 | 0 |
+| clawhdf5 | 121 | 26 | 0 | 0 | 0 | 0 |
 | h5dump 1.14.6 | 16 | 129 | 0 | 2 | 0 | 0 |
 | h5py 3.16.0 / HDF5 2.0.0 | 115 | 31 | 0 | 1 | 0 | 0 |
 
@@ -113,39 +99,39 @@ columns are.
 
 | file | h5dump | h5py | clawhdf5 | class |
 |---|---|---|---|---|
-| cvefiles/cve-2016-4330.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2016-4330.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2016-4331.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
-| cvefiles/cve-2016-4332-mtime-new.h5 | error exit | read 25 obj, 1 errors | read 25 obj | ok |
+| cvefiles/cve-2016-4332-mtime-new.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
-| cvefiles/cve-2016-4332-mtime.h5 | error exit | read 4 obj, 3 errors | read 4 obj, 1 errors | ok |
+| cvefiles/cve-2016-4332-mtime.h5 | error exit | read 4 obj, 3 errors | read 4 obj, 3 errors | ok |
-| cvefiles/cve-2016-4332-stab.h5 | error exit | open error | read 65 obj | h5py-cannot-read |
+| cvefiles/cve-2016-4332-stab.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
-| cvefiles/cve-2016-4333.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2016-4333.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2017-17505.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2017-17506.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2017-17507.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/cve-2017-17508.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
+| cvefiles/cve-2017-17508.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2017-17509.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2018-11202.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2018-11203.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/cve-2018-11204.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
+| cvefiles/cve-2018-11204.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/cve-2018-11205.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
+| cvefiles/cve-2018-11205.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2018-11206-new.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
-| cvefiles/cve-2018-11206-old.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2018-11206-old.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2018-11207.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2018-13866.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2018-13867.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2018-13868.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2018-13868.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2018-13869.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2018-13870.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2018-13871.h5 | error exit | read 2 obj | read 2 obj | ok |
 | cvefiles/cve-2018-13872.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2018-13873.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
+| cvefiles/cve-2018-13873.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2018-13874.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2018-13874.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2018-13875.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2018-13875.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
-| cvefiles/cve-2018-13876.h5 | error exit | open error | read 2 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2018-13876.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2018-14031.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2018-14031.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2018-14033.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2018-14034.h5 | error exit | read 1 obj, 2 errors | read 1 obj | ok |
-| cvefiles/cve-2018-14035.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2018-14035.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2018-14460.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
 | cvefiles/cve-2018-15671.h5 | ok | read 1 obj | read 1 obj | ok |
 | cvefiles/cve-2018-15672.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
@@ -159,102 +145,102 @@ columns are.
 | cvefiles/cve-2018-17435.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2018-17436 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2018-17437.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
-| cvefiles/cve-2018-17438 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | mismatch |
+| cvefiles/cve-2018-17438 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
-| cvefiles/cve-2018-17439 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | mismatch |
+| cvefiles/cve-2018-17439 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2019-8396.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
-| cvefiles/cve-2019-8397.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
+| cvefiles/cve-2019-8397.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/cve-2019-8398.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
+| cvefiles/cve-2019-8398.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/cve-2019-9151.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2019-9151.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | ok |
 | cvefiles/cve-2019-9152.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2020-10809 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2020-10810.h5 | error exit | open error | read 2 obj | h5py-cannot-read |
+| cvefiles/cve-2020-10810.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2020-10811.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
-| cvefiles/cve-2020-10812.h5 | error exit | open error | read 2 obj | h5py-cannot-read |
+| cvefiles/cve-2020-10812.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2020-18232.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
-| cvefiles/cve-2020-18494.h5 | ok | read 2 obj | read 2 obj, 1 errors | our-error |
+| cvefiles/cve-2020-18494.h5 | ok | read 2 obj | read 2 obj | ok |
-| cvefiles/cve-2021-36977.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | our-error |
+| cvefiles/cve-2021-36977.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2021-37501.h5 | error exit | read 18 obj, 1 errors | read 18 obj, 1 errors | ok |
 | cvefiles/cve-2021-45829.h5 | error exit | read 1 obj, 2 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2021-45830.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2021-45830.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2021-45833.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2021-46242.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
-| cvefiles/cve-2021-46243.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
+| cvefiles/cve-2021-46243.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/cve-2021-46244.h5 | error exit | read 2 obj, 1 errors | read 6 obj, 3 errors | mismatch |
+| cvefiles/cve-2021-46244.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2024-29157.h5 | error exit | read 4 obj, 7 errors | read 4 obj, 7 errors | ok |
 | cvefiles/cve-2024-29158.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2024-29159.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2024-29160.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
-| cvefiles/cve-2024-29161.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 2 errors | ok |
+| cvefiles/cve-2024-29161.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2024-29162.h5 | error exit | read 17 obj, 4 errors | read 17 obj, 3 errors | ok |
+| cvefiles/cve-2024-29162.h5 | error exit | read 17 obj, 4 errors | read 17 obj, 4 errors | ok |
 | cvefiles/cve-2024-29163.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
-| cvefiles/cve-2024-29164.h5 | ok | read 3 obj | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2024-29164.h5 | ok | read 3 obj | read 3 obj | ok |
 | cvefiles/cve-2024-29165.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2024-29166.h5 | error exit | read 17 obj, 2 errors | read 17 obj | ok |
 | cvefiles/cve-2024-32605.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
 | cvefiles/cve-2024-32606.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2024-32607-1.h5 | ok | read 10 obj | read 10 obj | ok |
 | cvefiles/cve-2024-32607-2.h5 | error exit | read 9 obj, 1 errors | read 9 obj, 1 errors | ok |
-| cvefiles/cve-2024-32608.h5 | error exit | read 6 obj, 1 errors | read 6 obj | ok |
+| cvefiles/cve-2024-32608.h5 | error exit | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
-| cvefiles/cve-2024-32609.h5 | error exit | SIGSEGV | read 4 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2024-32609.h5 | error exit | SIGSEGV | read 3 obj, 1 errors | h5py-cannot-read |
 | cvefiles/cve-2024-32610.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2024-32611.h5 | ok | read 6 obj | read 6 obj | ok |
-| cvefiles/cve-2024-32612.h5 | ok | read 3 obj | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2024-32612.h5 | ok | read 3 obj | read 3 obj | ok |
-| cvefiles/cve-2024-32613.h5 | error exit | read 7 obj, 1 errors | read 11 obj | mismatch |
+| cvefiles/cve-2024-32613.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
-| cvefiles/cve-2024-32614.h5 | error exit | read 25 obj, 2 errors | read 25 obj, 1 errors | ok |
+| cvefiles/cve-2024-32614.h5 | error exit | read 25 obj, 2 errors | read 25 obj, 2 errors | ok |
 | cvefiles/cve-2024-32615.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
-| cvefiles/cve-2024-32616.h5 | error exit | read 10 obj, 7 errors | read 11 obj, 6 errors | mismatch |
+| cvefiles/cve-2024-32616.h5 | error exit | read 10 obj, 7 errors | read 10 obj, 6 errors | ok |
 | cvefiles/cve-2024-32617.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2024-32618.h5 | error exit | read 4 obj, 2 errors | read 3 obj | mismatch |
+| cvefiles/cve-2024-32618.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
-| cvefiles/cve-2024-32619.h5 | error exit | read 3 obj, 2 errors | read 3 obj | ok |
+| cvefiles/cve-2024-32619.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
 | cvefiles/cve-2024-32620.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2024-32621.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2024-32622.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
-| cvefiles/cve-2024-32623.h5 | ok | read 6 obj | read 6 obj, 1 errors | our-error |
+| cvefiles/cve-2024-32623.h5 | ok | read 6 obj | read 6 obj | ok |
-| cvefiles/cve-2024-32624.h5 | error exit | read 6 obj, 1 errors | read 6 obj | ok |
+| cvefiles/cve-2024-32624.h5 | error exit | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
-| cvefiles/cve-2024-33873.h5 | error exit | read 4 obj, 1 errors | read 4 obj | ok |
+| cvefiles/cve-2024-33873.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
-| cvefiles/cve-2024-33874.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | our-error |
+| cvefiles/cve-2024-33874.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
 | cvefiles/cve-2024-33875.h5 | ok | read 2 obj | read 2 obj | ok |
 | cvefiles/cve-2024-33876.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2024-33877.h5 | error exit | read 8 obj, 1 errors | read 8 obj, 1 errors | ok |
 | cvefiles/cve-2025-2153.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
 | cvefiles/cve-2025-2308.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
-| cvefiles/cve-2025-2309.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | our-error |
+| cvefiles/cve-2025-2309.h5 | ok | read 6 obj, 1 errors | read 6 obj | ok |
 | cvefiles/cve-2025-2310.h5 | error exit | read 24 obj, 8 errors | read 24 obj, 8 errors | ok |
-| cvefiles/cve-2025-2912.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2912.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-2913.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2913.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-2914.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2914.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-2915.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2915.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-2923.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2923.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2025-2924.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-2925.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2025-2926.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2926.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | mismatch |
+| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
-| cvefiles/cve-2025-44905.h5 | error exit | read 25 obj, 3 errors | read 25 obj, 3 errors | mismatch |
+| cvefiles/cve-2025-44905.h5 | error exit | read 25 obj, 3 errors | read 25 obj, 3 errors | ok |
 | cvefiles/cve-2025-6269-1.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-6269-2.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-6269-3.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-6269-4.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2025-6270-1.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6270-1.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6270-2.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6270-2.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6270-3.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6270-3.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2025-6516.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-6750.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
-| cvefiles/cve-2025-6816.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6816.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6817.h5 | error exit | open error | read 1 obj | h5py-cannot-read |
+| cvefiles/cve-2025-6817.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6818.h5 | error exit | open error | read 1 obj | h5py-cannot-read |
+| cvefiles/cve-2025-6818.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6856.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6856.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2025-6857.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2025-6858.h5 | SIGSEGV | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6858.h5 | SIGSEGV | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2025-7067.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2025-7068.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-7068.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-7069.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-7069.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2026-26200.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2026-34734.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
+| cvefiles/cve-2026-34734.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2026-92627.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/unknown-1.h5 | error exit | read 11 obj, 1 errors | read 11 obj, 5 errors | our-error |
+| cvefiles/unknown-1.h5 | error exit | read 11 obj, 1 errors | read 11 obj, 1 errors | ok |
 | fuzzerfiles/gh-4431-poc-03.h5 | error exit | read 1 obj | read 1 obj | ok |
 | fuzzerfiles/gh-4432-poc-05.h5 | SIGSEGV | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| fuzzerfiles/gh-4433-poc-08.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
+| fuzzerfiles/gh-4433-poc-08.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | fuzzerfiles/gh-4434-poc-09.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
 | fuzzerfiles/gh-4435-poc-10.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | fuzzerfiles/gh-4585.h5 | error exit | open error | open error | h5py-cannot-read |
@@ -278,15 +264,18 @@ columns are.
 - **Types h5py widens.** Where h5py reads a type into a numpy type of a different size
   (FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not
   compared (shape and presence still are): dataset file type size 1 -> numpy float16 (2) (15x), attr file type size 1 -> numpy float16 (2) (15x), dataset file type size 2 -> numpy float32 (4) (2x), dataset file type size 8 -> numpy float128 (16) (1x), dataset file type size 12 -> numpy float128 (16) (1x), attr file type size 2 -> numpy float32 (4) (1x), dataset file type size 2 -> numpy >f4 (4) (1x), attr file type size 2 -> numpy >f4 (4) (1x).
+- **Corrupt data HDF5 2.0 reads through a bug.** clawhdf5 refuses these objects; h5py 3.16 /
+  HDF5 2.0 returns values for them that the file does not hold:
+  - `cve_hdf5/cvefiles/cve-2025-2308.h5` `/Scale_offset_long_long_data_le`: scale-offset codes run past the end of the chunk: HDF5 2.0 reads past its buffer; libhdf5's develop branch refuses the chunk ("Buffer too short").
+  - `cve_hdf5/cvefiles/cve-2025-44904.h5` `/Scale_offset_float_data_le`: unfiltered chunks of 38 and 37 bytes for 48-byte chunks: HDF5 2.0 fills the rest with whatever its buffer held; libhdf5's develop branch refuses them ("incorrect chunk size returned from index for unfiltered chunk").
+  - `hdf5/test/testfiles/bad_nbit_parms_walk.h5` `/Nbit_int_data_le`: an N-Bit parameter list one value short: HDF5 2.0 reads past the list; libhdf5's own test (`test_filter_bad_params`, test/dsets.c) now requires the read to fail.
 - **References** are compared by presence only (`R`), not by target.
 
 ## Objects h5py fails on but clawhdf5 reads
 
 - 19 x `OSError: Can't synchronously read data (no appropriate function for conversion path)`
-- 17 x `KeyError: '…'`
 - 1 x `TypeError: unhandled dtype kind M (dtype('…'))`
-- 1 x `OSError: Can't synchronously read data (bad coordinate offset)`
+- 1 x `TypeError: No NumPy equivalent for TypeTimeID exists`
-- 1 x `KeyError: "…"`
 - 1 x `ValueError: Insufficient precision in available types to represent (N, N, N, N, N)`
 
 ## Reproduce

Cargo.toml

@@ -16,6 +16,9 @@ members = [
     "crates/clawhdf5-cli",
     "crates/clawhdf5-napi",
     "crates/clawhdf5-bench",
+    "crates/clawhdf5-tools",
+    "crates/clawhdf5-wasm",
+    "crates/clawhdf5-remote",
     "crates/libaec-sys",
 ]
 resolver = "2"
@@ -34,3 +37,12 @@ tempfile = "3"
 criterion = { version = "0.5", features = ["html_reports"] }
 half = "2.7"
 serde = { version = "1", features = ["derive"] }
+
+# The browser build of clawhdf5-wasm (examples/wasm-viewer/build.sh): size
+# over speed, whole-program optimisation. Native profiles are unaffected.
+[profile.wasm-release]
+inherits = "release"
+opt-level = "s"
+lto = true
+codegen-units = 1
+panic = "abort"

README.md

@@ -73,8 +73,9 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
 - Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
   been discarding the retrieval score, costing the Markdown backend 40.6pp of
   Hit@1; fixed in v2.6.0.
-- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
+- Selection reads whose bounding box covers at most half the dataset decode
-  0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
+  only the chunks they touch (a 64×64 window: 105 ms to 0.39 ms), and full
+  reads are 1.2–1.9× faster (v2.5.0).
 
 **Memory**
 - A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
@@ -95,6 +96,12 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
   filtered top-k, never slower than unfiltered), and opt-in re-ranking and
   confidence rejection, which used to be reachable only through `ClawhdfBackend`.
 
+**Remote files (unreleased)**
+- New crate `clawhdf5-remote`: `open_url("http://…")` reads a file on an
+  HTTP server (or in S3/GCS/Azure, opt-in) by range requests through a
+  block cache, without downloading it; `h5rs` takes URLs with its `remote`
+  feature. See [Reading remote files](#reading-remote-files).
+
 **Tooling**
 - CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
   silently) and runs an aarch64 job for the NEON kernels.
@@ -407,6 +414,144 @@ let values = ds.read_f64()?;
 assert_eq!(values, vec![22.5, 23.1, 21.8]);
 ```
 
+### Groups and links
+
+```rust
+use clawhdf5::{AttrValue, FileBuilder};
+
+let mut b = FileBuilder::new();
+// A path creates its missing intermediate groups, as in h5py.
+b.create_dataset("run/2026/temps").with_f64_data(&[22.5, 23.1]);
+// Builders nest; a group added at an existing path is merged into it.
+let mut run = b.create_group("run");
+run.set_attr("operator", AttrValue::String("ana".into()));
+let mut cal = run.create_group("calibration");
+cal.track_order(true); // h5py lists members in insertion order
+cal.create_dataset("offset").with_f64_data(&[0.1]);
+run.add_group(cal.finish());
+b.add_group(run.finish());
+b.add_soft_link("latest", "/run/2026");    // h5py.SoftLink
+b.add_hard_link("temps", "/run/2026/temps"); // f["temps"] = f["run/2026/temps"]
+b.add_external_link("raw", "raw.h5", "/data");
+b.write("groups.h5")?;
+```
+
+A group holds at most 65 535 links; more is an error, as is a link over
+65 515 bytes (a very long soft-link target) in a group of more than 8 links.
+
+### Modifying an existing file
+
+```rust
+use clawhdf5::{AttrValue, FileEditor, Selection};
+
+// A file from h5py or clawhdf5, dataset "x" chunked with maxshape=(None,).
+let mut ed = FileEditor::open("data.h5")?; // exclusive lock, like libhdf5
+ed.resize("x", &[1100])?;                   // h5py: ds.resize((1100,))
+let sel = Selection::Hyperslab { start: vec![1000], stride: vec![1], count: vec![100], block: vec![1] };
+ed.write_values("x", &sel, &[0.5f64; 100])?; // ds[1000:1100] = 0.5
+ed.set_attr("x", "units", &AttrValue::String("m/s".into()))?;
+ed.resize("x", &[900])?;                    // shrinking prunes chunks, like h5py
+```
+
+Each call changes the file in place (no rewrite) and syncs it. Any chunk
+index (version-2 B-trees for several unlimited dimensions included) and
+attributes in compact or dense storage are handled as libhdf5 handles
+them; space an edit frees is reused by later edits of the same editor. What it
+cannot change safely is refused before anything is written; see
+[known issues](docs/known-issues.md) for the limits.
+
+### Reading remote files
+
+[`clawhdf5-remote`](crates/clawhdf5-remote/README.md) opens a file on an
+HTTP server (or, with its `s3`/`gcs`/`azure` features, in an object store)
+without downloading it: the read API is the same `clawhdf5::File`, and
+only the bytes an operation needs are fetched, by `Range` requests through
+a block cache (1 MiB blocks; opening fetches the first one). A file that
+changes on the server while it is open is an error, never a mix of old and
+new bytes.
+
+```rust
+let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/tall.h5")?;
+let values = file.dataset("/g2/dset2.1")?.read_f64()?;
+```
+
+To try it without a server of your own, the crate's test server serves a
+directory with range support:
+
+```bash
+cargo run -p clawhdf5-remote --example range_server -- crates/clawhdf5/tests/fixtures 127.0.0.1:8000
+# in another shell: list the file, read one dataset, print what it cost
+cargo run -p clawhdf5-remote --example read_url -- http://127.0.0.1:8000/tall.h5 /g2/dset2.1
+```
+
+```text
+/g1  group
+/g2  group
+/g2/dset2.1  dataset [10] F32
+/g2/dset2.2  dataset [3, 5] F32
+/g1/g1.1  group
+/g1/g1.2  group
+/g1/g1.2/g1.2.1  group
+/g1/g1.1/dset1.1.1  dataset [10, 10] I32
+/g1/g1.1/dset1.1.2  dataset [20] I32
+/g2/dset2.1: 10 values, first [1.0, 1.100000023841858, 1.2000000476837158, ...]
+1 range requests (the one at open included), 9968 bytes fetched, 9968 bytes cached
+```
+
+(`tall.h5` is 9 968 bytes, so the first block holds all of it.) `h5rs`
+built with `--features remote` takes the same URLs:
+`h5rs ls -r http://127.0.0.1:8000/tall.h5`. Plain HTTP builds no C;
+`https://` is the `https` feature (rustls with ring, which compiles C).
+Limits are in [known issues](docs/known-issues.md).
+
+### Python
+
+`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
+an h5py-shaped API and no libhdf5. It is not on PyPI; build it with
+[maturin](https://www.maturin.rs) into a virtualenv:
+
+```bash
+python -m venv .venv && . .venv/bin/activate
+pip install maturin numpy
+maturin develop --release -m crates/clawhdf5-py/Cargo.toml
+python -c "import clawhdf5; print(clawhdf5.__version__)"
+```
+
+```python
+import numpy as np
+import clawhdf5
+
+with clawhdf5.File("data.h5", "r") as f:
+    print(list(f.keys()))              # sorted member names, like h5py
+    ds = f["group/temperatures"]       # relative or absolute ("/group/...") paths
+    print(ds.shape, ds.dtype)          # dtype is the numpy dtype h5py reports
+    block = ds[100:200, ::4]           # a small selection reads only its chunks
+    row = ds[-1]                       # integers drop the axis
+    picked = ds[[1, 5, 9], :]          # one increasing index list per key
+    units = ds.attrs["units"]          # attributes come back as h5py returns them
+    everything = np.asarray(ds)
+
+    records = f["table"]               # compound -> numpy structured array
+    ids = records["id"]                # one field
+```
+
+Reads cover integers and IEEE floats of every width in either byte order,
+`bool`, enums, complex, fixed and variable-length strings, variable-length
+sequences, opaque, HDF5 array types and compounds; other types (references,
+bitfields, ...) raise `TypeError` instead of returning guessed data. Keys
+follow h5py (negative steps, `None` and boolean masks are refused). The
+read itself runs with the GIL released, so Python threads read in parallel.
+A selection whose bounding box covers at most half the dataset decodes only
+the chunks (or contiguous rows) that box overlaps; a larger one — including
+a strided slice across the whole dataset — decodes the whole dataset, as
+do datasets that are compact, virtual, unwritten, or chunked with a
+non-default fill value (`docs/known-issues.md`). An index list is read one
+group of neighbouring chunks at a time.
+Writing (`File(path, "w")`, `create_dataset`, `create_group`, `attrs[...] =`)
+covers `float64`, `float32`, `int64`, `int32` and `uint8` arrays. The tests
+in `crates/clawhdf5-py/tests` compare every read with h5py; run them with
+`pip install pytest h5py && pytest crates/clawhdf5-py/tests`.
+
 ### Agent Memory
 
 ```rust
@@ -587,19 +732,20 @@ let exported = backend.export_markdown("MEMORY.md")?;
 ## Crate Map
 
 ```
-clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
+clawhdf5 workspace (19 crates, ~86K lines of Rust in src/, ~104K with tests
                      and benches; plus libaec-sys, an internal FFI bindings
                      crate for the optional szip feature)
 │
 ├── Core HDF5
 │   ├── clawhdf5-format      — Binary parser/writer (no_std-capable), shared type definitions
 │   ├── clawhdf5-io          — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
-│   ├── clawhdf5-filters     — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format
+│   ├── clawhdf5-filters     — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc/Blosc2 filters live in clawhdf5-format
 │   ├── clawhdf5-derive      — Proc macros
 │   ├── clawhdf5             — High-level API
 │   ├── clawhdf5-netcdf4     — NetCDF-4 support
 │   ├── clawhdf5-accel       — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
-│   └── clawhdf5-gpu         — GPU compute (wgpu, hand-written WGSL compute shaders)
+│   ├── clawhdf5-gpu         — GPU compute (wgpu, hand-written WGSL compute shaders)
+│   └── clawhdf5-remote      — Remote files: HTTP(S) range requests, object stores, block cache
 │
 ├── Agent Memory
 │   ├── clawhdf5-agent       — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
@@ -610,9 +756,11 @@ clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
 │
 ├── Bindings
 │   ├── clawhdf5-py          — Python (PyO3)
-│   └── clawhdf5-napi        — Node.js (napi-rs)
+│   ├── clawhdf5-napi        — Node.js (napi-rs)
+│   └── clawhdf5-wasm        — Browser (WebAssembly, wasm-bindgen; read-only)
 │
 └── Tooling
+    ├── clawhdf5-tools       — h5rs: ls, dump, stat, diff, check
     └── clawhdf5-bench       — Benchmark suite
 ```
 
@@ -700,6 +848,24 @@ stores keep their setting. Opt out with `float16 = false` or
 | `system-zlib` | no | System zlib backend for deflate (C) |
 | `blake3_hash` | no | BLAKE3 content hashing for provenance |
 | `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) |
+| `lzf` | **yes** | LZF filter (id 32000), h5py's built-in `compression="lzf"`: read and write. No dependencies |
+| `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) |
+| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
+| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
+| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
+| `zfp` | no | ZFP filter (id 32013, H5Z-ZFP), read only: every mode (rate, precision, accuracy, reversible, expert) for int32, int64, float and double, 1-4-D, returning exactly libzfp's values. Pure Rust, no dependencies |
+| `plugin-filters` | no | All six above |
+
+clawhdf5 cannot write Blosc2 or ZFP. Any other
+filter can be supplied at run time with `filter_registry::register_filter` (a
+decoder closure, or a `FilterCodec` that also encodes). The facade
+(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2`, `zfp`
+and `plugin-filters`. Write
+with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
+and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
+`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
+encoder has one level (about zstd's level 1); no speed or ratio claims are
+made for these codecs.
 
 ### `clawhdf5-ann`
 

conformance/baseline.json

@@ -1,43 +1,40 @@
 {
  "comment": "conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
- "commit": "42b81d9f1c3d9bef6050ad8a1326ac8c97f641d3",
+ "commit": "f37e7ae3263277319dba4bc39be5397194eb00c3",
- "date": "2026-09-26 03:05 UTC",
+ "date": "2026-09-27 00:34 UTC",
  "reference": "h5py 3.16.0 / HDF5 2.0.0",
  "files": 697,
- "ok": 467,
+ "ok": 600,
  "counts": {
   "h5py-cannot-read": 92,
-  "mismatch": 15,
+  "mismatch": 2,
-  "ok": 467,
+  "ok": 600,
-  "our-error": 123
+  "our-error": 3
  },
  "per_corpus": {
   "NCAS-CMS_pyfive": {
    "mismatch": 1,
-   "ok": 31,
+   "ok": 32
-   "our-error": 1
   },
   "cve_hdf5": {
    "h5py-cannot-read": 32,
-   "mismatch": 11,
+   "ok": 113,
-   "ok": 87,
+   "our-error": 2
-   "our-error": 17
   },
   "h5py_data": {
    "ok": 4
   },
   "hdf5": {
    "h5py-cannot-read": 60,
-   "mismatch": 3,
+   "mismatch": 1,
-   "ok": 300,
+   "ok": 404,
-   "our-error": 103
+   "our-error": 1
   },
   "netcdf-c": {
    "ok": 20
   },
   "netcdf4-python": {
-   "ok": 16,
+   "ok": 18
-   "our-error": 2
   },
   "usnistgov_h5wasm": {
    "ok": 5
@@ -50,6 +47,7 @@
   "NCAS-CMS_pyfive/tests/compact.hdf5",
   "NCAS-CMS_pyfive/tests/data/btreev2.hdf5",
   "NCAS-CMS_pyfive/tests/data/chunked.hdf5",
+  "NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc",
   "NCAS-CMS_pyfive/tests/data/compressed.hdf5",
   "NCAS-CMS_pyfive/tests/data/compressed_v1.hdf5",
   "NCAS-CMS_pyfive/tests/data/dataset_datatypes.hdf5",
@@ -78,9 +76,11 @@
   "NCAS-CMS_pyfive/tests/data/resizable.hdf5",
   "NCAS-CMS_pyfive/tests/opaque_datetime.hdf5",
   "NCAS-CMS_pyfive/tests/opaque_fixed.hdf5",
+  "cve_hdf5/cvefiles/cve-2016-4330.h5",
   "cve_hdf5/cvefiles/cve-2016-4331.h5",
   "cve_hdf5/cvefiles/cve-2016-4332-mtime-new.h5",
   "cve_hdf5/cvefiles/cve-2016-4332-mtime.h5",
+  "cve_hdf5/cvefiles/cve-2016-4333.h5",
   "cve_hdf5/cvefiles/cve-2017-17505.h5",
   "cve_hdf5/cvefiles/cve-2017-17506.h5",
   "cve_hdf5/cvefiles/cve-2017-17507.h5",
@@ -91,15 +91,20 @@
   "cve_hdf5/cvefiles/cve-2018-11204.h5",
   "cve_hdf5/cvefiles/cve-2018-11205.h5",
   "cve_hdf5/cvefiles/cve-2018-11206-new.h5",
+  "cve_hdf5/cvefiles/cve-2018-11206-old.h5",
   "cve_hdf5/cvefiles/cve-2018-11207.h5",
   "cve_hdf5/cvefiles/cve-2018-13867.h5",
+  "cve_hdf5/cvefiles/cve-2018-13868.h5",
   "cve_hdf5/cvefiles/cve-2018-13869.h5",
   "cve_hdf5/cvefiles/cve-2018-13870.h5",
   "cve_hdf5/cvefiles/cve-2018-13871.h5",
   "cve_hdf5/cvefiles/cve-2018-13872.h5",
   "cve_hdf5/cvefiles/cve-2018-13873.h5",
+  "cve_hdf5/cvefiles/cve-2018-13875.h5",
+  "cve_hdf5/cvefiles/cve-2018-14031.h5",
   "cve_hdf5/cvefiles/cve-2018-14033.h5",
   "cve_hdf5/cvefiles/cve-2018-14034.h5",
+  "cve_hdf5/cvefiles/cve-2018-14035.h5",
   "cve_hdf5/cvefiles/cve-2018-14460.h5",
   "cve_hdf5/cvefiles/cve-2018-15671.h5",
   "cve_hdf5/cvefiles/cve-2018-15672.h5",
@@ -111,13 +116,22 @@
   "cve_hdf5/cvefiles/cve-2018-17434.h5",
   "cve_hdf5/cvefiles/cve-2018-17435.h5",
   "cve_hdf5/cvefiles/cve-2018-17437.h5",
+  "cve_hdf5/cvefiles/cve-2018-17438",
+  "cve_hdf5/cvefiles/cve-2018-17439",
   "cve_hdf5/cvefiles/cve-2019-8396.h5",
+  "cve_hdf5/cvefiles/cve-2019-8397.h5",
+  "cve_hdf5/cvefiles/cve-2019-8398.h5",
+  "cve_hdf5/cvefiles/cve-2019-9151.h5",
   "cve_hdf5/cvefiles/cve-2019-9152.h5",
   "cve_hdf5/cvefiles/cve-2020-10811.h5",
   "cve_hdf5/cvefiles/cve-2020-18232.h5",
+  "cve_hdf5/cvefiles/cve-2020-18494.h5",
+  "cve_hdf5/cvefiles/cve-2021-36977.h5",
   "cve_hdf5/cvefiles/cve-2021-37501.h5",
   "cve_hdf5/cvefiles/cve-2021-45829.h5",
   "cve_hdf5/cvefiles/cve-2021-45833.h5",
+  "cve_hdf5/cvefiles/cve-2021-46243.h5",
+  "cve_hdf5/cvefiles/cve-2021-46244.h5",
   "cve_hdf5/cvefiles/cve-2024-29157.h5",
   "cve_hdf5/cvefiles/cve-2024-29158.h5",
   "cve_hdf5/cvefiles/cve-2024-29159.h5",
@@ -125,6 +139,7 @@
   "cve_hdf5/cvefiles/cve-2024-29161.h5",
   "cve_hdf5/cvefiles/cve-2024-29162.h5",
   "cve_hdf5/cvefiles/cve-2024-29163.h5",
+  "cve_hdf5/cvefiles/cve-2024-29164.h5",
   "cve_hdf5/cvefiles/cve-2024-29165.h5",
   "cve_hdf5/cvefiles/cve-2024-29166.h5",
   "cve_hdf5/cvefiles/cve-2024-32605.h5",
@@ -134,21 +149,29 @@
   "cve_hdf5/cvefiles/cve-2024-32608.h5",
   "cve_hdf5/cvefiles/cve-2024-32610.h5",
   "cve_hdf5/cvefiles/cve-2024-32611.h5",
+  "cve_hdf5/cvefiles/cve-2024-32612.h5",
+  "cve_hdf5/cvefiles/cve-2024-32613.h5",
   "cve_hdf5/cvefiles/cve-2024-32614.h5",
   "cve_hdf5/cvefiles/cve-2024-32615.h5",
+  "cve_hdf5/cvefiles/cve-2024-32616.h5",
   "cve_hdf5/cvefiles/cve-2024-32617.h5",
+  "cve_hdf5/cvefiles/cve-2024-32618.h5",
   "cve_hdf5/cvefiles/cve-2024-32619.h5",
   "cve_hdf5/cvefiles/cve-2024-32620.h5",
   "cve_hdf5/cvefiles/cve-2024-32621.h5",
   "cve_hdf5/cvefiles/cve-2024-32622.h5",
+  "cve_hdf5/cvefiles/cve-2024-32623.h5",
   "cve_hdf5/cvefiles/cve-2024-32624.h5",
   "cve_hdf5/cvefiles/cve-2024-33873.h5",
+  "cve_hdf5/cvefiles/cve-2024-33874.h5",
   "cve_hdf5/cvefiles/cve-2024-33875.h5",
   "cve_hdf5/cvefiles/cve-2024-33876.h5",
   "cve_hdf5/cvefiles/cve-2024-33877.h5",
+  "cve_hdf5/cvefiles/cve-2025-2309.h5",
   "cve_hdf5/cvefiles/cve-2025-2310.h5",
   "cve_hdf5/cvefiles/cve-2025-2924.h5",
   "cve_hdf5/cvefiles/cve-2025-2925.h5",
+  "cve_hdf5/cvefiles/cve-2025-44905.h5",
   "cve_hdf5/cvefiles/cve-2025-6269-1.h5",
   "cve_hdf5/cvefiles/cve-2025-6269-2.h5",
   "cve_hdf5/cvefiles/cve-2025-6269-3.h5",
@@ -159,6 +182,7 @@
   "cve_hdf5/cvefiles/cve-2026-26200.h5",
   "cve_hdf5/cvefiles/cve-2026-34734.h5",
   "cve_hdf5/cvefiles/cve-2026-92627.h5",
+  "cve_hdf5/cvefiles/unknown-1.h5",
   "cve_hdf5/fuzzerfiles/gh-4431-poc-03.h5",
   "cve_hdf5/fuzzerfiles/gh-4432-poc-05.h5",
   "cve_hdf5/fuzzerfiles/gh-4433-poc-08.h5",
@@ -170,10 +194,30 @@
   "h5py_data/vlen_string_dset_utc.h5",
   "h5py_data/vlen_string_s390x.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bitgroom.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc2.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bshuf.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bzip2.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_granularbr.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_jpeg.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lz4.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lzf.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zfp.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zstd.h5",
+  "hdf5/HDF5Examples/C/H5G/16/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/C/H5G/16/h5ex_g_traverse.h5",
+  "hdf5/HDF5Examples/C/H5G/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/C/H5G/h5ex_g_traverse.h5",
+  "hdf5/HDF5Examples/C/H5G/h5ex_g_visit.h5",
+  "hdf5/HDF5Examples/FORTRAN/H5G/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/FORTRAN/H5G/h5ex_g_traverse.h5",
+  "hdf5/HDF5Examples/FORTRAN/H5G/h5ex_g_visit.h5",
+  "hdf5/HDF5Examples/JAVA/H5G/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/JAVA/H5G/h5ex_g_visit.h5",
+  "hdf5/HDF5Examples/JAVA/compat/H5G/110/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/JAVA/compat/H5G/110/h5ex_g_visit.h5",
+  "hdf5/HDF5Examples/JAVA/compat/H5G/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/JAVA/compat/H5G/h5ex_g_visit.h5",
   "hdf5/c++/test/th5s.h5",
   "hdf5/hl/test/testfiles/test_ds_be.h5",
   "hdf5/hl/test/testfiles/test_ds_be_new_ref-32bit.h5",
@@ -181,6 +225,9 @@
   "hdf5/hl/test/testfiles/test_ds_le.h5",
   "hdf5/hl/test/testfiles/test_ds_le_new_ref.h5",
   "hdf5/hl/test/testfiles/test_ld.h5",
+  "hdf5/hl/test/testfiles/test_table_be.h5",
+  "hdf5/hl/test/testfiles/test_table_cray.h5",
+  "hdf5/hl/test/testfiles/test_table_le.h5",
   "hdf5/test/testfiles/aggr.h5",
   "hdf5/test/testfiles/bad_chunk_ndims.h5",
   "hdf5/test/testfiles/bad_compound.h5",
@@ -188,12 +235,16 @@
   "hdf5/test/testfiles/be_data.h5",
   "hdf5/test/testfiles/be_extlink1.h5",
   "hdf5/test/testfiles/be_extlink2.h5",
+  "hdf5/test/testfiles/btree_idx_1_6.h5",
   "hdf5/test/testfiles/btree_idx_1_8.h5",
   "hdf5/test/testfiles/charsets.h5",
   "hdf5/test/testfiles/corrupt_stab_msg.h5",
+  "hdf5/test/testfiles/deflate.h5",
   "hdf5/test/testfiles/file_image_core_test.h5",
+  "hdf5/test/testfiles/filespace_1_6.h5",
   "hdf5/test/testfiles/filespace_1_8.h5",
   "hdf5/test/testfiles/fill18.h5",
+  "hdf5/test/testfiles/fill_old.h5",
   "hdf5/test/testfiles/filter_error.h5",
   "hdf5/test/testfiles/fsm_aggr_nopersist.h5",
   "hdf5/test/testfiles/fsm_aggr_persist.h5",
@@ -214,17 +265,28 @@
   "hdf5/test/testfiles/paged_nopersist.h5",
   "hdf5/test/testfiles/paged_persist.h5",
   "hdf5/test/testfiles/specmetaread.h5",
+  "hdf5/test/testfiles/tarrold.h5",
   "hdf5/test/testfiles/tbad_msg_count.h5",
   "hdf5/test/testfiles/tbogus.h5",
+  "hdf5/test/testfiles/test_filters_be.h5",
+  "hdf5/test/testfiles/test_filters_le.h5",
   "hdf5/test/testfiles/th5s.h5",
   "hdf5/test/testfiles/tlayouto.h5",
   "hdf5/test/testfiles/tmisc38a.h5",
   "hdf5/test/testfiles/tmisc38b.h5",
+  "hdf5/test/testfiles/tmtimen.h5",
+  "hdf5/test/testfiles/tmtimeo.h5",
   "hdf5/test/testfiles/tnullspace.h5",
   "hdf5/test/testfiles/tsizeslheap.h5",
+  "hdf5/tools/test/testfiles/bigendian/tall.h5",
   "hdf5/tools/test/testfiles/bigendian/tdset2.h5",
+  "hdf5/tools/test/testfiles/binfp64.h5",
+  "hdf5/tools/test/testfiles/binin16.h5",
   "hdf5/tools/test/testfiles/binin32.h5",
+  "hdf5/tools/test/testfiles/binin8.h5",
   "hdf5/tools/test/testfiles/binin8w.h5",
+  "hdf5/tools/test/testfiles/binuin16.h5",
+  "hdf5/tools/test/testfiles/binuin32.h5",
   "hdf5/tools/test/testfiles/bounds_latest_latest.h5",
   "hdf5/tools/test/testfiles/charsets.h5",
   "hdf5/tools/test/testfiles/compounds_array_vlen1.h5",
@@ -233,7 +295,11 @@
   "hdf5/tools/test/testfiles/file_space.h5",
   "hdf5/tools/test/testfiles/filter_fail.h5",
   "hdf5/tools/test/testfiles/h5clear_fsm_persist_equal.h5",
+  "hdf5/tools/test/testfiles/h5clear_fsm_persist_less.h5",
   "hdf5/tools/test/testfiles/h5clear_fsm_persist_noclose.h5",
+  "hdf5/tools/test/testfiles/h5clear_fsm_persist_user_equal.h5",
+  "hdf5/tools/test/testfiles/h5clear_fsm_persist_user_less.h5",
+  "hdf5/tools/test/testfiles/h5clear_mdc_image.h5",
   "hdf5/tools/test/testfiles/h5clear_sec2_v0.h5",
   "hdf5/tools/test/testfiles/h5clear_sec2_v2.h5",
   "hdf5/tools/test/testfiles/h5copy_extlinks_src.h5",
@@ -287,6 +353,7 @@
   "hdf5/tools/test/testfiles/h5diff_softlinks.h5",
   "hdf5/tools/test/testfiles/h5diff_strings1.h5",
   "hdf5/tools/test/testfiles/h5diff_strings2.h5",
+  "hdf5/tools/test/testfiles/h5diff_types.h5",
   "hdf5/tools/test/testfiles/h5fc_edge_v3.h5",
   "hdf5/tools/test/testfiles/h5fc_err_level.h5",
   "hdf5/tools/test/testfiles/h5fc_ext1_f.h5",
@@ -341,9 +408,12 @@
   "hdf5/tools/test/testfiles/h5repack_uint8be_ex.h5",
   "hdf5/tools/test/testfiles/h5stat_err_old_fill.h5",
   "hdf5/tools/test/testfiles/h5stat_err_old_layout.h5",
+  "hdf5/tools/test/testfiles/h5stat_err_refcount.h5",
   "hdf5/tools/test/testfiles/h5stat_filters.h5",
   "hdf5/tools/test/testfiles/h5stat_idx.h5",
+  "hdf5/tools/test/testfiles/h5stat_newgrat.h5",
   "hdf5/tools/test/testfiles/h5stat_threshold.h5",
+  "hdf5/tools/test/testfiles/h5stat_tsohm.h5",
   "hdf5/tools/test/testfiles/mod_h5clear_mdc_image.h5",
   "hdf5/tools/test/testfiles/non_comparables1.h5",
   "hdf5/tools/test/testfiles/non_comparables2.h5",
@@ -359,8 +429,15 @@
   "hdf5/tools/test/testfiles/t128bit_float.h5",
   "hdf5/tools/test/testfiles/tCVE-2021-37501_attr_decode.h5",
   "hdf5/tools/test/testfiles/tCVE_2018_11206_fill_new.h5",
+  "hdf5/tools/test/testfiles/tCVE_2018_11206_fill_old.h5",
   "hdf5/tools/test/testfiles/taindices.h5",
+  "hdf5/tools/test/testfiles/tall.h5",
+  "hdf5/tools/test/testfiles/tarray1.h5",
   "hdf5/tools/test/testfiles/tarray1_big.h5",
+  "hdf5/tools/test/testfiles/tarray2.h5",
+  "hdf5/tools/test/testfiles/tarray3.h5",
+  "hdf5/tools/test/testfiles/tarray4.h5",
+  "hdf5/tools/test/testfiles/tarray5.h5",
   "hdf5/tools/test/testfiles/tarray8.h5",
   "hdf5/tools/test/testfiles/tattr.h5",
   "hdf5/tools/test/testfiles/tattr2.h5",
@@ -372,20 +449,26 @@
   "hdf5/tools/test/testfiles/tbigdims.h5",
   "hdf5/tools/test/testfiles/tbinary.h5",
   "hdf5/tools/test/testfiles/tbitnopaque.h5",
+  "hdf5/tools/test/testfiles/tchar.h5",
   "hdf5/tools/test/testfiles/tcmpdattrintsize.h5",
   "hdf5/tools/test/testfiles/tcmpdintarray.h5",
   "hdf5/tools/test/testfiles/tcmpdints.h5",
   "hdf5/tools/test/testfiles/tcmpdintsize.h5",
   "hdf5/tools/test/testfiles/tcomplex.h5",
+  "hdf5/tools/test/testfiles/tcompound.h5",
+  "hdf5/tools/test/testfiles/tcompound_complex.h5",
   "hdf5/tools/test/testfiles/tcompound_complex2.h5",
+  "hdf5/tools/test/testfiles/tdatareg.h5",
   "hdf5/tools/test/testfiles/tdset.h5",
   "hdf5/tools/test/testfiles/tdset2.h5",
   "hdf5/tools/test/testfiles/tdset_idx.h5",
+  "hdf5/tools/test/testfiles/tempty.h5",
   "hdf5/tools/test/testfiles/textlink.h5",
   "hdf5/tools/test/testfiles/textlinkfar.h5",
   "hdf5/tools/test/testfiles/textlinksrc.h5",
   "hdf5/tools/test/testfiles/textlinktar.h5",
   "hdf5/tools/test/testfiles/textpfe.h5",
+  "hdf5/tools/test/testfiles/tfcontents1.h5",
   "hdf5/tools/test/testfiles/tfcontents2.h5",
   "hdf5/tools/test/testfiles/tfilters.h5",
   "hdf5/tools/test/testfiles/tfloat16.h5",
@@ -412,6 +495,7 @@
   "hdf5/tools/test/testfiles/tloop.h5",
   "hdf5/tools/test/testfiles/tnamed_dtype_attr.h5",
   "hdf5/tools/test/testfiles/tnestedcmpddt.h5",
+  "hdf5/tools/test/testfiles/tnestedcomp.h5",
   "hdf5/tools/test/testfiles/tno-subset.h5",
   "hdf5/tools/test/testfiles/tnullspace.h5",
   "hdf5/tools/test/testfiles/torderattr.h5",
@@ -426,6 +510,7 @@
   "hdf5/tools/test/testfiles/trefer_param.h5",
   "hdf5/tools/test/testfiles/trefer_reg.h5",
   "hdf5/tools/test/testfiles/trefer_reg_1d.h5",
+  "hdf5/tools/test/testfiles/tsaf.h5",
   "hdf5/tools/test/testfiles/tscalarattrintsize.h5",
   "hdf5/tools/test/testfiles/tscalarintattrsize.h5",
   "hdf5/tools/test/testfiles/tscalarintsize.h5",
@@ -435,39 +520,82 @@
   "hdf5/tools/test/testfiles/tst_onion_dset_1d.h5",
   "hdf5/tools/test/testfiles/tst_onion_dset_ext.h5",
   "hdf5/tools/test/testfiles/tst_onion_objs.h5",
+  "hdf5/tools/test/testfiles/tstr.h5",
+  "hdf5/tools/test/testfiles/tstr2.h5",
   "hdf5/tools/test/testfiles/tstr3.h5",
   "hdf5/tools/test/testfiles/tudfilter.h5",
   "hdf5/tools/test/testfiles/tudfilter2.h5",
+  "hdf5/tools/test/testfiles/tudlink.h5",
+  "hdf5/tools/test/testfiles/tvldtypes1.h5",
+  "hdf5/tools/test/testfiles/tvldtypes2.h5",
+  "hdf5/tools/test/testfiles/tvldtypes3.h5",
+  "hdf5/tools/test/testfiles/tvldtypes4.h5",
+  "hdf5/tools/test/testfiles/tvldtypes5.h5",
   "hdf5/tools/test/testfiles/tvlenstr_array.h5",
   "hdf5/tools/test/testfiles/tvlstr.h5",
   "hdf5/tools/test/testfiles/tvms.h5",
+  "hdf5/tools/test/testfiles/twithub.h5",
+  "hdf5/tools/test/testfiles/twithub513.h5",
+  "hdf5/tools/test/testfiles/txtfp32.h5",
+  "hdf5/tools/test/testfiles/txtfp64.h5",
+  "hdf5/tools/test/testfiles/txtin16.h5",
+  "hdf5/tools/test/testfiles/txtin32.h5",
+  "hdf5/tools/test/testfiles/txtin8.h5",
   "hdf5/tools/test/testfiles/txtstr.h5",
+  "hdf5/tools/test/testfiles/txtuin16.h5",
+  "hdf5/tools/test/testfiles/txtuin32.h5",
   "hdf5/tools/test/testfiles/vds/1_a.h5",
   "hdf5/tools/test/testfiles/vds/1_b.h5",
   "hdf5/tools/test/testfiles/vds/1_c.h5",
   "hdf5/tools/test/testfiles/vds/1_d.h5",
   "hdf5/tools/test/testfiles/vds/1_e.h5",
   "hdf5/tools/test/testfiles/vds/1_f.h5",
+  "hdf5/tools/test/testfiles/vds/1_vds.h5",
   "hdf5/tools/test/testfiles/vds/2_a.h5",
   "hdf5/tools/test/testfiles/vds/2_b.h5",
   "hdf5/tools/test/testfiles/vds/2_c.h5",
   "hdf5/tools/test/testfiles/vds/2_d.h5",
   "hdf5/tools/test/testfiles/vds/2_e.h5",
+  "hdf5/tools/test/testfiles/vds/2_vds.h5",
+  "hdf5/tools/test/testfiles/vds/3_1_vds.h5",
+  "hdf5/tools/test/testfiles/vds/3_2_vds.h5",
   "hdf5/tools/test/testfiles/vds/4_0.h5",
   "hdf5/tools/test/testfiles/vds/4_1.h5",
   "hdf5/tools/test/testfiles/vds/4_2.h5",
+  "hdf5/tools/test/testfiles/vds/4_vds.h5",
   "hdf5/tools/test/testfiles/vds/5_a.h5",
   "hdf5/tools/test/testfiles/vds/5_b.h5",
   "hdf5/tools/test/testfiles/vds/5_c.h5",
+  "hdf5/tools/test/testfiles/vds/5_vds.h5",
   "hdf5/tools/test/testfiles/vds/a.h5",
   "hdf5/tools/test/testfiles/vds/b.h5",
   "hdf5/tools/test/testfiles/vds/c.h5",
   "hdf5/tools/test/testfiles/vds/d.h5",
   "hdf5/tools/test/testfiles/vds/f-0.h5",
   "hdf5/tools/test/testfiles/vds/f-3.h5",
+  "hdf5/tools/test/testfiles/vds/vds-eiger.h5",
+  "hdf5/tools/test/testfiles/vds/vds-percival-unlim-maxmin.h5",
+  "hdf5/tools/test/testfiles/xml/tbitfields.h5",
+  "hdf5/tools/test/testfiles/xml/tcompound2.h5",
+  "hdf5/tools/test/testfiles/xml/tdset2.h5",
+  "hdf5/tools/test/testfiles/xml/tenum.h5",
   "hdf5/tools/test/testfiles/xml/test35.nc",
   "hdf5/tools/test/testfiles/xml/tloop2.h5",
+  "hdf5/tools/test/testfiles/xml/tmany.h5",
+  "hdf5/tools/test/testfiles/xml/tname-amp.h5",
+  "hdf5/tools/test/testfiles/xml/tname-apos.h5",
+  "hdf5/tools/test/testfiles/xml/tname-gt.h5",
+  "hdf5/tools/test/testfiles/xml/tname-lt.h5",
+  "hdf5/tools/test/testfiles/xml/tname-quot.h5",
+  "hdf5/tools/test/testfiles/xml/tname-sp.h5",
+  "hdf5/tools/test/testfiles/xml/tnodata.h5",
+  "hdf5/tools/test/testfiles/xml/tobjref.h5",
   "hdf5/tools/test/testfiles/xml/topaque.h5",
+  "hdf5/tools/test/testfiles/xml/tref-escapes-at.h5",
+  "hdf5/tools/test/testfiles/xml/tref-escapes.h5",
+  "hdf5/tools/test/testfiles/xml/tref.h5",
+  "hdf5/tools/test/testfiles/xml/tstring-at.h5",
+  "hdf5/tools/test/testfiles/xml/tstring.h5",
   "hdf5/tools/test/testfiles/zerodim.h5",
   "netcdf-c/h5_test/ref_tst_h_compounds.h5",
   "netcdf-c/h5_test/ref_tst_h_compounds2.h5",
@@ -504,6 +632,8 @@
   "netcdf4-python/examples/data/rtofs_glo_3dz_f006_6hrly_reg3.nc",
   "netcdf4-python/test/20171025_2056.Cloud_Top_Height.nc",
   "netcdf4-python/test/issue1152.nc",
+  "netcdf4-python/test/issue671.nc",
+  "netcdf4-python/test/issue672.nc",
   "netcdf4-python/test/test_gold.nc",
   "usnistgov_h5wasm/test/array.h5",
   "usnistgov_h5wasm/test/compressed.h5",

conformance/compare.py

@@ -146,7 +146,13 @@ for rel in files:
             if a.get("kind") != b.get("kind") and "error" not in b and "error" not in a:
                 issues.append(("mismatch", f"{p}: kind {a.get('kind')} vs ours {b.get('kind')}", "kind", b))
                 ok = False
+            # h5py could not open the object at all: it read none of its
+            # attributes or links, so there is nothing to compare ours with
+            # (the object's own error is compared above and below).
+            ref_unopened = a.get("kind") == "unknown" and "error" in a
             for k in ("error", "list_error", "attrs_error"):
+                if ref_unopened and k != "error":
+                    continue
                 if k in b and k not in a:
                     issues.append(("our-error", f"{p}: {k}: {b[k]}", b[k], b))
                     ok = False
@@ -164,7 +170,7 @@ for rel in files:
                     issues.append(("mismatch", f"{p}: values differ (h5py {a.get('dtype')} vs ours {b.get('dtype')})", "values", b | {"ref_head": a.get("head"), "ref_dtype": a.get("dtype")}))
                     ok = False
             ra, oa = a.get("attrs") or {}, b.get("attrs") or {}
-            if "attrs_error" not in b and "attrs_error" not in a:
+            if "attrs_error" not in b and "attrs_error" not in a and not ref_unopened:
                 for an in sorted(set(ra) | set(oa)):
                     x, y = ra.get(an), oa.get(an)
                     if x is None:

conformance/probe/Cargo.lock

@@ -29,6 +29,15 @@ version = "1.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
 
+[[package]]
+name = "bzip2"
+version = "0.6.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c"
+dependencies = [
+ "libbz2-rs-sys",
+]
+
 [[package]]
 name = "cc"
 version = "1.5.1"
@@ -52,12 +61,16 @@ name = "clawhdf5-format"
 version = "2.7.0"
 dependencies = [
  "byteorder",
+ "bzip2",
  "flate2",
  "libaec-sys",
+ "libc",
  "lz4_flex",
  "pco",
  "portable-atomic",
+ "ruzstd",
  "sha2",
+ "snap",
  "zstd",
 ]
 
@@ -192,6 +205,12 @@ dependencies = [
  "pkg-config",
 ]
 
+[[package]]
+name = "libbz2-rs-sys"
+version = "0.2.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "34b357333733e8260735ba5894eb928c02ecc69c78715f01a8019e7fa7f2db4c"
+
 [[package]]
 name = "libc"
 version = "0.2.189"
@@ -286,6 +305,15 @@ dependencies = [
  "rand_core",
 ]
 
+[[package]]
+name = "ruzstd"
+version = "0.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a252f5e20f038fe7b4ea53e073e65398d652c864cc162fc77c56c2f13717b888"
+dependencies = [
+ "twox-hash",
+]
+
 [[package]]
 name = "serde"
 version = "1.0.229"
@@ -351,6 +379,12 @@ version = "0.3.10"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
 
+[[package]]
+name = "snap"
+version = "1.1.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "199905e6153d6405f9728fe44daace35f8f837bbf830bb6e85fbd5828709a886"
+
 [[package]]
 name = "syn"
 version = "2.0.119"

conformance/probe/Cargo.toml

@@ -12,7 +12,7 @@ description = "Walks an HDF5 file with clawhdf5-format and prints a canonical JS
 [workspace]
 
 [dependencies]
-clawhdf5-format = { path = "../../crates/clawhdf5-format", features = ["lz4", "zstd", "szip", "pcodec"] }
+clawhdf5-format = { path = "../../crates/clawhdf5-format", features = ["lz4", "zstd", "szip", "pcodec", "plugin-filters"] }
 serde_json = "1"
 sha2 = "0.10"
 

conformance/probe/src/main.rs

@@ -19,9 +19,8 @@
 //! with its message, location and the clawhdf5 frames of its backtrace.
 
 use std::cell::RefCell;
-use std::collections::{HashMap, HashSet};
+use std::collections::HashSet;
 use std::panic::{self, AssertUnwindSafe};
-use std::rc::Rc;
 
 use clawhdf5_format::attribute::extract_attributes_full;
 use clawhdf5_format::data_layout::DataLayout;
@@ -29,14 +28,14 @@ use clawhdf5_format::data_read;
 use clawhdf5_format::dataspace::{Dataspace, DataspaceType};
 use clawhdf5_format::datatype::{Datatype, DatatypeByteOrder};
 use clawhdf5_format::filter_pipeline::FilterPipeline;
-use clawhdf5_format::global_heap::GlobalHeapCollection;
 use clawhdf5_format::group_v1::{self, GroupEntry};
 use clawhdf5_format::group_v2;
 use clawhdf5_format::message_type::MessageType;
-use clawhdf5_format::object_header::ObjectHeader;
+use clawhdf5_format::object_header::{ObjectClass, ObjectHeader};
 use clawhdf5_format::signature;
 use clawhdf5_format::superblock::Superblock;
 use clawhdf5_format::symbol_table::SymbolTableMessage;
+use clawhdf5_format::vl_data::{VlResolver, check_element_size};
 use serde_json::{Map, Value, json};
 use sha2::{Digest, Sha256};
 
@@ -111,7 +110,10 @@ struct Ctx<'a> {
     os: u8,
     ls: u8,
     base_dir: std::path::PathBuf,
-    heaps: RefCell<HashMap<u64, Result<Rc<GlobalHeapCollection>, String>>>,
+    /// Resolves variable-length elements as the library does (null
+    /// elements, strings cut at a NUL, heap objects of the wrong size
+    /// refused), caching each heap collection.
+    vl: RefCell<VlResolver<'a>>,
 }
 
 impl<'a> Ctx<'a> {
@@ -130,33 +132,6 @@ impl<'a> Ctx<'a> {
         }
     }
 
-    fn heap_obj(&self, addr: u64, idx: u32) -> Result<Vec<u8>, String> {
-        let coll = {
-            let mut cache = self.heaps.borrow_mut();
-            cache
-                .entry(addr)
-                .or_insert_with(|| {
-                    GlobalHeapCollection::parse(self.data, addr as usize, self.ls)
-                        .map(Rc::new)
-                        .map_err(e)
-                })
-                .clone()?
-        };
-        coll.get_object(idx as u16)
-            .map(|o| o.data.clone())
-            .ok_or_else(|| {
-                format!("GlobalHeapObjectNotFound {{ collection_address: {addr}, index: {idx} }}")
-            })
-    }
-
-    fn read_offset(&self, b: &[u8]) -> u64 {
-        let mut v = 0u64;
-        for (i, x) in b.iter().take(self.os as usize).enumerate() {
-            v |= (*x as u64) << (8 * i);
-        }
-        v
-    }
-
     fn canon(&self, dt: &Datatype, b: &[u8], out: &mut Vec<u8>) -> Result<(), String> {
         let size = dt.type_size() as usize;
         if b.len() < size {
@@ -204,41 +179,27 @@ impl<'a> Ctx<'a> {
                 }
             }
             Datatype::VariableLength {
+                size: vl_size,
                 is_string,
                 base_type,
                 ..
             } => {
-                let len = u32::from_le_bytes([b[0], b[1], b[2], b[3]]) as usize;
+                check_element_size(*vl_size, self.os).map_err(e)?;
-                let addr = self.read_offset(&b[4..]);
+                let el = &b[..size];
-                let idx_off = 4 + self.os as usize;
-                let idx = u32::from_le_bytes([
-                    b[idx_off],
-                    b[idx_off + 1],
-                    b[idx_off + 2],
-                    b[idx_off + 3],
-                ]);
-                let obj = if len == 0 || addr == 0 || addr == u64::MAX >> (64 - 8 * self.os as u32)
-                {
-                    Vec::new()
-                } else {
-                    self.heap_obj(addr, idx)?
-                };
                 if *is_string {
-                    let l = len.min(obj.len());
+                    let s = self.vl.borrow_mut().string_bytes(el).map_err(e)?;
-                    canon_str(&obj[..l], out);
+                    canon_str(&s[0], out);
                 } else {
                     let bs = base_type.type_size() as usize;
-                    if bs == 0 {
+                    // The borrow ends here: the base type may itself be
-                        return Err("canon: VL base size 0".into());
+                    // variable-length.
-                    }
+                    let seq = self.vl.borrow_mut().sequences(el, bs).map_err(e)?;
-                    let need = len.checked_mul(bs).ok_or("canon: VL overflow")?;
+                    let seq = &seq[0];
-                    if len > 0 && obj.len() < need {
+                    let len = seq.len() / bs;
-                        return Err(format!("canon: VL object {} < {need}", obj.len()));
-                    }
                     out.push(b'V');
                     out.extend_from_slice(&(len as u32).to_le_bytes());
                     for i in 0..len {
-                        self.canon(base_type, &obj[i * bs..], out)?;
+                        self.canon(base_type, &seq[i * bs..], out)?;
                     }
                 }
             }
@@ -286,21 +247,66 @@ impl<'a> Ctx<'a> {
         Ok(())
     }
 
+    /// VDS source files resolve next to the virtual file; like the library,
+    /// refuse absolute paths and `..`.
+    fn vds_resolver(
+        &self,
+    ) -> impl Fn(&str) -> Result<Option<Vec<u8>>, clawhdf5_format::error::FormatError> + use<> {
+        let base = self.base_dir.clone();
+        move |name: &str| {
+            use clawhdf5_format::error::FormatError;
+            let p = std::path::Path::new(name);
+            if p.is_absolute()
+                || p.components()
+                    .any(|c| matches!(c, std::path::Component::ParentDir))
+            {
+                return Err(FormatError::ChunkedReadError(format!("refused {name}")));
+            }
+            match std::fs::read(base.join(p)) {
+                Ok(b) => Ok(Some(b)),
+                Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(None),
+                Err(err) => Err(FormatError::ChunkedReadError(err.to_string())),
+            }
+        }
+    }
+
+    fn read_named_datatype(&self, h: &ObjectHeader) -> Result<(), String> {
+        let dtb = self
+            .payload(h, MessageType::Datatype)?
+            .ok_or("MissingMessage(Datatype)")?;
+        Datatype::parse_in_header(&dtb, h.version).map_err(e)?;
+        Ok(())
+    }
+
     fn read_dataset(&self, h: &ObjectHeader, rec: &mut Map<String, Value>) -> Result<(), String> {
         let dtb = self
             .payload(h, MessageType::Datatype)?
             .ok_or("MissingMessage(Datatype)")?;
-        let (dt, _) = Datatype::parse(&dtb).map_err(e)?;
+        let (dt, _) = Datatype::parse_in_header(&dtb, h.version).map_err(e)?;
         rec.insert("dtype".into(), Value::String(dtype_str(&dt)));
         let dsb = self
             .payload(h, MessageType::Dataspace)?
             .ok_or("MissingMessage(Dataspace)")?;
-        let ds = Dataspace::parse(&dsb, self.ls).map_err(e)?;
+        let mut ds = Dataspace::parse(&dsb, self.ls).map_err(e)?;
-        let (shape, n) = Self::shape(&ds);
+        // A virtual dataset's extent can come from its sources (unlimited /
-        rec.insert("shape".into(), shape);
+        // printf mappings), as h5py reports it, rather than the stored one.
-        if n.saturating_mul(dt.type_size() as u64) > MAX_BYTES {
+        if let Some(lm) = h
-            rec.insert("skipped".into(), Value::String("too large".into()));
+            .messages
-            return Ok(());
+            .iter()
+            .find(|m| m.msg_type == MessageType::DataLayout)
+            && let Ok(dl @ DataLayout::Virtual { .. }) =
+                DataLayout::parse(&lm.data, self.os, self.ls)
+        {
+            let resolver = self.vds_resolver();
+            ds.dimensions = clawhdf5_format::vds::virtual_dataset_extent(
+                self.data,
+                &dl,
+                &ds,
+                self.os,
+                self.ls,
+                Some(&resolver),
+            )
+            .map_err(e)?;
         }
         let lm = h
             .messages
@@ -308,6 +314,14 @@ impl<'a> Ctx<'a> {
             .find(|m| m.msg_type == MessageType::DataLayout)
             .ok_or("MissingMessage(DataLayout)")?;
         let dl = DataLayout::parse(&lm.data, self.os, self.ls).map_err(e)?;
+        // What libhdf5 checks when it opens the dataset (as File::dataset).
+        data_read::check_dataset_storage(&dl, &ds, &dt, self.data.len() as u64).map_err(e)?;
+        let (shape, n) = Self::shape(&ds);
+        rec.insert("shape".into(), shape);
+        if n.saturating_mul(dt.type_size() as u64) > MAX_BYTES {
+            rec.insert("skipped".into(), Value::String("too large".into()));
+            return Ok(());
+        }
         rec.insert(
             "layout".into(),
             Value::String(
@@ -331,28 +345,26 @@ impl<'a> Ctx<'a> {
             );
         }
         let raw = if matches!(dl, DataLayout::Virtual { .. }) {
-            let base = self.base_dir.clone();
+            let resolver = self.vds_resolver();
-            let resolver = move |name: &str| -> Option<Vec<u8>> {
+            let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
-                let p = std::path::Path::new(name);
+                self.data,
-                if p.is_absolute()
+                &h.messages,
-                    || p.components()
+                self.os,
-                        .any(|c| matches!(c, std::path::Component::ParentDir))
+                self.ls,
-                {
+            )
-                    return None;
+            .map_err(e)?;
-                }
+            clawhdf5_format::vds::read_virtual_dataset(
-                std::fs::read(base.join(p)).ok()
-            };
-            data_read::read_raw_data_full_with_resolver(
                 self.data,
                 &dl,
                 &ds,
                 &dt,
-                pipeline.as_ref(),
+                fill.as_deref(),
                 self.os,
                 self.ls,
                 Some(&resolver),
             )
             .map_err(e)?
+            .data
         } else {
             let cache = clawhdf5_format::chunk_cache::ChunkCache::new();
             clawhdf5_format::fill_value::read_full_with_fill::<clawhdf5_format::error::FormatError>(
@@ -647,6 +659,20 @@ fn is_group(h: &ObjectHeader) -> bool {
     })
 }
 
+/// The probe's kind for an object header: libhdf5's object class
+/// ([`ObjectHeader::object_class`]: group, then dataset — a datatype *and* a
+/// dataspace — then named datatype), which is what h5py opens the object as.
+/// The root group, and a header with only link messages, count as groups.
+fn kind_of(h: &ObjectHeader, is_root: bool) -> &'static str {
+    match h.object_class() {
+        Some(ObjectClass::Group) => "group",
+        Some(ObjectClass::Dataset) => "dataset",
+        _ if is_root || is_group(h) => "group",
+        Some(ObjectClass::NamedDatatype) => "datatype",
+        None => "unknown",
+    }
+}
+
 fn main() {
     install_hook();
     let path = std::env::args().nth(1).expect("usage: probe <file>");
@@ -660,10 +686,13 @@ fn main() {
             return;
         }
     };
-    let sb = guarded(|| {
+    // Every address is relative to the superblock: look at the file from
-        let off = signature::find_signature(&data).map_err(e)?;
+    // there on (past any user block), as libhdf5 does.
-        Superblock::parse(&data, off).map_err(e)
+    let hdf5: &[u8] = match signature::find_signature(&data) {
-    });
+        Ok(off) => &data[off..],
+        Err(_) => &data,
+    };
+    let sb = guarded(|| Superblock::parse(hdf5, 0).map_err(e));
     let sb = match sb {
         Ok(sb) => sb,
         Err(msg) => {
@@ -672,16 +701,63 @@ fn main() {
             return;
         }
     };
+    // libhdf5 refuses a truncated file and reads nothing past the recorded
+    // end of file.
+    let base = (data.len() - hdf5.len()) as u64;
+    let hdf5 = match sb.data_end(base, data.len() as u64) {
+        Ok(end) => &hdf5[..end as usize],
+        Err(err) => {
+            top.insert("open_error".into(), Value::String(e(err)));
+            println!("{}", Value::Object(top));
+            return;
+        }
+    };
+    // libhdf5 decodes the superblock extension at open (an error refuses
+    // the file), and loads a metadata cache image over the file's own
+    // metadata. It loads the image only when it first reads metadata — the
+    // root group — so a file whose image it cannot load still opens and
+    // that read fails. The library decides all three cases with the same
+    // `cache_image_state`: `File` and `MmapFile` open such a file and fail
+    // every object lookup with the image's error, which is what the probe
+    // records here (on the root group, where libhdf5 reports it).
+    use clawhdf5_format::superblock_ext::{self, CacheImageState};
+    let state = match guarded(|| superblock_ext::cache_image_state(hdf5, &sb).map_err(e)) {
+        Ok(x) => x,
+        Err(msg) => {
+            top.insert("open_error".into(), Value::String(msg));
+            println!("{}", Value::Object(top));
+            return;
+        }
+    };
+    let mut image_error = None;
+    let view = match state {
+        CacheImageState::Absent => None,
+        CacheImageState::Unloadable(err) => {
+            image_error = Some(e(err));
+            None
+        }
+        CacheImageState::Loaded(image) => {
+            let mut v = hdf5.to_vec();
+            match image.block(hdf5).and_then(|b| image.apply(b, &mut v)) {
+                Ok(()) => Some(v),
+                Err(err) => {
+                    image_error = Some(e(err));
+                    None
+                }
+            }
+        }
+    };
+    let hdf5: &[u8] = view.as_deref().unwrap_or(hdf5);
     top.insert("superblock_version".into(), json!(sb.version));
     let ctx = Ctx {
-        data: &data,
+        data: hdf5,
         os: sb.offset_size,
         ls: sb.length_size,
         base_dir: std::path::Path::new(&path)
             .parent()
             .map(|p| p.to_path_buf())
             .unwrap_or_default(),
-        heaps: RefCell::new(HashMap::new()),
+        vl: RefCell::new(VlResolver::new(hdf5, sb.offset_size, sb.length_size)),
     };
     let mut objects: Vec<Value> = Vec::new();
     let mut visited = HashSet::new();
@@ -699,6 +775,9 @@ fn main() {
         let mut rec = Map::new();
         rec.insert("path".into(), Value::String(p.clone()));
         let r = guarded(|| {
+            if let Some(msg) = &image_error {
+                return Err(msg.clone());
+            }
             let h = ctx.header(addr)?;
             Ok(h)
         });
@@ -711,29 +790,20 @@ fn main() {
                 continue;
             }
         };
-        let is_ds = h
+        let kind = kind_of(&h, addr == sb.root_group_address);
-            .messages
-            .iter()
-            .any(|m| m.msg_type == MessageType::DataLayout);
-        let kind = if is_ds {
-            "dataset"
-        } else if is_group(&h) || addr == sb.root_group_address {
-            "group"
-        } else if h
-            .messages
-            .iter()
-            .any(|m| m.msg_type == MessageType::Datatype)
-        {
-            "datatype"
-        } else {
-            "unknown"
-        };
         rec.insert("kind".into(), Value::String(kind.into()));
         if kind == "dataset"
             && let Err(msg) = guarded(|| ctx.read_dataset(&h, &mut rec))
         {
             rec.insert("error".into(), Value::String(msg));
         }
+        // Opening a committed datatype decodes it (h5py's `f[name]` fails on
+        // one libhdf5 cannot decode), so decode it here too.
+        if kind == "datatype"
+            && let Err(msg) = guarded(|| ctx.read_named_datatype(&h))
+        {
+            rec.insert("error".into(), Value::String(msg));
+        }
         if kind != "datatype" {
             match guarded(|| ctx.attrs(&h)) {
                 Ok(m) => {
@@ -836,6 +906,42 @@ mod tests {
         assert!(ieee_layout(&f32le));
     }
 
+    #[test]
+    fn kind_follows_libhdf5_object_class() {
+        use clawhdf5_format::object_header::HeaderMessage;
+        let header = |types: &[MessageType]| ObjectHeader {
+            version: 2,
+            messages: types
+                .iter()
+                .map(|&msg_type| HeaderMessage {
+                    msg_type,
+                    size: 0,
+                    flags: 0,
+                    creation_order: None,
+                    data: Vec::new(),
+                })
+                .collect(),
+            reference_count: None,
+            flags: 0,
+            access_time: None,
+            modification_time: None,
+            change_time: None,
+            birth_time: None,
+        };
+        use MessageType::*;
+        // cve-2024-33874 `/Dset1`: a datatype and a layout but no dataspace
+        // is a named datatype to libhdf5 (h5py opens it as one).
+        assert_eq!(kind_of(&header(&[Datatype, DataLayout]), false), "datatype");
+        assert_eq!(
+            kind_of(&header(&[Datatype, Dataspace, DataLayout]), false),
+            "dataset"
+        );
+        assert_eq!(kind_of(&header(&[SymbolTable]), false), "group");
+        assert_eq!(kind_of(&header(&[Link]), false), "group");
+        assert_eq!(kind_of(&header(&[]), true), "group");
+        assert_eq!(kind_of(&header(&[]), false), "unknown");
+    }
+
     #[test]
     fn partial_precision_int_is_shifted_and_sign_extended() {
         let dt = Datatype::FixedPoint {

conformance/ref.py

@@ -111,8 +111,11 @@ def note_conversion(tid, dt, rec):
 
 
 def hash_values(arr, dt, rec):
-    if dt.subdtype is not None:
+    # h5py expands an HDF5 array element type into trailing array dims, a
-        # h5py expands an HDF5 array element type into trailing array dims
+    # nested array type (an array of arrays) into all of them. Converting the
+    # expanded array back to the inner subarray type would broadcast every
+    # element into a whole subarray, so strip every level.
+    while dt.subdtype is not None:
         dt = dt.subdtype[0]
     arr = np.asarray(arr, dtype=dt)
     if simple(dt):
@@ -173,9 +176,13 @@ def main(path):
         return
     objects = []
     seen = set()
-    stack = [("/", None)]
+    # Objects h5py cannot open have no ObjectID to deduplicate by; they are
+    # deduplicated by the address their hard link points at instead, as the
+    # probe deduplicates every object by header address.
+    seen_unopenable = set()
+    stack = [("/", None, None)]
     while stack:
-        p, obj = stack.pop()
+        p, obj, link_addr = stack.pop()
         if len(objects) >= MAX_OBJECTS:
             top["truncated"] = True
             break
@@ -185,6 +192,10 @@ def main(path):
                 obj = f[p]
             key = hash(obj.id)  # h5py ObjectID hash = (fileno, object address/token)
         except Exception as e:  # noqa: BLE001
+            if link_addr is not None:
+                if link_addr in seen_unopenable:
+                    continue
+                seen_unopenable.add(link_addr)
             rec["kind"] = "unknown"
             rec["error"] = err(e)
             objects.append(rec)
@@ -232,15 +243,18 @@ def main(path):
                 base = "" if p == "/" else p
                 kids = []
                 for n in names:
+                    # The link's own type: `obj.get(n, getlink=True)` reports
+                    # a user-defined link (type 64-255) as a HardLink.
                     try:
-                        link = obj.get(n, getlink=True)
+                        info = obj.id.links.get_info(n.encode("utf-8", "surrogateescape"))
                     except Exception:  # noqa: BLE001
-                        link = None
+                        info = None
-                    if link is not None and not isinstance(link, h5py.HardLink):
+                    if info is not None and info.type != h5py.h5l.TYPE_HARD:
                         continue
-                    kids.append(f"{base}/{n}")
+                    addr = info.u if info is not None else None
-                for k in reversed(kids):
+                    kids.append((f"{base}/{n}", addr))
-                    stack.append((k, None))
+                for k, addr in reversed(kids):
+                    stack.append((k, None, addr))
             except Exception as e:  # noqa: BLE001
                 rec["list_error"] = err(e)
         objects.append(rec)

conformance/report.py

@@ -98,13 +98,37 @@ def is_h5py_be_vlen(i):
             and ">" in (i.get("ours_dtype") or ""))
 
 
+# Objects the reference (h5py 3.16 / HDF5 2.0) reads only because of an
+# HDF5 2.0 bug, and that clawhdf5 refuses: each one reads past a buffer or
+# returns bytes the file does not hold, and libhdf5's develop branch refuses all
+# three. (file, object) -> why. Checked 2026-09-26 against HDF5 2.0.0
+# and HDFGroup/hdf5 develop sources; see docs/known-issues.md.
+LIBHDF5_BUGS = {
+    ("cve_hdf5/cvefiles/cve-2025-2308.h5", "/Scale_offset_long_long_data_le"):
+        "scale-offset codes run past the end of the chunk: HDF5 2.0 reads past its buffer; "
+        "libhdf5's develop branch refuses the chunk (\"Buffer too short\")",
+    ("cve_hdf5/cvefiles/cve-2025-44904.h5", "/Scale_offset_float_data_le"):
+        "unfiltered chunks of 38 and 37 bytes for 48-byte chunks: HDF5 2.0 fills the rest with "
+        "whatever its buffer held; libhdf5's develop branch refuses them (\"incorrect chunk size returned "
+        "from index for unfiltered chunk\")",
+    ("hdf5/test/testfiles/bad_nbit_parms_walk.h5", "/Nbit_int_data_le"):
+        "an N-Bit parameter list one value short: HDF5 2.0 reads past the list; libhdf5's own "
+        "test (`test_filter_bad_params`, test/dsets.c) now requires the read to fail",
+}
+
+
+def is_libhdf5_bug(rel, i):
+    return i["kind"] == "our-error" and any(
+        f == rel and i["detail"].startswith(obj + ":") for (f, obj) in LIBHDF5_BUGS)
+
+
 known = collections.defaultdict(list)
 for r in rows:
-    if r["class"] != "mismatch":
-        continue
     iss = issues.get(r["file"], [])
-    if iss and all(is_h5py_be_vlen(i) for i in iss):
+    if r["class"] == "mismatch" and iss and all(is_h5py_be_vlen(i) for i in iss):
         known["h5py-be-vlen"].append(r["file"])
+    if r["class"] == "our-error" and iss and all(is_libhdf5_bug(r["file"], i) for i in iss):
+        known["libhdf5-2.0"].append(r["file"])
 
 
 # --- the CVE corpus: clawhdf5 vs h5dump vs h5py ------------------------------
@@ -230,9 +254,13 @@ for c in sorted(by_corpus):
     w(f"| {c} | {sum(cnt.values())} | " + " | ".join(str(cnt.get(k, 0)) for k in CLASSES) + " |")
 w(f"| **all** | **{len(rows)}** | " + " | ".join(f"**{total.get(k, 0)}**" for k in CLASSES) + " |")
 w("")
-n_known = sum(len(v) for v in known.values())
+if known["h5py-be-vlen"]:
-if n_known:
+    w(f"{len(known['h5py-be-vlen'])} of the {total.get('mismatch', 0)} mismatches are a known h5py bug, "
-    w(f"{n_known} of the {total.get('mismatch', 0)} mismatches are a known h5py bug, not ours (see *Known not-our-bug*).")
+      "not ours (see *Known not-our-bug*).")
+    w("")
+if known["libhdf5-2.0"]:
+    w(f"{len(known['libhdf5-2.0'])} of the {total.get('our-error', 0)} our-errors are corrupt data that "
+      "HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).")
     w("")
 w("Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):")
 w("")
@@ -311,6 +339,11 @@ if res["incomparable"]:
     w("  (FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not")
     w("  compared (shape and presence still are): "
       + ", ".join(f"{k} ({n}x)" for k, n in res["incomparable"]) + ".")
+w("- **Corrupt data HDF5 2.0 reads through a bug.** clawhdf5 refuses these objects; h5py 3.16 /")
+w("  HDF5 2.0 returns values for them that the file does not hold:")
+for (f, obj), why in sorted(LIBHDF5_BUGS.items()):
+    here = "" if f in known["libhdf5-2.0"] else " (not an our-error in this run)"
+    w(f"  - `{f}` `{obj}`: {why}{here}.")
 w("- **References** are compared by presence only (`R`), not by target.")
 w("")
 if res.get("ref_only_errors"):

crates/clawhdf5-accel/src/lib.rs

@@ -237,7 +237,10 @@ pub fn f16_to_f32_batch(input: &[u16], output: &mut [f32]) {
     convert::f16_to_f32_batch(input, output);
 }
 
-/// Compute Fletcher-32 checksum.
+/// Compute a textbook Fletcher-32 checksum (both sums start at 0xffff).
+///
+/// This is not HDF5's checksum; the Fletcher-32 I/O filter uses
+/// `clawhdf5_format::checksum::fletcher32`.
 pub fn checksum_fletcher32(data: &[u8]) -> u32 {
     checksum::checksum_fletcher32(data)
 }

crates/clawhdf5-agent/src/schema.rs

@@ -477,10 +477,34 @@ fn write_string_dataset(
     }
 }
 
+/// `/meta`'s attributes, failing if any of them cannot be read.
+///
+/// `Group::attrs` leaves out an attribute it cannot decode. For the store's
+/// settings that would silently fall back to defaults (e.g. `float16`, the
+/// WAL mark), so an unreadable attribute is an error here, as it was before
+/// `attrs` became tolerant.
+fn meta_attrs(
+    file: &clawhdf5::File,
+) -> Result<std::collections::HashMap<String, AttrValue>, MemoryError> {
+    let meta = file
+        .group("meta")
+        .map_err(|e| MemoryError::Schema(format!("missing /meta group: {e}")))?;
+    let (attrs, errors) = meta
+        .attrs_with_errors()
+        .map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
+    if let Some(e) = errors.first() {
+        return Err(MemoryError::Schema(format!(
+            "cannot read /meta attrs: {} unreadable, first: {e}",
+            errors.len()
+        )));
+    }
+    Ok(attrs)
+}
+
 /// Validate an HDF5 file has the correct schema and load all data.
 /// Read the checkpoint's [`WalMark`] from `/meta`, if it has one.
 pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
-    let attrs = file.group("meta").ok()?.attrs().ok()?;
+    let attrs = meta_attrs(file).ok()?;
     let len = match attrs.get(WAL_APPLIED_LEN_ATTR)? {
         AttrValue::I64(v) => u64::try_from(*v).ok()?,
         _ => return None,
@@ -498,10 +522,7 @@ pub fn read_signature(
     file: &clawhdf5::File,
 ) -> Result<Option<crate::signing::StoredSignature>, MemoryError> {
     use crate::signing::{Manifest, StoredSignature, from_hex};
-    let attrs = file
+    let attrs = meta_attrs(file)?;
-        .group("meta")
-        .and_then(|g| g.attrs())
-        .map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
     let version = match attrs.get(SIG_VERSION_ATTR) {
         None => return Ok(None),
         Some(AttrValue::I64(v)) => *v,
@@ -552,18 +573,14 @@ pub fn read_signature(
 
 /// Read the checkpoint bookkeeping from `/meta`.
 pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
-    let ann_generation = file
+    let ann_generation =
-        .group("meta")
+        meta_attrs(file)
             .ok()
-        .and_then(|g| g.attrs().ok())
             .and_then(|attrs| match attrs.get(ANN_GENERATION_ATTR) {
                 Some(AttrValue::I64(v)) => Some(*v as u64),
                 _ => None,
             });
-    let signed = file
+    let signed = meta_attrs(file).is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
-        .group("meta")
-        .and_then(|g| g.attrs())
-        .is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
     CheckpointMeta {
         wal_applied: read_wal_mark(file),
         ann_generation,
@@ -575,12 +592,7 @@ pub fn validate_and_load(
     file: &clawhdf5::File,
 ) -> Result<(MemoryConfig, MemoryCache, SessionCache, KnowledgeCache), MemoryError> {
     // Read /meta group attributes
-    let meta = file
+    let attrs = meta_attrs(file)?;
-        .group("meta")
-        .map_err(|e| MemoryError::Schema(format!("missing /meta group: {e}")))?;
-    let attrs = meta
-        .attrs()
-        .map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
 
     let schema_version = match attrs.get("schema_version") {
         Some(AttrValue::String(s)) => s.clone(),

crates/clawhdf5-agent/tests/float16_store.rs

@@ -258,3 +258,43 @@ fn an_existing_f32_store_stays_f32() {
     assert_eq!(&values[..before.1.len()], before.1.as_slice());
     assert_eq!(&values[before.1.len()..], odd.as_slice());
 }
+
+/// `Group::attrs` leaves out an attribute it cannot decode. A store whose
+/// `float16` setting is unreadable must not open as `float16 = false` (or with
+/// any other default in place of a setting it has): it is an error.
+#[test]
+fn unreadable_meta_attribute_fails_open_instead_of_defaulting() {
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("store.h5");
+    {
+        let mut m = HDF5Memory::create(config(&dir, "store.h5", true)).unwrap();
+        m.save(entry(1)).unwrap();
+        m.flush_wal().unwrap();
+    }
+    assert!(HDF5Memory::open_read_only(&path).is_ok());
+
+    // Give the `float16` attribute message an unknown version (the name is
+    // at +8 in a version-1 message and +9 in a version-3 one).
+    let mut bytes = std::fs::read(&path).unwrap();
+    let name = b"float16\0";
+    let mut hit = false;
+    let positions: Vec<usize> = (9..bytes.len() - name.len())
+        .filter(|&p| &bytes[p..p + name.len()] == name)
+        .collect();
+    for pos in positions {
+        for (back, version) in [(8, 1u8), (9, 3u8)] {
+            if bytes[pos - back] == version {
+                bytes[pos - back] = 0x7f;
+                hit = true;
+            }
+        }
+    }
+    assert!(hit, "float16 attribute message not found");
+    std::fs::write(&path, &bytes).unwrap();
+
+    match HDF5Memory::open_read_only(&path) {
+        Err(MemoryError::Schema(msg)) => assert!(msg.contains("/meta"), "{msg}"),
+        Err(e) => panic!("unexpected error: {e}"),
+        Ok(_) => panic!("store opened with an unreadable float16 setting"),
+    }
+}

crates/clawhdf5-ann/src/hnsw.rs

@@ -13,7 +13,7 @@ use clawhdf5_format::filter_pipeline::FilterPipeline;
 use clawhdf5_format::group_v2::resolve_path_any;
 use clawhdf5_format::message_type::MessageType;
 use clawhdf5_format::object_header::ObjectHeader;
-use clawhdf5_format::signature::find_signature;
+use clawhdf5_format::signature::split_user_block;
 use clawhdf5_format::superblock::Superblock;
 use clawhdf5_io::FileWriter as IoFileWriter;
 
@@ -861,8 +861,9 @@ impl HnswIndex {
     /// The HDF5 data must contain the `/ann/vectors`, `/ann/graph_layer_*`,
     /// and `/ann/config` datasets as produced by [`to_hdf5_bytes`].
     pub fn load_from_hdf5(data: &[u8]) -> Result<Self, FormatError> {
-        let sig_offset = find_signature(data)?;
+        // Addresses are relative to the superblock: skip any user block.
-        let sb = Superblock::parse(data, sig_offset)?;
+        let (_, data) = split_user_block(data)?;
+        let sb = Superblock::parse(data, 0)?;
 
         // Read config dataset and its attributes
         let config_attrs = read_dataset_attrs(data, &sb, "ann/config")?;

crates/clawhdf5-bench/Cargo.toml

@@ -34,6 +34,10 @@ path = "src/bin/consolidation_efficiency.rs"
 name = "ephemeral_perf"
 path = "src/bin/ephemeral_perf.rs"
 
+[[bin]]
+name = "concurrent_read"
+path = "src/bin/concurrent_read.rs"
+
 [[bin]]
 name = "mpi_io_bench"
 path = "src/bin/mpi_io_bench.rs"
@@ -64,6 +68,10 @@ clawhdf5-io = { path = "../clawhdf5-io" }
 mpi = { version = "0.8", optional = true }
 serde = { workspace = true }
 serde_json = "1"
+# concurrent_read: size the decode pool (--decode-threads) and evict files
+# from the page cache (--cold, posix_fadvise). Both pure Rust / bindings only.
+rayon = "1"
+libc = "0.2"
 tempfile = { workspace = true }
 # Optional: libhdf5 C wrapper for side-by-side comparison (requires system libhdf5).
 # Enable with: cargo bench -p clawhdf5-bench --features libhdf5-compare

crates/clawhdf5-bench/scripts/compare_concurrent_read.py

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+"""Tabulate concurrent_read JSON results (clawhdf5, h5py threads/processes).
+
+    python compare_concurrent_read.py clawhdf5.json h5py-threads.json h5py-procs.json
+
+Prints one Markdown table: for each layout, mode and thread count, every
+tool's MB/s and scaling efficiency, and the first file's MB/s relative to each
+of the others. Refuses to compare runs whose workload parameters differ.
+"""
+
+import json
+import sys
+
+COMPARED = ("datasets", "rows", "cols", "chunk", "deflate_level", "slab", "slabs", "seed")
+
+
+def main(paths):
+    if len(paths) < 2:
+        sys.exit(__doc__)
+    docs = []
+    for p in paths:
+        with open(p) as fh:
+            docs.append(json.load(fh))
+    ref = docs[0]
+    for d, p in zip(docs[1:], paths[1:]):
+        diff = [k for k in COMPARED if d["params"].get(k) != ref["params"].get(k)]
+        if diff:
+            sys.exit(f"{p}: workload differs from {paths[0]} in {', '.join(diff)}")
+        if d["cache"] != ref["cache"]:
+            print(f"warning: {p} ran {d['cache']!r}, {paths[0]} ran {ref['cache']!r}",
+                  file=sys.stderr)
+        if d.get("host") != ref.get("host"):
+            print(f"warning: {p} ran on {d.get('host')}, {paths[0]} on {ref.get('host')}",
+                  file=sys.stderr)
+
+    names = [d["tool"] for d in docs]
+    for d in docs:
+        extra = f", HDF5 {d['hdf5_version']}" if "hdf5_version" in d else ""
+        print(f"- {d['tool']} {d['version']}{extra}: host {d.get('host')}, "
+              f"{d.get('cpus')} CPUs, cache {d['cache']}, decode threads per read "
+              f"{d.get('decode_threads')}")
+    p = ref["params"]
+    print(f"\n{p['datasets']} datasets of {p['rows']} x {p['cols']} f32, chunks "
+          f"{p['chunk'][0]} x {p['chunk'][1]} (deflate {p['deflate_level']}); "
+          f"`same`: {p['slabs']} slabs of {p['slab']} x {p['slab']}\n")
+
+    index = [{(r["layout"], r["mode"], r["threads"]): r for r in d["results"]} for d in docs]
+    keys = [(r["layout"], r["mode"], r["threads"]) for r in ref["results"]]
+
+    head = ["layout", "mode", "threads"]
+    head += [f"{n} MB/s (eff)" for n in names]
+    head += [f"{names[0]} / {n}" for n in names[1:]]
+    print("| " + " | ".join(head) + " |")
+    print("|---|---|" + "---:|" * (len(head) - 2))
+    for key in keys:
+        cells = [key[0], key[1], str(key[2])]
+        rs = [ix.get(key) for ix in index]
+        for r in rs:
+            if r is None:
+                cells.append("-")
+            else:
+                eff = "-" if r["efficiency"] is None else f"{r['efficiency']:.2f}"
+                cells.append(f"{r['mb_s']:.0f} ({eff})")
+        for r in rs[1:]:
+            cells.append("-" if r is None else f"{rs[0]['mb_s'] / r['mb_s']:.2f}x")
+        print("| " + " | ".join(cells) + " |")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

crates/clawhdf5-bench/scripts/concurrent_read_h5py.py

@@ -0,0 +1,265 @@
+#!/usr/bin/env python3
+"""The concurrent_read workload with h5py, on the files concurrent_read wrote.
+
+libhdf5 serialises every API call under one global lock, and h5py holds its
+own global lock around every call as well, so h5py *threads* cannot decode in
+parallel. h5py users scale with *processes* instead; ``--executor processes``
+measures that (each worker opens the file itself).
+
+The workload mirrors ``crates/clawhdf5-bench/src/bin/concurrent_read.rs``:
+
+* ``distinct``: every dataset read in full once per repetition; worker ``t``
+  of ``T`` reads datasets ``t, t + T, ...``.
+* ``same``: ``--slabs`` random ``--slab`` x ``--slab`` hyperslabs of ``d00``
+  (slab ``j`` to worker ``j % T``), offsets from the same splitmix64 stream.
+
+Each worker times itself from a start barrier; a repetition spans the earliest
+start to the latest finish (CLOCK_MONOTONIC, comparable across processes).
+Threads share one ``h5py.File`` per repetition; process workers open the file
+inside the timed region (a few ms against reads of many MiB).
+
+Generate the files first with the Rust harness (it writes ``manifest.json``),
+then, for example::
+
+    python concurrent_read_h5py.py --dir DIR --executor threads --json h5py-threads.json
+    python concurrent_read_h5py.py --dir DIR --executor processes --json h5py-procs.json
+"""
+
+import argparse
+import json
+import multiprocessing as mp
+import os
+import platform
+import socket
+import sys
+import threading
+import time
+
+import h5py
+import numpy as np
+
+M64 = (1 << 64) - 1
+
+
+def splitmix64(state):
+    """Return (new_state, value); the same stream as the Rust harness."""
+    state = (state + 0x9E3779B97F4A7C15) & M64
+    z = state
+    z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & M64
+    z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & M64
+    return state, z ^ (z >> 31)
+
+
+def value(k, i):
+    """Element i (row-major) of dataset k, exactly as concurrent_read writes it."""
+    _, noise = splitmix64(i ^ (k << 40))
+    return np.float32((((i >> 6) % 16384) + k) + (noise & 0xFF) / 256.0)
+
+
+def slab_offsets(seed, count, rows, cols, slab):
+    s = seed
+    out = []
+    for _ in range(count):
+        s, r = splitmix64(s)
+        s, c = splitmix64(s)
+        out.append((r % (rows - slab + 1), c % (cols - slab + 1)))
+    return out
+
+
+def now():
+    return time.clock_gettime(time.CLOCK_MONOTONIC)
+
+
+def work(f, mode, t, threads, m, slabs, slab, verify):
+    """Worker t's share of one repetition on an open h5py.File."""
+    n = m["rows"] * m["cols"]
+    if mode == "distinct":
+        for k in range(t, m["datasets"], threads):
+            got = f[f"d{k:02d}"][...]
+            assert got.size == n
+            if verify:
+                flat = got.reshape(-1)
+                for i in (0, n // 3, n - 1):
+                    assert flat[i] == value(k, i), f"d{k:02d}[{i}]"
+    else:
+        ds = f["d00"]
+        cols = m["cols"]
+        for r, c in slabs[t::threads]:
+            got = ds[r : r + slab, c : c + slab]
+            assert got.shape == (slab, slab)
+            if verify:
+                assert got[0, 0] == value(0, r * cols + c)
+                last = (r + slab - 1) * cols + c + slab - 1
+                assert got[-1, -1] == value(0, last)
+
+
+# ----- process workers ------------------------------------------------------
+
+_barrier = None
+
+
+def _init(barrier):
+    global _barrier
+    _barrier = barrier
+
+
+def _proc_task(task):
+    path, mode, t, threads, m, slabs, slab = task
+    _barrier.wait()
+    start = now()
+    with h5py.File(path, "r") as f:
+        work(f, mode, t, threads, m, slabs, slab, False)
+    return start, now()
+
+
+def _noop(_):
+    return os.getpid()
+
+
+def run_threads(path, mode, threads, m, slabs, slab):
+    spans = [None] * threads
+    barrier = threading.Barrier(threads)
+    with h5py.File(path, "r") as f:
+
+        def body(t):
+            barrier.wait()
+            start = now()
+            work(f, mode, t, threads, m, slabs, slab, False)
+            spans[t] = (start, now())
+
+        ts = [threading.Thread(target=body, args=(t,)) for t in range(threads)]
+        for th in ts:
+            th.start()
+        for th in ts:
+            th.join()
+    return max(e for _, e in spans) - min(s for s, _ in spans)
+
+
+def run_processes(pool, path, mode, threads, m, slabs, slab):
+    tasks = [(path, mode, t, threads, m, slabs, slab) for t in range(threads)]
+    # One task per worker: each blocks in the barrier until all T have
+    # started, so no worker can take a second task.
+    spans = pool.map(_proc_task, tasks, chunksize=1)
+    return max(e for _, e in spans) - min(s for s, _ in spans)
+
+
+def warm(path):
+    with open(path, "rb") as fh:
+        while fh.read(1 << 24):
+            pass
+
+
+def evict(path):
+    fd = os.open(path, os.O_RDONLY)
+    try:
+        os.posix_fadvise(fd, 0, 0, os.POSIX_FADV_DONTNEED)
+    finally:
+        os.close(fd)
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    ap.add_argument("--dir", default="concurrent-read-data")
+    ap.add_argument("--executor", choices=["threads", "processes"], default="threads")
+    ap.add_argument("--threads", default="1,2,4,8,16")
+    ap.add_argument("--reps", type=int, default=3)
+    ap.add_argument("--slab", type=int, default=256)
+    ap.add_argument("--slabs", type=int, default=1024)
+    ap.add_argument("--seed", type=int, default=42)
+    ap.add_argument("--cold", action="store_true")
+    ap.add_argument("--modes", default="distinct,same")
+    ap.add_argument("--layouts", default="deflate,contiguous")
+    ap.add_argument("--json")
+    a = ap.parse_args()
+
+    # The Rust harness pins this value (splitmix64_reference).
+    assert splitmix64(42)[1] == 0xBDD732262FEB6E95, "splitmix64 port is wrong"
+
+    try:
+        with open(os.path.join(a.dir, "manifest.json")) as fh:
+            m = json.load(fh)
+    except FileNotFoundError:
+        sys.exit(f"{a.dir}/manifest.json not found: generate the files with "
+                 "`cargo run --release -p clawhdf5-bench --bin concurrent_read -- --dir ...` first")
+    threads_list = [int(x) for x in a.threads.split(",")]
+    modes = a.modes.split(",")
+    layouts = a.layouts.split(",")
+    if a.slab < 1 or a.slab > min(m["rows"], m["cols"]):
+        sys.exit(f"--slab must be 1..={min(m['rows'], m['cols'])}")
+    files = dict(m["files"])
+    slabs = slab_offsets(a.seed, a.slabs, m["rows"], m["cols"], a.slab)
+    dataset_bytes = m["rows"] * m["cols"] * 4
+    tool = f"h5py-{a.executor}"
+
+    ctx = mp.get_context("spawn")  # never fork a process holding HDF5 state
+    pools = {}
+    if a.executor == "processes":
+        for t in threads_list:
+            pool = ctx.Pool(t, initializer=_init, initargs=(ctx.Barrier(t),))
+            pool.map(_noop, range(t))  # start the workers outside the timing
+            pools[t] = pool
+
+    rows = []
+    print("| layout | mode | threads | MB/s | efficiency | median s |")
+    print("|---|---|---:|---:|---:|---:|")
+    try:
+        for layout in layouts:
+            path = os.path.join(a.dir, files[layout])
+            if not a.cold:
+                warm(path)
+            for mode in modes:
+                with h5py.File(path, "r") as f:  # untimed, checked pass
+                    work(f, mode, 0, 1, m, slabs, a.slab, True)
+                nbytes = (dataset_bytes * m["datasets"] if mode == "distinct"
+                          else a.slab * a.slab * 4 * a.slabs)
+                base = None
+                for t in threads_list:
+                    times = []
+                    for _ in range(a.reps):
+                        if a.cold:
+                            evict(path)
+                        if a.executor == "threads":
+                            times.append(run_threads(path, mode, t, m, slabs, a.slab))
+                        else:
+                            times.append(run_processes(pools[t], path, mode, t, m, slabs, a.slab))
+                    med = sorted(times)[len(times) // 2]
+                    mb_s = nbytes / (1 << 20) / med
+                    if t == 1:
+                        base = mb_s
+                    eff = mb_s / (t * base) if base else None
+                    print(f"| {layout} | {mode} | {t} | {mb_s:.0f} | "
+                          f"{'-' if eff is None else f'{eff:.2f}'} | {med:.4f} |")
+                    rows.append({
+                        "layout": layout, "mode": mode, "threads": t, "bytes": nbytes,
+                        "times_s": times, "median_s": med, "mb_s": mb_s, "efficiency": eff,
+                    })
+    finally:
+        for pool in pools.values():
+            pool.terminate()
+
+    if a.json:
+        doc = {
+            "tool": tool,
+            "version": h5py.__version__,
+            "hdf5_version": h5py.version.hdf5_version,
+            "python": platform.python_version(),
+            "host": socket.gethostname(),
+            "cpus": os.cpu_count(),
+            "unix_time": int(time.time()),
+            "cache": ("cold (posix_fadvise DONTNEED before each repetition)"
+                      if a.cold else "warm"),
+            "decode_threads": 1,
+            "params": {
+                "datasets": m["datasets"], "rows": m["rows"], "cols": m["cols"],
+                "chunk": m["chunk"], "deflate_level": m["deflate_level"],
+                "mib": dataset_bytes // (1 << 20), "slab": a.slab, "slabs": a.slabs,
+                "seed": a.seed, "reps": a.reps, "dir": a.dir,
+            },
+            "results": rows,
+        }
+        with open(a.json, "w") as fh:
+            json.dump(doc, fh, indent=2)
+
+
+if __name__ == "__main__":
+    main()

crates/clawhdf5-bench/src/bin/concurrent_read.rs

@@ -0,0 +1,523 @@
+//! Concurrent-read harness: how does decoded read throughput scale with the
+//! number of threads reading one open file?
+//!
+//! libhdf5 (threadsafe build) serialises every API call under one global
+//! mutex, and h5py holds it too, so threads cannot decode in parallel there.
+//! A clawhdf5 [`File`] is `Send + Sync`; this harness measures what that buys.
+//! `crates/clawhdf5-bench/scripts/concurrent_read_h5py.py` runs the same
+//! workload on the same files with h5py (threads, and processes), and
+//! `compare_concurrent_read.py` tabulates the JSON both write.
+//!
+//! Files (generated on first use, reused while `manifest.json` matches):
+//!
+//! * `<dir>/deflate.h5`: `--datasets` datasets `d00`, `d01`, ... of `f32`,
+//!   `--mib` MiB decoded each, shape `[mib * 256, 1024]`, chunks `256 x 256`,
+//!   deflate level 4.
+//! * `<dir>/contiguous.h5`: the same datasets, contiguous.
+//!
+//! Modes, for each layout and each thread count `T` (strong scaling: the total
+//! work per repetition is fixed, split among the threads):
+//!
+//! * `distinct`: every dataset is read in full once; thread `t` reads datasets
+//!   `t, t + T, t + 2T, ...`.
+//! * `same`: all threads read `d00`, `--slabs` random `--slab` x `--slab`
+//!   hyperslabs in total (slab `j` goes to thread `j % T`). The offsets come
+//!   from a splitmix64 stream seeded with `--seed`, identical in the h5py
+//!   script.
+//!
+//! One `File` per layout per repetition is shared by all threads (opened
+//! fresh each repetition, so no chunk cache carries over). Page cache:
+//! `warm` (default) reads every file once before timing; `--cold` evicts the
+//! files from the page cache with `posix_fadvise(POSIX_FADV_DONTNEED)` before
+//! every repetition (no root needed; it only evicts clean, unmapped pages, so
+//! it is best effort — the JSON says which was used).
+//!
+//! Decode inside one read is itself parallel when clawhdf5-format's `parallel`
+//! feature is on (it is in this binary, via clawhdf5-agent). `--decode-threads
+//! N` sizes that rayon pool; `--decode-threads 1` measures the API's own
+//! thread scaling, comparable with h5py where each call decodes on the
+//! calling thread.
+//!
+//! ```text
+//! cargo run --release -p clawhdf5-bench --bin concurrent_read -- \
+//!     --dir /data/concurrent-read --json clawhdf5.json
+//! cargo run --release -p clawhdf5-bench --bin concurrent_read -- \
+//!     --dir /tmp/cr --datasets 4 --mib 1 --threads 1,2 --slabs 16 --reps 1   # smoke
+//! ```
+
+use std::path::{Path, PathBuf};
+use std::sync::Barrier;
+use std::time::Instant;
+
+use clawhdf5::{File, FileBuilder, Selection};
+use serde::{Deserialize, Serialize};
+
+const COLS: u64 = 1024;
+const ROWS_PER_MIB: u64 = 256; // 256 rows x 1024 cols x 4 bytes = 1 MiB
+const CHUNK: u64 = 256;
+const DEFLATE_LEVEL: u32 = 4;
+const LAYOUTS: [&str; 2] = ["deflate", "contiguous"];
+const MANIFEST_VERSION: u32 = 1;
+
+/// splitmix64 — shared with the h5py script, which must produce the same
+/// stream (both the data and the hyperslab offsets depend on it).
+fn splitmix64(state: &mut u64) -> u64 {
+    *state = state.wrapping_add(0x9E37_79B9_7F4A_7C15);
+    let mut z = *state;
+    z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+    z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+    z ^ (z >> 31)
+}
+
+/// Element `i` (row-major) of dataset `k`: a slowly varying integer part plus
+/// 8 bits of noise, so deflate has real work to do (about 3.1x) and every value
+/// is exact in `f32` (< 2^15 with 8 fraction bits), which lets both harnesses
+/// check what they read against this formula.
+fn value(k: u64, i: u64) -> f32 {
+    let mut s = i ^ (k << 40);
+    let noise = splitmix64(&mut s) & 0xff;
+    (((i >> 6) % 16384) + k) as f32 + noise as f32 / 256.0
+}
+
+#[derive(Serialize, Deserialize, PartialEq, Debug, Clone)]
+struct Manifest {
+    version: u32,
+    datasets: u64,
+    rows: u64,
+    cols: u64,
+    chunk: [u64; 2],
+    deflate_level: u32,
+    files: Vec<(String, String)>, // (layout, file name)
+    writer: String,
+}
+
+fn manifest_for(datasets: u64, mib: u64) -> Manifest {
+    Manifest {
+        version: MANIFEST_VERSION,
+        datasets,
+        rows: mib * ROWS_PER_MIB,
+        cols: COLS,
+        chunk: [CHUNK, CHUNK],
+        deflate_level: DEFLATE_LEVEL,
+        files: LAYOUTS
+            .iter()
+            .map(|l| (l.to_string(), format!("{l}.h5")))
+            .collect(),
+        writer: format!("clawhdf5 {}", env!("CARGO_PKG_VERSION")),
+    }
+}
+
+fn dataset_values(k: u64, n: u64) -> Vec<f32> {
+    (0..n).map(|i| value(k, i)).collect()
+}
+
+/// Write the files unless `dir` already holds ones matching `want`.
+fn ensure_files(dir: &Path, want: &Manifest) -> std::io::Result<bool> {
+    let manifest_path = dir.join("manifest.json");
+    if let Ok(text) = std::fs::read_to_string(&manifest_path)
+        && let Ok(have) = serde_json::from_str::<Manifest>(&text)
+        && have.version == want.version
+        && have.datasets == want.datasets
+        && have.rows == want.rows
+        && have.cols == want.cols
+        && have.chunk == want.chunk
+        && have.deflate_level == want.deflate_level
+        && have.files == want.files
+        && want.files.iter().all(|(_, f)| dir.join(f).exists())
+    {
+        return Ok(false);
+    }
+    std::fs::create_dir_all(dir)?;
+    // A stale manifest must not survive a half-written regeneration.
+    let _ = std::fs::remove_file(&manifest_path);
+    let n = want.rows * want.cols;
+    for (layout, file) in &want.files {
+        // One layout at a time keeps the peak memory to about twice one
+        // file's decoded size.
+        let mut b = FileBuilder::new();
+        for k in 0..want.datasets {
+            let ds = b.create_dataset(&format!("d{k:02}"));
+            ds.with_f32_data(&dataset_values(k, n))
+                .with_shape(&[want.rows, want.cols]);
+            if layout == "deflate" {
+                ds.with_chunks(&[CHUNK.min(want.rows), CHUNK])
+                    .with_deflate(DEFLATE_LEVEL);
+            }
+        }
+        b.write(dir.join(file)).map_err(std::io::Error::other)?;
+    }
+    std::fs::write(
+        &manifest_path,
+        serde_json::to_string_pretty(want).map_err(std::io::Error::other)?,
+    )?;
+    Ok(true)
+}
+
+fn slab_offsets(seed: u64, count: usize, rows: u64, cols: u64, slab: u64) -> Vec<(u64, u64)> {
+    let mut s = seed;
+    (0..count)
+        .map(|_| {
+            let r = splitmix64(&mut s) % (rows - slab + 1);
+            let c = splitmix64(&mut s) % (cols - slab + 1);
+            (r, c)
+        })
+        .collect()
+}
+
+/// Warm the page cache by reading every byte of `path`.
+fn warm(path: &Path) -> std::io::Result<()> {
+    let mut f = std::fs::File::open(path)?;
+    std::io::copy(&mut f, &mut std::io::sink())?;
+    Ok(())
+}
+
+/// Ask the kernel to drop `path`'s pages from the page cache.
+fn evict(path: &Path) -> std::io::Result<()> {
+    use std::os::fd::AsRawFd;
+    let f = std::fs::File::open(path)?;
+    // SAFETY: plain syscall on a valid, open file descriptor.
+    let rc = unsafe { libc::posix_fadvise(f.as_raw_fd(), 0, 0, libc::POSIX_FADV_DONTNEED) };
+    if rc != 0 {
+        return Err(std::io::Error::from_raw_os_error(rc));
+    }
+    Ok(())
+}
+
+#[derive(Serialize)]
+struct Row {
+    layout: String,
+    mode: String,
+    threads: usize,
+    /// Decoded (selected) bytes read per repetition.
+    bytes: u64,
+    times_s: Vec<f64>,
+    median_s: f64,
+    mb_s: f64,
+    /// `mb_s / (threads * mb_s at threads = 1)`; null without a 1-thread row.
+    efficiency: Option<f64>,
+}
+
+struct Args {
+    dir: PathBuf,
+    datasets: u64,
+    mib: u64,
+    threads: Vec<usize>,
+    reps: usize,
+    slab: u64,
+    slabs: usize,
+    seed: u64,
+    cold: bool,
+    decode_threads: usize,
+    modes: Vec<String>,
+    layouts: Vec<String>,
+    json: Option<PathBuf>,
+}
+
+const USAGE: &str = "\
+usage: concurrent_read [--dir DIR] [--datasets N] [--mib N] [--threads 1,2,4,8,16]
+                       [--reps N] [--slab N] [--slabs N] [--seed N] [--cold]
+                       [--decode-threads N] [--modes distinct,same]
+                       [--layouts deflate,contiguous] [--json FILE]";
+
+fn parse_list<T: std::str::FromStr>(s: &str) -> Result<Vec<T>, String> {
+    s.split(',')
+        .map(|x| x.trim().parse().map_err(|_| format!("bad list item {x:?}")))
+        .collect()
+}
+
+fn parse_args() -> Result<Args, String> {
+    let mut a = Args {
+        dir: PathBuf::from("concurrent-read-data"),
+        datasets: 64,
+        mib: 64,
+        threads: vec![1, 2, 4, 8, 16],
+        reps: 3,
+        slab: 256,
+        slabs: 1024,
+        seed: 42,
+        cold: false,
+        decode_threads: 0,
+        modes: vec!["distinct".into(), "same".into()],
+        layouts: LAYOUTS.iter().map(|s| s.to_string()).collect(),
+        json: None,
+    };
+    let mut it = std::env::args().skip(1);
+    while let Some(flag) = it.next() {
+        if flag == "--cold" {
+            a.cold = true;
+            continue;
+        }
+        if flag == "-h" || flag == "--help" {
+            return Err(USAGE.into());
+        }
+        let v = it.next().ok_or(format!("{flag} needs a value\n{USAGE}"))?;
+        let num = |v: &str| {
+            v.parse::<u64>()
+                .map_err(|_| format!("{flag}: bad number {v:?}"))
+        };
+        match flag.as_str() {
+            "--dir" => a.dir = v.into(),
+            "--datasets" => a.datasets = num(&v)?,
+            "--mib" => a.mib = num(&v)?,
+            "--threads" => a.threads = parse_list(&v)?,
+            "--reps" => a.reps = num(&v)? as usize,
+            "--slab" => a.slab = num(&v)?,
+            "--slabs" => a.slabs = num(&v)? as usize,
+            "--seed" => a.seed = num(&v)?,
+            "--decode-threads" => a.decode_threads = num(&v)? as usize,
+            "--modes" => a.modes = parse_list(&v)?,
+            "--layouts" => a.layouts = parse_list(&v)?,
+            "--json" => a.json = Some(v.into()),
+            _ => return Err(format!("unknown flag {flag}\n{USAGE}")),
+        }
+    }
+    if a.datasets == 0 || a.datasets > 100 {
+        return Err("--datasets must be 1..=100".into());
+    }
+    if a.mib == 0 || a.reps == 0 || a.slabs == 0 || a.threads.contains(&0) {
+        return Err("--mib, --reps, --slabs and every --threads value must be > 0".into());
+    }
+    if a.slab == 0 || a.slab > COLS || a.slab > a.mib * ROWS_PER_MIB {
+        return Err(format!(
+            "--slab must be 1..={}",
+            COLS.min(a.mib * ROWS_PER_MIB)
+        ));
+    }
+    for m in &a.modes {
+        if m != "distinct" && m != "same" {
+            return Err(format!("unknown mode {m:?}"));
+        }
+    }
+    for l in &a.layouts {
+        if !LAYOUTS.contains(&l.as_str()) {
+            return Err(format!("unknown layout {l:?}"));
+        }
+    }
+    Ok(a)
+}
+
+/// One timed repetition: `T` threads on one shared `File`. Returns seconds.
+fn run_once(
+    path: &Path,
+    mode: &str,
+    threads: usize,
+    m: &Manifest,
+    slabs: &[(u64, u64)],
+    slab: u64,
+    verify: bool,
+) -> f64 {
+    let file = File::open(path).expect("open");
+    let barrier = Barrier::new(threads + 1); // + the spawning thread
+    let n = m.rows * m.cols;
+    // Each thread times itself from the barrier; the repetition spans the
+    // earliest start to the latest finish (timing on the spawning thread
+    // instead undercounts whenever it is scheduled after the workers ran).
+    let spans: Vec<(Instant, Instant)> = std::thread::scope(|s| {
+        let handles: Vec<_> = (0..threads)
+            .map(|t| {
+                let (file, barrier) = (&file, &barrier);
+                s.spawn(move || {
+                    barrier.wait();
+                    let start = Instant::now();
+                    match mode {
+                        "distinct" => {
+                            for k in (t as u64..m.datasets).step_by(threads) {
+                                let got = file.dataset(&format!("d{k:02}")).unwrap().read_f32();
+                                let got = got.unwrap();
+                                assert_eq!(got.len() as u64, n);
+                                if verify {
+                                    for i in [0, n / 3, n - 1] {
+                                        assert_eq!(got[i as usize], value(k, i), "d{k:02}[{i}]");
+                                    }
+                                }
+                                std::hint::black_box(got);
+                            }
+                        }
+                        _ => {
+                            let ds = file.dataset("d00").unwrap();
+                            for &(r, c) in slabs.iter().skip(t).step_by(threads) {
+                                let sel = Selection::Hyperslab {
+                                    start: vec![r, c],
+                                    stride: vec![1, 1],
+                                    count: vec![slab, slab],
+                                    block: vec![1, 1],
+                                };
+                                let got = ds.read_f32_selection(&sel).unwrap();
+                                assert_eq!(got.len() as u64, slab * slab);
+                                if verify {
+                                    let last = (r + slab - 1) * m.cols + c + slab - 1;
+                                    assert_eq!(got[0], value(0, r * m.cols + c));
+                                    assert_eq!(*got.last().unwrap(), value(0, last));
+                                }
+                                std::hint::black_box(got);
+                            }
+                        }
+                    }
+                    (start, Instant::now())
+                })
+            })
+            .collect();
+        barrier.wait();
+        handles.into_iter().map(|h| h.join().unwrap()).collect()
+    });
+    let start = spans.iter().map(|s| s.0).min().unwrap();
+    let end = spans.iter().map(|s| s.1).max().unwrap();
+    (end - start).as_secs_f64()
+}
+
+fn median(v: &[f64]) -> f64 {
+    let mut s = v.to_vec();
+    s.sort_by(f64::total_cmp);
+    s[s.len() / 2]
+}
+
+fn hostname() -> String {
+    std::fs::read_to_string("/proc/sys/kernel/hostname")
+        .map(|s| s.trim().to_string())
+        .unwrap_or_else(|_| "unknown".into())
+}
+
+fn main() {
+    let args = match parse_args() {
+        Ok(a) => a,
+        Err(e) => {
+            eprintln!("{e}");
+            std::process::exit(2);
+        }
+    };
+    if cfg!(debug_assertions) {
+        eprintln!("warning: debug build — numbers are meaningless. Use --release.");
+    }
+    if args.decode_threads > 0 {
+        rayon::ThreadPoolBuilder::new()
+            .num_threads(args.decode_threads)
+            .build_global()
+            .expect("configure rayon pool");
+    }
+
+    let manifest = manifest_for(args.datasets, args.mib);
+    let t = Instant::now();
+    match ensure_files(&args.dir, &manifest) {
+        Ok(true) => eprintln!(
+            "generated {} in {:.1} s",
+            args.dir.display(),
+            t.elapsed().as_secs_f64()
+        ),
+        Ok(false) => eprintln!("reusing {}", args.dir.display()),
+        Err(e) => {
+            eprintln!("cannot write test files in {}: {e}", args.dir.display());
+            std::process::exit(1);
+        }
+    }
+    let path_of = |layout: &str| args.dir.join(format!("{layout}.h5"));
+    let slabs = slab_offsets(
+        args.seed,
+        args.slabs,
+        manifest.rows,
+        manifest.cols,
+        args.slab,
+    );
+    let dataset_bytes = manifest.rows * manifest.cols * 4;
+
+    let mut rows: Vec<Row> = Vec::new();
+    println!("| layout | mode | threads | MB/s | efficiency | median s |");
+    println!("|---|---|---:|---:|---:|---:|");
+    for layout in &args.layouts {
+        let path = path_of(layout);
+        // Untimed pass: page cache warm (unless --cold), results checked.
+        if !args.cold {
+            warm(&path).expect("warm page cache");
+        }
+        for mode in &args.modes {
+            run_once(&path, mode, 1, &manifest, &slabs, args.slab, true);
+            let bytes = match mode.as_str() {
+                "distinct" => dataset_bytes * manifest.datasets,
+                _ => args.slab * args.slab * 4 * args.slabs as u64,
+            };
+            let mut base: Option<f64> = None;
+            for &threads in &args.threads {
+                let times: Vec<f64> = (0..args.reps)
+                    .map(|_| {
+                        if args.cold {
+                            evict(&path).expect("posix_fadvise");
+                        }
+                        run_once(&path, mode, threads, &manifest, &slabs, args.slab, false)
+                    })
+                    .collect();
+                let med = median(&times);
+                let mb_s = bytes as f64 / (1 << 20) as f64 / med;
+                if threads == 1 {
+                    base = Some(mb_s);
+                }
+                let efficiency = base.map(|b| mb_s / (threads as f64 * b));
+                println!(
+                    "| {layout} | {mode} | {threads} | {mb_s:.0} | {} | {med:.4} |",
+                    efficiency.map_or("-".into(), |e| format!("{e:.2}"))
+                );
+                rows.push(Row {
+                    layout: layout.clone(),
+                    mode: mode.clone(),
+                    threads,
+                    bytes,
+                    times_s: times,
+                    median_s: med,
+                    mb_s,
+                    efficiency,
+                });
+            }
+        }
+    }
+
+    if let Some(out) = &args.json {
+        let doc = serde_json::json!({
+            "tool": "clawhdf5",
+            "version": env!("CARGO_PKG_VERSION"),
+            "host": hostname(),
+            "cpus": std::thread::available_parallelism().map_or(0, |n| n.get()),
+            "unix_time": std::time::SystemTime::now()
+                .duration_since(std::time::UNIX_EPOCH)
+                .map_or(0, |d| d.as_secs()),
+            "cache": if args.cold { "cold (posix_fadvise DONTNEED before each repetition)" } else { "warm" },
+            "decode_threads": rayon::current_num_threads(),
+            "params": {
+                "datasets": manifest.datasets,
+                "mib": args.mib,
+                "rows": manifest.rows,
+                "cols": manifest.cols,
+                "chunk": manifest.chunk,
+                "deflate_level": manifest.deflate_level,
+                "slab": args.slab,
+                "slabs": args.slabs,
+                "seed": args.seed,
+                "reps": args.reps,
+                "dir": args.dir,
+            },
+            "results": rows,
+        });
+        std::fs::write(out, serde_json::to_string_pretty(&doc).unwrap()).expect("write json");
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn values_are_exact_in_f32() {
+        for k in [0, 7, 63] {
+            for i in [0u64, 1, 4095, 1 << 20, (1 << 24) - 1] {
+                let v = value(k, i);
+                assert_eq!(v, (v as f64) as f32);
+                assert!(v < 32768.0);
+                assert_eq!((v * 256.0).fract(), 0.0);
+            }
+        }
+    }
+
+    /// The h5py script hard-codes this vector to check its splitmix64 port.
+    #[test]
+    fn splitmix64_reference() {
+        let mut s = 42;
+        assert_eq!(splitmix64(&mut s), 0xBDD7_3226_2FEB_6E95);
+    }
+}

crates/clawhdf5-bench/tests/concurrent_read_smoke.rs

@@ -0,0 +1,148 @@
+//! Keeps the concurrent-read harnesses working: runs `concurrent_read`, the
+//! h5py script (threads and processes) and the comparison script end to end
+//! on tiny files. h5py reading the files also checks, element by element at
+//! spot positions, that both harnesses generate the same data and slabs.
+//!
+//! The h5py half is skipped when python3 with h5py is unavailable, unless
+//! `CLAWHDF5_REQUIRE_INTEROP=1`; `CLAWHDF5_PYTHON` picks the interpreter.
+
+use std::path::{Path, PathBuf};
+use std::process::Command;
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+fn python_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+fn scripts() -> PathBuf {
+    Path::new(env!("CARGO_MANIFEST_DIR")).join("scripts")
+}
+
+fn run(cmd: &mut Command) -> String {
+    let out = cmd.output().expect("spawn");
+    assert!(
+        out.status.success(),
+        "{cmd:?} failed\nSTDOUT:\n{}\nSTDERR:\n{}",
+        String::from_utf8_lossy(&out.stdout),
+        String::from_utf8_lossy(&out.stderr)
+    );
+    String::from_utf8_lossy(&out.stdout).into_owned()
+}
+
+const SMALL: [&str; 8] = [
+    "--threads",
+    "1,2",
+    "--slabs",
+    "8",
+    "--reps",
+    "1",
+    "--slab",
+    "64",
+];
+
+fn results(path: &Path) -> serde_json::Value {
+    serde_json::from_str(&std::fs::read_to_string(path).unwrap()).unwrap()
+}
+
+#[test]
+fn harnesses_run_end_to_end_on_tiny_files() {
+    let dir = tempfile::TempDir::new().unwrap();
+    let data = dir.path().join("data");
+    let claw = dir.path().join("claw.json");
+
+    let bin = env!("CARGO_BIN_EXE_concurrent_read");
+    run(Command::new(bin)
+        .arg("--dir")
+        .arg(&data)
+        .args(["--datasets", "3", "--mib", "1"])
+        .args(SMALL)
+        .arg("--json")
+        .arg(&claw));
+    // Second run reuses the files (and exercises --cold).
+    let out = Command::new(bin)
+        .arg("--dir")
+        .arg(&data)
+        .args(["--datasets", "3", "--mib", "1", "--cold"])
+        .args(SMALL)
+        .output()
+        .unwrap();
+    assert!(out.status.success());
+    assert!(String::from_utf8_lossy(&out.stderr).contains("reusing"));
+
+    let doc = results(&claw);
+    assert_eq!(doc["tool"], "clawhdf5");
+    // 2 layouts x 2 modes x 2 thread counts.
+    assert_eq!(doc["results"].as_array().unwrap().len(), 8);
+    for r in doc["results"].as_array().unwrap() {
+        assert!(r["mb_s"].as_f64().unwrap() > 0.0, "{r}");
+    }
+
+    if !python_available() {
+        assert!(
+            !interop_required(),
+            "CLAWHDF5_REQUIRE_INTEROP=1 but {} has no h5py",
+            python()
+        );
+        eprintln!("skipping the h5py half: no h5py in {}", python());
+        return;
+    }
+    let mut jsons = vec![claw];
+    for executor in ["threads", "processes"] {
+        let out = dir.path().join(format!("h5py-{executor}.json"));
+        run(Command::new(python())
+            .arg(scripts().join("concurrent_read_h5py.py"))
+            .arg("--dir")
+            .arg(&data)
+            .args(["--executor", executor])
+            .args(SMALL)
+            .arg("--json")
+            .arg(&out));
+        let doc = results(&out);
+        assert_eq!(doc["tool"], format!("h5py-{executor}"));
+        assert_eq!(doc["results"].as_array().unwrap().len(), 8);
+        jsons.push(out);
+    }
+    let table = run(Command::new(python())
+        .arg(scripts().join("compare_concurrent_read.py"))
+        .args(&jsons));
+    assert!(table.contains("| deflate | same | 2 |"), "{table}");
+    assert!(table.contains("clawhdf5 / h5py-processes"), "{table}");
+
+    // A different workload must not be compared.
+    let other = dir.path().join("other.json");
+    run(Command::new(python())
+        .arg(scripts().join("concurrent_read_h5py.py"))
+        .arg("--dir")
+        .arg(&data)
+        .args([
+            "--threads",
+            "1",
+            "--slabs",
+            "4",
+            "--reps",
+            "1",
+            "--slab",
+            "64",
+        ])
+        .arg("--json")
+        .arg(&other));
+    let out = Command::new(python())
+        .arg(scripts().join("compare_concurrent_read.py"))
+        .arg(&jsons[0])
+        .arg(&other)
+        .output()
+        .unwrap();
+    assert!(!out.status.success());
+    assert!(String::from_utf8_lossy(&out.stderr).contains("slabs"));
+}

crates/clawhdf5-format/Cargo.toml

@@ -22,6 +22,17 @@ zstd = { version = "0.13", optional = true }
 blake3 = { version = "1", optional = true }
 libaec-sys = { path = "../libaec-sys", version = "0.1", optional = true }
 pco = { version = "1.0", optional = true }
+# Pure-Rust Zstandard, for the plugin filters that embed zstd (bitshuffle,
+# blosc). The `zstd` feature (filter 32015) links libzstd instead.
+ruzstd = { version = "0.9", optional = true }
+# bzip2 with its default backend, libbz2-rs-sys: a pure-Rust port of
+# libbzip2 (no C is compiled, despite the -sys name).
+bzip2 = { version = "0.6", optional = true }
+snap = { version = "1", optional = true }
+
+[target.'cfg(target_os = "linux")'.dependencies]
+# madvise(MADV_HUGEPAGE) for large read buffers (see src/bulk_alloc.rs).
+libc = { version = "0.2", default-features = false }
 
 [dev-dependencies]
 half = { workspace = true }
@@ -37,7 +48,7 @@ harness = false
 # Deflate backend: `zlib-rs` (pure Rust) by default. `fast-deflate` selects
 # zlib-ng instead (C, built with cmake); flate2 prefers a C zlib whenever one
 # is enabled, so turning it on anywhere in the build overrides the default.
-default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress"]
+default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress", "lzf"]
 std = []
 checksum = []
 deflate = ["flate2"]
@@ -56,6 +67,25 @@ zstd = ["dep:zstd"]
 blake3_hash = ["blake3"]
 szip = ["libaec-sys"]
 pcodec = ["dep:pco"]
+# Plugin filters, pure Rust. LZF (32000) is h5py's built-in compression; it
+# has no dependencies, so it is on by default.
+lzf = []
+# Bitshuffle (32008), with its LZ4 and Zstandard modes.
+bitshuffle = ["lz4_flex", "ruzstd"]
+# bzip2 (307).
+bzip2 = ["dep:bzip2", "std"]
+# Blosc 1 (32001) with its BloscLZ, LZ4, Snappy, Zlib and Zstandard codecs.
+blosc = ["lz4_flex", "ruzstd", "snap", "deflate", "std"]
+# Blosc2 (32026), read-only: frames, B2ND arrays, and the Blosc codecs above.
+blosc2 = ["blosc"]
+# ZFP (32013, H5Z-ZFP), read-only: every mode, for int32, int64, float and
+# double fields of 1 to 4 dimensions.
+zfp = []
+# Every plugin filter above.
+plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2", "zfp"]
+# Test instrumentation: per-thread counts of heap objects read (see
+# `lookup_stats`), so tests can bound the cost of a name lookup.
+lookup-stats = ["std"]
 
 [[bench]]
 name = "parallel_decompress_bench"

crates/clawhdf5-format/src/addr.rs

@@ -0,0 +1,121 @@
+//! File address and length → in-memory index conversion.
+//!
+//! HDF5 addresses and lengths are 64-bit; the file is parsed through a
+//! `&[u8]` indexed by `usize`. On a 64-bit target every `u64` fits, but on a
+//! 32-bit one (`wasm32`, `i686`, `thumbv7em`) an address past `usize::MAX`
+//! used to be truncated by an `as usize` cast — silently pointing at another
+//! part of the file — or to panic. [`to_usize`] is the one conversion the
+//! parsers use instead: such an address is a clean
+//! [`FormatError::Overflow`]. It cannot be inside the data anyway: no slice
+//! is longer than `isize::MAX` bytes.
+
+#[cfg(not(feature = "std"))]
+use alloc::format;
+
+use crate::error::FormatError;
+
+/// A file address, offset or length from the file as a `usize` index.
+///
+/// Fails with [`FormatError::Overflow`] when the value does not fit this
+/// platform's `usize` (only possible on targets narrower than 64 bits).
+#[inline]
+pub fn to_usize(value: u64) -> Result<usize, FormatError> {
+    to_index::<usize>(value)
+}
+
+/// A file address for a [`crate::storage::Storage`] read, checked as
+/// [`to_usize`] checks it: the parsers read through 64-bit offsets, but an
+/// address that could not index an in-memory file on this platform is the
+/// same [`FormatError::Overflow`] the slice parsers gave for it.
+#[inline]
+pub fn checked_addr(value: u64) -> Result<u64, FormatError> {
+    to_usize(value).map(|_| value)
+}
+
+/// [`to_usize`] for an index type of any width. `usize` is 64 bits wide on
+/// the hosts CI tests on, where the error path cannot be reached through
+/// `usize`; tests run the same code with `u32` in its place, as on a 32-bit
+/// target.
+#[inline]
+fn to_index<T: TryFrom<u64>>(value: u64) -> Result<T, FormatError> {
+    T::try_from(value).map_err(|_| too_large(value))
+}
+
+/// A count or offset into an in-memory buffer (a codec's progress counter,
+/// a size the writer computed from data it holds) as a `usize`, saturating
+/// at `usize::MAX` instead of truncating.
+///
+/// For values that are bounded by the length of something in memory, so
+/// always fit; if one ever did not, a saturated index fails its bounds check
+/// or allocation instead of silently addressing the wrong bytes. A value
+/// read from the file uses [`to_usize`].
+#[inline]
+pub fn saturating_usize(value: u64) -> usize {
+    saturating_index(value, usize::MAX)
+}
+
+/// [`saturating_usize`] for an index type of any width, whose largest
+/// value is `max` (see [`to_index`]).
+#[inline]
+fn saturating_index<T: TryFrom<u64>>(value: u64, max: T) -> T {
+    T::try_from(value).unwrap_or(max)
+}
+
+#[cold]
+#[inline(never)]
+fn too_large(value: u64) -> FormatError {
+    FormatError::Overflow(format!(
+        "file address or length {value:#x} exceeds this platform's address space"
+    ))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn values_that_fit_convert_exactly() {
+        assert_eq!(to_usize(0), Ok(0));
+        assert_eq!(to_usize(0x1234), Ok(0x1234));
+        assert_eq!(to_usize(usize::MAX as u64), Ok(usize::MAX));
+    }
+
+    #[test]
+    fn saturating_conversion_never_wraps() {
+        assert_eq!(saturating_usize(0), 0);
+        assert_eq!(saturating_usize(0x1234), 0x1234);
+        assert_eq!(saturating_usize(usize::MAX as u64), usize::MAX);
+        // Past usize::MAX (32-bit targets) or at u64::MAX: saturates.
+        assert_eq!(saturating_usize(u64::MAX), usize::MAX);
+    }
+
+    #[test]
+    fn values_past_usize_max_are_an_error_not_truncated() {
+        // Reachable through `usize` only where it is narrower than u64 (no
+        // such target runs tests in CI), so the same conversion is run with
+        // u32 standing in for a 32-bit usize.
+        let max = u64::from(u32::MAX);
+        assert_eq!(to_index::<u32>(max), Ok(u32::MAX));
+        for past in [max + 1, max + 0x10, 0x1_0000_1234, u64::MAX] {
+            let err = to_index::<u32>(past).unwrap_err();
+            assert!(
+                matches!(err, FormatError::Overflow(_)),
+                "{past:#x}: {err:?}"
+            );
+        }
+        // Where an `as` cast would have wrapped to a small, valid-looking
+        // index, it is not returned.
+        assert_eq!(0x1_0000_1234_u64 as u32, 0x1234);
+        assert!(to_index::<u32>(0x1_0000_1234).is_err());
+
+        assert_eq!(saturating_index(max + 1, u32::MAX), u32::MAX);
+        assert_eq!(saturating_index(0x1_0000_1234, u32::MAX), u32::MAX);
+        assert_eq!(saturating_index(0x1234, u32::MAX), 0x1234);
+
+        // And through `usize` itself, whichever width it has here.
+        match (usize::MAX as u64).checked_add(1) {
+            Some(past) => assert!(matches!(to_usize(past), Err(FormatError::Overflow(_)))),
+            None => assert_eq!(to_usize(u64::MAX), Ok(usize::MAX)),
+        }
+    }
+}

crates/clawhdf5-format/src/attribute.rs

@@ -5,8 +5,10 @@ use alloc::{borrow::Cow, string::String, vec::Vec};
 #[cfg(feature = "std")]
 use std::borrow::Cow;
 
+use crate::addr::to_usize;
 use crate::attribute_info::AttributeInfoMessage;
-use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records_in, find_btree_v2_records_in};
+use crate::checksum::jenkins_lookup3;
 use crate::data_read;
 use crate::dataspace::Dataspace;
 use crate::datatype::Datatype;
@@ -15,6 +17,7 @@ use crate::fractal_heap::FractalHeapHeader;
 use crate::message_type::MessageType;
 use crate::object_header::ObjectHeader;
 use crate::shared_message;
+use crate::storage::Storage;
 use crate::vl_data;
 
 /// A parsed HDF5 attribute message.
@@ -50,7 +53,7 @@ impl AttributeMessage {
     ///
     /// `length_size` is needed for dataspace dimension parsing.
     pub fn parse(data: &[u8], length_size: u8) -> Result<AttributeMessage, FormatError> {
-        Self::parse_impl(data, length_size, None)
+        Self::parse_impl(data, length_size, None::<(&[u8], u8)>)
     }
 
     /// [`AttributeMessage::parse`] with access to the rest of the file, which
@@ -65,13 +68,24 @@ impl AttributeMessage {
         offset_size: u8,
         length_size: u8,
     ) -> Result<AttributeMessage, FormatError> {
-        Self::parse_impl(data, length_size, Some((file_data, offset_size)))
+        Self::parse_in_storage(data, file_data, offset_size, length_size)
     }
 
-    fn parse_impl(
+    /// [`AttributeMessage::parse_in_file`] with the file behind any
+    /// [`Storage`].
+    pub fn parse_in_storage<S: Storage + ?Sized>(
+        data: &[u8],
+        file: &S,
+        offset_size: u8,
+        length_size: u8,
+    ) -> Result<AttributeMessage, FormatError> {
+        Self::parse_impl(data, length_size, Some((file, offset_size)))
+    }
+
+    fn parse_impl<S: Storage + ?Sized>(
         data: &[u8],
         length_size: u8,
-        file: Option<(&[u8], u8)>,
+        file: Option<(&S, u8)>,
     ) -> Result<AttributeMessage, FormatError> {
         ensure_len(data, 0, 2)?;
         let version = data[0];
@@ -86,19 +100,19 @@ impl AttributeMessage {
 
     /// The bytes of an embedded datatype/dataspace message, following the
     /// shared-message reference when `shared` is set.
-    fn embedded_message<'a>(
+    fn embedded_message<'a, S: Storage + ?Sized>(
         bytes: &'a [u8],
         shared: bool,
         msg_type: MessageType,
         length_size: u8,
-        file: Option<(&[u8], u8)>,
+        file: Option<(&S, u8)>,
     ) -> Result<Cow<'a, [u8]>, FormatError> {
         if !shared {
             return Ok(Cow::Borrowed(bytes));
         }
         let (file_data, offset_size) = file.ok_or(FormatError::UnresolvedSharedMessage)?;
-        let shared_ref = shared_message::parse_shared_ref(bytes, offset_size)?;
+        let shared_ref = shared_message::parse_shared_ref_sized(bytes, offset_size, length_size)?;
-        shared_message::resolve_shared_message(
+        shared_message::resolve_shared_message_in(
             file_data,
             &shared_ref,
             msg_type,
@@ -143,10 +157,10 @@ impl AttributeMessage {
         })
     }
 
-    fn parse_v2(
+    fn parse_v2<S: Storage + ?Sized>(
         data: &[u8],
         length_size: u8,
-        file: Option<(&[u8], u8)>,
+        file: Option<(&S, u8)>,
     ) -> Result<AttributeMessage, FormatError> {
         // Flags: bit 0 = datatype is shared, bit 1 = dataspace is shared.
         let flags = data.get(1).copied().unwrap_or(0);
@@ -197,10 +211,10 @@ impl AttributeMessage {
         })
     }
 
-    fn parse_v3(
+    fn parse_v3<S: Storage + ?Sized>(
         data: &[u8],
         length_size: u8,
-        file: Option<(&[u8], u8)>,
+        file: Option<(&S, u8)>,
     ) -> Result<AttributeMessage, FormatError> {
         // Flags: bit 0 = datatype is shared, bit 1 = dataspace is shared.
         let flags = data.get(1).copied().unwrap_or(0);
@@ -322,9 +336,19 @@ impl AttributeMessage {
         file_data: &[u8],
         offset_size: u8,
         length_size: u8,
+    ) -> Result<Vec<String>, FormatError> {
+        self.read_vl_strings_in(file_data, offset_size, length_size)
+    }
+
+    /// [`Self::read_vl_strings`] over any [`Storage`].
+    pub fn read_vl_strings_in<S: Storage + ?Sized>(
+        &self,
+        file_data: &S,
+        offset_size: u8,
+        length_size: u8,
     ) -> Result<Vec<String>, FormatError> {
         let num_elements = self.dataspace.num_elements();
-        vl_data::read_vl_strings(
+        vl_data::read_vl_strings_in(
             file_data,
             &self.raw_data,
             num_elements,
@@ -341,7 +365,8 @@ fn compute_raw_data(
     dataspace: &Dataspace,
     datatype: &Datatype,
 ) -> Vec<u8> {
-    let num_elements = dataspace.num_elements() as usize;
+    // Saturating, like the product: the size is capped at what is there.
+    let num_elements = usize::try_from(dataspace.num_elements()).unwrap_or(usize::MAX);
     let elem_size = datatype.type_size() as usize;
     let expected_size = num_elements.saturating_mul(elem_size);
     let available = data.len().saturating_sub(pos);
@@ -362,6 +387,18 @@ fn extract_name(bytes: &[u8]) -> String {
     String::from_utf8_lossy(&bytes[..end]).into_owned()
 }
 
+/// An attribute's datatype gets libhdf5's extra check for a header without
+/// a checksum (see [`Datatype::check_unused_bits`]).
+fn check_in_header(
+    attr: AttributeMessage,
+    header: &ObjectHeader,
+) -> Result<AttributeMessage, FormatError> {
+    if header.version == 1 {
+        attr.datatype.check_unused_bits()?;
+    }
+    Ok(attr)
+}
+
 /// Extract all attribute messages from an object header.
 pub fn extract_attributes(
     header: &ObjectHeader,
@@ -371,7 +408,7 @@ pub fn extract_attributes(
     for msg in &header.messages {
         if msg.msg_type == MessageType::Attribute {
             let attr = AttributeMessage::parse(&msg.data, length_size)?;
-            attrs.push(attr);
+            attrs.push(check_in_header(attr, header)?);
         }
     }
     Ok(attrs)
@@ -394,57 +431,302 @@ pub fn find_attribute<'a>(
 ///
 /// Use this instead of `extract_attributes` when reading files that may use dense storage
 /// (e.g., objects with many attributes, typically >8).
+///
+/// Fails if any attribute cannot be read; see [`extract_attributes_tolerant`]
+/// to read the others.
 pub fn extract_attributes_full(
     file_data: &[u8],
     header: &ObjectHeader,
     offset_size: u8,
     length_size: u8,
 ) -> Result<Vec<AttributeMessage>, FormatError> {
-    let mut attrs = Vec::new();
+    extract_attributes_full_in(file_data, header, offset_size, length_size)
+}
 
-    // Collect compact attributes (inline in OH)
+/// [`extract_attributes_full`] over any [`Storage`]. Dense attribute
+/// storage is indexed by a v2 B-tree, which is not read over [`Storage`]
+/// yet: on a backend without the whole file in memory an object with dense
+/// attributes is [`FormatError::ContiguousStorageRequired`].
+pub fn extract_attributes_full_in<S: Storage + ?Sized>(
+    file: &S,
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<AttributeMessage>, FormatError> {
+    extract_attributes_with(file, header, offset_size, length_size, &mut Err)
+}
+
+/// Like [`extract_attributes_full`], but an attribute that cannot be read
+/// (a corrupt or unsupported attribute message, or a heap object that cannot
+/// be located) is left out and its error returned alongside the attributes
+/// that could be read, instead of failing them all.
+///
+/// Errors in the structures that index the attributes (the Attribute Info
+/// message, the dense-storage heap header or B-tree) still fail the call:
+/// then it is unknown which attributes exist at all.
+pub fn extract_attributes_tolerant(
+    file_data: &[u8],
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
+    extract_attributes_tolerant_core(file_data, header, offset_size, length_size)
+}
+
+/// [`extract_attributes_tolerant`] over any [`Storage`] (see
+/// [`extract_attributes_full_in`] for dense storage). One with the whole
+/// file in memory is read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
+pub fn extract_attributes_tolerant_in<S: Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => extract_attributes_tolerant(all, header, offset_size, length_size),
+        None => extract_attributes_tolerant_core(file_data, header, offset_size, length_size),
+    }
+}
+
+fn extract_attributes_tolerant_core<S: Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
+    let mut errors = Vec::new();
+    let attrs = extract_attributes_with(file_data, header, offset_size, length_size, &mut |e| {
+        errors.push(e);
+        Ok(())
+    })?;
+    Ok((attrs, errors))
+}
+
+/// Read every attribute; each one that fails goes to `on_error`, which
+/// either stops the read (returns the error) or skips that attribute.
+fn extract_attributes_with<S: Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
+    on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
+) -> Result<Vec<AttributeMessage>, FormatError> {
+    let mut attrs = Vec::new();
+    // Each attribute's creation order, where the file records one.
+    let mut orders: Vec<u32> = Vec::new();
+
+    extract_compact_attributes(
+        file_data,
+        header,
+        offset_size,
+        length_size,
+        &mut attrs,
+        &mut orders,
+        on_error,
+    )?;
+
+    // Check for dense attributes via AttributeInfo message
+    let attr_info = find_attribute_info(header, offset_size)?;
+    if let Some(info) = &attr_info
+        && let Some(fh_addr) = info.fractal_heap_address
+    {
+        extract_dense_attributes(
+            file_data,
+            info,
+            fh_addr,
+            offset_size,
+            length_size,
+            &mut attrs,
+            &mut orders,
+            on_error,
+        )?;
+    }
+
+    // An object that tracks attribute creation order lists its attributes
+    // in that order (h5py's `track_order=True`), as libhdf5 does; otherwise
+    // they come in storage order.
+    if attr_info.is_some_and(|i| i.max_creation_index.is_some()) {
+        let mut paired: Vec<(u32, AttributeMessage)> = orders.into_iter().zip(attrs).collect();
+        paired.sort_by_key(|(o, _)| *o);
+        attrs = paired.into_iter().map(|(_, a)| a).collect();
+    }
+
+    Ok(attrs)
+}
+
+/// B-tree v2 record type of dense attribute storage's name index.
+const ATTRIBUTE_NAME_INDEX: u8 = 8;
+
+/// The attribute called `name` on the object with header `header`: the
+/// first one [`extract_attributes_tolerant`] returns under that name, or
+/// `None` if it returns none (an attribute that cannot be read is not
+/// returned there either).
+///
+/// Compact attributes are in the header and are scanned. Dense attributes
+/// are found through the name index (a v2 B-tree of lookup3 name hashes,
+/// record type 8): only the attributes whose names hash like `name` are read
+/// from the heap, O(log n) instead of all of them. Errors in the structures
+/// that index the attributes fail the call, as they fail a listing.
+pub fn find_attribute_in_file(
+    file_data: &[u8],
+    header: &ObjectHeader,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Option<AttributeMessage>, FormatError> {
+    find_attribute_core(file_data, header, name, offset_size, length_size)
+}
+
+/// [`find_attribute_in_file`] over any [`Storage`] (see
+/// [`extract_attributes_full_in`] for dense storage, whose name index still
+/// needs the whole file in memory). One with the whole file in memory is
+/// read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
+pub fn find_attribute_in<S: Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Option<AttributeMessage>, FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => find_attribute_in_file(all, header, name, offset_size, length_size),
+        None => find_attribute_core(file_data, header, name, offset_size, length_size),
+    }
+}
+
+fn find_attribute_core<S: Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Option<AttributeMessage>, FormatError> {
+    let attr_info = find_attribute_info(header, offset_size)?;
+    let dense = attr_info
+        .as_ref()
+        .and_then(|i| Some((i.fractal_heap_address?, i.btree_name_index_address?)));
+    let Some((fh_addr, btree_addr)) = dense else {
+        // Compact only (or dense storage without a name index, which a
+        // listing reports): as a listing finds it.
+        return Ok(
+            extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?
+                .0
+                .into_iter()
+                .find(|a| a.name == name),
+        );
+    };
+    let btree_hdr = BTreeV2Header::parse_in(
+        file_data,
+        to_usize(btree_addr)? as u64,
+        offset_size,
+        length_size,
+    )?;
+    let fh = FractalHeapHeader::parse_in(file_data, fh_addr, offset_size, length_size)?;
+    if btree_hdr.tree_type != ATTRIBUTE_NAME_INDEX || btree_hdr.record_size < 4 {
+        return Ok(
+            extract_attributes_tolerant_in(file_data, header, offset_size, length_size)?
+                .0
+                .into_iter()
+                .find(|a| a.name == name),
+        );
+    }
+
+    // A listing has the compact attributes first.
+    let mut compact = Vec::new();
+    extract_compact_attributes(
+        file_data,
+        header,
+        offset_size,
+        length_size,
+        &mut compact,
+        &mut Vec::new(),
+        &mut |_| Ok(()),
+    )?;
+    if let Some(a) = compact.into_iter().find(|a| a.name == name) {
+        return Ok(Some(a));
+    }
+
+    // Record: heap ID + message flags(1) + creation order(4) + hash(4); the
+    // hash is the last field.
+    let hash = jenkins_lookup3(name.as_bytes());
+    let hash_at = usize::from(btree_hdr.record_size) - 4;
+    let records = find_btree_v2_records_in(file_data, &btree_hdr, offset_size, &mut |r| match r
+        .get(hash_at..hash_at + 4)
+    {
+        Some(h) => u32::from_le_bytes([h[0], h[1], h[2], h[3]]).cmp(&hash),
+        None => core::cmp::Ordering::Less,
+    })?;
+    let id_len = usize::from(fh.heap_id_length);
+    for record in &records {
+        let Some(id_bytes) = record.data.get(..id_len) else {
+            continue;
+        };
+        let attr = fh
+            .read_managed_object_in(file_data, id_bytes, offset_size)
+            .and_then(|d| {
+                AttributeMessage::parse_in_storage(&d, file_data, offset_size, length_size)
+            });
+        // One that cannot be read is left out, as from a listing.
+        if let Ok(attr) = attr
+            && attr.name == name
+        {
+            return Ok(Some(attr));
+        }
+    }
+    Ok(None)
+}
+
+/// The attributes stored in the object header itself (compact storage), and
+/// each one's creation order into `orders`.
+fn extract_compact_attributes<S: Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
+    attrs: &mut Vec<AttributeMessage>,
+    orders: &mut Vec<u32>,
+    on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
+) -> Result<(), FormatError> {
     for msg in &header.messages {
         if msg.msg_type == MessageType::Attribute {
-            if shared_message::is_shared(msg.flags) {
+            let attr = if shared_message::is_shared(msg.flags) {
                 // Shared attribute: resolve the reference to get actual attribute data
-                let shared_ref = shared_message::parse_shared_ref(&msg.data, offset_size)?;
+                shared_message::parse_shared_ref_sized(&msg.data, offset_size, length_size)
-                let resolved_data = shared_message::resolve_shared_message(
+                    .and_then(|shared_ref| {
+                        shared_message::resolve_shared_message_in(
                             file_data,
                             &shared_ref,
                             MessageType::Attribute,
                             offset_size,
                             length_size,
-                )?;
+                        )
-                let attr = AttributeMessage::parse_in_file(
+                    })
-                    &resolved_data,
+                    .and_then(|resolved| {
+                        AttributeMessage::parse_in_storage(
+                            &resolved,
                             file_data,
                             offset_size,
                             length_size,
-                )?;
+                        )
-                attrs.push(attr);
+                    })
             } else {
-                let attr = AttributeMessage::parse_in_file(
+                AttributeMessage::parse_in_storage(&msg.data, file_data, offset_size, length_size)
-                    &msg.data,
+            };
-                    file_data,
+            let attr = attr.and_then(|a| check_in_header(a, header));
-                    offset_size,
+            match attr {
-                    length_size,
+                Ok(attr) => {
-                )?;
                     attrs.push(attr);
+                    orders.push(msg.creation_order.map_or(0, u32::from));
+                }
+                Err(e) => on_error(e)?,
             }
         }
     }
-
+    Ok(())
-    // Check for dense attributes via AttributeInfo message
-    let attr_info = find_attribute_info(header, offset_size)?;
-    if let Some(info) = attr_info
-        && let Some(fh_addr) = info.fractal_heap_address
-    {
-        let dense_attrs =
-            extract_dense_attributes(file_data, &info, fh_addr, offset_size, length_size)?;
-        attrs.extend(dense_attrs);
-    }
-
-    Ok(attrs)
 }
 
 /// Find and parse the Attribute Info message from an object header.
@@ -461,16 +743,21 @@ fn find_attribute_info(
     Ok(None)
 }
 
-/// Extract attributes from dense storage (fractal heap + B-tree v2).
+/// Extract attributes from dense storage (fractal heap + B-tree v2), and
-fn extract_dense_attributes(
+/// each one's creation order into `orders`.
-    file_data: &[u8],
+#[allow(clippy::too_many_arguments)]
+fn extract_dense_attributes<S: Storage + ?Sized>(
+    file_data: &S,
     attr_info: &AttributeInfoMessage,
     fh_addr: u64,
     offset_size: u8,
     length_size: u8,
-) -> Result<Vec<AttributeMessage>, FormatError> {
+    attrs: &mut Vec<AttributeMessage>,
+    orders: &mut Vec<u32>,
+    on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
+) -> Result<(), FormatError> {
     // Parse fractal heap
-    let fh = FractalHeapHeader::parse(file_data, fh_addr as usize, offset_size, length_size)?;
+    let fh = FractalHeapHeader::parse_in(file_data, fh_addr, offset_size, length_size)?;
 
     // Parse B-tree v2 for name index (type 8)
     let btree_addr = attr_info
@@ -479,31 +766,47 @@ fn extract_dense_attributes(
             expected: 1,
             available: 0,
         })?;
-    let btree_hdr = BTreeV2Header::parse(file_data, btree_addr as usize, offset_size, length_size)?;
+    let btree_hdr = BTreeV2Header::parse_in(
-    let records = collect_btree_v2_records(file_data, &btree_hdr, offset_size, length_size)?;
+        file_data,
+        to_usize(btree_addr)? as u64,
+        offset_size,
+        length_size,
+    )?;
+    let records = collect_btree_v2_records_in(file_data, &btree_hdr, offset_size, length_size)?;
 
-    let mut attrs = Vec::new();
     for record in &records {
         // Per HDF5 spec, both type 8 and type 9 records start with heap_id:
         //   Type 8: heap_id(8) + msg_flags(1) + creation_order(4) + hash(4)
         //   Type 9: heap_id(8) + msg_flags(1) + creation_order(4)
-        let id_offset = 0;
+        let id_len = fh.heap_id_length as usize;
-
+        let Some(id_bytes) = record.data.get(..id_len) else {
-        if record.data.len() < id_offset + fh.heap_id_length as usize {
+            on_error(FormatError::UnexpectedEof {
+                expected: id_len,
+                available: record.data.len(),
+            })?;
             continue;
-        }
+        };
-        let id_bytes = &record.data[id_offset..id_offset + fh.heap_id_length as usize];
-
-        // Read attribute message from fractal heap
-        let attr_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
 
         // The data in the heap is a complete attribute message
-        let attr =
+        let attr = fh
-            AttributeMessage::parse_in_file(&attr_data, file_data, offset_size, length_size)?;
+            .read_managed_object_in(file_data, id_bytes, offset_size)
+            .and_then(|attr_data| {
+                AttributeMessage::parse_in_storage(&attr_data, file_data, offset_size, length_size)
+            });
+        match attr {
+            Ok(attr) => {
                 attrs.push(attr);
+                let order = record
+                    .data
+                    .get(id_len + 1..id_len + 5)
+                    .map_or(0, |b| u32::from_le_bytes([b[0], b[1], b[2], b[3]]));
+                orders.push(order);
+            }
+            Err(e) => on_error(e)?,
+        }
     }
 
-    Ok(attrs)
+    Ok(())
 }
 
 #[cfg(test)]
@@ -523,7 +826,8 @@ mod tests {
 
     /// Build an f64 LE datatype message.
     fn build_f64_dt() -> Vec<u8> {
-        let mut buf = build_dt_header(1, 1, [0x00, 0x00, 0x02], 8);
+        // Sign bit 63 (bits 8-15 of the class bits).
+        let mut buf = build_dt_header(1, 1, [0x20, 63, 0x00], 8);
         let mut props = [0u8; 12];
         props[2..4].copy_from_slice(&64u16.to_le_bytes()); // bit_precision
         props[4] = 52; // exp_location
@@ -897,4 +1201,73 @@ mod tests {
         let strs = attr.read_as_strings().unwrap();
         assert_eq!(strs, vec!["abcd", "EFGH"]);
     }
+
+    /// Every object's attributes in h5py-written files read identically
+    /// through a read_at-only CountingStorage — compact ones, shared ones,
+    /// those behind an Attribute Info message and dense storage (its v2
+    /// B-tree name index included) — and through a slice as Storage.
+    #[test]
+    fn storage_reads_match_slice_reads() {
+        use crate::storage::CountingStorage;
+        let files: [(&str, &[u8]); 5] = [
+            ("attrs", include_bytes!("../tests/fixtures/attrs.h5")),
+            (
+                "mixed_attrs",
+                include_bytes!("../tests/fixtures/mixed_attrs.h5"),
+            ),
+            (
+                "dense_attrs",
+                include_bytes!("../tests/fixtures/dense_attrs.h5"),
+            ),
+            (
+                "dense_attrs_root",
+                include_bytes!("../tests/fixtures/dense_attrs_root.h5"),
+            ),
+            (
+                "shared_fill_value",
+                include_bytes!("../tests/fixtures/shared_fill_value.h5"),
+            ),
+        ];
+        let (mut same, mut dense, mut attrs) = (0, 0, 0);
+        for (name, file) in files {
+            let sb = crate::superblock::Superblock::parse(file, 0).unwrap();
+            let (os, ls) = (sb.offset_size, sb.length_size);
+            let mut addrs = vec![sb.root_group_address];
+            addrs.extend(
+                crate::group_v2::resolve_group_children(file, &sb, sb.root_group_address)
+                    .unwrap()
+                    .iter()
+                    .map(|e| e.object_header_address),
+            );
+            let storage = CountingStorage::new(file.to_vec());
+            for addr in addrs {
+                let header = ObjectHeader::parse(file, addr as usize, os, ls).unwrap();
+                let want = extract_attributes_full(file, &header, os, ls);
+                let slice_storage = extract_attributes_full_in(&file, &header, os, ls);
+                assert_eq!(format!("{slice_storage:?}"), format!("{want:?}"));
+                let got = extract_attributes_full_in(&storage, &header, os, ls);
+                let got_t = extract_attributes_tolerant_in(&storage, &header, os, ls);
+                let is_dense = find_attribute_info(&header, os)
+                    .unwrap()
+                    .is_some_and(|i| i.fractal_heap_address.is_some());
+                if is_dense {
+                    dense += 1;
+                }
+                attrs += want.as_ref().map_or(0, Vec::len);
+                assert_eq!(format!("{got:?}"), format!("{want:?}"), "{name}");
+                let want_t = extract_attributes_tolerant(file, &header, os, ls);
+                assert_eq!(format!("{got_t:?}"), format!("{want_t:?}"), "{name}");
+                same += 1;
+                for a in want.iter().flatten() {
+                    let one = find_attribute_in(&storage, &header, &a.name, os, ls);
+                    let want_one = find_attribute_in_file(file, &header, &a.name, os, ls);
+                    assert_eq!(format!("{one:?}"), format!("{want_one:?}"), "{name}");
+                }
+            }
+        }
+        assert!(
+            same >= 5 && dense >= 2 && attrs >= 5,
+            "{same} {dense} {attrs}"
+        );
+    }
 }

crates/clawhdf5-format/src/btree_v1.rs

@@ -4,6 +4,7 @@
 use alloc::vec::Vec;
 
 use crate::error::FormatError;
+use crate::storage::{Storage, read_exact_at};
 
 /// A parsed B-tree v1 node.
 #[derive(Debug, Clone)]
@@ -74,13 +75,28 @@ impl BTreeV1Node {
         file_data: &[u8],
         offset: usize,
         offset_size: u8,
+        length_size: u8,
+    ) -> Result<BTreeV1Node, FormatError> {
+        Self::parse_in(file_data, offset as u64, offset_size, length_size)
+    }
+
+    /// [`Self::parse`] over any [`Storage`]: one read of the node's header,
+    /// one of its keys and children.
+    pub fn parse_in<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        offset_size: u8,
         _length_size: u8,
     ) -> Result<BTreeV1Node, FormatError> {
         // signature(4) + node_type(1) + node_level(1) + entries_used(2) = 8
         // + left_sibling(offset_size) + right_sibling(offset_size)
         let os = offset_size as usize;
         let header_size = 8 + os * 2;
-        ensure_len(file_data, offset, header_size)?;
+        let header = read_exact_at(file, offset, header_size)?;
+        let file_data: &[u8] = &header;
+        // The header's read checked that `offset + header_size` fits.
+        let body_start = offset + header_size as u64;
+        let offset = 0usize;
 
         if &file_data[offset..offset + 4] != b"TREE" {
             return Err(FormatError::InvalidBTreeSignature);
@@ -102,31 +118,30 @@ impl BTreeV1Node {
         } else {
             Some(read_offset(file_data, pos, offset_size)?)
         };
-        pos += os;
 
         // For type 0: keys are offset_size bytes, children are offset_size bytes
         // Layout: key[0], child[0], key[1], child[1], ..., key[N-1], child[N-1], key[N]
         let eu = entries_used as usize;
         let key_size = os; // For type 0, key = offset_size
         let needed = eu * (key_size + os) + key_size; // eu children + (eu+1) keys
-        ensure_len(file_data, pos, needed)?;
+        let body = read_exact_at(file, body_start, needed)?;
+        let file_data: &[u8] = &body;
 
         let mut keys = Vec::with_capacity(eu + 1);
         let mut children = Vec::with_capacity(eu);
 
-        for _i in 0..eu {
+        if os == 0 {
-            // key[i]
+            // What reading the first key reports (and keeps `chunks_exact`
-            let key = read_offset(file_data, pos, offset_size)?;
+            // below from being given a zero size).
-            keys.push(key);
+            return Err(FormatError::InvalidOffsetSize(offset_size));
-            pos += key_size;
-            // child[i]
-            let child = read_offset(file_data, pos, offset_size)?;
-            children.push(child);
-            pos += os;
         }
-        // final key
+        // `needed` bytes: key[0], child[0], ..., child[eu - 1], key[eu].
-        let key = read_offset(file_data, pos, offset_size)?;
+        let (pairs, last) = file_data.split_at(eu * (key_size + os));
-        keys.push(key);
+        for pair in pairs.chunks_exact(key_size + os) {
+            keys.push(read_offset(pair, 0, offset_size)?);
+            children.push(read_offset(pair, key_size, offset_size)?);
+        }
+        keys.push(read_offset(last, 0, offset_size)?);
 
         Ok(BTreeV1Node {
             node_type,
@@ -150,11 +165,21 @@ pub fn collect_symbol_table_nodes(
     offset_size: u8,
     length_size: u8,
 ) -> Result<Vec<u64>, FormatError> {
-    collect_symbol_table_nodes_inner(file_data, btree_address, offset_size, length_size, 0)
+    collect_symbol_table_nodes_in(file_data, btree_address, offset_size, length_size)
 }
 
-fn collect_symbol_table_nodes_inner(
+/// [`collect_symbol_table_nodes`] over any [`Storage`]: two reads per node.
-    file_data: &[u8],
+pub fn collect_symbol_table_nodes_in<S: Storage + ?Sized>(
+    file: &S,
+    btree_address: u64,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<u64>, FormatError> {
+    collect_symbol_table_nodes_inner(file, btree_address, offset_size, length_size, 0)
+}
+
+fn collect_symbol_table_nodes_inner<S: Storage + ?Sized>(
+    file: &S,
     btree_address: u64,
     offset_size: u8,
     length_size: u8,
@@ -164,7 +189,7 @@ fn collect_symbol_table_nodes_inner(
         return Err(FormatError::NestingDepthExceeded);
     }
 
-    let node = BTreeV1Node::parse(file_data, btree_address as usize, offset_size, length_size)?;
+    let node = BTreeV1Node::parse_in(file, btree_address, offset_size, length_size)?;
 
     if node.node_type != 0 {
         return Err(FormatError::InvalidBTreeNodeType(node.node_type));
@@ -178,7 +203,7 @@ fn collect_symbol_table_nodes_inner(
         let mut result = Vec::new();
         for &child_addr in &node.children {
             let child_snods = collect_symbol_table_nodes_inner(
-                file_data,
+                file,
                 child_addr,
                 offset_size,
                 length_size,
@@ -317,4 +342,48 @@ mod tests {
         assert_eq!(node.entries_used, 1);
         assert_eq!(node.children, vec![0x50]);
     }
+
+    /// Nodes and trees, cut at every length, parse identically through a
+    /// `read_at`-only storage.
+    #[test]
+    fn storage_parse_matches_slice_parse() {
+        use crate::storage::CountingStorage;
+        let nodes = [
+            build_btree_node(0, 0, &[0, 5, 10], &[0x100, 0x200], None, None, 8),
+            build_btree_node(0, 0, &[0, 5], &[0x100], Some(0x40), Some(0x80), 4),
+            build_btree_node(1, 2, &[0, 5], &[0x100], None, Some(0x80), 8),
+        ];
+        for (n, node) in nodes.iter().enumerate() {
+            let os = if n == 1 { 4 } else { 8 };
+            for cut in 0..=node.len() {
+                let f = &node[..cut];
+                let storage = CountingStorage::new(f.to_vec());
+                let want = BTreeV1Node::parse(f, 0, os, 8);
+                let got = BTreeV1Node::parse_in(&storage, 0, os, 8);
+                assert_eq!(format!("{got:?}"), format!("{want:?}"));
+            }
+        }
+        let leaf1 = build_btree_node(0, 0, &[0, 5], &[0xA00], None, None, 8);
+        let leaf2 = build_btree_node(0, 0, &[5, 10], &[0xB00], None, None, 8);
+        let internal = build_btree_node(0, 1, &[0, 5, 10], &[0, 256], None, None, 8);
+        let mut file = vec![0u8; 512 + internal.len()];
+        file[..leaf1.len()].copy_from_slice(&leaf1);
+        file[256..256 + leaf2.len()].copy_from_slice(&leaf2);
+        file[512..].copy_from_slice(&internal);
+        for cut in [file.len(), 300, 260, 100, 10] {
+            let mut f = file.clone();
+            if cut < 512 {
+                // Truncate the leaves, keep the root.
+                f[cut..512].fill(0);
+            }
+            let storage = CountingStorage::new(f.clone());
+            assert_eq!(
+                collect_symbol_table_nodes_in(&storage, 512, 8, 8),
+                collect_symbol_table_nodes(&f, 512, 8, 8)
+            );
+        }
+        let storage = CountingStorage::new(file);
+        collect_symbol_table_nodes_in(&storage, 512, 8, 8).unwrap();
+        assert_eq!(storage.reads(), 6);
+    }
 }

crates/clawhdf5-format/src/btree_v2.rs

@@ -2,11 +2,14 @@
 
 #[cfg(not(feature = "std"))]
 use alloc::vec::Vec;
+use core::cmp::Ordering;
 
 #[cfg(feature = "checksum")]
 use byteorder::{ByteOrder, LittleEndian};
 
+use crate::addr::to_usize;
 use crate::error::FormatError;
+use crate::storage::{Storage, Window, len_usize};
 
 /// Parsed B-tree v2 header (signature "BTHD").
 #[derive(Debug, Clone)]
@@ -71,7 +74,7 @@ fn ensure_len(data: &[u8], pos: usize, needed: usize) -> Result<(), FormatError>
 
 /// Compute the number of bytes needed to represent a count, using variable-width encoding.
 /// B-tree v2 uses this for the number of records fields in internal nodes.
-fn bytes_for_max_records(max_nrec: u64) -> usize {
+pub(crate) fn bytes_for_max_records(max_nrec: u64) -> usize {
     if max_nrec == 0 {
         return 1;
     }
@@ -97,38 +100,52 @@ impl BTreeV2Header {
         offset_size: u8,
         length_size: u8,
     ) -> Result<BTreeV2Header, FormatError> {
-        ensure_len(file_data, offset, 4)?;
+        Self::parse_in(file_data, offset as u64, offset_size, length_size)
-        if &file_data[offset..offset + 4] != b"BTHD" {
+    }
+
+    /// [`Self::parse`] over any [`Storage`]: one bounded read of the
+    /// header.
+    pub fn parse_in<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        offset_size: u8,
+        length_size: u8,
+    ) -> Result<BTreeV2Header, FormatError> {
+        // Every field and the checksum; the window holds all of it or ends
+        // at the end of the file, so its bounds checks are the whole-file
+        // ones.
+        let full = 16 + usize::from(offset_size) + 2 + usize::from(length_size) + 4;
+        let w = Window::read(file, offset, full)?;
+        let d = &w.bytes;
+        w.ensure(0, 4)?;
+        if &d[..4] != b"BTHD" {
             return Err(FormatError::InvalidBTreeV2Signature);
         }
 
-        ensure_len(file_data, offset, 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1)?;
+        w.ensure(0, 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1)?;
-        let version = file_data[offset + 4];
+        let version = d[4];
         if version != 0 {
             return Err(FormatError::InvalidBTreeV2Version(version));
         }
 
-        let tree_type = file_data[offset + 5];
+        let tree_type = d[5];
-        let node_size = u32::from_le_bytes([
+        let node_size = u32::from_le_bytes([d[6], d[7], d[8], d[9]]);
-            file_data[offset + 6],
+        let record_size = u16::from_le_bytes([d[10], d[11]]);
-            file_data[offset + 7],
+        let depth = u16::from_le_bytes([d[12], d[13]]);
-            file_data[offset + 8],
+        let _split_percent = d[14];
-            file_data[offset + 9],
+        let _merge_percent = d[15];
-        ]);
-        let record_size = u16::from_le_bytes([file_data[offset + 10], file_data[offset + 11]]);
-        let depth = u16::from_le_bytes([file_data[offset + 12], file_data[offset + 13]]);
-        let _split_percent = file_data[offset + 14];
-        let _merge_percent = file_data[offset + 15];
 
-        let mut pos = offset + 16;
+        let mut pos = 16;
-        let root_node_address = read_offset(file_data, pos, offset_size)?;
+        w.ensure(pos, usize::from(offset_size))?;
+        let root_node_address = read_offset(d, pos, offset_size)?;
         pos += offset_size as usize;
 
-        ensure_len(file_data, pos, 2)?;
+        w.ensure(pos, 2)?;
-        let num_records_in_root = u16::from_le_bytes([file_data[pos], file_data[pos + 1]]);
+        let num_records_in_root = u16::from_le_bytes([d[pos], d[pos + 1]]);
         pos += 2;
 
-        let total_records = read_offset(file_data, pos, length_size)?;
+        w.ensure(pos, usize::from(length_size))?;
+        let total_records = read_offset(d, pos, length_size)?;
         #[allow(unused_assignments)]
         {
             pos += length_size as usize;
@@ -137,9 +154,9 @@ impl BTreeV2Header {
         // Validate header checksum
         #[cfg(feature = "checksum")]
         {
-            ensure_len(file_data, pos, 4)?;
+            w.ensure(pos, 4)?;
-            let stored = LittleEndian::read_u32(&file_data[pos..pos + 4]);
+            let stored = LittleEndian::read_u32(&d[pos..pos + 4]);
-            let computed = crate::checksum::jenkins_lookup3(&file_data[offset..pos]);
+            let computed = crate::checksum::jenkins_lookup3(&d[..pos]);
             if computed != stored {
                 return Err(FormatError::ChecksumMismatch {
                     expected: stored,
@@ -163,7 +180,7 @@ impl BTreeV2Header {
 /// Compute maximum records per node for a given depth level.
 /// leaf: (node_size - overhead) / record_size
 /// internal: depends on pointers
-fn max_records_leaf(node_size: u32, record_size: u16) -> u64 {
+pub(crate) fn max_records_leaf(node_size: u32, record_size: u16) -> u64 {
     // Leaf overhead: signature(4) + version(1) + type(1) + checksum(4) = 10
     let overhead = 10u32;
     if node_size <= overhead || record_size == 0 {
@@ -189,6 +206,17 @@ pub fn collect_btree_v2_records(
     header: &BTreeV2Header,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<BTreeV2Record>, FormatError> {
+    collect_btree_v2_records_in(file_data, header, offset_size, length_size)
+}
+
+/// [`collect_btree_v2_records`] over any [`Storage`]: one bounded read per
+/// node.
+pub fn collect_btree_v2_records_in<S: Storage + ?Sized>(
+    file: &S,
+    header: &BTreeV2Header,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<BTreeV2Record>, FormatError> {
     if header.total_records == 0 || header.num_records_in_root == 0 {
         return Ok(Vec::new());
@@ -208,24 +236,25 @@ pub fn collect_btree_v2_records(
     // millions of records from a few kilobytes. Counting against what the
     // file could physically contain bounds that without trusting the
     // header's own `total_records`.
-    let mut budget = file_data.len() / usize::from(header.record_size.max(1));
+    let mut budget = len_usize(file) / usize::from(header.record_size.max(1));
 
     let max_leaf_nrec = max_records_leaf(header.node_size, header.record_size);
 
     if header.depth == 0 {
         // Root is a leaf
         parse_leaf_records(
-            file_data,
+            file,
-            header.root_node_address as usize,
+            to_usize(header.root_node_address)?,
             header.num_records_in_root,
             header.record_size,
+            header.node_size,
         )
     } else {
         // Root is internal; traverse recursively
         let mut records = Vec::new();
         collect_internal_records(
-            file_data,
+            file,
-            header.root_node_address as usize,
+            to_usize(header.root_node_address)?,
             header.num_records_in_root,
             header.depth,
             header.record_size,
@@ -240,36 +269,72 @@ pub fn collect_btree_v2_records(
     }
 }
 
+/// A node's bytes: `want` bytes at `offset` (fewer only at the end of the
+/// file), after checking its 4-byte signature. A node is read in one piece
+/// when it fits in `node_size` (every valid node does); a larger claimed
+/// extent — record counts from a damaged parent — is first checked against
+/// the end of the file, so it costs a read only of bytes the file has.
+/// Bounds errors are the whole-file ones: the signature check needs the
+/// first 6 bytes, then `checks` — `(position, length)` pairs relative to
+/// the node, in the order the parser checks them — must lie in the file.
+fn read_node<'a, S: Storage + ?Sized>(
+    file: &'a S,
+    offset: usize,
+    want: usize,
+    node_size: u32,
+    signature: &[u8; 4],
+    checks: &[(usize, usize)],
+) -> Result<Window<'a>, FormatError> {
+    let one_read = usize::try_from(node_size).unwrap_or(usize::MAX).max(6);
+    let w = Window::read(file, offset as u64, want.min(one_read))?;
+    w.ensure(0, 6)?;
+    if &w.bytes[..4] != signature {
+        return Err(FormatError::InvalidBTreeV2Signature);
+    }
+    if want <= one_read {
+        return Ok(w);
+    }
+    for &(rel, len) in checks {
+        Window::check_extent(file, offset as u64, rel, len)?;
+    }
+    Window::read(file, offset as u64, want)
+}
+
 /// Parse records from a leaf node (signature "BTLF").
-fn parse_leaf_records(
+fn parse_leaf_records<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file: &S,
     offset: usize,
     num_records: u16,
     record_size: u16,
+    node_size: u32,
 ) -> Result<Vec<BTreeV2Record>, FormatError> {
     // signature(4) + version(1) + type(1) = 6 bytes header
-    ensure_len(file_data, offset, 6)?;
+    let pos = 6;
-    if &file_data[offset..offset + 4] != b"BTLF" {
-        return Err(FormatError::InvalidBTreeV2Signature);
-    }
-
-    let pos = offset + 6;
     let rs = record_size as usize;
     let total = (num_records as usize)
         .checked_mul(rs)
         .ok_or(FormatError::UnexpectedEof {
             expected: usize::MAX,
-            available: file_data.len(),
+            available: len_usize(file),
         })?;
-    ensure_len(file_data, pos, total)?;
+    let w = read_node(
+        file,
+        offset,
+        pos + total + 4,
+        node_size,
+        b"BTLF",
+        &[(pos, total)],
+    )?;
+    let d = &w.bytes;
+    w.ensure(pos, total)?;
 
     // Validate checksum: 4 bytes after records + padding
     #[cfg(feature = "checksum")]
     {
         let checksum_pos = pos + total;
-        if file_data.len() >= checksum_pos + 4 {
+        if d.len() >= checksum_pos + 4 {
-            let stored = LittleEndian::read_u32(&file_data[checksum_pos..checksum_pos + 4]);
+            let stored = LittleEndian::read_u32(&d[checksum_pos..checksum_pos + 4]);
-            let computed = crate::checksum::jenkins_lookup3(&file_data[offset..checksum_pos]);
+            let computed = crate::checksum::jenkins_lookup3(&d[..checksum_pos]);
             if computed != stored {
                 return Err(FormatError::ChecksumMismatch {
                     expected: stored,
@@ -283,16 +348,135 @@ fn parse_leaf_records(
     for i in 0..num_records as usize {
         let start = pos + i * rs;
         records.push(BTreeV2Record {
-            data: file_data[start..start + rs].to_vec(),
+            data: d[start..start + rs].to_vec(),
         });
     }
     Ok(records)
 }
 
+/// An internal node read from the file: its bytes (from the signature on),
+/// where its records start, and its children as `(address, record count)`.
+struct InternalNode<'a> {
+    node: Window<'a>,
+    records_start: usize,
+    children: Vec<(u64, u16)>,
+}
+
+impl InternalNode<'_> {
+    /// Record `i`, `rs` bytes long.
+    fn record(&self, i: usize, rs: usize) -> Result<&[u8], FormatError> {
+        let overflow = || FormatError::UnexpectedEof {
+            expected: usize::MAX,
+            available: usize::MAX,
+        };
+        let rec_start = i
+            .checked_mul(rs)
+            .and_then(|o| self.records_start.checked_add(o))
+            .ok_or_else(overflow)?;
+        self.node.ensure(rec_start, rs)?;
+        Ok(&self.node.bytes[rec_start..rec_start + rs])
+    }
+}
+
+/// An internal node's layout: where its records start, and its children as
+/// `(address, record count)`.
+#[allow(clippy::too_many_arguments)]
+fn read_internal_node<S: Storage + ?Sized>(
+    file: &S,
+    offset: usize,
+    num_records: u16,
+    depth: u16,
+    record_size: u16,
+    node_size: u32,
+    offset_size: u8,
+    max_leaf_nrec: u64,
+) -> Result<InternalNode<'_>, FormatError> {
+    let nr = num_records as usize;
+    let rs = record_size as usize;
+
+    // Records first
+    let records_total = nr.checked_mul(rs).ok_or(FormatError::UnexpectedEof {
+        expected: usize::MAX,
+        available: len_usize(file),
+    })?;
+
+    // Child pointer layout, as libhdf5 computes it (H5B2__hdr_init): the
+    // child's record count is always encoded in the width needed for a
+    // *leaf's* maximum, and — below the first internal level — the child
+    // subtree's total record count in the width needed for the most records
+    // a subtree of that depth can hold.
+    let child_depth = depth - 1;
+    let nrec_width = bytes_for_max_records(max_leaf_nrec);
+    let total_nrec_width = if depth > 1 {
+        bytes_for_max_records(cum_max_records(
+            node_size,
+            record_size,
+            offset_size,
+            max_leaf_nrec,
+            child_depth,
+        ))
+    } else {
+        0
+    };
+
+    let num_children = nr + 1;
+    let child_ptr_size = offset_size as usize + nrec_width + total_nrec_width;
+    let pointers = num_children * child_ptr_size;
+
+    // signature(4) + version(1) + type(1) = 6, records, pointers, checksum.
+    let w = read_node(
+        file,
+        offset,
+        6 + records_total + pointers + 4,
+        node_size,
+        b"BTIN",
+        &[(6, records_total), (6 + records_total, pointers)],
+    )?;
+    let d = &w.bytes;
+    let mut pos = 6;
+    w.ensure(pos, records_total)?;
+    let records_start = pos;
+    pos += records_total;
+
+    w.ensure(pos, pointers)?;
+
+    let mut children = Vec::with_capacity(num_children);
+    for _ in 0..num_children {
+        let addr = read_offset(d, pos, offset_size)?;
+        pos += offset_size as usize;
+        let child_nrec = read_var_uint(d, pos, nrec_width)? as u16;
+        pos += nrec_width;
+        pos += total_nrec_width; // skip total records in subtree
+        children.push((addr, child_nrec));
+    }
+
+    // The checksum follows the child pointers and covers the node up to it.
+    // Lookups prune children by the keys in this node, so an unverified
+    // internal node could hide a record without any error: libhdf5 refuses
+    // a mismatch here, and so does this.
+    #[cfg(feature = "checksum")]
+    {
+        w.ensure(pos, 4)?;
+        let stored = LittleEndian::read_u32(&d[pos..pos + 4]);
+        let computed = crate::checksum::jenkins_lookup3(&d[..pos]);
+        if computed != stored {
+            return Err(FormatError::ChecksumMismatch {
+                expected: stored,
+                computed,
+            });
+        }
+    }
+    Ok(InternalNode {
+        node: w,
+        records_start,
+        children,
+    })
+}
+
 /// Recursively collect records from an internal node.
 #[allow(clippy::too_many_arguments, clippy::only_used_in_recursion)]
-fn collect_internal_records(
+fn collect_internal_records<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file: &S,
     offset: usize,
     num_records: u16,
     depth: u16,
@@ -304,90 +488,38 @@ fn collect_internal_records(
     budget: &mut usize,
     out: &mut Vec<BTreeV2Record>,
 ) -> Result<(), FormatError> {
-    // signature(4) + version(1) + type(1) = 6
-    ensure_len(file_data, offset, 6)?;
-    if &file_data[offset..offset + 4] != b"BTIN" {
-        return Err(FormatError::InvalidBTreeV2Signature);
-    }
-
     let nr = num_records as usize;
     let rs = record_size as usize;
-    let mut pos = offset + 6;
+    let node = read_internal_node(
-
+        file,
-    // Read all records first
+        offset,
-    let records_total = nr.checked_mul(rs).ok_or(FormatError::UnexpectedEof {
+        num_records,
-        expected: usize::MAX,
+        depth,
-        available: file_data.len(),
+        record_size,
-    })?;
+        node_size,
-    ensure_len(file_data, pos, records_total)?;
+        offset_size,
-    let records_start = pos;
+        max_leaf_nrec,
-    pos += records_total;
+    )?;
-
-    // Compute sizes for child pointers
-    // max_records at child depth - for variable-width nrec encoding
     let child_depth = depth - 1;
-    let max_nrec_child = if child_depth == 0 {
-        max_leaf_nrec
-    } else {
-        // For internal nodes at child_depth, the true max_nrec depends on the
-        // node size, record size, and the recursive width of child pointer
-        // entries (which themselves depend on max_nrec at deeper levels).
-        // Computing the exact value requires iterating from the leaf level
-        // upward, as described in the HDF5 spec (III.A.2 "Computing the Size
-        // of B-tree Nodes").
-        //
-        // We use `max_leaf_nrec * 2` as a conservative upper bound. This
-        // over-estimates the nrec encoding width, which means we may read
-        // slightly more bytes per child pointer than strictly necessary, but
-        // never fewer. The over-read bytes are harmless because we only
-        // decode `num_records` entries (the actual count from the node header).
-        //
-        // Known limitation: for very deep trees (depth > 3) with small record
-        // sizes, the true max could exceed this estimate, causing us to
-        // under-allocate the nrec encoding width and misparse child pointers.
-        // In practice, HDF5 B-tree v2 depths rarely exceed 2-3.
-        max_leaf_nrec * 2
-    };
-    let nrec_width = bytes_for_max_records(max_nrec_child);
-
-    // Total records in subtree width (only if depth > 1)
-    let total_nrec_width = if depth > 1 {
-        // Width to hold total records in a subtree
-        // We compute max possible total records at this subtree depth
-        let max_total = header_max_total_records(max_leaf_nrec, depth - 1);
-        bytes_for_max_records(max_total)
-    } else {
-        0
-    };
-
-    let num_children = nr + 1;
-    let child_ptr_size = offset_size as usize + nrec_width + total_nrec_width;
-    ensure_len(file_data, pos, num_children * child_ptr_size)?;
-
-    // Read child pointers
-    let mut children = Vec::with_capacity(num_children);
-    for _ in 0..num_children {
-        let addr = read_offset(file_data, pos, offset_size)?;
-        pos += offset_size as usize;
-        let child_nrec = read_var_uint(file_data, pos, nrec_width)? as u16;
-        pos += nrec_width;
-        pos += total_nrec_width; // skip total records in subtree
-        children.push((addr, child_nrec));
-    }
 
     // Interleave: child[0], record[0], child[1], record[1], ..., child[nr]
     // We collect child[0] records, then record[0], then child[1], etc.
-    for (i, &(child_addr, child_nrec)) in children.iter().enumerate() {
+    for (i, &(child_addr, child_nrec)) in node.children.iter().enumerate() {
         if child_depth == 0 {
             // Before parsing, so a refused tree is not also a large allocation.
             spend(budget, usize::from(child_nrec))?;
-            let leaf_recs =
+            let leaf_recs = parse_leaf_records(
-                parse_leaf_records(file_data, child_addr as usize, child_nrec, record_size)?;
+                file,
+                to_usize(child_addr)?,
+                child_nrec,
+                record_size,
+                node_size,
+            )?;
             out.extend(leaf_recs);
         } else {
             collect_internal_records(
-                file_data,
+                file,
-                child_addr as usize,
+                to_usize(child_addr)?,
                 child_nrec,
                 child_depth,
                 record_size,
@@ -402,32 +534,10 @@ fn collect_internal_records(
 
         // Add record[i] (except after the last child)
         if i < nr {
-            let rec_offset = i.checked_mul(rs).ok_or(FormatError::UnexpectedEof {
+            let data = node.record(i, rs)?;
-                expected: usize::MAX,
-                available: file_data.len(),
-            })?;
-            let rec_start =
-                records_start
-                    .checked_add(rec_offset)
-                    .ok_or(FormatError::UnexpectedEof {
-                        expected: usize::MAX,
-                        available: file_data.len(),
-                    })?;
-            let rec_end = rec_start
-                .checked_add(rs)
-                .ok_or(FormatError::UnexpectedEof {
-                    expected: usize::MAX,
-                    available: file_data.len(),
-                })?;
-            if rec_end > file_data.len() {
-                return Err(FormatError::UnexpectedEof {
-                    expected: rec_end,
-                    available: file_data.len(),
-                });
-            }
             spend(budget, 1)?;
             out.push(BTreeV2Record {
-                data: file_data[rec_start..rec_end].to_vec(),
+                data: data.to_vec(),
             });
         }
     }
@@ -435,14 +545,218 @@ fn collect_internal_records(
     Ok(())
 }
 
-/// Estimate maximum total records at a given depth (for variable-width encoding).
+/// The records of a B-tree v2 that fall in one key range, found by
-fn header_max_total_records(max_leaf_nrec: u64, depth: u16) -> u64 {
+/// descending the tree instead of reading all of it.
-    // Conservative: branching factor * max_leaf at each level
+///
-    let mut total = max_leaf_nrec;
+/// `cmp` places a record relative to the range: `Less` if the record sorts
-    for _ in 0..depth {
+/// before it, `Greater` if after, `Equal` if the record is in it. The tree
-        total = total.saturating_mul(max_leaf_nrec.max(2));
+/// must be ordered consistently with `cmp`, as libhdf5 orders it (a link or
+/// attribute name index by name hash, so all records with one hash form a
+/// range whatever order their names are in). Only the nodes whose key
+/// interval overlaps the range are read: O(depth) nodes plus those holding
+/// the matches. Matches come in tree order.
+pub fn find_btree_v2_records(
+    file_data: &[u8],
+    header: &BTreeV2Header,
+    offset_size: u8,
+    cmp: &mut dyn FnMut(&[u8]) -> Ordering,
+) -> Result<Vec<BTreeV2Record>, FormatError> {
+    find_btree_v2_records_in(file_data, header, offset_size, cmp)
+}
+
+/// [`find_btree_v2_records`] over any [`Storage`]: one bounded read per
+/// node visited.
+pub fn find_btree_v2_records_in<S: Storage + ?Sized>(
+    file: &S,
+    header: &BTreeV2Header,
+    offset_size: u8,
+    cmp: &mut dyn FnMut(&[u8]) -> Ordering,
+) -> Result<Vec<BTreeV2Record>, FormatError> {
+    if header.total_records == 0 || header.num_records_in_root == 0 {
+        return Ok(Vec::new());
     }
-    total
+    if header.depth > MAX_DEPTH {
+        return Err(FormatError::NestingDepthExceeded);
+    }
+    // As in `collect_btree_v2_records`: a valid tree cannot hold more
+    // records than the file has room for, however its children are shared.
+    let mut budget = len_usize(file) / usize::from(header.record_size.max(1));
+    let max_leaf_nrec = max_records_leaf(header.node_size, header.record_size);
+    let mut out = Vec::new();
+    find_in_node(
+        file,
+        header,
+        to_usize(header.root_node_address)?,
+        header.num_records_in_root,
+        header.depth,
+        offset_size,
+        max_leaf_nrec,
+        cmp,
+        &mut budget,
+        &mut out,
+    )?;
+    Ok(out)
+}
+
+#[allow(clippy::too_many_arguments)]
+fn find_in_node<S: Storage + ?Sized>(
+    file: &S,
+    header: &BTreeV2Header,
+    offset: usize,
+    num_records: u16,
+    depth: u16,
+    offset_size: u8,
+    max_leaf_nrec: u64,
+    cmp: &mut dyn FnMut(&[u8]) -> Ordering,
+    budget: &mut usize,
+    out: &mut Vec<BTreeV2Record>,
+) -> Result<(), FormatError> {
+    spend(budget, usize::from(num_records))?;
+    if depth == 0 {
+        let records = parse_leaf_records(
+            file,
+            offset,
+            num_records,
+            header.record_size,
+            header.node_size,
+        )?;
+        out.extend(
+            records
+                .into_iter()
+                .filter(|r| cmp(&r.data) == Ordering::Equal),
+        );
+        return Ok(());
+    }
+    let rs = usize::from(header.record_size);
+    let node = read_internal_node(
+        file,
+        offset,
+        num_records,
+        depth,
+        header.record_size,
+        header.node_size,
+        offset_size,
+        max_leaf_nrec,
+    )?;
+    let nr = usize::from(num_records);
+    let mut order = Vec::with_capacity(nr);
+    for i in 0..nr {
+        order.push(cmp(node.record(i, rs)?));
+    }
+    // Child `i` holds the keys between record `i - 1` and record `i`: it can
+    // hold a match unless the record before it is already past the range or
+    // the record after it is still before it.
+    for (i, &(child_addr, child_nrec)) in node.children.iter().enumerate() {
+        let after_left = i == 0 || order[i - 1] != Ordering::Greater;
+        let before_right = i == nr || order[i] != Ordering::Less;
+        if after_left && before_right {
+            find_in_node(
+                file,
+                header,
+                to_usize(child_addr)?,
+                child_nrec,
+                depth - 1,
+                offset_size,
+                max_leaf_nrec,
+                cmp,
+                budget,
+                out,
+            )?;
+        }
+        if i < nr && order[i] == Ordering::Equal {
+            out.push(BTreeV2Record {
+                data: node.record(i, rs)?.to_vec(),
+            });
+        }
+    }
+    Ok(())
+}
+
+/// Most records a subtree whose root is at `depth` can hold (libhdf5's
+/// `cum_max_nrec`). See [`node_info`].
+fn cum_max_records(
+    node_size: u32,
+    record_size: u16,
+    offset_size: u8,
+    max_leaf_nrec: u64,
+    depth: u16,
+) -> u64 {
+    node_info_from_leaf(node_size, record_size, offset_size, max_leaf_nrec, depth)
+        .last()
+        .map_or(max_leaf_nrec, |n| n.cum_max_nrec)
+}
+
+/// Capacity of a B-tree v2 node at one depth, as libhdf5 computes it
+/// (`H5B2__hdr_init`'s `node_info`).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) struct NodeInfo {
+    /// Most records one node at this depth holds.
+    pub(crate) max_nrec: u64,
+    /// Most records a subtree rooted at this depth holds.
+    pub(crate) cum_max_nrec: u64,
+    /// Bytes a subtree's total record count takes in a pointer to a node
+    /// at this depth (0 for a leaf, whose count is its own).
+    pub(crate) cum_max_nrec_size: usize,
+}
+
+/// Node capacities for depths `0..=depth` (entry `d` for depth `d`): a leaf
+/// holds `max_nrec(0)` records; an internal node at depth `d` holds
+/// `max_nrec(d)` records and `max_nrec(d) + 1` subtrees of depth `d - 1`,
+/// where `max_nrec(d)` is what fits in a node once each record is paired
+/// with a child pointer of the width depth `d` needs (address, the child's
+/// record count in the width a *leaf's* maximum needs, and below the first
+/// internal level the child subtree's total in the width its maximum
+/// needs), with one pointer more than records.
+pub(crate) fn node_info(
+    node_size: u32,
+    record_size: u16,
+    offset_size: u8,
+    depth: u16,
+) -> Vec<NodeInfo> {
+    let max_leaf = max_records_leaf(node_size, record_size);
+    node_info_from_leaf(node_size, record_size, offset_size, max_leaf, depth)
+}
+
+fn node_info_from_leaf(
+    node_size: u32,
+    record_size: u16,
+    offset_size: u8,
+    max_leaf_nrec: u64,
+    depth: u16,
+) -> Vec<NodeInfo> {
+    // Internal node overhead: signature(4) + version(1) + type(1) + checksum(4).
+    const PREFIX: u64 = 10;
+    let nrec_width = bytes_for_max_records(max_leaf_nrec) as u64;
+    let mut info = Vec::with_capacity(usize::from(depth) + 1);
+    info.push(NodeInfo {
+        max_nrec: max_leaf_nrec,
+        cum_max_nrec: max_leaf_nrec,
+        cum_max_nrec_size: 0,
+    });
+    for d in 1..=depth {
+        let below = info[usize::from(d) - 1];
+        let ptr = u64::from(offset_size)
+            + nrec_width
+            + if d > 1 {
+                below.cum_max_nrec_size as u64
+            } else {
+                0
+            };
+        let max_nrec = u64::from(node_size)
+            .saturating_sub(PREFIX)
+            .saturating_sub(ptr)
+            / (u64::from(record_size) + ptr).max(1);
+        let cum = max_nrec
+            .saturating_add(1)
+            .saturating_mul(below.cum_max_nrec)
+            .saturating_add(max_nrec);
+        info.push(NodeInfo {
+            max_nrec,
+            cum_max_nrec: cum,
+            cum_max_nrec_size: bytes_for_max_records(cum),
+        });
+    }
+    info
 }
 
 #[cfg(test)]
@@ -512,9 +826,15 @@ mod tests {
         child_nrec: u64,
     ) -> Vec<u8> {
         let max_leaf = max_records_leaf(node_size, record_size);
-        let nrec_width = bytes_for_max_records(if depth == 1 { max_leaf } else { max_leaf * 2 });
+        let nrec_width = bytes_for_max_records(max_leaf);
         let total_width = if depth > 1 {
-            bytes_for_max_records(header_max_total_records(max_leaf, depth - 1))
+            bytes_for_max_records(cum_max_records(
+                node_size,
+                record_size,
+                8,
+                max_leaf,
+                depth - 1,
+            ))
         } else {
             0
         };
@@ -526,6 +846,8 @@ mod tests {
             buf.extend_from_slice(&child_nrec.to_le_bytes()[..nrec_width]);
             buf.resize(buf.len() + total_width, 0);
         }
+        let sum = crate::checksum::jenkins_lookup3(&buf);
+        buf.extend_from_slice(&sum.to_le_bytes());
         buf
     }
 
@@ -673,4 +995,18 @@ mod tests {
         let records = collect_btree_v2_records(&header, &hdr, 8, 8).unwrap();
         assert!(records.is_empty());
     }
+
+    #[test]
+    fn subtree_capacity_matches_libhdf5() {
+        // A link-name index (11-byte records, 512-byte nodes, 8-byte
+        // addresses): libhdf5's H5B2__hdr_init gives 45 records per leaf,
+        // then cum_max_nrec 1 149 at depth 1 and 26 449 at depth 2 — two
+        // bytes of subtree count in a depth-3 root's child pointers, where
+        // leaf_max^3 = 91 125 would need three.
+        let leaf = max_records_leaf(512, 11);
+        assert_eq!(leaf, 45);
+        assert_eq!(cum_max_records(512, 11, 8, leaf, 0), 45);
+        assert_eq!(cum_max_records(512, 11, 8, leaf, 1), 1_149);
+        assert_eq!(cum_max_records(512, 11, 8, leaf, 2), 26_449);
+    }
 }

crates/clawhdf5-format/src/btree_v2_write.rs

@@ -0,0 +1,503 @@
+//! Writing version-2 B-trees: a header (`BTHD`) and its nodes, leaves
+//! (`BTLF`) and, for more records than one leaf holds, internal nodes
+//! (`BTIN`) to any depth.
+//!
+//! Node capacities come from [`crate::btree_v2::node_info`], the arithmetic
+//! libhdf5 uses (`H5B2__hdr_init`) and the reader decodes pointers with, so
+//! the pointer widths the writer encodes are the ones every reader expects.
+
+use crate::addr::saturating_usize;
+#[cfg(not(feature = "std"))]
+use alloc::{format, vec, vec::Vec};
+
+use crate::btree_v2::{NodeInfo, bytes_for_max_records, node_info};
+use crate::checksum::jenkins_lookup3;
+use crate::error::FormatError;
+
+/// How a B-tree is laid out: its record type and node geometry, as the
+/// header records them.
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct BTreeV2Params {
+    /// Record type (5: link names, 6: link creation order, 8: attribute
+    /// names, 9: attribute creation order, 10/11: chunks).
+    pub(crate) tree_type: u8,
+    /// Bytes per node.
+    pub(crate) node_size: u32,
+    /// Bytes per record.
+    pub(crate) record_size: u16,
+    /// Split and merge percentages. The writer fills nodes itself; these
+    /// only tell libhdf5 when to split and merge as it modifies the tree.
+    pub(crate) split_percent: u8,
+    pub(crate) merge_percent: u8,
+}
+
+/// Size of a B-tree v2 header.
+pub(crate) fn header_size(offset_size: u8, length_size: u8) -> usize {
+    4 + 1 + 1 + 4 + 2 + 2 + 1 + 1 + offset_size as usize + 2 + length_size as usize + 4
+}
+
+/// Deepest tree the writer builds. Even at the smallest fan-out libhdf5's
+/// arithmetic allows, a few levels hold more records than any file could.
+const MAX_WRITE_DEPTH: u16 = 32;
+
+/// Write a B-tree v2 holding `records` (`record_size` bytes each,
+/// concatenated, already in the tree's key order) at `addr`: the header,
+/// then its nodes, each `node_size` bytes. No records gives a header with
+/// an undefined root.
+///
+/// The tree is as shallow as the node size allows: a single leaf when the
+/// records fit one, otherwise internal nodes above leaves. Records are
+/// spread evenly over each node's children, so every node but the root is
+/// at least about half full (above libhdf5's merge threshold, which is below
+/// half), and each node holds at most its depth's maximum.
+pub(crate) fn build_btree_v2(
+    p: BTreeV2Params,
+    records: &[u8],
+    addr: u64,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<u8>, FormatError> {
+    let rs = usize::from(p.record_size);
+    if rs == 0 || !records.len().is_multiple_of(rs) {
+        return Err(FormatError::SerializationError(format!(
+            "B-tree v2 records are {} bytes, not a multiple of the record size {rs}",
+            records.len()
+        )));
+    }
+    let n = (records.len() / rs) as u64;
+    let hdr_len = header_size(offset_size, length_size);
+
+    // The shallowest depth whose subtree can hold every record.
+    let mut info = node_info(p.node_size, p.record_size, offset_size, 0);
+    let max_leaf = info[0].max_nrec;
+    if max_leaf == 0 || max_leaf > u64::from(u16::MAX) {
+        return Err(FormatError::SerializationError(format!(
+            "a {}-byte B-tree v2 node holds {max_leaf} {}-byte records; \
+             a node holds 1 to 65535",
+            p.node_size, p.record_size
+        )));
+    }
+    let mut depth = 0u16;
+    while info[usize::from(depth)].cum_max_nrec < n {
+        depth += 1;
+        if depth > MAX_WRITE_DEPTH {
+            return Err(FormatError::SerializationError(format!(
+                "{n} records do not fit a B-tree v2 of {}-byte nodes",
+                p.node_size
+            )));
+        }
+        info = node_info(p.node_size, p.record_size, offset_size, depth);
+        let max = info[usize::from(depth)].max_nrec;
+        if max == 0 || max > u64::from(u16::MAX) {
+            return Err(FormatError::SerializationError(format!(
+                "a {}-byte B-tree v2 internal node holds {max} records; \
+                 a node holds 1 to 65535",
+                p.node_size
+            )));
+        }
+    }
+
+    let mut w = TreeWriter {
+        p,
+        records,
+        info: &info,
+        nrec_width: bytes_for_max_records(max_leaf),
+        offset_size,
+        first_node: addr + hdr_len as u64,
+        nodes: Vec::new(),
+    };
+    let root = (n > 0)
+        .then(|| w.node(depth, 0, saturating_usize(n)))
+        .transpose()?;
+
+    let mut out = Vec::with_capacity(hdr_len + w.nodes.len() * p.node_size as usize);
+    out.extend_from_slice(b"BTHD");
+    out.push(0); // version
+    out.push(p.tree_type);
+    out.extend_from_slice(&p.node_size.to_le_bytes());
+    out.extend_from_slice(&p.record_size.to_le_bytes());
+    out.extend_from_slice(&depth.to_le_bytes());
+    out.push(p.split_percent);
+    out.push(p.merge_percent);
+    match root {
+        Some(r) => push_uint(&mut out, r.addr, offset_size as usize),
+        None => out.extend(core::iter::repeat_n(0xFF, offset_size as usize)),
+    }
+    let root_nrec = root.map_or(0, |r| r.nrec);
+    out.extend_from_slice(&(root_nrec as u16).to_le_bytes());
+    push_uint(&mut out, n, length_size as usize);
+    let sum = jenkins_lookup3(&out);
+    out.extend_from_slice(&sum.to_le_bytes());
+    debug_assert_eq!(out.len(), hdr_len);
+    for node in &w.nodes {
+        out.extend_from_slice(node);
+    }
+    Ok(out)
+}
+
+/// A written node, as its parent points at it.
+#[derive(Debug, Clone, Copy)]
+struct NodeRef {
+    addr: u64,
+    /// Records in the node itself.
+    nrec: u64,
+    /// Records in the subtree it roots.
+    all_nrec: u64,
+}
+
+struct TreeWriter<'a> {
+    p: BTreeV2Params,
+    records: &'a [u8],
+    info: &'a [NodeInfo],
+    /// Width of a child's record count: what a leaf's maximum needs.
+    nrec_width: usize,
+    offset_size: u8,
+    /// Address of the first node (right after the header).
+    first_node: u64,
+    /// Nodes in file order (children before their parent).
+    nodes: Vec<Vec<u8>>,
+}
+
+impl TreeWriter<'_> {
+    fn record(&self, i: usize) -> &[u8] {
+        let rs = usize::from(self.p.record_size);
+        &self.records[i * rs..(i + 1) * rs]
+    }
+
+    fn push_node(&mut self, mut node: Vec<u8>) -> u64 {
+        // The checksum covers the node up to it, not the padding after.
+        let sum = jenkins_lookup3(&node);
+        node.extend_from_slice(&sum.to_le_bytes());
+        debug_assert!(node.len() <= self.p.node_size as usize);
+        node.resize(self.p.node_size as usize, 0);
+        let addr = self.first_node + self.nodes.len() as u64 * u64::from(self.p.node_size);
+        self.nodes.push(node);
+        addr
+    }
+
+    /// Write the subtree of `depth` holding records `first..first + n`.
+    fn node(&mut self, depth: u16, first: usize, n: usize) -> Result<NodeRef, FormatError> {
+        let rs = usize::from(self.p.record_size);
+        let mut node = Vec::with_capacity(self.p.node_size as usize);
+        if depth == 0 {
+            debug_assert!(n as u64 <= self.info[0].max_nrec);
+            node.extend_from_slice(b"BTLF");
+            node.push(0); // version
+            node.push(self.p.tree_type);
+            node.extend_from_slice(&self.records[first * rs..(first + n) * rs]);
+            let addr = self.push_node(node);
+            return Ok(NodeRef {
+                addr,
+                nrec: n as u64,
+                all_nrec: n as u64,
+            });
+        }
+
+        // As few children as hold the records, at least two, with the
+        // records spread evenly: `k` children and `k - 1` records between
+        // them.
+        let below = self.info[usize::from(depth) - 1].cum_max_nrec;
+        let k = (n as u64 + 1).div_ceil(below + 1).max(2);
+        let max = self.info[usize::from(depth)].max_nrec;
+        if k - 1 > max || (n as u64) < k - 1 + k {
+            return Err(FormatError::SerializationError(format!(
+                "cannot spread {n} B-tree v2 records over {k} children at depth {depth}"
+            )));
+        }
+        let k = saturating_usize(k);
+        let in_children = n - (k - 1);
+        let (base, extra) = (in_children / k, in_children % k);
+
+        let mut children = Vec::with_capacity(k);
+        let mut separators = Vec::with_capacity(k - 1);
+        let mut next = first;
+        for c in 0..k {
+            let m = base + usize::from(c < extra);
+            children.push(self.node(depth - 1, next, m)?);
+            next += m;
+            if c + 1 < k {
+                separators.push(next);
+                next += 1;
+            }
+        }
+        debug_assert_eq!(next, first + n);
+
+        node.extend_from_slice(b"BTIN");
+        node.push(0); // version
+        node.push(self.p.tree_type);
+        for &s in &separators {
+            node.extend_from_slice(self.record(s));
+        }
+        let total_width = if depth > 1 {
+            self.info[usize::from(depth) - 1].cum_max_nrec_size
+        } else {
+            0
+        };
+        for c in &children {
+            push_uint(&mut node, c.addr, self.offset_size as usize);
+            push_uint(&mut node, c.nrec, self.nrec_width);
+            if depth > 1 {
+                push_uint(&mut node, c.all_nrec, total_width);
+            }
+        }
+        let addr = self.push_node(node);
+        Ok(NodeRef {
+            addr,
+            nrec: (k - 1) as u64,
+            all_nrec: n as u64,
+        })
+    }
+}
+
+/// Append `v` as a `width`-byte little-endian integer.
+fn push_uint(buf: &mut Vec<u8>, v: u64, width: usize) {
+    let bytes = v.to_le_bytes();
+    buf.extend_from_slice(&bytes[..width.min(8)]);
+    buf.extend(vec![0u8; width.saturating_sub(8)]);
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+
+    fn params(node_size: u32, record_size: u16) -> BTreeV2Params {
+        BTreeV2Params {
+            tree_type: 5,
+            node_size,
+            record_size,
+            split_percent: 100,
+            merge_percent: 40,
+        }
+    }
+
+    /// `n` 11-byte records: a big-endian counter, so byte order is key order.
+    fn records(n: usize, rs: usize) -> Vec<u8> {
+        let mut out = Vec::with_capacity(n * rs);
+        for i in 0..n {
+            let mut r = vec![0u8; rs];
+            r[..8].copy_from_slice(&(i as u64).to_be_bytes());
+            out.extend_from_slice(&r);
+        }
+        out
+    }
+
+    fn roundtrip(node_size: u32, rs: u16, n: usize, os: u8, ls: u8) -> BTreeV2Header {
+        let recs = records(n, usize::from(rs));
+        let base = 4096u64;
+        let tree = build_btree_v2(params(node_size, rs), &recs, base, os, ls).unwrap();
+        let mut file = vec![0u8; base as usize];
+        file.extend_from_slice(&tree);
+        let hdr = BTreeV2Header::parse(&file, base as usize, os, ls).unwrap();
+        assert_eq!(hdr.total_records, n as u64);
+        let got = collect_btree_v2_records(&file, &hdr, os, ls).unwrap();
+        assert_eq!(got.len(), n);
+        let flat: Vec<u8> = got.into_iter().flat_map(|r| r.data).collect();
+        assert_eq!(flat, recs, "node {node_size} rs {rs} n {n}");
+        hdr
+    }
+
+    #[test]
+    fn one_leaf_then_deeper_trees_read_back_in_order() {
+        // 512-byte nodes of 11-byte records: 45 per leaf, 1149 at depth 1,
+        // 26 449 at depth 2.
+        let info = node_info(512, 11, 8, 3);
+        assert_eq!(
+            info.iter().map(|i| i.cum_max_nrec).collect::<Vec<_>>(),
+            [45, 1149, 26_449, 608_349]
+        );
+        for (n, depth) in [
+            (0, 0),
+            (1, 0),
+            (45, 0),
+            (46, 1),
+            (1149, 1),
+            (1150, 2),
+            (26_449, 2),
+            (26_450, 3),
+            (100_000, 3),
+        ] {
+            let hdr = roundtrip(512, 11, n, 8, 8);
+            assert_eq!(hdr.depth, depth, "{n} records");
+        }
+    }
+
+    #[test]
+    fn pointer_widths_follow_the_offset_and_length_sizes() {
+        for (os, ls) in [(4, 4), (8, 4), (4, 8), (2, 2)] {
+            roundtrip(512, 11, 5000, os, ls);
+        }
+        // Wide counts: a leaf of 2048 bytes / 9-byte records (226, one byte)
+        // and deeper subtree totals of three bytes.
+        roundtrip(2048, 9, 300_000, 8, 8);
+    }
+
+    #[test]
+    fn every_node_is_within_its_capacity_and_above_the_merge_threshold() {
+        let rs = 17u16;
+        let n = 70_000usize;
+        let info = node_info(512, rs, 8, 3);
+        let recs = records(n, usize::from(rs));
+        let tree = build_btree_v2(params(512, rs), &recs, 0, 8, 8).unwrap();
+        let hdr_len = header_size(8, 8);
+        let nodes = (tree.len() - hdr_len) / 512;
+        for i in 0..nodes {
+            let node = &tree[hdr_len + i * 512..hdr_len + (i + 1) * 512];
+            let sig = &node[..4];
+            if sig == b"BTLF" {
+                continue; // counts checked through the parents below
+            }
+            assert_eq!(sig, b"BTIN");
+        }
+        // Walk from the header: each child's count within [40%, 100%].
+        let hdr = BTreeV2Header::parse(&tree, 0, 8, 8).unwrap();
+        assert_eq!(hdr.depth, 3);
+        assert!(u64::from(hdr.num_records_in_root) <= info[3].max_nrec);
+        fn walk(tree: &[u8], addr: usize, nrec: usize, depth: usize, info: &[NodeInfo], rs: usize) {
+            if depth == 0 {
+                return;
+            }
+            let nrec_w = bytes_for_max_records(info[0].max_nrec);
+            let tot_w = if depth > 1 {
+                info[depth - 1].cum_max_nrec_size
+            } else {
+                0
+            };
+            let mut pos = addr + 6 + nrec * rs;
+            for _ in 0..=nrec {
+                let a = u64::from_le_bytes(tree[pos..pos + 8].try_into().unwrap()) as usize;
+                pos += 8;
+                let mut c = 0usize;
+                for b in 0..nrec_w {
+                    c |= usize::from(tree[pos + b]) << (8 * b);
+                }
+                pos += nrec_w + tot_w;
+                let max = info[depth - 1].max_nrec as usize;
+                assert!(c <= max && c * 100 > max * 40, "{c} of {max}");
+                walk(tree, a, c, depth - 1, info, rs);
+            }
+        }
+        walk(
+            &tree,
+            hdr.root_node_address as usize,
+            usize::from(hdr.num_records_in_root),
+            3,
+            &info,
+            usize::from(rs),
+        );
+        assert!(nodes > 0);
+    }
+
+    /// Descending to a key range finds exactly the records a full read
+    /// holds in it — runs of equal keys that straddle node boundaries
+    /// included — at every depth, and nothing for keys not in the tree.
+    #[test]
+    fn a_key_range_search_matches_a_full_scan() {
+        use crate::btree_v2::find_btree_v2_records;
+        use core::cmp::Ordering;
+        let rs = 11usize;
+        // Keys 0, 0, 0, 2, 2, 2, 4, ...: runs of three, odd keys missing.
+        for n in [1usize, 45, 46, 1150, 30_000] {
+            let mut recs = Vec::with_capacity(n * rs);
+            for i in 0..n {
+                let mut r = vec![0u8; rs];
+                r[..8].copy_from_slice(&((i / 3 * 2) as u64).to_be_bytes());
+                r[8..].copy_from_slice(&[(i % 3) as u8, 0, 0]);
+                recs.extend_from_slice(&r);
+            }
+            let base = 4096u64;
+            let tree = build_btree_v2(params(512, 11), &recs, base, 8, 8).unwrap();
+            let mut file = vec![0u8; base as usize];
+            file.extend_from_slice(&tree);
+            let hdr = BTreeV2Header::parse(&file, base as usize, 8, 8).unwrap();
+            let all = collect_btree_v2_records(&file, &hdr, 8, 8).unwrap();
+            let key = |r: &[u8]| u64::from_be_bytes(r[..8].try_into().unwrap());
+            let last = key(&all[n - 1].data);
+            let probes = (0..=last + 1).step_by(if n > 1000 { 37 } else { 1 });
+            for k in probes.chain([last, last + 1, u64::MAX]) {
+                let found =
+                    find_btree_v2_records(&file, &hdr, 8, &mut |r: &[u8]| key(r).cmp(&k)).unwrap();
+                let want: Vec<&[u8]> = all
+                    .iter()
+                    .map(|r| r.data.as_slice())
+                    .filter(|r| key(r) == k)
+                    .collect();
+                let got: Vec<&[u8]> = found.iter().map(|r| r.data.as_slice()).collect();
+                assert_eq!(got, want, "n {n} key {k}");
+                assert_eq!(
+                    got.len(),
+                    if k % 2 == 0 && k <= last {
+                        want.len()
+                    } else {
+                        0
+                    }
+                );
+            }
+            // Every record, or none, when the whole tree is in or out of range.
+            let every = find_btree_v2_records(&file, &hdr, 8, &mut |_| Ordering::Equal).unwrap();
+            assert_eq!(every.len(), n);
+            let none = find_btree_v2_records(&file, &hdr, 8, &mut |_| Ordering::Less).unwrap();
+            assert!(none.is_empty());
+        }
+    }
+
+    /// A two-level tree read through a `read_at`-only storage gives what
+    /// the slice gives — records, descents and errors — whole, truncated
+    /// at every length, and with each byte of its nodes flipped, and each
+    /// node costs one read.
+    #[test]
+    fn storage_reads_match_slice_reads() {
+        use crate::btree_v2::{
+            collect_btree_v2_records_in, find_btree_v2_records, find_btree_v2_records_in,
+        };
+        use crate::storage::CountingStorage;
+        let (rs, n, base) = (11usize, 120usize, 64usize);
+        let recs = records(n, rs);
+        let tree = build_btree_v2(params(128, 11), &recs, base as u64, 8, 8).unwrap();
+        let mut whole = vec![0u8; base];
+        whole.extend_from_slice(&tree);
+        let hdr = BTreeV2Header::parse(&whole, base, 8, 8).unwrap();
+        assert!(hdr.depth >= 1, "{hdr:?}");
+        let key = |r: &[u8]| u64::from_be_bytes(r[..8].try_into().unwrap());
+        let mut files = Vec::new();
+        for cut in base..=whole.len() {
+            files.push(whole[..cut].to_vec());
+        }
+        for at in base..whole.len() {
+            let mut bad = whole.clone();
+            bad[at] ^= 0x5a;
+            files.push(bad);
+        }
+        let mut ok = 0;
+        for f in &files {
+            let st = CountingStorage::new(f.clone());
+            let want_h = BTreeV2Header::parse(f, base, 8, 8);
+            let got_h = BTreeV2Header::parse_in(&st, base as u64, 8, 8);
+            assert_eq!(format!("{got_h:?}"), format!("{want_h:?}"));
+            // The nodes of the intact header, over each damaged file.
+            let want = collect_btree_v2_records(f, &hdr, 8, 8);
+            st.reset();
+            let got = collect_btree_v2_records_in(&st, &hdr, 8, 8);
+            assert_eq!(format!("{got:?}"), format!("{want:?}"));
+            if want.is_ok() {
+                ok += 1;
+                assert!(st.reads() <= 1 + n as u64 / 3, "{} reads", st.reads());
+            }
+            for k in [0u64, 7, 60, 119, 500] {
+                let want = find_btree_v2_records(f, &hdr, 8, &mut |r: &[u8]| key(r).cmp(&k));
+                let got = find_btree_v2_records_in(&st, &hdr, 8, &mut |r: &[u8]| key(r).cmp(&k));
+                assert_eq!(format!("{got:?}"), format!("{want:?}"));
+            }
+        }
+        assert!(ok > 1);
+    }
+
+    #[test]
+    fn a_node_too_small_or_too_big_is_an_error() {
+        assert!(build_btree_v2(params(16, 11), &records(1, 11), 0, 8, 8).is_err());
+        // A leaf with room for more than 65 535 records.
+        assert!(build_btree_v2(params(1 << 20, 11), &records(1, 11), 0, 8, 8).is_err());
+        // Records that are not whole.
+        assert!(build_btree_v2(params(512, 11), &[0u8; 12], 0, 8, 8).is_err());
+    }
+}

crates/clawhdf5-format/src/bulk_alloc.rs

@@ -0,0 +1,79 @@
+//! Large output buffers backed by transparent huge pages where the OS offers
+//! them.
+//!
+//! A fresh multi-megabyte `Vec` is mapped lazily by the kernel: the first
+//! write to each 4 KiB page takes a page fault, and the kernel zeroes the page
+//! before handing it over. For a 64 MiB read that is 16384 faults, and they
+//! cost far more than the copy that fills the buffer — single-threaded
+//! contiguous reads ran at about a quarter of h5py's speed because of them.
+//! numpy (so h5py) avoids this by asking for transparent huge pages
+//! (`madvise(MADV_HUGEPAGE)`) on every allocation of 4 MiB or more, which
+//! turns 512 faults into one; this module does the same.
+//!
+//! The advice only changes how the pages are backed, never their contents, so
+//! it is harmless when it cannot be honoured (THP disabled, not Linux, a
+//! region that is part of the heap): the buffer is then exactly what it would
+//! have been without it.
+
+#[cfg(not(feature = "std"))]
+use alloc::vec::Vec;
+
+/// Buffers smaller than this are left alone (numpy uses the same threshold).
+#[cfg(any(target_os = "linux", test))]
+pub(crate) const HUGE_PAGE_THRESHOLD: usize = 4 << 20;
+
+/// Advise the kernel to back `[ptr, ptr + len)` with transparent huge pages,
+/// when `len` is large enough to benefit. Call it before the first write so
+/// the faults happen at huge-page granularity.
+#[inline]
+pub(crate) fn advise_huge_pages(ptr: *const u8, len: usize) {
+    #[cfg(target_os = "linux")]
+    if len >= HUGE_PAGE_THRESHOLD {
+        const PAGE: usize = 4096;
+        let start = (ptr as usize).next_multiple_of(PAGE);
+        let end = (ptr as usize + len) & !(PAGE - 1);
+        if end > start {
+            // SAFETY: `[start, end)` lies inside an allocation of `len` bytes
+            // at `ptr` that the caller owns, and is page aligned as madvise
+            // requires. MADV_HUGEPAGE does not change the memory's contents or
+            // validity; on failure (EINVAL when THP is compiled out, etc.) the
+            // region is simply left as it was, so the result is ignored.
+            unsafe {
+                libc::madvise(start as *mut libc::c_void, end - start, libc::MADV_HUGEPAGE);
+            }
+        }
+    }
+    #[cfg(not(target_os = "linux"))]
+    let _ = (ptr, len);
+}
+
+/// `Vec::with_capacity(count)` for a buffer about to be filled in bulk, with
+/// huge-page advice when it is large (see the module docs).
+#[inline]
+pub(crate) fn vec_for_bulk<T>(count: usize) -> Vec<T> {
+    let v: Vec<T> = Vec::with_capacity(count);
+    advise_huge_pages(
+        v.as_ptr().cast::<u8>(),
+        v.capacity().saturating_mul(core::mem::size_of::<T>()),
+    );
+    v
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn bulk_vec_is_an_ordinary_vec() {
+        for count in [0usize, 1, 1000, HUGE_PAGE_THRESHOLD / 4 + 3] {
+            let mut v: Vec<u32> = vec_for_bulk(count);
+            assert!(v.capacity() >= count);
+            v.extend((0..count as u32).map(|i| i.wrapping_mul(2654435761)));
+            assert!(
+                v.iter()
+                    .enumerate()
+                    .all(|(i, &x)| x == (i as u32).wrapping_mul(2654435761))
+            );
+        }
+    }
+}

crates/clawhdf5-format/src/checksum.rs

@@ -14,6 +14,45 @@ pub fn jenkins_lookup3(data: &[u8]) -> u32 {
     hashlittle(data, 0)
 }
 
+/// HDF5's Fletcher-32 checksum, as the Fletcher-32 I/O filter (filter id 3)
+/// stores it after each chunk.
+///
+/// A line-for-line port of `H5_checksum_fletcher32` (H5checksum.c, libhdf5
+/// 1.8 through 1.14): big-endian 16-bit words summed in blocks of 360, each
+/// sum reduced after a block by the ones'-complement fold
+/// `(s & 0xffff) + (s >> 16)` rather than `% 65535`, an odd trailing byte
+/// taken as the high byte of a last word, and a final fold of both sums.
+/// The fold and `% 65535` differ whenever a sum is a non-zero multiple of
+/// 65535: the fold leaves 0xffff where the modulo gives 0, so the two
+/// disagree on about one chunk in 32768 and libhdf5 rejects the other's
+/// checksum. This must stay the only implementation.
+pub fn fletcher32(data: &[u8]) -> u32 {
+    let mut sum1: u32 = 0;
+    let mut sum2: u32 = 0;
+    // 360 words keep both sums inside 32 bits between folds (the bound
+    // libhdf5 uses: after a fold sum1 < 0x10200, so sum2 stays below
+    // 360 * 361 / 2 * 0xffff + 360 * 0x10200 + 0x1fffe < 2^32). The adds wrap
+    // like the C unsigned arithmetic all the same.
+    let (words, odd) = data.as_chunks::<2>();
+    for block in words.chunks(360) {
+        for w in block {
+            sum1 = sum1.wrapping_add((u32::from(w[0]) << 8) | u32::from(w[1]));
+            sum2 = sum2.wrapping_add(sum1);
+        }
+        sum1 = (sum1 & 0xffff) + (sum1 >> 16);
+        sum2 = (sum2 & 0xffff) + (sum2 >> 16);
+    }
+    if let [last] = odd {
+        sum1 = sum1.wrapping_add(u32::from(*last) << 8);
+        sum2 = sum2.wrapping_add(sum1);
+        sum1 = (sum1 & 0xffff) + (sum1 >> 16);
+        sum2 = (sum2 & 0xffff) + (sum2 >> 16);
+    }
+    sum1 = (sum1 & 0xffff) + (sum1 >> 16);
+    sum2 = (sum2 & 0xffff) + (sum2 >> 16);
+    (sum2 << 16) | sum1
+}
+
 /// Compute CRC32 (IEEE / ISO 3309) over data.
 ///
 /// When the `fast-checksum` feature is enabled, this uses hardware CRC32
@@ -207,6 +246,19 @@ fn hashlittle(data: &[u8], initval: u32) -> u32 {
 mod tests {
     use super::*;
 
+    /// Values of libhdf5's `H5_checksum_fletcher32` (h5py 3.x's bundled
+    /// libhdf5, called through ctypes). The first three are sums that are
+    /// multiples of 65535, where `% 65535` gave 0 instead of 0xffff.
+    #[test]
+    fn fletcher32_matches_libhdf5() {
+        assert_eq!(fletcher32(&[0x00, 0x01, 0xff, 0xfe]), 0x0001_ffff);
+        assert_eq!(fletcher32(&[0xff; 720]), 0xffff_ffff);
+        assert_eq!(fletcher32(&[0xff; 721]), 0xff00_ff00);
+        assert_eq!(fletcher32(&[0xff; 1441]), 0xff00_ff00);
+        assert_eq!(fletcher32(&[]), 0);
+        assert_eq!(fletcher32(&[7]), 0x0700_0700);
+    }
+
     #[test]
     fn empty_input() {
         // Empty input should return the initial state after no mixing

crates/clawhdf5-format/src/chunk_index.rs

@@ -18,6 +18,7 @@ use alloc::collections::BTreeMap;
 #[cfg(feature = "std")]
 use std::collections::HashMap;
 
+use crate::addr::to_usize;
 use crate::chunk_cache::ChunkCoord;
 use crate::chunked_read::ChunkInfo;
 
@@ -167,7 +168,15 @@ impl ChunkLayout {
 
         for (_coord, ci) in index.iter() {
             let coord: ChunkCoord = ci.offsets.iter().take(rank).copied().collect();
-            let chunk_offsets: Vec<usize> = coord.iter().map(|&o| o as usize).collect();
+            // `ds_dims` are `usize`: a chunk at an offset past `usize::MAX`
+            // (only on a 32-bit target) lies outside the dataset.
+            let Ok(chunk_offsets) = coord
+                .iter()
+                .map(|&o| to_usize(o))
+                .collect::<Result<Vec<usize>, _>>()
+            else {
+                continue;
+            };
 
             let copies = if rank == 0 {
                 // Scalar dataset — single copy

crates/clawhdf5-format/src/chunked_write.rs

@@ -3,19 +3,22 @@
 #[cfg(not(feature = "std"))]
 extern crate alloc;
 
+use crate::addr::saturating_usize;
 #[cfg(not(feature = "std"))]
 use alloc::{format, vec, vec::Vec};
 
+use crate::btree_v2_write::{BTreeV2Params, build_btree_v2};
 use crate::checksum::jenkins_lookup3;
 use crate::chunk_cache::{CACHE_LINE_SIZE, align_to_cache_line};
 use crate::chunk_grid::ChunkGrid;
 use crate::ea_writer;
 use crate::error::FormatError;
 use crate::filter_pipeline::{
-    FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZ4, FILTER_PCODEC, FILTER_PCODEC_NAME,
+    FILTER_BITSHUFFLE, FILTER_BLOSC, FILTER_BZIP2, FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZ4,
-    FILTER_SHUFFLE, FILTER_ZSTD, FilterDescription, FilterPipeline,
+    FILTER_LZF, FILTER_PCODEC, FILTER_PCODEC_NAME, FILTER_SHUFFLE, FILTER_ZSTD, FilterDescription,
+    FilterPipeline,
 };
-use crate::filters::compress_chunk;
+use crate::filters::compress_chunk_masked;
 /// Round a file offset up to the next cache-line boundary.
 ///
 /// This ensures chunk data starts at an address that is a multiple of the
@@ -48,6 +51,167 @@ pub struct ChunkOptions {
     /// Pcodec lossless numerical compression. Private, unregistered filter
     /// ID [`FILTER_PCODEC`] (480): only clawhdf5 can read it.
     pub pcodec: bool,
+    /// A plugin compression filter (LZF, ...). Takes priority over the
+    /// codecs above. Each needs its cargo feature to be written.
+    pub plugin: Option<PluginFilter>,
+}
+
+/// A compression filter from the common HDF5 plugin set, written in the
+/// format the libhdf5 plugin (h5py / hdf5plugin) reads.
+#[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
+pub enum PluginFilter {
+    /// LZF (filter 32000), h5py's built-in `compression="lzf"`. Needs the
+    /// `lzf` feature.
+    Lzf,
+    /// Bitshuffle (filter 32008): a bit transpose of each block of
+    /// `block_size` elements (0 = bitshuffle's default, else a multiple of
+    /// 8), optionally compressed. Needs the `bitshuffle` feature.
+    Bitshuffle {
+        /// Block size in elements; 0 for the default.
+        block_size: u32,
+        /// Compression after the transpose.
+        compression: BitshuffleCompression,
+    },
+    /// bzip2 (filter 307) at block size `level` (1-9). Needs the `bzip2`
+    /// feature.
+    Bzip2 {
+        /// Block size 1-9 (9 = hdf5plugin's default).
+        level: u32,
+    },
+    /// Blosc 1 (filter 32001): `codec` at `level` (0-9; 0 stores), after
+    /// `shuffle`. Needs the `blosc` feature.
+    Blosc {
+        /// The codec inside the Blosc frame.
+        codec: BloscCodec,
+        /// Compression level 0-9 (0 stores the data uncompressed).
+        level: u32,
+        /// The shuffle Blosc applies first.
+        shuffle: BloscShuffle,
+    },
+}
+
+/// The codec inside a Blosc frame that clawhdf5 can write. (It reads
+/// BloscLZ too, but cannot write it.)
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum BloscCodec {
+    /// LZ4.
+    Lz4,
+    /// Snappy.
+    Snappy,
+    /// Zlib, at the Blosc level.
+    Zlib,
+    /// Zstandard (clawhdf5's pure-Rust encoder has one level, about zstd 1).
+    Zstd,
+}
+
+/// The shuffle Blosc applies before compressing.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum BloscShuffle {
+    /// None.
+    None,
+    /// Byte shuffle (Blosc's default).
+    Byte,
+    /// Bit shuffle.
+    Bit,
+}
+
+/// What bitshuffle compresses its blocks with.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum BitshuffleCompression {
+    /// Transpose only.
+    None,
+    /// LZ4 (bitshuffle's `cname="lz4"`, the common choice).
+    Lz4,
+    /// Zstandard. clawhdf5's pure-Rust encoder has a single level (about
+    /// zstd's level 1); `level` is recorded in the file for other writers.
+    Zstd {
+        /// Level recorded in `cd_values[5]`.
+        level: u32,
+    },
+}
+
+impl PluginFilter {
+    /// Whether the filter reorders bytes itself, so the automatic shuffle
+    /// pre-filter would only get in its way.
+    fn shuffles_itself(&self) -> bool {
+        match self {
+            PluginFilter::Lzf => false,
+            PluginFilter::Bitshuffle { .. } => true,
+            PluginFilter::Bzip2 { .. } => false,
+            PluginFilter::Blosc { .. } => true,
+        }
+    }
+
+    /// The pipeline entry for this filter. `chunk_bytes` is one chunk's
+    /// uncompressed size (0 if unknown).
+    fn description(&self, element_size: u32, chunk_bytes: u32) -> FilterDescription {
+        match self {
+            // h5py's lzf_set_local: filter version, liblzf version, chunk
+            // size in bytes. Optional, as h5py flags it: a chunk the filter
+            // cannot shrink may then be stored unfiltered.
+            PluginFilter::Lzf => FilterDescription {
+                filter_id: FILTER_LZF,
+                name: Some("lzf".into()),
+                flags: 1,
+                client_data: vec![4, 0x0105, chunk_bytes],
+            },
+            // bshuf_h5_set_local: version 0.4, element size, block size,
+            // compression (0 none, 2 LZ4, 3 Zstandard), Zstandard level.
+            // hdf5-blosc's blosc_set_local: filter revision 2, Blosc format
+            // 2, type size, chunk size, then level, shuffle, compressor.
+            PluginFilter::Blosc {
+                codec,
+                level,
+                shuffle,
+            } => FilterDescription {
+                filter_id: FILTER_BLOSC,
+                name: Some("blosc".into()),
+                flags: 1,
+                client_data: vec![
+                    2,
+                    2,
+                    element_size,
+                    chunk_bytes,
+                    (*level).min(9),
+                    match shuffle {
+                        BloscShuffle::None => 0,
+                        BloscShuffle::Byte => 1,
+                        BloscShuffle::Bit => 2,
+                    },
+                    match codec {
+                        BloscCodec::Lz4 => 1,
+                        BloscCodec::Snappy => 3,
+                        BloscCodec::Zlib => 4,
+                        BloscCodec::Zstd => 5,
+                    },
+                ],
+            },
+            PluginFilter::Bzip2 { level } => FilterDescription {
+                filter_id: FILTER_BZIP2,
+                name: Some("bzip2".into()),
+                flags: 1,
+                client_data: vec![(*level).clamp(1, 9)],
+            },
+            PluginFilter::Bitshuffle {
+                block_size,
+                compression,
+            } => {
+                let mut cd = vec![0, 4, element_size, *block_size];
+                match compression {
+                    BitshuffleCompression::None => cd.push(0),
+                    BitshuffleCompression::Lz4 => cd.push(2),
+                    BitshuffleCompression::Zstd { level } => cd.extend([3, *level]),
+                }
+                FilterDescription {
+                    filter_id: FILTER_BITSHUFFLE,
+                    name: Some("bitshuffle; see https://github.com/kiyo-masui/bitshuffle".into()),
+                    flags: 1,
+                    client_data: cd,
+                }
+            }
+        }
+    }
 }
 
 /// Largest chunk the automatic choice produces, in bytes.
@@ -92,14 +256,33 @@ impl ChunkOptions {
             || self.lz4
             || self.zstd_level.is_some()
             || self.pcodec
+            || self.plugin.is_some()
     }
 
     /// Build a FilterPipeline from the options.
     pub fn build_pipeline(&self, element_size: u32) -> Option<FilterPipeline> {
+        self.build_pipeline_for_chunk(element_size, 0)
+    }
+
+    /// Build a FilterPipeline for chunks of `chunk_bytes` uncompressed bytes
+    /// (0 if unknown). Some plugin filters record the chunk size in their
+    /// client data.
+    pub fn build_pipeline_for_chunk(
+        &self,
+        element_size: u32,
+        chunk_bytes: u32,
+    ) -> Option<FilterPipeline> {
         let mut filters = Vec::new();
 
-        let has_compression =
+        let plugin_shuffles = self
-            self.deflate_level.is_some() || self.zstd_level.is_some() || self.lz4 || self.pcodec;
+            .plugin
+            .as_ref()
+            .is_some_and(PluginFilter::shuffles_itself);
+        let has_compression = self.deflate_level.is_some()
+            || self.zstd_level.is_some()
+            || self.lz4
+            || self.pcodec
+            || (self.plugin.is_some() && !plugin_shuffles);
 
         // Shuffle before compression. Applied if explicitly requested OR if compression
         // is active and the caller hasn't disabled it — matches h5py default behavior
@@ -113,8 +296,11 @@ impl ChunkOptions {
             });
         }
 
-        // Compression filters (mutually exclusive, priority: pcodec > zstd > lz4 > deflate)
+        // Compression filters (mutually exclusive, priority: plugin > pcodec >
-        if self.pcodec {
+        // zstd > lz4 > deflate)
+        if let Some(plugin) = &self.plugin {
+            filters.push(plugin.description(element_size, chunk_bytes));
+        } else if self.pcodec {
             filters.push(FilterDescription {
                 filter_id: FILTER_PCODEC,
                 name: Some(FILTER_PCODEC_NAME.into()),
@@ -229,18 +415,18 @@ pub fn split_into_chunks(
     // Dataset strides (row-major)
     let mut ds_strides = vec![1usize; rank];
     for i in (0..rank.saturating_sub(1)).rev() {
-        ds_strides[i] = ds_strides[i + 1] * shape[i + 1] as usize;
+        ds_strides[i] = ds_strides[i + 1] * saturating_usize(shape[i + 1]);
     }
 
     // Chunk strides
     let mut chunk_strides = vec![1usize; rank];
     for i in (0..rank.saturating_sub(1)).rev() {
-        chunk_strides[i] = chunk_strides[i + 1] * chunk_dims[i + 1] as usize;
+        chunk_strides[i] = chunk_strides[i + 1] * saturating_usize(chunk_dims[i + 1]);
     }
 
-    let chunk_total_elements: usize = chunk_dims.iter().map(|&d| d as usize).product();
+    let chunk_total_elements: usize = chunk_dims.iter().map(|&d| saturating_usize(d)).product();
 
-    let mut result = Vec::with_capacity(total_chunks as usize);
+    let mut result = Vec::with_capacity(saturating_usize(total_chunks));
 
     for linear_idx in 0..total_chunks {
         // Convert linear index to chunk grid coordinates
@@ -268,8 +454,8 @@ pub fn split_into_chunks(
                 let coord_in_chunk = remaining_idx / chunk_strides[d];
                 remaining_idx %= chunk_strides[d];
 
-                let global_coord = offsets[d] as usize + coord_in_chunk;
+                let global_coord = saturating_usize(offsets[d]) + coord_in_chunk;
-                if global_coord >= shape[d] as usize {
+                if global_coord >= saturating_usize(shape[d]) {
                     out_of_bounds = true;
                     break;
                 }
@@ -304,7 +490,12 @@ pub fn split_into_chunks(
 #[cfg(feature = "parallel")]
 const PARALLEL_COMPRESS_THRESHOLD: usize = 2;
 
-/// Compress all chunks, using parallel compression when beneficial.
+/// Compress all chunks, using parallel compression when beneficial, and
+/// return each chunk's stored bytes with its filter mask.
+///
+/// Chunks run through the pipeline as libhdf5 runs them
+/// ([`compress_chunk_masked`]): an optional filter that fails — LZF or Blosc
+/// output no smaller than its input — is skipped and its mask bit set.
 ///
 /// With the `parallel` feature and more than [`PARALLEL_COMPRESS_THRESHOLD`]
 /// filtered chunks, compression runs across rayon threads; otherwise it is
@@ -314,7 +505,7 @@ fn compress_all_chunks(
     chunks: &[(Vec<u64>, Vec<u8>)],
     pipeline: &Option<FilterPipeline>,
     element_size: u32,
-) -> Result<Vec<Vec<u8>>, FormatError> {
+) -> Result<Vec<(Vec<u8>, u32)>, FormatError> {
     #[cfg(feature = "parallel")]
     {
         if let Some(pl) = pipeline
@@ -323,7 +514,7 @@ fn compress_all_chunks(
             use rayon::prelude::*;
             return chunks
                 .par_iter()
-                .map(|(_offsets, chunk_bytes)| compress_chunk(chunk_bytes, pl, element_size))
+                .map(|(_offsets, chunk_bytes)| compress_chunk_masked(chunk_bytes, pl, element_size))
                 .collect();
         }
     }
@@ -333,9 +524,9 @@ fn compress_all_chunks(
         .iter()
         .map(|(_offsets, chunk_bytes)| {
             if let Some(pl) = pipeline {
-                compress_chunk(chunk_bytes, pl, element_size)
+                compress_chunk_masked(chunk_bytes, pl, element_size)
             } else {
-                Ok(chunk_bytes.clone())
+                Ok((chunk_bytes.clone(), 0))
             }
         })
         .collect()
@@ -383,39 +574,7 @@ fn serialize_v4_single_chunk(
     let ndims = chunk_dims.len() as u8 + 1;
     buf.push(ndims);
 
-    // dim_size_encoded_length: how many bytes per dimension
+    push_v4_chunk_dims(&mut buf, chunk_dims, element_size);
-    // We need to figure out the minimum encoding width
-    let max_dim = chunk_dims
-        .iter()
-        .map(|&d| d as u64)
-        .chain(core::iter::once(element_size as u64))
-        .max()
-        .unwrap_or(1);
-    let dim_encoded_len: u8 = if max_dim <= 0xFF {
-        1
-    } else if max_dim <= 0xFFFF {
-        2
-    } else {
-        4
-    };
-    buf.push(dim_encoded_len);
-
-    // dimension sizes (chunk dims + element size)
-    for &d in chunk_dims {
-        match dim_encoded_len {
-            1 => buf.push(d as u8),
-            2 => buf.extend_from_slice(&(d as u16).to_le_bytes()),
-            4 => buf.extend_from_slice(&d.to_le_bytes()),
-            _ => {}
-        }
-    }
-    // Element size dimension
-    match dim_encoded_len {
-        1 => buf.push(element_size as u8),
-        2 => buf.extend_from_slice(&(element_size as u16).to_le_bytes()),
-        4 => buf.extend_from_slice(&element_size.to_le_bytes()),
-        _ => {}
-    }
 
     // chunk index type = 1 (single chunk)
     buf.push(1);
@@ -465,6 +624,25 @@ fn serialize_v4_fixed_array(
 
 /// The part of a v4 chunked layout message before the chunk index type:
 /// version, class, flags and the chunk dimensions (plus the element size).
+/// Append a v4 layout's dimension width and its dimensions (the chunk
+/// dimensions, then the element size). Each takes the fewest bytes that hold
+/// the largest, as libhdf5 computes it (`H5D__chunk_set_sizes`:
+/// `(log2(dim) + 8) / 8`); HDF5 2.0.0 refuses any other width.
+pub(crate) fn push_v4_chunk_dims(buf: &mut Vec<u8>, chunk_dims: &[u32], element_size: u32) {
+    let max_dim = chunk_dims
+        .iter()
+        .copied()
+        .chain(core::iter::once(element_size))
+        .max()
+        .unwrap_or(1)
+        .max(1);
+    let width = (32 - max_dim.leading_zeros()).div_ceil(8) as usize;
+    buf.push(width as u8);
+    for &d in chunk_dims.iter().chain(core::iter::once(&element_size)) {
+        buf.extend_from_slice(&d.to_le_bytes()[..width]);
+    }
+}
+
 fn layout_v4_chunked_prefix(chunk_dims: &[u32], element_size: u32) -> Vec<u8> {
     let mut buf = Vec::new();
     buf.push(4); // version
@@ -476,35 +654,7 @@ fn layout_v4_chunked_prefix(chunk_dims: &[u32], element_size: u32) -> Vec<u8> {
     let ndims = chunk_dims.len() as u8 + 1;
     buf.push(ndims);
 
-    let max_dim = chunk_dims
+    push_v4_chunk_dims(&mut buf, chunk_dims, element_size);
-        .iter()
-        .map(|&d| d as u64)
-        .chain(core::iter::once(element_size as u64))
-        .max()
-        .unwrap_or(1);
-    let dim_encoded_len: u8 = if max_dim <= 0xFF {
-        1
-    } else if max_dim <= 0xFFFF {
-        2
-    } else {
-        4
-    };
-    buf.push(dim_encoded_len);
-
-    for &d in chunk_dims {
-        match dim_encoded_len {
-            1 => buf.push(d as u8),
-            2 => buf.extend_from_slice(&(d as u16).to_le_bytes()),
-            4 => buf.extend_from_slice(&d.to_le_bytes()),
-            _ => {}
-        }
-    }
-    match dim_encoded_len {
-        1 => buf.push(element_size as u8),
-        2 => buf.extend_from_slice(&(element_size as u16).to_le_bytes()),
-        4 => buf.extend_from_slice(&element_size.to_le_bytes()),
-        _ => {}
-    }
     buf
 }
 
@@ -654,8 +804,10 @@ pub fn build_fixed_array_at(
 /// writer passes eliminates the double-compression that the two-pass layout
 /// algorithm previously performed.
 pub struct PrecompressedChunks {
-    /// Per-chunk: (raw_size_bytes, compressed_bytes).
+    /// Per-chunk: (raw_size_bytes, stored_bytes, filter_mask). Bit `i` of
-    pub chunks: Vec<(u64, Vec<u8>)>,
+    /// the mask is set when filter `i` was skipped (an optional filter that
+    /// failed); 0 for every chunk of an unfiltered dataset.
+    pub chunks: Vec<(u64, Vec<u8>, u32)>,
     pub has_filters: bool,
     pub element_size: usize,
     pub shape: Vec<u64>,
@@ -675,7 +827,12 @@ pub fn precompress_chunks(
     element_size: usize,
     options: &ChunkOptions,
 ) -> Result<PrecompressedChunks, FormatError> {
-    let pipeline = options.build_pipeline(element_size as u32);
+    let chunk_bytes = chunk_dims
+        .iter()
+        .try_fold(element_size as u64, |acc, &d| acc.checked_mul(d))
+        .and_then(|b| u32::try_from(b).ok())
+        .unwrap_or(0);
+    let pipeline = options.build_pipeline_for_chunk(element_size as u32, chunk_bytes);
     let has_filters = pipeline.is_some();
     let pipeline_message = pipeline.as_ref().map(|pl| pl.serialize());
 
@@ -685,7 +842,7 @@ pub fn precompress_chunks(
     let chunks = raw_chunks
         .into_iter()
         .zip(compressed)
-        .map(|((_offsets, raw_bytes), c)| (raw_bytes.len() as u64, c))
+        .map(|((_offsets, raw_bytes), (c, mask))| (raw_bytes.len() as u64, c, mask))
         .collect();
 
     Ok(PrecompressedChunks {
@@ -718,7 +875,7 @@ pub fn build_chunked_data_from_precompressed(
     let mut data_buf = Vec::new();
     let mut written_chunks = Vec::with_capacity(num_chunks);
 
-    for (raw_size, compressed) in &pre.chunks {
+    for (raw_size, compressed, filter_mask) in &pre.chunks {
         let aligned_offset = align_to_cache_line(data_buf.len());
         if aligned_offset > data_buf.len() {
             data_buf.resize(aligned_offset, 0u8);
@@ -730,7 +887,7 @@ pub fn build_chunked_data_from_precompressed(
             address,
             compressed_size,
             raw_size: *raw_size,
-            filter_mask: 0,
+            filter_mask: *filter_mask,
         });
     }
 
@@ -767,7 +924,7 @@ pub fn build_chunked_data_from_precompressed(
             } else {
                 None
             };
-            let filter_mask = if pre.has_filters { Some(0u32) } else { None };
+            let filter_mask = pre.has_filters.then_some(written_chunks[0].filter_mask);
             serialize_v4_single_chunk(
                 &chunk_dims_u32,
                 chunk_addr,
@@ -887,7 +1044,7 @@ impl ChunkIndexPlan {
                     Ok(Self::SingleChunk)
                 } else {
                     let grid = ChunkGrid::fixed_array(shape, Some(max), chunk_dims)?;
-                    Ok(Self::FixedArray(grid, nslots as usize))
+                    Ok(Self::FixedArray(grid, saturating_usize(nslots)))
                 }
             }
             1 => Ok(Self::ExtensibleArray(ChunkGrid::extensible_array(
@@ -957,11 +1114,12 @@ const BT2_CHUNK_FILTERED: u8 = 11;
 ///
 /// `records` are `(scaled coordinates, chunk)` in lexicographic order of the
 /// coordinates, which is the order the library's comparator
-/// (`H5VM_vector_cmp_u`) keeps them in. The tree is a single leaf: the
+/// (`H5VM_vector_cmp_u`) keeps them in. Up to 65 535 chunks go in a single
-/// library's 2048-byte node when the records fit, otherwise a leaf node
+/// leaf: the library's 2048-byte node when the records fit, otherwise a leaf
-/// sized to hold them all (the root's record count is 16-bit, so at most
+/// node sized to hold them all (the layout the writer has always used, kept
-/// 65535 chunks). Returns the bytes and the node size the layout message
+/// so those files do not change). More chunks get the library's 2048-byte
-/// must record.
+/// nodes with internal nodes above the leaves. Returns the bytes and the
+/// node size the layout message must record.
 fn build_btree_v2_chunk_index_at(
     rank: usize,
     records: &[(Vec<u64>, &WrittenChunk)],
@@ -971,73 +1129,49 @@ fn build_btree_v2_chunk_index_at(
     base_address: u64,
 ) -> Result<(Vec<u8>, u32), FormatError> {
     let os = offset_size as usize;
-    let nrec = u16::try_from(records.len()).map_err(|_| {
-        FormatError::ChunkedReadError(
-            "more than 65535 chunks with more than one unlimited dimension: \
-             use larger chunks"
-                .into(),
-        )
-    })?;
     let chunk_size_bytes = has_filters.then(|| {
         let slots: Vec<Option<WrittenChunk>> =
             records.iter().map(|(_, c)| Some((*c).clone())).collect();
         filtered_chunk_size_len(&slots)
     });
     let record_size = os + chunk_size_bytes.map_or(0, |n| n + 4) + 8 * rank;
+    let record_size_u16 = u16::try_from(record_size)
+        .map_err(|_| FormatError::Overflow("B-tree v2 record size".into()))?;
+    let node_size = if records.len() <= usize::from(u16::MAX) {
         // Leaf: signature, version, type, records, checksum.
         let leaf_len = 4 + 1 + 1 + records.len() * record_size + 4;
-    let node_size = u32::try_from(leaf_len)
+        u32::try_from(leaf_len)
             .map_err(|_| FormatError::Overflow("B-tree v2 leaf size".into()))?
-        .max(BT2_NODE_SIZE);
+            .max(BT2_NODE_SIZE)
+    } else {
+        BT2_NODE_SIZE
+    };
     let tree_type = if has_filters {
         BT2_CHUNK_FILTERED
     } else {
         BT2_CHUNK_UNFILTERED
     };
 
-    let hdr_len = 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1 + os + 2 + length_size as usize + 4;
+    let mut flat = Vec::with_capacity(records.len() * record_size);
-    let leaf_address = base_address + hdr_len as u64;
-
-    let mut out = Vec::with_capacity(hdr_len + node_size as usize);
-    out.extend_from_slice(b"BTHD");
-    out.push(0); // version
-    out.push(tree_type);
-    out.extend_from_slice(&node_size.to_le_bytes());
-    out.extend_from_slice(&(record_size as u16).to_le_bytes());
-    out.extend_from_slice(&0u16.to_le_bytes()); // depth
-    out.push(BT2_SPLIT_PERCENT);
-    out.push(BT2_MERGE_PERCENT);
-    if records.is_empty() {
-        out.extend(core::iter::repeat_n(0xFF, os));
-    } else {
-        push_addr(&mut out, leaf_address, offset_size);
-    }
-    out.extend_from_slice(&nrec.to_le_bytes());
-    match length_size {
-        4 => out.extend_from_slice(&(records.len() as u32).to_le_bytes()),
-        _ => out.extend_from_slice(&(records.len() as u64).to_le_bytes()),
-    }
-    let sum = jenkins_lookup3(&out);
-    out.extend_from_slice(&sum.to_le_bytes());
-    debug_assert_eq!(out.len(), hdr_len);
-    if records.is_empty() {
-        return Ok((out, node_size));
-    }
-
-    let leaf_start = out.len();
-    out.extend_from_slice(b"BTLF");
-    out.push(0); // version
-    out.push(tree_type);
     for (scaled, chunk) in records {
-        push_index_element(&mut out, Some(chunk), offset_size, chunk_size_bytes);
+        push_index_element(&mut flat, Some(chunk), offset_size, chunk_size_bytes);
         for &c in scaled {
-            out.extend_from_slice(&c.to_le_bytes());
+            flat.extend_from_slice(&c.to_le_bytes());
         }
     }
-    let sum = jenkins_lookup3(&out[leaf_start..]);
+    let out = build_btree_v2(
-    out.extend_from_slice(&sum.to_le_bytes());
+        BTreeV2Params {
-    // The library reads whole nodes; pad the leaf out to the node size.
+            tree_type,
-    out.resize(leaf_start + node_size as usize, 0);
+            node_size,
+            record_size: record_size_u16,
+            split_percent: BT2_SPLIT_PERCENT,
+            merge_percent: BT2_MERGE_PERCENT,
+        },
+        &flat,
+        base_address,
+        offset_size,
+        length_size,
+    )?;
     Ok((out, node_size))
 }
 
@@ -1125,7 +1259,7 @@ pub fn write_selection_to_buffer(
             let rank = dims.len();
             let mut ds_strides = vec![1usize; rank];
             for i in (0..rank.saturating_sub(1)).rev() {
-                ds_strides[i] = ds_strides[i + 1] * dims[i + 1] as usize;
+                ds_strides[i] = ds_strides[i + 1] * saturating_usize(dims[i + 1]);
             }
 
             let mut src_offset = 0usize;
@@ -1175,7 +1309,7 @@ pub fn write_selection_to_buffer(
                                 buffer,
                                 new_data,
                                 src_offset,
-                                current_ds_offset + coord as usize * ds_strides[d],
+                                current_ds_offset + saturating_usize(coord) * ds_strides[d],
                             );
                         }
                     }
@@ -1202,14 +1336,14 @@ pub fn write_selection_to_buffer(
             let rank = dims.len();
             let mut ds_strides = vec![1usize; rank];
             for i in (0..rank.saturating_sub(1)).rev() {
-                ds_strides[i] = ds_strides[i + 1] * dims[i + 1] as usize;
+                ds_strides[i] = ds_strides[i + 1] * saturating_usize(dims[i + 1]);
             }
 
             for (pi, pt) in pts.iter().enumerate() {
                 let flat: usize = pt
                     .iter()
                     .zip(ds_strides.iter())
-                    .map(|(&p, &s)| p as usize * s)
+                    .map(|(&p, &s)| saturating_usize(p) * s)
                     .sum();
                 let dst = flat * elem_size;
                 let src = pi * elem_size;
@@ -1569,6 +1703,35 @@ mod tests {
         assert_eq!(pl.filters[1].client_data, vec![3]);
     }
 
+    #[test]
+    fn chunk_options_pipeline_lzf() {
+        let options = ChunkOptions {
+            plugin: Some(PluginFilter::Lzf),
+            ..Default::default()
+        };
+        assert!(options.is_chunked());
+        let pl = options.build_pipeline_for_chunk(8, 800).unwrap();
+        assert_eq!(pl.filters.len(), 2);
+        assert_eq!(pl.filters[0].filter_id, FILTER_SHUFFLE);
+        assert_eq!(pl.filters[1].filter_id, FILTER_LZF);
+        assert_eq!(pl.filters[1].client_data, vec![4, 0x0105, 800]);
+    }
+
+    #[test]
+    fn chunk_options_pipeline_bitshuffle_has_no_auto_shuffle() {
+        let options = ChunkOptions {
+            plugin: Some(PluginFilter::Bitshuffle {
+                block_size: 0,
+                compression: BitshuffleCompression::Zstd { level: 5 },
+            }),
+            ..Default::default()
+        };
+        let pl = options.build_pipeline(4).unwrap();
+        assert_eq!(pl.filters.len(), 1);
+        assert_eq!(pl.filters[0].filter_id, FILTER_BITSHUFFLE);
+        assert_eq!(pl.filters[0].client_data, vec![0, 4, 4, 0, 3, 5]);
+    }
+
     #[test]
     fn chunk_options_zstd_priority_over_deflate() {
         let options = ChunkOptions {
@@ -1788,6 +1951,98 @@ mod tests {
         bytes_to_f64(&output)
     }
 
+    /// Every chunk index the writer builds records each chunk's real filter
+    /// mask: LZF output no smaller than the chunk is skipped (bit 1, behind
+    /// shuffle) and the chunk stored shuffled only; compressible chunks keep
+    /// mask 0. The data reads back through both kinds of chunk.
+    #[cfg(feature = "lzf")]
+    #[test]
+    fn skipped_lzf_chunks_are_masked_in_every_index() {
+        let c = 64usize;
+        // Chunks alternate: random bytes (LZF cannot shrink them), then 7s.
+        let mut state = 0x1234_5678_u64;
+        let data: Vec<f64> = (0..4 * c)
+            .map(|i| {
+                if (i / c).is_multiple_of(2) {
+                    state ^= state << 13;
+                    state ^= state >> 7;
+                    state ^= state << 17;
+                    f64::from_bits(state)
+                } else {
+                    7.0
+                }
+            })
+            .collect();
+        let raw = f64_to_bytes(&data);
+        let options = ChunkOptions {
+            plugin: Some(PluginFilter::Lzf),
+            ..Default::default()
+        };
+        let c64 = c as u64;
+        #[allow(clippy::type_complexity)]
+        let cases: [(&[u64], &[u64], Option<&[u64]>, u8, &[u32]); 4] = [
+            (&[c64], &[c64], None, 1, &[2]),
+            (&[4 * c64], &[c64], None, 3, &[2, 0, 2, 0]),
+            (&[4 * c64], &[c64], Some(&[u64::MAX]), 4, &[2, 0, 2, 0]),
+            (
+                &[2, 2 * c64],
+                &[1, c64],
+                Some(&[u64::MAX, u64::MAX]),
+                5,
+                &[2, 0, 2, 0],
+            ),
+        ];
+        let base = 0x1000u64;
+        for (shape, chunks, maxshape, index_type, want_masks) in cases {
+            let n: u64 = shape.iter().product();
+            let raw = &raw[..n as usize * 8];
+            let result =
+                build_chunked_data_at_ext(raw, shape, chunks, 8, &options, base, maxshape).unwrap();
+            let mut file = vec![0u8; base as usize];
+            file.extend_from_slice(&result.data_bytes);
+            let layout = DataLayout::parse(&result.layout_message, 8, 8).unwrap();
+            assert!(
+                matches!(&layout, DataLayout::Chunked { chunk_index_type, .. }
+                    if *chunk_index_type == Some(index_type)),
+                "{layout:?}"
+            );
+            let dataspace = Dataspace {
+                space_type: DataspaceType::Simple,
+                rank: shape.len() as u8,
+                dimensions: shape.to_vec(),
+                max_dimensions: maxshape.map(<[u64]>::to_vec),
+            };
+            let (mut infos, _) =
+                crate::chunked_read::list_chunks(&file, &layout, &dataspace, 8, 8, 8).unwrap();
+            infos.sort_by(|a, b| a.offsets.cmp(&b.offsets));
+            let masks: Vec<u32> = infos.iter().map(|i| i.filter_mask).collect();
+            assert_eq!(masks, want_masks, "index type {index_type}");
+            for info in &infos {
+                // Skipped chunks are stored at the chunk's size (shuffled).
+                assert_eq!(
+                    info.chunk_size == (c * 8) as u32,
+                    info.filter_mask != 0,
+                    "{info:?}"
+                );
+            }
+            let pipeline = crate::filter_pipeline::FilterPipeline::parse(
+                result.pipeline_message.as_ref().unwrap(),
+            )
+            .unwrap();
+            let out = read_chunked_data(
+                &file,
+                &layout,
+                &dataspace,
+                &make_f64_type(),
+                Some(&pipeline),
+                8,
+                8,
+            )
+            .unwrap();
+            assert_eq!(out, raw, "index type {index_type}");
+        }
+    }
+
     #[test]
     fn ea_roundtrip_1d_inline_only() {
         let values: Vec<f64> = (0..10).map(|i| i as f64).collect();

crates/clawhdf5-format/src/data_layout.rs

@@ -1,12 +1,14 @@
 //! HDF5 Data Layout message parsing (message type 0x0008).
 
 #[cfg(not(feature = "std"))]
-use alloc::{string::String, vec::Vec};
+use alloc::{format, string::String, vec::Vec};
 
 #[cfg(feature = "std")]
 use std::string::String;
 
+use crate::addr::to_usize;
 use crate::error::FormatError;
+use crate::storage::Storage;
 
 /// A single VDS (Virtual Dataset) source mapping.
 ///
@@ -24,6 +26,34 @@ pub struct VdsMapping {
     pub virtual_selection: Vec<u8>,
 }
 
+/// Most dimensions a layout message can list (libhdf5 `H5O_LAYOUT_NDIMS`):
+/// 32 dataspace dimensions plus the element size.
+const MAX_LAYOUT_NDIMS: usize = 33;
+
+/// libhdf5's checks on a chunked layout message's dimensions
+/// (`H5O__layout_decode`): at most [`MAX_LAYOUT_NDIMS`], no dimension 0, and
+/// before version 4 at least one dataspace dimension plus the element size.
+/// A zero chunk dimension used to read the dataset as all fill values.
+fn check_chunk_dims(dims: Vec<u32>, layout_version: u8) -> Result<Vec<u32>, FormatError> {
+    if dims.len() > MAX_LAYOUT_NDIMS {
+        return Err(FormatError::InvalidChunkDimensions(
+            "dimensionality is too large".into(),
+        ));
+    }
+    if layout_version < 4 && dims.len() < 2 {
+        return Err(FormatError::InvalidChunkDimensions(
+            "bad dimensions for chunked storage".into(),
+        ));
+    }
+    if let Some(u) = dims.iter().position(|&d| d == 0) {
+        return Err(FormatError::InvalidChunkDimensions(format!(
+            "bad chunk dimension value when parsing layout message - chunk dimension must be \
+             positive: mesg->u.chunk.dim[{u}] = 0"
+        )));
+    }
+    Ok(dims)
+}
+
 /// Parsed HDF5 data layout message.
 #[derive(Debug, Clone, PartialEq)]
 pub enum DataLayout {
@@ -45,7 +75,9 @@ pub enum DataLayout {
         chunk_dimensions: Vec<u32>,
         /// B-tree address, or `None` if undefined.
         btree_address: Option<u64>,
-        /// Layout version (3 or 4).
+        /// Layout version (3 or 4). Version 1/2 messages (HDF5 1.4/1.6-era)
+        /// use the same version-1 B-tree chunk index as version 3 and are
+        /// reported as 3.
         version: u8,
         /// Chunk index type (v4 only).
         chunk_index_type: Option<u8>,
@@ -72,21 +104,33 @@ pub enum DataLayout {
     },
 }
 
+/// Version-1 VDS mapping flag: the source file name is stored by an earlier
+/// entry, whose index follows in place of the name.
+const VDS_SOURCE_FILE_SHARED: u8 = 0x01;
+/// Version-1 VDS mapping flag: likewise for the source dataset name.
+const VDS_SOURCE_DSET_SHARED: u8 = 0x02;
+/// Version-1 VDS mapping flag: the source is in the virtual file itself
+/// (`"."`); no file name is stored.
+const VDS_SOURCE_SAME_FILE: u8 = 0x04;
+const VDS_ALL_FLAGS: u8 = VDS_SOURCE_FILE_SHARED | VDS_SOURCE_DSET_SHARED | VDS_SOURCE_SAME_FILE;
+
 /// Parse VDS mappings from global-heap object data.
 ///
 /// The global-heap block holding a VDS mapping list is laid out as
-/// (reverse-engineered and validated against HDF5 2.0):
+/// (`H5D__virtual_store_layout` / `H5D__virtual_load_layout` in libhdf5):
 ///
 /// ```text
 /// version(1) · nused(length_size, LE) · entry[nused] · checksum(4)
 /// ```
 ///
 /// Each entry is:
-/// - source file name — a null-terminated string in **block version 0**; in
+/// - **block version 1 only:** a flags byte. `0x04`: the source is in the
-///   **block version 1** a same-file reference is encoded as a single `0x04`
+///   virtual file itself and no file name is stored; `0x01`/`0x02`: the
-///   marker byte (the source file is the virtual file itself) in place of the
+///   source file/dataset name is that of an earlier entry, whose index
-///   name;
+///   (`length_size` bytes) is stored instead of the name. libhdf5 2.0 writes
-/// - source dataset name (null-terminated string);
+///   version 1 when the file's low version bound is 2.0 and it saves space;
+/// - source file name (null-terminated string, unless flagged above);
+/// - source dataset name (null-terminated string, unless flagged above);
 /// - source selection (serialized `H5S` dataspace selection — self-describing
 ///   in length);
 /// - virtual selection (serialized `H5S` dataspace selection).
@@ -112,7 +156,7 @@ pub fn parse_vds_mappings(
     // `nused` is untrusted; don't pre-allocate from it. Each entry consumes at
     // least a few bytes, so the loop is naturally bounded by the heap data and
     // a bogus `nused` simply errors out on the first short read.
-    let mut mappings = Vec::new();
+    let mut mappings: Vec<VdsMapping> = Vec::new();
     // Reads one self-describing selection at `pos`, returning its raw bytes and
     // advancing past it — bounds-checked so a corrupt selection can't overrun.
     let read_selection = |heap_data: &[u8], pos: &mut usize| -> Result<Vec<u8>, FormatError> {
@@ -132,17 +176,57 @@ pub fn parse_vds_mappings(
         Ok(bytes)
     };
 
-    for _ in 0..nused {
+    if version > 1 {
-        // Source file name (with the version-1 same-file marker handled).
+        return Err(FormatError::ChunkedReadError(
-        let source_file = if version >= 1 && heap_data.get(pos) == Some(&0x04) {
+            "unsupported VDS mapping block version".into(),
+        ));
+    }
+    for i in 0..nused {
+        // Version 1 prefixes each entry with a flags byte; a name may then be
+        // omitted (same file) or replaced by the index of an earlier entry
+        // holding the same name (`H5D__virtual_load_layout`).
+        let flags = if version >= 1 {
+            let f = *heap_data.get(pos).ok_or(FormatError::UnexpectedEof {
+                expected: pos + 1,
+                available: heap_data.len(),
+            })?;
             pos += 1;
+            if f & !VDS_ALL_FLAGS != 0 {
+                return Err(FormatError::ChunkedReadError(
+                    "unknown VDS mapping flags".into(),
+                ));
+            }
+            f
+        } else {
+            0
+        };
+        // Index of an earlier entry, for a shared name.
+        let earlier = |pos: &mut usize| -> Result<usize, FormatError> {
+            let idx = read_length(heap_data, *pos, length_size)?;
+            *pos += ls;
+            if idx >= i {
+                return Err(FormatError::ChunkedReadError(
+                    "VDS mapping shares a name with a later entry".into(),
+                ));
+            }
+            to_usize(idx)
+        };
+
+        let source_file = if flags & VDS_SOURCE_SAME_FILE != 0 {
             String::from(".")
+        } else if flags & VDS_SOURCE_FILE_SHARED != 0 {
+            let idx = earlier(&mut pos)?;
+            mappings[idx].source_file.clone()
         } else {
             read_null_terminated_string(heap_data, &mut pos)?
         };
 
-        // Source dataset name.
+        let source_dataset = if flags & VDS_SOURCE_DSET_SHARED != 0 {
-        let source_dataset = read_null_terminated_string(heap_data, &mut pos)?;
+            let idx = earlier(&mut pos)?;
+            mappings[idx].source_dataset.clone()
+        } else {
+            read_null_terminated_string(heap_data, &mut pos)?
+        };
 
         // Source selection, then virtual selection (both self-describing length).
         let source_selection = read_selection(heap_data, &mut pos)?;
@@ -227,6 +311,16 @@ impl DataLayout {
         &mut self,
         file_data: &[u8],
         length_size: u8,
+    ) -> Result<(), FormatError> {
+        self.resolve_vds_mappings_in(file_data, length_size)
+    }
+
+    /// [`Self::resolve_vds_mappings`] over any [`Storage`]: one read of the
+    /// global heap collection holding the mappings.
+    pub fn resolve_vds_mappings_in<S: Storage + ?Sized>(
+        &mut self,
+        file_data: &S,
+        length_size: u8,
     ) -> Result<(), FormatError> {
         if let DataLayout::Virtual {
             global_heap_address,
@@ -236,11 +330,8 @@ impl DataLayout {
         } = self
             && let Some(addr) = *global_heap_address
         {
-            let coll = crate::global_heap::GlobalHeapCollection::parse(
+            let coll =
-                file_data,
+                crate::global_heap::GlobalHeapCollection::parse_in(file_data, addr, length_size)?;
-                addr as usize,
-                length_size,
-            )?;
             let obj = coll.get_object(*global_heap_index as u16).ok_or(
                 FormatError::GlobalHeapObjectNotFound {
                     collection_address: addr,
@@ -261,6 +352,7 @@ impl DataLayout {
         let layout_class = data[1];
 
         match version {
+            1 | 2 => Self::parse_v1_v2(data, offset_size),
             3 => Self::parse_v3(data, layout_class, offset_size, length_size),
             // v5 (emitted by HDF5 1.14+/2.0 with `libver=latest`) uses the same
             // message structure as v4 — only the version number was bumped.
@@ -269,6 +361,87 @@ impl DataLayout {
         }
     }
 
+    /// Layout message versions 1 and 2 (HDF5 before 1.6.3):
+    ///
+    /// ```text
+    /// version(1) · dimensionality(1) · layout class(1) · reserved(5)
+    /// · address(offset_size)      — contiguous and chunked only
+    /// · dimension sizes(4 × dimensionality)
+    /// · compact data size(4) · compact raw data — compact only
+    /// ```
+    ///
+    /// The dimension sizes are the dataset's (contiguous/compact) or the
+    /// chunk's (chunked) extent plus a trailing element-size dimension, as in
+    /// version 3's chunked form. libhdf5 ignores them for contiguous storage
+    /// and sizes the data from the dataspace; the product of the stored
+    /// dimensions is that same size, and a disagreement (a dimension that was
+    /// truncated to 32 bits) is caught by the reader's size check rather than
+    /// returning wrong data.
+    fn parse_v1_v2(data: &[u8], offset_size: u8) -> Result<DataLayout, FormatError> {
+        ensure_len(data, 0, 8)?;
+        let dimensionality = data[1] as usize;
+        let layout_class = data[2];
+        // H5O_LAYOUT_NDIMS: 32 dataspace dimensions + the element-size one.
+        if dimensionality > 33 {
+            return Err(FormatError::Overflow(format!(
+                "data layout dimensionality {dimensionality} exceeds 33"
+            )));
+        }
+        let mut p = 8;
+        let os = offset_size as usize;
+        let address = match layout_class {
+            1 | 2 => {
+                ensure_len(data, p, os)?;
+                let a = if is_undefined(data, p, offset_size) {
+                    None
+                } else {
+                    Some(read_offset(data, p, offset_size)?)
+                };
+                p += os;
+                a
+            }
+            0 => None,
+            _ => return Err(FormatError::InvalidLayoutClass(layout_class)),
+        };
+        ensure_len(data, p, dimensionality * 4)?;
+        let dims: Vec<u32> = data[p..p + dimensionality * 4]
+            .as_chunks::<4>()
+            .0
+            .iter()
+            .map(|c| u32::from_le_bytes(*c))
+            .collect();
+        p += dimensionality * 4;
+        match layout_class {
+            0 => {
+                ensure_len(data, p, 4)?;
+                let size =
+                    u32::from_le_bytes([data[p], data[p + 1], data[p + 2], data[p + 3]]) as usize;
+                ensure_len(data, p + 4, size)?;
+                Ok(DataLayout::Compact {
+                    data: data[p + 4..p + 4 + size].to_vec(),
+                })
+            }
+            1 => {
+                let size = dims
+                    .iter()
+                    .try_fold(1u64, |acc, &d| acc.checked_mul(d as u64))
+                    .ok_or_else(|| {
+                        FormatError::Overflow(format!("contiguous layout size {dims:?}"))
+                    })?;
+                Ok(DataLayout::Contiguous { address, size })
+            }
+            _ => Ok(DataLayout::Chunked {
+                chunk_dimensions: check_chunk_dims(dims, 2)?,
+                btree_address: address,
+                version: 3,
+                chunk_index_type: None,
+                single_chunk_filtered_size: None,
+                single_chunk_filter_mask: None,
+                dont_filter_partial_edge_chunks: false,
+            }),
+        }
+    }
+
     fn parse_v3(
         data: &[u8],
         layout_class: u8,
@@ -321,7 +494,7 @@ impl DataLayout {
                     p += 4;
                 }
                 Ok(DataLayout::Chunked {
-                    chunk_dimensions,
+                    chunk_dimensions: check_chunk_dims(chunk_dimensions, 3)?,
                     btree_address,
                     version: 3,
                     chunk_index_type: None,
@@ -370,47 +543,40 @@ impl DataLayout {
                 let dimensionality = data[pos + 1] as usize;
                 let dim_size_encoded_length = data[pos + 2] as usize;
                 let mut p = pos + 3;
+                if dimensionality > MAX_LAYOUT_NDIMS {
+                    return Err(FormatError::InvalidChunkDimensions(
+                        "dimensionality is too large".into(),
+                    ));
+                }
 
-                // dimension sizes
+                // Each dimension takes 1 to 8 bytes (libhdf5 writes the
+                // fewest that hold the largest one, so 3, 5, 6 and 7 occur:
+                // a chunk dimension of 70 000 takes 3). libhdf5 refuses 0
+                // and more than 8.
+                if dim_size_encoded_length == 0 || dim_size_encoded_length > 8 {
+                    return Err(FormatError::InvalidChunkDimensions(
+                        "encoded chunk dimension size is too large".into(),
+                    ));
+                }
                 ensure_len(data, p, dimensionality * dim_size_encoded_length)?;
                 let mut chunk_dimensions = Vec::with_capacity(dimensionality);
                 for _ in 0..dimensionality {
-                    let val = match dim_size_encoded_length {
+                    let val = data[p..p + dim_size_encoded_length]
-                        1 => data[p] as u32,
+                        .iter()
-                        2 => u16::from_le_bytes([data[p], data[p + 1]]) as u32,
+                        .rev()
-                        4 => u32::from_le_bytes([data[p], data[p + 1], data[p + 2], data[p + 3]]),
+                        .fold(0u64, |acc, &b| (acc << 8) | u64::from(b));
-                        8 => {
+                    // Chunk dimensions are held as u32; HDF5 2.0 can write
-                            // V4 chunked encodes dimension sizes as 8 bytes, but
+                    // larger ones (layout version 5), which are refused
-                            // our ChunkedStorageV4 stores them as u32. We read only
+                    // rather than truncated.
-                            // the low 4 bytes (little-endian). This silently
+                    let val = u32::try_from(val).map_err(|_| {
-                            // truncates dimensions > 4 GiB, which are not expected
+                        FormatError::InvalidChunkDimensions(format!(
-                            // in practice (HDF5 chunk dimensions are always small).
+                            "chunk dimension {val} is larger than 2^32 - 1, which is not supported"
-                            // If the high bytes are non-zero, the file is malformed
+                        ))
-                            // or uses dimensions we cannot represent.
+                    })?;
-                            let high = u32::from_le_bytes([
-                                data[p + 4],
-                                data[p + 5],
-                                data[p + 6],
-                                data[p + 7],
-                            ]);
-                            if high != 0 {
-                                return Err(FormatError::UnexpectedEof {
-                                    expected: p + 8,
-                                    available: data.len(),
-                                });
-                            }
-                            u32::from_le_bytes([data[p], data[p + 1], data[p + 2], data[p + 3]])
-                        }
-                        _ => {
-                            return Err(FormatError::UnexpectedEof {
-                                expected: p + dim_size_encoded_length,
-                                available: data.len(),
-                            });
-                        }
-                    };
                     chunk_dimensions.push(val);
                     p += dim_size_encoded_length;
                 }
+                let chunk_dimensions = check_chunk_dims(chunk_dimensions, 4)?;
 
                 // chunk index type
                 ensure_len(data, p, 1)?;
@@ -546,6 +712,202 @@ impl DataLayout {
 mod tests {
     use super::*;
 
+    /// Version 1/2 header: version, dimensionality, class, reserved(5).
+    fn v1v2_header(version: u8, ndims: u8, class: u8) -> Vec<u8> {
+        vec![version, ndims, class, 0, 0, 0, 0, 0]
+    }
+
+    #[test]
+    fn v2_compact() {
+        let mut buf = v1v2_header(2, 2, 0);
+        // dims (3 elements of 2 bytes) — no address for compact
+        buf.extend_from_slice(&3u32.to_le_bytes());
+        buf.extend_from_slice(&2u32.to_le_bytes());
+        buf.extend_from_slice(&6u32.to_le_bytes()); // compact size (u32 in v1/v2)
+        buf.extend_from_slice(&[1, 0, 2, 0, 3, 0]);
+        assert_eq!(
+            DataLayout::parse(&buf, 8, 8).unwrap(),
+            DataLayout::Compact {
+                data: vec![1, 0, 2, 0, 3, 0]
+            }
+        );
+    }
+
+    #[test]
+    fn v1_contiguous_size_from_dimensions() {
+        let mut buf = v1v2_header(1, 3, 1);
+        buf.extend_from_slice(&0x800u32.to_le_bytes()); // 4-byte address
+        for d in [10u32, 20, 4] {
+            buf.extend_from_slice(&d.to_le_bytes());
+        }
+        assert_eq!(
+            DataLayout::parse(&buf, 4, 4).unwrap(),
+            DataLayout::Contiguous {
+                address: Some(0x800),
+                size: 800,
+            }
+        );
+    }
+
+    #[test]
+    fn v1_contiguous_undefined_address() {
+        let mut buf = v1v2_header(1, 2, 1);
+        buf.extend_from_slice(&[0xFF; 8]);
+        buf.extend_from_slice(&5u32.to_le_bytes());
+        buf.extend_from_slice(&8u32.to_le_bytes());
+        assert_eq!(
+            DataLayout::parse(&buf, 8, 8).unwrap(),
+            DataLayout::Contiguous {
+                address: None,
+                size: 40,
+            }
+        );
+    }
+
+    #[test]
+    fn v1_chunked_maps_to_btree_v1_index() {
+        let mut buf = v1v2_header(1, 3, 2);
+        buf.extend_from_slice(&0x1234u64.to_le_bytes());
+        for d in [50u32, 50, 4] {
+            buf.extend_from_slice(&d.to_le_bytes());
+        }
+        assert_eq!(
+            DataLayout::parse(&buf, 8, 8).unwrap(),
+            DataLayout::Chunked {
+                chunk_dimensions: vec![50, 50, 4],
+                btree_address: Some(0x1234),
+                version: 3,
+                chunk_index_type: None,
+                single_chunk_filtered_size: None,
+                single_chunk_filter_mask: None,
+                dont_filter_partial_edge_chunks: false,
+            }
+        );
+    }
+
+    /// A v3 chunked layout message with these dims (element size last).
+    fn v3_chunked_msg(dims: &[u32]) -> Vec<u8> {
+        let mut buf = vec![3u8, 2, dims.len() as u8];
+        buf.extend_from_slice(&0x1000u64.to_le_bytes());
+        for d in dims {
+            buf.extend_from_slice(&d.to_le_bytes());
+        }
+        buf
+    }
+
+    #[test]
+    fn chunk_dimensions_are_checked_when_the_layout_is_parsed() {
+        assert!(DataLayout::parse(&v3_chunked_msg(&[4, 4, 8]), 8, 8).is_ok());
+        // A zero chunk dimension used to read as all fill values.
+        let err = DataLayout::parse(&v3_chunked_msg(&[4, 0, 8]), 8, 8).unwrap_err();
+        assert!(
+            matches!(&err, FormatError::InvalidChunkDimensions(m) if m.contains("dim[1] = 0")),
+            "{err:?}"
+        );
+        // Only the element-size dimension: libhdf5 "bad dimensions".
+        assert_eq!(
+            DataLayout::parse(&v3_chunked_msg(&[8]), 8, 8).unwrap_err(),
+            FormatError::InvalidChunkDimensions("bad dimensions for chunked storage".into())
+        );
+        assert_eq!(
+            DataLayout::parse(&v3_chunked_msg(&[1; 34]), 8, 8).unwrap_err(),
+            FormatError::InvalidChunkDimensions("dimensionality is too large".into())
+        );
+        // v1/v2 and v4 messages get the zero check too.
+        let mut v1 = v1v2_header(1, 2, 2);
+        v1.extend_from_slice(&0x1000u64.to_le_bytes());
+        v1.extend_from_slice(&0u32.to_le_bytes());
+        v1.extend_from_slice(&8u32.to_le_bytes());
+        assert!(matches!(
+            DataLayout::parse(&v1, 8, 8),
+            Err(FormatError::InvalidChunkDimensions(_))
+        ));
+        let mut v4 = vec![4u8, 2, 0, 2, 4];
+        v4.extend_from_slice(&0u32.to_le_bytes());
+        v4.extend_from_slice(&8u32.to_le_bytes());
+        v4.push(3); // fixed array index
+        v4.push(0); // page bits
+        v4.extend_from_slice(&0x1000u64.to_le_bytes());
+        assert!(matches!(
+            DataLayout::parse(&v4, 8, 8),
+            Err(FormatError::InvalidChunkDimensions(_))
+        ));
+    }
+
+    /// A v4 chunked layout (fixed array index) whose `dims` are each
+    /// encoded in `width` bytes.
+    fn v4_chunked_msg(width: u8, dims: &[u64]) -> Vec<u8> {
+        let mut m = vec![4u8, 2, 0, dims.len() as u8, width];
+        for &d in dims {
+            m.extend_from_slice(&d.to_le_bytes()[..width.min(8) as usize]);
+        }
+        m.push(3); // fixed array index
+        m.push(0); // page bits
+        m.extend_from_slice(&0x1000u64.to_le_bytes());
+        m
+    }
+
+    #[test]
+    fn v4_chunk_dimensions_take_1_to_8_bytes() {
+        // libhdf5 encodes each dimension in the fewest bytes that hold the
+        // largest: a chunk dimension of 70 000 takes 3, and 3, 5, 6 and 7
+        // were refused ("UnexpectedEof").
+        for width in 1..=8u8 {
+            let dims = [if width >= 3 { 70_000 } else { 200 }, 8];
+            let layout = DataLayout::parse(&v4_chunked_msg(width, &dims), 8, 8)
+                .unwrap_or_else(|e| panic!("width {width}: {e:?}"));
+            assert!(
+                matches!(&layout, DataLayout::Chunked { chunk_dimensions, .. }
+                    if chunk_dimensions.iter().map(|&d| u64::from(d)).eq(dims)),
+                "width {width}: {layout:?}"
+            );
+        }
+        // libhdf5 refuses 0 and more than 8 bytes.
+        for width in [0u8, 9] {
+            assert_eq!(
+                DataLayout::parse(&v4_chunked_msg(width, &[4, 8]), 8, 8).unwrap_err(),
+                FormatError::InvalidChunkDimensions(
+                    "encoded chunk dimension size is too large".into()
+                )
+            );
+        }
+        // A dimension past u32 cannot be represented and is refused, not
+        // truncated.
+        assert!(matches!(
+            DataLayout::parse(&v4_chunked_msg(5, &[1 << 32, 8]), 8, 8),
+            Err(FormatError::InvalidChunkDimensions(m)) if m.contains("2^32")
+        ));
+    }
+
+    #[test]
+    fn v1v2_rejects_bad_class_dimensionality_and_truncation() {
+        assert_eq!(
+            DataLayout::parse(&v1v2_header(1, 1, 3), 8, 8).unwrap_err(),
+            FormatError::InvalidLayoutClass(3)
+        );
+        assert!(matches!(
+            DataLayout::parse(&v1v2_header(2, 34, 1), 8, 8).unwrap_err(),
+            FormatError::Overflow(_)
+        ));
+        // Chunked, dims cut short.
+        let mut buf = v1v2_header(1, 2, 2);
+        buf.extend_from_slice(&0x10u64.to_le_bytes());
+        buf.extend_from_slice(&7u32.to_le_bytes());
+        assert!(matches!(
+            DataLayout::parse(&buf, 8, 8).unwrap_err(),
+            FormatError::UnexpectedEof { .. }
+        ));
+        // Compact, raw data shorter than its declared size.
+        let mut buf = v1v2_header(2, 1, 0);
+        buf.extend_from_slice(&4u32.to_le_bytes());
+        buf.extend_from_slice(&100u32.to_le_bytes());
+        buf.extend_from_slice(&[0; 4]);
+        assert!(matches!(
+            DataLayout::parse(&buf, 8, 8).unwrap_err(),
+            FormatError::UnexpectedEof { .. }
+        ));
+    }
+
     #[test]
     fn v3_compact() {
         let mut buf = vec![3u8, 0]; // version=3, class=0 (compact)
@@ -849,6 +1211,62 @@ mod tests {
         assert_eq!(v1.iter_linear_1d(8).unwrap(), vec![4, 5, 6, 7]);
     }
 
+    #[test]
+    fn parse_vds_mappings_v1_shared_names() {
+        // Written by HDF5 2.0 (h5py, libver=("v200", "v200")) for three
+        // mappings from `a_rather_long_source_file.h5:a_rather_long_dataset_name`
+        // and one from the same file: the entries carry flags 0x00, 0x03, 0x03
+        // and 0x06, so names after the first are stored as entry indices.
+        let blob: &[u8] = &[
+            0x01, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x61, 0x5f, 0x72, 0x61,
+            0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x73, 0x6f, 0x75, 0x72,
+            0x63, 0x65, 0x5f, 0x66, 0x69, 0x6c, 0x65, 0x2e, 0x68, 0x35, 0x00, 0x61, 0x5f, 0x72,
+            0x61, 0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x64, 0x61, 0x74,
+            0x61, 0x73, 0x65, 0x74, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x00, 0x02, 0x00, 0x00, 0x00,
+            0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00,
+            0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02,
+            0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00,
+            0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03,
+            0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x04, 0x00, 0x01, 0x00, 0x01,
+            0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02,
+            0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+            0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00,
+            0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x00, 0x01, 0x00,
+            0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02, 0x00,
+            0x00, 0x00, 0x02, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00,
+            0x01, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02,
+            0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00,
+            0x00, 0x01, 0x02, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01,
+            0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x8e, 0xa7, 0xea, 0x7a,
+        ];
+        let mappings = parse_vds_mappings(blob, 8).unwrap();
+        let names: Vec<(&str, &str)> = mappings
+            .iter()
+            .map(|m| (m.source_file.as_str(), m.source_dataset.as_str()))
+            .collect();
+        let (file, dset) = ("a_rather_long_source_file.h5", "a_rather_long_dataset_name");
+        assert_eq!(
+            names,
+            vec![(file, dset), (file, dset), (file, dset), (".", dset)]
+        );
+    }
+
+    #[test]
+    fn parse_vds_mappings_v1_forward_reference_is_error() {
+        // Entry 0 claiming to share entry 0's file name must not index past
+        // the entries decoded so far.
+        let mut blob = vec![0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x01];
+        blob.extend_from_slice(&[0u8; 8]);
+        blob.extend_from_slice(b"d\0");
+        assert!(parse_vds_mappings(&blob, 8).is_err());
+        // Unknown flag bits are refused.
+        let blob = [0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x08, b'd', 0];
+        assert!(parse_vds_mappings(&blob, 8).is_err());
+    }
+
     #[test]
     fn parse_vds_mappings_external_v0() {
         // Block version 0 with an explicit (external) source file name.
@@ -896,4 +1314,44 @@ mod tests {
         let blob = [0x01u8, 0, 0, 0, 0, 0, 0, 0, 0];
         assert!(parse_vds_mappings(&blob, 8).unwrap().is_empty());
     }
+
+    /// A virtual dataset's mappings resolve identically through a
+    /// read_at-only CountingStorage, in two reads of the global heap.
+    #[test]
+    fn vds_mappings_through_storage_match_slice() {
+        use crate::message_type::MessageType;
+        use crate::object_header::ObjectHeader;
+        use crate::storage::CountingStorage;
+        let file: &[u8] = include_bytes!("../tests/fixtures/vds_same_file.h5");
+        let sb = crate::superblock::Superblock::parse(file, 0).unwrap();
+        let (os, ls) = (sb.offset_size, sb.length_size);
+        let storage = CountingStorage::new(file.to_vec());
+        let mut virtuals = 0;
+        for child in
+            crate::group_v2::resolve_group_children(file, &sb, sb.root_group_address).unwrap()
+        {
+            let h =
+                ObjectHeader::parse(file, child.object_header_address as usize, os, ls).unwrap();
+            let Some(msg) = h
+                .messages
+                .iter()
+                .find(|m| m.msg_type == MessageType::DataLayout)
+            else {
+                continue;
+            };
+            let mut want = DataLayout::parse(&msg.data, os, ls).unwrap();
+            if !matches!(want, DataLayout::Virtual { .. }) {
+                continue;
+            }
+            let mut got = want.clone();
+            want.resolve_vds_mappings(file, ls).unwrap();
+            storage.reset();
+            got.resolve_vds_mappings_in(&storage, ls).unwrap();
+            assert_eq!(format!("{got:?}"), format!("{want:?}"));
+            assert!(matches!(&got, DataLayout::Virtual { mappings, .. } if !mappings.is_empty()));
+            assert_eq!(storage.reads(), 2);
+            virtuals += 1;
+        }
+        assert!(virtuals >= 1);
+    }
 }

crates/clawhdf5-format/src/dataspace.rs

@@ -7,6 +7,9 @@ use alloc::vec::Vec;
 
 use crate::error::FormatError;
 
+/// Most dimensions a dataspace can have (`H5S_MAX_RANK`).
+pub const MAX_RANK: u8 = 32;
+
 /// Type of dataspace.
 #[derive(Debug, Clone, PartialEq)]
 pub enum DataspaceType {
@@ -67,6 +70,12 @@ impl Dataspace {
         let version = data[0];
         let rank = data[1];
         let flags = data[2];
+        // H5O__sdspace_decode's checks.
+        if rank > MAX_RANK {
+            return Err(FormatError::InvalidDataspace(
+                "simple dataspace dimensionality is too large",
+            ));
+        }
 
         let (space_type, header_size) = match version {
             1 => {
@@ -88,6 +97,11 @@ impl Dataspace {
                     2 => DataspaceType::Null,
                     _ => return Err(FormatError::InvalidDataspaceType(type_byte)),
                 };
+                if st != DataspaceType::Simple && rank > 0 {
+                    return Err(FormatError::InvalidDataspace(
+                        "invalid rank for scalar or NULL dataspace",
+                    ));
+                }
                 (st, 4usize)
             }
             _ => return Err(FormatError::InvalidDataspaceVersion(version)),
@@ -107,8 +121,13 @@ impl Dataspace {
         // Read max dimensions if flags bit 0 is set
         let max_dimensions = if flags & 0x01 != 0 {
             let mut max_dims = Vec::with_capacity(rank as usize);
-            for _ in 0..rank {
+            for &dim in &dimensions {
                 let val = read_length(data, pos, length_size)?;
+                if dim > val {
+                    return Err(FormatError::InvalidDataspace(
+                        "dataspace dimension size is greater than its maximum size",
+                    ));
+                }
                 max_dims.push(val);
                 pos += ls;
             }
@@ -176,7 +195,6 @@ impl Dataspace {
         match self.space_type {
             DataspaceType::Null => Ok(0),
             DataspaceType::Scalar => Ok(1),
-            DataspaceType::Simple if self.dimensions.is_empty() => Ok(0),
             DataspaceType::Simple => self
                 .dimensions
                 .iter()
@@ -195,18 +213,14 @@ impl Dataspace {
         match self.space_type {
             DataspaceType::Null => 0,
             DataspaceType::Scalar => 1,
-            DataspaceType::Simple => {
+            // A simple dataspace of rank 0 holds one element, as in libhdf5
-                if self.dimensions.is_empty() {
+            // (the product of no dimensions). Saturate rather than wrap: a
-                    0
+            // wrapped product could under-size a buffer. Size-critical
-                } else {
+            // callers use `checked_num_elements`.
-                    // Saturate rather than wrap: a wrapped product could
+            DataspaceType::Simple => self
-                    // under-size a buffer. Size-critical callers use
+                .dimensions
-                    // `checked_num_elements`.
-                    self.dimensions
                 .iter()
-                        .fold(1u64, |acc, &d| acc.saturating_mul(d))
+                .fold(1u64, |acc, &d| acc.saturating_mul(d)),
-                }
-            }
         }
     }
 }
@@ -352,4 +366,44 @@ mod tests {
         let ds = Dataspace::parse(&data, 8).unwrap();
         assert_eq!(ds.max_dimensions, Some(vec![10]));
     }
+
+    /// A simple dataspace of rank 0 (cve-2020-18494's `/dset1`) holds one
+    /// element in libhdf5, which h5py reads as shape `()`. It was 0.
+    #[test]
+    fn simple_rank_zero_holds_one_element() {
+        let data = build_v2_dataspace(0, 0, 1, &[], None);
+        let ds = Dataspace::parse(&data, 8).unwrap();
+        assert_eq!(ds.space_type, DataspaceType::Simple);
+        assert_eq!(ds.num_elements(), 1);
+        assert_eq!(ds.checked_num_elements().unwrap(), 1);
+    }
+
+    /// `H5O__sdspace_decode`'s checks.
+    #[test]
+    fn refuses_what_libhdf5_refuses() {
+        let too_many = build_v2_dataspace(33, 0, 1, &[1; 33], None);
+        assert!(matches!(
+            Dataspace::parse(&too_many, 8),
+            Err(FormatError::InvalidDataspace(_))
+        ));
+        let scalar_with_rank = build_v2_dataspace(1, 0, 0, &[4], None);
+        assert!(matches!(
+            Dataspace::parse(&scalar_with_rank, 8),
+            Err(FormatError::InvalidDataspace(_))
+        ));
+        let null_with_rank = build_v2_dataspace(1, 0, 2, &[4], None);
+        assert!(matches!(
+            Dataspace::parse(&null_with_rank, 8),
+            Err(FormatError::InvalidDataspace(_))
+        ));
+        let over_max = build_v1_dataspace(2, 0x01, &[5, 20], Some(&[10, 10]));
+        assert!(matches!(
+            Dataspace::parse(&over_max, 8),
+            Err(FormatError::InvalidDataspace(_))
+        ));
+        // 32 dimensions, and a size equal to the maximum or unlimited, are fine.
+        assert!(Dataspace::parse(&build_v2_dataspace(32, 0, 1, &[1; 32], None), 8).is_ok());
+        let at_max = build_v1_dataspace(2, 0x01, &[10, 20], Some(&[10, u64::MAX]));
+        assert!(Dataspace::parse(&at_max, 8).is_ok());
+    }
 }

crates/clawhdf5-format/src/datatype.rs

@@ -125,6 +125,11 @@ pub enum Datatype {
     },
     /// Class 9: Variable-length type.
     VariableLength {
+        /// Size of one element as stored in the file: a sequence length (4
+        /// bytes), a global heap collection address (the file's
+        /// `offset_size`) and an object index (4 bytes) — 16 in a file with
+        /// 8-byte offsets, 12 with 4-byte offsets.
+        size: u32,
         is_string: bool,
         padding: Option<StringPadding>,
         charset: Option<CharacterSet>,
@@ -208,6 +213,31 @@ fn offset_bytes_for_size(compound_size: u32) -> usize {
 }
 
 /// Read an unsigned integer of 1, 2, 4, or 8 bytes (LE).
+/// The size field of the datatype message at `pos`, as stored (a
+/// variable-length type's stored size is not modelled in [`Datatype`]).
+fn stored_type_size(data: &[u8], pos: usize) -> Result<u32, FormatError> {
+    ensure_len(data, pos, 8)?;
+    Ok(LittleEndian::read_u32(&data[pos + 4..pos + 8]))
+}
+
+/// libhdf5 refuses an array type of more than `H5S_MAX_RANK` (32)
+/// dimensions.
+fn check_array_rank(ndims: usize) -> Result<(), FormatError> {
+    if ndims > 32 {
+        return Err(invalid("too many dimensions for array datatype"));
+    }
+    Ok(())
+}
+
+/// A zero-sized array dimension makes a zero-sized type, which libhdf5
+/// cannot open ("unable to retrieve size of datatype").
+fn check_array_dims(dims: &[u32]) -> Result<(), FormatError> {
+    if dims.contains(&0) {
+        return Err(invalid("zero-sized dimension specified"));
+    }
+    Ok(())
+}
+
 fn read_uint(data: &[u8], offset: usize, nbytes: usize) -> Result<u64, FormatError> {
     ensure_len(data, offset, nbytes)?;
     let slice = &data[offset..offset + nbytes];
@@ -232,10 +262,104 @@ fn read_uint(data: &[u8], offset: usize, nbytes: usize) -> Result<u64, FormatErr
 /// available stack is a few KB.
 const MAX_DATATYPE_DEPTH: u16 = 64;
 
+fn invalid(why: impl Into<String>) -> FormatError {
+    FormatError::InvalidDatatype(why.into())
+}
+
+/// libhdf5's bounds checks on an integer type's bit offset and precision
+/// (`H5O__dtype_decode_helper`): both must lie inside the type. (Newer
+/// libhdf5 checks bit fields the same way; HDF5 2.0, which h5py 3.16 ships,
+/// does not, and opens such a type.)
+fn check_integer_bits(size: u32, bit_offset: u16, bit_precision: u16) -> Result<(), FormatError> {
+    let bits = u64::from(size) * 8;
+    if u64::from(bit_offset) >= bits {
+        return Err(invalid("integer offset out of bounds"));
+    }
+    if bit_precision == 0 {
+        return Err(invalid("precision is zero"));
+    }
+    if u64::from(bit_offset) + u64::from(bit_precision) > bits {
+        return Err(invalid("integer offset+precision out of bounds"));
+    }
+    Ok(())
+}
+
+/// Whether the closed bit ranges `[a0, a1]` and `[b0, b1]` share a bit.
+fn ranges_overlap(a0: u64, a1: u64, b0: u64, b1: u64) -> bool {
+    a0 <= b1 && b0 <= a1
+}
+
+/// libhdf5's checks on a floating-point type's fields: exponent and mantissa
+/// must lie inside the type, be non-empty, and not overlap each other or the
+/// sign bit. (libhdf5 does not check a float's bit offset and precision.)
+///
+/// One libhdf5 check is left out on purpose: a sign bit position outside the
+/// type ("sign bit position out of bounds"). clawhdf5 up to v2.7.0 wrote 63
+/// there for every float, so every `f32` it wrote (every agent store's
+/// embeddings) would stop opening. The position is not used to decode an
+/// IEEE float, so reading such a type returns the right values.
+fn check_float_fields(
+    size: u32,
+    sign: u8,
+    epos: u8,
+    esize: u8,
+    mpos: u8,
+    msize: u8,
+) -> Result<(), FormatError> {
+    let bits = u64::from(size) * 8;
+    let (sign, epos, esize, mpos, msize) = (
+        u64::from(sign),
+        u64::from(epos),
+        u64::from(esize),
+        u64::from(mpos),
+        u64::from(msize),
+    );
+    if esize == 0 {
+        return Err(invalid("exponent size can't be zero"));
+    }
+    if epos >= bits {
+        return Err(invalid("exponent starting position out of bounds"));
+    }
+    if epos + esize > bits {
+        return Err(invalid("exponent range out of bounds"));
+    }
+    if msize == 0 {
+        return Err(invalid("mantissa size can't be zero"));
+    }
+    if mpos >= bits {
+        return Err(invalid("mantissa starting position out of bounds"));
+    }
+    if mpos + msize > bits {
+        return Err(invalid("mantissa range out of bounds"));
+    }
+    let (e_end, m_end) = (epos + esize - 1, mpos + msize - 1);
+    if ranges_overlap(sign, sign, epos, e_end) {
+        return Err(invalid("exponent and sign positions overlap"));
+    }
+    if ranges_overlap(sign, sign, mpos, m_end) {
+        return Err(invalid("mantissa and sign positions overlap"));
+    }
+    if ranges_overlap(epos, e_end, mpos, m_end) {
+        return Err(invalid("mantissa and exponent positions overlap"));
+    }
+    Ok(())
+}
+
 impl Datatype {
     /// Parse a datatype message from raw bytes.
     ///
     /// Returns `(Datatype, bytes_consumed)` for recursive parsing.
+    ///
+    /// A type libhdf5 refuses to decode is refused here too, with
+    /// [`FormatError::InvalidDatatype`] carrying libhdf5's reason: size 0,
+    /// integer/bit-field/float bit fields outside the type or overlapping,
+    /// a compound with no members, a member outside its compound, a
+    /// duplicate or overlapping member, an enum whose size differs from its
+    /// base type's or with an empty name, an array of more than 32
+    /// dimensions or a zero-sized one, an unaligned opaque tag length.
+    /// Reading such a type used to return data from a corrupt file. Checks
+    /// newer libhdf5 releases add but HDF5 2.0 (h5py 3.16) lacks are left
+    /// out, so a file h5py opens still opens here.
     pub fn parse(data: &[u8]) -> Result<(Datatype, usize), FormatError> {
         Self::parse_with_depth(data, 0)
     }
@@ -259,6 +383,14 @@ impl Datatype {
 
         let size = LittleEndian::read_u32(&data[4..8]);
         let mut pos = 8;
+        // libhdf5 refuses size 0 for every class. A fixed-length string is
+        // exempt: clawhdf5 up to v2.7.0 wrote an empty-string attribute
+        // with a size-0 string type, and refusing it would fail every
+        // attribute of such objects, while reading it (an empty string) is
+        // harmless.
+        if size == 0 && class_id != 3 {
+            return Err(invalid("invalid datatype size"));
+        }
 
         match class_id {
             0 => {
@@ -272,6 +404,7 @@ impl Datatype {
                 let signed = (bf0 >> 3) & 0x01 == 1;
                 let bit_offset = LittleEndian::read_u16(&data[pos..pos + 2]);
                 let bit_precision = LittleEndian::read_u16(&data[pos + 2..pos + 4]);
+                check_integer_bits(size, bit_offset, bit_precision)?;
                 pos += 4;
                 Ok((
                     Datatype::FixedPoint {
@@ -289,13 +422,23 @@ impl Datatype {
                 ensure_len(data, pos, 12)?;
                 let bo_low = bf0 & 0x01;
                 let bo_high = (bf0 >> 6) & 0x01;
+                // Bit 6 (with bit 0) is VAX order, defined by version 3; libhdf5
+                // ignores bit 6 in older versions, which this read as VAX,
+                // byte-swapping a little-endian float.
+                let bo_high = if version >= 3 { bo_high } else { 0 };
                 let byte_order = match (bo_high, bo_low) {
                     (0, 0) => DatatypeByteOrder::LittleEndian,
                     (0, 1) => DatatypeByteOrder::BigEndian,
-                    (1, 0) => DatatypeByteOrder::Vax,
+                    (1, 0) => {
+                        return Err(invalid("bad byte order for datatype message"));
+                    }
                     (1, 1) => DatatypeByteOrder::Vax,
                     _ => unreachable!(),
                 };
+                // Bits 4-5: mantissa normalization; 3 is undefined.
+                if (bf0 >> 4) & 0x03 == 3 {
+                    return Err(invalid("unknown floating-point normalization"));
+                }
                 let bit_offset = LittleEndian::read_u16(&data[pos..pos + 2]);
                 let bit_precision = LittleEndian::read_u16(&data[pos + 2..pos + 4]);
                 let exponent_location = data[pos + 4];
@@ -303,6 +446,14 @@ impl Datatype {
                 let mantissa_location = data[pos + 6];
                 let mantissa_size = data[pos + 7];
                 let exponent_bias = LittleEndian::read_u32(&data[pos + 8..pos + 12]);
+                check_float_fields(
+                    size,
+                    bf1,
+                    exponent_location,
+                    exponent_size,
+                    mantissa_location,
+                    mantissa_size,
+                )?;
                 pos += 12;
                 Ok((
                     Datatype::FloatingPoint {
@@ -371,6 +522,10 @@ impl Datatype {
             5 => {
                 // Opaque
                 let tag_len = bf0 as usize;
+                // libhdf5 writes the NUL-padded length, a multiple of 8.
+                if !tag_len.is_multiple_of(8) {
+                    return Err(invalid("opaque flag field must be aligned"));
+                }
                 ensure_len(data, pos, tag_len)?;
                 // The stored tag is NUL-padded to a multiple of 8 bytes; the
                 // tag itself ends at the first NUL (libhdf5 reads it with
@@ -384,7 +539,45 @@ impl Datatype {
             6 => {
                 // Compound
                 let num_members = (bf0 as u16) | ((bf1 as u16) << 8);
-                let mut members = Vec::with_capacity(num_members as usize);
+                if num_members == 0 {
+                    return Err(invalid("invalid number of members: 0"));
+                }
+                let mut members: Vec<CompoundMember> = Vec::with_capacity(num_members as usize);
+                // Each member's size in the compound as libhdf5 decodes it:
+                // its stored size, times a v1 member's array dimensions. A
+                // variable-length member takes 4 + offset size + 4 bytes on
+                // disk, not the 16 of `Datatype::type_size`.
+                let mut member_sizes: Vec<u64> = Vec::with_capacity(num_members as usize);
+                // libhdf5 checks each member as it is decoded: it must fit in
+                // the compound (by its own stored size, before a v1 member's
+                // array dimensions are applied), and must not repeat a name
+                // or overlap an earlier member (by its final size).
+                let check_member = |members: &[CompoundMember],
+                                    member_sizes: &[u64],
+                                    name: &str,
+                                    byte_offset: u64,
+                                    stored_size: u32,
+                                    final_size: u64|
+                 -> Result<(), FormatError> {
+                    if byte_offset + u64::from(stored_size) > u64::from(size) {
+                        return Err(invalid(
+                            "member type extends outside its parent compound type",
+                        ));
+                    }
+                    if let Some(j) = members.iter().position(|m| m.name == name) {
+                        return Err(invalid(format!(
+                            "duplicated compound field name '{name}', for fields {j} and {}",
+                            members.len()
+                        )));
+                    }
+                    let end = byte_offset + final_size;
+                    if members.iter().zip(member_sizes).any(|(m, &m_size)| {
+                        byte_offset < m.byte_offset + m_size && m.byte_offset < end
+                    }) {
+                        return Err(invalid("member overlaps with previous member"));
+                    }
+                    Ok(())
+                };
 
                 if (3..=5).contains(&version) {
                     // v3, v4 and v5 share the compact member encoding (name,
@@ -396,9 +589,20 @@ impl Datatype {
                         pos += name_len;
                         let byte_offset = read_uint(data, pos, ob)?;
                         pos += ob;
+                        let stored_size = stored_type_size(data, pos)?;
                         let (member_dt, consumed) =
                             Self::parse_with_depth(&data[pos..], depth + 1)?;
                         pos += consumed;
+                        let final_size = u64::from(stored_size);
+                        check_member(
+                            &members,
+                            &member_sizes,
+                            &name,
+                            byte_offset,
+                            stored_size,
+                            final_size,
+                        )?;
+                        member_sizes.push(final_size);
                         members.push(CompoundMember {
                             name,
                             byte_offset,
@@ -423,13 +627,57 @@ impl Datatype {
                         ensure_len(data, pos, 4)?;
                         let byte_offset = LittleEndian::read_u32(&data[pos..pos + 4]) as u64;
                         pos += 4;
+                        // v1 members can be fixed-size arrays of the member
+                        // type (libhdf5 builds an array type from these
+                        // fields; the permutation is ignored, as libhdf5
+                        // does). Skipping them read a `[4] i32` member as
+                        // one `i32`.
+                        let mut array_dims = Vec::new();
                         if version == 1 {
                             ensure_len(data, pos, 28)?;
+                            let ndims = data[pos] as usize;
+                            // libhdf5 refuses more than four dimensions and,
+                            // when building the array type, a zero-sized one.
+                            let zero_dim = (0..ndims.min(4)).any(|j| {
+                                let at = pos + 12 + 4 * j;
+                                LittleEndian::read_u32(&data[at..at + 4]) == 0
+                            });
+                            if ndims > 4 {
+                                return Err(invalid("invalid number of dimensions for array"));
+                            }
+                            if zero_dim {
+                                return Err(invalid("zero-sized dimension specified"));
+                            }
+                            array_dims = (0..ndims)
+                                .map(|j| {
+                                    let at = pos + 12 + 4 * j;
+                                    LittleEndian::read_u32(&data[at..at + 4])
+                                })
+                                .collect();
                             pos += 28;
                         }
-                        let (member_dt, consumed) =
+                        let stored_size = stored_type_size(data, pos)?;
+                        let (mut member_dt, consumed) =
                             Self::parse_with_depth(&data[pos..], depth + 1)?;
                         pos += consumed;
+                        let final_size = array_dims.iter().fold(u64::from(stored_size), |a, &d| {
+                            a.saturating_mul(u64::from(d))
+                        });
+                        if !array_dims.is_empty() {
+                            member_dt = Datatype::Array {
+                                base_type: Box::new(member_dt),
+                                dimensions: array_dims,
+                            };
+                        }
+                        check_member(
+                            &members,
+                            &member_sizes,
+                            &name,
+                            byte_offset,
+                            stored_size,
+                            final_size,
+                        )?;
+                        member_sizes.push(final_size);
                         members.push(CompoundMember {
                             name,
                             byte_offset,
@@ -468,6 +716,9 @@ impl Datatype {
                 let (base_type, base_consumed) = Self::parse_with_depth(&data[pos..], depth + 1)?;
                 pos += base_consumed;
                 let base_size = base_type.type_size();
+                if base_size != size {
+                    return Err(invalid("ENUM datatype size does not match parent"));
+                }
                 let mut members = Vec::with_capacity(num_members as usize);
                 // Enum layout: base_type, then all names (null-terminated), then all values
                 // v1/v2: names are padded to 8-byte boundaries
@@ -475,6 +726,9 @@ impl Datatype {
                 let mut member_names = Vec::with_capacity(num_members as usize);
                 for _ in 0..num_members {
                     let (name, name_len) = read_null_terminated_string(data, pos)?;
+                    if name.is_empty() {
+                        return Err(invalid("0 length enum name"));
+                    }
                     if version < 3 {
                         let padded = (name_len + 7) & !7;
                         pos += padded;
@@ -522,6 +776,7 @@ impl Datatype {
                 pos += consumed;
                 Ok((
                     Datatype::VariableLength {
+                        size,
                         is_string,
                         padding,
                         charset,
@@ -535,6 +790,7 @@ impl Datatype {
                 if version == 2 {
                     ensure_len(data, pos, 4)?;
                     let ndims = data[pos] as usize;
+                    check_array_rank(ndims)?;
                     pos += 4; // ndims(1) + reserved(3)
                     ensure_len(data, pos, ndims * 4 + ndims * 4)?;
                     let mut dimensions = Vec::with_capacity(ndims);
@@ -542,6 +798,7 @@ impl Datatype {
                         dimensions.push(LittleEndian::read_u32(&data[pos..pos + 4]));
                         pos += 4;
                     }
+                    check_array_dims(&dimensions)?;
                     // skip permutation indices
                     pos += ndims * 4;
                     let (base_type, consumed) = Self::parse_with_depth(&data[pos..], depth + 1)?;
@@ -558,6 +815,7 @@ impl Datatype {
                     // type); HDF5 1.14+/2.0 with `libver=latest` emits v5.
                     ensure_len(data, pos, 1)?;
                     let ndims = data[pos] as usize;
+                    check_array_rank(ndims)?;
                     pos += 1;
                     ensure_len(data, pos, ndims * 4)?;
                     let mut dimensions = Vec::with_capacity(ndims);
@@ -565,6 +823,7 @@ impl Datatype {
                         dimensions.push(LittleEndian::read_u32(&data[pos..pos + 4]));
                         pos += 4;
                     }
+                    check_array_dims(&dimensions)?;
                     let (base_type, consumed) = Self::parse_with_depth(&data[pos..], depth + 1)?;
                     pos += consumed;
                     Ok((
@@ -621,6 +880,70 @@ impl Datatype {
         }
     }
 
+    /// [`Self::parse`] for the datatype message of an object whose header
+    /// has version `header_version`: a version-1 header, which has no
+    /// checksum, additionally gets [`Self::check_unused_bits`], as libhdf5
+    /// does. Use this wherever the header is at hand.
+    pub fn parse_in_header(
+        data: &[u8],
+        header_version: u8,
+    ) -> Result<(Datatype, usize), FormatError> {
+        let parsed = Self::parse(data)?;
+        if header_version == 1 {
+            parsed.0.check_unused_bits()?;
+        }
+        Ok(parsed)
+    }
+
+    /// libhdf5's guard against a corrupt numeric type in a header without
+    /// a checksum (`H5T_is_numeric_with_unusual_unused_bits`, HDF5 1.14.4+):
+    /// an integer, float or bit field wider than a byte whose precision and
+    /// offset leave more than half its bits unused is taken for corruption
+    /// (e.g. a 3-bit integer in 4 bytes, `cve-2024-29162`, or a 32-bit float
+    /// in 65525 bytes, `cve-2024-32614`), anywhere in the type. libhdf5
+    /// skips the check for checksummed (version-2) headers and when the
+    /// file is opened with `H5Pset_relax_file_integrity_checks`; so does
+    /// [`Self::parse_in_header`], which has no such option.
+    pub fn check_unused_bits(&self) -> Result<(), FormatError> {
+        match self {
+            Datatype::FixedPoint {
+                size,
+                bit_offset,
+                bit_precision,
+                ..
+            }
+            | Datatype::FloatingPoint {
+                size,
+                bit_offset,
+                bit_precision,
+                ..
+            }
+            | Datatype::BitField {
+                size,
+                bit_offset,
+                bit_precision,
+                ..
+            } => {
+                let bits = u64::from(*size) * 8;
+                let prec = u64::from(*bit_precision);
+                if *size > 1 && prec < bits && bits > 2 * (prec + u64::from(*bit_offset)) {
+                    return Err(invalid(format!(
+                        "datatype has unusually large # of unused bits (prec = {prec} bits, \
+                         size = {size} bytes), possibly corrupted file"
+                    )));
+                }
+                Ok(())
+            }
+            Datatype::Compound { members, .. } => members
+                .iter()
+                .try_for_each(|m| m.datatype.check_unused_bits()),
+            Datatype::Enumeration { base_type, .. }
+            | Datatype::VariableLength { base_type, .. }
+            | Datatype::Array { base_type, .. } => base_type.check_unused_bits(),
+            _ => Ok(()),
+        }
+    }
+
     /// Serialize datatype to HDF5 message bytes.
     pub fn serialize(&self) -> Vec<u8> {
         match self {
@@ -700,6 +1023,7 @@ impl Datatype {
                 Self::build_header(3, 1, [bf0, 0, 0], *size)
             }
             Datatype::VariableLength {
+                size,
                 is_string,
                 padding,
                 charset,
@@ -722,7 +1046,7 @@ impl Datatype {
                 } else {
                     0
                 };
-                let mut buf = Self::build_header(9, 1, [bf0, bf1, 0], 16);
+                let mut buf = Self::build_header(9, 1, [bf0, bf1, 0], *size);
                 buf.extend_from_slice(&base_type.serialize());
                 buf
             }
@@ -832,9 +1156,23 @@ impl Datatype {
     }
 
     /// Check that this datatype can be written: every part of it has an
-    /// on-disk encoding. [`Self::serialize`] cannot report errors, so the
+    /// on-disk encoding, and the encoding is one the reader (and libhdf5)
-    /// writer calls this first.
+    /// accepts. [`Self::serialize`] cannot report errors, so the writer calls
+    /// this first. A compound with no fields or a repeated field name, or an
+    /// enum member with an empty name, is refused here: libhdf5 and h5py
+    /// refuse such types, and so does [`Self::parse`], so writing one made a
+    /// file that could not be read back.
     pub fn check_encodable(&self) -> Result<(), FormatError> {
+        self.check_encodable_parts()?;
+        Self::parse(&self.serialize()).map_err(|e| {
+            FormatError::SerializationError(format!(
+                "datatype cannot be written: HDF5 readers refuse it ({e})"
+            ))
+        })?;
+        Ok(())
+    }
+
+    fn check_encodable_parts(&self) -> Result<(), FormatError> {
         match self {
             Datatype::Opaque { tag, .. } if opaque_tag_text(tag).len() > MAX_OPAQUE_TAG_LEN => {
                 Err(FormatError::SerializationError(format!(
@@ -847,10 +1185,10 @@ impl Datatype {
             )),
             Datatype::Compound { members, .. } => members
                 .iter()
-                .try_for_each(|m| m.datatype.check_encodable()),
+                .try_for_each(|m| m.datatype.check_encodable_parts()),
             Datatype::Enumeration { base_type, .. }
             | Datatype::VariableLength { base_type, .. }
-            | Datatype::Array { base_type, .. } => base_type.check_encodable(),
+            | Datatype::Array { base_type, .. } => base_type.check_encodable_parts(),
             _ => Ok(()),
         }
     }
@@ -877,7 +1215,7 @@ impl Datatype {
             Datatype::Compound { size, .. } => *size,
             Datatype::Reference { size, .. } => *size,
             Datatype::Enumeration { size, .. } => *size,
-            Datatype::VariableLength { .. } => 16, // typically pointer + length
+            Datatype::VariableLength { size, .. } => *size,
             Datatype::Array {
                 base_type,
                 dimensions,
@@ -954,7 +1292,8 @@ mod tests {
     ) -> Vec<u8> {
         // LE byte order: bo_low=0, bo_high=0
         let bf0 = 0x00u8;
-        let bf1 = 0x00u8;
+        // Sign bit: the top bit.
+        let bf1 = (size * 8 - 1) as u8;
         // mantissa norm = 2 (MSB not stored) in bits 24-31... wait, that's bf2
         let bf2 = 0x02u8; // norm = 2
         let mut buf = build_dt_header(1, 1, [bf0, bf1, bf2], size);
@@ -981,7 +1320,7 @@ mod tests {
         let levels = MAX_DATATYPE_DEPTH as usize + 10;
         let mut data = Vec::new();
         for _ in 0..levels {
-            data.extend_from_slice(&build_dt_header(9, 3, [0, 0, 0], 0));
+            data.extend_from_slice(&build_dt_header(9, 3, [0, 0, 0], 16));
         }
         data.extend_from_slice(&build_fixed_point(4, false, false, 0, 32));
 
@@ -995,7 +1334,7 @@ mod tests {
         let levels = MAX_DATATYPE_DEPTH as usize - 1;
         let mut data = Vec::new();
         for _ in 0..levels {
-            data.extend_from_slice(&build_dt_header(9, 3, [0, 0, 0], 0));
+            data.extend_from_slice(&build_dt_header(9, 3, [0, 0, 0], 16));
         }
         data.extend_from_slice(&build_fixed_point(4, false, false, 0, 32));
 
@@ -1145,8 +1484,8 @@ mod tests {
 
     #[test]
     fn test_opaque() {
-        // tag_len = 4, tag = "BLOB"
+        // tag = "BLOB"; the stored length is the NUL-padded length, 8
-        let mut buf = build_dt_header(5, 1, [4, 0, 0], 64);
+        let mut buf = build_dt_header(5, 1, [8, 0, 0], 64);
         buf.extend_from_slice(b"BLOB");
         // Pad to 8 bytes
         buf.extend_from_slice(&[0, 0, 0, 0]);
@@ -1336,6 +1675,64 @@ mod tests {
         assert_xyid_compound(dt);
     }
 
+    #[test]
+    fn test_compound_v1_member_array_fields() {
+        // HDF5 1.6 wrote array members of a v1 compound through the legacy
+        // per-member fields (as in libhdf5's tools/test/testfiles/
+        // tcompound.h5 `type2`: `int_array` [4] i32, `float_array` [5][6]
+        // f32). They used to be skipped, reading each member as a scalar.
+        let i32le: [u8; 12] = [
+            0x10, 0x08, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x00,
+        ];
+        let mut b = vec![0x16, 0x02, 0x00, 0x00, 0x88, 0x00, 0x00, 0x00];
+        for (name, offset, dims) in [
+            (&b"int_array"[..], 0u32, &[4u32][..]),
+            (&b"xy"[..], 16, &[5u32, 6][..]),
+        ] {
+            let mut padded = name.to_vec();
+            padded.resize((name.len() + 1 + 7) & !7, 0);
+            b.extend_from_slice(&padded);
+            b.extend_from_slice(&offset.to_le_bytes());
+            b.push(dims.len() as u8);
+            b.extend_from_slice(&[0u8; 3 + 4 + 4]); // reserved, permutation, reserved
+            for j in 0..4 {
+                b.extend_from_slice(&dims.get(j).copied().unwrap_or(0).to_le_bytes());
+            }
+            b.extend_from_slice(&i32le);
+        }
+        let (dt, consumed) = Datatype::parse(&b).unwrap();
+        assert_eq!(consumed, b.len());
+        let Datatype::Compound { members, .. } = dt else {
+            panic!("expected Compound, got {dt:?}");
+        };
+        let got: Vec<(&str, u64, u32, Option<Vec<u32>>)> = members
+            .iter()
+            .map(|m| {
+                let dims = match &m.datatype {
+                    Datatype::Array { dimensions, .. } => Some(dimensions.clone()),
+                    _ => None,
+                };
+                (m.name.as_str(), m.byte_offset, m.datatype.type_size(), dims)
+            })
+            .collect();
+        assert_eq!(
+            got,
+            vec![
+                ("int_array", 0, 16, Some(vec![4])),
+                ("xy", 16, 120, Some(vec![5, 6])),
+            ]
+        );
+
+        // More than four dimensions cannot be encoded, and libhdf5 refuses a
+        // zero-sized dimension (a fuzzed tcompound.h5, cve-2024-32616.h5).
+        let mut bad = b.clone();
+        bad[8 + 16 + 4] = 5;
+        assert!(Datatype::parse(&bad).is_err());
+        let mut bad = b.clone();
+        bad[8 + 16 + 4] = 2; // [4, 0]
+        assert!(Datatype::parse(&bad).is_err());
+    }
+
     #[test]
     fn test_compound_v1_truncated_is_error_not_panic() {
         let bytes = compound_v1_bytes();
@@ -1499,11 +1896,13 @@ mod tests {
         let (dt, _) = Datatype::parse(&buf).unwrap();
         match dt {
             Datatype::VariableLength {
+                size,
                 is_string,
                 padding,
                 charset,
                 base_type,
             } => {
+                assert_eq!(size, 16);
                 assert!(is_string);
                 assert_eq!(padding, Some(StringPadding::NullTerminate));
                 assert_eq!(charset, Some(CharacterSet::Utf8));
@@ -1524,11 +1923,13 @@ mod tests {
         let (dt, _) = Datatype::parse(&buf).unwrap();
         match dt {
             Datatype::VariableLength {
+                size,
                 is_string,
                 padding,
                 charset,
                 base_type,
             } => {
+                assert_eq!(size, 16);
                 assert!(!is_string);
                 assert_eq!(padding, None);
                 assert_eq!(charset, None);
@@ -1538,6 +1939,19 @@ mod tests {
         }
     }
 
+    #[test]
+    fn variable_length_size_is_the_stored_size() {
+        // A file with 4-byte offsets stores 12-byte VL elements (length 4 +
+        // address 4 + index 4); the type used to report 16 regardless, so
+        // every read laid the elements out 16 bytes apart.
+        let mut buf = build_dt_header(9, 1, [0x01, 0x00, 0], 12);
+        buf.extend_from_slice(&build_fixed_point(1, false, false, 0, 8));
+        let (dt, _) = Datatype::parse(&buf).unwrap();
+        assert_eq!(dt.type_size(), 12);
+        // And it is written back as stored.
+        assert_eq!(dt.serialize()[4..8], 12u32.to_le_bytes());
+    }
+
     #[test]
     fn test_array_2d() {
         // Array [3][4] of i32 LE, version 3
@@ -1960,4 +2374,273 @@ mod tests {
         };
         assert_eq!(dt.type_size(), 48);
     }
+    /// Every check here mirrors one in libhdf5's `H5O__dtype_decode_helper`;
+    /// the error text is libhdf5's.
+    fn invalid_reason(data: &[u8]) -> String {
+        match Datatype::parse(data) {
+            Err(FormatError::InvalidDatatype(why)) => why,
+            other => panic!("expected InvalidDatatype, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn size_zero_is_refused() {
+        // cve-2017-17508: a variable-length string member of stored size 0.
+        let mut data = build_dt_header(9, 1, [1, 0, 0], 0);
+        data.extend_from_slice(&build_fixed_point(1, false, false, 0, 8));
+        assert_eq!(invalid_reason(&data), "invalid datatype size");
+        // Except a fixed-length string, which clawhdf5 <= v2.7.0 wrote for an
+        // empty-string attribute.
+        assert!(Datatype::parse(&build_dt_header(3, 1, [0, 0, 0], 0)).is_ok());
+        assert_eq!(
+            invalid_reason(&build_fixed_point(0, false, false, 0, 0)),
+            "invalid datatype size"
+        );
+    }
+
+    #[test]
+    fn integer_bits_must_lie_inside_the_type() {
+        assert_eq!(
+            invalid_reason(&build_fixed_point(4, false, false, 32, 1)),
+            "integer offset out of bounds"
+        );
+        assert_eq!(
+            invalid_reason(&build_fixed_point(4, false, false, 0, 0)),
+            "precision is zero"
+        );
+        assert_eq!(
+            invalid_reason(&build_fixed_point(4, false, false, 8, 25)),
+            "integer offset+precision out of bounds"
+        );
+        // A partial-precision integer inside its bytes is fine.
+        assert!(Datatype::parse(&build_fixed_point(4, false, false, 12, 8)).is_ok());
+    }
+
+    #[test]
+    fn float_fields_must_lie_inside_the_type_and_not_overlap() {
+        // (sign, epos, esize, mpos, msize) on an f32
+        let f32_with = |sign: u8, epos: u8, esize: u8, mpos: u8, msize: u8| {
+            let mut data = build_dt_header(1, 1, [0x20, sign, 0], 4);
+            data.extend_from_slice(&0u16.to_le_bytes());
+            data.extend_from_slice(&32u16.to_le_bytes());
+            data.extend_from_slice(&[epos, esize, mpos, msize]);
+            data.extend_from_slice(&127u32.to_le_bytes());
+            data
+        };
+        assert!(Datatype::parse(&f32_with(31, 23, 8, 0, 23)).is_ok());
+        for (fields, why) in [
+            ((31, 23, 0, 0, 23), "exponent size can't be zero"),
+            (
+                (31, 32, 8, 0, 23),
+                "exponent starting position out of bounds",
+            ),
+            ((31, 30, 8, 0, 23), "exponent range out of bounds"),
+            ((31, 23, 8, 0, 0), "mantissa size can't be zero"),
+            (
+                (31, 23, 8, 40, 1),
+                "mantissa starting position out of bounds",
+            ),
+            // cve-2024-29163: a 128-bit mantissa in a 4-byte float.
+            ((31, 23, 8, 0, 128), "mantissa range out of bounds"),
+            ((23, 23, 8, 0, 23), "exponent and sign positions overlap"),
+            ((0, 23, 8, 0, 23), "mantissa and sign positions overlap"),
+            // cve-2026-34734.
+            (
+                (31, 20, 8, 0, 23),
+                "mantissa and exponent positions overlap",
+            ),
+        ] {
+            let (sign, epos, esize, mpos, msize) = fields;
+            assert_eq!(
+                invalid_reason(&f32_with(sign, epos, esize, mpos, msize)),
+                why,
+                "{fields:?}"
+            );
+        }
+        // Normalization 3 is undefined; bit 6 (VAX) needs bit 0 from v3.
+        let mut data = f32_with(31, 23, 8, 0, 23);
+        data[1] = 0x30;
+        assert_eq!(
+            invalid_reason(&data),
+            "unknown floating-point normalization"
+        );
+        let mut data = f32_with(31, 23, 8, 0, 23);
+        data[0] = 0x31; // version 3
+        data[1] = 0x60;
+        assert_eq!(invalid_reason(&data), "bad byte order for datatype message");
+    }
+
+    #[test]
+    fn unusual_unused_bits_are_refused_in_version_1_headers_only() {
+        // cve-2024-29162: a 3-bit integer in 4 bytes.
+        let data = build_fixed_point(4, false, true, 0, 3);
+        assert!(Datatype::parse_in_header(&data, 2).is_ok());
+        assert_eq!(
+            match Datatype::parse_in_header(&data, 1) {
+                Err(FormatError::InvalidDatatype(why)) => why,
+                other => panic!("{other:?}"),
+            },
+            "datatype has unusually large # of unused bits (prec = 3 bits, size = 4 bytes), \
+             possibly corrupted file"
+        );
+        // Half the bits used (with the offset) is not unusual; nor is a
+        // 1-byte type; nor a full-precision one.
+        for (size, offset, prec) in [(4u32, 0u16, 16u16), (4, 8, 8), (1, 0, 1), (8, 0, 64)] {
+            let data = build_fixed_point(size, false, true, offset, prec);
+            assert!(
+                Datatype::parse_in_header(&data, 1).is_ok(),
+                "{size} {offset} {prec}"
+            );
+        }
+        // Nested: a compound member's type is checked too.
+        let member = build_fixed_point(4, false, true, 0, 15);
+        let data = compound_v3(4, &[("a", 0, member)]);
+        assert!(Datatype::parse_in_header(&data, 2).is_ok());
+        assert!(Datatype::parse_in_header(&data, 1).is_err());
+    }
+
+    #[test]
+    fn f32_written_by_clawhdf5_up_to_2_7_0_still_parses() {
+        // Those versions put the sign bit at 63 whatever the float's size;
+        // libhdf5 refuses it ("sign bit position out of bounds").
+        let mut data = build_dt_header(1, 1, [0x20, 63, 0], 4);
+        data.extend_from_slice(&0u16.to_le_bytes());
+        data.extend_from_slice(&32u16.to_le_bytes());
+        data.extend_from_slice(&[23, 8, 0, 23]);
+        data.extend_from_slice(&127u32.to_le_bytes());
+        assert!(Datatype::parse(&data).is_ok());
+    }
+
+    #[test]
+    fn float_bit_6_is_vax_order_only_from_version_3() {
+        // h5py opens a v1 float with bit 6 set as an ordinary little-endian
+        // float; it used to be read as VAX order.
+        let mut data = build_float(4, 23, 8, 0, 23, 127);
+        data[1] |= 0x40;
+        match Datatype::parse(&data).unwrap().0 {
+            Datatype::FloatingPoint { byte_order, .. } => {
+                assert_eq!(byte_order, DatatypeByteOrder::LittleEndian)
+            }
+            other => panic!("{other:?}"),
+        }
+        data[0] = 0x31;
+        data[1] |= 0x01;
+        match Datatype::parse(&data).unwrap().0 {
+            Datatype::FloatingPoint { byte_order, .. } => {
+                assert_eq!(byte_order, DatatypeByteOrder::Vax)
+            }
+            other => panic!("{other:?}"),
+        }
+    }
+
+    #[test]
+    fn opaque_tag_length_must_be_padded() {
+        let mut data = build_dt_header(5, 1, [4, 0, 0], 4);
+        data.extend_from_slice(b"BLOB");
+        assert_eq!(invalid_reason(&data), "opaque flag field must be aligned");
+    }
+
+    /// A v3 compound of `size` bytes with `(name, offset, member)` members.
+    fn compound_v3(size: u32, members: &[(&str, u8, Vec<u8>)]) -> Vec<u8> {
+        let n = members.len() as u8;
+        let mut data = build_dt_header(6, 3, [n, 0, 0], size);
+        for (name, off, dt) in members {
+            data.extend_from_slice(name.as_bytes());
+            data.push(0);
+            data.push(*off);
+            data.extend_from_slice(dt);
+        }
+        data
+    }
+
+    #[test]
+    fn compound_members_are_checked() {
+        let i4 = build_fixed_point(4, false, true, 0, 32);
+        // cve-2016-4332: no members.
+        assert_eq!(
+            invalid_reason(&compound_v3(8, &[])),
+            "invalid number of members: 0"
+        );
+        assert_eq!(
+            invalid_reason(&compound_v3(
+                8,
+                &[("a", 0, i4.clone()), ("b", 6, i4.clone())]
+            )),
+            "member type extends outside its parent compound type"
+        );
+        assert_eq!(
+            invalid_reason(&compound_v3(
+                8,
+                &[("a", 0, i4.clone()), ("a", 4, i4.clone())]
+            )),
+            "duplicated compound field name 'a', for fields 0 and 1"
+        );
+        assert_eq!(
+            invalid_reason(&compound_v3(
+                8,
+                &[("a", 0, i4.clone()), ("b", 2, i4.clone())]
+            )),
+            "member overlaps with previous member"
+        );
+        assert_eq!(
+            invalid_reason(&compound_v3(
+                8,
+                &[("b", 4, i4.clone()), ("a", 2, i4.clone())]
+            )),
+            "member overlaps with previous member"
+        );
+        // Members out of offset order, and gaps, are fine.
+        assert!(Datatype::parse(&compound_v3(12, &[("b", 8, i4.clone()), ("a", 0, i4)])).is_ok());
+    }
+
+    #[test]
+    fn enum_is_checked() {
+        let base = build_fixed_point(4, false, true, 0, 32);
+        let enum_of = |size: u32, names: &[&str]| {
+            let mut data = build_dt_header(8, 3, [names.len() as u8, 0, 0], size);
+            data.extend_from_slice(&base);
+            for n in names {
+                data.extend_from_slice(n.as_bytes());
+                data.push(0);
+            }
+            for i in 0..names.len() as u32 {
+                data.extend_from_slice(&i.to_le_bytes());
+            }
+            data
+        };
+        assert!(Datatype::parse(&enum_of(4, &["RED", "GREEN"])).is_ok());
+        // cve-2024-32618.
+        assert_eq!(
+            invalid_reason(&enum_of(4, &["", "GREEN"])),
+            "0 length enum name"
+        );
+        assert_eq!(
+            invalid_reason(&enum_of(2, &["RED"])),
+            "ENUM datatype size does not match parent"
+        );
+    }
+
+    #[test]
+    fn array_dimensions_are_checked() {
+        let base = build_fixed_point(4, false, true, 0, 32);
+        let array_v3 = |dims: &[u32]| {
+            let n = dims.iter().product::<u32>().max(1);
+            let mut data = build_dt_header(10, 3, [0, 0, 0], 4 * n);
+            data.push(dims.len() as u8);
+            for d in dims {
+                data.extend_from_slice(&d.to_le_bytes());
+            }
+            data.extend_from_slice(&base);
+            data
+        };
+        assert!(Datatype::parse(&array_v3(&[2, 3])).is_ok());
+        assert_eq!(
+            invalid_reason(&array_v3(&[2, 0])),
+            "zero-sized dimension specified"
+        );
+        assert_eq!(
+            invalid_reason(&array_v3(&[1; 33])),
+            "too many dimensions for array datatype"
+        );
+    }
 }

crates/clawhdf5-format/src/ea_writer.rs

@@ -3,11 +3,14 @@
 #[cfg(not(feature = "std"))]
 extern crate alloc;
 
+use crate::addr::saturating_usize;
 #[cfg(not(feature = "std"))]
 use alloc::{vec, vec::Vec};
 
 use crate::checksum::jenkins_lookup3;
-use crate::chunked_write::{WrittenChunk, filtered_chunk_size_len, push_addr, push_index_element};
+use crate::chunked_write::{
+    WrittenChunk, filtered_chunk_size_len, push_addr, push_index_element, push_v4_chunk_dims,
+};
 
 /// Serialize a v4 Extensible Array layout message.
 pub(crate) fn serialize_v4_extensible_array(
@@ -24,35 +27,7 @@ pub(crate) fn serialize_v4_extensible_array(
     let ndims = chunk_dims.len() as u8 + 1;
     buf.push(ndims);
 
-    let max_dim = chunk_dims
+    push_v4_chunk_dims(&mut buf, chunk_dims, element_size);
-        .iter()
-        .map(|&d| d as u64)
-        .chain(core::iter::once(element_size as u64))
-        .max()
-        .unwrap_or(1);
-    let dim_encoded_len: u8 = if max_dim <= 0xFF {
-        1
-    } else if max_dim <= 0xFFFF {
-        2
-    } else {
-        4
-    };
-    buf.push(dim_encoded_len);
-
-    for &d in chunk_dims {
-        match dim_encoded_len {
-            1 => buf.push(d as u8),
-            2 => buf.extend_from_slice(&(d as u16).to_le_bytes()),
-            4 => buf.extend_from_slice(&d.to_le_bytes()),
-            _ => unreachable!("unexpected dim_encoded_len: {dim_encoded_len}"),
-        }
-    }
-    match dim_encoded_len {
-        1 => buf.push(element_size as u8),
-        2 => buf.extend_from_slice(&(element_size as u16).to_le_bytes()),
-        4 => buf.extend_from_slice(&element_size.to_le_bytes()),
-        _ => unreachable!("unexpected dim_encoded_len: {dim_encoded_len}"),
-    }
 
     // chunk index type = 4 (Extensible Array)
     buf.push(4);
@@ -273,7 +248,7 @@ pub fn build_extensible_array_at(
 
     // Header (EAHD). The six statistics are, in order: super blocks, their
     // bytes, data blocks, their bytes, max index set, elements realised.
-    let mut out = Vec::with_capacity((cursor - ea_base_address) as usize);
+    let mut out = Vec::with_capacity(saturating_usize(cursor - ea_base_address));
     out.extend_from_slice(b"EAHD");
     out.push(0); // version
     out.push(client_id);

crates/clawhdf5-format/src/error.rs

@@ -12,7 +12,11 @@ use std::string::String;
 use core::fmt;
 
 /// Errors that can occur when parsing HDF5 binary format structures.
+///
+/// Non-exhaustive: new failure modes (new storage backends, new file
+/// features) add variants, so a `match` needs a wildcard arm.
 #[derive(Debug, Clone, PartialEq, Eq)]
+#[non_exhaustive]
 pub enum FormatError {
     /// The HDF5 magic signature was not found at any valid offset.
     SignatureNotFound,
@@ -80,6 +84,9 @@ pub enum FormatError {
     InvalidLocalHeapSignature,
     /// Invalid local heap version.
     InvalidLocalHeapVersion(u8),
+    /// A local heap's free list points outside its data segment (libhdf5:
+    /// "bad heap free list").
+    InvalidLocalHeapFreeList,
     /// Invalid B-tree v1 signature.
     InvalidBTreeSignature,
     /// Invalid B-tree node type.
@@ -117,6 +124,14 @@ pub enum FormatError {
     /// A message is marked shared but was parsed without access to the file,
     /// so the reference to the real message could not be followed.
     UnresolvedSharedMessage,
+    /// A shared-message reference points at an object header that holds no
+    /// (unshared) message of the referenced type (raw message type id).
+    SharedMessageTargetMissing(u16),
+    /// A superblock was parsed at a non-zero offset of the buffer (the file
+    /// has a user block of this many bytes). HDF5 addresses are relative to
+    /// the superblock, so the buffer must start there: see
+    /// `signature::split_user_block`.
+    UserBlockNotStripped(u64),
     /// A selection does not fit the dataset it was applied to (wrong rank, or
     /// it reaches past a dimension's extent).
     SelectionOutOfBounds(String),
@@ -190,6 +205,55 @@ pub enum FormatError {
     DuplicateDatasetName(String),
     /// Integer overflow in size computation (malformed data protection).
     Overflow(String),
+    /// An object header that libhdf5 refuses to load (the reason is
+    /// libhdf5's own error text): a misaligned or overrunning message, a
+    /// wrong message count, contradictory message flags, a message of a
+    /// class that cannot be shared flagged shareable, …
+    InvalidObjectHeader(&'static str),
+    /// A datatype message libhdf5 refuses to decode (the reason is
+    /// libhdf5's own error text): size 0, bit fields outside the type,
+    /// an empty enum name, a compound member outside its compound, …
+    InvalidDatatype(String),
+    /// A chunked layout whose chunk dimensions libhdf5 refuses: a zero
+    /// dimension, a rank that does not match the dataspace, an element size
+    /// that is not the datatype's, or a chunk of 4 GiB or more indexed by a
+    /// version-1 B-tree.
+    InvalidChunkDimensions(String),
+    /// The superblock's end-of-file address lies past the end of the file:
+    /// the file was truncated (libhdf5 refuses to open it).
+    TruncatedFile {
+        /// End of file recorded in the superblock (relative to byte 0).
+        stored_eof: u64,
+        /// The file's actual length in bytes.
+        actual_len: u64,
+    },
+    /// A link libhdf5 refuses to list: a symbol-table entry with an empty
+    /// name ("invalid link name"). Listing the group fails, as in libhdf5.
+    InvalidLinkName,
+    /// A dataspace message libhdf5 refuses to decode (the reason is
+    /// libhdf5's own error text): more than 32 dimensions, a rank on a
+    /// scalar or null dataspace, a dimension larger than its maximum.
+    InvalidDataspace(&'static str),
+    /// A dataset whose storage libhdf5 refuses when it opens the dataset
+    /// (the reason is libhdf5's own error text): an element count times
+    /// element size that overflows, contiguous storage past the end of the
+    /// file, compact data of the wrong size.
+    InvalidDatasetStorage(&'static str),
+    /// A superblock extension message libhdf5 refuses to decode when it
+    /// opens the file (the reason is libhdf5's own error text): a File Space
+    /// Info message that runs off its end or has a bad page size, a metadata
+    /// cache image outside the file, …
+    InvalidSuperblockExtension(&'static str),
+    /// A metadata cache image block libhdf5 refuses to load (the reason is
+    /// libhdf5's own error text).
+    InvalidCacheImage(&'static str),
+    /// The [`Storage`](crate::storage::Storage) backend failed to serve a
+    /// read (an I/O or network error, or a short read inside the file).
+    Storage(String),
+    /// The operation still needs the whole file as one slice and the
+    /// [`Storage`](crate::storage::Storage) backend has no contiguous view
+    /// (`as_contiguous()` is `None`); the text names the operation.
+    ContiguousStorageRequired(&'static str),
 }
 
 impl fmt::Display for FormatError {
@@ -270,6 +334,9 @@ impl fmt::Display for FormatError {
             FormatError::InvalidLocalHeapSignature => {
                 write!(f, "invalid local heap signature")
             }
+            FormatError::InvalidLocalHeapFreeList => {
+                write!(f, "bad local heap free list")
+            }
             FormatError::InvalidLocalHeapVersion(v) => {
                 write!(f, "invalid local heap version: {v}")
             }
@@ -339,6 +406,16 @@ impl fmt::Display for FormatError {
             FormatError::SelectionOutOfBounds(msg) => {
                 write!(f, "selection out of bounds: {msg}")
             }
+            FormatError::UserBlockNotStripped(n) => write!(
+                f,
+                "file has a {n}-byte user block: parse the bytes from the superblock on \
+                 (signature::split_user_block)"
+            ),
+            FormatError::SharedMessageTargetMissing(t) => write!(
+                f,
+                "shared message reference points at an object header with no message of type \
+                 {t:#06x}"
+            ),
             FormatError::UnresolvedSharedMessage => write!(
                 f,
                 "message is shared but no file data was available to resolve it"
@@ -382,9 +459,17 @@ impl fmt::Display for FormatError {
             FormatError::InvalidFilterPipelineVersion(v) => {
                 write!(f, "invalid filter pipeline version: {v}")
             }
-            FormatError::UnsupportedFilter(id) => {
+            FormatError::UnsupportedFilter(id) => match crate::filter_registry::known_filter(*id) {
-                write!(f, "unsupported filter: {id}")
+                Some((name, Some(feature))) => write!(
-            }
+                    f,
+                    "unsupported filter: {id} ({name}; this build lacks the `{feature}` feature)"
+                ),
+                Some((name, None)) => write!(
+                    f,
+                    "unsupported filter: {id} ({name}, not implemented by clawhdf5)"
+                ),
+                None => write!(f, "unsupported filter: {id}"),
+            },
             FormatError::FilterError(msg) => {
                 write!(f, "filter error: {msg}")
             }
@@ -421,6 +506,50 @@ impl fmt::Display for FormatError {
             FormatError::Overflow(msg) => {
                 write!(f, "integer overflow: {msg}")
             }
+            FormatError::InvalidObjectHeader(why) => {
+                write!(f, "corrupt object header: {why}")
+            }
+            FormatError::InvalidDatatype(why) => {
+                write!(f, "invalid datatype: {why}")
+            }
+            FormatError::InvalidChunkDimensions(why) => {
+                write!(f, "invalid chunk dimensions: {why}")
+            }
+            FormatError::TruncatedFile {
+                stored_eof,
+                actual_len,
+            } => {
+                write!(
+                    f,
+                    "truncated file: the superblock records end of file {stored_eof}, \
+                     but the file is {actual_len} bytes"
+                )
+            }
+            FormatError::InvalidLinkName => {
+                write!(f, "invalid link name: a group entry has an empty name")
+            }
+            FormatError::InvalidDataspace(why) => {
+                write!(f, "invalid dataspace: {why}")
+            }
+            FormatError::InvalidDatasetStorage(why) => {
+                write!(f, "invalid dataset storage: {why}")
+            }
+            FormatError::InvalidSuperblockExtension(why) => {
+                write!(f, "invalid superblock extension: {why}")
+            }
+            FormatError::InvalidCacheImage(why) => {
+                write!(f, "invalid metadata cache image: {why}")
+            }
+            FormatError::Storage(why) => {
+                write!(f, "storage read failed: {why}")
+            }
+            FormatError::ContiguousStorageRequired(what) => {
+                write!(
+                    f,
+                    "{what} needs the whole file in memory, which this storage backend does \
+                     not provide"
+                )
+            }
         }
     }
 }

crates/clawhdf5-format/src/extensible_array.rs

@@ -9,19 +9,23 @@ extern crate alloc;
 #[cfg(not(feature = "std"))]
 use alloc::{format, vec, vec::Vec};
 
+use crate::addr::to_usize;
 use crate::chunk_grid::ChunkGrid;
 use crate::chunked_read::ChunkInfo;
 use crate::error::FormatError;
+use crate::storage::{PAGED_BLOCK_ONE_READ_MAX, Storage, Window, read_exact_at};
 
 /// Verify the Jenkins lookup3 checksum stored immediately after
-/// `data[start..end]`, as every Extensible Array structure carries one.
+/// `data[start..end]`, as every Extensible Array structure carries one. `w`
+/// is a window of the file and `start`/`end` are relative to it.
 ///
 /// A corrupt chunk index yields addresses pointing at the wrong bytes, so a
 /// mismatch is an error: otherwise the damage surfaces as plausible data read
 /// from the wrong chunk.
 #[cfg(feature = "checksum")]
-fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatError> {
+fn verify_checksum(w: &Window<'_>, start: usize, end: usize) -> Result<(), FormatError> {
-    ensure_len(data, end, 4)?;
+    w.ensure(end, 4)?;
+    let data: &[u8] = &w.bytes;
     let stored = u32::from_le_bytes([data[end], data[end + 1], data[end + 2], data[end + 3]]);
     let computed = crate::checksum::jenkins_lookup3(&data[start..end]);
     if computed != stored {
@@ -34,7 +38,7 @@ fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatEr
 }
 
 #[cfg(not(feature = "checksum"))]
-fn verify_checksum(_data: &[u8], _start: usize, _end: usize) -> Result<(), FormatError> {
+fn verify_checksum(_w: &Window<'_>, _start: usize, _end: usize) -> Result<(), FormatError> {
     Ok(())
 }
 
@@ -80,19 +84,6 @@ fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
     })
 }
 
-fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
-    if offset
-        .checked_add(needed)
-        .is_none_or(|end| end > data.len())
-    {
-        return Err(FormatError::UnexpectedEof {
-            expected: offset.saturating_add(needed),
-            available: data.len(),
-        });
-    }
-    Ok(())
-}
-
 fn is_undefined_addr(addr: u64, offset_size: u8) -> bool {
     match offset_size {
         2 => addr == 0xFFFF,
@@ -130,6 +121,16 @@ impl ExtensibleArrayHeader {
         offset: usize,
         offset_size: u8,
         length_size: u8,
+    ) -> Result<Self, FormatError> {
+        Self::parse_in(file_data, offset as u64, offset_size, length_size)
+    }
+
+    /// [`Self::parse`] over any [`Storage`]: one read of the header.
+    pub fn parse_in<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        offset_size: u8,
+        length_size: u8,
     ) -> Result<Self, FormatError> {
         // EAHD: signature(4) + version(1) + client_id(1) + element_size(1) +
         //   max_nelmts_bits(1) + idx_blk_elmts(1) + min_dblk_nelmts(1) +
@@ -137,9 +138,10 @@ impl ExtensibleArrayHeader {
         //   6 stats fields (each length_size) + index_block_address(offset_size) + checksum(4)
         let min_size =
             4 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 6 * length_size as usize + offset_size as usize + 4;
-        ensure_len(file_data, offset, min_size)?;
+        let w = Window::read(file, offset, min_size)?;
+        w.ensure(0, min_size)?;
 
-        let d = &file_data[offset..];
+        let d: &[u8] = &w.bytes;
         if &d[0..4] != b"EAHD" {
             return Err(FormatError::ChunkedReadError(
                 "invalid Extensible Array header signature".into(),
@@ -172,7 +174,7 @@ impl ExtensibleArrayHeader {
         pos += ls; // skip max_idx_set (6th stats field)
         let index_block_address = read_offset(d, pos, offset_size)?;
         pos += offset_size as usize;
-        verify_checksum(file_data, offset, offset + pos)?;
+        verify_checksum(&w, 0, pos)?;
 
         Ok(ExtensibleArrayHeader {
             client_id,
@@ -193,11 +195,11 @@ impl ExtensibleArrayHeader {
     }
 }
 
-/// Read a single element from the extensible array element data.
+/// Read a single element at offset `pos` of the window `w`.
 /// Returns (chunk_info, bytes_consumed) or None if unallocated.
 #[allow(clippy::too_many_arguments)]
 fn read_element(
-    data: &[u8],
+    w: &Window<'_>,
     pos: usize,
     client_id: u8,
     element_size: u8,
@@ -207,15 +209,11 @@ fn read_element(
     grid: &ChunkGrid,
 ) -> Result<(Option<ChunkInfo>, usize), FormatError> {
     let os = offset_size as usize;
+    let data: &[u8] = &w.bytes;
 
     if client_id == 0 {
         // Non-filtered: just address
-        if pos + os > data.len() {
+        w.ensure(pos, os)?;
-            return Err(FormatError::UnexpectedEof {
-                expected: pos + os,
-                available: data.len(),
-            });
-        }
         if is_undefined(data, pos, offset_size) {
             return Ok((None, os));
         }
@@ -243,15 +241,7 @@ fn read_element(
         }
         let chunk_size_bytes = es - os - 4;
         let elem_total = os + chunk_size_bytes + 4;
-        if pos
+        w.ensure(pos, elem_total)?;
-            .checked_add(elem_total)
-            .is_none_or(|end| end > data.len())
-        {
-            return Err(FormatError::UnexpectedEof {
-                expected: pos.saturating_add(elem_total),
-                available: data.len(),
-            });
-        }
         if is_undefined(data, pos, offset_size) {
             return Ok((None, elem_total));
         }
@@ -315,9 +305,9 @@ fn page_nelmts(header: &ExtensibleArrayHeader) -> Option<usize> {
 /// paged. The bitmap lives in the super block, not here — a paged data block
 /// stores only its prefix, then one slot per page.
 #[allow(clippy::too_many_arguments)]
-fn read_data_block_elements(
+fn read_data_block_elements<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file: &S,
-    db_offset: usize,
+    db_offset: u64,
     nelmts: usize,
     header: &ExtensibleArrayHeader,
     offset_size: u8,
@@ -330,21 +320,28 @@ fn read_data_block_elements(
     // EADB: signature(4) + version(1) + client_id(1) + header_address(offset_size)
     //       + block offset(arr_off_size)
     let db_header_size = 4 + 1 + 1 + offset_size as usize + arr_off_size(header);
-    ensure_len(file_data, db_offset, db_header_size)?;
+    let prefix = read_exact_at(file, db_offset, db_header_size)?;
 
-    if &file_data[db_offset..db_offset + 4] != b"EADB" {
+    if &prefix[0..4] != b"EADB" {
         return Err(FormatError::ChunkedReadError(
             "invalid Extensible Array data block signature".into(),
         ));
     }
 
-    let mut pos = db_offset + db_header_size;
+    // Positions below are relative to the data block.
+    let mut pos = db_header_size;
     let page = page_nelmts(header).ok_or_else(|| {
         FormatError::Overflow("Extensible Array page element count overflows usize".into())
     })?;
+    let elem_bytes = if header.client_id == 0 {
+        offset_size as usize
+    } else {
+        header.element_size as usize
+    };
 
     let mut chunks = Vec::new();
-    let read_run = |from: usize,
+    let read_run = |w: &Window<'_>,
+                    from: usize,
                     count: usize,
                     first_index: usize,
                     chunks: &mut Vec<ChunkInfo>|
@@ -352,7 +349,7 @@ fn read_data_block_elements(
         let mut p = from;
         for i in 0..count {
             let (info, consumed) = read_element(
-                file_data,
+                w,
                 p,
                 header.client_id,
                 header.element_size,
@@ -370,18 +367,19 @@ fn read_data_block_elements(
     };
 
     if nelmts <= page {
-        // Prefix and elements are covered by one checksum.
+        // Prefix and elements are covered by one checksum. One window holds
-        let elem_bytes = if header.client_id == 0 {
+        // all of it (or ends at the end of the file), so its bounds checks
-            offset_size as usize
+        // are the whole-file ones.
-        } else {
-            header.element_size as usize
-        };
         let end = nelmts
             .checked_mul(elem_bytes)
             .and_then(|b| pos.checked_add(b))
             .ok_or_else(|| FormatError::Overflow("Extensible Array data block span".into()))?;
-        verify_checksum(file_data, db_offset, end)?;
+        // The checksum's bounds check comes first: make it before reading.
-        read_run(pos, nelmts, start_index, &mut chunks)?;
+        #[cfg(feature = "checksum")]
+        Window::check_extent(file, db_offset, end, 4)?;
+        let w = Window::read(file, db_offset, end.saturating_add(4))?;
+        verify_checksum(&w, 0, end)?;
+        read_run(&w, pos, nelmts, start_index, &mut chunks)?;
         return Ok(chunks);
     }
 
@@ -389,18 +387,32 @@ fn read_data_block_elements(
     // each holding `page` elements followed by a checksum. Pages whose bit is
     // clear were never written; their slot still occupies the file, so stride
     // over it rather than reading zeros as addresses.
-    verify_checksum(file_data, db_offset, pos)?;
+    let npages = nelmts.div_ceil(page);
-    pos += 4;
+    // The whole data block in one window when it is small: every position
-    let elem_bytes = if header.client_id == 0 {
+    // checked below lies inside it (or past the end of the file). A larger
-        offset_size as usize
+    // block is read as its prefix, then each page in use on its own.
+    let block_len = pos
+        .saturating_add(4)
+        .saturating_add(npages.saturating_mul(page.saturating_mul(elem_bytes).saturating_add(4)));
+    let whole = if block_len <= PAGED_BLOCK_ONE_READ_MAX {
+        Some(Window::read(file, db_offset, block_len)?)
     } else {
-        header.element_size as usize
+        None
     };
+    let head_w;
+    let head = match &whole {
+        Some(w) => w,
+        None => {
+            head_w = Window::read(file, db_offset, pos + 4)?;
+            &head_w
+        }
+    };
+    verify_checksum(head, 0, pos)?;
+    pos += 4;
     let page_stride = page
         .checked_mul(elem_bytes)
         .and_then(|b| b.checked_add(4))
         .ok_or_else(|| FormatError::Overflow("Extensible Array page stride".into()))?;
-    let npages = nelmts.div_ceil(page);
     for p in 0..npages {
         // One bit per page across the whole super block, packed contiguously
         // and MSB-first within each byte, as H5VM_bit_get reads it.
@@ -410,10 +422,20 @@ fn read_data_block_elements(
             .is_some_and(|byte| byte & (0x80 >> (bit % 8)) != 0);
         if initialised {
             let count = core::cmp::min(page, nelmts - p * page);
+            // `w` holds the page from `base` on (positions below are
+            // relative to it, and `pos` to the data block).
+            let page_w;
+            let (w, base) = match &whole {
+                Some(w) => (w, 0),
+                None => {
+                    page_w = Window::read(file, db_offset.saturating_add(pos as u64), page_stride)?;
+                    (&page_w, pos)
+                }
+            };
             // Each page carries its own checksum, over a full page's worth of
             // slots even when the last one holds fewer live elements.
-            verify_checksum(file_data, pos, pos + page * elem_bytes)?;
+            verify_checksum(w, pos - base, pos - base + page * elem_bytes)?;
-            read_run(pos, count, start_index + p * page, &mut chunks)?;
+            read_run(w, pos - base, count, start_index + p * page, &mut chunks)?;
         }
         pos = pos
             .checked_add(page_stride)
@@ -435,6 +457,32 @@ pub fn read_extensible_array_chunks(
     chunk_dimensions: &[u32],
     element_size: u32,
     offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<ChunkInfo>, FormatError> {
+    read_extensible_array_chunks_in(
+        &file_data,
+        header,
+        dataset_dims,
+        max_dims,
+        chunk_dimensions,
+        element_size,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`read_extensible_array_chunks`] over any [`Storage`]: one read of the
+/// index block's prefix, one of the whole index block, and the same for
+/// every super block and data block it references.
+#[allow(clippy::too_many_arguments)]
+pub fn read_extensible_array_chunks_in<S: Storage + ?Sized>(
+    file: &S,
+    header: &ExtensibleArrayHeader,
+    dataset_dims: &[u64],
+    max_dims: Option<&[u64]>,
+    chunk_dimensions: &[u32],
+    element_size: u32,
+    offset_size: u8,
     _length_size: u8,
 ) -> Result<Vec<ChunkInfo>, FormatError> {
     let os = offset_size as usize;
@@ -451,19 +499,20 @@ pub fn read_extensible_array_chunks(
     // Parse index block (EAIB): signature(4) + version(1) + client_id(1)
     // + header address(offset_size), then the inline elements, then the
     // direct data block addresses, then the super block addresses.
-    let ib_offset = header.index_block_address as usize;
+    // Positions below are relative to the index block.
+    let ib_offset = header.index_block_address;
     let ib_header_size = 4 + 1 + 1 + os;
-    ensure_len(file_data, ib_offset, ib_header_size)?;
+    let prefix = read_exact_at(file, ib_offset, ib_header_size)?;
 
-    if &file_data[ib_offset..ib_offset + 4] != b"EAIB" {
+    if &prefix[0..4] != b"EAIB" {
         return Err(FormatError::ChunkedReadError(
             "invalid Extensible Array index block signature".into(),
         ));
     }
-    let mut pos = ib_offset + ib_header_size;
+    let mut pos = ib_header_size;
 
     let mut chunks = Vec::new();
-    let total_elements = header.num_elements as usize;
+    let total_elements = to_usize(header.num_elements)?;
 
     let dmin = header.min_dblk_nelmts as usize;
     if dmin == 0 || !dmin.is_power_of_two() {
@@ -520,13 +569,19 @@ pub fn read_extensible_array_chunks(
                 .and_then(|n| n.checked_mul(os).and_then(|b| p.checked_add(b)))
         })
         .ok_or_else(|| FormatError::Overflow("Extensible Array index block span".into()))?;
-    verify_checksum(file_data, ib_offset, ib_end)?;
+    // The whole index block in one window: every position read below is
+    // before `ib_end`.
+    // The checksum's bounds check comes first: make it before reading.
+    #[cfg(feature = "checksum")]
+    Window::check_extent(file, ib_offset, ib_end, 4)?;
+    let w = Window::read(file, ib_offset, ib_end.saturating_add(4))?;
+    verify_checksum(&w, 0, ib_end)?;
 
     // 1. Elements stored inline in the index block.
     let n_inline = (header.idx_blk_elmts as usize).min(total_elements);
     for i in 0..n_inline {
         let (info, consumed) = read_element(
-            file_data,
+            &w,
             pos,
             header.client_id,
             header.element_size,
@@ -550,8 +605,8 @@ pub fn read_extensible_array_chunks(
         if global_index >= total_elements {
             return Ok(chunks);
         }
-        ensure_len(file_data, pos, os)?;
+        w.ensure(pos, os)?;
-        let addr = read_offset(file_data, pos, offset_size)?;
+        let addr = read_offset(&w.bytes, pos, offset_size)?;
         pos += os;
         if !is_undefined_addr(addr, offset_size) {
             if dblk_nelmts > page_nelmts(header).unwrap_or(usize::MAX) {
@@ -562,8 +617,8 @@ pub fn read_extensible_array_chunks(
                 ));
             }
             chunks.extend(read_data_block_elements(
-                file_data,
+                file,
-                addr as usize,
+                addr,
                 dblk_nelmts,
                 header,
                 offset_size,
@@ -583,16 +638,16 @@ pub fn read_extensible_array_chunks(
         if global_index >= total_elements {
             break;
         }
-        ensure_len(file_data, pos, os)?;
+        w.ensure(pos, os)?;
-        let sb_addr = read_offset(file_data, pos, offset_size)?;
+        let sb_addr = read_offset(&w.bytes, pos, offset_size)?;
         pos += os;
         let (ndblks, dblk_nelmts) = sblk_info(u, dmin).ok_or_else(|| {
             FormatError::Overflow("Extensible Array super block layout overflows usize".into())
         })?;
         if !is_undefined_addr(sb_addr, offset_size) {
             chunks.extend(read_super_block(
-                file_data,
+                file,
-                sb_addr as usize,
+                sb_addr,
                 ndblks,
                 dblk_nelmts,
                 header,
@@ -617,9 +672,9 @@ pub fn read_extensible_array_chunks(
 /// + block offset + the page-init bitmap for every data block it owns
 /// + one address per data block + checksum.
 #[allow(clippy::too_many_arguments)]
-fn read_super_block(
+fn read_super_block<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file: &S,
-    sb_offset: usize,
+    sb_offset: u64,
     ndblks: usize,
     dblk_nelmts: usize,
     header: &ExtensibleArrayHeader,
@@ -630,9 +685,9 @@ fn read_super_block(
 ) -> Result<Vec<ChunkInfo>, FormatError> {
     let os = offset_size as usize;
     let sb_header_size = 4 + 1 + 1 + os + arr_off_size(header);
-    ensure_len(file_data, sb_offset, sb_header_size)?;
+    let prefix = read_exact_at(file, sb_offset, sb_header_size)?;
 
-    if &file_data[sb_offset..sb_offset + 4] != b"EASB" {
+    if &prefix[0..4] != b"EASB" {
         return Err(FormatError::ChunkedReadError(
             "invalid Extensible Array super block signature".into(),
         ));
@@ -654,29 +709,38 @@ fn read_super_block(
     let bitmap_bytes = per_dblk_bitmap
         .checked_mul(ndblks)
         .ok_or_else(|| FormatError::Overflow("Extensible Array page bitmap size".into()))?;
-    let bitmap_start = sb_offset + sb_header_size;
+    // Positions below are relative to the super block, whose bytes (up to
-    ensure_len(file_data, bitmap_start, bitmap_bytes)?;
+    // its checksum) are all in one window.
-    let bitmap = &file_data[bitmap_start..bitmap_start + bitmap_bytes];
+    let bitmap_start = sb_header_size;
-
+    // The bitmap's bounds check, then (with checksums) the checksum's, come
+    // before anything else is read from the block: make them before reading
+    // it, so size fields stretching it past the end of the file cost no read.
+    Window::check_extent(file, sb_offset, bitmap_start, bitmap_bytes)?;
     let mut pos = bitmap_start + bitmap_bytes;
-    let mut chunks = Vec::new();
-    let mut global_idx = start_index;
 
     // One checksum covers the prefix, the bitmap and every data block address.
     let sb_end = ndblks
         .checked_mul(os)
         .and_then(|b| pos.checked_add(b))
         .ok_or_else(|| FormatError::Overflow("Extensible Array super block span".into()))?;
-    verify_checksum(file_data, sb_offset, sb_end)?;
+    #[cfg(feature = "checksum")]
+    Window::check_extent(file, sb_offset, sb_end, 4)?;
+    let w = Window::read(file, sb_offset, sb_end.saturating_add(4))?;
+    w.ensure(bitmap_start, bitmap_bytes)?;
+    let bitmap = &w.bytes[bitmap_start..bitmap_start + bitmap_bytes];
+
+    let mut chunks = Vec::new();
+    let mut global_idx = start_index;
+    verify_checksum(&w, 0, sb_end)?;
 
     for i in 0..ndblks {
-        ensure_len(file_data, pos, os)?;
+        w.ensure(pos, os)?;
-        let addr = read_offset(file_data, pos, offset_size)?;
+        let addr = read_offset(&w.bytes, pos, offset_size)?;
         pos += os;
         if !is_undefined_addr(addr, offset_size) {
             chunks.extend(read_data_block_elements(
-                file_data,
+                file,
-                addr as usize,
+                addr,
                 dblk_nelmts,
                 header,
                 offset_size,
@@ -887,11 +951,11 @@ mod tests {
         assert_eq!(chunks[1].offsets, vec![20]);
     }
 
-    /// Build a synthetic EA with inline elements + one direct data block.
+    /// A synthetic EA with inline elements + one direct data block: the
-    #[test]
+    /// file, with the header at 0x100 (8-byte offsets and lengths, 4 chunks
-    fn read_inline_plus_data_blocks() {
+    /// of 10 elements from 0x1000 on).
+    fn build_inline_plus_data_blocks() -> Vec<u8> {
         let os: u8 = 8;
-        let ls: u8 = 8;
         let osv = os as usize;
         let chunk_byte_size = 10u64 * 8; // 10 elements × 8 bytes
         let idx_blk_elmts = 2u8;
@@ -981,8 +1045,17 @@ mod tests {
             dbpos += osv;
         }
         stamp_checksum(&mut file_data, aedb_offset, dbpos);
+        file_data
+    }
 
-        let header = ExtensibleArrayHeader::parse(&file_data, aehd_offset, os, ls).unwrap();
+    /// Build a synthetic EA with inline elements + one direct data block.
+    #[test]
+    fn read_inline_plus_data_blocks() {
+        let (os, ls) = (8u8, 8u8);
+        let chunk_byte_size = 10u64 * 8;
+        let base_addr = 0x1000u64;
+        let file_data = build_inline_plus_data_blocks();
+        let header = ExtensibleArrayHeader::parse(&file_data, 0x100, os, ls).unwrap();
         let ds_dims = vec![40u64];
         let chunk_dims = vec![10u32];
         let chunks = read_extensible_array_chunks(
@@ -1018,7 +1091,8 @@ mod tests {
     fn read_element_unallocated() {
         let data = vec![0xFFu8; 16];
         let grid = ChunkGrid::fixed_array(&[50], None, &[10]).unwrap();
-        let (info, consumed) = read_element(&data, 0, 0, 8, 8, 80, 0, &grid).unwrap();
+        let (info, consumed) =
+            read_element(&Window::whole(&data), 0, 0, 8, 8, 80, 0, &grid).unwrap();
         assert!(info.is_none());
         assert_eq!(consumed, 8);
     }
@@ -1038,8 +1112,17 @@ mod tests {
         data[12..16].copy_from_slice(&0u32.to_le_bytes());
 
         let grid = ChunkGrid::fixed_array(&[50], None, &[10]).unwrap();
-        let (info, consumed) =
+        let (info, consumed) = read_element(
-            read_element(&data, 0, 1, elem_size as u8, os, 80, 2, &grid).unwrap();
+            &Window::whole(&data),
+            0,
+            1,
+            elem_size as u8,
+            os,
+            80,
+            2,
+            &grid,
+        )
+        .unwrap();
         let ci = info.unwrap();
         assert_eq!(ci.address, 0x2000);
         assert_eq!(ci.chunk_size, 120);
@@ -1047,4 +1130,38 @@ mod tests {
         assert_eq!(ci.offsets, vec![20]);
         assert_eq!(consumed, elem_size);
     }
+
+    /// The Storage path reads exactly what the slice path reads: the array
+    /// whole, cut at every length through its structures, and with a byte
+    /// damaged in each of them, through a read_at-only CountingStorage.
+    #[test]
+    fn storage_reads_match_slice_reads() {
+        use crate::storage::CountingStorage;
+        let full = build_inline_plus_data_blocks();
+        let mut files = Vec::new();
+        for cut in 0x100..0x340 {
+            files.push(full[..cut].to_vec());
+        }
+        for at in [0x104, 0x150, 0x204, 0x216, 0x230, 0x304, 0x318] {
+            let mut damaged = full.clone();
+            damaged[at] ^= 1;
+            files.push(damaged);
+        }
+        files.push(full);
+        let mut compared = 0;
+        for f in files {
+            let storage = CountingStorage::new(f.clone());
+            let want = ExtensibleArrayHeader::parse(&f, 0x100, 8, 8);
+            let got = ExtensibleArrayHeader::parse_in(&storage, 0x100, 8, 8);
+            assert_eq!(format!("{got:?}"), format!("{want:?}"));
+            let Ok(h) = want else { continue };
+            for dims in [&[40u64][..], &[25]] {
+                let want = read_extensible_array_chunks(&f, &h, dims, None, &[10], 8, 8, 8);
+                let got = read_extensible_array_chunks_in(&storage, &h, dims, None, &[10], 8, 8, 8);
+                assert_eq!(format!("{got:?}"), format!("{want:?}"), "{} bytes", f.len());
+                compared += 1;
+            }
+        }
+        assert!(compared > 100);
+    }
 }

crates/clawhdf5-format/src/fill_value.rs

@@ -12,7 +12,8 @@
 #[cfg(not(feature = "std"))]
 use alloc::{format, vec, vec::Vec};
 
-use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks};
+use crate::addr::to_usize;
+use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks_in};
 use crate::data_layout::DataLayout;
 use crate::dataspace::Dataspace;
 use crate::error::FormatError;
@@ -116,9 +117,22 @@ pub fn dataset_fill_value_in(
     messages: &[HeaderMessage],
     offset_size: u8,
     length_size: u8,
+) -> Result<Option<Vec<u8>>, FormatError> {
+    dataset_fill_value_from_storage(&file_data, messages, offset_size, length_size)
+}
+
+/// [`dataset_fill_value_in`] with the file behind any
+/// [`Storage`](crate::storage::Storage) (a `&dyn Storage` too). (The trait
+/// is not imported here: its `len` would shadow the slice method in this
+/// module.)
+pub fn dataset_fill_value_from_storage<S: crate::storage::Storage + ?Sized>(
+    file: &S,
+    messages: &[HeaderMessage],
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Option<Vec<u8>>, FormatError> {
     fill_value_from(messages, |msg| {
-        crate::shared_message::message_data_with_sohm(file_data, msg, offset_size, length_size)
+        crate::shared_message::message_data_with_sohm_in(file, msg, offset_size, length_size)
             .map(|data| data.into_owned())
     })
 }
@@ -199,6 +213,30 @@ pub fn read_full_with_fill<E: From<FormatError>>(
     offset_size: u8,
     length_size: u8,
     read: impl FnOnce() -> Result<Vec<u8>, E>,
+) -> Result<Vec<u8>, E> {
+    read_full_with_fill_in(
+        messages,
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        offset_size,
+        length_size,
+        read,
+    )
+}
+
+/// [`read_full_with_fill`] over any [`Storage`](crate::storage::Storage).
+#[allow(clippy::too_many_arguments)]
+pub fn read_full_with_fill_in<E: From<FormatError>, S: crate::storage::Storage + ?Sized>(
+    messages: &[HeaderMessage],
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    offset_size: u8,
+    length_size: u8,
+    read: impl FnOnce() -> Result<Vec<u8>, E>,
 ) -> Result<Vec<u8>, E> {
     // A dataset with external raw data also has no data address in this
     // file. It is NOT unallocated — its values live elsewhere — so it must
@@ -209,12 +247,12 @@ pub fn read_full_with_fill<E: From<FormatError>>(
     {
         return Err(FormatError::ExternalDataFilesUnsupported.into());
     }
-    let fill = dataset_fill_value_in(file_data, messages, offset_size, length_size)?;
+    let fill = dataset_fill_value_from_storage(file_data, messages, offset_size, length_size)?;
     if !has_storage(layout) {
         return Ok(filled_dataset(dataspace, elem_size, fill.as_deref())?);
     }
     let mut output = read()?;
-    apply_to_unallocated_chunks(
+    apply_to_unallocated_chunks_in(
         &mut output,
         file_data,
         layout,
@@ -240,6 +278,30 @@ pub fn apply_to_unallocated_chunks(
     fill: Option<&[u8]>,
     offset_size: u8,
     length_size: u8,
+) -> Result<(), FormatError> {
+    apply_to_unallocated_chunks_in(
+        output,
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        fill,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`apply_to_unallocated_chunks`] over any [`Storage`](crate::storage::Storage).
+#[allow(clippy::too_many_arguments)]
+pub fn apply_to_unallocated_chunks_in<S: crate::storage::Storage + ?Sized>(
+    output: &mut [u8],
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    fill: Option<&[u8]>,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<(), FormatError> {
     let Some(fill) = fill.filter(|f| f.len() == elem_size && !is_default(Some(f))) else {
         return Ok(());
@@ -247,7 +309,7 @@ pub fn apply_to_unallocated_chunks(
     if !matches!(layout, DataLayout::Chunked { .. }) || elem_size == 0 {
         return Ok(());
     }
-    let (chunks, chunk_dims) = list_chunks(
+    let (chunks, chunk_dims) = list_chunks_in(
         file_data,
         layout,
         dataspace,
@@ -256,7 +318,11 @@ pub fn apply_to_unallocated_chunks(
         length_size,
     )?;
     let rank = chunk_dims.len();
-    let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
+    let ds_dims: Vec<usize> = dataspace
+        .dimensions
+        .iter()
+        .map(|&d| to_usize(d))
+        .collect::<Result<_, _>>()?;
     if rank == 0 || ds_dims.len() != rank || chunk_dims.contains(&0) {
         return Ok(());
     }
@@ -288,7 +354,7 @@ pub fn apply_to_unallocated_chunks(
         let mut cell = 0usize;
         let mut in_range = true;
         for d in 0..rank {
-            let coord = chunk.offsets[d] as usize / chunk_dims[d];
+            let coord = to_usize(chunk.offsets[d])? / chunk_dims[d];
             if coord >= grid[d] {
                 in_range = false;
                 break;
@@ -439,4 +505,37 @@ mod tests {
             .collect();
         assert_eq!(filled, [2, 3, 7, 8]);
     }
+
+    /// Fill values, shared ones in the SOHM heap included, resolve
+    /// identically through a read_at-only CountingStorage.
+    #[test]
+    fn storage_reads_match_slice_reads() {
+        use crate::object_header::ObjectHeader;
+        use crate::storage::CountingStorage;
+        let file: &[u8] = include_bytes!("../tests/fixtures/shared_fill_value.h5");
+        let sb = crate::superblock::Superblock::parse(file, 0).unwrap();
+        let (os, ls) = (sb.offset_size, sb.length_size);
+        let storage = CountingStorage::new(file.to_vec());
+        let mut shared = 0;
+        let children =
+            crate::group_v2::resolve_group_children(file, &sb, sb.root_group_address).unwrap();
+        assert!(children.len() >= 3);
+        for child in children {
+            let h =
+                ObjectHeader::parse(file, child.object_header_address as usize, os, ls).unwrap();
+            shared += h
+                .messages
+                .iter()
+                .filter(|m| {
+                    m.msg_type == MessageType::FillValue
+                        && crate::shared_message::is_shared(m.flags)
+                })
+                .count();
+            let want = dataset_fill_value_in(file, &h.messages, os, ls);
+            assert_eq!(want, Ok(Some((-7i32).to_le_bytes().to_vec())));
+            let got = dataset_fill_value_from_storage(&storage, &h.messages, os, ls);
+            assert_eq!(got, want, "{}", child.name);
+        }
+        assert!(shared >= 2);
+    }
 }

crates/clawhdf5-format/src/filter_pipeline.rs

@@ -19,6 +19,19 @@ pub const FILTER_SCALEOFFSET: u16 = 6;
 pub const FILTER_LZ4: u16 = 32004;
 /// Zstandard compression.
 pub const FILTER_ZSTD: u16 = 32015;
+/// bzip2 (registered by PyTables; hdf5plugin's `BZip2`).
+pub const FILTER_BZIP2: u16 = 307;
+/// LZF — h5py's built-in `compression="lzf"`.
+pub const FILTER_LZF: u16 = 32000;
+/// Blosc 1 (hdf5-blosc; hdf5plugin's `Blosc`).
+pub const FILTER_BLOSC: u16 = 32001;
+/// Bitshuffle, optionally with LZ4 or Zstandard (hdf5plugin's `Bitshuffle`).
+pub const FILTER_BITSHUFFLE: u16 = 32008;
+/// ZFP lossy (and lossless) compression of numeric arrays (H5Z-ZFP;
+/// hdf5plugin's `Zfp`). Read-only, with the `zfp` feature.
+pub const FILTER_ZFP: u16 = 32013;
+/// Blosc 2 (hdf5plugin's `Blosc2`).
+pub const FILTER_BLOSC2: u16 = 32026;
 /// Pcodec lossless numerical codec — a **private, unregistered** clawhdf5
 /// filter. Pcodec has no ID in the HDF Group's filter registry (checked
 /// 2026-09-25, `hdf5_plugins/docs/RegisteredFilterPlugins.md`), so it uses an

crates/clawhdf5-format/src/filter_registry.rs

@@ -0,0 +1,496 @@
+//! Filter registry: every filter is looked up here by its HDF5 filter ID.
+//!
+//! Two tiers:
+//!
+//! * **Built-in filters** — a static table of the filters compiled into this
+//!   build: the HDF5 standard filters (deflate, shuffle, Fletcher32, szip,
+//!   N-Bit, scale-offset) and the plugin filters whose cargo features are
+//!   enabled (LZ4, Zstandard, pcodec, LZF, bitshuffle, bzip2, blosc,
+//!   blosc2, zfp).
+//!   [`builtin_filters`] lists them.
+//! * **Registered filters** (`std` only) — codecs the application supplies
+//!   for any other ID with [`register_filter`] (a [`FilterCodec`], or just a
+//!   decoding closure). A registered codec cannot shadow a built-in one,
+//!   except under 32023: that ID belongs to Granular BitRound, and the
+//!   built-in entry there only reads the pcodec chunks clawhdf5 <= 2.7.0
+//!   wrote (filter name `"pcodec"`), so a codec registered for 32023 handles
+//!   every other chunk with that ID, and writes.
+//!
+//! An ID in neither tier fails with [`FormatError::UnsupportedFilter`], as it
+//! always has.
+//!
+//! ```
+//! # #[cfg(feature = "std")] {
+//! use clawhdf5_format::filter_registry::{self, FilterContext};
+//! use clawhdf5_format::error::FormatError;
+//!
+//! // A toy filter in the private-use range: every byte XORed with 0x5A.
+//! filter_registry::register_filter(300, |input: &[u8], _ctx: &FilterContext<'_>| {
+//!     Ok::<_, FormatError>(input.iter().map(|b| b ^ 0x5A).collect())
+//! })
+//! .unwrap();
+//! assert!(filter_registry::is_filter_available(300));
+//! filter_registry::unregister_filter(300);
+//! # }
+//! ```
+
+#[cfg(not(feature = "std"))]
+extern crate alloc;
+
+#[cfg(not(feature = "std"))]
+use alloc::vec::Vec;
+
+use crate::error::FormatError;
+use crate::filter_pipeline::FilterDescription;
+
+/// What a codec is told about the filter it is applying.
+#[derive(Debug, Clone, Copy)]
+pub struct FilterContext<'a> {
+    /// The filter as recorded in the dataset's filter pipeline: its ID, name,
+    /// flags and client data (`cd_values`).
+    pub filter: &'a FilterDescription,
+    /// Size in bytes of one dataset element (the datatype's size).
+    pub element_size: usize,
+    /// Decoding only: the most bytes this stage may produce — what entered
+    /// the filter when the chunk was written. 0 means unknown; a decoder then
+    /// falls back to a fixed ceiling. Always 0 when encoding.
+    pub max_output: usize,
+}
+
+impl FilterContext<'_> {
+    /// The filter's client data (`cd_values`).
+    pub fn client_data(&self) -> &[u32] {
+        &self.filter.client_data
+    }
+
+    /// The largest output a decoder should allow: [`Self::max_output`], or
+    /// 256 MiB when that is unknown.
+    pub fn output_limit(&self) -> usize {
+        if self.max_output != 0 {
+            self.max_output
+        } else {
+            crate::filters::MAX_DECOMPRESS_SIZE
+        }
+    }
+}
+
+/// A filter implementation.
+///
+/// `decode` undoes the filter (the read direction). `encode` applies it (the
+/// write direction); the default refuses with
+/// [`FormatError::UnsupportedFilter`], which is right for a read-only codec.
+pub trait FilterCodec: Send + Sync {
+    /// Undo the filter on one chunk. The output must not exceed
+    /// [`FilterContext::output_limit`]; the pipeline rejects a larger one.
+    fn decode(&self, input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError>;
+
+    /// Apply the filter to one chunk.
+    fn encode(&self, input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+        let _ = input;
+        Err(FormatError::UnsupportedFilter(ctx.filter.filter_id))
+    }
+}
+
+/// Any `Fn(&[u8], &FilterContext) -> Result<Vec<u8>, FormatError>` is a
+/// decode-only codec.
+impl<F> FilterCodec for F
+where
+    F: Fn(&[u8], &FilterContext<'_>) -> Result<Vec<u8>, FormatError> + Send + Sync,
+{
+    fn decode(&self, input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+        self(input, ctx)
+    }
+}
+
+/// Signature of a built-in filter's decoder or encoder.
+pub type BuiltinFn = fn(&[u8], &FilterContext<'_>) -> Result<Vec<u8>, FormatError>;
+
+/// A filter compiled into this build.
+#[derive(Debug, Clone, Copy)]
+pub struct BuiltinFilter {
+    /// HDF5 filter ID.
+    pub id: u16,
+    /// Human-readable name.
+    pub name: &'static str,
+    /// Decoder.
+    pub(crate) decode: BuiltinFn,
+    /// Encoder, if this build can write the filter.
+    pub(crate) encode: Option<BuiltinFn>,
+}
+
+impl BuiltinFilter {
+    /// Whether this build can write the filter as well as read it.
+    pub fn can_encode(&self) -> bool {
+        self.encode.is_some()
+    }
+
+    /// Whether the built-in entry only borrows its ID for some chunks, so a
+    /// registered codec may take the rest: the legacy pcodec entry under
+    /// Granular BitRound's 32023, which claims only chunks named `"pcodec"`.
+    fn is_shared(&self) -> bool {
+        self.id == crate::filter_pipeline::FILTER_PCODEC_LEGACY
+    }
+
+    /// Whether this entry decodes chunks written with `filter`.
+    fn claims(&self, filter: &crate::filter_pipeline::FilterDescription) -> bool {
+        !self.is_shared()
+            || filter.name.as_deref() == Some(crate::filter_pipeline::FILTER_PCODEC_LEGACY_NAME)
+    }
+}
+
+/// The filters compiled into this build, in ID order.
+pub fn builtin_filters() -> &'static [BuiltinFilter] {
+    crate::filters::BUILTIN_FILTERS
+}
+
+/// The built-in filter with this ID, if it is compiled in.
+pub fn builtin_filter(id: u16) -> Option<&'static BuiltinFilter> {
+    builtin_filters().iter().find(|f| f.id == id)
+}
+
+/// Why a filter ID may be missing from this build: the filter's name, and
+/// the cargo feature that provides it (`None`: clawhdf5 does not implement
+/// it — register a codec for it with [`register_filter`]). `None` for an ID
+/// clawhdf5 knows nothing about.
+pub fn known_filter(id: u16) -> Option<(&'static str, Option<&'static str>)> {
+    Some(match id {
+        1 => ("deflate", Some("deflate")),
+        4 => ("SZIP", Some("szip")),
+        307 => ("bzip2", Some("bzip2")),
+        480 => ("pcodec", Some("pcodec")),
+        32000 => ("LZF", Some("lzf")),
+        32001 => ("Blosc", Some("blosc")),
+        32004 => ("LZ4", Some("lz4")),
+        32008 => ("bitshuffle", Some("bitshuffle")),
+        32013 => ("ZFP", Some("zfp")),
+        32015 => ("Zstandard", Some("zstd")),
+        32019 => ("JPEG", None),
+        32022 => ("BitGroom", None),
+        32023 => ("Granular BitRound", None),
+        32026 => ("Blosc2", Some("blosc2")),
+        _ => return None,
+    })
+}
+
+/// Whether a chunk filtered with `id` can be decoded: a built-in filter or a
+/// registered one.
+pub fn is_filter_available(id: u16) -> bool {
+    if builtin_filter(id).is_some() {
+        return true;
+    }
+    #[cfg(feature = "std")]
+    {
+        registered(id).is_some()
+    }
+    #[cfg(not(feature = "std"))]
+    {
+        false
+    }
+}
+
+/// Whether chunks filtered with `id` may be decoded by a codec the
+/// application registered (whose stored sizes this crate cannot bound).
+pub(crate) fn may_be_registered(id: u16) -> bool {
+    if builtin_filter(id).is_some_and(|b| !b.is_shared()) {
+        return false;
+    }
+    #[cfg(feature = "std")]
+    {
+        registered(id).is_some()
+    }
+    #[cfg(not(feature = "std"))]
+    {
+        false
+    }
+}
+
+#[cfg(feature = "std")]
+mod custom {
+    use super::FilterCodec;
+    use std::collections::BTreeMap;
+    use std::sync::{Arc, PoisonError, RwLock};
+
+    pub(super) type Registry = BTreeMap<u16, Arc<dyn FilterCodec>>;
+
+    static REGISTRY: RwLock<Registry> = RwLock::new(BTreeMap::new());
+
+    pub(super) fn with_read<R>(f: impl FnOnce(&Registry) -> R) -> R {
+        // A panic while holding the lock cannot leave the map half-updated
+        // (every update is a single insert/remove), so poisoning is ignored.
+        f(&REGISTRY.read().unwrap_or_else(PoisonError::into_inner))
+    }
+
+    pub(super) fn with_write<R>(f: impl FnOnce(&mut Registry) -> R) -> R {
+        f(&mut REGISTRY.write().unwrap_or_else(PoisonError::into_inner))
+    }
+}
+
+/// Register a codec for filter `id`, process-wide. It is used for every
+/// chunk read (and, if it implements [`FilterCodec::encode`], written) with
+/// that filter ID, by every file.
+///
+/// A plain closure `Fn(&[u8], &FilterContext) -> Result<Vec<u8>, FormatError>`
+/// registers a decoder. Replaces (and returns) an earlier registration for
+/// the same ID. Fails with [`FormatError::FilterError`] if `id` is a built-in
+/// filter of this build: those cannot be overridden. The exception is 32023
+/// (Granular BitRound): with the `pcodec` feature the built-in entry there
+/// reads only chunks whose filter is named `"pcodec"` (clawhdf5 <= 2.7.0's
+/// files); a codec registered for 32023 decodes every other chunk with that
+/// ID and does all the writing.
+#[cfg(feature = "std")]
+pub fn register_filter<C>(
+    id: u16,
+    codec: C,
+) -> Result<Option<std::sync::Arc<dyn FilterCodec>>, FormatError>
+where
+    C: FilterCodec + 'static,
+{
+    if let Some(builtin) = builtin_filter(id).filter(|b| !b.is_shared()) {
+        return Err(FormatError::FilterError(format!(
+            "filter {id} ({}) is built in and cannot be re-registered",
+            builtin.name
+        )));
+    }
+    let codec: std::sync::Arc<dyn FilterCodec> = std::sync::Arc::new(codec);
+    Ok(custom::with_write(|r| r.insert(id, codec)))
+}
+
+/// Remove the codec registered for `id`. Returns whether one was registered.
+#[cfg(feature = "std")]
+pub fn unregister_filter(id: u16) -> bool {
+    custom::with_write(|r| r.remove(&id).is_some())
+}
+
+/// The codec registered for `id`, if any.
+#[cfg(feature = "std")]
+pub fn registered(id: u16) -> Option<std::sync::Arc<dyn FilterCodec>> {
+    custom::with_read(|r| r.get(&id).cloned())
+}
+
+/// Undo filter `ctx.filter` on `input`: the built-in decoder if there is one
+/// that claims the chunk, else a registered one, else the built-in decoder's
+/// own refusal or [`FormatError::UnsupportedFilter`].
+pub(crate) fn decode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+    let id = ctx.filter.filter_id;
+    let builtin = builtin_filter(id);
+    if let Some(builtin) = builtin.filter(|b| b.claims(ctx.filter)) {
+        return (builtin.decode)(input, ctx);
+    }
+    #[cfg(feature = "std")]
+    if let Some(codec) = registered(id) {
+        let out = codec.decode(input, ctx)?;
+        // A registered codec is outside our control: hold it to the same
+        // bound the built-in decoders enforce.
+        if out.len() > ctx.output_limit() {
+            return Err(FormatError::DecompressionError(format!(
+                "filter {id}: decoded {} bytes, more than the {} the chunk can hold",
+                out.len(),
+                ctx.output_limit()
+            )));
+        }
+        return Ok(out);
+    }
+    match builtin {
+        Some(builtin) => (builtin.decode)(input, ctx),
+        None => Err(FormatError::UnsupportedFilter(id)),
+    }
+}
+
+/// Apply filter `ctx.filter` to `input`.
+pub(crate) fn encode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+    let id = ctx.filter.filter_id;
+    #[cfg(feature = "std")]
+    if builtin_filter(id).is_some_and(|b| b.is_shared())
+        && let Some(codec) = registered(id)
+    {
+        return codec.encode(input, ctx);
+    }
+    if let Some(builtin) = builtin_filter(id) {
+        return match builtin.encode {
+            Some(encode) => encode(input, ctx),
+            None => Err(FormatError::UnsupportedFilter(id)),
+        };
+    }
+    #[cfg(feature = "std")]
+    if let Some(codec) = registered(id) {
+        return codec.encode(input, ctx);
+    }
+    Err(FormatError::UnsupportedFilter(id))
+}
+
+#[cfg(all(test, feature = "std"))]
+pub(crate) mod tests {
+    use super::*;
+    use crate::filter_pipeline::{FILTER_FLETCHER32, FILTER_SHUFFLE, FilterPipeline};
+    use crate::filters::{compress_chunk, decompress_chunk};
+
+    fn pipeline(id: u16) -> FilterPipeline {
+        FilterPipeline {
+            version: 2,
+            filters: vec![FilterDescription {
+                filter_id: id,
+                name: Some("test".into()),
+                flags: 0,
+                client_data: vec![7],
+            }],
+        }
+    }
+
+    struct Xor;
+    impl FilterCodec for Xor {
+        fn decode(&self, input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+            let k = ctx.client_data()[0] as u8;
+            Ok(input.iter().map(|b| b ^ k).collect())
+        }
+        fn encode(&self, input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+            self.decode(input, ctx)
+        }
+    }
+
+    // Each test uses its own ID: the registry is process-wide and tests run
+    // in parallel.
+
+    #[test]
+    fn unknown_filter_keeps_its_error() {
+        let err = decompress_chunk(b"abc", &pipeline(311), 3, 1).unwrap_err();
+        assert_eq!(err, FormatError::UnsupportedFilter(311));
+        let err = compress_chunk(b"abc", &pipeline(311), 1).unwrap_err();
+        assert_eq!(err, FormatError::UnsupportedFilter(311));
+    }
+
+    #[test]
+    fn registered_codec_round_trips_through_the_pipeline() {
+        assert!(!is_filter_available(312));
+        assert!(register_filter(312, Xor).unwrap().is_none());
+        assert!(is_filter_available(312));
+        let data = b"hello, registry".to_vec();
+        let enc = compress_chunk(&data, &pipeline(312), 1).unwrap();
+        assert_ne!(enc, data);
+        assert_eq!(
+            decompress_chunk(&enc, &pipeline(312), data.len(), 1).unwrap(),
+            data
+        );
+        assert!(unregister_filter(312));
+        assert!(!unregister_filter(312));
+        assert_eq!(
+            decompress_chunk(&enc, &pipeline(312), data.len(), 1).unwrap_err(),
+            FormatError::UnsupportedFilter(312)
+        );
+    }
+
+    #[test]
+    fn closure_registers_a_decoder_only() {
+        register_filter(313, |input: &[u8], _ctx: &FilterContext<'_>| {
+            Ok(input.iter().rev().copied().collect())
+        })
+        .unwrap();
+        assert_eq!(
+            decompress_chunk(b"abc", &pipeline(313), 3, 1).unwrap(),
+            b"cba"
+        );
+        assert_eq!(
+            compress_chunk(b"abc", &pipeline(313), 1).unwrap_err(),
+            FormatError::UnsupportedFilter(313)
+        );
+        unregister_filter(313);
+    }
+
+    #[test]
+    fn registered_decoder_output_is_bounded() {
+        register_filter(314, |_input: &[u8], _ctx: &FilterContext<'_>| {
+            Ok(vec![0u8; 1000])
+        })
+        .unwrap();
+        let err = decompress_chunk(b"abc", &pipeline(314), 10, 1).unwrap_err();
+        assert!(matches!(err, FormatError::DecompressionError(_)), "{err:?}");
+        unregister_filter(314);
+    }
+
+    #[test]
+    fn builtins_cannot_be_overridden() {
+        for id in [FILTER_SHUFFLE, FILTER_FLETCHER32] {
+            let Err(err) = register_filter(id, Xor) else {
+                panic!("built-in filter {id} was re-registered");
+            };
+            assert!(matches!(err, FormatError::FilterError(_)), "{err:?}");
+        }
+        assert!(builtin_filter(FILTER_SHUFFLE).is_some());
+    }
+
+    /// Serialises the tests that register or read filter 32023 (the
+    /// registry is process-wide).
+    pub(crate) static ID_32023: std::sync::Mutex<()> = std::sync::Mutex::new(());
+
+    /// 32023 is Granular BitRound's ID; the `pcodec` build's built-in entry
+    /// there reads only clawhdf5 <= 2.7.0's pcodec chunks (named "pcodec"),
+    /// so a codec can be registered for the rest, and writes with it.
+    #[test]
+    fn a_codec_can_be_registered_for_granular_bitround() {
+        let _guard = ID_32023
+            .lock()
+            .unwrap_or_else(std::sync::PoisonError::into_inner);
+        let named = |name: Option<&str>| FilterPipeline {
+            version: 2,
+            filters: vec![FilterDescription {
+                filter_id: 32023,
+                name: name.map(Into::into),
+                flags: 0,
+                client_data: vec![7],
+            }],
+        };
+        let prev = register_filter(32023, Xor).expect("32023 must be registrable");
+        assert!(prev.is_none());
+        let data = b"granular bitround".to_vec();
+        for name in [None, Some("granular_bitround"), Some("test")] {
+            let pl = named(name);
+            let enc = compress_chunk(&data, &pl, 1).unwrap();
+            assert_ne!(enc, data);
+            assert_eq!(decompress_chunk(&enc, &pl, data.len(), 1).unwrap(), data);
+        }
+        // clawhdf5 <= 2.7.0's pcodec chunks still go to the built-in reader.
+        #[cfg(feature = "pcodec")]
+        {
+            let raw: Vec<u8> = (0..64)
+                .flat_map(|i| (f64::from(i) * 0.5).to_le_bytes())
+                .collect();
+            let comp = crate::filters::pcodec_compress(&raw, 8).unwrap();
+            let mut pl = named(Some("pcodec"));
+            pl.filters[0].client_data = vec![8];
+            assert_eq!(decompress_chunk(&comp, &pl, raw.len(), 8).unwrap(), raw);
+        }
+        assert!(unregister_filter(32023));
+        let pl = named(None);
+        assert!(matches!(
+            decompress_chunk(&data, &pl, data.len(), 1),
+            Err(FormatError::UnsupportedFilter(32023))
+        ));
+    }
+
+    #[test]
+    fn unsupported_filter_error_names_the_filter() {
+        let msg = FormatError::UnsupportedFilter(32026).to_string();
+        assert!(msg.contains("Blosc2") && msg.contains("`blosc2`"), "{msg}");
+        let msg = FormatError::UnsupportedFilter(32013).to_string();
+        assert!(msg.contains("ZFP") && msg.contains("`zfp`"), "{msg}");
+        let msg = FormatError::UnsupportedFilter(32019).to_string();
+        assert!(
+            msg.contains("JPEG") && msg.contains("not implemented"),
+            "{msg}"
+        );
+        let msg = FormatError::UnsupportedFilter(32000).to_string();
+        assert!(msg.contains("LZF") && msg.contains("`lzf`"), "{msg}");
+        assert_eq!(
+            FormatError::UnsupportedFilter(399).to_string(),
+            "unsupported filter: 399"
+        );
+    }
+
+    #[test]
+    fn builtin_table_is_sorted_and_unique() {
+        let ids: Vec<u16> = builtin_filters().iter().map(|f| f.id).collect();
+        let mut sorted = ids.clone();
+        sorted.sort_unstable();
+        sorted.dedup();
+        assert_eq!(ids, sorted);
+    }
+}

crates/clawhdf5-format/src/filters_bitshuffle.rs

@@ -0,0 +1,446 @@
+//! Bitshuffle (HDF5 filter 32008) and the bit transpose it shares with blosc.
+//!
+//! **The transform.** A block of `n` elements (`n` a multiple of 8) of
+//! `es` bytes each is viewed as an `n × 8·es` bit matrix — row *i* is
+//! element *i*, column `8·j + k` is bit *k* (LSB first) of its byte *j* — and
+//! transposed: the output is `8·es` rows of `n` bits, row `8·j + k` holding
+//! bit *k* of byte *j* of every element in order, packed LSB first. That is
+//! what `bshuf_trans_bit_elem` produces (checked against hdf5plugin's
+//! library bit for bit).
+//!
+//! **The filter** (`bshuf_h5filter.c`). `cd_values`: `[0..2]` bitshuffle
+//! version, `[2]` element size, `[3]` block size in elements (0 = default:
+//! 8192 bytes' worth, rounded down to a multiple of 8, at least 128),
+//! `[4]` compression (0 none, 2 LZ4, 3 Zstandard), `[5]` Zstandard level.
+//! The chunk is cut into blocks of `block size` elements; the tail shorter
+//! than a block is transposed as one block rounded down to a multiple of 8
+//! elements, and the last `n mod 8` elements are stored as they are.
+//! Uncompressed, that is the whole chunk. Compressed, the chunk starts with a
+//! 12-byte header — the decoded size (u64 big-endian) and the block size in
+//! bytes (u32 big-endian) — and each transposed block is stored as a u32
+//! big-endian length and an LZ4 block / Zstandard frame; the untransposed
+//! tail follows the last block.
+
+#[cfg(not(feature = "std"))]
+extern crate alloc;
+#[cfg(not(feature = "std"))]
+use alloc::{format, vec, vec::Vec};
+
+use crate::error::FormatError;
+#[cfg(feature = "bitshuffle")]
+use crate::filter_registry::FilterContext;
+
+/// Transpose an 8×8 bit matrix packed in a u64 (byte *r* = row *r*, bit *c*
+/// of that byte = column *c*). An involution.
+#[inline]
+fn transpose8(mut x: u64) -> u64 {
+    let t = (x ^ (x >> 7)) & 0x00AA_00AA_00AA_00AA;
+    x = x ^ t ^ (t << 7);
+    let t = (x ^ (x >> 14)) & 0x0000_CCCC_0000_CCCC;
+    x = x ^ t ^ (t << 14);
+    let t = (x ^ (x >> 28)) & 0x0000_0000_F0F0_F0F0;
+    x ^ t ^ (t << 28)
+}
+
+/// Bit-transpose one block: `input` and `out` are `n * es` bytes, `n` a
+/// multiple of 8.
+pub(crate) fn bitshuffle_block(input: &[u8], out: &mut [u8], n: usize, es: usize) {
+    debug_assert!(n.is_multiple_of(8) && input.len() == n * es && out.len() == n * es);
+    let row = n / 8;
+    for j in 0..es {
+        for g in 0..row {
+            let mut x = 0u64;
+            for t in 0..8 {
+                x |= u64::from(input[(8 * g + t) * es + j]) << (8 * t);
+            }
+            let y = transpose8(x);
+            for k in 0..8 {
+                out[(8 * j + k) * row + g] = (y >> (8 * k)) as u8;
+            }
+        }
+    }
+}
+
+/// Undo [`bitshuffle_block`].
+pub(crate) fn bitunshuffle_block(input: &[u8], out: &mut [u8], n: usize, es: usize) {
+    debug_assert!(n.is_multiple_of(8) && input.len() == n * es && out.len() == n * es);
+    let row = n / 8;
+    for j in 0..es {
+        for g in 0..row {
+            let mut y = 0u64;
+            for k in 0..8 {
+                y |= u64::from(input[(8 * j + k) * row + g]) << (8 * k);
+            }
+            let x = transpose8(y);
+            for t in 0..8 {
+                out[(8 * g + t) * es + j] = (x >> (8 * t)) as u8;
+            }
+        }
+    }
+}
+
+/// `bshuf_default_block_size`: 8 KiB of elements, a multiple of 8, >= 128.
+#[cfg(feature = "bitshuffle")]
+fn default_block_size(es: usize) -> usize {
+    ((8192 / es) / 8 * 8).max(128)
+}
+
+#[cfg(feature = "bitshuffle")]
+fn err(msg: &str) -> FormatError {
+    FormatError::DecompressionError(format!("bitshuffle: {msg}"))
+}
+
+/// `cd_values[4]`: the compression bitshuffle applies after the transpose.
+#[cfg(feature = "bitshuffle")]
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum Codec {
+    None,
+    Lz4,
+    Zstd,
+}
+
+#[cfg(feature = "bitshuffle")]
+fn codec(cd: &[u32]) -> Result<Codec, FormatError> {
+    match cd.get(4).copied().unwrap_or(0) {
+        0 => Ok(Codec::None),
+        2 => Ok(Codec::Lz4),
+        3 => Ok(Codec::Zstd),
+        other => Err(FormatError::FilterError(format!(
+            "bitshuffle: unknown compression {other}"
+        ))),
+    }
+}
+
+/// The element counts of the transposed blocks for `size` elements.
+#[cfg(feature = "bitshuffle")]
+fn blocks(size: usize, block: usize) -> impl Iterator<Item = usize> {
+    let full = size / block;
+    let last = (size % block) / 8 * 8;
+    core::iter::repeat_n(block, full).chain((last > 0).then_some(last))
+}
+
+/// Decode a bitshuffle-filtered chunk.
+#[cfg(feature = "bitshuffle")]
+pub(crate) fn bitshuffle_decode(
+    input: &[u8],
+    ctx: &FilterContext<'_>,
+) -> Result<Vec<u8>, FormatError> {
+    let cd = ctx.client_data();
+    let es = match cd.get(2) {
+        Some(&e) if e != 0 => e as usize,
+        _ => return Err(err("missing element size")),
+    };
+    let codec = codec(cd)?;
+    let limit = ctx.output_limit();
+    if codec == Codec::None {
+        if input.len() > limit {
+            return Err(err("output exceeds the chunk size"));
+        }
+        let block = match cd.get(3) {
+            Some(&b) if b != 0 => b as usize,
+            _ => default_block_size(es),
+        };
+        if !block.is_multiple_of(8) {
+            return Err(err("block size is not a multiple of 8"));
+        }
+        if !input.len().is_multiple_of(es) {
+            return Err(err("chunk is not a whole number of elements"));
+        }
+        let size = input.len() / es;
+        let mut out = vec![0u8; input.len()];
+        let mut pos = 0;
+        for n in blocks(size, block) {
+            let bytes = n * es;
+            bitunshuffle_block(&input[pos..pos + bytes], &mut out[pos..pos + bytes], n, es);
+            pos += bytes;
+        }
+        out[pos..].copy_from_slice(&input[pos..]);
+        return Ok(out);
+    }
+
+    let header = input.get(..12).ok_or_else(|| err("truncated header"))?;
+    let total = u64::from_be_bytes(header[..8].try_into().unwrap());
+    let block_bytes = u32::from_be_bytes(header[8..12].try_into().unwrap()) as usize;
+    let total = usize::try_from(total)
+        .ok()
+        .filter(|&t| t <= limit)
+        .ok_or_else(|| err("decoded size exceeds the chunk size"))?;
+    if !total.is_multiple_of(es) {
+        return Err(err("chunk is not a whole number of elements"));
+    }
+    if block_bytes == 0 || !block_bytes.is_multiple_of(es) {
+        return Err(err("bad block size"));
+    }
+    let block = block_bytes / es;
+    if !block.is_multiple_of(8) {
+        return Err(err("block size is not a multiple of 8"));
+    }
+    let size = total / es;
+    let mut out = vec![0u8; total];
+    let mut tmp = vec![0u8; block_bytes.min(total)];
+    let mut ip = 12usize;
+    let mut op = 0usize;
+    let mut zstd = None;
+    for n in blocks(size, block) {
+        let bytes = n * es;
+        let len = input
+            .get(ip..ip + 4)
+            .map(|b| u32::from_be_bytes(b.try_into().unwrap()) as usize)
+            .ok_or_else(|| err("truncated block header"))?;
+        ip += 4;
+        let comp = input
+            .get(ip..ip.saturating_add(len))
+            .ok_or_else(|| err("truncated block"))?;
+        ip += len;
+        let dst = &mut tmp[..bytes];
+        let got = match codec {
+            Codec::Lz4 => lz4_flex::block::decompress_into(comp, dst)
+                .map_err(|e| err(&format!("lz4: {e}")))?,
+            Codec::Zstd => zstd_decode_into(
+                zstd.get_or_insert_with(ruzstd::decoding::FrameDecoder::new),
+                comp,
+                dst,
+            )?,
+            Codec::None => unreachable!(),
+        };
+        if got != bytes {
+            return Err(err("block decoded to the wrong size"));
+        }
+        bitunshuffle_block(dst, &mut out[op..op + bytes], n, es);
+        op += bytes;
+    }
+    let tail = total - op;
+    let rest = input
+        .get(ip..ip + tail)
+        .ok_or_else(|| err("truncated trailing elements"))?;
+    out[op..].copy_from_slice(rest);
+    Ok(out)
+}
+
+/// Decode Zstandard frames into exactly `dst`, failing if they hold more.
+///
+/// ruzstd reserves a frame's declared window (by default up to 100 MiB)
+/// before decoding it, so the window is capped at what the output could
+/// need: twice `dst` (window sizes are rounded up), and at least 128 KiB.
+/// The encoders behind these filters (c-blosc, c-blosc2, bitshuffle)
+/// compress each block in one call with its size known, so libzstd's
+/// window never exceeds the block.
+#[cfg(any(feature = "bitshuffle", feature = "blosc"))]
+pub(crate) fn zstd_decode_into(
+    decoder: &mut ruzstd::decoding::FrameDecoder,
+    frames: &[u8],
+    dst: &mut [u8],
+) -> Result<usize, FormatError> {
+    decoder.set_max_window_size((2 * dst.len()).max(1 << 17) as u64);
+    decoder
+        .decode_all(frames, dst)
+        .map_err(|e| FormatError::DecompressionError(format!("zstd: {e}")))
+}
+
+/// Compress with ruzstd. It implements one level (roughly zstd's level 1),
+/// so the requested level only matters to other encoders.
+#[cfg(any(feature = "bitshuffle", feature = "blosc"))]
+pub(crate) fn zstd_encode(data: &[u8]) -> Vec<u8> {
+    ruzstd::encoding::compress_to_vec(data, ruzstd::encoding::CompressionLevel::Fastest)
+}
+
+/// Encode a chunk with the bitshuffle filter.
+#[cfg(feature = "bitshuffle")]
+pub(crate) fn bitshuffle_encode(
+    input: &[u8],
+    ctx: &FilterContext<'_>,
+) -> Result<Vec<u8>, FormatError> {
+    let cd = ctx.client_data();
+    let es = match cd.get(2) {
+        Some(&e) if e != 0 => e as usize,
+        _ => ctx.element_size.max(1),
+    };
+    let codec = codec(cd)?;
+    let block = match cd.get(3) {
+        Some(&b) if b != 0 => b as usize,
+        _ => default_block_size(es),
+    };
+    let cerr = |m: &str| FormatError::CompressionError(format!("bitshuffle: {m}"));
+    if !block.is_multiple_of(8) {
+        return Err(cerr("block size is not a multiple of 8"));
+    }
+    if !input.len().is_multiple_of(es) {
+        return Err(cerr("chunk is not a whole number of elements"));
+    }
+    let size = input.len() / es;
+    let mut out = Vec::with_capacity(input.len() + 12 + input.len() / 64);
+    if codec != Codec::None {
+        out.extend_from_slice(&(input.len() as u64).to_be_bytes());
+        let block_bytes =
+            u32::try_from(block * es).map_err(|_| cerr("block size does not fit in 32 bits"))?;
+        out.extend_from_slice(&block_bytes.to_be_bytes());
+    }
+    let mut tmp = vec![0u8; (block * es).min(input.len())];
+    let mut pos = 0;
+    for n in blocks(size, block) {
+        let bytes = n * es;
+        let dst = &mut tmp[..bytes];
+        bitshuffle_block(&input[pos..pos + bytes], dst, n, es);
+        match codec {
+            Codec::None => out.extend_from_slice(dst),
+            Codec::Lz4 | Codec::Zstd => {
+                let comp = if codec == Codec::Lz4 {
+                    lz4_flex::block::compress(dst)
+                } else {
+                    zstd_encode(dst)
+                };
+                out.extend_from_slice(&(comp.len() as u32).to_be_bytes());
+                out.extend_from_slice(&comp);
+            }
+        }
+        pos += bytes;
+    }
+    out.extend_from_slice(&input[pos..]);
+    Ok(out)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The definition, one bit at a time.
+    fn naive(input: &[u8], n: usize, es: usize) -> Vec<u8> {
+        let mut out = vec![0u8; n * es];
+        for i in 0..n {
+            for j in 0..es {
+                for k in 0..8 {
+                    if input[i * es + j] >> k & 1 == 1 {
+                        let p = (8 * j + k) * n + i;
+                        out[p / 8] |= 1 << (p % 8);
+                    }
+                }
+            }
+        }
+        out
+    }
+
+    #[test]
+    fn transpose_matches_the_definition_and_inverts() {
+        for (n, es) in [(8, 1), (16, 2), (24, 4), (128, 8), (64, 3), (8, 16)] {
+            let input: Vec<u8> = (0..n * es)
+                .map(|i| (i as u32).wrapping_mul(2_654_435_761).rotate_left(7) as u8)
+                .collect();
+            let mut out = vec![0u8; n * es];
+            bitshuffle_block(&input, &mut out, n, es);
+            assert_eq!(out, naive(&input, n, es), "n={n} es={es}");
+            let mut back = vec![0u8; n * es];
+            bitunshuffle_block(&out, &mut back, n, es);
+            assert_eq!(back, input);
+        }
+    }
+
+    #[cfg(feature = "bitshuffle")]
+    fn ctx_for(cd: Vec<u32>) -> crate::filter_pipeline::FilterDescription {
+        crate::filter_pipeline::FilterDescription {
+            filter_id: crate::filter_pipeline::FILTER_BITSHUFFLE,
+            name: None,
+            flags: 0,
+            client_data: cd,
+        }
+    }
+
+    #[cfg(feature = "bitshuffle")]
+    #[test]
+    fn filter_round_trips_every_mode() {
+        for es in [1usize, 2, 4, 8] {
+            for n in [0usize, 1, 7, 8, 100, 1000, 5003] {
+                let data: Vec<u8> = (0..n * es)
+                    .map(|i| (i % 97) as u8 ^ (i / 300) as u8)
+                    .collect();
+                for (comp, block) in [(0, 0), (0, 16), (2, 0), (2, 64), (3, 0), (3, 1024)] {
+                    let f = ctx_for(vec![0, 4, es as u32, block, comp]);
+                    let ctx = FilterContext {
+                        filter: &f,
+                        element_size: es,
+                        max_output: data.len(),
+                    };
+                    let enc = bitshuffle_encode(&data, &ctx).unwrap();
+                    let dec = bitshuffle_decode(&enc, &ctx).unwrap();
+                    assert_eq!(dec, data, "es={es} n={n} comp={comp} block={block}");
+                }
+            }
+        }
+    }
+
+    #[cfg(feature = "bitshuffle")]
+    #[test]
+    fn rejects_oversized_and_truncated_chunks() {
+        let data = vec![5u8; 4096];
+        let f = ctx_for(vec![0, 4, 4, 0, 2]);
+        let mut ctx = FilterContext {
+            filter: &f,
+            element_size: 4,
+            max_output: data.len(),
+        };
+        let enc = bitshuffle_encode(&data, &ctx).unwrap();
+        assert!(bitshuffle_decode(&enc[..enc.len() - 1], &ctx).is_err());
+        ctx.max_output = 100;
+        assert!(bitshuffle_decode(&enc, &ctx).is_err());
+    }
+
+    /// Random and mutated chunks, in every mode, and hostile `cd_values`:
+    /// errors are fine, panics are not.
+    #[cfg(feature = "bitshuffle")]
+    #[test]
+    fn fuzzed_chunks_never_panic() {
+        use crate::test_fuzz::{Rng, fuzz_decoder};
+        let data: Vec<u8> = (0..3001u32)
+            .flat_map(|i| ((i / 7) as u16).to_le_bytes())
+            .collect();
+        for (comp, block) in [(0, 0), (0, 16), (2, 0), (2, 64), (3, 0), (3, 1024)] {
+            let f = ctx_for(vec![0, 4, 2, block, comp]);
+            let ctx = FilterContext {
+                filter: &f,
+                element_size: 2,
+                max_output: data.len(),
+            };
+            let seeds = vec![
+                bitshuffle_encode(&data, &ctx).unwrap(),
+                bitshuffle_encode(&data[..34], &ctx).unwrap(),
+                bitshuffle_encode(&data[..512], &ctx).unwrap(),
+            ];
+            fuzz_decoder(
+                0xb5 + comp as u64 * 7 + block as u64,
+                &seeds,
+                4_000,
+                data.len(),
+                |s| bitshuffle_decode(s, &ctx),
+            );
+        }
+        // Hostile filter parameters on a valid chunk.
+        let mut rng = Rng::new(0xcd);
+        let good = ctx_for(vec![0, 4, 2, 0, 2]);
+        let enc = bitshuffle_encode(
+            &data,
+            &FilterContext {
+                filter: &good,
+                element_size: 2,
+                max_output: data.len(),
+            },
+        )
+        .unwrap();
+        for _ in 0..3_000 {
+            let cd: Vec<u32> = (0..rng.below(7))
+                .map(|_| match rng.below(4) {
+                    0 => rng.below(5) as u32,
+                    1 => u32::MAX - rng.below(4) as u32,
+                    2 => 1 << rng.below(32),
+                    _ => rng.next_u64() as u32,
+                })
+                .collect();
+            let f = ctx_for(cd);
+            let ctx = FilterContext {
+                filter: &f,
+                element_size: 2,
+                max_output: data.len(),
+            };
+            let _ = bitshuffle_decode(&enc, &ctx);
+            let _ = bitshuffle_decode(&data, &ctx);
+        }
+    }
+}

crates/clawhdf5-format/src/filters_blosc.rs

@@ -0,0 +1,711 @@
+//! Blosc 1 (HDF5 filter 32001, `hdf5-blosc`, hdf5plugin's `Blosc`), in pure
+//! Rust: the Blosc 1 frame, its byte shuffle and bit shuffle, and the
+//! BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard codecs inside it.
+//!
+//! **Frame** (c-blosc 1.x, format version 2). A 16-byte header — version
+//! (2), codec format version (1), flags, type size, then little-endian `u32`
+//! decoded size, block size and frame size. Flags: bit 0 byte shuffle, bit
+//! 1 stored raw ("memcpyed": the data follows the header), bit 2 bit
+//! shuffle, bit 4 "do not split", bits 5-7 the codec (0 BloscLZ, 1 LZ4 and
+//! LZ4HC, 2 Snappy, 3 Zlib, 4 Zstandard). Unless stored raw, a table of
+//! `u32` block offsets follows, one per block of `block size` bytes (the
+//! last one may be shorter). A block is one stream, or — when the "do not
+//! split" flag is clear, the type size is at most 16, the block holds at
+//! least 128 elements, and it is not the short last block — `type size`
+//! streams, one per byte plane. Each stream is a `u32` length and the
+//! codec's output; a length equal to the stream's decoded size means the
+//! bytes are stored raw. The decoded block is then unshuffled (byte shuffle
+//! for type size > 1; bit shuffle when the block holds a multiple of 8
+//! elements, the trailing partial element copied as is).
+//!
+//! **Filter** (`blosc_filter.c`) `cd_values`: `[0]` filter revision, `[1]`
+//! Blosc format version, `[2]` type size, `[3]` chunk size in bytes, `[4]`
+//! compression level, `[5]` shuffle (0 none, 1 byte, 2 bit), `[6]`
+//! compressor (0 blosclz, 1 lz4, 2 lz4hc, 3 snappy, 4 zlib, 5 zstd). The
+//! decoder needs only the frame.
+
+use crate::error::FormatError;
+use crate::filter_registry::FilterContext;
+use crate::filters_bitshuffle::{bitshuffle_block, bitunshuffle_block};
+
+const HEADER: usize = 16;
+const FLAG_SHUFFLE: u8 = 0x01;
+const FLAG_MEMCPYED: u8 = 0x02;
+const FLAG_BITSHUFFLE: u8 = 0x04;
+const FLAG_FUTURE: u8 = 0x08;
+const FLAG_DONT_SPLIT: u8 = 0x10;
+const MAX_SPLITS: usize = 16;
+const MIN_BUFFERSIZE: usize = 128;
+
+fn err(msg: &str) -> FormatError {
+    FormatError::DecompressionError(format!("blosc: {msg}"))
+}
+
+fn le32(b: &[u8], at: usize) -> Result<usize, FormatError> {
+    b.get(at..at + 4)
+        .map(|s| u32::from_le_bytes(s.try_into().unwrap()) as usize)
+        .ok_or_else(|| err("truncated frame"))
+}
+
+/// The codec inside a Blosc frame (flags bits 5-7).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum Codec {
+    BloscLz,
+    Lz4,
+    Snappy,
+    Zlib,
+    Zstd,
+}
+
+impl Codec {
+    pub(crate) fn from_flags(flags: u8) -> Result<Codec, FormatError> {
+        match flags >> 5 {
+            0 => Ok(Codec::BloscLz),
+            1 => Ok(Codec::Lz4),
+            2 => Ok(Codec::Snappy),
+            3 => Ok(Codec::Zlib),
+            4 => Ok(Codec::Zstd),
+            other => Err(err(&format!("unknown codec {other}"))),
+        }
+    }
+}
+
+/// Decode one codec stream into exactly `dst`.
+pub(crate) fn decode_stream(
+    codec: Codec,
+    src: &[u8],
+    dst: &mut [u8],
+    zstd: &mut Option<ruzstd::decoding::FrameDecoder>,
+) -> Result<(), FormatError> {
+    let n = match codec {
+        Codec::BloscLz => blosclz_decompress(src, dst),
+        Codec::Lz4 => {
+            lz4_flex::block::decompress_into(src, dst).map_err(|e| err(&format!("lz4: {e}")))?
+        }
+        Codec::Snappy => {
+            let len = snap::raw::decompress_len(src).map_err(|e| err(&format!("snappy: {e}")))?;
+            if len != dst.len() {
+                return Err(err("snappy stream has the wrong size"));
+            }
+            snap::raw::Decoder::new()
+                .decompress(src, dst)
+                .map_err(|e| err(&format!("snappy: {e}")))?
+        }
+        Codec::Zlib => {
+            let out = crate::filters::inflate_bounded(src, dst.len(), dst.len())
+                .map_err(|e| err(&format!("zlib: {e}")))?;
+            let n = out.len();
+            if n == dst.len() {
+                dst.copy_from_slice(&out);
+            }
+            n
+        }
+        Codec::Zstd => crate::filters_bitshuffle::zstd_decode_into(
+            zstd.get_or_insert_with(ruzstd::decoding::FrameDecoder::new),
+            src,
+            dst,
+        )?,
+    };
+    if n != dst.len() {
+        return Err(err("stream decoded to the wrong size"));
+    }
+    Ok(())
+}
+
+/// Decode a Blosc-filtered chunk: one Blosc 1 frame.
+///
+/// An HDF5 chunk is never empty, so a frame that decodes to nothing where
+/// the chunk size is known is corrupt (libhdf5's filter fails it too).
+pub(crate) fn blosc_decode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+    let out = blosc_decompress(input, ctx.output_limit())?;
+    if out.is_empty() && ctx.max_output != 0 {
+        return Err(err("empty frame for a non-empty chunk"));
+    }
+    Ok(out)
+}
+
+/// Decompress a Blosc 1 frame, refusing more than `limit` bytes of output.
+pub fn blosc_decompress(input: &[u8], limit: usize) -> Result<Vec<u8>, FormatError> {
+    if input.len() < HEADER {
+        return Err(err("truncated header"));
+    }
+    let version = input[0];
+    let codec_version = input[1];
+    let flags = input[2];
+    let typesize = input[3] as usize;
+    let nbytes = le32(input, 4)?;
+    let blocksize = le32(input, 8)?;
+    let cbytes = le32(input, 12)?;
+    if version != 1 && version != 2 {
+        return Err(err(&format!(
+            "frame format version {version} is not Blosc 1 (a Blosc 2 chunk?)"
+        )));
+    }
+    if flags & FLAG_FUTURE != 0 {
+        return Err(err("unknown header flags"));
+    }
+    if nbytes > limit {
+        return Err(err("decoded size exceeds the chunk size"));
+    }
+    if cbytes > input.len() {
+        return Err(err("frame is longer than the chunk"));
+    }
+    if cbytes < HEADER {
+        return Err(err("truncated frame"));
+    }
+    let src = &input[..cbytes];
+    if nbytes == 0 {
+        return Ok(Vec::new());
+    }
+    if blocksize == 0 || typesize == 0 {
+        return Err(err("bad block or type size"));
+    }
+    let mut out = vec![0u8; nbytes];
+    if flags & FLAG_MEMCPYED != 0 {
+        if cbytes != nbytes + HEADER {
+            return Err(err("stored frame has the wrong size"));
+        }
+        out.copy_from_slice(&src[HEADER..]);
+        return Ok(out);
+    }
+    let codec = Codec::from_flags(flags)?;
+    if codec_version != 1 {
+        return Err(err(&format!(
+            "unsupported {codec:?} format version {codec_version}"
+        )));
+    }
+    let nblocks = nbytes.div_ceil(blocksize);
+    let leftover = nbytes % blocksize;
+    if nblocks > (cbytes - HEADER) / 4 {
+        return Err(err("block table is truncated"));
+    }
+    let block_len = blocksize.min(nbytes);
+    let mut tmp = vec![0u8; block_len];
+    let mut zstd = None;
+    let dont_split = flags & FLAG_DONT_SPLIT != 0;
+    for j in 0..nblocks {
+        let is_leftover = j == nblocks - 1 && leftover > 0;
+        let bsize = if is_leftover { leftover } else { blocksize };
+        let nsplits = if !dont_split
+            && typesize <= MAX_SPLITS
+            && bsize / typesize >= MIN_BUFFERSIZE
+            && !is_leftover
+        {
+            typesize
+        } else {
+            1
+        };
+        let neblock = bsize / nsplits;
+        let mut pos = le32(src, HEADER + 4 * j)?;
+        let tmp = &mut tmp[..bsize];
+        for s in 0..nsplits {
+            let clen = src
+                .get(pos..)
+                .and_then(|rest| rest.get(..4))
+                .map(|b| u32::from_le_bytes(b.try_into().unwrap()) as usize)
+                .ok_or_else(|| err("block offset out of range"))?;
+            pos += 4;
+            let stream = src
+                .get(pos..pos.saturating_add(clen))
+                .ok_or_else(|| err("stream runs past the frame"))?;
+            let dst = &mut tmp[s * neblock..(s + 1) * neblock];
+            if clen == neblock {
+                dst.copy_from_slice(stream);
+            } else {
+                decode_stream(codec, stream, dst, &mut zstd)?;
+            }
+            pos += clen;
+        }
+        // `bsize` is a whole number of splits by construction (`nsplits` > 1
+        // only for full blocks, and c-blosc sizes those in whole elements).
+        if nsplits * neblock != bsize {
+            return Err(err("block is not a whole number of streams"));
+        }
+        let dest = &mut out[j * blocksize..j * blocksize + bsize];
+        unshuffle_block(flags, typesize, tmp, dest);
+    }
+    Ok(out)
+}
+
+/// Undo the frame's shuffle on one decoded block.
+fn unshuffle_block(flags: u8, typesize: usize, src: &[u8], dest: &mut [u8]) {
+    let bsize = src.len();
+    if flags & FLAG_SHUFFLE != 0 && typesize > 1 {
+        let n = bsize / typesize;
+        for i in 0..n {
+            for b in 0..typesize {
+                dest[i * typesize + b] = src[b * n + i];
+            }
+        }
+        dest[n * typesize..].copy_from_slice(&src[n * typesize..]);
+    } else if flags & FLAG_BITSHUFFLE != 0 && bsize >= typesize {
+        let n = bsize / typesize;
+        if n.is_multiple_of(8) {
+            let body = n * typesize;
+            bitunshuffle_block(&src[..body], &mut dest[..body], n, typesize);
+            dest[body..].copy_from_slice(&src[body..]);
+        } else {
+            dest.copy_from_slice(src);
+        }
+    } else {
+        dest.copy_from_slice(src);
+    }
+}
+
+/// BloscLZ decompression (c-blosc 1.21 `blosclz_decompress`): returns the
+/// number of bytes written, or 0 on malformed input — exactly as the C
+/// decoder, including stopping before a match that ends the stream, so a
+/// stream libblosc rejects is rejected here too.
+///
+/// Instructions: a control byte `ctrl`. Below 32, a literal run of
+/// `ctrl + 1` bytes. Otherwise a match: length `(ctrl >> 5) + 2`, extended
+/// by following bytes while they are 255 when the top three bits are all
+/// set; distance `((ctrl & 31) << 8) + next byte + 1`, or — when that byte
+/// is 255 and the high bits are 31 — a 16-bit big-endian distance plus 8192.
+/// The first instruction is always a literal.
+pub(crate) fn blosclz_decompress(input: &[u8], out: &mut [u8]) -> usize {
+    const MAX_DISTANCE: usize = 8191;
+    let limit = input.len();
+    if limit == 0 {
+        return 0;
+    }
+    let mut ip = 1usize;
+    let mut op = 0usize;
+    let mut ctrl = (input[0] & 31) as usize;
+    loop {
+        if ctrl >= 32 {
+            let mut len = (ctrl >> 5) - 1;
+            let ofs = (ctrl & 31) << 8;
+            if len == 6 {
+                loop {
+                    if ip + 1 >= limit {
+                        return 0;
+                    }
+                    let code = input[ip] as usize;
+                    ip += 1;
+                    len += code;
+                    if code != 255 {
+                        break;
+                    }
+                }
+            } else if ip + 1 >= limit {
+                return 0;
+            }
+            let code = input[ip] as usize;
+            ip += 1;
+            len += 3;
+            // The copy source is `distance` bytes back.
+            let mut distance = ofs + code + 1;
+            if code == 255 && ofs == 31 << 8 {
+                if ip + 1 >= limit {
+                    return 0;
+                }
+                let far = ((input[ip] as usize) << 8) + input[ip + 1] as usize;
+                ip += 2;
+                distance = far + MAX_DISTANCE + 1;
+            }
+            if op + len > out.len() {
+                return 0;
+            }
+            if distance > op {
+                return 0;
+            }
+            if ip >= limit {
+                break;
+            }
+            ctrl = input[ip] as usize;
+            ip += 1;
+            let start = op - distance;
+            if distance >= len {
+                out.copy_within(start..start + len, op);
+            } else {
+                for k in 0..len {
+                    out[op + k] = out[start + k];
+                }
+            }
+            op += len;
+        } else {
+            let run = ctrl + 1;
+            if op + run > out.len() || ip + run > limit {
+                return 0;
+            }
+            out[op..op + run].copy_from_slice(&input[ip..ip + run]);
+            op += run;
+            ip += run;
+            if ip >= limit {
+                break;
+            }
+            ctrl = input[ip] as usize;
+            ip += 1;
+        }
+    }
+    op
+}
+
+/// The codec our encoder puts inside the frame.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum EncodeCodec {
+    Lz4,
+    Snappy,
+    Zlib,
+    Zstd,
+}
+
+impl EncodeCodec {
+    /// From the filter's `cd_values[6]` compressor code.
+    fn from_cd(code: u32) -> Result<EncodeCodec, FormatError> {
+        match code {
+            1 | 2 => Ok(EncodeCodec::Lz4),
+            3 => Ok(EncodeCodec::Snappy),
+            4 => Ok(EncodeCodec::Zlib),
+            5 => Ok(EncodeCodec::Zstd),
+            0 => Err(FormatError::CompressionError(
+                "blosc: clawhdf5 cannot write BloscLZ; choose lz4, snappy, zlib or zstd".into(),
+            )),
+            other => Err(FormatError::CompressionError(format!(
+                "blosc: unknown compressor {other}"
+            ))),
+        }
+    }
+
+    fn flags(self) -> u8 {
+        (match self {
+            EncodeCodec::Lz4 => 1,
+            EncodeCodec::Snappy => 2,
+            EncodeCodec::Zlib => 3,
+            EncodeCodec::Zstd => 4,
+        }) << 5
+    }
+
+    fn encode(self, data: &[u8], level: u32) -> Result<Vec<u8>, FormatError> {
+        match self {
+            EncodeCodec::Lz4 => Ok(lz4_flex::block::compress(data)),
+            EncodeCodec::Snappy => snap::raw::Encoder::new()
+                .compress_vec(data)
+                .map_err(|e| FormatError::CompressionError(format!("blosc: snappy: {e}"))),
+            EncodeCodec::Zlib => crate::filters::deflate_bounded(data, level.min(9))
+                .map_err(|e| FormatError::CompressionError(format!("blosc: zlib: {e}"))),
+            EncodeCodec::Zstd => Ok(crate::filters_bitshuffle::zstd_encode(data)),
+        }
+    }
+}
+
+/// Block size our encoder uses: at most 256 KiB, a whole number of
+/// elements (and, for bit shuffle, of 8-element groups).
+fn encode_block_size(nbytes: usize, typesize: usize, bitshuffle: bool) -> usize {
+    let unit = if bitshuffle { 8 * typesize } else { typesize };
+    let target = (256 * 1024).min(nbytes);
+    if target < unit {
+        return nbytes.max(1);
+    }
+    target / unit * unit
+}
+
+/// Encode a chunk as one Blosc 1 frame. `cd_values` as hdf5-blosc:
+/// `[2]` type size, `[4]` level (0 = store), `[5]` shuffle, `[6]` codec.
+pub(crate) fn blosc_encode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+    let cd = ctx.client_data();
+    let cerr = |m: &str| FormatError::CompressionError(format!("blosc: {m}"));
+    let typesize = match cd.get(2) {
+        Some(&t) if t != 0 => t as usize,
+        _ => ctx.element_size.max(1),
+    };
+    // Blosc records the type size in one byte; c-blosc treats larger types
+    // as bytes.
+    let typesize = if typesize > 255 { 1 } else { typesize };
+    let level = cd.get(4).copied().unwrap_or(5);
+    let shuffle = cd.get(5).copied().unwrap_or(1);
+    let codec = EncodeCodec::from_cd(cd.get(6).copied().unwrap_or(1))?;
+    let nbytes = input.len();
+    if nbytes > i32::MAX as usize - HEADER {
+        return Err(cerr("chunk too large for a Blosc frame"));
+    }
+    let mut flags = codec.flags();
+    match shuffle {
+        0 => {}
+        1 => flags |= FLAG_SHUFFLE,
+        2 => flags |= FLAG_BITSHUFFLE,
+        other => return Err(cerr(&format!("unknown shuffle mode {other}"))),
+    }
+    let blocksize = encode_block_size(nbytes, typesize, shuffle == 2);
+    let header = |flags: u8, blocksize: usize, cbytes: usize| {
+        let mut h = Vec::with_capacity(HEADER);
+        h.extend_from_slice(&[2, 1, flags, typesize as u8]);
+        h.extend_from_slice(&(nbytes as u32).to_le_bytes());
+        h.extend_from_slice(&(blocksize as u32).to_le_bytes());
+        h.extend_from_slice(&(cbytes as u32).to_le_bytes());
+        h
+    };
+    let stored = || {
+        let mut out = header(
+            FLAG_MEMCPYED | (flags & !(FLAG_SHUFFLE | FLAG_BITSHUFFLE)),
+            blocksize,
+            nbytes + HEADER,
+        );
+        out.extend_from_slice(input);
+        out
+    };
+    if level == 0 || nbytes == 0 {
+        return Ok(stored());
+    }
+
+    let nblocks = nbytes.div_ceil(blocksize);
+    let leftover = nbytes % blocksize;
+    let mut body = Vec::with_capacity(nbytes / 2);
+    let mut starts = Vec::with_capacity(nblocks);
+    let table_end = HEADER + 4 * nblocks;
+    let mut shuffled = vec![0u8; blocksize];
+    for j in 0..nblocks {
+        let is_leftover = j == nblocks - 1 && leftover > 0;
+        let bsize = if is_leftover { leftover } else { blocksize };
+        let block = &input[j * blocksize..j * blocksize + bsize];
+        let sh = &mut shuffled[..bsize];
+        shuffle_block(flags, typesize, block, sh);
+        starts.push(table_end + body.len());
+        let nsplits = if typesize <= MAX_SPLITS
+            && bsize / typesize >= MIN_BUFFERSIZE
+            && !is_leftover
+            && bsize.is_multiple_of(typesize)
+        {
+            typesize
+        } else {
+            1
+        };
+        let neblock = bsize / nsplits;
+        for s in 0..nsplits {
+            let part = &sh[s * neblock..(s + 1) * neblock];
+            let comp = codec.encode(part, level)?;
+            if comp.len() < neblock {
+                body.extend_from_slice(&(comp.len() as u32).to_le_bytes());
+                body.extend_from_slice(&comp);
+            } else {
+                body.extend_from_slice(&(neblock as u32).to_le_bytes());
+                body.extend_from_slice(part);
+            }
+        }
+        if table_end + body.len() >= nbytes + HEADER {
+            // Incompressible: store instead, as c-blosc does.
+            return Ok(stored());
+        }
+    }
+    // A split block must decode as split: the decoder infers splitting from
+    // the same rule, which requires a whole number of elements per block.
+    let cbytes = table_end + body.len();
+    let mut out = header(flags, blocksize, cbytes);
+    for s in starts {
+        out.extend_from_slice(&(s as u32).to_le_bytes());
+    }
+    out.extend_from_slice(&body);
+    Ok(out)
+}
+
+/// Apply the frame's shuffle to one block (the inverse of
+/// [`unshuffle_block`]).
+fn shuffle_block(flags: u8, typesize: usize, src: &[u8], dest: &mut [u8]) {
+    let bsize = src.len();
+    if flags & FLAG_SHUFFLE != 0 && typesize > 1 {
+        let n = bsize / typesize;
+        for i in 0..n {
+            for b in 0..typesize {
+                dest[b * n + i] = src[i * typesize + b];
+            }
+        }
+        dest[n * typesize..].copy_from_slice(&src[n * typesize..]);
+    } else if flags & FLAG_BITSHUFFLE != 0 && bsize >= typesize {
+        let n = bsize / typesize;
+        if n.is_multiple_of(8) {
+            let body = n * typesize;
+            bitshuffle_block(&src[..body], &mut dest[..body], n, typesize);
+            dest[body..].copy_from_slice(&src[body..]);
+        } else {
+            dest.copy_from_slice(src);
+        }
+    } else {
+        dest.copy_from_slice(src);
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::filter_pipeline::{FILTER_BLOSC, FilterDescription};
+
+    /// A blosclz stream: literal "abc", then a 9-byte match 3 back (a run
+    /// of "abc"), then literal "Z".
+    #[test]
+    fn blosclz_decodes_literals_and_overlapping_matches() {
+        // Match: length (ctrl >> 5) + 2 = 8, distance ofs + code + 1 = 3.
+        let stream = [2, b'a', b'b', b'c', (6 << 5), 2, 0, b'Z'];
+        let mut out = [0u8; 12];
+        assert_eq!(blosclz_decompress(&stream, &mut out), 12);
+        assert_eq!(&out, b"abcabcabcabZ");
+        // A stream cut inside a match is malformed.
+        let mut out = [0u8; 11];
+        assert_eq!(blosclz_decompress(&stream[..6], &mut out), 0);
+        // A match before the start of the output is malformed.
+        assert_eq!(blosclz_decompress(&[0, b'a', 32, 5, 0, b'x'], &mut out), 0);
+    }
+
+    fn desc(cd: Vec<u32>) -> FilterDescription {
+        FilterDescription {
+            filter_id: FILTER_BLOSC,
+            name: None,
+            flags: 0,
+            client_data: cd,
+        }
+    }
+
+    #[test]
+    fn frame_round_trips_every_codec_and_shuffle() {
+        for ts in [1usize, 2, 4, 8, 3, 32] {
+            for n in [0usize, 5, 100, 1000, 70_000, 300_001] {
+                if n * ts > 1 << 20 && ts > 1 {
+                    continue;
+                }
+                let data: Vec<u8> = (0..n * ts)
+                    .map(|i| ((i / ts) % 200) as u8 ^ (i % ts) as u8)
+                    .collect();
+                for codec in [1u32, 3, 4, 5] {
+                    for shuffle in [0u32, 1, 2] {
+                        for level in [0u32, 5] {
+                            let f = desc(vec![2, 2, ts as u32, 0, level, shuffle, codec]);
+                            let ctx = FilterContext {
+                                filter: &f,
+                                element_size: ts,
+                                max_output: data.len(),
+                            };
+                            let enc = blosc_encode(&data, &ctx).unwrap();
+                            let dec = blosc_decode(&enc, &ctx).unwrap_or_else(|e| {
+                                panic!("ts={ts} n={n} codec={codec} shuffle={shuffle}: {e}")
+                            });
+                            assert!(
+                                dec == data,
+                                "ts={ts} n={n} codec={codec} shuffle={shuffle} level={level}"
+                            );
+                        }
+                    }
+                }
+            }
+        }
+    }
+
+    #[test]
+    fn rejects_bad_frames() {
+        let data = vec![9u8; 50_000];
+        let f = desc(vec![2, 2, 4, 0, 5, 1, 1]);
+        let ctx = FilterContext {
+            filter: &f,
+            element_size: 4,
+            max_output: data.len(),
+        };
+        let enc = blosc_encode(&data, &ctx).unwrap();
+        assert!(blosc_decode(&enc[..enc.len() - 3], &ctx).is_err());
+        let small = FilterContext {
+            max_output: 49_999,
+            ..ctx
+        };
+        assert!(blosc_decode(&enc, &small).is_err());
+        let mut v3 = enc.clone();
+        v3[0] = 3;
+        assert!(blosc_decode(&v3, &ctx).is_err());
+        let f0 = desc(vec![2, 2, 4, 0, 5, 1, 0]);
+        let ctx0 = FilterContext { filter: &f0, ..ctx };
+        assert!(blosc_encode(&data, &ctx0).is_err());
+    }
+
+    /// A frame that declares no data, for a chunk that has some.
+    #[test]
+    fn empty_frame_for_a_non_empty_chunk_is_an_error() {
+        let mut frame = vec![2u8, 1, 0x20, 4];
+        for v in [0u32, 64, 16] {
+            frame.extend_from_slice(&v.to_le_bytes());
+        }
+        assert_eq!(blosc_decompress(&frame, 64).unwrap(), b"");
+        let f = desc(vec![2, 2, 4, 64, 5, 1, 1]);
+        let ctx = FilterContext {
+            filter: &f,
+            element_size: 4,
+            max_output: 64,
+        };
+        assert!(blosc_decode(&frame, &ctx).is_err());
+    }
+
+    /// A frame whose header claims a compressed size smaller than the
+    /// header itself, not stored raw: an error, not an arithmetic overflow
+    /// (it panicked in debug builds).
+    #[test]
+    fn frame_size_below_the_header_is_an_error() {
+        let mut frame = vec![2u8, 1, 1 << 5, 4];
+        for v in [64u32, 64, 8] {
+            frame.extend_from_slice(&v.to_le_bytes());
+        }
+        frame.extend_from_slice(&[0; 40]);
+        assert!(blosc_decompress(&frame, 1000).is_err());
+        for cbytes in 0..16u32 {
+            frame[12..16].copy_from_slice(&cbytes.to_le_bytes());
+            assert!(blosc_decompress(&frame, 1000).is_err(), "cbytes={cbytes}");
+        }
+    }
+
+    /// A BloscLZ frame (our encoder cannot write one): a single block,
+    /// one stream, no shuffle.
+    fn blosclz_frame() -> Vec<u8> {
+        let stream = [2, b'a', b'b', b'c', (6 << 5), 2, 0, b'Z'];
+        let mut f = vec![2u8, 1, 0, 1];
+        for v in [12u32, 12, (HEADER + 4 + 4 + stream.len()) as u32] {
+            f.extend_from_slice(&v.to_le_bytes());
+        }
+        f.extend_from_slice(&((HEADER + 4) as u32).to_le_bytes());
+        f.extend_from_slice(&(stream.len() as u32).to_le_bytes());
+        f.extend_from_slice(&stream);
+        f
+    }
+
+    /// Random and mutated frames, every codec and shuffle: errors are fine,
+    /// panics are not.
+    #[test]
+    fn fuzzed_frames_never_panic() {
+        let limit = 6000;
+        let data: Vec<u8> = (0..1500u32).flat_map(|i| (i / 5).to_le_bytes()).collect();
+        let mut seeds = vec![blosclz_frame()];
+        for codec in [1u32, 3, 4, 5] {
+            for shuffle in [0u32, 1, 2] {
+                for (ts, n) in [(4usize, data.len()), (4, 520), (1, 300), (2, 4)] {
+                    let f = desc(vec![2, 2, ts as u32, 0, 5, shuffle, codec]);
+                    let ctx = FilterContext {
+                        filter: &f,
+                        element_size: ts,
+                        max_output: n,
+                    };
+                    seeds.push(blosc_encode(&data[..n], &ctx).unwrap());
+                }
+            }
+        }
+        // Stored raw.
+        let f = desc(vec![2, 2, 4, 0, 0, 1, 1]);
+        let ctx = FilterContext {
+            filter: &f,
+            element_size: 4,
+            max_output: 64,
+        };
+        seeds.push(blosc_encode(&data[..64], &ctx).unwrap());
+        crate::test_fuzz::fuzz_decoder(0xb10, &seeds, 30_000, limit, |s| {
+            blosc_decompress(s, limit)
+        });
+    }
+
+    /// BloscLZ streams on their own, random and mutated.
+    #[test]
+    fn fuzzed_blosclz_streams_never_panic() {
+        let seed = blosclz_frame()[HEADER + 8..].to_vec();
+        let mut out = [0u8; 64];
+        crate::test_fuzz::fuzz_decoder(0xb11, &[seed], 30_000, 64, |s| {
+            let n = blosclz_decompress(s, &mut out);
+            if n == 0 {
+                Err(err("malformed"))
+            } else {
+                Ok(out[..n].to_vec())
+            }
+        });
+    }
+}

crates/clawhdf5-format/src/filters_bzip2.rs

@@ -0,0 +1,133 @@
+//! bzip2 (HDF5 filter 307, PyTables' `H5Zbzip2.c`, hdf5plugin's `BZip2`).
+//!
+//! The chunk is one bzip2 stream; `cd_values[0]` is the block size (1-9,
+//! the compression level). Decoded with the `bzip2` crate's default backend,
+//! `libbz2-rs-sys`, a pure-Rust port of libbzip2.
+
+use crate::addr::saturating_usize;
+use crate::error::FormatError;
+use crate::filter_registry::FilterContext;
+
+fn err(msg: &str) -> FormatError {
+    FormatError::DecompressionError(format!("bzip2: {msg}"))
+}
+
+/// Decode a bzip2-filtered chunk, refusing output beyond the chunk size.
+pub(crate) fn bzip2_decode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+    use bzip2::{Decompress, Status};
+    let limit = ctx.output_limit();
+    let max_capacity = limit.saturating_add(1);
+    let hint = if ctx.max_output != 0 {
+        ctx.max_output
+    } else {
+        input.len().saturating_mul(4)
+    };
+    let mut out = Vec::new();
+    out.try_reserve_exact(hint.clamp(1, max_capacity))
+        .map_err(|_| err("cannot allocate the output buffer"))?;
+    let mut dec = Decompress::new(false);
+    loop {
+        let (in_before, out_before) = (dec.total_in(), dec.total_out());
+        let status = dec
+            .decompress_vec(&input[saturating_usize(in_before)..], &mut out)
+            .map_err(|e| err(&e.to_string()))?;
+        if out.len() > limit {
+            return Err(err("output exceeds the chunk size"));
+        }
+        if status == Status::StreamEnd {
+            return Ok(out);
+        }
+        if out.len() == out.capacity() {
+            let grow = out
+                .capacity()
+                .min(max_capacity.saturating_sub(out.capacity()))
+                .max(1);
+            out.try_reserve_exact(grow)
+                .map_err(|_| err("cannot allocate the output buffer"))?;
+        } else if saturating_usize(dec.total_in()) >= input.len()
+            || (dec.total_in(), dec.total_out()) == (in_before, out_before)
+        {
+            return Err(err("truncated stream"));
+        }
+    }
+}
+
+/// Encode a chunk as one bzip2 stream at block size `cd_values[0]`
+/// (default 9, as hdf5plugin).
+pub(crate) fn bzip2_encode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+    use bzip2::{Action, Compress, Compression, Status};
+    let level = ctx.client_data().first().copied().unwrap_or(9).clamp(1, 9);
+    let cerr = |m: String| FormatError::CompressionError(format!("bzip2: {m}"));
+    let mut enc = Compress::new(Compression::new(level), 0);
+    // bzip2's worst case is about 1% + 600 bytes over the input.
+    let mut out = Vec::with_capacity(input.len() + input.len() / 100 + 600);
+    loop {
+        let consumed = saturating_usize(enc.total_in());
+        let status = enc
+            .compress_vec(&input[consumed..], &mut out, Action::Finish)
+            .map_err(|e| cerr(e.to_string()))?;
+        if status == Status::StreamEnd {
+            return Ok(out);
+        }
+        if out.len() == out.capacity() {
+            out.reserve(out.capacity().max(4096));
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::filter_pipeline::{FILTER_BZIP2, FilterDescription};
+
+    fn desc(level: u32) -> FilterDescription {
+        FilterDescription {
+            filter_id: FILTER_BZIP2,
+            name: None,
+            flags: 0,
+            client_data: vec![level],
+        }
+    }
+
+    #[test]
+    fn round_trips_and_bounds() {
+        let data: Vec<u8> = (0..100_000u32)
+            .flat_map(|i| (i % 777).to_le_bytes())
+            .collect();
+        for level in [1, 5, 9] {
+            let f = desc(level);
+            let ctx = FilterContext {
+                filter: &f,
+                element_size: 4,
+                max_output: data.len(),
+            };
+            let enc = bzip2_encode(&data, &ctx).unwrap();
+            assert!(enc.len() < data.len() / 4);
+            assert_eq!(bzip2_decode(&enc, &ctx).unwrap(), data);
+            // Truncated, and larger than the chunk: errors, not data.
+            assert!(bzip2_decode(&enc[..enc.len() / 2], &ctx).is_err());
+            let small = FilterContext {
+                max_output: data.len() - 1,
+                ..ctx
+            };
+            assert!(bzip2_decode(&enc, &small).is_err());
+        }
+    }
+
+    /// Random and mutated streams: errors are fine, panics are not.
+    #[test]
+    fn fuzzed_streams_never_panic() {
+        let f = desc(9);
+        let data: Vec<u8> = (0..4000u32).flat_map(|i| (i % 91).to_le_bytes()).collect();
+        let ctx = FilterContext {
+            filter: &f,
+            element_size: 4,
+            max_output: data.len(),
+        };
+        let seeds = vec![
+            bzip2_encode(&data, &ctx).unwrap(),
+            bzip2_encode(&data[..40], &ctx).unwrap(),
+        ];
+        crate::test_fuzz::fuzz_decoder(0xb2, &seeds, 3_000, data.len(), |s| bzip2_decode(s, &ctx));
+    }
+}

crates/clawhdf5-format/src/filters_lzf.rs

@@ -0,0 +1,260 @@
+//! LZF (HDF5 filter 32000) — h5py's built-in compression filter
+//! (`compression="lzf"`), in pure Rust.
+//!
+//! The chunk is one raw LZF stream (liblzf 3.x format, no header). The
+//! stream is a sequence of instructions, each starting with a control byte:
+//!
+//! * `000LLLLL` — a literal run: the next `L + 1` bytes (1..=32) are copied.
+//! * `LLLOOOOO [E] OOOOOOOO` — a back reference: copy `len + 2` bytes from
+//!   `distance` bytes back, where `len` is the top three bits (1..=6), or
+//!   `7 + E` when they are all ones, and `distance` is the 13-bit offset
+//!   (high five bits in the control byte, low eight in the last byte) plus 1.
+//!
+//! h5py's filter (`lzf_filter.c`) records the chunk's size in bytes in
+//! `cd_values[2]` (slots 0 and 1 hold the filter and liblzf versions) and
+//! sizes its output buffer from it.
+
+#[cfg(not(feature = "std"))]
+extern crate alloc;
+#[cfg(not(feature = "std"))]
+use alloc::{format, vec, vec::Vec};
+
+use crate::error::FormatError;
+use crate::filter_registry::FilterContext;
+
+/// `H5PY_FILTER_LZF_VERSION`, written to `cd_values[0]`.
+pub const LZF_FILTER_VERSION: u32 = 4;
+/// `LZF_VERSION` (liblzf 1.5), written to `cd_values[1]`.
+pub const LZF_API_VERSION: u32 = 0x0105;
+
+const MAX_LITERAL: usize = 32;
+const MAX_OFFSET: usize = 1 << 13;
+const MAX_REF: usize = (1 << 8) + (1 << 3);
+const HASH_LOG: u32 = 14;
+
+fn err(msg: &str) -> FormatError {
+    FormatError::DecompressionError(format!("lzf: {msg}"))
+}
+
+/// Decode an LZF-filtered chunk.
+pub(crate) fn lzf_decode(input: &[u8], ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+    let limit = ctx.output_limit();
+    let hint = match ctx.client_data().get(2) {
+        Some(&n) if n != 0 => n as usize,
+        _ => input.len().saturating_mul(2),
+    };
+    lzf_decompress(input, hint.min(limit), limit)
+}
+
+/// Decompress a raw LZF stream, refusing to produce more than `limit` bytes.
+pub fn lzf_decompress(
+    input: &[u8],
+    size_hint: usize,
+    limit: usize,
+) -> Result<Vec<u8>, FormatError> {
+    let mut out: Vec<u8> = Vec::new();
+    out.try_reserve(size_hint)
+        .map_err(|_| err("cannot allocate the output buffer"))?;
+    let mut ip = 0usize;
+    while ip < input.len() {
+        let ctrl = input[ip] as usize;
+        ip += 1;
+        if ctrl < 32 {
+            let run = ctrl + 1;
+            let lit = input
+                .get(ip..ip + run)
+                .ok_or_else(|| err("literal run past the end of the input"))?;
+            if out.len() + run > limit {
+                return Err(err("output exceeds the chunk size"));
+            }
+            out.extend_from_slice(lit);
+            ip += run;
+        } else {
+            let mut len = ctrl >> 5;
+            if len == 7 {
+                len += *input
+                    .get(ip)
+                    .ok_or_else(|| err("truncated back reference"))?
+                    as usize;
+                ip += 1;
+            }
+            let low = *input
+                .get(ip)
+                .ok_or_else(|| err("truncated back reference"))? as usize;
+            ip += 1;
+            let distance = ((ctrl & 0x1f) << 8) + low + 1;
+            let len = len + 2;
+            if distance > out.len() {
+                return Err(err("back reference before the start of the output"));
+            }
+            if out.len() + len > limit {
+                return Err(err("output exceeds the chunk size"));
+            }
+            let start = out.len() - distance;
+            if distance >= len {
+                out.extend_from_within(start..start + len);
+            } else {
+                // Overlapping copy: repeats the last `distance` bytes.
+                for k in 0..len {
+                    let b = out[start + k];
+                    out.push(b);
+                }
+            }
+        }
+    }
+    Ok(out)
+}
+
+/// Encode a chunk with the LZF filter.
+pub(crate) fn lzf_encode(input: &[u8], _ctx: &FilterContext<'_>) -> Result<Vec<u8>, FormatError> {
+    Ok(lzf_compress(input))
+}
+
+fn hash3(b: &[u8]) -> usize {
+    let v = (u32::from(b[0]) << 16) | (u32::from(b[1]) << 8) | u32::from(b[2]);
+    (v.wrapping_mul(2_654_435_761) >> (32 - HASH_LOG)) as usize
+}
+
+fn flush_literals(out: &mut Vec<u8>, lit: &[u8]) {
+    for run in lit.chunks(MAX_LITERAL) {
+        out.push((run.len() - 1) as u8);
+        out.extend_from_slice(run);
+    }
+}
+
+/// Compress `input` into a raw LZF stream any liblzf decoder reads.
+///
+/// Incompressible input grows by one byte per 32. (h5py's own filter gives
+/// up on such a chunk and stores it unfiltered; storing the slightly larger
+/// stream is equally readable.)
+pub fn lzf_compress(input: &[u8]) -> Vec<u8> {
+    let n = input.len();
+    let mut out = Vec::with_capacity(n + n / MAX_LITERAL + 1);
+    let mut table = vec![0u32; 1 << HASH_LOG];
+    let mut lit_start = 0usize;
+    let mut i = 0usize;
+    while i + 2 < n {
+        let h = hash3(&input[i..]);
+        let cand = table[h] as usize;
+        table[h] = (i + 1) as u32;
+        if cand != 0 {
+            let r = cand - 1;
+            let distance = i - r;
+            if distance <= MAX_OFFSET && input[r..r + 3] == input[i..i + 3] {
+                let max_len = (n - i).min(MAX_REF);
+                let mut len = 3;
+                while len < max_len && input[r + len] == input[i + len] {
+                    len += 1;
+                }
+                flush_literals(&mut out, &input[lit_start..i]);
+                let code = len - 2;
+                let off = distance - 1;
+                if code < 7 {
+                    out.push(((code << 5) | (off >> 8)) as u8);
+                } else {
+                    out.push(((7 << 5) | (off >> 8)) as u8);
+                    out.push((code - 7) as u8);
+                }
+                out.push((off & 0xff) as u8);
+                // Index the positions the match covered so later data can
+                // refer back into it.
+                let end = i + len;
+                let mut j = i + 1;
+                while j < end && j + 2 < n {
+                    table[hash3(&input[j..])] = (j + 1) as u32;
+                    j += 1;
+                }
+                i = end;
+                lit_start = i;
+                continue;
+            }
+        }
+        i += 1;
+    }
+    flush_literals(&mut out, &input[lit_start..]);
+    out
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn round_trip(data: &[u8]) {
+        let c = lzf_compress(data);
+        assert_eq!(lzf_decompress(&c, data.len(), data.len()).unwrap(), data);
+    }
+
+    #[test]
+    fn round_trips() {
+        round_trip(b"");
+        round_trip(b"a");
+        round_trip(b"abcabcabcabcabcabcabcabcabcabcabcabc");
+        round_trip(&[7u8; 10_000]);
+        let noise: Vec<u8> = (0..70_000u32)
+            .map(|i| (i.wrapping_mul(2_654_435_761) >> 13) as u8)
+            .collect();
+        round_trip(&noise);
+        let ramp: Vec<u8> = (0..100_000u32)
+            .flat_map(|i| (i % 1000).to_le_bytes())
+            .collect();
+        round_trip(&ramp);
+    }
+
+    #[test]
+    fn compresses_repetitive_data() {
+        let data = [42u8; 4096];
+        assert!(lzf_compress(&data).len() < 100);
+    }
+
+    /// The chunk h5py 3.16's bundled liblzf writes for
+    /// `b"hello hello hello hello"` (read back with `read_direct_chunk`): a
+    /// 7-byte literal, a 14-byte back reference 6 bytes back (extended
+    /// length), and a 2-byte literal.
+    #[test]
+    fn decodes_liblzf_output() {
+        let stream = b"\x06hello h\xe0\x05\x05\x01lo";
+        assert_eq!(
+            lzf_decompress(stream, 23, 23).unwrap(),
+            b"hello hello hello hello"
+        );
+    }
+
+    #[test]
+    fn rejects_corrupt_streams() {
+        // Back reference before the start.
+        assert!(lzf_decompress(&[0x20, 0x00], 10, 10).is_err());
+        // Literal run past the end.
+        assert!(lzf_decompress(&[0x05, 1, 2], 10, 10).is_err());
+        // Output over the limit.
+        let c = lzf_compress(&[1u8; 100]);
+        assert!(lzf_decompress(&c, 10, 99).is_err());
+    }
+
+    /// Random and mutated streams: errors are fine, panics are not.
+    #[test]
+    fn fuzzed_streams_never_panic() {
+        let seeds: Vec<Vec<u8>> = [
+            b"hello hello hello hello".to_vec(),
+            vec![7u8; 3000],
+            (0..2000u32).flat_map(|i| (i % 37).to_le_bytes()).collect(),
+            (0..500u32)
+                .map(|i| (i.wrapping_mul(2_654_435_761) >> 13) as u8)
+                .collect(),
+        ]
+        .iter()
+        .map(|d| lzf_compress(d))
+        .collect();
+        for limit in [0usize, 23, 4096, 8000] {
+            crate::test_fuzz::fuzz_decoder(
+                0x1f2 + limit as u64,
+                &seeds[..1],
+                5_000,
+                limit.max(23),
+                |s| lzf_decompress(s, limit, limit.max(23)),
+            );
+        }
+        crate::test_fuzz::fuzz_decoder(0x1f3, &seeds, 20_000, 8000, |s| {
+            lzf_decompress(s, 8000, 8000)
+        });
+    }
+}

crates/clawhdf5-format/src/filters_szip.rs

@@ -34,6 +34,7 @@ const SZ_NN_OPTION_MASK: u32 = 32;
 ///
 /// The chunk is a 4-byte little-endian uncompressed size followed by the
 /// szlib stream.
+#[cfg_attr(not(feature = "szip"), allow(dead_code))]
 pub(crate) fn szip_decompress(
     _data: &[u8],
     _cd: &[u32],

crates/clawhdf5-format/src/fixed_array.rs

@@ -6,19 +6,23 @@ extern crate alloc;
 #[cfg(not(feature = "std"))]
 use alloc::{format, vec, vec::Vec};
 
+use crate::addr::to_usize;
 use crate::chunk_grid::ChunkGrid;
 use crate::chunked_read::ChunkInfo;
 use crate::error::FormatError;
+use crate::storage::{PAGED_BLOCK_ONE_READ_MAX, Storage, Window, len_usize, read_exact_at};
 
 /// Verify the Jenkins lookup3 checksum stored immediately after
-/// `data[start..end]`, as every Fixed Array structure carries one.
+/// `data[start..end]`, as every Fixed Array structure carries one. `w` is
+/// a window of the file and `start`/`end` are relative to it.
 ///
 /// A corrupt chunk index silently yields addresses pointing at the wrong
 /// bytes, so a mismatch has to be an error rather than a shrug: without this
 /// the damage surfaces as plausible-looking data from the wrong chunk.
 #[cfg(feature = "checksum")]
-fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatError> {
+fn verify_checksum(w: &Window<'_>, start: usize, end: usize) -> Result<(), FormatError> {
-    ensure_len(data, end, 4)?;
+    w.ensure(end, 4)?;
+    let data = &w.bytes;
     let stored = u32::from_le_bytes([data[end], data[end + 1], data[end + 2], data[end + 3]]);
     let computed = crate::checksum::jenkins_lookup3(&data[start..end]);
     if computed != stored {
@@ -31,7 +35,7 @@ fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatEr
 }
 
 #[cfg(not(feature = "checksum"))]
-fn verify_checksum(_data: &[u8], _start: usize, _end: usize) -> Result<(), FormatError> {
+fn verify_checksum(_w: &Window<'_>, _start: usize, _end: usize) -> Result<(), FormatError> {
     Ok(())
 }
 
@@ -73,19 +77,6 @@ fn read_length(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
     read_offset(data, pos, size)
 }
 
-fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
-    if offset
-        .checked_add(needed)
-        .is_none_or(|end| end > data.len())
-    {
-        return Err(FormatError::UnexpectedEof {
-            expected: offset.saturating_add(needed),
-            available: data.len(),
-        });
-    }
-    Ok(())
-}
-
 fn is_undefined(data: &[u8], pos: usize, size: u8) -> bool {
     let s = size as usize;
     if pos + s > data.len() {
@@ -101,13 +92,24 @@ impl FixedArrayHeader {
         offset: usize,
         offset_size: u8,
         length_size: u8,
+    ) -> Result<Self, FormatError> {
+        Self::parse_in(file_data, offset as u64, offset_size, length_size)
+    }
+
+    /// [`Self::parse`] over any [`Storage`]: one read of the header.
+    pub fn parse_in<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        offset_size: u8,
+        length_size: u8,
     ) -> Result<Self, FormatError> {
         // FAHD signature(4) + version(1) + client_id(1) + element_size(1) +
         // max_nelmts_bits(1) + num_elements(length_size) + data_block_addr(offset_size) + checksum(4)
         let min_size = 4 + 1 + 1 + 1 + 1 + length_size as usize + offset_size as usize + 4;
-        ensure_len(file_data, offset, min_size)?;
+        let w = Window::read(file, offset, min_size)?;
+        w.ensure(0, min_size)?;
 
-        let d = &file_data[offset..];
+        let d: &[u8] = &w.bytes;
         if &d[0..4] != b"FAHD" {
             return Err(FormatError::ChunkedReadError(
                 "invalid Fixed Array header signature".into(),
@@ -130,7 +132,7 @@ impl FixedArrayHeader {
         pos += length_size as usize;
         let data_block_address = read_offset(d, pos, offset_size)?;
         pos += offset_size as usize;
-        verify_checksum(file_data, offset, offset + pos)?;
+        verify_checksum(&w, 0, pos)?;
 
         Ok(FixedArrayHeader {
             client_id,
@@ -156,15 +158,39 @@ pub fn read_fixed_array_chunks(
     chunk_dimensions: &[u32],
     element_size: u32,
     offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<ChunkInfo>, FormatError> {
+    read_fixed_array_chunks_in(
+        &file_data,
+        header,
+        dataset_dims,
+        max_dims,
+        chunk_dimensions,
+        element_size,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`read_fixed_array_chunks`] over any [`Storage`]: one read of the data
+/// block's prefix, one of the whole data block (pages included).
+#[allow(clippy::too_many_arguments)]
+pub fn read_fixed_array_chunks_in<S: Storage + ?Sized>(
+    file: &S,
+    header: &FixedArrayHeader,
+    dataset_dims: &[u64],
+    max_dims: Option<&[u64]>,
+    chunk_dimensions: &[u32],
+    element_size: u32,
+    offset_size: u8,
     _length_size: u8,
 ) -> Result<Vec<ChunkInfo>, FormatError> {
-    let db_offset = header.data_block_address as usize;
+    let file_len = len_usize(file);
+    let db_offset = to_usize(header.data_block_address)?;
 
     // Parse data block header: FADB(4) + version(1) + client_id(1) + header_address(offset_size)
     let db_header_size = 4 + 1 + 1 + offset_size as usize;
-    ensure_len(file_data, db_offset, db_header_size)?;
+    let d = read_exact_at(file, db_offset as u64, db_header_size)?;
-
-    let d = &file_data[db_offset..];
     if &d[0..4] != b"FADB" {
         return Err(FormatError::ChunkedReadError(
             "invalid Fixed Array data block signature".into(),
@@ -174,11 +200,11 @@ pub fn read_fixed_array_chunks(
     // Elements start immediately after the data block prefix.
     let elements_start = db_offset + db_header_size;
 
-    let num_elements = header.num_elements as usize;
+    let num_elements = to_usize(header.num_elements)?;
     // A chunk index cannot describe more elements than the file has bytes (each
     // element occupies at least `offset_size` bytes). Reject a corrupt count
     // before it can drive a huge loop or overflow an offset computation.
-    if num_elements > file_data.len() {
+    if num_elements > file_len {
         return Err(FormatError::ChunkedReadError(
             "Fixed Array element count exceeds file size".into(),
         ));
@@ -208,11 +234,15 @@ pub fn read_fixed_array_chunks(
         chunk_dimensions.iter().map(|&d| d as u64).product::<u64>() * element_size as u64;
 
     let mut chunks = Vec::new();
-    let push_element =
+    // `rel` is relative to the data block, whose bytes are in `w`.
-        |i: usize, abs: usize, chunks: &mut Vec<ChunkInfo>| -> Result<(), FormatError> {
+    let push_element = |w: &Window<'_>,
+                        i: usize,
+                        rel: usize,
+                        chunks: &mut Vec<ChunkInfo>|
+     -> Result<(), FormatError> {
         if let Some((address, chunk_size, filter_mask)) = parse_fa_element(
-                file_data,
+            w,
-                abs,
+            rel,
             header.client_id,
             offset_size,
             header.element_size,
@@ -246,10 +276,16 @@ pub fn read_fixed_array_chunks(
 
     if !is_paged {
         // Non-paged: prefix, then `num_elements` elements packed directly,
-        // then a checksum over both.
+        // then a checksum over both. One window holds all of it (or ends at
-        verify_checksum(file_data, db_offset, elem_at(elements_start, num_elements)?)?;
+        // the end of the file), so its bounds checks are the whole-file ones.
+        let end = elem_at(elements_start, num_elements)?;
+        // The checksum's bounds check comes first: make it before reading.
+        #[cfg(feature = "checksum")]
+        Window::check_extent(file, db_offset as u64, end - db_offset, 4)?;
+        let w = Window::read(file, db_offset as u64, end.saturating_add(4) - db_offset)?;
+        verify_checksum(&w, 0, end - db_offset)?;
         for i in 0..num_elements {
-            push_element(i, elem_at(elements_start, i)?, &mut chunks)?;
+            push_element(&w, i, elem_at(elements_start, i)? - db_offset, &mut chunks)?;
         }
         return Ok(chunks);
     }
@@ -272,22 +308,40 @@ pub fn read_fixed_array_chunks(
         .and_then(|x| x.checked_add(4))
         .ok_or_else(stride_overflow)?;
 
-    if bitmap_start + bitmap_size > file_data.len() {
+    if bitmap_start + bitmap_size > file_len {
         return Err(FormatError::UnexpectedEof {
             expected: bitmap_start + bitmap_size,
-            available: file_data.len(),
+            available: file_len,
         });
     }
+    // The whole data block in one window when it is small: every page slot
+    // is at most `page_stride` bytes, so every position checked below lies
+    // inside it (or past the end of the file). A larger block is read as its
+    // prefix and bitmap, then each page in use on its own.
+    let block_len = (pages_start - db_offset).saturating_add(npages.saturating_mul(page_stride));
+    let whole = if block_len <= PAGED_BLOCK_ONE_READ_MAX {
+        Some(Window::read(file, db_offset as u64, block_len)?)
+    } else {
+        None
+    };
+    let head_w;
+    let head = match &whole {
+        Some(w) => w,
+        None => {
+            head_w = Window::read(file, db_offset as u64, pages_start - db_offset)?;
+            &head_w
+        }
+    };
     // The prefix and page bitmap are covered by their own checksum, and each
     // initialised page by one of its own.
-    verify_checksum(file_data, db_offset, bitmap_start + bitmap_size)?;
+    verify_checksum(head, 0, bitmap_start + bitmap_size - db_offset)?;
 
     for p in 0..npages {
         let page_first = p * page_nelmts; // < num_elements, cannot overflow
         let page_count = core::cmp::min(page_nelmts, num_elements - page_first);
 
         // Check the page-init bit (MSB-first within each byte).
-        let bit_byte = file_data[bitmap_start + p / 8];
+        let bit_byte = head.bytes[bitmap_start + p / 8 - db_offset];
         let bit_mask = 1u8 << (7 - (p % 8));
         if bit_byte & bit_mask == 0 {
             continue; // entire page unallocated
@@ -297,21 +351,33 @@ pub fn read_fixed_array_chunks(
             .checked_mul(page_stride)
             .and_then(|o| pages_start.checked_add(o))
             .ok_or_else(stride_overflow)?;
-        verify_checksum(file_data, page_off, elem_at(page_off, page_count)?)?;
+        let page_end = elem_at(page_off, page_count)?;
+        // `w` holds the page from `base` on (positions below are relative
+        // to it).
+        let page_w;
+        let (w, base) = match &whole {
+            Some(w) => (w, db_offset),
+            None => {
+                page_w =
+                    Window::read(file, page_off as u64, page_end.saturating_add(4) - page_off)?;
+                (&page_w, page_off)
+            }
+        };
+        verify_checksum(w, page_off - base, page_end - base)?;
         for e in 0..page_count {
-            push_element(page_first + e, elem_at(page_off, e)?, &mut chunks)?;
+            push_element(w, page_first + e, elem_at(page_off, e)? - base, &mut chunks)?;
         }
     }
 
     Ok(chunks)
 }
 
-/// Parse a single Fixed Array element at absolute file offset `abs`.
+/// Parse a single Fixed Array element at offset `abs` of the window `w`.
 ///
 /// Returns `Some((address, chunk_size, filter_mask))` for an allocated chunk, or
 /// `None` if the element is undefined (an unallocated chunk, address all-`0xFF`).
 fn parse_fa_element(
-    file_data: &[u8],
+    w: &Window<'_>,
     abs: usize,
     client_id: u8,
     offset_size: u8,
@@ -321,12 +387,8 @@ fn parse_fa_element(
     let os = offset_size as usize;
     if client_id == 0 {
         // Non-filtered: element is just the chunk address.
-        if abs + os > file_data.len() {
+        w.ensure(abs, os)?;
-            return Err(FormatError::UnexpectedEof {
+        let file_data: &[u8] = &w.bytes;
-                expected: abs + os,
-                available: file_data.len(),
-            });
-        }
         if is_undefined(file_data, abs, offset_size) {
             return Ok(None);
         }
@@ -341,17 +403,14 @@ fn parse_fa_element(
             ));
         }
         let chunk_size_bytes = es - os - 4;
-        if abs + es > file_data.len() {
+        w.ensure(abs, es)?;
-            return Err(FormatError::UnexpectedEof {
+        let file_data: &[u8] = &w.bytes;
-                expected: abs + es,
-                available: file_data.len(),
-            });
-        }
         if is_undefined(file_data, abs, offset_size) {
             return Ok(None);
         }
         let address = read_offset(file_data, abs, offset_size)?;
-        let chunk_size = read_variable_length(&file_data[abs + os..], chunk_size_bytes)?;
+        let chunk_size =
+            read_variable_length(&file_data[abs + os..abs + es - 4], chunk_size_bytes)?;
         let fm_off = abs + os + chunk_size_bytes;
         let filter_mask = u32::from_le_bytes([
             file_data[fm_off],
@@ -813,4 +872,126 @@ mod tests {
             .collect();
         assert_eq!(got, expect);
     }
+
+    /// A fixed array (header at 0x100, data block at 0x200) of `n` chunks,
+    /// filtered or not, paged when `n` exceeds `1 << page_bits`; every
+    /// page initialised except page 1.
+    fn build_fixed_array(n: usize, filtered: bool, page_bits: u8) -> Vec<u8> {
+        let os = 8usize;
+        let es = if filtered { os + 4 + 4 } else { os };
+        let (fahd, db) = (0x100usize, 0x200usize);
+        let mut f = vec![0u8; 0x2000];
+        f[fahd..fahd + 4].copy_from_slice(b"FAHD");
+        f[fahd + 5] = u8::from(filtered);
+        f[fahd + 6] = es as u8;
+        f[fahd + 7] = page_bits;
+        f[fahd + 8..fahd + 16].copy_from_slice(&(n as u64).to_le_bytes());
+        f[fahd + 16..fahd + 24].copy_from_slice(&(db as u64).to_le_bytes());
+        stamp_checksum(&mut f, fahd, fahd + 24);
+        f[db..db + 4].copy_from_slice(b"FADB");
+        f[db + 5] = u8::from(filtered);
+        f[db + 6..db + 14].copy_from_slice(&(fahd as u64).to_le_bytes());
+        let elems = db + 6 + os;
+        let write = |f: &mut Vec<u8>, at: usize, i: usize| {
+            let addr = if i == 2 {
+                u64::MAX
+            } else {
+                0x1000 + i as u64 * 0x100
+            };
+            f[at..at + os].copy_from_slice(&addr.to_le_bytes());
+            if filtered {
+                f[at + os..at + os + 4].copy_from_slice(&(100 + i as u32).to_le_bytes());
+                f[at + os + 4..at + os + 8].copy_from_slice(&(i as u32 & 1).to_le_bytes());
+            }
+        };
+        let page = 1usize << page_bits;
+        if n <= page {
+            for i in 0..n {
+                write(&mut f, elems + i * es, i);
+            }
+            stamp_checksum(&mut f, db, elems + n * es);
+        } else {
+            let npages = n.div_ceil(page);
+            let bitmap = npages.div_ceil(8);
+            for p in 0..npages {
+                if p != 1 {
+                    f[elems + p / 8] |= 0x80 >> (p % 8);
+                }
+            }
+            stamp_checksum(&mut f, db, elems + bitmap);
+            let pages_start = elems + bitmap + 4;
+            for p in (0..npages).filter(|&p| p != 1) {
+                let at = pages_start + p * (page * es + 4);
+                let count = page.min(n - p * page);
+                for e in 0..count {
+                    write(&mut f, at + e * es, p * page + e);
+                }
+                stamp_checksum(&mut f, at, at + count * es);
+            }
+        }
+        f
+    }
+
+    /// Non-paged and paged, filtered and unfiltered arrays, cut at every
+    /// length through the data block and with a damaged byte, read
+    /// identically through a `read_at`-only storage.
+    #[test]
+    fn storage_reads_match_slice_reads() {
+        use crate::storage::CountingStorage;
+        for (n, filtered, bits) in [(3, false, 10), (3, true, 10), (11, false, 2), (11, true, 2)] {
+            let full = build_fixed_array(n, filtered, bits);
+            let es = if filtered { 16 } else { 8 };
+            let dims = [n as u64 * 20];
+            let h = FixedArrayHeader::parse(&full, 0x100, 8, 8).unwrap();
+            let chunks = read_fixed_array_chunks(&full, &h, &dims, None, &[20], 8, 8, 8).unwrap();
+            // Chunk 2 is unallocated, and so is page 1 of a paged array.
+            let expect = if n > 4 { n - 1 - 4 } else { n - 1 };
+            assert_eq!(chunks.len(), expect);
+            let mut files = Vec::new();
+            for cut in (0x100..0x200 + 40 + n * (es + 4) + 16).step_by(3) {
+                files.push(full[..cut].to_vec());
+            }
+            for at in [0x104, 0x210, 0x21a, 0x230] {
+                let mut damaged = full.clone();
+                damaged[at] ^= 1;
+                files.push(damaged);
+            }
+            files.push(full);
+            for f in files {
+                let storage = CountingStorage::new(f.clone());
+                let want = FixedArrayHeader::parse(&f, 0x100, 8, 8);
+                let got = FixedArrayHeader::parse_in(&storage, 0x100, 8, 8);
+                assert_eq!(format!("{got:?}"), format!("{want:?}"));
+                let Ok(h) = want else { continue };
+                let want = read_fixed_array_chunks(&f, &h, &dims, None, &[20], 8, 8, 8);
+                let got = read_fixed_array_chunks_in(&storage, &h, &dims, None, &[20], 8, 8, 8);
+                assert_eq!(format!("{got:?}"), format!("{want:?}"), "{} bytes", f.len());
+            }
+        }
+    }
+
+    /// A header whose element count stretches its data block (one checksum
+    /// over the whole block) far past the end of a 16 MiB file: the
+    /// checksum's bounds check fails before the block is read, with the
+    /// slice read's error.
+    #[cfg(feature = "checksum")]
+    #[test]
+    fn oversized_block_fails_before_reading() {
+        use crate::storage::CountingStorage;
+        let mut f = build_fixed_array(3, false, 10);
+        f.resize(16 << 20, 0);
+        let mut h = FixedArrayHeader::parse(&f, 0x100, 8, 8).unwrap();
+        h.max_nelmts_bits = 30;
+        h.num_elements = 4 << 20;
+        let dims = [h.num_elements * 20];
+        let want = read_fixed_array_chunks(&f, &h, &dims, None, &[20], 8, 8, 8);
+        assert!(
+            matches!(want, Err(FormatError::UnexpectedEof { .. })),
+            "{want:?}"
+        );
+        let storage = CountingStorage::new(f);
+        let got = read_fixed_array_chunks_in(&storage, &h, &dims, None, &[20], 8, 8, 8);
+        assert_eq!(format!("{got:?}"), format!("{want:?}"));
+        assert!(storage.bytes_read() < 64, "{} bytes", storage.bytes_read());
+    }
 }

crates/clawhdf5-format/src/gather.rs

@@ -0,0 +1,597 @@
+//! Copying a selection out of a row-major buffer one contiguous run at a time.
+//!
+//! A selection's elements, in output order, fall into runs that are adjacent
+//! in the source: a whole block along the last dimension, blocks that touch
+//! (`stride == block`), and whole rows when the inner dimensions are selected
+//! in full. Copying run by run turns a 256 x 256 hyperslab of a 1024-wide
+//! dataset into 256 `memcpy`s of 1 KiB, where the old extractor recursed and
+//! bounds-checked once per element.
+
+#[cfg(not(feature = "std"))]
+use alloc::{vec, vec::Vec};
+
+use crate::data_read::NativeElement;
+use crate::error::FormatError;
+use crate::selection::Selection;
+use crate::storage::{ExtentBytes, ExtentReq, Storage, raw_batches};
+
+/// Row-major element strides of `dims` (the last dimension has stride 1).
+fn strides(dims: &[u64]) -> Vec<u64> {
+    let mut s = vec![1u64; dims.len()];
+    for d in (0..dims.len().saturating_sub(1)).rev() {
+        s[d] = s[d + 1].wrapping_mul(dims[d + 1]);
+    }
+    s
+}
+
+/// Merges adjacent runs before handing them on.
+struct Coalesce<F: FnMut(u64, u64)> {
+    start: u64,
+    len: u64,
+    emit: F,
+}
+
+impl<F: FnMut(u64, u64)> Coalesce<F> {
+    #[inline]
+    fn push(&mut self, start: u64, len: u64) {
+        if len == 0 {
+            return;
+        }
+        if self.len > 0 && self.start.wrapping_add(self.len) == start {
+            self.len += len;
+            return;
+        }
+        self.flush();
+        self.start = start;
+        self.len = len;
+    }
+
+    fn flush(&mut self) {
+        if self.len > 0 {
+            (self.emit)(self.start, self.len);
+            self.len = 0;
+        }
+    }
+}
+
+/// Call `emit(first_element, element_count)` for each run of a hyperslab's
+/// elements that is contiguous in a row-major dataset of shape `dims`, in
+/// the order the selection returns them. Adjacent runs are merged.
+///
+/// Coordinates at or past a dimension's extent are skipped, as the
+/// element-wise extractor always did; callers that want them to be an error
+/// validate the selection first. The four vectors must have `dims.len()`
+/// entries.
+pub(crate) fn hyperslab_runs(
+    dims: &[u64],
+    start: &[u64],
+    stride: &[u64],
+    count: &[u64],
+    block: &[u64],
+    emit: impl FnMut(u64, u64),
+) {
+    let rank = dims.len();
+    let mut out = Coalesce {
+        start: 0,
+        len: 0,
+        emit,
+    };
+    if rank == 0 {
+        out.push(0, 1);
+        out.flush();
+        return;
+    }
+    if (0..rank).any(|d| count[d] == 0 || block[d] == 0) {
+        return;
+    }
+    let strides = strides(dims);
+    let last = rank - 1;
+    // Odometer over the outer dimensions: (block index, offset in block).
+    let mut ci = vec![0u64; last];
+    let mut bi = vec![0u64; last];
+    'outer: loop {
+        // Base offset of this row, or skip it if a coordinate is out of range.
+        let mut base = 0u64;
+        let mut in_range = true;
+        for d in 0..last {
+            let coord = start[d]
+                .saturating_add(ci[d].saturating_mul(stride[d]))
+                .saturating_add(bi[d]);
+            if coord >= dims[d] {
+                in_range = false;
+                break;
+            }
+            base = base.wrapping_add(coord.wrapping_mul(strides[d]));
+        }
+        if in_range && (stride[last] == block[last] || count[last] == 1) {
+            // Blocks that touch (the common unit-stride case: block 1,
+            // stride 1) are one range; don't split it into per-element runs.
+            let s = start[last];
+            let e = s
+                .saturating_add(count[last].saturating_mul(block[last]))
+                .min(dims[last]);
+            if s < e {
+                out.push(base.wrapping_add(s), e - s);
+            }
+        } else if in_range {
+            for c in 0..count[last] {
+                let s = start[last].saturating_add(c.saturating_mul(stride[last]));
+                if s >= dims[last] {
+                    continue;
+                }
+                let e = s.saturating_add(block[last]).min(dims[last]);
+                out.push(base.wrapping_add(s), e - s);
+            }
+        }
+        // Advance the odometer, last outer dimension fastest.
+        let mut d = last;
+        loop {
+            if d == 0 {
+                break 'outer;
+            }
+            d -= 1;
+            bi[d] += 1;
+            if bi[d] < block[d] {
+                break;
+            }
+            bi[d] = 0;
+            ci[d] += 1;
+            if ci[d] < count[d] {
+                break;
+            }
+            ci[d] = 0;
+        }
+    }
+    out.flush();
+}
+
+/// The selected elements of `src` — a row-major dataset of shape `dims` and
+/// `elem_size`-byte elements — copied into a fresh `Vec<T>`, one `memcpy` per
+/// contiguous run, with no zero-filling of the output first.
+///
+/// For `T` other than `u8`, `elem_size` must equal `size_of::<T>()`. The
+/// selection must be a validated hyperslab, point list or `None` (`All` is the
+/// caller's to handle); `src` must hold exactly the dataset. Anything that
+/// would read outside `src` is an error, never a partial result.
+pub(crate) fn gather<T: NativeElement>(
+    src: &[u8],
+    dims: &[u64],
+    elem_size: usize,
+    selection: &Selection,
+) -> Result<Vec<T>, FormatError> {
+    let t_size = core::mem::size_of::<T>();
+    if elem_size == 0 || (t_size != 1 && t_size != elem_size) {
+        return Err(FormatError::DataSizeMismatch {
+            expected: t_size,
+            actual: elem_size,
+        });
+    }
+    let n_elements = match selection {
+        Selection::None => 0,
+        Selection::Hyperslab { count, block, .. } => count
+            .iter()
+            .zip(block)
+            .try_fold(1u64, |acc, (&c, &b)| acc.checked_mul(c.checked_mul(b)?))
+            .ok_or_else(|| FormatError::Overflow("hyperslab count x block overflows".into()))?,
+        Selection::Points(points) => points.len() as u64,
+        Selection::All => {
+            return Err(FormatError::SelectionOutOfBounds(
+                "gather does not take Selection::All".into(),
+            ));
+        }
+    };
+    let out_bytes = crate::chunked_read::checked_byte_len(n_elements, elem_size)?;
+    let out_len = out_bytes / t_size;
+    let mut out: Vec<T> = crate::bulk_alloc::vec_for_bulk(out_len);
+    let dst = out.as_mut_ptr().cast::<u8>();
+    let mut written = 0usize;
+    let mut failed = false;
+    let mut copy_run = |first: u64, n: u64| {
+        if failed {
+            return;
+        }
+        let range = usize::try_from(first)
+            .ok()
+            .and_then(|f| f.checked_mul(elem_size))
+            .zip(
+                usize::try_from(n)
+                    .ok()
+                    .and_then(|n| n.checked_mul(elem_size)),
+            )
+            .and_then(|(at, len)| Some((at, len, at.checked_add(len)?)));
+        match range {
+            Some((at, len, end)) if end <= src.len() && written + len <= out_bytes => {
+                // SAFETY: `src[at..end]` is in bounds (checked above), and
+                // `dst + written .. + len` lies within `out`'s capacity of
+                // `out_bytes` bytes (checked above); `out` is a fresh
+                // allocation, so the regions do not overlap.
+                unsafe {
+                    core::ptr::copy_nonoverlapping(src.as_ptr().add(at), dst.add(written), len)
+                };
+                written += len;
+            }
+            _ => failed = true,
+        }
+    };
+    let mut bad_point = false;
+    match selection {
+        Selection::Hyperslab {
+            start,
+            stride,
+            count,
+            block,
+        } => {
+            let rank = dims.len();
+            if [start.len(), stride.len(), count.len(), block.len()] != [rank; 4] {
+                return Err(FormatError::SelectionOutOfBounds(
+                    "hyperslab rank does not match dataset rank".into(),
+                ));
+            }
+            hyperslab_runs(dims, start, stride, count, block, &mut copy_run);
+        }
+        Selection::Points(points) => {
+            let strides = strides(dims);
+            let mut runs = Coalesce {
+                start: 0,
+                len: 0,
+                emit: &mut copy_run,
+            };
+            for p in points {
+                if p.len() != dims.len() || p.iter().zip(dims).any(|(c, n)| c >= n) {
+                    bad_point = true;
+                    break;
+                }
+                let at = p
+                    .iter()
+                    .zip(&strides)
+                    .fold(0u64, |acc, (c, s)| acc.wrapping_add(c.wrapping_mul(*s)));
+                runs.push(at, 1);
+            }
+            runs.flush();
+        }
+        Selection::None | Selection::All => {}
+    }
+    if failed || bad_point || written != out_bytes {
+        return Err(FormatError::SelectionOutOfBounds(
+            "selection addresses elements outside the dataset".into(),
+        ));
+    }
+    // SAFETY: all `out_bytes` bytes, i.e. `out_len` values of `T`, were
+    // written above, and every bit pattern is a valid `T` (`NativeElement`).
+    unsafe { out.set_len(out_len) };
+    Ok(out)
+}
+
+/// Largest gap between two of a selection's runs that [`gather_storage`]
+/// reads through rather than asking for the runs separately: skipping a
+/// few KiB costs a remote backend far less than another request (and a
+/// local one less than another call and allocation).
+pub(crate) const GATHER_GAP_BYTES: usize = 4 << 10;
+
+/// Largest single read [`gather_storage`] makes of a selection's runs: runs
+/// are merged into reads up to this size, and a longer run is split.
+pub(crate) const GATHER_SPAN_BYTES: usize = 8 << 20;
+
+/// Call `emit(first_element, element_count)` for each run of a validated
+/// hyperslab or point selection (in output order; see [`hyperslab_runs`]),
+/// or the error for a hyperslab of the wrong rank or a point outside `dims`
+/// (runs before that point have been emitted).
+fn selection_runs(
+    dims: &[u64],
+    selection: &Selection,
+    emit: &mut dyn FnMut(u64, u64),
+) -> Result<(), FormatError> {
+    match selection {
+        Selection::Hyperslab {
+            start,
+            stride,
+            count,
+            block,
+        } => {
+            let rank = dims.len();
+            if [start.len(), stride.len(), count.len(), block.len()] != [rank; 4] {
+                return Err(FormatError::SelectionOutOfBounds(
+                    "hyperslab rank does not match dataset rank".into(),
+                ));
+            }
+            hyperslab_runs(dims, start, stride, count, block, emit);
+        }
+        Selection::Points(points) => {
+            let strides = strides(dims);
+            let mut coalesce = Coalesce {
+                start: 0,
+                len: 0,
+                emit,
+            };
+            for p in points {
+                if p.len() != dims.len() || p.iter().zip(dims).any(|(c, n)| c >= n) {
+                    return Err(FormatError::SelectionOutOfBounds(
+                        "selection addresses elements outside the dataset".into(),
+                    ));
+                }
+                let at = p
+                    .iter()
+                    .zip(&strides)
+                    .fold(0u64, |acc, (c, s)| acc.wrapping_add(c.wrapping_mul(*s)));
+                coalesce.push(at, 1);
+            }
+            coalesce.flush();
+        }
+        Selection::None | Selection::All => {}
+    }
+    Ok(())
+}
+
+/// One read of [`gather_storage`]: bytes `[start, end)` of the dataset,
+/// which hold the output's bytes up to `out_end` (from where the previous
+/// span's end left off).
+#[derive(Clone, Copy)]
+struct Span {
+    start: usize,
+    end: usize,
+    out_end: usize,
+}
+
+/// [`gather`] of bytes (`T = u8`) from a dataset that is not in memory: the
+/// dataset's `src_len` bytes start at `base` in `file`, which must hold all
+/// of them (the caller checks). Same checks and errors as [`gather`].
+///
+/// The selection's runs are walked twice. The first walk checks them and
+/// plans the reads: runs in increasing order with at most
+/// [`GATHER_GAP_BYTES`] between them are read as one span (the gap is read
+/// and dropped), up to [`GATHER_SPAN_BYTES`] per span. So a strided
+/// selection is a few large reads, not one per element, and nothing is
+/// allocated per run. The spans are fetched batch by batch (one
+/// [`Storage::read_ranges`] call per [`crate::storage::RAW_BATCH_BYTES`])
+/// while the second walk copies each run out of its span.
+pub(crate) fn gather_storage<S: Storage + ?Sized>(
+    file: &S,
+    base: u64,
+    src_len: usize,
+    dims: &[u64],
+    elem_size: usize,
+    selection: &Selection,
+) -> Result<Vec<u8>, FormatError> {
+    if elem_size == 0 {
+        return Err(FormatError::DataSizeMismatch {
+            expected: 1,
+            actual: elem_size,
+        });
+    }
+    let n_elements = match selection {
+        Selection::None => 0,
+        Selection::Hyperslab { count, block, .. } => count
+            .iter()
+            .zip(block)
+            .try_fold(1u64, |acc, (&c, &b)| acc.checked_mul(c.checked_mul(b)?))
+            .ok_or_else(|| FormatError::Overflow("hyperslab count x block overflows".into()))?,
+        Selection::Points(points) => points.len() as u64,
+        Selection::All => {
+            return Err(FormatError::SelectionOutOfBounds(
+                "gather does not take Selection::All".into(),
+            ));
+        }
+    };
+    let out_bytes = crate::chunked_read::checked_byte_len(n_elements, elem_size)?;
+    let outside = || {
+        FormatError::SelectionOutOfBounds("selection addresses elements outside the dataset".into())
+    };
+
+    // First walk: check every run and plan the spans.
+    let mut spans: Vec<Span> = Vec::new();
+    let mut total = 0usize;
+    let mut failed = false;
+    selection_runs(dims, selection, &mut |first: u64, n: u64| {
+        if failed {
+            return;
+        }
+        let range = usize::try_from(first)
+            .ok()
+            .and_then(|f| f.checked_mul(elem_size))
+            .zip(
+                usize::try_from(n)
+                    .ok()
+                    .and_then(|n| n.checked_mul(elem_size)),
+            )
+            .and_then(|(at, len)| Some((at, len, at.checked_add(len)?)));
+        let Some((mut at, mut len)) = range
+            .filter(|&(_, len, end)| end <= src_len && len <= out_bytes - total)
+            .map(|(at, len, _)| (at, len))
+        else {
+            failed = true;
+            return;
+        };
+        while len > 0 {
+            let room = match spans.last_mut() {
+                Some(s)
+                    if at >= s.end
+                        && at - s.end <= GATHER_GAP_BYTES
+                        && at - s.start < GATHER_SPAN_BYTES =>
+                {
+                    let take = len.min(GATHER_SPAN_BYTES - (at - s.start));
+                    s.end = at + take;
+                    s.out_end += take;
+                    take
+                }
+                _ => {
+                    let take = len.min(GATHER_SPAN_BYTES);
+                    spans.push(Span {
+                        start: at,
+                        end: at + take,
+                        out_end: total + take,
+                    });
+                    take
+                }
+            };
+            total += room;
+            at += room;
+            len -= room;
+        }
+    })?;
+    if failed || total != out_bytes {
+        return Err(outside());
+    }
+
+    // The spans' reads, and the batches they are fetched in.
+    let reqs: Vec<ExtentReq> = spans
+        .iter()
+        .map(|s| ExtentReq {
+            addr: base + s.start as u64,
+            len: s.end - s.start,
+            fetch: Some(s.end - s.start),
+        })
+        .collect();
+    let batches = raw_batches(reqs.len(), false, |i| reqs[i].len);
+
+    // Second walk: copy each run out of its span, fetching each batch of
+    // spans when the walk reaches it (and dropping the previous one).
+    let mut out = crate::bulk_alloc::vec_for_bulk(out_bytes);
+    let mut span = 0usize;
+    let mut batch = 0usize;
+    let mut fetched: Option<ExtentBytes<'_>> = None;
+    let mut error: Option<FormatError> = None;
+    selection_runs(dims, selection, &mut |first: u64, n: u64| {
+        if error.is_some() {
+            return;
+        }
+        // Checked by the first walk (these cannot saturate or wrap).
+        let mut at = crate::addr::saturating_usize(first).wrapping_mul(elem_size);
+        let mut len = crate::addr::saturating_usize(n).wrapping_mul(elem_size);
+        while len > 0 {
+            while spans.get(span).is_some_and(|s| s.out_end <= out.len()) {
+                span += 1;
+            }
+            if fetched.is_none() || span >= batches[batch].end {
+                fetched = None;
+                while batches.get(batch).is_some_and(|b| span >= b.end) {
+                    batch += 1;
+                }
+                let (Some(b), Some(_)) = (batches.get(batch).cloned(), spans.get(span)) else {
+                    // The second walk emitted more than the first.
+                    error = Some(outside());
+                    return;
+                };
+                match ExtentBytes::fetch(file, &reqs[b.clone()], b.start) {
+                    Ok(f) => fetched = Some(f),
+                    Err(e) => {
+                        error = Some(e);
+                        return;
+                    }
+                }
+            }
+            let s = spans[span];
+            let take = len.min(s.out_end - out.len());
+            let bytes = match fetched
+                .as_ref()
+                .map(|f| f.get(span, &reqs[span]))
+                .unwrap_or_else(|| Err(outside()))
+            {
+                Ok(b) => b,
+                Err(e) => {
+                    error = Some(e);
+                    return;
+                }
+            };
+            match at
+                .checked_sub(s.start)
+                .and_then(|o| bytes.get(o..o.checked_add(take)?))
+            {
+                Some(b) => out.extend_from_slice(b),
+                None => {
+                    error = Some(outside());
+                    return;
+                }
+            }
+            at += take;
+            len -= take;
+        }
+    })?;
+    if let Some(e) = error {
+        return Err(e);
+    }
+    if out.len() != out_bytes {
+        return Err(outside());
+    }
+    Ok(out)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn runs(dims: &[u64], sel: [&[u64]; 4]) -> Vec<(u64, u64)> {
+        let mut v = Vec::new();
+        hyperslab_runs(dims, sel[0], sel[1], sel[2], sel[3], |s, n| v.push((s, n)));
+        v
+    }
+
+    #[test]
+    fn runs_merge_blocks_and_whole_rows() {
+        // A box: one run per row.
+        assert_eq!(
+            runs(&[4, 10], [&[1, 2], &[1, 1], &[2, 3], &[1, 1]]),
+            vec![(12, 3), (22, 3)]
+        );
+        // Whole rows: one run.
+        assert_eq!(
+            runs(&[4, 10], [&[1, 0], &[1, 1], &[3, 10], &[1, 1]]),
+            vec![(10, 30)]
+        );
+        // stride == block: blocks merge.
+        assert_eq!(
+            runs(&[1, 10], [&[0, 1], &[1, 2], &[1, 4], &[1, 2]]),
+            vec![(1, 8)]
+        );
+        // Strided with blocks along both dimensions.
+        assert_eq!(
+            runs(&[6, 10], [&[0, 1], &[3, 4], &[2, 2], &[2, 2]]),
+            vec![
+                (1, 2),
+                (5, 2),
+                (11, 2),
+                (15, 2),
+                (31, 2),
+                (35, 2),
+                (41, 2),
+                (45, 2)
+            ]
+        );
+        // Empty.
+        assert!(runs(&[4, 10], [&[0, 0], &[1, 1], &[0, 3], &[1, 1]]).is_empty());
+        // Scalar.
+        assert_eq!(runs(&[], [&[], &[], &[], &[]]), vec![(0, 1)]);
+    }
+
+    #[test]
+    fn gather_matches_element_order_and_rejects_out_of_range() {
+        let dims = [3u64, 4];
+        let src: Vec<u8> = (0..12u16).flat_map(|v| v.to_le_bytes()).collect();
+        let sel = Selection::Hyperslab {
+            start: vec![0, 1],
+            stride: vec![2, 2],
+            count: vec![2, 2],
+            block: vec![1, 1],
+        };
+        let got: Vec<u8> = gather(&src, &dims, 2, &sel).unwrap();
+        let want: Vec<u8> = [1u16, 3, 9, 11]
+            .iter()
+            .flat_map(|v| v.to_le_bytes())
+            .collect();
+        assert_eq!(got, want);
+        let pts = Selection::Points(vec![vec![2, 3], vec![0, 0], vec![0, 1]]);
+        let got: Vec<u8> = gather(&src, &dims, 2, &pts).unwrap();
+        let want: Vec<u8> = [11u16, 0, 1].iter().flat_map(|v| v.to_le_bytes()).collect();
+        assert_eq!(got, want);
+        // Past the extent, or a source shorter than the dataset: an error.
+        let bad = Selection::Points(vec![vec![3, 0]]);
+        assert!(gather::<u8>(&src, &dims, 2, &bad).is_err());
+        let past = Selection::Hyperslab {
+            start: vec![2, 0],
+            stride: vec![1, 1],
+            count: vec![2, 4],
+            block: vec![1, 1],
+        };
+        assert!(gather::<u8>(&src, &dims, 2, &past).is_err());
+        assert!(gather::<u8>(&src[..20], &dims, 2, &pts).is_err());
+    }
+}

crates/clawhdf5-format/src/global_heap.rs

@@ -1,9 +1,12 @@
 //! HDF5 Global Heap collection parsing.
 
 #[cfg(not(feature = "std"))]
-use alloc::vec::Vec;
+use alloc::{borrow::Cow, format, string::String, vec::Vec};
+#[cfg(feature = "std")]
+use std::borrow::Cow;
 
 use crate::error::FormatError;
+use crate::storage::{Storage, len_usize, read_exact_at};
 
 /// Magic signature for global heap collections.
 const GCOL_SIGNATURE: [u8; 4] = *b"GCOL";
@@ -28,19 +31,20 @@ pub struct GlobalHeapObject {
     pub data: Vec<u8>,
 }
 
-fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
+/// Checks that `[offset, offset + needed)` ends by `data_len`.
+fn ensure_len(data_len: usize, offset: usize, needed: usize) -> Result<(), FormatError> {
     match offset.checked_add(needed) {
-        Some(end) if end <= data.len() => Ok(()),
+        Some(end) if end <= data_len => Ok(()),
         _ => Err(FormatError::UnexpectedEof {
             expected: offset.saturating_add(needed),
-            available: data.len(),
+            available: data_len,
         }),
     }
 }
 
 fn read_length(data: &[u8], offset: usize, length_size: u8) -> Result<u64, FormatError> {
     let s = length_size as usize;
-    ensure_len(data, offset, s)?;
+    ensure_len(data.len(), offset, s)?;
     let slice = &data[offset..offset + s];
     Ok(match length_size {
         2 => u16::from_le_bytes([slice[0], slice[1]]) as u64,
@@ -52,11 +56,42 @@ fn read_length(data: &[u8], offset: usize, length_size: u8) -> Result<u64, Forma
     })
 }
 
+fn object_overrun_msg(index: u16, size: usize, collection_size: u64) -> String {
+    format!(
+        "global heap object {index} ({size} bytes) runs past the end of its \
+         {collection_size}-byte collection"
+    )
+}
+
 /// Round up to next multiple of 8.
 fn pad8(x: usize) -> usize {
     (x + 7) & !7
 }
 
+/// Where one object of a global heap collection lies in the file, without
+/// its data: see [`GlobalHeapCollection::parse_index`].
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct GlobalHeapObjectRef {
+    /// Object index (1-based; 0 is the free space marker).
+    pub index: u16,
+    /// Reference count.
+    pub reference_count: u16,
+    /// Offset of the object's data in the file data the collection was
+    /// parsed from.
+    pub offset: usize,
+    /// Size of the object's data in bytes.
+    pub size: usize,
+}
+
+/// A global heap collection's objects, located but not copied.
+#[derive(Debug, Clone)]
+pub struct GlobalHeapIndex {
+    /// Total size of this collection including header.
+    pub collection_size: u64,
+    /// The objects, in file order.
+    pub objects: Vec<GlobalHeapObjectRef>,
+}
+
 impl GlobalHeapCollection {
     /// Parse a global heap collection at the given offset in the file data.
     pub fn parse(
@@ -64,71 +99,153 @@ impl GlobalHeapCollection {
         offset: usize,
         length_size: u8,
     ) -> Result<GlobalHeapCollection, FormatError> {
-        // signature(4) + version(1) + reserved(3) + collection_size(length_size)
+        Self::parse_in(file_data, offset as u64, length_size)
-        let header_size = 8 + length_size as usize;
+    }
-        ensure_len(file_data, offset, header_size)?;
 
-        if file_data[offset..offset + 4] != GCOL_SIGNATURE {
+    /// [`Self::parse`] over any [`Storage`]: one read of the header, one of
+    /// the collection.
+    pub fn parse_in<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        length_size: u8,
+    ) -> Result<GlobalHeapCollection, FormatError> {
+        let (bytes, base, index) = Self::read_collection(file, offset, length_size)?;
+        Ok(GlobalHeapCollection {
+            collection_size: index.collection_size,
+            objects: index
+                .objects
+                .iter()
+                .map(|o| GlobalHeapObject {
+                    index: o.index,
+                    reference_count: o.reference_count,
+                    data: bytes[o.offset - base..o.offset - base + o.size].to_vec(),
+                })
+                .collect(),
+        })
+    }
+
+    /// Locate the objects of the global heap collection at `offset` without
+    /// copying their data, so a caller can keep many collections indexed
+    /// for the cost of their object headers.
+    ///
+    /// The collection must lie inside `file_data`, and every object inside
+    /// the collection, as libhdf5 lays them out; an object that runs past
+    /// its collection is an error.
+    pub fn parse_index(
+        file_data: &[u8],
+        offset: usize,
+        length_size: u8,
+    ) -> Result<GlobalHeapIndex, FormatError> {
+        Self::parse_index_in(file_data, offset as u64, length_size)
+    }
+
+    /// [`Self::parse_index`] over any [`Storage`]: one read of the header,
+    /// one of the collection. The object offsets are file offsets.
+    pub fn parse_index_in<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        length_size: u8,
+    ) -> Result<GlobalHeapIndex, FormatError> {
+        Ok(Self::read_collection(file, offset, length_size)?.2)
+    }
+
+    /// Read the collection at `offset` and index its objects: the
+    /// collection's bytes, its offset as a `usize`, and the index (with
+    /// file offsets).
+    pub(crate) fn read_collection<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        length_size: u8,
+    ) -> Result<(Cow<'_, [u8]>, usize, GlobalHeapIndex), FormatError> {
+        let file_len = len_usize(file);
+        // signature(4) + version(1) + reserved(3) + collection_size(length_size),
+        // padded to a multiple of 8 as libhdf5 lays it out (`H5HG_SIZEOF_HDR`).
+        // With 8-byte lengths the padding is 0; with 4-byte lengths it is 4,
+        // and reading without it put every object 4 bytes early.
+        let header_size = pad8(8 + length_size as usize);
+        let header = read_exact_at(file, offset, header_size)?;
+        let offset = usize::try_from(offset).map_err(|_| FormatError::UnexpectedEof {
+            expected: usize::MAX,
+            available: file_len,
+        })?;
+
+        if header[..4] != GCOL_SIGNATURE {
             return Err(FormatError::InvalidGlobalHeapSignature);
         }
 
-        let version = file_data[offset + 4];
+        let version = header[4];
         if version != 1 {
             return Err(FormatError::InvalidGlobalHeapVersion(version));
         }
 
-        let collection_size = read_length(file_data, offset + 8, length_size)?;
+        let collection_size = read_length(&header, 8, length_size)?;
-        let collection_size_usize =
+        let collection_end = usize::try_from(collection_size)
-            usize::try_from(collection_size).map_err(|_| FormatError::UnexpectedEof {
+            .ok()
-                expected: u64::MAX as usize,
+            .and_then(|size| offset.checked_add(size))
-                available: file_data.len(),
-            })?;
-        let collection_end =
-            offset
-                .checked_add(collection_size_usize)
             .ok_or(FormatError::UnexpectedEof {
                 expected: usize::MAX,
-                    available: file_data.len(),
+                available: file_len,
             })?;
+        if collection_end > file_len {
+            return Err(FormatError::UnexpectedEof {
+                expected: collection_end,
+                available: file_len,
+            });
+        }
+        let collection = read_exact_at(file, offset as u64, collection_end - offset)?;
+        // Positions below are file offsets; `file_data(p)` is the byte at `p`.
+        let file_data = |p: usize| collection[p - offset];
 
         let mut pos = offset + header_size;
         let mut objects = Vec::new();
 
         // Parse objects until we hit index 0 (free space) or run out of space
         while pos + 2 <= collection_end {
-            ensure_len(file_data, pos, 2)?;
+            let object_index = u16::from_le_bytes([file_data(pos), file_data(pos + 1)]);
-            let object_index = u16::from_le_bytes([file_data[pos], file_data[pos + 1]]);
 
             if object_index == 0 {
                 // Free space marker — done
                 break;
             }
 
-            // object_index(2) + reference_count(2) + reserved(4) + object_size(length_size)
+            // object_index(2) + reference_count(2) + reserved(4) +
-            let obj_header_size = 8 + length_size as usize;
+            // object_size(length_size), padded to 8 (`H5HG_SIZEOF_OBJHDR`).
-            ensure_len(file_data, pos, obj_header_size)?;
+            let obj_header_size = pad8(8 + length_size as usize);
+            ensure_len(collection_end, pos, obj_header_size)?;
 
-            let reference_count = u16::from_le_bytes([file_data[pos + 2], file_data[pos + 3]]);
+            let reference_count = u16::from_le_bytes([file_data(pos + 2), file_data(pos + 3)]);
-            let object_size = read_length(file_data, pos + 8, length_size)? as usize;
+            let object_size =
+                usize::try_from(read_length(&collection[pos - offset..], 8, length_size)?)
+                    .map_err(|_| FormatError::Overflow("global heap object size".into()))?;
 
             pos += obj_header_size;
-            ensure_len(file_data, pos, object_size)?;
+            if pos
-            let data = file_data[pos..pos + object_size].to_vec();
+                .checked_add(object_size)
+                .is_none_or(|end| end > collection_end)
+            {
+                return Err(FormatError::VlDataError(object_overrun_msg(
+                    object_index,
+                    object_size,
+                    collection_size,
+                )));
+            }
 
-            objects.push(GlobalHeapObject {
+            objects.push(GlobalHeapObjectRef {
                 index: object_index,
                 reference_count,
-                data,
+                offset: pos,
+                size: object_size,
             });
 
             // Advance past data + padding to 8-byte boundary
-            pos += pad8(object_size);
+            pos = pos.saturating_add(pad8(object_size));
         }
 
-        Ok(GlobalHeapCollection {
+        let index = GlobalHeapIndex {
             collection_size,
             objects,
-        })
+        };
+        Ok((collection, offset, index))
     }
 
     /// Get an object by its index.
@@ -149,11 +266,12 @@ mod tests {
         let ls = length_size as usize;
 
         // Calculate total size
-        let header_size = 8 + ls;
+        // libhdf5 pads both headers to a multiple of 8.
+        let header_size = pad8(8 + ls);
         let mut obj_size_total = 0usize;
         for (_, _, data) in objects {
-            let obj_header = 8 + ls;
+            let obj_header = pad8(8 + ls);
-            obj_size_total += obj_header + pad8(data.len());
+            obj_size_total += obj_header + pad8(<[u8]>::len(data));
         }
         // Free space marker (2 bytes for index 0)
         obj_size_total += 2;
@@ -170,6 +288,7 @@ mod tests {
             8 => buf.extend_from_slice(&(collection_size as u64).to_le_bytes()),
             _ => panic!("unsupported length_size"),
         }
+        buf.resize(header_size, 0);
 
         // Objects
         for (index, ref_count, data) in objects {
@@ -177,14 +296,17 @@ mod tests {
             buf.extend_from_slice(&ref_count.to_le_bytes());
             buf.extend_from_slice(&[0u8; 4]); // reserved
             match length_size {
-                4 => buf.extend_from_slice(&(data.len() as u32).to_le_bytes()),
+                // `<[u8]>::len`: with `Storage` in scope `data.len()` on a
-                8 => buf.extend_from_slice(&(data.len() as u64).to_le_bytes()),
+                // `&&[u8]` resolves to `Storage::len` (a `u64`).
+                4 => buf.extend_from_slice(&(<[u8]>::len(data) as u32).to_le_bytes()),
+                8 => buf.extend_from_slice(&(<[u8]>::len(data) as u64).to_le_bytes()),
                 _ => panic!("unsupported"),
             }
+            buf.resize(buf.len() + (pad8(8 + ls) - (8 + ls)), 0);
             buf.extend_from_slice(data);
             // Pad to 8 bytes
-            let padded = pad8(data.len());
+            let padded = pad8(<[u8]>::len(data));
-            buf.resize(buf.len() + (padded - data.len()), 0);
+            buf.resize(buf.len() + (padded - <[u8]>::len(data)), 0);
         }
 
         // Free space marker
@@ -252,4 +374,44 @@ mod tests {
         assert_eq!(coll.objects.len(), 1);
         assert_eq!(coll.objects[0].data, b"test");
     }
+
+    /// Collections, and every truncation of them, index and parse
+    /// identically through a `read_at`-only storage: two reads each.
+    #[test]
+    fn storage_parse_matches_slice_parse() {
+        use crate::storage::CountingStorage;
+        let objs: &[(u16, u16, &[u8])] = &[(1, 1, b"hello"), (2, 3, b"a longer object")];
+        for ls in [4u8, 8] {
+            let coll = build_collection(objs, ls);
+            let mut corrupt = coll.clone();
+            corrupt[8] = 200; // collection size past the end of the file
+            let mut overrun = coll.clone();
+            let size_at = pad8(8 + ls as usize) + 8;
+            overrun[size_at] = 250; // first object runs past the collection
+            for full in [coll, corrupt, overrun] {
+                for at in [0usize, 5] {
+                    for cut in 0..=full.len() {
+                        let mut f = vec![0u8; at];
+                        f.extend_from_slice(&full[..cut]);
+                        let storage = CountingStorage::new(f.clone());
+                        let want = GlobalHeapCollection::parse(&f, at, ls);
+                        let got = GlobalHeapCollection::parse_in(&storage, at as u64, ls);
+                        assert_eq!(format!("{got:?}"), format!("{want:?}"));
+                        let want = GlobalHeapCollection::parse_index(&f, at, ls);
+                        let got = GlobalHeapCollection::parse_index_in(&storage, at as u64, ls);
+                        assert_eq!(format!("{got:?}"), format!("{want:?}"));
+                    }
+                }
+            }
+        }
+        let storage = CountingStorage::new(build_collection(objs, 8));
+        assert_eq!(
+            GlobalHeapCollection::parse_in(&storage, 0, 8)
+                .unwrap()
+                .objects
+                .len(),
+            2
+        );
+        assert_eq!(storage.reads(), 2);
+    }
 }

crates/clawhdf5-format/src/group_v1.rs

@@ -3,11 +3,13 @@
 #[cfg(not(feature = "std"))]
 use alloc::{string::String, vec::Vec};
 
-use crate::btree_v1::collect_symbol_table_nodes;
+use crate::addr::checked_addr;
+use crate::btree_v1::collect_symbol_table_nodes_in;
 use crate::error::FormatError;
 use crate::local_heap::LocalHeap;
 use crate::message_type::MessageType;
 use crate::object_header::ObjectHeader;
+use crate::storage::Storage;
 use crate::symbol_table::{SymbolTableMessage, SymbolTableNode};
 
 /// A resolved group entry (child name + object header address).
@@ -21,23 +23,56 @@ pub struct GroupEntry {
     pub cache_type: u32,
 }
 
-/// Given a SymbolTableMessage, resolve all group children.
+/// Given a SymbolTableMessage, resolve all group children: the group's
+/// listing.
+///
+/// An entry with an empty name fails the listing with
+/// [`FormatError::InvalidLinkName`], as it fails libhdf5's link iteration
+/// (`H5G__ent_to_link`: "invalid link name"). Looking a name up
+/// ([`resolve_path`], and the path resolution in
+/// [`crate::group_v2::resolve_path_any`]) still works in such a group, as it
+/// does in libhdf5.
 pub fn resolve_v1_group_entries(
     file_data: &[u8],
     sym_table_msg: &SymbolTableMessage,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    resolve_v1_group_entries_in(file_data, sym_table_msg, offset_size, length_size)
+}
+
+/// [`resolve_v1_group_entries`] over any [`Storage`].
+pub fn resolve_v1_group_entries_in<S: Storage + ?Sized>(
+    file_data: &S,
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    let entries = v1_group_entries(file_data, sym_table_msg, offset_size, length_size)?;
+    if entries.iter().any(|e| e.name.is_empty()) {
+        return Err(FormatError::InvalidLinkName);
+    }
+    Ok(entries)
+}
+
+/// Every entry of a v1 group, empty names included — for looking a name up,
+/// which never matches an empty name.
+pub(crate) fn v1_group_entries<S: Storage + ?Sized>(
+    file_data: &S,
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<GroupEntry>, FormatError> {
     // Parse local heap
-    let heap = LocalHeap::parse(
+    let heap = LocalHeap::parse_in(
         file_data,
-        sym_table_msg.local_heap_address as usize,
+        checked_addr(sym_table_msg.local_heap_address)?,
         offset_size,
         length_size,
     )?;
 
     // Collect all SNOD addresses from B-tree
-    let snod_addrs = collect_symbol_table_nodes(
+    let snod_addrs = collect_symbol_table_nodes_in(
         file_data,
         sym_table_msg.btree_address,
         offset_size,
@@ -45,10 +80,17 @@ pub fn resolve_v1_group_entries(
     )?;
 
     let mut entries = Vec::new();
+    let mut heap_checked = false;
     for snod_addr in snod_addrs {
-        let snod = SymbolTableNode::parse(file_data, snod_addr as usize, offset_size)?;
+        let snod = SymbolTableNode::parse_in(file_data, checked_addr(snod_addr)?, offset_size)?;
         for entry in &snod.entries {
-            let name = heap.read_string(file_data, entry.link_name_offset)?;
+            // Like libhdf5, look at the heap's free list only once a name is
+            // needed: an empty group with a damaged heap still lists.
+            if !heap_checked {
+                heap.validate_free_list_in(file_data, length_size)?;
+                heap_checked = true;
+            }
+            let name = heap.read_string_in(file_data, entry.link_name_offset)?;
             entries.push(GroupEntry {
                 name,
                 object_header_address: entry.object_header_address,
@@ -73,25 +115,99 @@ pub fn find_v1_soft_link(
     offset_size: u8,
     length_size: u8,
 ) -> Result<Option<String>, FormatError> {
-    let heap = LocalHeap::parse(
+    find_v1_soft_link_in(file_data, sym_table_msg, name, offset_size, length_size)
+}
+
+/// [`find_v1_soft_link`] over any [`Storage`].
+pub fn find_v1_soft_link_in<S: Storage + ?Sized>(
+    file_data: &S,
+    sym_table_msg: &SymbolTableMessage,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Option<String>, FormatError> {
+    let mut found = None;
+    for_each_v1_soft_link(
         file_data,
-        sym_table_msg.local_heap_address as usize,
+        sym_table_msg,
+        offset_size,
+        length_size,
+        |link_name| link_name == name,
+        |_, target| {
+            found = Some(target);
+            false
+        },
+    )?;
+    Ok(found)
+}
+
+/// Every soft link in a v1 group, as `(name, target path)`.
+pub fn v1_soft_links(
+    file_data: &[u8],
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<(String, String)>, FormatError> {
+    v1_soft_links_in(file_data, sym_table_msg, offset_size, length_size)
+}
+
+/// [`v1_soft_links`] over any [`Storage`].
+pub fn v1_soft_links_in<S: Storage + ?Sized>(
+    file_data: &S,
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<(String, String)>, FormatError> {
+    let mut links = Vec::new();
+    for_each_v1_soft_link(
+        file_data,
+        sym_table_msg,
+        offset_size,
+        length_size,
+        |_| true,
+        |name, target| {
+            links.push((String::from(name), target));
+            true
+        },
+    )?;
+    Ok(links)
+}
+
+/// Visit the soft links of a v1 group whose name passes `wanted`, with their
+/// target paths, until `visit` returns false.
+fn for_each_v1_soft_link<S: Storage + ?Sized>(
+    file_data: &S,
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
+    wanted: impl Fn(&str) -> bool,
+    mut visit: impl FnMut(&str, String) -> bool,
+) -> Result<(), FormatError> {
+    let heap = LocalHeap::parse_in(
+        file_data,
+        checked_addr(sym_table_msg.local_heap_address)?,
         offset_size,
         length_size,
     )?;
-    let snod_addrs = collect_symbol_table_nodes(
+    let snod_addrs = collect_symbol_table_nodes_in(
         file_data,
         sym_table_msg.btree_address,
         offset_size,
         length_size,
     )?;
+    let mut heap_checked = false;
     for snod_addr in snod_addrs {
-        let snod = SymbolTableNode::parse(file_data, snod_addr as usize, offset_size)?;
+        let snod = SymbolTableNode::parse_in(file_data, checked_addr(snod_addr)?, offset_size)?;
         for entry in &snod.entries {
             if entry.cache_type != CACHE_TYPE_SOFT_LINK {
                 continue;
             }
-            if heap.read_string(file_data, entry.link_name_offset)? != name {
+            if !heap_checked {
+                heap.validate_free_list_in(file_data, length_size)?;
+                heap_checked = true;
+            }
+            let name = heap.read_string_in(file_data, entry.link_name_offset)?;
+            if !wanted(&name) {
                 continue;
             }
             let value_offset = u32::from_le_bytes([
@@ -100,12 +216,19 @@ pub fn find_v1_soft_link(
                 entry.scratch_pad[2],
                 entry.scratch_pad[3],
             ]);
-            return heap
+            let target = heap.read_string_in(file_data, u64::from(value_offset))?;
-                .read_string(file_data, u64::from(value_offset))
+            if !visit(&name, target) {
-                .map(Some);
+                return Ok(());
             }
         }
-    Ok(None)
+    }
+    Ok(())
+}
+
+/// Whether a v1 symbol-table entry is a soft link (no object header of its
+/// own; its target path is in the local heap).
+pub fn is_v1_soft_link(entry: &GroupEntry) -> bool {
+    entry.cache_type == CACHE_TYPE_SOFT_LINK
 }
 
 /// Extract the SymbolTableMessage from an object header's messages.
@@ -131,6 +254,17 @@ pub fn resolve_path(
     path: &str,
     offset_size: u8,
     length_size: u8,
+) -> Result<u64, FormatError> {
+    resolve_path_in(file_data, root_sym_table, path, offset_size, length_size)
+}
+
+/// [`resolve_path`] over any [`Storage`].
+pub fn resolve_path_in<S: Storage + ?Sized>(
+    file_data: &S,
+    root_sym_table: &SymbolTableMessage,
+    path: &str,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<u64, FormatError> {
     let components: Vec<&str> = path.split('/').filter(|s| !s.is_empty()).collect();
     if components.is_empty() {
@@ -140,8 +274,7 @@ pub fn resolve_path(
     let mut current_sym_table = root_sym_table.clone();
 
     for (i, component) in components.iter().enumerate() {
-        let entries =
+        let entries = v1_group_entries(file_data, &current_sym_table, offset_size, length_size)?;
-            resolve_v1_group_entries(file_data, &current_sym_table, offset_size, length_size)?;
 
         let found = entries.iter().find(|e| e.name == *component);
         match found {
@@ -151,9 +284,9 @@ pub fn resolve_path(
                     return Ok(entry.object_header_address);
                 }
                 // Not last — must be a group, parse its object header to get symbol table
-                let obj_header = ObjectHeader::parse(
+                let obj_header = ObjectHeader::parse_in(
                     file_data,
-                    entry.object_header_address as usize,
+                    checked_addr(entry.object_header_address)?,
                     offset_size,
                     length_size,
                 )?;
@@ -358,6 +491,22 @@ mod tests {
         assert_eq!(entries[1].object_header_address, 0x2000);
     }
 
+    /// cve-2021-46244 `/BAG_root`: a symbol-table entry with an empty name.
+    /// libhdf5 fails the group's listing ("invalid link name"); a lookup of
+    /// the other names still works.
+    #[test]
+    fn empty_entry_name_fails_the_listing_not_a_lookup() {
+        let (file, msg) = build_synthetic_group(&[("", 0x1000, 0), ("elevation", 0x2000, 0)], 8, 8);
+        assert_eq!(
+            resolve_v1_group_entries(&file, &msg, 8, 8).unwrap_err(),
+            FormatError::InvalidLinkName
+        );
+        assert_eq!(
+            resolve_path(&file, &msg, "elevation", 8, 8).unwrap(),
+            0x2000
+        );
+    }
+
     #[test]
     fn resolve_path_single_level() {
         let (file, msg) =

crates/clawhdf5-format/src/group_v2.rs

@@ -6,7 +6,14 @@
 #[cfg(not(feature = "std"))]
 use alloc::{string::String, vec::Vec};
 
-use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+#[cfg(not(feature = "std"))]
+use alloc::collections::BTreeSet;
+#[cfg(feature = "std")]
+use std::collections::BTreeSet;
+
+use crate::addr::checked_addr;
+use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records_in, find_btree_v2_records_in};
+use crate::checksum::jenkins_lookup3;
 use crate::error::FormatError;
 use crate::fractal_heap::FractalHeapHeader;
 use crate::group_v1::{self, GroupEntry};
@@ -14,6 +21,7 @@ use crate::link_info::LinkInfoMessage;
 use crate::link_message::{LinkMessage, LinkTarget};
 use crate::message_type::MessageType;
 use crate::object_header::ObjectHeader;
+use crate::storage::Storage;
 use crate::superblock::Superblock;
 use crate::symbol_table::SymbolTableMessage;
 
@@ -25,6 +33,16 @@ pub fn resolve_v2_group_entries(
     object_header: &ObjectHeader,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    resolve_v2_group_entries_in(file_data, object_header, offset_size, length_size)
+}
+
+/// [`resolve_v2_group_entries`] over any [`Storage`].
+pub fn resolve_v2_group_entries_in<S: Storage + ?Sized>(
+    file_data: &S,
+    object_header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<GroupEntry>, FormatError> {
     // Look for Link Info message to determine storage type
     let link_info = find_link_info(object_header, offset_size)?;
@@ -38,6 +56,24 @@ pub fn resolve_v2_group_entries(
     }
 }
 
+/// First user-defined link type (HDF5 reserves 2-63; 64 is external).
+const FIRST_USER_DEFINED_LINK_TYPE: u8 = 65;
+
+/// Parse a Link message, or `None` for a user-defined link (type 65-255).
+///
+/// A user-defined link's target is only meaningful to the application that
+/// registered its class, so, like libhdf5 without that class, we cannot
+/// follow it. Leaving it out lets the rest of the group be listed and
+/// resolved instead of one such link failing the whole group; reserved
+/// types (2-63) are still an error.
+fn parse_link(data: &[u8], offset_size: u8) -> Result<Option<LinkMessage>, FormatError> {
+    match LinkMessage::parse(data, offset_size) {
+        Ok(link) => Ok(Some(link)),
+        Err(FormatError::InvalidLinkType(t)) if t >= FIRST_USER_DEFINED_LINK_TYPE => Ok(None),
+        Err(e) => Err(e),
+    }
+}
+
 /// Extract link entries from Link messages directly in the object header (compact storage).
 fn resolve_compact_entries(
     object_header: &ObjectHeader,
@@ -46,7 +82,9 @@ fn resolve_compact_entries(
     let mut entries = Vec::new();
     for msg in &object_header.messages {
         if msg.msg_type == MessageType::Link {
-            let link = LinkMessage::parse(&msg.data, offset_size)?;
+            let Some(link) = parse_link(&msg.data, offset_size)? else {
+                continue;
+            };
             if let LinkTarget::Hard {
                 object_header_address,
             } = link.link_target
@@ -64,8 +102,8 @@ fn resolve_compact_entries(
 }
 
 /// Visit every link in dense storage (fractal heap + B-tree v2 name index).
-fn for_each_dense_link(
+fn for_each_dense_link<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     link_info: &LinkInfoMessage,
     fh_addr: u64,
     offset_size: u8,
@@ -73,14 +111,20 @@ fn for_each_dense_link(
     mut visit: impl FnMut(LinkMessage),
 ) -> Result<(), FormatError> {
     // Parse fractal heap
-    let fh = FractalHeapHeader::parse(file_data, fh_addr as usize, offset_size, length_size)?;
+    let fh =
+        FractalHeapHeader::parse_in(file_data, checked_addr(fh_addr)?, offset_size, length_size)?;
 
     // Parse B-tree v2 for name index
     let btree_addr = link_info
         .btree_name_index_address
         .ok_or_else(|| FormatError::PathNotFound(String::from("no B-tree v2 name index")))?;
-    let btree_hdr = BTreeV2Header::parse(file_data, btree_addr as usize, offset_size, length_size)?;
+    let btree_hdr = BTreeV2Header::parse_in(
-    let records = collect_btree_v2_records(file_data, &btree_hdr, offset_size, length_size)?;
+        file_data,
+        checked_addr(btree_addr)?,
+        offset_size,
+        length_size,
+    )?;
+    let records = collect_btree_v2_records_in(file_data, &btree_hdr, offset_size, length_size)?;
 
     for record in &records {
         // For type 5 (name index): hash(4) + heap_id(heap_id_length)
@@ -97,15 +141,17 @@ fn for_each_dense_link(
         let id_bytes = &record.data[id_offset..id_offset + fh.heap_id_length as usize];
 
         // Read managed object from fractal heap
-        let link_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
+        let link_data = fh.read_managed_object_in(file_data, id_bytes, offset_size)?;
-        visit(LinkMessage::parse(&link_data, offset_size)?);
+        if let Some(link) = parse_link(&link_data, offset_size)? {
+            visit(link);
+        }
     }
     Ok(())
 }
 
 /// Resolve entries from dense storage (fractal heap + B-tree v2).
-fn resolve_dense_entries(
+fn resolve_dense_entries<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     link_info: &LinkInfoMessage,
     fh_addr: u64,
     offset_size: u8,
@@ -134,17 +180,16 @@ fn resolve_dense_entries(
     Ok(entries)
 }
 
-/// The soft or external link called `name` in this group, if there is one.
+/// The soft link called `name` in a v1 (symbol table) group, if there is
-/// Hard links are what `resolve_group_entries` returns; this is consulted only
+/// one. Hard links are what `resolve_group_entries` returns; this is
-/// when a path component isn't among them.
+/// consulted only when a path component isn't among them.
-fn find_symbolic_link(
+fn find_v1_symbolic_link<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     object_header: &ObjectHeader,
     name: &str,
     offset_size: u8,
     length_size: u8,
 ) -> Result<Option<LinkTarget>, FormatError> {
-    if is_v1_group(object_header) {
     let Some(sym_msg) = object_header
         .messages
         .iter()
@@ -153,16 +198,56 @@ fn find_symbolic_link(
         return Ok(None);
     };
     let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
-        return group_v1::find_v1_soft_link(file_data, &stm, name, offset_size, length_size)
+    group_v1::find_v1_soft_link_in(file_data, &stm, name, offset_size, length_size)
-            .map(|target| target.map(|target_path| LinkTarget::Soft { target_path }));
+        .map(|target| target.map(|target_path| LinkTarget::Soft { target_path }))
-    }
+}
-    if !is_v2_group(object_header) {
+
-        return Ok(None);
+/// B-tree v2 record type of a dense group's link name index.
-    }
+const LINK_NAME_INDEX: u8 = 5;
-    let is_symbolic = |t: &LinkTarget| !matches!(t, LinkTarget::Hard { .. });
+
+/// The links called `name` in a v2 group (a valid group has at most one),
+/// in storage order: header message order for a compact group, name index
+/// order for a dense one.
+///
+/// In dense storage the link name index (a v2 B-tree of lookup3 name
+/// hashes, record type 5) is descended to the records with the name's hash,
+/// and only their links are read from the heap — O(log n) instead of every
+/// link. libhdf5 orders records with equal hashes by name; all of them are
+/// read and compared here, so that order does not matter. An index of
+/// another type is scanned in full.
+fn links_named<S: Storage + ?Sized>(
+    file_data: &S,
+    object_header: &ObjectHeader,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<LinkMessage>, FormatError> {
+    let mut found = Vec::new();
     let link_info = find_link_info(object_header, offset_size)?;
-    let mut found = None;
+    let Some(fh_addr) = link_info.fractal_heap_address else {
-    if let Some(fh_addr) = link_info.fractal_heap_address {
+        for msg in &object_header.messages {
+            if msg.msg_type == MessageType::Link
+                && let Some(link) = parse_link(&msg.data, offset_size)?
+                && link.name == name
+            {
+                found.push(link);
+            }
+        }
+        return Ok(found);
+    };
+
+    let fh =
+        FractalHeapHeader::parse_in(file_data, checked_addr(fh_addr)?, offset_size, length_size)?;
+    let btree_addr = link_info
+        .btree_name_index_address
+        .ok_or_else(|| FormatError::PathNotFound(String::from("no B-tree v2 name index")))?;
+    let btree_hdr = BTreeV2Header::parse_in(
+        file_data,
+        checked_addr(btree_addr)?,
+        offset_size,
+        length_size,
+    )?;
+    if btree_hdr.tree_type != LINK_NAME_INDEX {
         for_each_dense_link(
             file_data,
             &link_info,
@@ -170,24 +255,176 @@ fn find_symbolic_link(
             offset_size,
             length_size,
             |link| {
-                if link.name == name && is_symbolic(&link.link_target) {
+                if link.name == name {
-                    found = Some(link.link_target);
+                    found.push(link);
                 }
             },
         )?;
-    } else {
+        return Ok(found);
-        for msg in &object_header.messages {
-            if msg.msg_type == MessageType::Link {
-                let link = LinkMessage::parse(&msg.data, offset_size)?;
-                if link.name == name && is_symbolic(&link.link_target) {
-                    found = Some(link.link_target);
     }
+
+    // Record: hash(4) + heap ID.
+    let hash = jenkins_lookup3(name.as_bytes());
+    let records = find_btree_v2_records_in(file_data, &btree_hdr, offset_size, &mut |r| {
+        match r.get(..4) {
+            Some(h) => u32::from_le_bytes([h[0], h[1], h[2], h[3]]).cmp(&hash),
+            // Too short to hold a hash (a corrupt record size): never a match.
+            None => core::cmp::Ordering::Less,
         }
+    })?;
+    let id_len = usize::from(fh.heap_id_length);
+    for record in &records {
+        let Some(id_bytes) = record.data.get(4..4 + id_len) else {
+            continue;
+        };
+        let link_data = fh.read_managed_object_in(file_data, id_bytes, offset_size)?;
+        if let Some(link) = parse_link(&link_data, offset_size)?
+            && link.name == name
+        {
+            found.push(link);
         }
     }
     Ok(found)
 }
 
+/// The link called `name` in a v2 group, if any.
+///
+/// A valid group has at most one; libhdf5 cannot create two. If a damaged
+/// or hand-made group has several, the first wins and the rest are
+/// ignored, whatever their kind and even if the first cannot be followed.
+/// That is libhdf5's rule for a compact group (`H5G__compact_lookup` stops
+/// at the first Link message of that name; h5py then fails to open a
+/// dangling first link although a later one resolves). For a dense group
+/// "first" is first in name index order; libhdf5 binary-searches the index
+/// and may land on another of several exact duplicates. The listing
+/// ([`resolve_group_children`]), [`resolve_child`] and path resolution all
+/// apply this rule, so they agree.
+fn first_link_named<S: Storage + ?Sized>(
+    file_data: &S,
+    object_header: &ObjectHeader,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Option<LinkMessage>, FormatError> {
+    Ok(
+        links_named(file_data, object_header, name, offset_size, length_size)?
+            .into_iter()
+            .next(),
+    )
+}
+
+/// The link [`resolve_path_any`] follows for one path component `name` of
+/// the group with header `object_header`: a hard link (as `Hard`), else a
+/// soft or external link of that name, else `None`. Fails with
+/// `PathNotFound` if the object is not a group.
+fn lookup_link<S: Storage + ?Sized>(
+    file_data: &S,
+    object_header: &ObjectHeader,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Option<LinkTarget>, FormatError> {
+    if is_v1_group(object_header) {
+        let entries = resolve_group_entries(file_data, object_header, offset_size, length_size)?;
+        if let Some(e) = entries
+            .iter()
+            .find(|e| e.name == name && e.object_header_address != u64::MAX)
+        {
+            return Ok(Some(LinkTarget::Hard {
+                object_header_address: e.object_header_address,
+            }));
+        }
+        return find_v1_symbolic_link(file_data, object_header, name, offset_size, length_size);
+    }
+    if !is_v2_group(object_header) {
+        return Err(FormatError::PathNotFound(String::from(
+            "object header is not a group",
+        )));
+    }
+    Ok(
+        first_link_named(file_data, object_header, name, offset_size, length_size)?
+            .map(|link| link.link_target)
+            .filter(|t| {
+                !matches!(
+                    t,
+                    LinkTarget::Hard {
+                        object_header_address: u64::MAX
+                    }
+                )
+            }),
+    )
+}
+
+/// The object header address of the child called `name` of the group at
+/// `group_address`: the address [`resolve_group_children`] lists under that
+/// name, or `PathNotFound` if it lists none.
+///
+/// A dense group's child is found through its link name index (see
+/// [`links_named`]) and only the named link is read and, if it is a soft
+/// link, followed — not every link in the group. A v1 group is listed.
+pub fn resolve_child(
+    file_data: &[u8],
+    superblock: &Superblock,
+    group_address: u64,
+    name: &str,
+) -> Result<u64, FormatError> {
+    resolve_child_core(file_data, superblock, group_address, name)
+}
+
+/// [`resolve_child`] over any [`Storage`]. One with the whole file in memory
+/// is read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
+pub fn resolve_child_in<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
+    name: &str,
+) -> Result<u64, FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => resolve_child(all, superblock, group_address, name),
+        None => resolve_child_core(file_data, superblock, group_address, name),
+    }
+}
+
+fn resolve_child_core<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
+    name: &str,
+) -> Result<u64, FormatError> {
+    let os = superblock.offset_size;
+    let ls = superblock.length_size;
+    let not_found = || FormatError::PathNotFound(String::from(name));
+    let header = ObjectHeader::parse_in(file_data, checked_addr(group_address)?, os, ls)?;
+    if !is_v2_group(&header) || is_v1_group(&header) {
+        return resolve_group_children_in(file_data, superblock, group_address)?
+            .into_iter()
+            .find(|e| e.name == name)
+            .map(|e| e.object_header_address)
+            .ok_or_else(not_found);
+    }
+    // The first link of that name only, as the listing (see
+    // `first_link_named`).
+    match first_link_named(file_data, &header, name, os, ls)?.map(|l| l.link_target) {
+        Some(LinkTarget::Hard {
+            object_header_address,
+        }) => Ok(object_header_address),
+        Some(LinkTarget::Soft { target_path }) => {
+            match resolve_path_from_in(file_data, superblock, group_address, &target_path) {
+                // Left out of the listing: dangling, cyclic, or in another file.
+                Err(
+                    FormatError::PathNotFound(_)
+                    | FormatError::NestingDepthExceeded
+                    | FormatError::ExternalLinkUnsupported { .. },
+                ) => Err(not_found()),
+                other => other,
+            }
+        }
+        Some(LinkTarget::External { .. }) | None => Err(not_found()),
+    }
+}
+
 /// Find and parse the Link Info message from an object header.
 fn find_link_info(
     object_header: &ObjectHeader,
@@ -231,69 +468,230 @@ pub fn resolve_path_any(
     superblock: &Superblock,
     path: &str,
 ) -> Result<u64, FormatError> {
-    resolve_path_following_links(file_data, superblock, path, 0)
+    resolve_path_any_core(file_data, superblock, path)
+}
+
+/// [`resolve_path_any`] over any [`Storage`]. One with the whole file in memory
+/// is read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
+pub fn resolve_path_any_in<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    path: &str,
+) -> Result<u64, FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => resolve_path_any(all, superblock, path),
+        None => resolve_path_any_core(file_data, superblock, path),
+    }
+}
+
+fn resolve_path_any_core<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    path: &str,
+) -> Result<u64, FormatError> {
+    resolve_path_following_links(
+        file_data,
+        superblock,
+        superblock.root_group_address,
+        path,
+        0,
+    )
+}
+
+/// Resolve `path` relative to the group at `group_address` (an absolute path
+/// starts at the root group instead), following soft links. This is how a
+/// relative soft link's target is resolved: from the group holding the link.
+pub fn resolve_path_from(
+    file_data: &[u8],
+    superblock: &Superblock,
+    group_address: u64,
+    path: &str,
+) -> Result<u64, FormatError> {
+    resolve_path_from_in(file_data, superblock, group_address, path)
+}
+
+/// [`resolve_path_from`] over any [`Storage`].
+pub fn resolve_path_from_in<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
+    path: &str,
+) -> Result<u64, FormatError> {
+    let start = if path.starts_with('/') {
+        superblock.root_group_address
+    } else {
+        group_address
+    };
+    resolve_path_following_links(file_data, superblock, start, path, 0)
+}
+
+/// The children of the group at `group_address` that can be opened, as h5py
+/// lists them: hard links, and soft links resolved to the object they point
+/// at (under the soft link's own name). Links that cannot be followed are
+/// left out rather than failing the listing — a dangling or cyclic soft link
+/// (h5py lists its name but cannot open it), an external link (another
+/// file), and a user-defined link. An object header that is not a group has
+/// no children.
+///
+/// Any other error, such as a corrupt structure met while resolving a soft
+/// link, is returned.
+pub fn resolve_group_children(
+    file_data: &[u8],
+    superblock: &Superblock,
+    group_address: u64,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    resolve_group_children_core(file_data, superblock, group_address)
+}
+
+/// [`resolve_group_children`] over any [`Storage`]. One with the whole file in memory
+/// is read as the slice, by code compiled in this crate (see
+/// [`crate::storage`], "Slice entry points").
+#[inline]
+pub fn resolve_group_children_in<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    match file_data.as_contiguous() {
+        Some(all) => resolve_group_children(all, superblock, group_address),
+        None => resolve_group_children_core(file_data, superblock, group_address),
+    }
+}
+
+fn resolve_group_children_core<S: Storage + ?Sized>(
+    file_data: &S,
+    superblock: &Superblock,
+    group_address: u64,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    let os = superblock.offset_size;
+    let ls = superblock.length_size;
+    let header = ObjectHeader::parse_in(file_data, checked_addr(group_address)?, os, ls)?;
+
+    let mut entries = Vec::new();
+    let mut soft = Vec::new();
+    if is_v1_group(&header) {
+        let sym_msg = header
+            .messages
+            .iter()
+            .find(|m| m.msg_type == MessageType::SymbolTable)
+            .ok_or_else(|| FormatError::PathNotFound(String::from("no symbol table message")))?;
+        let stm = SymbolTableMessage::parse(&sym_msg.data, os)?;
+        let all = group_v1::resolve_v1_group_entries_in(file_data, &stm, os, ls)?;
+        if all.iter().any(group_v1::is_v1_soft_link) {
+            soft = group_v1::v1_soft_links_in(file_data, &stm, os, ls)?;
+        }
+        entries.extend(all.into_iter().filter(|e| !group_v1::is_v1_soft_link(e)));
+    } else if is_v2_group(&header) {
+        // Only the first link of each name counts (see `first_link_named`).
+        let mut seen = BTreeSet::new();
+        let mut visit = |link: LinkMessage| {
+            if !seen.insert(link.name.clone()) {
+                return;
+            }
+            match link.link_target {
+                LinkTarget::Hard {
+                    object_header_address,
+                } => entries.push(GroupEntry {
+                    name: link.name,
+                    object_header_address,
+                    cache_type: 0,
+                }),
+                LinkTarget::Soft { target_path } => soft.push((link.name, target_path)),
+                LinkTarget::External { .. } => {}
+            }
+        };
+        let link_info = find_link_info(&header, os)?;
+        if let Some(fh_addr) = link_info.fractal_heap_address {
+            for_each_dense_link(file_data, &link_info, fh_addr, os, ls, visit)?;
+        } else {
+            for msg in &header.messages {
+                if msg.msg_type == MessageType::Link
+                    && let Some(link) = parse_link(&msg.data, os)?
+                {
+                    visit(link);
+                }
+            }
+        }
+    }
+
+    for (name, target) in soft {
+        match resolve_path_from_in(file_data, superblock, group_address, &target) {
+            Ok(object_header_address) => entries.push(GroupEntry {
+                name,
+                object_header_address,
+                cache_type: 0,
+            }),
+            // Dangling, cyclic, or ending in another file: not openable here.
+            Err(
+                FormatError::PathNotFound(_)
+                | FormatError::NestingDepthExceeded
+                | FormatError::ExternalLinkUnsupported { .. },
+            ) => {}
+            Err(e) => return Err(e),
+        }
+    }
+    Ok(entries)
 }
 
 /// Soft links followed while resolving one path. Guards against link cycles
 /// (`a -> b -> a`), which are legal to create.
 const MAX_SOFT_LINK_DEPTH: u8 = 16;
 
-fn resolve_path_following_links(
+/// Walk `path` from the group at `start`, following soft links.
-    file_data: &[u8],
+fn resolve_path_following_links<S: Storage + ?Sized>(
+    file_data: &S,
     superblock: &Superblock,
+    start: u64,
     path: &str,
     depth: u8,
 ) -> Result<u64, FormatError> {
-    let components: Vec<&str> = path.split('/').filter(|s| !s.is_empty()).collect();
+    let components: Vec<&str> = path
+        .split('/')
+        .filter(|s| !s.is_empty() && *s != ".")
+        .collect();
     if components.is_empty() {
-        return Ok(superblock.root_group_address);
+        return Ok(start);
     }
 
     let os = superblock.offset_size;
     let ls = superblock.length_size;
 
-    let root_header =
+    let mut current_addr = start;
-        ObjectHeader::parse(file_data, superblock.root_group_address as usize, os, ls)?;
+    let mut current_header = ObjectHeader::parse_in(file_data, checked_addr(start)?, os, ls)?;
-
-    let mut current_addr = superblock.root_group_address;
-    let mut current_header = root_header;
 
     for (i, component) in components.iter().enumerate() {
-        let entries = resolve_group_entries(file_data, &current_header, os, ls)?;
+        match lookup_link(file_data, &current_header, component, os, ls)? {
-
+            Some(LinkTarget::Hard {
-        let found = entries
+                object_header_address,
-            .iter()
+            }) => {
-            .find(|e| e.name == *component && e.object_header_address != u64::MAX);
-        match found {
-            Some(entry) => {
                 if i == components.len() - 1 {
-                    return Ok(entry.object_header_address);
+                    return Ok(object_header_address);
                 }
-                current_addr = entry.object_header_address;
+                current_addr = object_header_address;
-                current_header = ObjectHeader::parse(file_data, current_addr as usize, os, ls)?;
+                current_header =
+                    ObjectHeader::parse_in(file_data, checked_addr(current_addr)?, os, ls)?;
             }
-            None => {
+            found => {
-                return match find_symbolic_link(file_data, &current_header, component, os, ls)? {
+                return match found {
                     Some(LinkTarget::Soft { target_path }) => {
                         if depth >= MAX_SOFT_LINK_DEPTH {
                             return Err(FormatError::NestingDepthExceeded);
                         }
                         // A relative target is relative to the group holding
                         // the link; then the rest of the original path.
-                        let mut full = String::new();
+                        let from = if target_path.starts_with('/') {
-                        if !target_path.starts_with('/') {
+                            superblock.root_group_address
-                            for parent in &components[..i] {
+                        } else {
-                                full.push('/');
+                            current_addr
-                                full.push_str(parent);
+                        };
-                            }
+                        let mut full = target_path;
-                        }
-                        full.push('/');
-                        full.push_str(&target_path);
                         for rest in &components[i + 1..] {
                             full.push('/');
                             full.push_str(rest);
                         }
-                        resolve_path_following_links(file_data, superblock, &full, depth + 1)
+                        resolve_path_following_links(file_data, superblock, from, &full, depth + 1)
                     }
                     Some(LinkTarget::External {
                         filename,
@@ -312,8 +710,8 @@ fn resolve_path_following_links(
 }
 
 /// Resolve group entries from an object header, auto-detecting v1 vs v2.
-fn resolve_group_entries(
+fn resolve_group_entries<S: Storage + ?Sized>(
-    file_data: &[u8],
+    file_data: &S,
     object_header: &ObjectHeader,
     offset_size: u8,
     length_size: u8,
@@ -326,9 +724,11 @@ fn resolve_group_entries(
             .find(|m| m.msg_type == MessageType::SymbolTable)
             .ok_or_else(|| FormatError::PathNotFound(String::from("no symbol table message")))?;
         let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
-        group_v1::resolve_v1_group_entries(file_data, &stm, offset_size, length_size)
+        // A lookup: an entry with an empty name (which fails a listing) is
+        // skipped by the name comparison, as in libhdf5.
+        group_v1::v1_group_entries(file_data, &stm, offset_size, length_size)
     } else if is_v2_group(object_header) {
-        resolve_v2_group_entries(file_data, object_header, offset_size, length_size)
+        resolve_v2_group_entries_in(file_data, object_header, offset_size, length_size)
     } else {
         Err(FormatError::PathNotFound(String::from(
             "object header is not a group",

crates/clawhdf5-format/src/lane_partition.rs

@@ -112,7 +112,8 @@ pub fn partition(
 
     for idx in 0..num_items {
         let h = fxhash_combine(seed, idx as u64);
-        let lane = (h % num_lanes as u64) as usize;
+        // Below `num_lanes`, so it fits.
+        let lane = crate::addr::saturating_usize(h % num_lanes as u64);
         lanes[lane].push(idx);
     }
 

crates/clawhdf5-format/src/lib.rs

@@ -26,12 +26,13 @@
 //! use clawhdf5_format::{signature, superblock, object_header, group_v2,
 //!     datatype, dataspace, data_layout, data_read, message_type::MessageType};
 //!
-//! let file_data = std::fs::read("output.h5").unwrap();
+//! let bytes = std::fs::read("output.h5").unwrap();
-//! let sig = signature::find_signature(&file_data).unwrap();
+//! // Addresses are relative to the superblock: skip any user block.
-//! let sb  = superblock::Superblock::parse(&file_data, sig).unwrap();
+//! let (_user_block, file_data) = signature::split_user_block(&bytes).unwrap();
-//! let addr = group_v2::resolve_path_any(&file_data, &sb, "data").unwrap();
+//! let sb  = superblock::Superblock::parse(file_data, 0).unwrap();
+//! let addr = group_v2::resolve_path_any(file_data, &sb, "data").unwrap();
 //! let hdr = object_header::ObjectHeader::parse(
-//!     &file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
+//!     file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
 //! ```
 //!
 //! # Features
@@ -42,16 +43,27 @@
 //! | `checksum` | yes | Jenkins lookup3 checksum validation |
 //! | `deflate` | yes | Deflate (gzip) compression via `flate2` |
 //! | `provenance` | yes | SHINES provenance — SHA-256 hashing & verification |
+//! | `lzf` | yes | LZF filter (32000), h5py's `compression="lzf"` |
+//! | `bitshuffle` | no | Bitshuffle filter (32008), none/LZ4/Zstandard |
+//! | `bzip2` | no | bzip2 filter (307) |
+//! | `blosc` | no | Blosc 1 filter (32001) |
+//! | `plugin-filters` | no | The four above |
+//!
+//! Filters are looked up by ID in [`filter_registry`], which also takes
+//! codecs registered at run time for other IDs.
 
 #![cfg_attr(not(feature = "std"), no_std)]
 
 #[cfg(not(feature = "std"))]
 extern crate alloc;
 
+pub mod addr;
 pub mod attribute;
 pub mod attribute_info;
 pub mod btree_v1;
 pub mod btree_v2;
+mod btree_v2_write;
+mod bulk_alloc;
 pub mod checksum;
 pub mod chunk_cache;
 mod chunk_grid;
@@ -70,11 +82,25 @@ pub mod extensible_array;
 pub mod file_writer;
 pub mod fill_value;
 pub mod filter_pipeline;
+pub mod filter_registry;
 pub mod filters;
+#[cfg(any(feature = "bitshuffle", feature = "blosc"))]
+mod filters_bitshuffle;
+#[cfg(feature = "blosc")]
+pub mod filters_blosc;
+#[cfg(feature = "blosc2")]
+pub mod filters_blosc2;
+#[cfg(feature = "bzip2")]
+mod filters_bzip2;
+#[cfg(feature = "lzf")]
+pub mod filters_lzf;
 mod filters_szip;
+#[cfg(feature = "zfp")]
+pub mod filters_zfp;
 pub mod fixed_array;
 pub mod float16;
 pub mod fractal_heap;
+mod gather;
 pub mod global_heap;
 pub mod group_info;
 pub mod group_v1;
@@ -84,6 +110,7 @@ pub mod lane_partition;
 pub mod link_info;
 pub mod link_message;
 pub mod local_heap;
+pub mod lookup_stats;
 pub mod message_type;
 pub mod metadata_cache;
 pub mod metadata_index;
@@ -97,10 +124,24 @@ pub mod property_list;
 pub mod selection;
 pub mod shared_message;
 pub mod signature;
+pub mod storage;
 pub mod superblock;
+pub mod superblock_ext;
 pub mod symbol_table;
+#[cfg(all(
+    test,
+    any(
+        feature = "lzf",
+        feature = "bitshuffle",
+        feature = "bzip2",
+        feature = "blosc"
+    )
+))]
+mod test_fuzz;
 pub mod type_builders;
+pub mod vds;
 pub mod vl_data;
+mod writer_tree;
 
 #[cfg(feature = "provenance")]
 pub mod provenance;

crates/clawhdf5-format/src/link_message.rs

@@ -3,6 +3,7 @@
 #[cfg(not(feature = "std"))]
 use alloc::{string::String, vec::Vec};
 
+use crate::addr::to_usize;
 use crate::datatype::CharacterSet;
 use crate::error::FormatError;
 
@@ -247,7 +248,7 @@ impl LinkMessage {
         };
 
         // Link name length
-        let name_len = read_offset(data, pos, name_size_field_width)? as usize;
+        let name_len = to_usize(read_offset(data, pos, name_size_field_width)?)?;
         pos += name_size_field_width as usize;
 
         // Link name

crates/clawhdf5-format/src/local_heap.rs

@@ -3,7 +3,9 @@
 #[cfg(not(feature = "std"))]
 use alloc::string::String;
 
+use crate::addr::to_usize;
 use crate::error::FormatError;
+use crate::storage::{Storage, len_usize, read_exact_at};
 
 /// Parsed HDF5 Local Heap header.
 #[derive(Debug, Clone)]
@@ -16,21 +18,6 @@ pub struct LocalHeap {
     pub data_segment_address: u64,
 }
 
-/// Checks that `[offset, offset + needed)` fits within `data`, guarding the
-/// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
-fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
-    if offset
-        .checked_add(needed)
-        .is_none_or(|end| end > data.len())
-    {
-        return Err(FormatError::UnexpectedEof {
-            expected: offset.saturating_add(needed),
-            available: data.len(),
-        });
-    }
-    Ok(())
-}
-
 fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
     let s = size as usize;
     if pos.checked_add(s).is_none_or(|end| end > data.len()) {
@@ -50,6 +37,10 @@ fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
     })
 }
 
+/// First read of a name on a backend without the file in memory: most link
+/// names are shorter than this.
+const NAME_READ_START: usize = 64;
+
 impl LocalHeap {
     /// Parse a local heap header at the given offset in the file data.
     pub fn parse(
@@ -57,12 +48,24 @@ impl LocalHeap {
         offset: usize,
         offset_size: u8,
         length_size: u8,
+    ) -> Result<LocalHeap, FormatError> {
+        Self::parse_in(file_data, offset as u64, offset_size, length_size)
+    }
+
+    /// [`Self::parse`] over any [`Storage`]: one read of the header.
+    pub fn parse_in<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        offset_size: u8,
+        length_size: u8,
     ) -> Result<LocalHeap, FormatError> {
         // signature(4) + version(1) + reserved(3) = 8, then length_size*2 + offset_size
         let ls = length_size as usize;
         let os = offset_size as usize;
         let total = 8 + ls * 2 + os;
-        ensure_len(file_data, offset, total)?;
+        let header = read_exact_at(file, offset, total)?;
+        let file_data: &[u8] = &header;
+        let offset = 0usize;
 
         if &file_data[offset..offset + 4] != b"HEAP" {
             return Err(FormatError::InvalidLocalHeapSignature);
@@ -87,45 +90,129 @@ impl LocalHeap {
         })
     }
 
+    /// Walk the free list the way libhdf5 does when it loads a heap's data
+    /// (`H5HL__fl_deserialize`), rejecting a heap whose free list points
+    /// outside the data segment. libhdf5 refuses such a heap ("bad heap free
+    /// list"), and names read from it would be garbage.
+    ///
+    /// libhdf5 only loads a heap when it needs a name from it (an empty
+    /// group's broken heap goes unnoticed), so call this before the first
+    /// [`Self::read_string`], not on parse.
+    ///
+    /// The end of the list is `H5HL_FREE_NULL` (1); an all-ones value (the
+    /// undefined address) is accepted as "no free list" too.
+    pub fn validate_free_list(&self, file_data: &[u8], length_size: u8) -> Result<(), FormatError> {
+        self.validate_free_list_in(file_data, length_size)
+    }
+
+    /// [`Self::validate_free_list`] over any [`Storage`]: two small reads
+    /// per free block.
+    pub fn validate_free_list_in<S: Storage + ?Sized>(
+        &self,
+        file: &S,
+        length_size: u8,
+    ) -> Result<(), FormatError> {
+        const FREE_NULL: u64 = 1;
+        let ls = length_size as usize;
+        let undefined = if ls >= 8 {
+            u64::MAX
+        } else {
+            (1u64 << (8 * ls)) - 1
+        };
+        let size = self.data_segment_size;
+        let seg = self.data_segment_address;
+        let mut next = self.free_list_head_offset;
+        // Each free block holds two lengths, so a list longer than this
+        // revisits a block: a cycle.
+        let max_blocks = size / (2 * ls as u64) + 1;
+        let mut walked = 0u64;
+        while next != FREE_NULL && next != undefined {
+            if next >= size || walked >= max_blocks {
+                return Err(FormatError::InvalidLocalHeapFreeList);
+            }
+            walked += 1;
+            let at = seg
+                .checked_add(next)
+                .and_then(|a| usize::try_from(a).ok())
+                .ok_or(FormatError::InvalidLocalHeapFreeList)?;
+            let block_offset = next;
+            next = read_offset(&read_exact_at(file, at as u64, ls)?, 0, length_size)?;
+            if next == 0 {
+                return Err(FormatError::InvalidLocalHeapFreeList);
+            }
+            let block_size =
+                read_offset(&read_exact_at(file, (at + ls) as u64, ls)?, 0, length_size)?;
+            if block_offset
+                .checked_add(block_size)
+                .is_none_or(|end| end > size)
+            {
+                return Err(FormatError::InvalidLocalHeapFreeList);
+            }
+        }
+        Ok(())
+    }
+
     /// Read a null-terminated string from the heap's data segment at the given byte offset.
     pub fn read_string(&self, file_data: &[u8], string_offset: u64) -> Result<String, FormatError> {
-        let seg_addr = self.data_segment_address as usize;
+        self.read_string_in(file_data, string_offset)
+    }
+
+    /// [`Self::read_string`] over any [`Storage`]: one read of up to 64
+    /// bytes for a short name, more (each four times the last) up to the end
+    /// of the data segment for a longer one.
+    pub fn read_string_in<S: Storage + ?Sized>(
+        &self,
+        file: &S,
+        string_offset: u64,
+    ) -> Result<String, FormatError> {
+        let file_len = len_usize(file);
+        let seg_addr = to_usize(self.data_segment_address)?;
         let str_start =
             seg_addr
-                .checked_add(string_offset as usize)
+                .checked_add(to_usize(string_offset)?)
                 .ok_or(FormatError::Overflow(
                     "local heap seg_addr + string_offset overflow".into(),
                 ))?;
         let seg_end = seg_addr
-            .checked_add(self.data_segment_size as usize)
+            .checked_add(to_usize(self.data_segment_size)?)
             .ok_or(FormatError::Overflow(
                 "local heap seg_addr + data_segment_size overflow".into(),
             ))?;
 
-        if str_start >= file_data.len() || str_start >= seg_end {
+        if str_start >= file_len || str_start >= seg_end {
             return Err(FormatError::UnexpectedEof {
                 expected: str_start + 1,
-                available: file_data.len(),
+                available: file_len,
             });
         }
 
-        // Find null terminator
+        // Find the null terminator, which lies before the end of the data
-        let search_end = seg_end.min(file_data.len());
+        // segment (or of the file). In memory that is one borrowed slice;
-        let mut end = str_start;
+        // otherwise the bytes are read in growing pieces, so a name costs a
-        while end < search_end && file_data[end] != 0 {
+        // read of about its own length, not of the rest of the segment
-            end += 1;
+        // (whose size is an untrusted header field).
+        let search_end = seg_end.min(file_len);
+        let total = search_end - str_start;
+        let mut want = if file.as_contiguous().is_some() {
+            total
+        } else {
+            total.min(NAME_READ_START)
+        };
+        loop {
+            let rest = read_exact_at(file, str_start as u64, want)?;
+            if let Some(len) = rest.iter().position(|&b| b == 0) {
+                let s = core::str::from_utf8(&rest[..len])
+                    .map_err(|_| FormatError::InvalidLocalHeapSignature)?;
+                return Ok(String::from(s));
             }
-
+            if want == total {
-        if end >= search_end {
                 return Err(FormatError::UnexpectedEof {
-                expected: end + 1,
+                    expected: search_end + 1,
                     available: search_end,
                 });
             }
-
+            want = want.saturating_mul(4).min(total);
-        let s = core::str::from_utf8(&file_data[str_start..end])
+        }
-            .map_err(|_| FormatError::InvalidLocalHeapSignature)?;
-        Ok(String::from(s))
     }
 }
 
@@ -162,8 +249,8 @@ mod tests {
         // data_segment_size
         write_val(&mut file, pos, data_seg_size as u64, length_size);
         pos += length_size as usize;
-        // free_list_head_offset
+        // free_list_head_offset: H5HL_FREE_NULL (no free space)
-        write_val(&mut file, pos, 0xFFFFFFFF, length_size);
+        write_val(&mut file, pos, 1, length_size);
         pos += length_size as usize;
         // data_segment_address
         write_val(&mut file, pos, data_seg_offset as u64, offset_size);
@@ -243,6 +330,50 @@ mod tests {
         assert_eq!(s, "test");
     }
 
+    /// Heap with data segment `[a, b, c, 0-padding]` whose free list starts
+    /// at `head` and has one block `(next, size)` at offset 8.
+    fn heap_with_free_block(head: u64, next: u64, size: u64) -> Vec<u8> {
+        let mut file = build_heap_file(0, 100, &["abcdefg"], 8, 8);
+        file.resize(200, 0);
+        write_val(&mut file, 8, 32, 8); // data segment size
+        write_val(&mut file, 16, head, 8);
+        write_val(&mut file, 108, next, 8);
+        write_val(&mut file, 116, size, 8);
+        file
+    }
+
+    #[test]
+    fn free_list_inside_the_segment_is_accepted() {
+        let file = heap_with_free_block(8, 1, 24);
+        let heap = LocalHeap::parse(&file, 0, 8, 8).unwrap();
+        heap.validate_free_list(&file, 8).unwrap();
+        assert_eq!(heap.read_string(&file, 0).unwrap(), "abcdefg");
+        // An all-ones head is "no free list" too.
+        let file = heap_with_free_block(u64::MAX, 0, 0);
+        let heap = LocalHeap::parse(&file, 0, 8, 8).unwrap();
+        assert!(heap.validate_free_list(&file, 8).is_ok());
+    }
+
+    #[test]
+    fn bad_free_list_is_rejected_like_libhdf5() {
+        for (head, next, size, why) in [
+            (40, 1, 8, "head past the segment"),
+            (8, 1, 25, "block runs past the segment"),
+            (8, 0, 8, "next offset of zero"),
+            (8, 8, 8, "cycle"),
+            (8, 999, 8, "next past the segment"),
+        ] {
+            let file = heap_with_free_block(head, next, size);
+            // The header itself parses; the free list is checked on use.
+            let heap = LocalHeap::parse(&file, 0, 8, 8).unwrap();
+            assert_eq!(
+                heap.validate_free_list(&file, 8).unwrap_err(),
+                FormatError::InvalidLocalHeapFreeList,
+                "{why}"
+            );
+        }
+    }
+
     #[test]
     fn invalid_version() {
         let mut file = build_heap_file(0, 100, &["x"], 8, 8);
@@ -250,4 +381,77 @@ mod tests {
         let err = LocalHeap::parse(&file, 0, 8, 8).unwrap_err();
         assert_eq!(err, FormatError::InvalidLocalHeapVersion(1));
     }
+
+    /// Header, free list and strings read identically through a
+    /// `read_at`-only storage, for every truncation of the file.
+    #[test]
+    fn storage_reads_match_slice_reads() {
+        use crate::storage::CountingStorage;
+        let plain = build_heap_file(0, 64, &["", "alpha", "beta"], 8, 8);
+        // A free block of 16 bytes at segment offset 12, ending the list.
+        let mut free = build_heap_file(0, 64, &["", "alpha", "beta", &"x".repeat(20)], 8, 8);
+        free[16..24].copy_from_slice(&12u64.to_le_bytes());
+        free[64 + 12..64 + 20].copy_from_slice(&1u64.to_le_bytes());
+        free[64 + 20..64 + 28].copy_from_slice(&16u64.to_le_bytes());
+        let mut bad_free = free.clone();
+        bad_free[64 + 20..64 + 28].copy_from_slice(&99u64.to_le_bytes());
+        for full in [plain, free, bad_free] {
+            for cut in 0..=full.len() {
+                let f = &full[..cut];
+                let storage = CountingStorage::new(f.to_vec());
+                let want = LocalHeap::parse(f, 0, 8, 8);
+                let got = LocalHeap::parse_in(&storage, 0, 8, 8);
+                assert_eq!(format!("{got:?}"), format!("{want:?}"));
+                let Ok(heap) = want else { continue };
+                assert_eq!(
+                    heap.validate_free_list_in(&storage, 8),
+                    heap.validate_free_list(f, 8)
+                );
+                for off in [0u64, 1, 2, 6, 7, 11, 100] {
+                    assert_eq!(heap.read_string_in(&storage, off), heap.read_string(f, off));
+                }
+            }
+        }
+    }
+
+    /// Names of every length around the first read's size, and one with no
+    /// terminator, read identically through a `read_at`-only storage; a
+    /// short name in a heap whose header claims a huge data segment costs
+    /// one small read, not a read of the rest of the file.
+    #[test]
+    fn long_names_and_hostile_segment_sizes() {
+        use crate::storage::CountingStorage;
+        let names: Vec<String> = [0usize, 1, 63, 64, 65, 255, 256, 257, 1000, 5000]
+            .iter()
+            .map(|&n| "n".repeat(n))
+            .collect();
+        let refs: Vec<&str> = names.iter().map(String::as_str).collect();
+        let mut file = build_heap_file(0, 64, &refs, 8, 8);
+        let heap = LocalHeap::parse(&file, 0, 8, 8).unwrap();
+        let storage = CountingStorage::new(file.clone());
+        let mut off = 0u64;
+        for name in &names {
+            let got = heap.read_string_in(&storage, off);
+            assert_eq!(got, heap.read_string(&file, off));
+            assert_eq!(got.unwrap(), *name);
+            off += name.len() as u64 + 1;
+        }
+        // The last name loses its terminator: both report the same error.
+        let seg_end = 64 + heap.data_segment_size as usize;
+        file[seg_end - 1] = b'n';
+        let storage = CountingStorage::new(file.clone());
+        let last = off - names[names.len() - 1].len() as u64 - 1;
+        let want = heap.read_string(&file, last);
+        assert!(want.is_err());
+        assert_eq!(heap.read_string_in(&storage, last), want);
+
+        // A 64 MiB file whose heap claims a data segment reaching its end.
+        let mut big = build_heap_file(0, 64, &["short", "names"], 8, 8);
+        big.resize(64 << 20, 0);
+        big[8..16].copy_from_slice(&((64u64 << 20) - 64).to_le_bytes());
+        let heap = LocalHeap::parse(&big, 0, 8, 8).unwrap();
+        let storage = CountingStorage::new(big.clone());
+        assert_eq!(heap.read_string_in(&storage, 6).unwrap(), "names");
+        assert_eq!((storage.reads(), storage.bytes_read()), (1, 64));
+    }
 }

crates/clawhdf5-format/src/lookup_stats.rs

@@ -0,0 +1,31 @@
+//! Work counters for tests of lookup cost (feature `lookup-stats`).
+//!
+//! Counts fractal-heap objects read — each is one link or attribute message
+//! decoded out of a dense group or dense attribute storage — so a test can
+//! check that finding one name reads a handful of them, not the whole group.
+//! Per thread, so tests running in parallel do not see each other's reads.
+//! Without the feature the counting compiles to nothing.
+
+#[cfg(feature = "lookup-stats")]
+std::thread_local! {
+    static HEAP_OBJECTS: core::cell::Cell<u64> = const { core::cell::Cell::new(0) };
+}
+
+/// Record one heap object read.
+#[inline(always)]
+pub(crate) fn heap_object_read() {
+    #[cfg(feature = "lookup-stats")]
+    HEAP_OBJECTS.with(|c| c.set(c.get() + 1));
+}
+
+/// Heap objects read on this thread since the last [`reset`].
+#[cfg(feature = "lookup-stats")]
+pub fn heap_objects_read() -> u64 {
+    HEAP_OBJECTS.with(core::cell::Cell::get)
+}
+
+/// Zero this thread's counters.
+#[cfg(feature = "lookup-stats")]
+pub fn reset() {
+    HEAP_OBJECTS.with(|c| c.set(0));
+}

crates/clawhdf5-format/src/object_header_writer.rs

@@ -12,9 +12,16 @@ use crate::message_type::MessageType;
 /// its size truncated to 16 bits produced files libhdf5 refuses.
 pub const MAX_MESSAGE_SIZE: usize = u16::MAX as usize;
 
+/// Object header flags: attribute creation order tracked (each message
+/// then carries a 2-byte creation order) and indexed.
+const OHDR_ATTR_CRT_ORDER_TRACKED: u8 = 0x04;
+const OHDR_ATTR_CRT_ORDER_INDEXED: u8 = 0x08;
+
 /// Writer for v2 object headers with proper checksums.
 pub struct ObjectHeaderWriter {
-    messages: Vec<(MessageType, Vec<u8>, u8)>, // (type, data, msg_flags)
+    messages: Vec<(MessageType, Vec<u8>, u8, u16)>, // (type, data, msg_flags, creation order)
+    /// Attribute creation order tracked and indexed.
+    attr_order: bool,
 }
 
 impl ObjectHeaderWriter {
@@ -22,17 +29,33 @@ impl ObjectHeaderWriter {
     pub fn new() -> Self {
         Self {
             messages: Vec::new(),
+            attr_order: false,
         }
     }
 
+    /// Track and index attribute creation order, as libhdf5 does for an
+    /// object created with `H5P_CRT_ORDER_TRACKED | H5P_CRT_ORDER_INDEXED`
+    /// (h5py's `track_order=True`): the header's flags say so, and every
+    /// message carries a creation order (an attribute's own; 0 for the
+    /// others). libhdf5 reads the setting back from these flags.
+    pub fn track_attr_order(&mut self) {
+        self.attr_order = true;
+    }
+
     /// Add a message to the header with default flags (0).
     pub fn add_message(&mut self, msg_type: MessageType, data: Vec<u8>) {
-        self.messages.push((msg_type, data, 0));
+        self.messages.push((msg_type, data, 0, 0));
     }
 
     /// Add a message with specific flags.
     pub fn add_message_with_flags(&mut self, msg_type: MessageType, data: Vec<u8>, flags: u8) {
-        self.messages.push((msg_type, data, flags));
+        self.messages.push((msg_type, data, flags, 0));
+    }
+
+    /// Add a message with its creation order, which is written only when
+    /// attribute creation order is tracked ([`Self::track_attr_order`]).
+    pub fn add_message_with_order(&mut self, msg_type: MessageType, data: Vec<u8>, order: u16) {
+        self.messages.push((msg_type, data, 0, order));
     }
 
     /// Serialize the complete v2 object header (OHDR + messages + checksum).
@@ -41,10 +64,10 @@ impl ObjectHeaderWriter {
     /// than [`MAX_MESSAGE_SIZE`] (e.g. an attribute over ~64 KiB, which would
     /// need dense attribute storage), rather than writing a corrupt header.
     pub fn serialize(&self) -> Result<Vec<u8>, FormatError> {
-        if let Some((msg_type, data, _)) = self
+        if let Some((msg_type, data, _, _)) = self
             .messages
             .iter()
-            .find(|(_, data, _)| data.len() > MAX_MESSAGE_SIZE)
+            .find(|(_, data, _, _)| data.len() > MAX_MESSAGE_SIZE)
         {
             return Err(FormatError::SerializationError(format!(
                 "{msg_type:?} message is {} bytes; an object header message holds at most \
@@ -52,11 +75,13 @@ impl ObjectHeaderWriter {
                 data.len()
             )));
         }
-        // Calculate total message bytes: each message has type(1) + size(2) + flags(1) + data
+        // Calculate total message bytes: each message has type(1) + size(2) +
+        // flags(1) [+ creation order(2)] + data
+        let msg_header = if self.attr_order { 6 } else { 4 };
         let msg_bytes_total: usize = self
             .messages
             .iter()
-            .map(|(_, data, _)| 4 + data.len())
+            .map(|(_, data, _, _)| msg_header + data.len())
             .sum();
 
         // Determine chunk size field width based on msg_bytes_total
@@ -68,6 +93,12 @@ impl ObjectHeaderWriter {
             (0x02u8, 4)
         };
 
+        let flags = if self.attr_order {
+            flags | OHDR_ATTR_CRT_ORDER_TRACKED | OHDR_ATTR_CRT_ORDER_INDEXED
+        } else {
+            flags
+        };
+
         let mut buf = Vec::new();
 
         // OHDR signature
@@ -85,7 +116,7 @@ impl ObjectHeaderWriter {
         }
 
         // Messages
-        for (msg_type, data, msg_flags) in &self.messages {
+        for (msg_type, data, msg_flags, order) in &self.messages {
             let type_id = msg_type.to_u16();
             assert!(
                 type_id <= 255,
@@ -94,6 +125,9 @@ impl ObjectHeaderWriter {
             buf.push(type_id as u8); // type (1 byte in v2)
             buf.extend_from_slice(&(data.len() as u16).to_le_bytes()); // size (2 bytes)
             buf.push(*msg_flags); // flags
+            if self.attr_order {
+                buf.extend_from_slice(&order.to_le_bytes()); // creation order
+            }
             buf.extend_from_slice(data);
         }
 
@@ -193,6 +227,21 @@ mod tests {
         assert_eq!(hdr.messages.len(), 0);
     }
 
+    #[test]
+    fn tracked_attribute_order_is_in_the_flags_and_every_message() {
+        let mut writer = ObjectHeaderWriter::new();
+        writer.track_attr_order();
+        writer.add_message(MessageType::Dataspace, vec![1, 2, 3, 4]);
+        writer.add_message_with_order(MessageType::Attribute, vec![5, 6], 7);
+        let bytes = writer.serialize().unwrap();
+        assert_eq!(bytes[5] & 0x0C, 0x0C);
+        let hdr = ObjectHeader::parse(&bytes, 0, 8, 8).unwrap();
+        assert_eq!(hdr.messages.len(), 2);
+        assert_eq!(hdr.messages[0].creation_order, Some(0));
+        assert_eq!(hdr.messages[1].creation_order, Some(7));
+        assert_eq!(hdr.messages[1].data, vec![5, 6]);
+    }
+
     #[test]
     fn two_messages_roundtrip() {
         let mut writer = ObjectHeaderWriter::new();

crates/clawhdf5-format/src/parallel_read.rs

@@ -10,8 +10,24 @@
 use crate::chunked_read::ChunkInfo;
 use crate::error::FormatError;
 use crate::filter_pipeline::FilterPipeline;
-use crate::filters::decompress_chunk_masked;
+use crate::filters::decompress_chunk_exact;
 use crate::lane_partition::{self, LaneStats, PartitionStats};
+use crate::storage::{ExtentReq, Storage, for_each_extent_batch};
+
+/// The extents of `chunks`' stored bytes (see
+/// [`crate::chunked_read::chunk_req`]), fetched batch by batch with
+/// [`for_each_extent_batch`] when the file is not in memory (each chunk's
+/// bounds error is reported when that chunk is decoded, as before).
+fn chunk_reqs(
+    chunks: &[ChunkInfo],
+    pipeline: Option<&FilterPipeline>,
+    chunk_total_bytes: usize,
+) -> Vec<ExtentReq> {
+    chunks
+        .iter()
+        .map(|c| crate::chunked_read::chunk_req(c, pipeline, chunk_total_bytes, true))
+        .collect()
+}
 
 /// Threshold: only use parallel decompression when chunk count exceeds this.
 const PARALLEL_THRESHOLD: usize = 4;
@@ -27,6 +43,146 @@ pub fn should_use_parallel(chunk_count: usize) -> bool {
     chunk_count > PARALLEL_THRESHOLD
 }
 
+/// Whether handing a read's chunks to rayon can decode them faster than the
+/// calling thread would alone.
+///
+/// `false` when the pool the work would go to (the current pool inside a
+/// rayon worker, else the global one) has a single thread. Handing work to
+/// that pool is then worse than useless: the caller blocks while the one
+/// worker decodes, and every other thread reading at the same time queues
+/// behind the same worker, so N reader threads decode on one core. (That is
+/// how full reads with `--decode-threads 1` stopped scaling at about 2x in
+/// the `concurrent_read` benchmark.)
+pub fn pool_can_parallelise() -> bool {
+    rayon::current_num_threads() > 1
+}
+
+/// How many rayon workers [`run_with_helpers`] should ask to help with
+/// `items` work items, given that the calling thread works too: the pool's
+/// other threads (all of them when the caller is not one), at most one per
+/// item beyond the caller's first.
+pub(crate) fn helper_count(items: usize) -> usize {
+    let pool = rayon::current_num_threads();
+    // A one-thread pool means "decode on the calling thread" (the setting
+    // benchmarks use to compare with h5py, where each call decodes on its
+    // caller): no helper, so one read never uses two cores.
+    if pool <= 1 {
+        return 0;
+    }
+    let others = if rayon::current_thread_index().is_some() {
+        pool.saturating_sub(1)
+    } else {
+        pool
+    };
+    others.min(items.saturating_sub(1))
+}
+
+/// Run `body` on the calling thread and on up to `helpers` rayon workers at
+/// once, returning when the caller's call has finished and every worker that
+/// started one has too. `body` shares its work out itself (typically by
+/// claiming items from an atomic counter until none are left).
+///
+/// The caller never waits for a worker to *become* free: helpers are queued
+/// on the pool, and one that only gets to run after the caller has finished
+/// returns without calling `body`. So a busy or small pool can only fail to
+/// speed a read up, never hold it back — with `par_iter`, the calling thread
+/// (not a pool worker) handed all the work to the pool and slept, and N
+/// threads reading through a 2-worker pool decoded on 2 cores.
+///
+/// A panic in `body`, on any thread, is resumed on the caller once every
+/// helper that started has stopped.
+pub(crate) fn run_with_helpers(helpers: usize, body: &(dyn Fn() + Sync)) {
+    use std::panic::{AssertUnwindSafe, catch_unwind, resume_unwind};
+    use std::sync::{Arc, Condvar, Mutex, PoisonError};
+
+    if helpers == 0 {
+        body();
+        return;
+    }
+
+    type Body = dyn Fn() + Sync + 'static;
+    struct Shared {
+        /// `body`, its lifetime erased. Only dereferenced by a helper that
+        /// registered in `state` while it was open (see below).
+        body: *const Body,
+        /// (closed, helpers inside `body`).
+        state: Mutex<(bool, usize)>,
+        idle: Condvar,
+        panic: Mutex<Option<Box<dyn core::any::Any + Send>>>,
+    }
+    // SAFETY: `body` points to a `Sync` closure, so calling it from other
+    // threads is allowed; the pointer is only used under the protocol below,
+    // which keeps it from outliving the closure.
+    unsafe impl Send for Shared {}
+    unsafe impl Sync for Shared {}
+
+    fn help(shared: &Shared) {
+        {
+            let mut state = shared.state.lock().unwrap_or_else(PoisonError::into_inner);
+            if state.0 {
+                return;
+            }
+            state.1 += 1;
+        }
+        // SAFETY: registered while open, so the caller of `run_with_helpers`
+        // is still inside it (it closes, then waits until no helper is
+        // registered, before returning), and `body` is alive.
+        let body = unsafe { &*shared.body };
+        if let Err(payload) = catch_unwind(AssertUnwindSafe(body)) {
+            shared
+                .panic
+                .lock()
+                .unwrap_or_else(PoisonError::into_inner)
+                .get_or_insert(payload);
+        }
+        let mut state = shared.state.lock().unwrap_or_else(PoisonError::into_inner);
+        state.1 -= 1;
+        if state.1 == 0 {
+            shared.idle.notify_all();
+        }
+    }
+
+    let body_ptr: *const (dyn Fn() + Sync + '_) = body;
+    // SAFETY: only the lifetime changes (same fat-pointer layout). The
+    // pointer is dereferenced only while this function is running: see
+    // `help` and the wait below.
+    let body_ptr: *const Body = unsafe { core::mem::transmute(body_ptr) };
+    let shared = Arc::new(Shared {
+        body: body_ptr,
+        state: Mutex::new((false, 0)),
+        idle: Condvar::new(),
+        panic: Mutex::new(None),
+    });
+    for _ in 0..helpers {
+        let shared = Arc::clone(&shared);
+        rayon::spawn(move || help(&shared));
+    }
+    let caller = catch_unwind(AssertUnwindSafe(body));
+    {
+        // Close, then wait for the helpers inside `body`; later ones return
+        // at once. This must happen even if `body` panicked on this thread.
+        let mut state = shared.state.lock().unwrap_or_else(PoisonError::into_inner);
+        state.0 = true;
+        while state.1 > 0 {
+            state = shared
+                .idle
+                .wait(state)
+                .unwrap_or_else(PoisonError::into_inner);
+        }
+    }
+    if let Err(payload) = caller {
+        resume_unwind(payload);
+    }
+    let helper_panic = shared
+        .panic
+        .lock()
+        .unwrap_or_else(PoisonError::into_inner)
+        .take();
+    if let Some(payload) = helper_panic {
+        resume_unwind(payload);
+    }
+}
+
 /// Decompress chunks in parallel using lane-partitioned assignment.
 ///
 /// Instead of naive `par_iter`, chunks are deterministically assigned to lanes
@@ -49,6 +205,27 @@ pub fn decompress_chunks_lane_partitioned(
     element_size: u32,
     seed: u64,
     num_lanes: Option<usize>,
+) -> Result<(Vec<Vec<u8>>, PartitionStats), FormatError> {
+    decompress_chunks_lane_partitioned_in(
+        file_data,
+        chunks,
+        pipeline,
+        chunk_total_bytes,
+        element_size,
+        seed,
+        num_lanes,
+    )
+}
+
+/// [`decompress_chunks_lane_partitioned`] over any [`Storage`].
+pub fn decompress_chunks_lane_partitioned_in<S: Storage + ?Sized>(
+    file_data: &S,
+    chunks: &[ChunkInfo],
+    pipeline: &FilterPipeline,
+    chunk_total_bytes: usize,
+    element_size: u32,
+    seed: u64,
+    num_lanes: Option<usize>,
 ) -> Result<(Vec<Vec<u8>>, PartitionStats), FormatError> {
     use rayon::prelude::*;
 
@@ -58,38 +235,36 @@ pub fn decompress_chunks_lane_partitioned(
             .unwrap_or(1)
     });
 
-    let assignments = lane_partition::partition_chunks(chunks.len(), lanes, seed);
+    let reqs = chunk_reqs(chunks, Some(pipeline), chunk_total_bytes);
-    let num_lanes = assignments.len();
+    let mut ordered: Vec<Vec<u8>> = Vec::with_capacity(chunks.len());
-
+    let mut partition_stats = PartitionStats::new(0);
+    partition_stats.total_chunks = chunks.len();
+    // Each batch of fetched chunks is partitioned into lanes and decoded
+    // before the next batch is fetched (with the file in memory there is
+    // one batch: all the chunks).
+    for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
+        let assignments = lane_partition::partition_chunks(batch.len(), lanes, seed);
         // Each lane processes its assigned chunks and returns results + stats.
-    let lane_results: Result<Vec<(Vec<DecompressedChunk>, LaneStats)>, FormatError> = assignments
+        let lane_results: Result<Vec<(Vec<DecompressedChunk>, LaneStats)>, FormatError> =
+            assignments
                 .into_par_iter()
                 .map(|indices| {
                     let mut results = Vec::with_capacity(indices.len());
                     let mut stats = LaneStats::default();
 
-            for &index in &indices {
+                    for &local in &indices {
+                        let index = batch.start + local;
                         let chunk_info = &chunks[index];
-                let c_addr = chunk_info.address as usize;
                         let size = chunk_info.chunk_size as usize;
+                        let raw_chunk = raw_bytes.get(index, &reqs[index])?;
 
-                if c_addr
+                        let decompressed = decompress_chunk_exact(
-                    .checked_add(size)
-                    .is_none_or(|end| end > file_data.len())
-                {
-                    return Err(FormatError::UnexpectedEof {
-                        expected: c_addr.saturating_add(size),
-                        available: file_data.len(),
-                    });
-                }
-                let raw_chunk = &file_data[c_addr..c_addr + size];
-
-                let decompressed = decompress_chunk_masked(
                             raw_chunk,
                             pipeline,
                             chunk_total_bytes,
                             element_size,
                             chunk_info.filter_mask,
+                            &chunk_info.offsets,
                         )?;
 
                         stats.chunks_processed += 1;
@@ -105,14 +280,19 @@ pub fn decompress_chunks_lane_partitioned(
                     Ok((results, stats))
                 })
                 .collect();
-
         let lane_results = lane_results?;
 
         // Aggregate stats
-    let mut partition_stats = PartitionStats::new(num_lanes);
+        if partition_stats.per_lane.len() < lane_results.len() {
-    partition_stats.total_chunks = chunks.len();
+            partition_stats
-    for (lane_idx, (_, stats)) in lane_results.iter().enumerate() {
+                .per_lane
-        partition_stats.per_lane[lane_idx] = stats.clone();
+                .resize_with(lane_results.len(), LaneStats::default);
+            partition_stats.num_lanes = lane_results.len();
+        }
+        for (lane, (_, stats)) in partition_stats.per_lane.iter_mut().zip(&lane_results) {
+            lane.chunks_processed += stats.chunks_processed;
+            lane.compressed_bytes += stats.compressed_bytes;
+            lane.decompressed_bytes += stats.decompressed_bytes;
         }
 
         // Flatten and sort by original index to restore order
@@ -121,8 +301,9 @@ pub fn decompress_chunks_lane_partitioned(
             .flat_map(|(chunks, _)| chunks)
             .collect();
         all_chunks.sort_by_key(|dc| dc.index);
-
+        ordered.extend(all_chunks.into_iter().map(|dc| dc.data));
-    let ordered = all_chunks.into_iter().map(|dc| dc.data).collect();
+        Ok(())
+    })?;
     Ok((ordered, partition_stats))
 }
 
@@ -140,32 +321,37 @@ pub fn decompress_chunks_parallel(
     pipeline: &FilterPipeline,
     chunk_total_bytes: usize,
     element_size: u32,
+) -> Result<Vec<Vec<u8>>, FormatError> {
+    decompress_chunks_parallel_in(file_data, chunks, pipeline, chunk_total_bytes, element_size)
+}
+
+/// [`decompress_chunks_parallel`] over any [`Storage`].
+pub fn decompress_chunks_parallel_in<S: Storage + ?Sized>(
+    file_data: &S,
+    chunks: &[ChunkInfo],
+    pipeline: &FilterPipeline,
+    chunk_total_bytes: usize,
+    element_size: u32,
 ) -> Result<Vec<Vec<u8>>, FormatError> {
     use rayon::prelude::*;
 
-    let results: Result<Vec<DecompressedChunk>, FormatError> = chunks
+    let reqs = chunk_reqs(chunks, Some(pipeline), chunk_total_bytes);
-        .par_iter()
+    let mut ordered: Vec<Vec<u8>> = Vec::with_capacity(chunks.len());
-        .enumerate()
+    for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
-        .map(|(index, chunk_info)| {
+        let results: Result<Vec<DecompressedChunk>, FormatError> = batch
-            let c_addr = chunk_info.address as usize;
+            .clone()
-            let size = chunk_info.chunk_size as usize;
+            .into_par_iter()
-            if c_addr
+            .map(|index| {
-                .checked_add(size)
+                let chunk_info = &chunks[index];
-                .is_none_or(|end| end > file_data.len())
+                let raw_chunk = raw_bytes.get(index, &reqs[index])?;
-            {
-                return Err(FormatError::UnexpectedEof {
-                    expected: c_addr.saturating_add(size),
-                    available: file_data.len(),
-                });
-            }
-            let raw_chunk = &file_data[c_addr..c_addr + size];
 
-            let decompressed = decompress_chunk_masked(
+                let decompressed = decompress_chunk_exact(
                     raw_chunk,
                     pipeline,
                     chunk_total_bytes,
                     element_size,
                     chunk_info.filter_mask,
+                    &chunk_info.offsets,
                 )?;
 
                 Ok(DecompressedChunk {
@@ -177,7 +363,10 @@ pub fn decompress_chunks_parallel(
 
         let mut result_vec = results?;
         result_vec.sort_by_key(|dc| dc.index);
-    Ok(result_vec.into_iter().map(|dc| dc.data).collect())
+        ordered.extend(result_vec.into_iter().map(|dc| dc.data));
+        Ok(())
+    })?;
+    Ok(ordered)
 }
 
 /// Decompress chunks sequentially (fallback when parallel is not warranted).
@@ -188,33 +377,150 @@ pub fn decompress_chunks_sequential(
     chunk_total_bytes: usize,
     element_size: u32,
 ) -> Result<Vec<Vec<u8>>, FormatError> {
+    decompress_chunks_sequential_in(file_data, chunks, pipeline, chunk_total_bytes, element_size)
+}
+
+/// [`decompress_chunks_sequential`] over any [`Storage`].
+pub fn decompress_chunks_sequential_in<S: Storage + ?Sized>(
+    file_data: &S,
+    chunks: &[ChunkInfo],
+    pipeline: Option<&FilterPipeline>,
+    chunk_total_bytes: usize,
+    element_size: u32,
+) -> Result<Vec<Vec<u8>>, FormatError> {
+    let reqs = chunk_reqs(chunks, pipeline, chunk_total_bytes);
     let mut result = Vec::with_capacity(chunks.len());
-    for chunk_info in chunks {
+    for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
-        let c_addr = chunk_info.address as usize;
+        for i in batch {
-        let size = chunk_info.chunk_size as usize;
+            let chunk_info = &chunks[i];
-        if c_addr
+            let raw_chunk = raw_bytes.get(i, &reqs[i])?;
-            .checked_add(size)
-            .is_none_or(|end| end > file_data.len())
-        {
-            return Err(FormatError::UnexpectedEof {
-                expected: c_addr.saturating_add(size),
-                available: file_data.len(),
-            });
-        }
-        let raw_chunk = &file_data[c_addr..c_addr + size];
 
             let decompressed = if let Some(pl) = pipeline {
-            decompress_chunk_masked(
+                decompress_chunk_exact(
                     raw_chunk,
                     pl,
                     chunk_total_bytes,
                     element_size,
                     chunk_info.filter_mask,
+                    &chunk_info.offsets,
                 )?
             } else {
                 raw_chunk.to_vec()
             };
             result.push(decompressed);
         }
+        Ok(())
+    })?;
     Ok(result)
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::filter_pipeline::{FILTER_SHUFFLE, FilterDescription};
+
+    /// Eight shuffled 32-byte chunks; chunk 5 is stored short when `short`.
+    fn chunks(short: bool) -> (Vec<u8>, Vec<ChunkInfo>) {
+        let mut file = Vec::new();
+        let mut infos = Vec::new();
+        for i in 0..8u64 {
+            let len = if short && i == 5 { 16 } else { 32 };
+            infos.push(ChunkInfo {
+                chunk_size: len as u32,
+                filter_mask: 0,
+                offsets: vec![i * 8],
+                address: file.len() as u64,
+            });
+            file.extend(core::iter::repeat_n(i as u8, len));
+        }
+        (file, infos)
+    }
+
+    /// Every item is processed exactly once, whatever mix of caller and
+    /// helpers ends up doing it.
+    #[test]
+    fn run_with_helpers_shares_all_work() {
+        use core::sync::atomic::{AtomicUsize, Ordering};
+        for helpers in [0, 1, 3, 16] {
+            let n = 1000;
+            let next = AtomicUsize::new(0);
+            let done: Vec<AtomicUsize> = (0..n).map(|_| AtomicUsize::new(0)).collect();
+            run_with_helpers(helpers, &|| {
+                loop {
+                    let i = next.fetch_add(1, Ordering::Relaxed);
+                    if i >= n {
+                        break;
+                    }
+                    done[i].fetch_add(1, Ordering::Relaxed);
+                }
+            });
+            assert!(done.iter().all(|d| d.load(Ordering::Relaxed) == 1));
+        }
+    }
+
+    /// A panic in the shared body reaches the caller whichever thread it
+    /// happened on, and only after the helpers inside the body have left it
+    /// (they borrow the caller's stack).
+    #[test]
+    fn run_with_helpers_propagates_panics() {
+        use core::sync::atomic::{AtomicUsize, Ordering};
+        use std::panic::{AssertUnwindSafe, catch_unwind};
+        let caller = std::thread::current().id();
+        for panic_on_caller in [true, false] {
+            let inside = AtomicUsize::new(0);
+            let calls = AtomicUsize::new(0);
+            let result = catch_unwind(AssertUnwindSafe(|| {
+                run_with_helpers(4, &|| {
+                    inside.fetch_add(1, Ordering::SeqCst);
+                    calls.fetch_add(1, Ordering::SeqCst);
+                    let on_caller = std::thread::current().id() == caller;
+                    std::thread::sleep(std::time::Duration::from_millis(20));
+                    inside.fetch_sub(1, Ordering::SeqCst);
+                    if on_caller == panic_on_caller {
+                        panic!("boom");
+                    }
+                });
+            }));
+            // A helper may never have run (the pool was slow to start it),
+            // in which case nothing panicked when `panic_on_caller` is false.
+            if panic_on_caller || calls.load(Ordering::SeqCst) > 1 {
+                assert!(result.is_err());
+            }
+            assert_eq!(inside.load(Ordering::SeqCst), 0);
+        }
+    }
+
+    /// Every parallel decoder refuses a chunk that decodes short, naming it.
+    #[test]
+    fn short_decoded_chunk_is_an_error() {
+        let pipeline = FilterPipeline {
+            version: 2,
+            filters: vec![FilterDescription {
+                filter_id: FILTER_SHUFFLE,
+                name: None,
+                flags: 0,
+                client_data: vec![4],
+            }],
+        };
+        let (file, good) = chunks(false);
+        assert_eq!(
+            decompress_chunks_parallel(&file, &good, &pipeline, 32, 4).unwrap()[5],
+            [5u8; 32]
+        );
+        let (file, bad) = chunks(true);
+        let errs = [
+            decompress_chunks_lane_partitioned(&file, &bad, &pipeline, 32, 4, 1, Some(3))
+                .map(|_| ())
+                .unwrap_err(),
+            decompress_chunks_parallel(&file, &bad, &pipeline, 32, 4)
+                .map(|_| ())
+                .unwrap_err(),
+            decompress_chunks_sequential(&file, &bad, Some(&pipeline), 32, 4)
+                .map(|_| ())
+                .unwrap_err(),
+        ];
+        for e in errs {
+            assert!(e.to_string().contains("[40]"), "{e}");
+        }
+    }
+}

crates/clawhdf5-format/src/partial_read.rs

@@ -3,11 +3,13 @@
 //!
 //! [`crate::data_read::read_raw_data_selection`] used to decode the *entire*
 //! dataset and then pick elements out of it, so reading a 64x64 window of a
-//! large dataset took about as long as reading all of it. Here the selection's
+//! large dataset took about as long as reading all of it. A contiguous
-//! bounding box is materialised instead — only the rows of a contiguous
+//! dataset's selection is now copied straight out of the file, one `memcpy`
-//! dataset, or only the chunks, that overlap it — and the existing extractor
+//! per contiguous run of selected elements (`crate::gather`). For chunked
-//! runs over that small buffer with the selection translated to the box's
+//! data the selection's bounding box is materialised — only the chunks that
-//! origin. Extraction semantics are therefore exactly the full-read ones.
+//! overlap it — and the extractor runs over that small buffer with the
+//! selection translated to the box's origin. Extraction semantics are
+//! therefore exactly the full-read ones.
 
 #[cfg(not(feature = "std"))]
 use alloc::string as alloc_or_std;
@@ -16,14 +18,15 @@ use alloc::{format, vec, vec::Vec};
 #[cfg(feature = "std")]
 use std::string as alloc_or_std;
 
-use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks};
+use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks_for_read_in};
 use crate::data_layout::DataLayout;
 use crate::data_read::extract_selection_from_buffer;
 use crate::dataspace::Dataspace;
 use crate::error::FormatError;
 use crate::filter_pipeline::FilterPipeline;
-use crate::filters::{all_filters_skipped, decompress_chunk_masked};
+use crate::filters::{all_filters_skipped, decompress_chunk_exact_with};
 use crate::selection::Selection;
+use crate::storage::{ExtentReq, Storage, for_each_extent_batch};
 
 /// The smallest axis-aligned box containing every selected element, as
 /// `(start, extent)` per dimension. `None` when there is nothing to gain or
@@ -201,7 +204,12 @@ fn copy_overlap(
     };
     let (src_strides, out_strides) = (strides(src_shape), strides(box_extent));
     let last = rank - 1;
-    let run = ((hi[last] - lo[last]) as usize) * elem_size;
+    // Byte offsets into the in-memory buffers; one that does not fit `usize`
+    // (a 32-bit target) is out of both buffers, like one past their ends.
+    let bytes = |elements: u64| usize::try_from(elements).ok()?.checked_mul(elem_size);
+    let Some(run) = bytes(hi[last] - lo[last]) else {
+        return;
+    };
 
     let mut idx = lo.clone();
     loop {
@@ -211,8 +219,12 @@ fn copy_overlap(
         let out_at: u64 = (0..rank)
             .map(|d| (idx[d] - box_start[d]) * out_strides[d])
             .sum();
-        let (s, o) = (src_at as usize * elem_size, out_at as usize * elem_size);
+        if let (Some(s), Some(o)) = (bytes(src_at), bytes(out_at))
-        if let (Some(from), Some(to)) = (src.get(s..s + run), out.get_mut(o..o + run)) {
+            && let (Some(from), Some(to)) = (
+                src.get(s..s.saturating_add(run)),
+                out.get_mut(o..o.saturating_add(run)),
+            )
+        {
             to.copy_from_slice(from);
         }
         // Advance over every dimension but the last.
@@ -245,15 +257,81 @@ pub fn read_selection(
     offset_size: u8,
     length_size: u8,
     selection: &Selection,
+) -> Result<Option<Vec<u8>>, FormatError> {
+    read_selection_in(
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        pipeline,
+        offset_size,
+        length_size,
+        selection,
+    )
+}
+
+/// [`read_selection`] over any [`Storage`].
+#[allow(clippy::too_many_arguments)]
+pub fn read_selection_in<S: Storage + ?Sized>(
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    selection: &Selection,
 ) -> Result<Option<Vec<u8>>, FormatError> {
     let dims = &dataspace.dimensions;
     if dims.is_empty() || elem_size == 0 {
         return Ok(None);
     }
+    let total = dataspace.checked_num_elements()?;
+    // Contiguous data is addressable in place: copy the selection's runs
+    // straight out of it, whatever fraction of the dataset it covers, with no
+    // intermediate box (and no full copy for a large selection).
+    if let (
+        DataLayout::Contiguous {
+            address: Some(address),
+            ..
+        },
+        Selection::Hyperslab { .. } | Selection::Points(_),
+    ) = (layout, selection)
+    {
+        validate(selection, dims)?;
+        let base = usize::try_from(*address)
+            .map_err(|_| FormatError::Overflow("data address exceeds usize".into()))?;
+        let file_len = crate::storage::len_usize(file_data);
+        let eof = FormatError::UnexpectedEof {
+            expected: base,
+            available: file_len,
+        };
+        if let Some(all) = file_data.as_contiguous() {
+            let data = all
+                .get(base..)
+                .and_then(|d| d.get(..checked_byte_len(total, elem_size).ok()?))
+                .ok_or(eof)?;
+            return crate::gather::gather::<u8>(data, dims, elem_size, selection).map(Some);
+        }
+        // Not in memory: the same bounds check, then only the selected runs
+        // are read.
+        let len = checked_byte_len(total, elem_size)
+            .ok()
+            .filter(|&len| base <= file_len && len <= file_len - base)
+            .ok_or(eof)?;
+        return crate::gather::gather_storage(
+            file_data,
+            base as u64,
+            len,
+            dims,
+            elem_size,
+            selection,
+        )
+        .map(Some);
+    }
     let Some((box_start, box_extent)) = bounding_box(selection, dims) else {
         return Ok(None);
     };
-    let total = dataspace.checked_num_elements()?;
     let box_elements = box_extent
         .iter()
         .try_fold(1u64, |acc, &e| acc.checked_mul(e))
@@ -265,79 +343,65 @@ pub fn read_selection(
     let mut boxed = alloc_output(checked_byte_len(box_elements, elem_size)?)?;
 
     match layout {
-        DataLayout::Contiguous {
-            address: Some(address),
-            ..
-        } => {
-            let base = usize::try_from(*address)
-                .map_err(|_| FormatError::Overflow("data address exceeds usize".into()))?;
-            let data = file_data
-                .get(base..)
-                .and_then(|d| d.get(..checked_byte_len(total, elem_size).ok()?))
-                .ok_or(FormatError::UnexpectedEof {
-                    expected: base,
-                    available: file_data.len(),
-                })?;
-            let origin = vec![0u64; dims.len()];
-            copy_overlap(
-                data,
-                &origin,
-                dims,
-                &mut boxed,
-                &box_start,
-                &box_extent,
-                elem_size,
-            );
-        }
         DataLayout::Chunked {
             btree_address: Some(_),
             ..
         } => {
-            let (chunks, chunk_dims) = list_chunks(
+            let (chunks, chunk_dims) = list_chunks_for_read_in(
                 file_data,
                 layout,
                 dataspace,
                 elem_size,
+                pipeline,
                 offset_size,
                 length_size,
             )?;
             let rank = dims.len();
             let chunk_shape: Vec<u64> = chunk_dims.iter().map(|&d| d as u64).collect();
             let chunk_bytes = crate::chunked_read::checked_chunk_byte_len(&chunk_dims, elem_size)?;
-            for chunk in &chunks {
+            // The chunks overlapping the box, in index order.
+            let wanted: Vec<&crate::chunked_read::ChunkInfo> = chunks
+                .iter()
+                .filter(|chunk| {
                     if chunk.offsets.len() < rank || chunk.address == u64::MAX {
-                    continue;
+                        return false;
                     }
                     let origin = &chunk.offsets[..rank];
-                let overlaps = (0..rank).all(|d| {
+                    (0..rank).all(|d| {
                         origin[d] < box_start[d] + box_extent[d]
                             && origin[d].saturating_add(chunk_shape[d]) > box_start[d]
-                });
+                    })
-                if !overlaps {
+                })
-                    continue;
+                .collect();
-                }
+            // Their stored bytes, batch by batch when the file is not in
-                let at = usize::try_from(chunk.address)
+            // memory; each batch's chunks are decoded into this thread's
-                    .map_err(|_| FormatError::Overflow("chunk address exceeds usize".into()))?;
+            // reusable buffers before the next batch is fetched.
-                let raw = at
+            let reqs: Vec<ExtentReq> = wanted
-                    .checked_add(chunk.chunk_size as usize)
+                .iter()
-                    .and_then(|end| file_data.get(at..end))
+                .map(|c| crate::chunked_read::chunk_req(c, pipeline, chunk_bytes, true))
-                    .ok_or(FormatError::UnexpectedEof {
+                .collect();
-                        expected: at.saturating_add(chunk.chunk_size as usize),
+            for_each_extent_batch(file_data, &reqs, |batch, raw_bytes| {
-                        available: file_data.len(),
+                crate::chunked_read::with_scratch(|scratch| -> Result<(), FormatError> {
+                    for i in batch {
+                        let chunk = wanted[i];
+                        let origin = &chunk.offsets[..rank];
+                        usize::try_from(chunk.address).map_err(|_| {
+                            FormatError::Overflow("chunk address exceeds usize".into())
                         })?;
-                // Mirrors the full-read path: filter-mask bit i set means
+                        let raw = raw_bytes.get(i, &reqs[i])?;
-                // filter i was not applied to this chunk.
+                        // Mirrors the full-read path: filter-mask bit i set
-                let decoded;
+                        // means filter i was not applied to this chunk.
                         let data: &[u8] = match pipeline {
                             Some(pl) if !all_filters_skipped(pl, chunk.filter_mask) => {
-                        decoded = decompress_chunk_masked(
+                                decompress_chunk_exact_with(
                                     raw,
                                     pl,
                                     chunk_bytes,
                                     elem_size as u32,
                                     chunk.filter_mask,
-                        )?;
+                                    &chunk.offsets[..rank],
-                        &decoded
+                                    scratch,
+                                )?
                             }
                             _ => raw,
                         };
@@ -351,6 +415,9 @@ pub fn read_selection(
                             elem_size,
                         );
                     }
+                    Ok(())
+                })
+            })?;
         }
         _ => return Ok(None),
     }

crates/clawhdf5-format/src/provenance.rs

@@ -13,7 +13,6 @@ use sha2::{Digest, Sha256};
 
 use crate::attribute::AttributeMessage;
 use crate::data_layout::DataLayout;
-use crate::data_read::read_raw_data;
 use crate::dataspace::Dataspace;
 use crate::datatype::Datatype;
 use crate::error::FormatError;
@@ -128,10 +127,20 @@ pub fn verify_dataset(
     header: &ObjectHeader,
     offset_size: u8,
     length_size: u8,
+) -> Result<VerifyResult, FormatError> {
+    verify_dataset_in(file_data, header, offset_size, length_size)
+}
+
+/// [`verify_dataset`] over any [`Storage`](crate::storage::Storage).
+pub fn verify_dataset_in<S: crate::storage::Storage + ?Sized>(
+    file_data: &S,
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<VerifyResult, FormatError> {
     // 1. Extract all attributes (compact + dense).
     let attrs =
-        crate::attribute::extract_attributes_full(file_data, header, offset_size, length_size)?;
+        crate::attribute::extract_attributes_full_in(file_data, header, offset_size, length_size)?;
 
     // 2. Find the stored hash.
     let stored_hash = attrs
@@ -174,7 +183,7 @@ pub fn verify_dataset(
         .transpose()?;
 
     let raw = match &dl {
-        DataLayout::Chunked { .. } => crate::chunked_read::read_chunked_data(
+        DataLayout::Chunked { .. } => crate::chunked_read::read_chunked_data_in(
             file_data,
             &dl,
             &ds,
@@ -183,7 +192,7 @@ pub fn verify_dataset(
             offset_size,
             length_size,
         )?,
-        _ => read_raw_data(file_data, &dl, &ds, &dt)?,
+        _ => crate::data_read::read_raw_data_in(file_data, &dl, &ds, &dt)?,
     };
 
     // 4. Compare.

crates/clawhdf5-format/src/selection.rs

@@ -19,6 +19,7 @@ use alloc::{vec, vec::Vec};
 
 use core::ops::Range;
 
+use crate::addr::to_usize;
 use crate::error::FormatError;
 
 /// A selection describing which elements of a dataset to access.
@@ -229,44 +230,47 @@ impl Selection {
     /// self-describing in length, so the count lets a caller walk a packed list
     /// of selections — as the Virtual Dataset global-heap block does).
     ///
-    /// Only the forms needed for VDS assembly are decoded: `ALL`, `NONE`, and
+    /// Decodes `ALL`, `NONE`, and hyperslabs at every version libhdf5 writes
-    /// **regular** hyperslabs serialized at **version 3** (the encoding HDF5
+    /// (1: irregular, 4-byte coordinates — the default-format encoding; 2:
-    /// 1.10+/2.0 emit). Point selections, irregular hyperslabs, and older
+    /// regular, 8-byte; 3: either, variable width). A regular hyperslab maps
-    /// hyperslab versions return an error rather than mis-decoding.
+    /// to [`Selection::Hyperslab`]; an *irregular* one (a union of blocks)
+    /// maps to a single-block hyperslab when it has one block, and otherwise to
+    /// [`Selection::Points`] listing the union in row-major order (the order
+    /// libhdf5 iterates it in). Unlimited counts/blocks decode as `u64::MAX`
+    /// (see [`SerializedSelection::decode`] for the raw form). Point
+    /// selections are refused: libhdf5 does not allow them in virtual datasets
+    /// either.
     pub fn decode_serialized(data: &[u8]) -> Result<(Selection, usize), FormatError> {
-        if data.len() < 8 {
+        let (raw, len) = SerializedSelection::decode(data)?;
-            return Err(FormatError::UnexpectedEof {
+        let sel = match raw {
-                expected: 8,
+            SerializedSelection::All => Selection::All,
-                available: data.len(),
+            SerializedSelection::None => Selection::None,
-            });
+            SerializedSelection::Regular {
+                start,
+                stride,
+                count,
+                block,
+            } => Selection::Hyperslab {
+                start,
+                stride,
+                count,
+                block,
+            },
+            SerializedSelection::Blocks { rank, starts, ends } => {
+                if starts.len() == rank {
+                    let block = starts.iter().zip(&ends).map(|(&s, &e)| e - s + 1).collect();
+                    Selection::Hyperslab {
+                        start: starts,
+                        stride: vec![1; rank],
+                        count: vec![1; rank],
+                        block,
                     }
-        let sel_type = u32::from_le_bytes([data[0], data[1], data[2], data[3]]);
-        let version = u32::from_le_bytes([data[4], data[5], data[6], data[7]]);
-
-        match sel_type {
-            // ALL / NONE: type(4) + version(4) + reserved(4) + length(4) = 16 bytes.
-            3 | 0 => {
-                if data.len() < 16 {
-                    return Err(FormatError::UnexpectedEof {
-                        expected: 16,
-                        available: data.len(),
-                    });
-                }
-                let sel = if sel_type == 3 {
-                    Selection::All
                 } else {
-                    Selection::None
+                    Selection::Points(blocks_union_coords(rank, &starts, &ends)?)
+                }
+            }
         };
-                Ok((sel, 16))
+        Ok((sel, len))
-            }
-            2 => decode_hyperslab_serialized(data, version),
-            1 => Err(FormatError::ChunkedReadError(
-                "VDS point selections are not supported".into(),
-            )),
-            _ => Err(FormatError::ChunkedReadError(
-                "unknown dataspace selection type".into(),
-            )),
-        }
     }
 
     /// Enumerate the selected element indices of a **1-D** dataspace of the
@@ -314,6 +318,11 @@ impl Selection {
                         "VDS selection rank does not match dataspace rank".into(),
                     ));
                 }
+                if count.iter().chain(block.iter()).any(|&v| v == UNLIMITED) {
+                    return Err(FormatError::ChunkedReadError(
+                        "unlimited selection must be clipped before it is enumerated".into(),
+                    ));
+                }
                 // Selected coordinates along each dimension, in order.
                 let mut per_dim: Vec<Vec<u64>> = Vec::with_capacity(rank);
                 for d in 0..rank {
@@ -400,59 +409,174 @@ impl Selection {
     }
 }
 
-/// Decode an `H5S_SEL_HYPER` selection in its serialized form. Only version-3
+/// Hyperslab count/block value meaning "unlimited" (`H5S_UNLIMITED`).
-/// **regular** hyperslabs are supported.
+pub const UNLIMITED: u64 = u64::MAX;
-fn decode_hyperslab_serialized(
+
-    data: &[u8],
+/// Largest number of elements an irregular selection is expanded to when it
-    version: u32,
+/// is converted to a point list by [`Selection::decode_serialized`].
-) -> Result<(Selection, usize), FormatError> {
+const MAX_EXPANDED_POINTS: u64 = 1 << 26;
-    if version != 3 {
+
-        return Err(FormatError::ChunkedReadError(
+/// A selection exactly as `H5S_select_serialize` stores it, before it is
-            "only version-3 hyperslab selections are supported".into(),
+/// applied to any dataspace.
-        ));
+///
+/// Unlike [`Selection`] this keeps an irregular hyperslab as its list of
+/// blocks, and a regular hyperslab's count/block may be [`UNLIMITED`] (the
+/// unlimited selections used by unlimited and "printf" virtual dataset
+/// mappings).
+#[derive(Debug, Clone, PartialEq)]
+pub enum SerializedSelection {
+    /// `H5S_SEL_ALL`.
+    All,
+    /// `H5S_SEL_NONE`.
+    None,
+    /// A regular hyperslab. `count[d]` or `block[d]` may be [`UNLIMITED`].
+    Regular {
+        start: Vec<u64>,
+        stride: Vec<u64>,
+        count: Vec<u64>,
+        block: Vec<u64>,
+    },
+    /// An irregular hyperslab: the union of `starts.len() / rank` blocks, each
+    /// given by its first (`starts`) and last (`ends`, inclusive) coordinate,
+    /// flattened block-major.
+    Blocks {
+        rank: usize,
+        starts: Vec<u64>,
+        ends: Vec<u64>,
+    },
+}
+
+fn sel_err(msg: &str) -> FormatError {
+    FormatError::ChunkedReadError(msg.into())
+}
+
+/// Bounds-checked little-endian reader over a serialized selection.
+struct SelReader<'a> {
+    data: &'a [u8],
+    pos: usize,
+}
+
+impl SelReader<'_> {
+    fn take(&mut self, n: usize) -> Result<&[u8], FormatError> {
+        let end = self.pos.checked_add(n).filter(|&e| e <= self.data.len());
+        let end = end.ok_or(FormatError::UnexpectedEof {
+            expected: self.pos.saturating_add(n),
+            available: self.data.len(),
+        })?;
+        let s = &self.data[self.pos..end];
+        self.pos = end;
+        Ok(s)
     }
-    // type(4) ver(4) flags(1) enc_size(1) rank(4) [start,stride,count,block]*rank
+
-    if data.len() < 14 {
+    fn uint(&mut self, size: usize) -> Result<u64, FormatError> {
-        return Err(FormatError::UnexpectedEof {
+        let bytes = self.take(size)?;
-            expected: 14,
+        Ok(bytes
-            available: data.len(),
+            .iter()
-        });
+            .enumerate()
+            .fold(0u64, |v, (i, &b)| v | (b as u64) << (i * 8)))
     }
-    let flags = data[8];
+
-    let enc_size = data[9] as usize;
+    fn remaining(&self) -> usize {
-    // Bit 0 set => regular hyperslab. Irregular hyperslabs list explicit blocks.
+        self.data.len() - self.pos
-    if flags & 0x01 == 0 {
-        return Err(FormatError::ChunkedReadError(
-            "irregular VDS hyperslab selections are not supported".into(),
-        ));
     }
-    if enc_size != 2 && enc_size != 4 && enc_size != 8 {
+}
-        return Err(FormatError::ChunkedReadError(
+
-            "unsupported hyperslab coordinate encoding size".into(),
+impl SerializedSelection {
-        ));
+    /// Decode a serialized selection, returning it and the number of bytes it
-    }
+    /// occupies. Mirrors libhdf5's `H5S_select_deserialize`: `ALL`/`NONE` and
-    let rank = u32::from_le_bytes([data[10], data[11], data[12], data[13]]) as usize;
+    /// hyperslab versions 1-3 are decoded; point selections (which libhdf5
-    // HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything larger so a
+    /// refuses in virtual datasets) and malformed input are errors.
-    // corrupt rank can't drive a huge allocation or read loop.
+    pub fn decode(data: &[u8]) -> Result<(SerializedSelection, usize), FormatError> {
-    if rank > 32 {
+        let mut r = SelReader { data, pos: 0 };
-        return Err(FormatError::ChunkedReadError(
+        let sel_type = r.uint(4)?;
-            "hyperslab selection rank exceeds maximum (32)".into(),
+        let version = r.uint(4)?;
-        ));
+        match sel_type {
-    }
+            // ALL / NONE: type(4) + version(4) + reserved(4) + length(4).
-    let mut pos = 14;
+            0 | 3 => {
-    let read_coord = |data: &[u8], pos: usize| -> Result<u64, FormatError> {
+                r.take(8)?;
-        if pos + enc_size > data.len() {
+                let sel = if sel_type == 3 {
-            return Err(FormatError::UnexpectedEof {
+                    SerializedSelection::All
-                expected: pos + enc_size,
+                } else {
-                available: data.len(),
+                    SerializedSelection::None
-            });
-        }
-        let mut v = 0u64;
-        for (i, &b) in data[pos..pos + enc_size].iter().enumerate() {
-            v |= (b as u64) << (i * 8);
-        }
-        Ok(v)
                 };
+                Ok((sel, r.pos))
+            }
+            2 => {
+                let sel = decode_hyperslab(&mut r, version)?;
+                Ok((sel, r.pos))
+            }
+            1 => Err(sel_err(
+                "VDS point selections are not supported (libhdf5 rejects them too)",
+            )),
+            _ => Err(sel_err("unknown dataspace selection type")),
+        }
+    }
+
+    /// The single dimension in which this selection is unlimited, if any.
+    pub fn unlimited_dim(&self) -> Option<usize> {
+        match self {
+            SerializedSelection::Regular { count, block, .. } => count
+                .iter()
+                .zip(block)
+                .position(|(&c, &b)| c == UNLIMITED || b == UNLIMITED),
+            _ => None,
+        }
+    }
+
+    /// The rank the selection was serialized with (`None` for ALL/NONE, which
+    /// carry no rank).
+    pub fn rank(&self) -> Option<usize> {
+        match self {
+            SerializedSelection::Regular { start, .. } => Some(start.len()),
+            SerializedSelection::Blocks { rank, .. } => Some(*rank),
+            _ => None,
+        }
+    }
+}
+
+/// `H5S__hyper_deserialize`: after the type and version words.
+fn decode_hyperslab(r: &mut SelReader, version: u64) -> Result<SerializedSelection, FormatError> {
+    const REGULAR: u8 = 0x01;
+    let (flags, enc_size) = match version {
+        // v1: reserved(4) + length(4), always irregular, 4-byte coordinates.
+        1 => {
+            r.take(8)?;
+            (0u8, 4usize)
+        }
+        // v2: flags(1) + length(4), 8-byte coordinates.
+        2 => {
+            let flags = r.take(1)?[0];
+            r.take(4)?;
+            (flags, 8)
+        }
+        // v3: flags(1) + encoding size(1).
+        3 => {
+            let flags = r.take(1)?[0];
+            let enc = r.take(1)?[0] as usize;
+            (flags, enc)
+        }
+        _ => return Err(sel_err("unsupported hyperslab selection version")),
+    };
+    if flags & !REGULAR != 0 {
+        return Err(sel_err("unknown hyperslab selection flags"));
+    }
+    if !matches!(enc_size, 2 | 4 | 8) {
+        return Err(sel_err("unsupported hyperslab coordinate encoding size"));
+    }
+    let rank = to_usize(r.uint(4)?)?;
+    // HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything else so a
+    // corrupt rank can't drive a huge allocation or read loop.
+    if rank == 0 || rank > 32 {
+        return Err(sel_err("hyperslab selection rank must be 1..=32"));
+    }
+    // The all-ones value of the encoding width means "unlimited".
+    let unlim_raw = if enc_size == 8 {
+        u64::MAX
+    } else {
+        (1u64 << (enc_size * 8)) - 1
+    };
+
+    if flags & REGULAR != 0 {
         let (mut start, mut stride, mut count, mut block) = (
             Vec::with_capacity(rank),
             Vec::with_capacity(rank),
@@ -460,24 +584,104 @@ fn decode_hyperslab_serialized(
             Vec::with_capacity(rank),
         );
         for _ in 0..rank {
-        start.push(read_coord(data, pos)?);
+            start.push(r.uint(enc_size)?);
-        pos += enc_size;
+            stride.push(r.uint(enc_size)?);
-        stride.push(read_coord(data, pos)?);
+            let c = r.uint(enc_size)?;
-        pos += enc_size;
+            count.push(if c == unlim_raw { UNLIMITED } else { c });
-        count.push(read_coord(data, pos)?);
+            let b = r.uint(enc_size)?;
-        pos += enc_size;
+            block.push(if b == unlim_raw { UNLIMITED } else { b });
-        block.push(read_coord(data, pos)?);
-        pos += enc_size;
         }
-    Ok((
+        let unlimited = count
-        Selection::Hyperslab {
+            .iter()
+            .zip(&block)
+            .filter(|&(&c, &b)| c == UNLIMITED || b == UNLIMITED)
+            .count();
+        if unlimited > 1 {
+            return Err(sel_err(
+                "hyperslab selection is unlimited in more than one dimension",
+            ));
+        }
+        for d in 0..rank {
+            // Overlapping blocks are not a valid regular hyperslab.
+            if count[d] > 1 && block[d] != UNLIMITED && block[d] > stride[d] {
+                return Err(sel_err("regular hyperslab blocks overlap"));
+            }
+        }
+        return Ok(SerializedSelection::Regular {
             start,
             stride,
             count,
             block,
-        },
+        });
-        pos,
+    }
-    ))
+
+    // Irregular: number of blocks, then each block's start and end corners.
+    let nblocks = r.uint(enc_size)?;
+    let per_block = (rank * 2 * enc_size) as u64;
+    // Untrusted count: it must fit in what is left of the buffer.
+    if nblocks
+        .checked_mul(per_block)
+        .is_none_or(|need| need > r.remaining() as u64)
+    {
+        return Err(FormatError::UnexpectedEof {
+            expected: r
+                .pos
+                .saturating_add(to_usize(nblocks.saturating_mul(per_block))?),
+            available: r.data.len(),
+        });
+    }
+    let n = to_usize(nblocks)? * rank;
+    let (mut starts, mut ends) = (Vec::with_capacity(n), Vec::with_capacity(n));
+    for _ in 0..nblocks {
+        for _ in 0..rank {
+            starts.push(r.uint(enc_size)?);
+        }
+        for _ in 0..rank {
+            ends.push(r.uint(enc_size)?);
+        }
+    }
+    if starts.iter().zip(&ends).any(|(s, e)| e < s) {
+        return Err(sel_err("hyperslab block ends before it starts"));
+    }
+    Ok(SerializedSelection::Blocks { rank, starts, ends })
+}
+
+/// The coordinates of the union of the given blocks, in row-major order.
+fn blocks_union_coords(
+    rank: usize,
+    starts: &[u64],
+    ends: &[u64],
+) -> Result<Vec<Vec<u64>>, FormatError> {
+    let mut total = 0u64;
+    for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
+        let vol = s
+            .iter()
+            .zip(e)
+            .try_fold(1u64, |acc, (&s, &e)| acc.checked_mul(e - s + 1));
+        total = vol
+            .and_then(|v| total.checked_add(v))
+            .filter(|&t| t <= MAX_EXPANDED_POINTS)
+            .ok_or_else(|| sel_err("irregular hyperslab selection is too large to expand"))?;
+    }
+    let mut out = Vec::with_capacity(to_usize(total)?);
+    for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
+        let mut cur = s.to_vec();
+        'block: loop {
+            out.push(cur.clone());
+            for d in (0..rank).rev() {
+                if cur[d] < e[d] {
+                    cur[d] += 1;
+                    continue 'block;
+                }
+                cur[d] = s[d];
+            }
+            break;
+        }
+    }
+    // Lexicographic order of coordinates is row-major order.
+    out.sort_unstable();
+    out.dedup();
+    Ok(out)
 }
 
 // ---------------------------------------------------------------------------
@@ -642,11 +846,100 @@ mod tests {
     }
 
     #[test]
-    fn decode_irregular_hyperslab_rejected() {
+    fn decode_truncated_irregular_hyperslab_is_error() {
+        // Irregular, rank 1, but the block count is missing.
         let bytes = [0x02u8, 0, 0, 0, 0x03, 0, 0, 0, 0x00, 0x02, 0x01, 0, 0, 0];
         assert!(Selection::decode_serialized(&bytes).is_err());
     }
 
+    /// Version 1 as libhdf5 writes it for the default (earliest) format bounds:
+    /// type, version, reserved(4), length(4), rank(4), nblocks(4), then each
+    /// block's start and inclusive end corner as 4-byte values.
+    fn v1_blocks(rank: u32, blocks: &[(&[u32], &[u32])]) -> Vec<u8> {
+        let mut b = Vec::new();
+        for w in [2u32, 1, 0, 0, rank, blocks.len() as u32] {
+            b.extend_from_slice(&w.to_le_bytes());
+        }
+        for (s, e) in blocks {
+            for v in s.iter().chain(e.iter()) {
+                b.extend_from_slice(&v.to_le_bytes());
+            }
+        }
+        b
+    }
+
+    #[test]
+    fn decode_v1_irregular_single_block() {
+        // Exactly what h5py/HDF5 2.0 writes for `[0:4]` with default libver.
+        let bytes = v1_blocks(1, &[(&[0], &[3])]);
+        let (sel, used) = Selection::decode_serialized(&bytes).unwrap();
+        assert_eq!(used, bytes.len());
+        assert_eq!(sel.iter_linear_1d(8).unwrap(), vec![0, 1, 2, 3]);
+    }
+
+    #[test]
+    fn decode_v1_irregular_union_is_row_major() {
+        // Blocks given out of order and overlapping still enumerate once each,
+        // in row-major order (libhdf5 iterates the union, not the list).
+        let bytes = v1_blocks(2, &[(&[1, 0], &[1, 1]), (&[0, 2], &[1, 2])]);
+        let (sel, used) = Selection::decode_serialized(&bytes).unwrap();
+        assert_eq!(used, bytes.len());
+        // (0,2) (1,0) (1,1) (1,2) in a 2x3 space.
+        assert_eq!(sel.iter_linear(&[2, 3]).unwrap(), vec![2, 3, 4, 5]);
+    }
+
+    #[test]
+    fn decode_v2_regular_with_unlimited_count() {
+        // v2: flags(1) + length(4), then 8-byte start/stride/count/block.
+        let mut b = Vec::new();
+        b.extend_from_slice(&2u32.to_le_bytes());
+        b.extend_from_slice(&2u32.to_le_bytes());
+        b.push(0x01);
+        b.extend_from_slice(&36u32.to_le_bytes());
+        b.extend_from_slice(&1u32.to_le_bytes());
+        for v in [0u64, 10, u64::MAX, 10] {
+            b.extend_from_slice(&v.to_le_bytes());
+        }
+        let (raw, used) = SerializedSelection::decode(&b).unwrap();
+        assert_eq!(used, b.len());
+        assert_eq!(raw.unlimited_dim(), Some(0));
+        assert_eq!(
+            raw,
+            SerializedSelection::Regular {
+                start: vec![0],
+                stride: vec![10],
+                count: vec![UNLIMITED],
+                block: vec![10],
+            }
+        );
+        // An unclipped unlimited selection cannot be enumerated.
+        let (sel, _) = Selection::decode_serialized(&b).unwrap();
+        assert!(sel.iter_linear_1d(100).is_err());
+    }
+
+    #[test]
+    fn decode_v3_two_byte_all_ones_is_unlimited() {
+        let bytes = [
+            0x02, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0x01, 0, 0, 0, //
+            0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0xFF, 0xFF,
+        ];
+        let (raw, _) = SerializedSelection::decode(&bytes).unwrap();
+        assert_eq!(raw.unlimited_dim(), Some(0));
+    }
+
+    #[test]
+    fn decode_irregular_block_count_beyond_buffer_is_error() {
+        let mut b = v1_blocks(1, &[(&[0], &[3])]);
+        b[20..24].copy_from_slice(&u32::MAX.to_le_bytes());
+        assert!(Selection::decode_serialized(&b).is_err());
+    }
+
+    #[test]
+    fn decode_point_selection_is_refused() {
+        let bytes = [1u8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
+        assert!(Selection::decode_serialized(&bytes).is_err());
+    }
+
     #[test]
     fn iter_linear_2d_block_row_major() {
         // A 2x2 block at the top-left of a 4x4 space => linear 0,1,4,5.

crates/clawhdf5-format/src/shared_message.rs

@@ -23,11 +23,12 @@ use alloc::vec::Vec;
 #[cfg(feature = "std")]
 use std::borrow::Cow;
 
-use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records_in};
 use crate::error::FormatError;
 use crate::fractal_heap::FractalHeapHeader;
 use crate::message_type::MessageType;
 use crate::object_header::ObjectHeader;
+use crate::storage::{Storage, Window, read_exact_at};
 
 /// Fractal heap ID length for SOHM entries (fixed at 8 bytes).
 const FHEAP_ID_LEN: usize = 8;
@@ -154,13 +155,29 @@ pub fn is_shared(msg_flags: u8) -> bool {
 ///
 /// When the shared flag is set on a message, the data contains a reference
 /// instead of the actual message content.
+///
+/// Assumes the file's length size equals its offset size, which only matters
+/// for version-1 references; use [`parse_shared_ref_sized`] when the
+/// superblock's length size is known.
 pub fn parse_shared_ref(data: &[u8], offset_size: u8) -> Result<SharedMessageRef, FormatError> {
+    parse_shared_ref_sized(data, offset_size, offset_size)
+}
+
+/// [`parse_shared_ref`] with the superblock's length size, which locates the
+/// object header address in a version-1 reference.
+pub fn parse_shared_ref_sized(
+    data: &[u8],
+    offset_size: u8,
+    length_size: u8,
+) -> Result<SharedMessageRef, FormatError> {
     ensure_len(data, 0, 2)?;
     let version = data[0];
     let ref_type = data[1];
 
     // Layouts (HDF5 spec IV.A.2 "Shared Message", and libhdf5's decoder):
-    //   v1: version, type, reserved(6), address     — always "committed"
+    //   v1: version, type, reserved(6), then an old-style symbol table
+    //       entry: link-name offset(length_size), object header address,
+    //       cache type(4), reserved(4), scratch(16)  — always "committed"
     //   v2: version, type, address                  — always "committed"
     //   v3: version, type, then a fractal-heap ID if type == SOHM, otherwise
     //       an address
@@ -177,7 +194,7 @@ pub fn parse_shared_ref(data: &[u8], offset_size: u8) -> Result<SharedMessageRef
         })
     };
     match version {
-        1 => address_at(2 + 6),
+        1 => address_at(2 + 6 + length_size as usize),
         2 => address_at(2),
         3 if ref_type == SHARE_TYPE_SOHM => {
             ensure_len(data, 2, FHEAP_ID_LEN)?;
@@ -237,17 +254,31 @@ pub fn parse_sohm_table(
     nindexes: u8,
     offset_size: u8,
 ) -> Result<SohmTable, FormatError> {
-    ensure_len(file_data, table_addr, 4)?;
+    parse_sohm_table_in(file_data, table_addr as u64, nindexes, offset_size)
-    if &file_data[table_addr..table_addr + 4] != b"SMTB" {
+}
+
+/// [`parse_sohm_table`] over any [`Storage`]: one read of the signature,
+/// one of every index entry.
+pub fn parse_sohm_table_in<S: Storage + ?Sized>(
+    file: &S,
+    table_addr: u64,
+    nindexes: u8,
+    offset_size: u8,
+) -> Result<SohmTable, FormatError> {
+    let sig = read_exact_at(file, table_addr, 4)?;
+    if *sig != *b"SMTB" {
         return Err(FormatError::InvalidSohmTableSignature);
     }
-    let mut pos = table_addr + 4;
     let os = offset_size as usize;
     let entry_size = 1 + 1 + 2 + 4 + 2 + 2 + 2 + os + os; // 14 + 2*offset_size
+    // Positions below are relative to the table.
+    let w = Window::read(file, table_addr, 4 + nindexes as usize * entry_size)?;
+    let file_data: &[u8] = &w.bytes;
+    let mut pos = 4;
 
     let mut indexes = Vec::with_capacity(nindexes as usize);
     for _ in 0..nindexes {
-        ensure_len(file_data, pos, entry_size)?;
+        w.ensure(pos, entry_size)?;
         let version = file_data[pos];
         if version != 0 {
             return Err(FormatError::InvalidSohmTableVersion(version));
@@ -353,16 +384,29 @@ pub fn parse_sohm_list(
     num_messages: u16,
     offset_size: u8,
 ) -> Result<Vec<SohmEntry>, FormatError> {
-    ensure_len(file_data, list_addr, 4)?;
+    parse_sohm_list_in(file_data, list_addr as u64, num_messages, offset_size)
-    if &file_data[list_addr..list_addr + 4] != b"SMLI" {
+}
+
+/// [`parse_sohm_list`] over any [`Storage`]: one read of the signature, one
+/// of every entry.
+pub fn parse_sohm_list_in<S: Storage + ?Sized>(
+    file: &S,
+    list_addr: u64,
+    num_messages: u16,
+    offset_size: u8,
+) -> Result<Vec<SohmEntry>, FormatError> {
+    let sig = read_exact_at(file, list_addr, 4)?;
+    if *sig != *b"SMLI" {
         return Err(FormatError::InvalidSohmListSignature);
     }
     let entry_sz = sohm_entry_size(offset_size);
-    let mut pos = list_addr + 4;
+    // Positions below are relative to the list.
+    let w = Window::read(file, list_addr, 4 + num_messages as usize * entry_sz)?;
+    let mut pos = 4;
     let mut entries = Vec::with_capacity(num_messages as usize);
     for _ in 0..num_messages {
-        ensure_len(file_data, pos, entry_sz)?;
+        w.ensure(pos, entry_sz)?;
-        let entry = parse_sohm_entry(&file_data[pos..], offset_size)?;
+        let entry = parse_sohm_entry(&w.bytes[pos..], offset_size)?;
         entries.push(entry);
         pos += entry_sz;
     }
@@ -376,8 +420,18 @@ pub fn parse_sohm_btree_entries(
     offset_size: u8,
     length_size: u8,
 ) -> Result<Vec<SohmEntry>, FormatError> {
-    let header = BTreeV2Header::parse(file_data, btree_addr, offset_size, length_size)?;
+    parse_sohm_btree_entries_in(file_data, btree_addr as u64, offset_size, length_size)
-    let records = collect_btree_v2_records(file_data, &header, offset_size, length_size)?;
+}
+
+/// [`parse_sohm_btree_entries`] over any [`Storage`].
+pub fn parse_sohm_btree_entries_in<S: Storage + ?Sized>(
+    file: &S,
+    btree_addr: u64,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<SohmEntry>, FormatError> {
+    let header = BTreeV2Header::parse_in(file, btree_addr, offset_size, length_size)?;
+    let records = collect_btree_v2_records_in(file, &header, offset_size, length_size)?;
     let mut entries = Vec::with_capacity(records.len());
     for rec in &records {
         let entry = parse_sohm_entry(&rec.data, offset_size)?;
@@ -397,15 +451,24 @@ pub fn load_sohm_table(
     offset_size: u8,
     length_size: u8,
 ) -> Result<Option<SohmTable>, FormatError> {
-    let sig = crate::signature::find_signature(file_data)?;
+    load_sohm_table_in(file_data, offset_size, length_size)
-    let sb = crate::superblock::Superblock::parse(file_data, sig)?;
+}
+
+/// [`load_sohm_table`] over any [`Storage`].
+pub fn load_sohm_table_in<S: Storage + ?Sized>(
+    file_data: &S,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Option<SohmTable>, FormatError> {
+    let sig = crate::signature::find_signature_in(file_data)?;
+    let sb = crate::superblock::Superblock::parse_in(file_data, sig)?;
     let Some(ext_addr) = sb
         .superblock_extension_address
         .filter(|&a| !is_undefined(a, offset_size))
     else {
         return Ok(None);
     };
-    let ext = ObjectHeader::parse(file_data, ext_addr as usize, offset_size, length_size)?;
+    let ext = ObjectHeader::parse_in(file_data, ext_addr, offset_size, length_size)?;
     let Some(msg) = ext
         .messages
         .iter()
@@ -414,9 +477,9 @@ pub fn load_sohm_table(
         return Ok(None);
     };
     let table_msg = parse_sohm_table_message(&msg.data, offset_size)?;
-    parse_sohm_table(
+    parse_sohm_table_in(
         file_data,
-        table_msg.table_address as usize,
+        table_msg.table_address,
         table_msg.nindexes,
         offset_size,
     )
@@ -430,17 +493,27 @@ pub fn message_data_with_sohm<'a>(
     msg: &'a crate::object_header::HeaderMessage,
     offset_size: u8,
     length_size: u8,
+) -> Result<Cow<'a, [u8]>, FormatError> {
+    message_data_with_sohm_in(file_data, msg, offset_size, length_size)
+}
+
+/// [`message_data_with_sohm`] over any [`Storage`].
+pub fn message_data_with_sohm_in<'a, S: Storage + ?Sized>(
+    file_data: &S,
+    msg: &'a crate::object_header::HeaderMessage,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Cow<'a, [u8]>, FormatError> {
     if !is_shared(msg.flags) {
         return Ok(Cow::Borrowed(&msg.data));
     }
-    let shared_ref = parse_shared_ref(&msg.data, offset_size)?;
+    let shared_ref = parse_shared_ref_sized(&msg.data, offset_size, length_size)?;
     let table = if shared_ref.heap_id.is_some() {
-        load_sohm_table(file_data, offset_size, length_size)?
+        load_sohm_table_in(file_data, offset_size, length_size)?
     } else {
         None
     };
-    resolve_shared_message_with_sohm(
+    resolve_shared_message_with_sohm_in(
         file_data,
         &shared_ref,
         msg.msg_type,
@@ -479,6 +552,25 @@ pub fn resolve_sohm_message(
     target_msg_type: MessageType,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<u8>, FormatError> {
+    resolve_sohm_message_in(
+        &file_data,
+        heap_id,
+        sohm_table,
+        target_msg_type,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`resolve_sohm_message`] over any [`Storage`].
+pub fn resolve_sohm_message_in<S: Storage + ?Sized>(
+    file_data: &S,
+    heap_id: &[u8; FHEAP_ID_LEN],
+    sohm_table: &SohmTable,
+    target_msg_type: MessageType,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<u8>, FormatError> {
     let index = find_index_for_msg_type(sohm_table, target_msg_type)
         .ok_or(FormatError::InvalidSharedMessageVersion(2))?;
@@ -487,13 +579,9 @@ pub fn resolve_sohm_message(
         return Err(FormatError::InvalidSharedMessageVersion(2));
     }
 
-    let fh_header = FractalHeapHeader::parse(
+    let fh_header =
-        file_data,
+        FractalHeapHeader::parse_in(file_data, index.heap_addr, offset_size, length_size)?;
-        index.heap_addr as usize,
+    fh_header.read_managed_object_in(file_data, heap_id, offset_size)
-        offset_size,
-        length_size,
-    )?;
-    fh_header.read_managed_object(file_data, heap_id, offset_size)
 }
 
 /// The payload of an object-header message, following the indirection if the
@@ -510,12 +598,22 @@ pub fn message_data<'a>(
     msg: &'a crate::object_header::HeaderMessage,
     offset_size: u8,
     length_size: u8,
+) -> Result<Cow<'a, [u8]>, FormatError> {
+    message_data_in(file_data, msg, offset_size, length_size)
+}
+
+/// [`message_data`] over any [`Storage`].
+pub fn message_data_in<'a, S: Storage + ?Sized>(
+    file_data: &S,
+    msg: &'a crate::object_header::HeaderMessage,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Cow<'a, [u8]>, FormatError> {
     if !is_shared(msg.flags) {
         return Ok(Cow::Borrowed(&msg.data));
     }
-    let shared_ref = parse_shared_ref(&msg.data, offset_size)?;
+    let shared_ref = parse_shared_ref_sized(&msg.data, offset_size, length_size)?;
-    resolve_shared_message(
+    resolve_shared_message_in(
         file_data,
         &shared_ref,
         msg.msg_type,
@@ -529,7 +627,8 @@ pub fn message_data<'a>(
 ///
 /// For type 1/3 (shared in another object header), reads the target object header
 /// and finds the message of the specified type.
-/// For type 2 (SOHM), uses the fractal heap from the SOHM table.
+/// For type 2 (SOHM), uses the fractal heap from the file's SOHM table,
+/// loaded from the superblock extension on demand.
 pub fn resolve_shared_message(
     file_data: &[u8],
     shared_ref: &SharedMessageRef,
@@ -537,13 +636,35 @@ pub fn resolve_shared_message(
     offset_size: u8,
     length_size: u8,
 ) -> Result<Vec<u8>, FormatError> {
-    resolve_shared_message_with_sohm(
+    resolve_shared_message_in(
+        &file_data,
+        shared_ref,
+        target_msg_type,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`resolve_shared_message`] over any [`Storage`].
+pub fn resolve_shared_message_in<S: Storage + ?Sized>(
+    file_data: &S,
+    shared_ref: &SharedMessageRef,
+    target_msg_type: MessageType,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<u8>, FormatError> {
+    let table = if shared_ref.heap_id.is_some() {
+        load_sohm_table_in(file_data, offset_size, length_size)?
+    } else {
+        None
+    };
+    resolve_shared_message_with_sohm_in(
         file_data,
         shared_ref,
         target_msg_type,
         offset_size,
         length_size,
-        None,
+        table.as_ref(),
     )
 }
 
@@ -555,6 +676,25 @@ pub fn resolve_shared_message_with_sohm(
     offset_size: u8,
     length_size: u8,
     sohm_table: Option<&SohmTable>,
+) -> Result<Vec<u8>, FormatError> {
+    resolve_shared_message_with_sohm_in(
+        &file_data,
+        shared_ref,
+        target_msg_type,
+        offset_size,
+        length_size,
+        sohm_table,
+    )
+}
+
+/// [`resolve_shared_message_with_sohm`] over any [`Storage`].
+pub fn resolve_shared_message_with_sohm_in<S: Storage + ?Sized>(
+    file_data: &S,
+    shared_ref: &SharedMessageRef,
+    target_msg_type: MessageType,
+    offset_size: u8,
+    length_size: u8,
+    sohm_table: Option<&SohmTable>,
 ) -> Result<Vec<u8>, FormatError> {
     // Dispatch on what the reference carries rather than on `ref_type`: v1/v2
     // references are always an object-header address whatever their type
@@ -564,8 +704,7 @@ pub fn resolve_shared_message_with_sohm(
         shared_ref.heap_id.as_ref(),
     ) {
         (Some(addr), _) => {
-            let target_header =
+            let target_header = ObjectHeader::parse_in(file_data, addr, offset_size, length_size)?;
-                ObjectHeader::parse(file_data, addr as usize, offset_size, length_size)?;
             for msg in &target_header.messages {
                 if msg.msg_type == target_msg_type && !is_shared(msg.flags) {
                     return Ok(msg.data.clone());
@@ -592,7 +731,7 @@ pub fn resolve_shared_message_with_sohm(
         }
         (None, Some(heap_id)) => {
             let table = sohm_table.ok_or(FormatError::InvalidSharedMessageVersion(2))?;
-            resolve_sohm_message(
+            resolve_sohm_message_in(
                 file_data,
                 heap_id,
                 table,
@@ -649,15 +788,26 @@ mod tests {
 
     #[test]
     fn parse_v1_ref() {
-        let mut data = Vec::new();
+        // Datatype message of `/group1/dset2` in HDF5's `tcompound.h5`
-        data.push(1); // version
+        // (written in 2000): version 1, six reserved bytes, then an old-style
-        data.push(0); // type
+        // symbol table entry — link-name offset 0x10, object header address
-        data.extend_from_slice(&[0u8; 6]); // reserved
+        // 0x590 (the committed datatype `/type1`), cache type, reserved and
-        data.extend_from_slice(&0x5678u64.to_le_bytes());
+        // scratch.
+        let mut data = vec![1, 0, 0, 0, 0, 0, 0, 0];
+        data.extend_from_slice(&0x10u64.to_le_bytes());
+        data.extend_from_slice(&0x590u64.to_le_bytes());
+        data.extend_from_slice(&[0; 24]);
 
-        let shared = parse_shared_ref(&data, 8).unwrap();
+        let shared = parse_shared_ref_sized(&data, 8, 8).unwrap();
         assert_eq!(shared.version, 1);
-        assert_eq!(shared.object_header_address, Some(0x5678));
+        assert_eq!(shared.object_header_address, Some(0x590));
+
+        // The name offset is a length: 4 bytes here, then an 8-byte address.
+        let mut data = vec![1, 0, 0, 0, 0, 0, 0, 0];
+        data.extend_from_slice(&0x10u32.to_le_bytes());
+        data.extend_from_slice(&0x590u64.to_le_bytes());
+        let shared = parse_shared_ref_sized(&data, 8, 4).unwrap();
+        assert_eq!(shared.object_header_address, Some(0x590));
     }
 
     #[test]
@@ -1019,4 +1169,82 @@ mod tests {
         // With 2-byte offsets: OH=2+2=4, heap=12, entry=1+4+12=17
         assert_eq!(sohm_entry_size(2), 17);
     }
+
+    /// SOHM tables and lists parse identically through a read_at-only
+    /// CountingStorage: at two offsets, with 4- and 8-byte offsets, cut at
+    /// every length and with a bad signature.
+    #[test]
+    fn storage_reads_match_slice_reads() {
+        use crate::storage::CountingStorage;
+        let idx = |t: u8, n: u16| SohmIndex {
+            index_type: t,
+            mesg_types: 0x0008,
+            min_mesg_size: 50,
+            list_max: 50,
+            btree_min: 40,
+            num_messages: n,
+            index_addr: 0x3000,
+            heap_addr: 0x4000,
+        };
+        let heap_entry = |h: u32| SohmEntry {
+            location: 0,
+            hash: h,
+            heap_id: Some([1, 2, 3, 4, 5, 6, 7, h as u8]),
+            ref_count: Some(h),
+            mesg_index: None,
+            oh_addr: None,
+        };
+        let oh_entry = SohmEntry {
+            location: 1,
+            hash: 9,
+            heap_id: None,
+            ref_count: None,
+            mesg_index: Some(3),
+            oh_addr: Some(0x7000),
+        };
+        let mut compared = 0;
+        for os in [4u8, 8] {
+            let smtb = build_smtb(&[idx(0, 2), idx(1, 7)], os);
+            let smli = build_smli(&[heap_entry(1), oh_entry.clone(), heap_entry(2)], os);
+            for (body, n) in [(smtb, 2u16), (smli, 3)] {
+                let is_table = &body[..4] == b"SMTB";
+                for at in [0usize, 0x40] {
+                    let mut full = vec![0u8; at];
+                    full.extend_from_slice(&body);
+                    let mut files = Vec::new();
+                    for cut in at..=full.len() {
+                        files.push(full[..cut].to_vec());
+                    }
+                    let mut bad = full.clone();
+                    bad[at] = b'X';
+                    files.push(bad);
+                    for f in files {
+                        let st = CountingStorage::new(f.clone());
+                        let (want, got) = if is_table {
+                            (
+                                format!("{:?}", parse_sohm_table(&f, at, n as u8, os)),
+                                format!("{:?}", parse_sohm_table_in(&st, at as u64, n as u8, os)),
+                            )
+                        } else {
+                            (
+                                format!("{:?}", parse_sohm_list(&f, at, n, os)),
+                                format!("{:?}", parse_sohm_list_in(&st, at as u64, n, os)),
+                            )
+                        };
+                        assert_eq!(got, want, "{} bytes", f.len());
+                        assert!(st.reads() <= 2);
+                        compared += 1;
+                    }
+                }
+            }
+        }
+        assert!(compared > 200);
+        // The B-tree index reads through Storage too, errors included.
+        let junk = vec![0u8; 64];
+        let st = CountingStorage::new(junk.clone());
+        assert_eq!(
+            format!("{:?}", parse_sohm_btree_entries_in(&st, 0, 8, 8)),
+            format!("{:?}", parse_sohm_btree_entries(&junk, 0, 8, 8))
+        );
+    }
 }

crates/clawhdf5-format/src/signature.rs

@@ -1,6 +1,7 @@
 //! HDF5 file signature (magic bytes) detection.
 
 use crate::error::FormatError;
+use crate::storage::{Storage, read_exact_at};
 
 /// The 8-byte HDF5 magic signature.
 pub const HDF5_SIGNATURE: [u8; 8] = [0x89, b'H', b'D', b'F', b'\r', b'\n', 0x1A, b'\n'];
@@ -11,6 +12,16 @@ pub const HDF5_SIGNATURE: [u8; 8] = [0x89, b'H', b'D', b'F', b'\r', b'\n', 0x1A,
 /// (powers of two starting at 512, plus offset 0).
 ///
 /// Returns the byte offset where the signature was found.
+///
+/// A non-zero offset means the file starts with a *user block*, and every
+/// address inside the file is relative to the superblock's position, not to
+/// byte 0 (libhdf5 uses the signature's position as the base address even
+/// when the stored base-address field disagrees). The parsers in this crate
+/// take addresses as indices into `file_data`, so they must be handed the
+/// bytes from the signature on — use [`split_user_block`]. [`Superblock::parse`]
+/// refuses a non-zero offset for this reason.
+///
+/// [`Superblock::parse`]: crate::superblock::Superblock::parse
 pub fn find_signature(data: &[u8]) -> Result<usize, FormatError> {
     // Check offset 0
     if data.len() >= 8 && data[..8] == HDF5_SIGNATURE {
@@ -29,6 +40,31 @@ pub fn find_signature(data: &[u8]) -> Result<usize, FormatError> {
     Err(FormatError::SignatureNotFound)
 }
 
+/// [`find_signature`] over any [`Storage`]: one 8-byte read per candidate
+/// offset.
+pub fn find_signature_in<S: Storage + ?Sized>(file: &S) -> Result<u64, FormatError> {
+    let len = file.len();
+    let mut offset = 0u64;
+    while offset.checked_add(8).is_some_and(|end| end <= len) {
+        if *read_exact_at(file, offset, 8)? == HDF5_SIGNATURE {
+            return Ok(offset);
+        }
+        offset = if offset == 0 { 512 } else { offset * 2 };
+    }
+    Err(FormatError::SignatureNotFound)
+}
+
+/// Split a file into its user block and its HDF5 bytes.
+///
+/// Returns `(user_block, hdf5)`: `user_block` is everything before the
+/// superblock signature (empty for most files) and `hdf5` is the rest, in
+/// which every HDF5 address is a plain index. Pass `hdf5` as `file_data` to
+/// every parser in this crate, and parse the superblock at offset 0 of it.
+pub fn split_user_block(data: &[u8]) -> Result<(&[u8], &[u8]), FormatError> {
+    let offset = find_signature(data)?;
+    Ok(data.split_at(offset))
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -88,6 +124,21 @@ mod tests {
         assert_eq!(find_signature(&data), Err(FormatError::SignatureNotFound));
     }
 
+    #[test]
+    fn split_user_block_rebases_at_the_signature() {
+        let mut data = vec![7u8; 1024];
+        data[512..520].copy_from_slice(&HDF5_SIGNATURE);
+        let (ub, hdf5) = split_user_block(&data).unwrap();
+        assert_eq!(ub.len(), 512);
+        assert_eq!(hdf5.len(), 512);
+        assert_eq!(&hdf5[..8], &HDF5_SIGNATURE);
+
+        data[..8].copy_from_slice(&HDF5_SIGNATURE);
+        let (ub, hdf5) = split_user_block(&data).unwrap();
+        assert!(ub.is_empty());
+        assert_eq!(hdf5.len(), 1024);
+    }
+
     #[test]
     fn signature_prefers_earliest() {
         // Signature at both 0 and 512, should return 0
@@ -96,4 +147,26 @@ mod tests {
         data[512..520].copy_from_slice(&HDF5_SIGNATURE);
         assert_eq!(find_signature(&data), Ok(0));
     }
+
+    #[test]
+    fn find_signature_in_matches_slice_search() {
+        use crate::storage::CountingStorage;
+        for (len, at) in [
+            (0, None),
+            (7, None),
+            (8, Some(0)),
+            (600, Some(512)),
+            (5000, Some(4096)),
+            (3000, Some(2048)),
+            (3000, None),
+        ] {
+            let mut data = vec![0u8; len];
+            if let Some(at) = at {
+                data[at..at + 8].copy_from_slice(&HDF5_SIGNATURE);
+            }
+            let want = find_signature(&data).map(|o| o as u64);
+            let got = find_signature_in(&CountingStorage::new(data));
+            assert_eq!(got, want, "{len} {at:?}");
+        }
+    }
 }

crates/clawhdf5-format/src/storage.rs

@@ -0,0 +1,723 @@
+//! Where the parsers read the file from: the [`Storage`] trait.
+//!
+//! Every parser used to take the whole file as one `&[u8]`. [`Storage`] is
+//! the abstraction that replaces it (see `docs/design/range-reads.md`,
+//! option (a)): a parser asks for the bytes it needs, `[offset, offset +
+//! len)`, with 64-bit offsets, and gets them back as a [`Cow`] — borrowed
+//! when the backend holds the file in memory (a `Vec`, an mmap), owned when
+//! it had to fetch them (a range request, a block cache).
+//!
+//! `impl Storage for [u8]` serves the in-memory case with no copy, and
+//! [`Storage::as_contiguous`] lets a hot loop borrow the whole file at once
+//! when the backend has it. Modules are converted one at a time: a converted
+//! parser has an `*_in<S: Storage + ?Sized>(file: &S, ..)` core and keeps
+//! its old `&[u8]` signature as a thin wrapper, so callers do not change.
+//!
+//! The cores are generic rather than taking `&dyn Storage` so that the
+//! wrappers monomorphise for `[u8]`: the bounds check of each structure read
+//! inlines to what the slice code did, with no indirect call and no copy,
+//! which keeps local files as fast as before the migration. A `&dyn Storage`
+//! still works (`S = dyn Storage`), and a remote backend pays one indirect
+//! call per structure read.
+//!
+//! # Slice entry points
+//!
+//! A generic core is instantiated in the crate that calls it, so a
+//! downstream crate calling `parse_in::<[u8]>` gets its own copy of the
+//! parser, compiled without this crate's private helpers inlined (there is
+//! no cross-crate inlining of non-`#[inline]` functions without LTO): a
+//! metadata walk through the facade ran about 6% slower that way than
+//! through the `&[u8]` wrappers. The `*_in` entry points on the facade's hot
+//! paths (object headers, group listing and lookup, attributes) therefore
+//! check [`Storage::as_contiguous`] first and hand an in-memory file to
+//! their non-generic `&[u8]` wrapper, compiled here; both run the one core.
+//!
+//! The trait is synchronous and `no_std`: parsing is CPU work, and a remote
+//! backend bridges to its own I/O.
+
+#[cfg(not(feature = "std"))]
+use alloc::{borrow::Cow, boxed::Box, vec::Vec};
+#[cfg(feature = "std")]
+use std::{borrow::Cow, boxed::Box, vec::Vec};
+
+use core::ops::Range;
+
+use crate::error::FormatError;
+
+/// A random-access source of file bytes.
+///
+/// Offsets are relative to the start of the HDF5 data (the superblock), like
+/// every address in the file.
+pub trait Storage {
+    /// Bytes `[offset, offset + len)`.
+    ///
+    /// The result is shorter than `len` only when the range runs past the
+    /// end of the storage (and empty when `offset` is at or past the end);
+    /// a backend that cannot serve a range returns an error instead of a
+    /// short read.
+    /// It is never longer than `len`; the parsers cut a longer result to
+    /// `len` (see [`exact_len`]) rather than read bytes from outside the
+    /// range.
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError>;
+
+    /// Current length of the storage in bytes.
+    fn len(&self) -> u64;
+
+    /// Whether the storage holds no bytes.
+    fn is_empty(&self) -> bool {
+        self.len() == 0
+    }
+
+    /// Several reads at once, in the order given. Backends that talk to a
+    /// remote store coalesce and parallelise these; the default reads them
+    /// one by one with [`Storage::read_at`].
+    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
+        ranges
+            .iter()
+            .map(|r| {
+                let len = usize::try_from(r.end.saturating_sub(r.start)).map_err(|_| {
+                    FormatError::Overflow("read range longer than the address space".into())
+                })?;
+                self.read_at(r.start, len)
+            })
+            .collect()
+    }
+
+    /// The whole storage as one slice, when the backend has it in memory
+    /// (a `Vec`, an mmap). Hot loops use this to keep their zero-copy path;
+    /// `None` means every byte has to go through [`Storage::read_at`].
+    fn as_contiguous(&self) -> Option<&[u8]> {
+        None
+    }
+}
+
+impl Storage for [u8] {
+    #[inline]
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        let n = self.len();
+        let start = usize::try_from(offset).map_or(n, |o| o.min(n));
+        let end = start.saturating_add(len).min(n);
+        Ok(Cow::Borrowed(&self[start..end]))
+    }
+
+    #[inline]
+    fn len(&self) -> u64 {
+        <[u8]>::len(self) as u64
+    }
+
+    #[inline]
+    fn as_contiguous(&self) -> Option<&[u8]> {
+        Some(self)
+    }
+}
+
+impl Storage for Vec<u8> {
+    #[inline]
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        self.as_slice().read_at(offset, len)
+    }
+
+    #[inline]
+    fn len(&self) -> u64 {
+        Vec::len(self) as u64
+    }
+
+    #[inline]
+    fn as_contiguous(&self) -> Option<&[u8]> {
+        Some(self.as_slice())
+    }
+}
+
+impl<T: Storage + ?Sized> Storage for &T {
+    #[inline]
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        (**self).read_at(offset, len)
+    }
+
+    #[inline]
+    fn len(&self) -> u64 {
+        (**self).len()
+    }
+
+    #[inline]
+    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
+        (**self).read_ranges(ranges)
+    }
+
+    #[inline]
+    fn as_contiguous(&self) -> Option<&[u8]> {
+        (**self).as_contiguous()
+    }
+}
+
+impl<T: Storage + ?Sized> Storage for Box<T> {
+    #[inline]
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        (**self).read_at(offset, len)
+    }
+
+    #[inline]
+    fn len(&self) -> u64 {
+        (**self).len()
+    }
+
+    #[inline]
+    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
+        (**self).read_ranges(ranges)
+    }
+
+    #[inline]
+    fn as_contiguous(&self) -> Option<&[u8]> {
+        (**self).as_contiguous()
+    }
+}
+
+#[cfg(feature = "std")]
+impl<T: Storage + ?Sized> Storage for std::sync::Arc<T> {
+    #[inline]
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        (**self).read_at(offset, len)
+    }
+
+    #[inline]
+    fn len(&self) -> u64 {
+        (**self).len()
+    }
+
+    #[inline]
+    fn read_ranges(&self, ranges: &[Range<u64>]) -> Result<Vec<Cow<'_, [u8]>>, FormatError> {
+        (**self).read_ranges(ranges)
+    }
+
+    #[inline]
+    fn as_contiguous(&self) -> Option<&[u8]> {
+        (**self).as_contiguous()
+    }
+}
+
+/// `storage.len()` as the `usize` the parsers' end-of-file errors report
+/// (saturating on targets where the file is larger than the address space).
+#[inline]
+pub(crate) fn len_usize<S: Storage + ?Sized>(file: &S) -> usize {
+    usize::try_from(file.len()).unwrap_or(usize::MAX)
+}
+
+/// Bytes `[offset, offset + len)`, all of them.
+///
+/// A range that runs past the end of the storage is
+/// [`FormatError::UnexpectedEof`] with `expected = offset + len` and
+/// `available = storage length` — the error the `&[u8]` parsers give for
+/// the same bounds check (`offset + len > file_data.len()`).
+#[inline]
+pub fn read_exact_at<S: Storage + ?Sized>(
+    file: &S,
+    offset: u64,
+    len: usize,
+) -> Result<Cow<'_, [u8]>, FormatError> {
+    let eof = || FormatError::UnexpectedEof {
+        expected: usize::try_from(offset)
+            .unwrap_or(usize::MAX)
+            .saturating_add(len),
+        available: len_usize(file),
+    };
+    // In-memory fast path: plain slicing (for `S = [u8]` this inlines to
+    // the slice code's bounds check).
+    if let Some(all) = file.as_contiguous() {
+        return usize::try_from(offset)
+            .ok()
+            .and_then(|start| all.get(start..start.checked_add(len)?))
+            .map(Cow::Borrowed)
+            .ok_or_else(eof);
+    }
+    match offset.checked_add(len as u64) {
+        Some(end) if end <= file.len() => {}
+        _ => return Err(eof()),
+    }
+    // A short read (the storage shrank, or the backend served less inside
+    // the file) is an error: never parse a partial structure.
+    exact_len(file.read_at(offset, len)?, len)
+}
+
+/// `bytes`, the result of asking a [`Storage`] for `len` bytes, as exactly
+/// `len` bytes: a longer result (a backend that broke
+/// [`Storage::read_at`]'s contract) is cut to `len`, so bytes from outside
+/// the range asked for are never parsed or returned; a shorter one is an
+/// error (the storage shrank, or the backend failed), never a partial
+/// structure.
+#[inline]
+pub fn exact_len(bytes: Cow<'_, [u8]>, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+    match bytes.len().cmp(&len) {
+        core::cmp::Ordering::Equal => Ok(bytes),
+        core::cmp::Ordering::Less => Err(short_read()),
+        core::cmp::Ordering::Greater => Ok(match bytes {
+            Cow::Borrowed(b) => Cow::Borrowed(&b[..len]),
+            Cow::Owned(mut v) => {
+                v.truncate(len);
+                Cow::Owned(v)
+            }
+        }),
+    }
+}
+
+#[cold]
+#[inline(never)]
+fn short_read() -> FormatError {
+    FormatError::Storage(
+        "short read inside the file (the storage shrank or the backend failed)".into(),
+    )
+}
+
+/// Largest paged data block (fixed or extensible array) read in one piece.
+/// A bigger one is read as its prefix and then page by page, only the pages
+/// in use, so a block whose size fields claim more than the file holds
+/// costs no more than the pages it really has.
+pub(crate) const PAGED_BLOCK_ONE_READ_MAX: usize = 1 << 20;
+
+/// A window of the file: up to `max` bytes read at `base`, fewer only at
+/// the end of the file. Its [`Window::ensure`] reports a bounds failure
+/// exactly as the whole-file check `ensure_len(file_data, base + rel, n)`
+/// did — with the absolute position and the file's length — as long as
+/// every position checked lies within the `max` bytes the window was asked
+/// for: then a position past the window is past the end of the file.
+pub(crate) struct Window<'a> {
+    /// The bytes, from `base` on.
+    pub bytes: Cow<'a, [u8]>,
+    base: usize,
+    file_len: usize,
+}
+
+impl<'a> Window<'a> {
+    /// Read up to `max` bytes at `base`.
+    pub fn read<S: Storage + ?Sized>(
+        file: &'a S,
+        base: u64,
+        max: usize,
+    ) -> Result<Self, FormatError> {
+        Ok(Window {
+            bytes: read_upto(file, base, max)?,
+            base: usize::try_from(base).unwrap_or(usize::MAX),
+            file_len: len_usize(file),
+        })
+    }
+
+    /// A whole in-memory file as one window (base 0).
+    #[cfg(test)]
+    pub fn whole(bytes: &'a [u8]) -> Self {
+        Window {
+            bytes: Cow::Borrowed(bytes),
+            base: 0,
+            file_len: bytes.len(),
+        }
+    }
+
+    /// [`Window::ensure`] for a window at `base` that has not been read:
+    /// whether `[rel, rel + needed)` lies in the file, with the same error.
+    /// Lets a parser whose first step is to check a structure's whole extent
+    /// (a checksum at its end) fail before reading a structure that a
+    /// hostile size field has stretched past the end of the file.
+    pub fn check_extent<S: Storage + ?Sized>(
+        file: &S,
+        base: u64,
+        rel: usize,
+        needed: usize,
+    ) -> Result<(), FormatError> {
+        let base = usize::try_from(base).unwrap_or(usize::MAX);
+        let file_len = len_usize(file);
+        match base.checked_add(rel).and_then(|p| p.checked_add(needed)) {
+            Some(end) if end <= file_len => Ok(()),
+            _ => Err(FormatError::UnexpectedEof {
+                expected: base.saturating_add(rel).saturating_add(needed),
+                available: file_len,
+            }),
+        }
+    }
+
+    /// Check that `[rel, rel + needed)` (relative to `base`) is in the file.
+    #[inline]
+    pub fn ensure(&self, rel: usize, needed: usize) -> Result<(), FormatError> {
+        match rel.checked_add(needed) {
+            Some(end) if end <= self.bytes.len() => Ok(()),
+            _ => Err(FormatError::UnexpectedEof {
+                expected: self.base.saturating_add(rel).saturating_add(needed),
+                available: self.file_len,
+            }),
+        }
+    }
+}
+
+/// Up to `max` bytes from `offset` on: fewer only at the end of the
+/// storage. For structures whose size is only known once their prefix has
+/// been parsed and whose parsers bound-check what they are given.
+#[inline]
+pub fn read_upto<S: Storage + ?Sized>(
+    file: &S,
+    offset: u64,
+    max: usize,
+) -> Result<Cow<'_, [u8]>, FormatError> {
+    if let Some(all) = file.as_contiguous() {
+        let start = usize::try_from(offset).map_or(all.len(), |o| o.min(all.len()));
+        let end = start.saturating_add(max).min(all.len());
+        return Ok(Cow::Borrowed(&all[start..end]));
+    }
+    let avail = file.len().saturating_sub(offset);
+    let len = usize::try_from(avail).map_or(max, |a| a.min(max));
+    exact_len(file.read_at(offset, len)?, len)
+}
+
+/// Most stored bytes fetched by one [`Storage::read_ranges`] call when a
+/// read gathers many extents (a chunked dataset's chunks, a selection's
+/// runs): a larger read is fetched and decoded batch by batch, so a backend
+/// without the file in memory never holds more than this much undecoded
+/// data per read (or one extent, when a single one is larger — and every
+/// chunk's extent is bounded by what the chunk can need, see
+/// [`crate::filters::stored_chunk_limit`]).
+pub const RAW_BATCH_BYTES: usize = 64 << 20;
+
+/// One extent of a raw-data read: `len` bytes stored at `addr`, whose
+/// bounds are checked against the file, of which the first `fetch` bytes
+/// are read (`None`: only checked, not read — its bytes are not needed).
+///
+/// `fetch` below `len` bounds what a crafted size field can make a read
+/// fetch: a chunk never needs more of its stored bytes than its decoded
+/// size allows, however large its index entry says it is.
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct ExtentReq {
+    pub addr: u64,
+    pub len: usize,
+    pub fetch: Option<usize>,
+}
+
+impl ExtentReq {
+    /// How many bytes are read for this extent.
+    #[inline]
+    fn fetch_len(&self) -> usize {
+        self.fetch.map_or(0, |f| f.min(self.len))
+    }
+}
+
+/// One extent's bytes on their own (see [`ExtentReq`]): the whole extent's
+/// bounds checked as [`read_exact_at`] checks them, and its first
+/// `req.fetch` bytes read (none when `fetch` is `None`).
+pub(crate) fn read_extent<'a, S: Storage + ?Sized>(
+    file: &'a S,
+    req: &ExtentReq,
+) -> Result<Cow<'a, [u8]>, FormatError> {
+    let start = usize::try_from(req.addr).unwrap_or(usize::MAX);
+    match start.checked_add(req.len) {
+        Some(end) if end <= len_usize(file) => read_exact_at(file, req.addr, req.fetch_len()),
+        _ => Err(FormatError::UnexpectedEof {
+            expected: start.saturating_add(req.len),
+            available: len_usize(file),
+        }),
+    }
+}
+
+/// The stored bytes of one batch of extents (chunks, contiguous runs),
+/// fetched together: [`Storage::read_ranges`] is called once for the batch,
+/// so a remote backend can coalesce and parallelise the requests. See
+/// [`for_each_extent_batch`], which is how every raw-data read gets them.
+///
+/// With the whole file in memory nothing is fetched: [`Self::get`] slices
+/// it, as the slice readers did. Either way an extent that does not lie in
+/// the file is the error the slice readers gave for it
+/// ([`FormatError::UnexpectedEof`] with its end and the file length, or
+/// [`FormatError::Overflow`] for an address past this platform's `usize`),
+/// reported when that extent is asked for — so a read reports the first
+/// failing extent in its own order, whatever fails after it.
+pub(crate) enum ExtentBytes<'a> {
+    /// The whole file.
+    Contiguous(&'a [u8]),
+    /// Each extent's bytes, or its bounds error; the first is extent
+    /// `base` of the read.
+    Fetched {
+        base: usize,
+        extents: Vec<Extent<'a>>,
+    },
+}
+
+/// One extent of [`ExtentBytes::Fetched`].
+pub(crate) enum Extent<'a> {
+    /// Its bytes.
+    Bytes(Cow<'a, [u8]>),
+    /// In the file, but not fetched (the caller did not want its bytes).
+    NotFetched,
+    /// The error reading it gives.
+    Err(FormatError),
+}
+
+impl<'a> ExtentBytes<'a> {
+    /// Fetch `reqs`, extents `base..base + reqs.len()` of the read: the
+    /// bytes of those wanted, and the bounds check of all of them.
+    pub(crate) fn fetch<S: Storage + ?Sized>(
+        file: &'a S,
+        reqs: &[ExtentReq],
+        base: usize,
+    ) -> Result<Self, FormatError> {
+        if let Some(all) = file.as_contiguous() {
+            return Ok(ExtentBytes::Contiguous(all));
+        }
+        let file_len = len_usize(file);
+        let mut ranges = Vec::new();
+        let mut out = Vec::with_capacity(reqs.len());
+        // Positions in `out` of the extents being read, in `ranges` order.
+        let mut slots = Vec::new();
+        for req in reqs {
+            let checked = crate::addr::to_usize(req.addr).and_then(|start| {
+                match start.checked_add(req.len) {
+                    Some(end) if end <= file_len => Ok(()),
+                    _ => Err(FormatError::UnexpectedEof {
+                        expected: start.saturating_add(req.len),
+                        available: file_len,
+                    }),
+                }
+            });
+            match checked {
+                Ok(()) if req.fetch.is_some() => {
+                    slots.push(out.len());
+                    ranges.push(req.addr..req.addr + req.fetch_len() as u64);
+                    out.push(Extent::NotFetched);
+                }
+                Ok(()) => out.push(Extent::NotFetched),
+                Err(e) => out.push(Extent::Err(e)),
+            }
+        }
+        if !ranges.is_empty() {
+            let got = file.read_ranges(&ranges)?;
+            if got.len() != ranges.len() {
+                return Err(FormatError::Storage(
+                    "read_ranges returned the wrong number of ranges".into(),
+                ));
+            }
+            for ((slot, bytes), r) in slots.into_iter().zip(got).zip(&ranges) {
+                let len = crate::addr::saturating_usize(r.end - r.start);
+                out[slot] = Extent::Bytes(exact_len(bytes, len)?);
+            }
+        }
+        Ok(ExtentBytes::Fetched { base, extents: out })
+    }
+
+    /// Whether extent `i` of the read (`req`) lies in the file: its bounds
+    /// error if not.
+    pub(crate) fn check(&self, i: usize, req: &ExtentReq) -> Result<(), FormatError> {
+        match self {
+            ExtentBytes::Contiguous(_) => self.get(i, req).map(|_| ()),
+            ExtentBytes::Fetched { base, extents } => {
+                match i.checked_sub(*base).and_then(|j| extents.get(j)) {
+                    Some(Extent::Err(e)) => Err(e.clone()),
+                    Some(_) => Ok(()),
+                    None => Err(not_fetched()),
+                }
+            }
+        }
+    }
+
+    /// Extent `i` of the read (`req`): its first `req.fetch` bytes, the
+    /// same whether the file is in memory or not.
+    pub(crate) fn get(&self, i: usize, req: &ExtentReq) -> Result<&[u8], FormatError> {
+        match self {
+            ExtentBytes::Contiguous(all) => {
+                let start = crate::addr::to_usize(req.addr)?;
+                start
+                    .checked_add(req.len)
+                    .and_then(|end| all.get(start..end))
+                    .map(|b| &b[..req.fetch_len()])
+                    .ok_or(FormatError::UnexpectedEof {
+                        expected: start.saturating_add(req.len),
+                        available: <[u8]>::len(all),
+                    })
+            }
+            ExtentBytes::Fetched { base, extents } => {
+                match i.checked_sub(*base).and_then(|j| extents.get(j)) {
+                    Some(Extent::Bytes(b)) => Ok(b),
+                    Some(Extent::Err(e)) => Err(e.clone()),
+                    _ => Err(not_fetched()),
+                }
+            }
+        }
+    }
+}
+
+#[cold]
+fn not_fetched() -> FormatError {
+    FormatError::Storage("an extent that was not fetched was asked for".into())
+}
+
+/// The one way raw-data reads fetch stored bytes: `reqs` are split into
+/// consecutive batches of at most [`RAW_BATCH_BYTES`] of fetched bytes (at
+/// least one extent each — and no extent fetches more than its
+/// [`ExtentReq::fetch`]), and for each batch in turn its bytes are fetched
+/// with one [`Storage::read_ranges`] call and `f(batch, &bytes)` is called,
+/// with `bytes` indexed by the extent's position in `reqs`. A batch's bytes
+/// are dropped before the next batch is fetched, and an error from `f`
+/// stops the read before anything more is fetched.
+///
+/// With the whole file in memory there is nothing to fetch: one call, over
+/// all of `reqs`, that slices the file.
+pub(crate) fn for_each_extent_batch<'a, S: Storage + ?Sized>(
+    file: &'a S,
+    reqs: &[ExtentReq],
+    mut f: impl FnMut(Range<usize>, &ExtentBytes<'a>) -> Result<(), FormatError>,
+) -> Result<(), FormatError> {
+    let contiguous = file.as_contiguous().is_some();
+    for batch in raw_batches(reqs.len(), contiguous, |i| reqs[i].fetch_len()) {
+        let bytes = ExtentBytes::fetch(file, &reqs[batch.clone()], batch.start)?;
+        f(batch, &bytes)?;
+    }
+    Ok(())
+}
+
+/// Split `n` extents, whose sizes `size(i)` gives, into consecutive batches
+/// of at most [`RAW_BATCH_BYTES`] (at least one extent each): the ranges of
+/// `0..n` to fetch together. With the whole file in memory (`contiguous`)
+/// there is nothing to fetch, and one batch.
+pub(crate) fn raw_batches(
+    n: usize,
+    contiguous: bool,
+    size: impl Fn(usize) -> usize,
+) -> Vec<Range<usize>> {
+    if contiguous || n == 0 {
+        return core::iter::once(0..n).collect();
+    }
+    let mut out = Vec::new();
+    let (mut start, mut bytes) = (0, 0usize);
+    for i in 0..n {
+        let s = size(i);
+        if i > start && bytes.saturating_add(s) > RAW_BATCH_BYTES {
+            out.push(start..i);
+            start = i;
+            bytes = 0;
+        }
+        bytes = bytes.saturating_add(s);
+    }
+    out.push(start..n);
+    out
+}
+
+/// Borrow the whole file for a code path that has not been converted to
+/// [`Storage`] yet. On a backend without a contiguous view this is the
+/// clean [`FormatError::ContiguousStorageRequired`] error, never a guess.
+#[inline]
+pub fn require_contiguous<'a, S: Storage + ?Sized>(
+    file: &'a S,
+    what: &'static str,
+) -> Result<&'a [u8], FormatError> {
+    file.as_contiguous()
+        .ok_or(FormatError::ContiguousStorageRequired(what))
+}
+
+/// A [`Storage`] over an in-memory buffer that serves every byte through
+/// [`Storage::read_at`] (its [`Storage::as_contiguous`] is `None`, so no
+/// parser can take the whole-slice shortcut), copies what it serves (as a
+/// remote backend would), and counts the reads and bytes.
+///
+/// It is the equivalence harness of the range-read migration: parsing a
+/// file through it must give exactly what parsing the `&[u8]` gives, and
+/// the counters are the request counts a cacheless range reader would make.
+#[derive(Debug)]
+pub struct CountingStorage {
+    data: Vec<u8>,
+    reads: portable_atomic::AtomicU64,
+    bytes: portable_atomic::AtomicU64,
+}
+
+impl CountingStorage {
+    /// Serve `data` (the file from the superblock on).
+    pub fn new(data: Vec<u8>) -> Self {
+        CountingStorage {
+            data,
+            reads: portable_atomic::AtomicU64::new(0),
+            bytes: portable_atomic::AtomicU64::new(0),
+        }
+    }
+
+    /// Number of `read_at` calls served so far.
+    pub fn reads(&self) -> u64 {
+        self.reads.load(portable_atomic::Ordering::Relaxed)
+    }
+
+    /// Number of bytes served so far.
+    pub fn bytes_read(&self) -> u64 {
+        self.bytes.load(portable_atomic::Ordering::Relaxed)
+    }
+
+    /// Reset both counters.
+    pub fn reset(&self) {
+        self.reads.store(0, portable_atomic::Ordering::Relaxed);
+        self.bytes.store(0, portable_atomic::Ordering::Relaxed);
+    }
+}
+
+impl Storage for CountingStorage {
+    fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
+        let got = self.data.as_slice().read_at(offset, len)?;
+        self.reads.fetch_add(1, portable_atomic::Ordering::Relaxed);
+        self.bytes
+            .fetch_add(got.len() as u64, portable_atomic::Ordering::Relaxed);
+        Ok(Cow::Owned(got.into_owned()))
+    }
+
+    fn len(&self) -> u64 {
+        self.data.len() as u64
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn slice_reads_are_borrowed_and_clamped() {
+        let data: Vec<u8> = (0u8..10).collect();
+        let s: &[u8] = &data;
+        let dynamic: &dyn Storage = &s;
+        assert_eq!(dynamic.len(), 10);
+        let r = dynamic.read_at(2, 3).unwrap();
+        assert!(matches!(r, Cow::Borrowed(_)));
+        assert_eq!(&*r, &[2, 3, 4]);
+        assert_eq!(&*dynamic.read_at(8, 5).unwrap(), &[8, 9]);
+        assert!(dynamic.read_at(10, 5).unwrap().is_empty());
+        assert!(dynamic.read_at(u64::MAX, 5).unwrap().is_empty());
+        assert_eq!(dynamic.as_contiguous(), Some(&data[..]));
+        let v: &dyn Storage = &data;
+        assert_eq!(v.as_contiguous(), Some(&data[..]));
+    }
+
+    #[test]
+    fn read_exact_matches_slice_bounds_errors() {
+        let data = [0u8; 10];
+        let s: &[u8] = &data;
+        assert_eq!(&*read_exact_at(&s, 4, 6).unwrap(), &[0; 6]);
+        assert_eq!(
+            read_exact_at(&s, 4, 7).unwrap_err(),
+            FormatError::UnexpectedEof {
+                expected: 11,
+                available: 10
+            }
+        );
+        assert!(read_exact_at(&s, u64::MAX, 1).is_err());
+        assert_eq!(read_upto(&s, 7, 100).unwrap().len(), 3);
+        assert_eq!(read_upto(&s, 70, 100).unwrap().len(), 0);
+    }
+
+    #[test]
+    fn counting_storage_counts_and_hides_the_slice() {
+        let c = CountingStorage::new((0u8..10).collect());
+        assert!(c.as_contiguous().is_none());
+        let r = c.read_at(3, 4).unwrap();
+        assert!(matches!(r, Cow::Owned(_)));
+        assert_eq!(&*r, &[3, 4, 5, 6]);
+        c.read_at(8, 4).unwrap();
+        assert_eq!((c.reads(), c.bytes_read()), (2, 6));
+        c.reset();
+        assert_eq!((c.reads(), c.bytes_read()), (0, 0));
+    }
+
+    #[test]
+    fn read_ranges_default_loops() {
+        let data: Vec<u8> = (0u8..10).collect();
+        let s: &[u8] = &data;
+        let got = s.read_ranges(&[1..3, 5..9]).unwrap();
+        assert_eq!(&*got[0], &[1, 2]);
+        assert_eq!(&*got[1], &[5, 6, 7, 8]);
+    }
+}

crates/clawhdf5-format/src/superblock.rs

@@ -7,6 +7,11 @@ use byteorder::{ByteOrder, LittleEndian};
 
 use crate::error::FormatError;
 use crate::signature::HDF5_SIGNATURE;
+use crate::storage::{Storage, read_upto};
+
+/// Bytes read to parse a superblock: more than the largest one (version 1
+/// with 8-byte offsets and lengths, 100 bytes).
+const SUPERBLOCK_READ_LEN: usize = 128;
 
 /// Parsed HDF5 superblock (all versions).
 #[derive(Debug, Clone, PartialEq, Eq)]
@@ -100,6 +105,42 @@ pub mod swmr_flags {
 }
 
 impl Superblock {
+    /// Where the HDF5 data ends, relative to the superblock, for a file of
+    /// `file_len` bytes whose superblock is at `user_block` (both counted
+    /// from the start of the file), with libhdf5's truncation check
+    /// (`H5F__super_read`).
+    ///
+    /// The superblock records the end of the file's data as an absolute
+    /// address. A file shorter than that was truncated, and libhdf5 refuses
+    /// to open it ("truncated file"); so does this, with
+    /// [`FormatError::TruncatedFile`]. Bytes past that address are not part
+    /// of the file: libhdf5 fails any read of them ("addr overflow" /
+    /// "address plus size exceeds file eoa"), so a reader should parse only
+    /// the data up to the returned end. As libhdf5 does for a SWMR reader,
+    /// the check is skipped for a version-3 superblock whose writer is still
+    /// writing it in SWMR mode (it extends the file as it goes); the data
+    /// then ends at the end of the file.
+    ///
+    /// When the superblock's recorded base address differs from where the
+    /// superblock actually is (a user block added or removed after the file
+    /// was written), libhdf5 moves the recorded end of file by the same
+    /// amount, and so does this.
+    pub fn data_end(&self, user_block: u64, file_len: u64) -> Result<u64, FormatError> {
+        let eof =
+            i128::from(self.eof_address) - i128::from(self.base_address) + i128::from(user_block);
+        if eof < 0 || eof > i128::from(file_len) {
+            if self.version >= 3 && self.is_swmr_write() {
+                return Ok(file_len.saturating_sub(user_block));
+            }
+            return Err(FormatError::TruncatedFile {
+                stored_eof: u64::try_from(eof).unwrap_or(self.eof_address),
+                actual_len: file_len,
+            });
+        }
+        // 0 <= eof <= file_len, so it fits a u64.
+        Ok((eof as u64).saturating_sub(user_block))
+    }
+
     /// Whether the file was opened with write access when the superblock was written.
     pub fn is_write_access(&self) -> bool {
         self.consistency_flags & swmr_flags::WRITE_ACCESS != 0
@@ -125,7 +166,16 @@ impl Superblock {
         file_data: &[u8],
         signature_offset: usize,
     ) -> Result<u64, FormatError> {
-        let refreshed = Superblock::parse(file_data, signature_offset)?;
+        self.refresh_eof_in(file_data, signature_offset as u64)
+    }
+
+    /// [`Self::refresh_eof`] over any [`Storage`].
+    pub fn refresh_eof_in<S: Storage + ?Sized>(
+        &mut self,
+        file: &S,
+        signature_offset: u64,
+    ) -> Result<u64, FormatError> {
+        let refreshed = Superblock::parse_in(file, signature_offset)?;
         self.eof_address = refreshed.eof_address;
         self.consistency_flags = refreshed.consistency_flags;
         Ok(self.eof_address)
@@ -174,14 +224,32 @@ impl Superblock {
 
     /// Parse a superblock from `data` starting at `signature_offset`.
     ///
-    /// The signature must be present at the given offset.
+    /// The signature must be present at the given offset, and that offset
+    /// must be 0: every address in an HDF5 file is relative to the
+    /// superblock, so when a file has a user block (signature at 512, 1024,
+    /// …) the caller must pass the bytes from the signature on — see
+    /// [`crate::signature::split_user_block`] — and use that slice as
+    /// `file_data` everywhere. A non-zero offset is refused with
+    /// [`FormatError::UserBlockNotStripped`] because the addresses in the
+    /// returned superblock would otherwise be applied to the wrong bytes.
     pub fn parse(data: &[u8], signature_offset: usize) -> Result<Superblock, FormatError> {
-        let d = data
+        Self::parse_in(data, signature_offset as u64)
-            .get(signature_offset..)
+    }
-            .ok_or(FormatError::UnexpectedEof {
+
-                expected: signature_offset + 1,
+    /// [`Self::parse`] over any [`Storage`]: one read of the first
-                available: data.len(),
+    /// [`SUPERBLOCK_READ_LEN`] bytes (fewer when the file is shorter, which
-            })?;
+    /// is then refused with the same end-of-file errors as a short slice).
+    pub fn parse_in<S: Storage + ?Sized>(
+        file: &S,
+        signature_offset: u64,
+    ) -> Result<Superblock, FormatError> {
+        if signature_offset != 0 {
+            return Err(FormatError::UserBlockNotStripped(signature_offset));
+        }
+        // Every bounds check below needs at most 100 bytes, so on a longer
+        // file none of them can fail and the window's length does not show.
+        let window = read_upto(file, 0, SUPERBLOCK_READ_LEN)?;
+        let d: &[u8] = &window;
         ensure_len(d, 9)?; // signature(8) + version(1)
 
         // Verify signature
@@ -527,6 +595,30 @@ mod tests {
         buf
     }
 
+    #[test]
+    fn data_end_refuses_truncated_files_like_libhdf5() {
+        // build_v2_bytes records base 0, end of file 2048.
+        let sb = Superblock::parse(&build_v2_bytes(8, 2), 0).unwrap();
+        assert_eq!(sb.data_end(0, 2048), Ok(2048));
+        // Bytes past the recorded end are not part of the file.
+        assert_eq!(sb.data_end(0, 4096), Ok(2048));
+        assert_eq!(
+            sb.data_end(0, 2047),
+            Err(FormatError::TruncatedFile {
+                stored_eof: 2048,
+                actual_len: 2047
+            })
+        );
+        // A user block added in front after the file was written (the
+        // recorded base address is still 0): the end moves with it.
+        assert_eq!(sb.data_end(512, 2560), Ok(2048));
+        assert!(sb.data_end(512, 2559).is_err());
+        // A v3 superblock still being written in SWMR mode is not checked.
+        let mut swmr = Superblock::parse(&build_v2_bytes(8, 3), 0).unwrap();
+        swmr.consistency_flags = swmr_flags::WRITE_ACCESS | swmr_flags::SWMR_WRITE;
+        assert_eq!(swmr.data_end(0, 1000), Ok(1000));
+    }
+
     #[test]
     fn parse_v0_8byte_offsets() {
         let data = build_v0_bytes(8);
@@ -676,7 +768,16 @@ mod tests {
         let mut data = vec![0u8; 1024];
         let v0 = build_v0_bytes(8);
         data[512..512 + v0.len()].copy_from_slice(&v0);
-        let sb = Superblock::parse(&data, 512).unwrap();
+        // Addresses are relative to the superblock, so parsing in place
+        // (where they would be applied to the whole buffer) is refused...
+        assert_eq!(
+            Superblock::parse(&data, 512),
+            Err(FormatError::UserBlockNotStripped(512))
+        );
+        // ...and the caller parses the bytes from the signature on.
+        let (ub, hdf5) = crate::signature::split_user_block(&data).unwrap();
+        assert_eq!(ub.len(), 512);
+        let sb = Superblock::parse(hdf5, 0).unwrap();
         assert_eq!(sb.version, 0);
         assert_eq!(sb.root_group_address, 96);
     }
@@ -815,4 +916,34 @@ mod tests {
         assert_eq!(parsed.version, 3);
         assert_eq!(parsed.page_size, None);
     }
+
+    /// Through a storage that serves only `read_at`, every version parses
+    /// to the same superblock, and every truncation to the same error, as
+    /// from a slice — in one read.
+    #[test]
+    fn parse_in_matches_slice_parse() {
+        use crate::storage::CountingStorage;
+        let mut files = vec![
+            build_v0_bytes(8),
+            build_v0_bytes(4),
+            build_v1_bytes(8),
+            build_v1_bytes(4),
+            build_v2_bytes(8, 2),
+            build_v2_bytes(4, 3),
+        ];
+        for f in files.clone() {
+            let mut long = f.clone();
+            long.resize(4096, 0xAB);
+            files.push(long);
+            for cut in [0, 5, 9, 13, 20, 30, f.len() - 1] {
+                files.push(f[..cut.min(f.len())].to_vec());
+            }
+        }
+        for f in files {
+            let want = Superblock::parse(&f, 0);
+            let storage = CountingStorage::new(f.clone());
+            assert_eq!(Superblock::parse_in(&storage, 0), want, "{} bytes", f.len());
+            assert_eq!(storage.reads(), 1);
+        }
+    }
 }

crates/clawhdf5-format/src/superblock_ext.rs

@@ -0,0 +1,922 @@
+//! The superblock extension of a version 2 or 3 superblock, and the
+//! metadata cache image it can point to.
+//!
+//! libhdf5 reads the extension when it opens a file (`H5F__super_read`) and
+//! decodes the messages that configure the file: v1 B-tree "K" values, File
+//! Space Info, and the Metadata Cache Image. A message that does not decode
+//! makes the file fail to open, so [`read_superblock_extension`] decodes and
+//! checks them the way libhdf5 does.
+//!
+//! A metadata cache image (written with `H5Pset_mdc_image_config`) is a
+//! block holding serialized metadata cache entries — object headers, B-tree
+//! nodes, heaps — each with its file address. libhdf5 loads it into its
+//! cache before it reads any other metadata (`H5C__load_cache_image`,
+//! `H5C__reconstruct_cache_contents`), and the entries take the place of
+//! the file's bytes at their addresses: the file itself may hold stale or
+//! no metadata there (in `h5clear_mdc_image.h5` the root group's header is
+//! only in the image). [`CacheImage::apply`] does the same with bytes: it
+//! writes every entry at its address, so every parser reads what libhdf5
+//! reads. It writes into whatever the opener gives it — a private
+//! copy-on-write mapping of the file, or a buffer the opener owns — so the
+//! file is never copied whole.
+
+#[cfg(not(feature = "std"))]
+use alloc::{borrow::Cow, collections::BTreeSet, vec::Vec};
+#[cfg(feature = "std")]
+use std::{borrow::Cow, collections::BTreeSet};
+
+use crate::error::FormatError;
+use crate::message_type::MessageType;
+use crate::object_header::ObjectHeader;
+use crate::storage::{Storage, read_exact_at};
+use crate::superblock::Superblock;
+
+/// Message type of the File Space Info message.
+const MSG_FSINFO: u16 = 0x0017;
+/// Message type of the Metadata Cache Image message.
+const MSG_MDCI: u16 = 0x0018;
+/// Header message flag: the library did not know the message when it wrote
+/// it back (`H5O_MSG_FLAG_WAS_UNKNOWN`); libhdf5 then ignores its contents.
+const MSG_FLAG_WAS_UNKNOWN: u8 = 0x20;
+
+/// `H5F_FILE_SPACE_PAGE_SIZE_MIN` / `_MAX`.
+const PAGE_SIZE_MIN: u64 = 512;
+const PAGE_SIZE_MAX: u64 = 1024 * 1024 * 1024;
+/// libhdf5's default file space page size, used for a version 0 message.
+const PAGE_SIZE_DEFAULT: u64 = 4096;
+/// Free-space managers whose addresses a persisting version 1 File Space
+/// Info message lists (`H5F_MEM_PAGE_SUPER` .. `H5F_MEM_PAGE_NTYPES`), and
+/// a version 0 one (`H5FD_MEM_SUPER` .. `H5FD_MEM_NTYPES`).
+const FSM_ADDRS_V1: usize = 12;
+const FSM_ADDRS_V0: usize = 6;
+
+/// Metadata cache image block limits (`H5Cimage.c`, `H5ACprivate.h`).
+const MDCI_SIGNATURE: &[u8; 4] = b"MDCI";
+const MDCI_HAVE_RESIZE_STATUS: u8 = 0x01;
+const MDCI_ENTRY_IS_FD_PARENT: u8 = 0x04;
+const MDCI_ENTRY_IS_FD_CHILD: u8 = 0x08;
+/// `H5AC_NTYPES`: entry type ids are below this.
+const MDCI_NTYPES: u8 = 30;
+/// `H5C_RING_NTYPES`.
+const MDCI_RING_NTYPES: u8 = 6;
+/// `H5AC__CACHE_IMAGE__ENTRY_AGEOUT__MAX`.
+const MDCI_AGE_MAX: u8 = 100;
+
+/// A decoded File Space Info message (0x0017), mapped to version 1 as
+/// libhdf5 maps a version 0 one.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct FileSpaceInfo {
+    /// Message version as stored (0 or 1).
+    pub version: u8,
+    /// File space strategy (`H5F_fspace_strategy_t`).
+    pub strategy: u8,
+    /// Whether free space is persisted.
+    pub persist: bool,
+    /// Free-space section threshold.
+    pub threshold: u64,
+    /// File space page size.
+    pub page_size: u64,
+}
+
+/// Where a metadata cache image block is (Metadata Cache Image message,
+/// 0x0018).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct CacheImageLocation {
+    /// Address of the image block.
+    pub address: u64,
+    /// Length of the image block in bytes.
+    pub length: u64,
+}
+
+/// The messages of a superblock extension that libhdf5 decodes at open.
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct SuperblockExtension {
+    /// v1 B-tree "K" values (chunk index, symbol table node, symbol table
+    /// leaf), when the extension overrides the defaults.
+    pub btree_k: Option<(u16, u16, u16)>,
+    /// The File Space Info message.
+    pub file_space_info: Option<FileSpaceInfo>,
+    /// The metadata cache image, when the file has one.
+    pub cache_image: Option<CacheImageLocation>,
+}
+
+fn ext_err(why: &'static str) -> FormatError {
+    FormatError::InvalidSuperblockExtension(why)
+}
+
+const RAN_OFF: &str = "ran off end of input buffer while decoding";
+
+/// A little-endian cursor over one message or block, failing with
+/// `overrun` when it runs off the end.
+struct Cursor<'a> {
+    data: &'a [u8],
+    pos: usize,
+    overrun: FormatError,
+}
+
+impl<'a> Cursor<'a> {
+    fn new(data: &'a [u8], overrun: FormatError) -> Self {
+        Cursor {
+            data,
+            pos: 0,
+            overrun,
+        }
+    }
+
+    fn take(&mut self, n: usize) -> Result<&'a [u8], FormatError> {
+        let end = self
+            .pos
+            .checked_add(n)
+            .filter(|&e| e <= self.data.len())
+            .ok_or_else(|| self.overrun.clone())?;
+        let s = &self.data[self.pos..end];
+        self.pos = end;
+        Ok(s)
+    }
+
+    fn u8(&mut self) -> Result<u8, FormatError> {
+        Ok(self.take(1)?[0])
+    }
+
+    fn uint(&mut self, width: u8) -> Result<u64, FormatError> {
+        let b = self.take(width as usize)?;
+        Ok(b.iter()
+            .rev()
+            .fold(0u64, |acc, &x| (acc << 8) | u64::from(x)))
+    }
+
+    /// An address of `width` bytes; `None` when undefined (all ones).
+    fn addr(&mut self, width: u8) -> Result<Option<u64>, FormatError> {
+        let v = self.uint(width)?;
+        let undef = if width >= 8 {
+            u64::MAX
+        } else {
+            (1u64 << (8 * u32::from(width))) - 1
+        };
+        Ok((v != undef).then_some(v))
+    }
+}
+
+/// Decode and check the superblock extension of `sb`, as libhdf5 does when
+/// it opens the file. `data` is the file from the superblock on, up to the
+/// end of file the superblock records (its end is libhdf5's "eoa").
+///
+/// Returns `Ok(None)` for a superblock without an extension (versions 0
+/// and 1 have none). A message libhdf5 fails to decode, or a cache image
+/// that does not lie inside the file, is an error: libhdf5 refuses to open
+/// such a file (`cve-2020-10810`: a File Space Info message too short for
+/// the free-space manager addresses it announces; `cve-2020-10812`: a cache
+/// image past the end of the file).
+pub fn read_superblock_extension(
+    data: &[u8],
+    sb: &Superblock,
+) -> Result<Option<SuperblockExtension>, FormatError> {
+    read_superblock_extension_in(data, sb)
+}
+
+/// [`read_superblock_extension`] over any [`Storage`]; its length is the
+/// end of file.
+pub fn read_superblock_extension_in<S: Storage + ?Sized>(
+    file: &S,
+    sb: &Superblock,
+) -> Result<Option<SuperblockExtension>, FormatError> {
+    let os = sb.offset_size;
+    let ls = sb.length_size;
+    let undef = if os >= 8 {
+        u64::MAX
+    } else {
+        (1u64 << (8 * u32::from(os))) - 1
+    };
+    let Some(addr) = sb.superblock_extension_address.filter(|&a| a != undef) else {
+        return Ok(None);
+    };
+    let addr = usize::try_from(addr).map_err(|_| ext_err("address out of range"))?;
+    let header = ObjectHeader::parse_in(file, addr as u64, os, ls)?;
+    let eoa = file.len();
+
+    let mut ext = SuperblockExtension::default();
+    for msg in &header.messages {
+        match msg.msg_type {
+            MessageType::BTreeKValues => {
+                let mut c = Cursor::new(&msg.data, ext_err(RAN_OFF));
+                if c.u8()? != 0 {
+                    return Err(ext_err("bad version number for v1 B-tree 'K' message"));
+                }
+                let chunk = c.uint(2)? as u16;
+                let snode = c.uint(2)? as u16;
+                let leaf = c.uint(2)? as u16;
+                ext.btree_k = Some((chunk, snode, leaf));
+            }
+            MessageType::Unknown(MSG_FSINFO) if msg.flags & MSG_FLAG_WAS_UNKNOWN == 0 => {
+                ext.file_space_info = Some(decode_fsinfo(&msg.data, os, ls)?);
+            }
+            MessageType::Unknown(MSG_MDCI) => {
+                let mut c = Cursor::new(&msg.data, ext_err(RAN_OFF));
+                if c.u8()? != 0 {
+                    return Err(ext_err(
+                        "bad version number for metadata cache image message",
+                    ));
+                }
+                let address = c.addr(os)?;
+                let length = c.uint(ls)?;
+                let Some(address) = address else {
+                    return Err(ext_err("metadata cache image address is undefined"));
+                };
+                if address.checked_add(length).is_none_or(|end| end > eoa) {
+                    return Err(ext_err(
+                        "metadata cache image: address plus size exceeds file eoa",
+                    ));
+                }
+                ext.cache_image = Some(CacheImageLocation { address, length });
+            }
+            _ => {}
+        }
+    }
+    Ok(Some(ext))
+}
+
+/// `H5O__fsinfo_decode` plus the checks `H5F__super_read` makes on it.
+fn decode_fsinfo(data: &[u8], os: u8, ls: u8) -> Result<FileSpaceInfo, FormatError> {
+    let mut c = Cursor::new(data, ext_err(RAN_OFF));
+    let version = c.u8()?;
+    let info = if version == 0 {
+        let old_strategy = c.u8()?;
+        let threshold = c.uint(ls)?;
+        // H5F_file_space_type_t: 1 ALL_PERSIST, 2 ALL, 3 AGGR_VFD, 4 VFD.
+        let (strategy, persist) = match old_strategy {
+            1 => {
+                for _ in 0..FSM_ADDRS_V0 {
+                    c.addr(os)?;
+                }
+                (0, true)
+            }
+            2 => (0, false),
+            3 => (2, false),
+            4 => (3, false),
+            _ => return Err(ext_err("invalid file space strategy")),
+        };
+        FileSpaceInfo {
+            version,
+            strategy,
+            persist,
+            threshold,
+            page_size: PAGE_SIZE_DEFAULT,
+        }
+    } else {
+        if version > 1 {
+            return Err(ext_err("File space info message's version out of bounds"));
+        }
+        let strategy = c.u8()?;
+        let persist = c.u8()? != 0;
+        let threshold = c.uint(ls)?;
+        let page_size = c.uint(ls)?;
+        if page_size == 0 || page_size > PAGE_SIZE_MAX {
+            return Err(ext_err("invalid page size in file space info"));
+        }
+        c.uint(2)?; // page end metadata threshold
+        c.addr(os)?; // EOA before the free-space managers
+        if persist {
+            for _ in 0..FSM_ADDRS_V1 {
+                c.addr(os)?;
+            }
+        }
+        FileSpaceInfo {
+            version,
+            strategy,
+            persist,
+            threshold,
+            page_size,
+        }
+    };
+    if info.page_size < PAGE_SIZE_MIN {
+        return Err(ext_err("file space page size too small"));
+    }
+    Ok(info)
+}
+
+/// One entry of a metadata cache image: `len` bytes at `image_offset` in
+/// the image block, belonging at file address `address`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct ImageEntry {
+    address: u64,
+    image_offset: usize,
+    len: usize,
+}
+
+/// A decoded metadata cache image: where its block is, and the entries it
+/// holds. [`CacheImage::apply`] writes the entries over a file's bytes.
+///
+/// Only the entry list is kept, never a copy of the file: an opener that
+/// maps the file applies the image to a private copy-on-write mapping, so
+/// only the pages the entries land on are copied.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CacheImage {
+    location: CacheImageLocation,
+    entries: Vec<ImageEntry>,
+}
+
+/// What an opener must do about a file's metadata cache image.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum CacheImageState {
+    /// The file has no image: its bytes are its metadata.
+    Absent,
+    /// The file has an image that loads: apply it with [`CacheImage::apply`].
+    Loaded(CacheImage),
+    /// The file has an image libhdf5 fails to load. libhdf5 still opens the
+    /// file (the image loads at the first metadata read), and that read
+    /// fails with this error.
+    Unloadable(FormatError),
+}
+
+impl CacheImage {
+    /// Decode the metadata cache image at `location` in `data` (the file
+    /// from the superblock on, up to its recorded end of file). The image is
+    /// checked as libhdf5 checks it (`H5C__decode_cache_image_header`,
+    /// `H5C__reconstruct_cache_entry`): signature and version, the image
+    /// length it records, entry types, rings and ages in range, entry
+    /// addresses inside the file and not repeated, flush-dependency parents
+    /// already in the cache.
+    ///
+    /// One check is stricter than libhdf5's: an entry must end inside the
+    /// file. libhdf5 checks only that it starts there, and serves the rest
+    /// from the image; the images libhdf5 writes never do this (every entry
+    /// lies below the image block, which is written last), and the bytes an
+    /// entry would put past the end of file have nowhere to go in a view of
+    /// the file.
+    ///
+    /// libhdf5 does not verify the block's trailing checksum when it loads
+    /// an image, so neither does this.
+    pub fn decode(
+        data: &[u8],
+        location: CacheImageLocation,
+        sb: &Superblock,
+    ) -> Result<Self, FormatError> {
+        Self::decode_in(data, location, sb)
+    }
+
+    /// [`Self::decode`] over any [`Storage`]: one read of the image block.
+    pub fn decode_in<S: Storage + ?Sized>(
+        file: &S,
+        location: CacheImageLocation,
+        sb: &Superblock,
+    ) -> Result<Self, FormatError> {
+        let (offset_size, length_size) = (sb.offset_size, sb.length_size);
+        let bad = FormatError::InvalidCacheImage;
+        let block = image_block_in(file, location)?;
+        let eoa = file.len();
+        let mut c = Cursor::new(&block, bad(RAN_OFF));
+
+        // Header: signature, version, flags, image data length, entry count.
+        if c.take(4)? != MDCI_SIGNATURE {
+            return Err(bad("bad metadata cache image header signature"));
+        }
+        if c.u8()? != 0 {
+            return Err(bad("bad metadata cache image version"));
+        }
+        if c.u8()? & MDCI_HAVE_RESIZE_STATUS != 0 {
+            return Err(bad("MDC resize status not yet supported"));
+        }
+        if c.uint(length_size)? != location.length {
+            return Err(bad("bad metadata cache image data length"));
+        }
+        let n_entries = c.uint(4)?;
+        if n_entries == 0 {
+            return Err(bad("bad metadata cache entry count"));
+        }
+
+        let mut entries = Vec::new();
+        // What is in libhdf5's cache when it loads the image: the superblock
+        // and the superblock extension's object header (read to find the
+        // image). Each entry's flush-dependency parents are looked up in the
+        // cache as the entry is inserted (`H5C__reconstruct_cache_contents`
+        // searches the index inside the loop that inserts the entries, in
+        // HDF5 1.14.6 and 2.0.0 alike), so a parent must be one of those or
+        // an earlier entry.
+        let mut cached = BTreeSet::new();
+        cached.insert(0);
+        if let Some(ext) = sb.superblock_extension_address {
+            cached.insert(ext);
+        }
+        let mut seen = BTreeSet::new();
+        for _ in 0..n_entries {
+            let type_id = c.u8()?;
+            if type_id >= MDCI_NTYPES {
+                return Err(bad("type id is out of valid range"));
+            }
+            let flags = c.u8()?;
+            if c.u8()? >= MDCI_RING_NTYPES {
+                return Err(bad("ring is out of valid range"));
+            }
+            if c.u8()? > MDCI_AGE_MAX {
+                return Err(bad("entry age is out of policy range"));
+            }
+            let children = c.uint(2)?;
+            // libhdf5 checks the parent flag against the child count only in
+            // debug builds (release builds refuse any entry with children);
+            // the image format's own rule is checked here.
+            if (flags & MDCI_ENTRY_IS_FD_PARENT != 0) != (children > 0) {
+                return Err(bad("flush dependency parent flag and child count disagree"));
+            }
+            c.uint(2)?; // dirty dependency children: reset for a read-only open
+            let parents = c.uint(2)?;
+            if (flags & MDCI_ENTRY_IS_FD_CHILD != 0) != (parents > 0) {
+                return Err(bad("flush dependency child flag and parent count disagree"));
+            }
+            c.uint(4)?; // LRU rank
+            let address = c
+                .addr(offset_size)?
+                .filter(|&a| a < eoa)
+                .ok_or(bad("invalid entry address range"))?;
+            let size = c.uint(length_size)?;
+            if size == 0 {
+                return Err(bad("invalid entry size"));
+            }
+            for _ in 0..parents {
+                let parent = c
+                    .addr(offset_size)?
+                    .ok_or(bad("invalid flush dependency parent offset"))?;
+                if !seen.contains(&parent) && !cached.contains(&parent) {
+                    return Err(bad("fd parent not in cache"));
+                }
+            }
+            let len = usize::try_from(size).map_err(|_| bad(RAN_OFF))?;
+            let image_offset = c.pos;
+            c.take(len)?;
+            if address.checked_add(size).is_none_or(|end| end > eoa) {
+                return Err(bad("entry extends past the end of file"));
+            }
+            if !seen.insert(address) {
+                return Err(bad("duplicate addresses in cache"));
+            }
+            entries.push(ImageEntry {
+                address,
+                image_offset,
+                len,
+            });
+        }
+        Ok(CacheImage { location, entries })
+    }
+
+    /// Where the image block is.
+    pub fn location(&self) -> CacheImageLocation {
+        self.location
+    }
+
+    /// The number of entries in the image.
+    pub fn len(&self) -> usize {
+        self.entries.len()
+    }
+
+    /// Whether the image has no entries (a decoded image always has some).
+    pub fn is_empty(&self) -> bool {
+        self.entries.is_empty()
+    }
+
+    /// The file ranges (address, length) the image's entries replace.
+    pub fn entry_ranges(&self) -> impl Iterator<Item = (u64, usize)> + '_ {
+        self.entries.iter().map(|e| (e.address, e.len))
+    }
+
+    /// The image block in `data`, the bytes [`Self::decode`] read it from.
+    pub fn block<'a>(&self, data: &'a [u8]) -> Result<&'a [u8], FormatError> {
+        image_block(data, self.location)
+    }
+
+    /// [`Self::block`] over any [`Storage`].
+    pub fn block_in<'a, S: Storage + ?Sized>(
+        &self,
+        file: &'a S,
+    ) -> Result<Cow<'a, [u8]>, FormatError> {
+        image_block_in(file, self.location)
+    }
+
+    /// Every entry as `(file address, its bytes)`, taken from `block` (the
+    /// image block, see [`Self::block_in`]), in the order [`Self::apply`]
+    /// writes them: for a reader that cannot write the image over the
+    /// file's bytes and lays the entries over each read instead.
+    pub fn entries<'b>(&self, block: &'b [u8]) -> Result<Vec<(u64, &'b [u8])>, FormatError> {
+        let short = || FormatError::InvalidCacheImage("image applied to the wrong file");
+        self.entries
+            .iter()
+            .map(|e| {
+                let src = block
+                    .get(e.image_offset..e.image_offset + e.len)
+                    .ok_or_else(short)?;
+                Ok((e.address, src))
+            })
+            .collect()
+    }
+
+    /// Write every entry over `dst`, the file's bytes from the superblock
+    /// on (as long as the `data` the image was decoded from), taking the
+    /// entries from `block` (the image block, see [`Self::block`]). `block`
+    /// must not alias `dst`: an entry may land on the block itself.
+    pub fn apply(&self, block: &[u8], dst: &mut [u8]) -> Result<(), FormatError> {
+        let short = || FormatError::InvalidCacheImage("image applied to the wrong file");
+        for e in &self.entries {
+            let src = block
+                .get(e.image_offset..e.image_offset + e.len)
+                .ok_or_else(short)?;
+            let at = usize::try_from(e.address).map_err(|_| short())?;
+            dst.get_mut(at..at + e.len)
+                .ok_or_else(short)?
+                .copy_from_slice(src);
+        }
+        Ok(())
+    }
+}
+
+fn image_block(data: &[u8], location: CacheImageLocation) -> Result<&[u8], FormatError> {
+    let (start, len) = image_block_range(data.len() as u64, location)?;
+    let start = crate::addr::to_usize(start)?;
+    Ok(&data[start..start + len])
+}
+
+fn image_block_in<S: Storage + ?Sized>(
+    file: &S,
+    location: CacheImageLocation,
+) -> Result<Cow<'_, [u8]>, FormatError> {
+    let (start, len) = image_block_range(file.len(), location)?;
+    read_exact_at(file, start, len)
+}
+
+/// Where the image block is, checked against a file of `file_len` bytes.
+fn image_block_range(
+    file_len: u64,
+    location: CacheImageLocation,
+) -> Result<(u64, usize), FormatError> {
+    let bad = FormatError::InvalidCacheImage;
+    let start = usize::try_from(location.address).map_err(|_| bad("address out of range"))?;
+    let len = usize::try_from(location.length).map_err(|_| bad("length out of range"))?;
+    start
+        .checked_add(len)
+        .filter(|&end| end as u64 <= file_len)
+        .ok_or(bad("image block extends past the end of the file"))?;
+    Ok((start as u64, len))
+}
+
+/// What an opener must do before reading a file's metadata: check the
+/// superblock extension ([`read_superblock_extension`]; an error means
+/// libhdf5 refuses to open the file) and decode any metadata cache image
+/// ([`CacheImage::decode`]). `data` is the file from the superblock on, up
+/// to its recorded end of file.
+pub fn cache_image_state(data: &[u8], sb: &Superblock) -> Result<CacheImageState, FormatError> {
+    cache_image_state_in(data, sb)
+}
+
+/// [`cache_image_state`] over any [`Storage`].
+pub fn cache_image_state_in<S: Storage + ?Sized>(
+    file: &S,
+    sb: &Superblock,
+) -> Result<CacheImageState, FormatError> {
+    match read_superblock_extension_in(file, sb)? {
+        Some(SuperblockExtension {
+            cache_image: Some(location),
+            ..
+        }) => Ok(match CacheImage::decode_in(file, location, sb) {
+            Ok(image) => CacheImageState::Loaded(image),
+            Err(e) => CacheImageState::Unloadable(e),
+        }),
+        _ => Ok(CacheImageState::Absent),
+    }
+}
+
+/// [`cache_image_state`] for a reader that holds the file's bytes in a
+/// buffer of its own: check the superblock extension and write any cache
+/// image over `data` in place (only the image block is copied). An image
+/// libhdf5 cannot load is an error here: such a reader has no way to open
+/// the file and fail each object instead.
+pub fn apply_cache_image_in_place(data: &mut [u8], sb: &Superblock) -> Result<(), FormatError> {
+    match cache_image_state(data, sb)? {
+        CacheImageState::Absent => Ok(()),
+        CacheImageState::Unloadable(e) => Err(e),
+        CacheImageState::Loaded(image) => {
+            let block = image.block(data)?.to_vec();
+            image.apply(&block, data)
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The file's bytes with the image at `loc` applied.
+    fn apply_cache_image(
+        data: &[u8],
+        loc: CacheImageLocation,
+        sb: &Superblock,
+    ) -> Result<Vec<u8>, FormatError> {
+        let image = CacheImage::decode(data, loc, sb)?;
+        let mut out = data.to_vec();
+        image.apply(image.block(data)?, &mut out)?;
+        Ok(out)
+    }
+
+    fn sb_v2(ext: u64) -> Superblock {
+        Superblock {
+            version: 2,
+            offset_size: 8,
+            length_size: 8,
+            base_address: 0,
+            eof_address: 0,
+            root_group_address: 0,
+            group_leaf_node_k: None,
+            group_internal_node_k: None,
+            indexed_storage_internal_node_k: None,
+            free_space_address: None,
+            driver_info_address: None,
+            consistency_flags: 0,
+            superblock_extension_address: Some(ext),
+            checksum: None,
+            page_size: None,
+        }
+    }
+
+    /// A file whose superblock extension (a version 1 object header at 48)
+    /// holds the given messages, padded to `len` bytes.
+    fn file_with_ext(messages: &[(u16, &[u8])], len: usize) -> Vec<u8> {
+        let mut body = Vec::new();
+        for &(t, d) in messages {
+            let padded = d.len().div_ceil(8) * 8;
+            body.extend_from_slice(&t.to_le_bytes());
+            body.extend_from_slice(&(padded as u16).to_le_bytes());
+            body.extend_from_slice(&[0x14, 0, 0, 0]);
+            body.extend_from_slice(d);
+            body.resize(body.len() + padded - d.len(), 0);
+        }
+        let mut f = vec![0u8; 48];
+        f.push(1);
+        f.push(0);
+        f.extend_from_slice(&(messages.len() as u16).to_le_bytes());
+        f.extend_from_slice(&1u32.to_le_bytes());
+        f.extend_from_slice(&(body.len() as u32).to_le_bytes());
+        f.extend_from_slice(&[0; 4]);
+        f.extend_from_slice(&body);
+        f.resize(len, 0);
+        f
+    }
+
+    fn fsinfo_v1(page_size: u64, persist: bool, n_addrs: usize) -> Vec<u8> {
+        let mut m = vec![1, 1, u8::from(persist)];
+        m.extend_from_slice(&1u64.to_le_bytes());
+        m.extend_from_slice(&page_size.to_le_bytes());
+        m.extend_from_slice(&0u16.to_le_bytes());
+        m.extend_from_slice(&u64::MAX.to_le_bytes());
+        for _ in 0..n_addrs {
+            m.extend_from_slice(&u64::MAX.to_le_bytes());
+        }
+        m
+    }
+
+    fn mdci(address: u64, length: u64) -> Vec<u8> {
+        let mut m = vec![0];
+        m.extend_from_slice(&address.to_le_bytes());
+        m.extend_from_slice(&length.to_le_bytes());
+        m
+    }
+
+    #[test]
+    fn no_extension() {
+        assert_eq!(
+            read_superblock_extension(&[0; 64], &sb_v2(u64::MAX)).unwrap(),
+            None
+        );
+    }
+
+    #[test]
+    fn file_space_info_as_libhdf5_decodes_it() {
+        // What FileWriter::with_page_size writes.
+        let f = file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, false, 0))], 256);
+        let ext = read_superblock_extension(&f, &sb_v2(48)).unwrap().unwrap();
+        assert_eq!(ext.file_space_info.unwrap().page_size, 4096);
+        let f = file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, true, 12))], 512);
+        assert!(read_superblock_extension(&f, &sb_v2(48)).is_ok());
+
+        let refused = |m: Vec<u8>| {
+            let f = file_with_ext(&[(MSG_FSINFO, &m)], 512);
+            read_superblock_extension(&f, &sb_v2(48)).unwrap_err()
+        };
+        // Persisting, but too short for the manager addresses.
+        let mut short = fsinfo_v1(4096, true, 12);
+        short.truncate(short.len() - 8);
+        assert_eq!(refused(short), ext_err(RAN_OFF));
+        assert!(matches!(
+            refused(fsinfo_v1(256, false, 0)),
+            FormatError::InvalidSuperblockExtension(_)
+        ));
+        assert!(matches!(
+            refused(fsinfo_v1(0, false, 0)),
+            FormatError::InvalidSuperblockExtension(_)
+        ));
+        let mut v2 = fsinfo_v1(4096, false, 0);
+        v2[0] = 2;
+        assert!(matches!(
+            refused(v2),
+            FormatError::InvalidSuperblockExtension(_)
+        ));
+        // cve-2020-10810: version 0, strategy ALL_PERSIST, and a message of
+        // 32 bytes that cannot hold the six addresses that follow.
+        let mut v0 = vec![0u8, 1];
+        v0.extend_from_slice(&[0, 1, 0, 0, 0, 0, 0, 0]);
+        v0.resize(32, 0xff);
+        assert_eq!(refused(v0), ext_err(RAN_OFF));
+        // A version 0 message without persistence is fine.
+        let mut v0 = vec![0u8, 2];
+        v0.extend_from_slice(&[0; 8]);
+        let f = file_with_ext(&[(MSG_FSINFO, &v0)], 256);
+        assert!(read_superblock_extension(&f, &sb_v2(48)).is_ok());
+    }
+
+    #[test]
+    fn cache_image_location_must_be_inside_the_file() {
+        let f = file_with_ext(&[(MSG_MDCI, &mdci(128, 64))], 192);
+        let ext = read_superblock_extension(&f, &sb_v2(48)).unwrap().unwrap();
+        assert_eq!(
+            ext.cache_image,
+            Some(CacheImageLocation {
+                address: 128,
+                length: 64
+            })
+        );
+        // cve-2020-10812: 256 MiB at 0x10100 in a 2565-byte file.
+        let f = file_with_ext(&[(MSG_MDCI, &mdci(0x10100, 0x1000_0000))], 2565);
+        assert!(matches!(
+            read_superblock_extension(&f, &sb_v2(48)),
+            Err(FormatError::InvalidSuperblockExtension(_))
+        ));
+        let f = file_with_ext(&[(MSG_MDCI, &mdci(u64::MAX, 8))], 256);
+        assert!(read_superblock_extension(&f, &sb_v2(48)).is_err());
+    }
+
+    /// A cache image block with `entries` of (address, bytes).
+    fn image(entries: &[(u64, &[u8])]) -> Vec<u8> {
+        let with_deps: Vec<_> = entries.iter().map(|&(a, b)| (a, b, 0, None)).collect();
+        image_with_deps(&with_deps)
+    }
+
+    /// A cache image block with `entries` of (address, bytes, flush
+    /// dependency children, flush dependency parent).
+    fn image_with_deps(entries: &[(u64, &[u8], u16, Option<u64>)]) -> Vec<u8> {
+        let mut b = Vec::new();
+        b.extend_from_slice(MDCI_SIGNATURE);
+        b.push(0);
+        b.push(0);
+        b.extend_from_slice(&0u64.to_le_bytes()); // length, patched below
+        b.extend_from_slice(&(entries.len() as u32).to_le_bytes());
+        for &(addr, bytes, children, parent) in entries {
+            let mut flags = 0x02; // in LRU
+            if children > 0 {
+                flags |= MDCI_ENTRY_IS_FD_PARENT;
+            }
+            if parent.is_some() {
+                flags |= MDCI_ENTRY_IS_FD_CHILD;
+            }
+            b.extend_from_slice(&[5, flags, 1, 0]); // type, flags, ring, age
+            b.extend_from_slice(&children.to_le_bytes());
+            b.extend_from_slice(&0u16.to_le_bytes()); // dirty children
+            b.extend_from_slice(&u16::from(parent.is_some()).to_le_bytes());
+            b.extend_from_slice(&0i32.to_le_bytes());
+            b.extend_from_slice(&addr.to_le_bytes());
+            b.extend_from_slice(&(bytes.len() as u64).to_le_bytes());
+            if let Some(p) = parent {
+                b.extend_from_slice(&p.to_le_bytes());
+            }
+            b.extend_from_slice(bytes);
+        }
+        b.extend_from_slice(&[0; 4]); // checksum (not verified, as in libhdf5)
+        let n = b.len() as u64;
+        b[6..14].copy_from_slice(&n.to_le_bytes());
+        b
+    }
+
+    #[test]
+    fn cache_image_entries_replace_the_file_bytes() {
+        let img = image(&[(16, b"HEADER"), (40, b"NODE")]);
+        let mut f = vec![0u8; 64];
+        let at = f.len() as u64;
+        f.extend_from_slice(&img);
+        let loc = CacheImageLocation {
+            address: at,
+            length: img.len() as u64,
+        };
+        let out = apply_cache_image(&f, loc, &sb_v2(u64::MAX)).unwrap();
+        assert_eq!(out.len(), f.len());
+        assert_eq!(&out[16..22], b"HEADER");
+        assert_eq!(&out[40..44], b"NODE");
+        assert_eq!(&out[..16], &f[..16]);
+
+        let bad = |img: Vec<u8>| {
+            let mut f = vec![0u8; 64];
+            f.extend_from_slice(&img);
+            let loc = CacheImageLocation {
+                address: 64,
+                length: img.len() as u64,
+            };
+            apply_cache_image(&f, loc, &sb_v2(u64::MAX)).unwrap_err()
+        };
+        let mut sig = image(&[(16, b"x")]);
+        sig[0] = b'X';
+        assert!(matches!(bad(sig), FormatError::InvalidCacheImage(_)));
+        assert!(matches!(
+            bad(image(&[(16, b"a"), (16, b"b")])),
+            FormatError::InvalidCacheImage("duplicate addresses in cache")
+        ));
+        assert!(matches!(
+            bad(image(&[(1 << 20, b"far")])),
+            FormatError::InvalidCacheImage("invalid entry address range")
+        ));
+        let mut len = image(&[(16, b"x")]);
+        len[6] ^= 1;
+        assert!(matches!(bad(len), FormatError::InvalidCacheImage(_)));
+        // An entry that starts inside the file (64 bytes, then a 60-byte
+        // image) but runs past its end.
+        assert!(matches!(
+            bad(image(&[(123, b"8 bytes!")])),
+            FormatError::InvalidCacheImage("entry extends past the end of file")
+        ));
+        let mut cut = image(&[(16, b"abcdef")]);
+        let n = cut.len() as u64 - 8;
+        cut.truncate(cut.len() - 8);
+        cut[6..14].copy_from_slice(&n.to_le_bytes());
+        assert!(matches!(bad(cut), FormatError::InvalidCacheImage(_)));
+    }
+
+    /// libhdf5 resolves an entry's flush-dependency parents as it inserts
+    /// the entry (`H5C__reconstruct_cache_contents`): a parent must be an
+    /// earlier entry, or the superblock or its extension's object header,
+    /// which are cached before the image loads. A parent listed after its
+    /// child fails ("fd parent not in cache?!?").
+    #[test]
+    fn flush_dependency_parents_must_already_be_cached() {
+        let load = |img: Vec<u8>| {
+            let mut f = vec![0u8; 64];
+            f.extend_from_slice(&img);
+            let loc = CacheImageLocation {
+                address: 64,
+                length: img.len() as u64,
+            };
+            apply_cache_image(&f, loc, &sb_v2(48))
+        };
+        // Parent first, as libhdf5 writes images.
+        assert!(
+            load(image_with_deps(&[
+                (16, b"P", 1, None),
+                (40, b"C", 0, Some(16))
+            ]))
+            .is_ok()
+        );
+        // Child first: libhdf5 does not find the parent.
+        assert_eq!(
+            load(image_with_deps(&[
+                (40, b"C", 0, Some(16)),
+                (16, b"P", 1, None)
+            ]))
+            .unwrap_err(),
+            FormatError::InvalidCacheImage("fd parent not in cache")
+        );
+        // The superblock extension's header (at 48 here) is in the cache.
+        assert!(load(image_with_deps(&[(40, b"C", 0, Some(48))])).is_ok());
+        // An entry cannot be its own parent.
+        assert!(load(image_with_deps(&[(40, b"C", 1, Some(40))])).is_err());
+    }
+
+    /// The extension and cache image decode identically through a
+    /// `read_at`-only storage, errors included.
+    #[test]
+    fn storage_parse_matches_slice_parse() {
+        use crate::storage::CountingStorage;
+        let img = image(&[(16, b"HEADER"), (40, b"NODE")]);
+        let mut with_image = file_with_ext(&[(MSG_MDCI, &mdci(256, img.len() as u64))], 256);
+        with_image.extend_from_slice(&img);
+        let mut bad_image = with_image.clone();
+        bad_image[256] = b'X';
+        let files = [
+            file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, false, 0))], 256),
+            file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(256, false, 0))], 256),
+            file_with_ext(&[(MSG_MDCI, &mdci(128, 64))], 192),
+            file_with_ext(&[(MSG_MDCI, &mdci(0x10100, 0x1000_0000))], 2565),
+            file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, true, 12))], 60),
+            with_image,
+            bad_image,
+        ];
+        for f in files {
+            let storage = CountingStorage::new(f.clone());
+            let sb = sb_v2(48);
+            assert_eq!(
+                read_superblock_extension_in(&storage, &sb),
+                read_superblock_extension(&f, &sb)
+            );
+            assert_eq!(
+                cache_image_state_in(&storage, &sb),
+                cache_image_state(&f, &sb)
+            );
+            if let Ok(CacheImageState::Loaded(image)) = cache_image_state(&f, &sb) {
+                assert_eq!(
+                    &*image.block_in(&storage).unwrap(),
+                    image.block(&f).unwrap()
+                );
+            }
+        }
+    }
+}

crates/clawhdf5-format/src/symbol_table.rs

@@ -4,6 +4,7 @@
 use alloc::vec::Vec;
 
 use crate::error::FormatError;
+use crate::storage::{Storage, read_exact_at};
 
 /// Symbol Table message (type 0x0011) found in v1 group object headers.
 #[derive(Debug, Clone, PartialEq)]
@@ -79,65 +80,49 @@ impl SymbolTableNode {
         offset: usize,
         offset_size: u8,
     ) -> Result<SymbolTableNode, FormatError> {
-        // signature(4) + version(1) + reserved(1) + number_of_symbols(2) = 8
+        Self::parse_in(file_data, offset as u64, offset_size)
-        if offset
-            .checked_add(8)
-            .is_none_or(|end| end > file_data.len())
-        {
-            return Err(FormatError::UnexpectedEof {
-                expected: offset.saturating_add(8),
-                available: file_data.len(),
-            });
     }
 
-        if &file_data[offset..offset + 4] != b"SNOD" {
+    /// [`Self::parse`] over any [`Storage`]: one read of the node's header,
+    /// one of its entries.
+    pub fn parse_in<S: Storage + ?Sized>(
+        file: &S,
+        offset: u64,
+        offset_size: u8,
+    ) -> Result<SymbolTableNode, FormatError> {
+        // signature(4) + version(1) + reserved(1) + number_of_symbols(2) = 8
+        let header = read_exact_at(file, offset, 8)?;
+
+        if &header[..4] != b"SNOD" {
             return Err(FormatError::InvalidSymbolTableNodeSignature);
         }
 
-        let version = file_data[offset + 4];
+        let version = header[4];
         if version != 1 {
             return Err(FormatError::InvalidSymbolTableNodeVersion(version));
         }
 
-        let num_symbols =
+        let num_symbols = u16::from_le_bytes([header[6], header[7]]) as usize;
-            u16::from_le_bytes([file_data[offset + 6], file_data[offset + 7]]) as usize;
 
         let os = offset_size as usize;
         // Each entry: link_name_offset(os) + obj_hdr_addr(os) + cache_type(4) + reserved(4) + scratch(16)
         let entry_size = os + os + 4 + 4 + 16;
-        let entries_start = offset + 8;
+        // `offset + 8` fits: the header's read checked it. The entries'
-        let needed = entries_start.checked_add(num_symbols * entry_size).ok_or(
+        // read is the bounds check (`offset + 8 + entries > file length`,
-            FormatError::UnexpectedEof {
+        // which cannot overflow: at most 65535 entries of 40 bytes).
-                expected: usize::MAX,
+        let body = read_exact_at(file, offset + 8, num_symbols * entry_size)?;
-                available: file_data.len(),
+        let file_data: &[u8] = &body;
-            },
-        )?;
-        if needed > file_data.len() {
-            return Err(FormatError::UnexpectedEof {
-                expected: needed,
-                available: file_data.len(),
-            });
-        }
 
         let mut entries = Vec::with_capacity(num_symbols);
-        let mut pos = entries_start;
+        for entry in file_data.chunks_exact(entry_size) {
-        for _ in 0..num_symbols {
+            let link_name_offset = read_offset(entry, 0, offset_size)?;
-            let link_name_offset = read_offset(file_data, pos, offset_size)?;
+            let object_header_address = read_offset(entry, os, offset_size)?;
-            pos += os;
+            let pos = 2 * os;
-            let object_header_address = read_offset(file_data, pos, offset_size)?;
+            let cache_type =
-            pos += os;
+                u32::from_le_bytes([entry[pos], entry[pos + 1], entry[pos + 2], entry[pos + 3]]);
-            let cache_type = u32::from_le_bytes([
-                file_data[pos],
-                file_data[pos + 1],
-                file_data[pos + 2],
-                file_data[pos + 3],
-            ]);
-            pos += 4;
             // reserved 4 bytes
-            pos += 4;
             let mut scratch_pad = [0u8; 16];
-            scratch_pad.copy_from_slice(&file_data[pos..pos + 16]);
+            scratch_pad.copy_from_slice(&entry[pos + 8..pos + 24]);
-            pos += 16;
 
             entries.push(SymbolTableEntry {
                 link_name_offset,
@@ -256,4 +241,28 @@ mod tests {
         let result = SymbolTableNode::parse(&data, usize::MAX / 2, 8);
         assert!(result.is_err());
     }
+
+    /// Nodes, cut at every length and at an offset, parse identically
+    /// through a `read_at`-only storage.
+    #[test]
+    fn storage_parse_matches_slice_parse() {
+        use crate::storage::CountingStorage;
+        for os in [4u8, 8] {
+            let node = build_snod(&[(0, 0x100, 0), (8, 0x200, 1), (16, 0x300, 2)], os);
+            let mut bad = node.clone();
+            bad[4] = 2;
+            for full in [node, bad] {
+                for at in [0usize, 7] {
+                    for cut in 0..=full.len() {
+                        let mut f = vec![0u8; at];
+                        f.extend_from_slice(&full[..cut]);
+                        let storage = CountingStorage::new(f.clone());
+                        let want = SymbolTableNode::parse(&f, at, os);
+                        let got = SymbolTableNode::parse_in(&storage, at as u64, os);
+                        assert_eq!(format!("{got:?}"), format!("{want:?}"));
+                    }
+                }
+            }
+        }
+    }
 }

crates/clawhdf5-format/src/test_fuzz.rs

@@ -0,0 +1,149 @@
+//! Mutation fuzzing for the filter decoders (tests only).
+//!
+//! A decoder fed a random or mutated frame may fail, but must not panic —
+//! tests build with overflow checks and debug assertions, so an unchecked
+//! subtraction, multiplication or shift on a header field, or an
+//! out-of-range slice, fails the test — and must not return more than its
+//! output limit.
+
+#[cfg(not(feature = "std"))]
+extern crate alloc;
+#[cfg(not(feature = "std"))]
+use alloc::vec::Vec;
+
+use crate::error::FormatError;
+
+/// xorshift64*: deterministic, so a failure reproduces.
+pub(crate) struct Rng(u64);
+
+impl Rng {
+    pub(crate) fn new(seed: u64) -> Rng {
+        Rng(seed.max(1))
+    }
+
+    pub(crate) fn next_u64(&mut self) -> u64 {
+        let mut x = self.0;
+        x ^= x >> 12;
+        x ^= x << 25;
+        x ^= x >> 27;
+        self.0 = x;
+        x.wrapping_mul(0x2545_F491_4F6C_DD1D)
+    }
+
+    /// Uniform in `0..n` (`n` > 0).
+    pub(crate) fn below(&mut self, n: usize) -> usize {
+        (self.next_u64() % n as u64) as usize
+    }
+
+    pub(crate) fn bytes(&mut self, n: usize) -> Vec<u8> {
+        (0..n).map(|_| self.next_u64() as u8).collect()
+    }
+
+    /// A u32 that tends to hit edge cases in size and offset fields.
+    fn interesting_u32(&mut self, len: usize) -> u32 {
+        match self.below(10) {
+            0 => 0,
+            1 => 1,
+            2 => self.below(20) as u32,
+            3 => 15 + self.below(3) as u32,
+            4 => u32::MAX - self.below(16) as u32,
+            5 => 1 << self.below(32),
+            6 => (len as u32)
+                .wrapping_add(self.below(9) as u32)
+                .wrapping_sub(4),
+            7 => i32::MAX as u32,
+            _ => self.next_u64() as u32,
+        }
+    }
+}
+
+/// One to four random edits of `seed`.
+pub(crate) fn mutate(rng: &mut Rng, seed: &[u8]) -> Vec<u8> {
+    let mut v = seed.to_vec();
+    for _ in 0..1 + rng.below(4) {
+        let len = v.len();
+        match rng.below(9) {
+            0 if len > 0 => {
+                let i = rng.below(len);
+                v[i] ^= 1 << rng.below(8);
+            }
+            1 if len > 0 => {
+                let i = rng.below(len);
+                v[i] = rng.next_u64() as u8;
+            }
+            2 if len > 0 => {
+                let i = rng.below(len);
+                v[i] = [0, 0xff, 0x7f, 0x80, 0x20, 0x1f][rng.below(6)];
+            }
+            // A size or offset field: little- or big-endian, anywhere, but
+            // most often in the first 32 bytes where headers live.
+            3 | 4 if len >= 4 => {
+                let span = if rng.below(2) == 0 { len.min(32) } else { len };
+                let i = rng.below(span - 3);
+                let x = rng.interesting_u32(len);
+                let b = if rng.below(2) == 0 {
+                    x.to_le_bytes()
+                } else {
+                    x.to_be_bytes()
+                };
+                v[i..i + 4].copy_from_slice(&b);
+            }
+            5 if len > 0 => v.truncate(rng.below(len)),
+            6 => {
+                let n = 1 + rng.below(64);
+                let extra = rng.bytes(n);
+                v.extend_from_slice(&extra);
+            }
+            7 if len > 1 => {
+                let a = rng.below(len);
+                let b = a + rng.below(len - a);
+                let copy = v[a..b].to_vec();
+                let at = rng.below(len);
+                v.splice(at..at, copy);
+            }
+            _ if len > 0 => {
+                let i = rng.below(len);
+                v[i] = v[i].wrapping_add(1 + rng.below(3) as u8);
+            }
+            _ => v.push(rng.next_u64() as u8),
+        }
+    }
+    v
+}
+
+/// Feed `iters` inputs to `decode`: mostly mutations of `seeds`, some pure
+/// noise and some truncated seeds. Asserts only "no panic, output within
+/// `limit`".
+pub(crate) fn fuzz_decoder(
+    seed: u64,
+    seeds: &[Vec<u8>],
+    iters: usize,
+    limit: usize,
+    mut decode: impl FnMut(&[u8]) -> Result<Vec<u8>, FormatError>,
+) {
+    assert!(!seeds.is_empty());
+    let mut rng = Rng::new(seed);
+    for s in seeds {
+        // The seeds themselves must be valid, or the fuzz explores nothing.
+        decode(s).expect("seed frame must decode");
+    }
+    for _ in 0..iters {
+        let input = match rng.below(16) {
+            0 => {
+                let n = rng.below(96);
+                rng.bytes(n)
+            }
+            1 => {
+                let s = &seeds[rng.below(seeds.len())];
+                s[..rng.below(s.len() + 1)].to_vec()
+            }
+            _ => {
+                let s = &seeds[rng.below(seeds.len())];
+                mutate(&mut rng, s)
+            }
+        };
+        if let Ok(out) = decode(&input) {
+            assert!(out.len() <= limit, "decoded {} > limit {limit}", out.len());
+        }
+    }
+}

crates/clawhdf5-format/src/type_builders.rs

@@ -296,7 +296,8 @@ impl EnumTypeBuilder {
 
 // ---- Attribute helper ----
 
-pub(crate) fn build_attr_message(name: &str, value: &AttrValue) -> AttributeMessage {
+/// The attribute message the writers store for `value` under `name`.
+pub fn build_attr_message(name: &str, value: &AttrValue) -> AttributeMessage {
     match value {
         AttrValue::F64(v) => AttributeMessage {
             name: name.to_string(),
@@ -503,6 +504,9 @@ pub struct DatasetBuilder {
     /// `data` field is ignored; instead the global heap blob is built from
     /// these mappings and a VDS layout message is emitted.
     pub(crate) virtual_sources: Option<Vec<VdsMapping>>,
+    /// Track (and index) attribute creation order; `None` follows the
+    /// file's default (`FileWriter::track_order`).
+    pub(crate) track_order: Option<bool>,
     #[cfg(feature = "provenance")]
     pub(crate) provenance: Option<ProvenanceConfig>,
 }
@@ -522,11 +526,22 @@ impl DatasetBuilder {
             compact: false,
             alignment: 0,
             virtual_sources: None,
+            track_order: None,
             #[cfg(feature = "provenance")]
             provenance: None,
         }
     }
 
+    /// Track the creation order of this dataset's attributes, and index it,
+    /// as h5py's `create_dataset(..., track_order=True)` does: libhdf5 (and
+    /// h5py) then list the attributes in the order they were set rather
+    /// than by name. libhdf5 numbers at most 65 535 attributes on an object
+    /// that tracks their order; more is an error when the file is written.
+    pub fn track_order(&mut self, track: bool) -> &mut Self {
+        self.track_order = Some(track);
+        self
+    }
+
     pub fn with_f64_data(&mut self, data: &[f64]) -> &mut Self {
         self.datatype = Some(make_f64_type());
         let mut b = Vec::with_capacity(data.len() * 8);
@@ -695,8 +710,13 @@ impl DatasetBuilder {
         self
     }
 
+    /// Set attribute `name`. Setting it again replaces the earlier value,
+    /// as `attrs[name] = v` does in h5py.
     pub fn set_attr(&mut self, name: &str, value: AttrValue) -> &mut Self {
-        self.attrs.push((name.to_string(), value));
+        match self.attrs.iter_mut().find(|(n, _)| n == name) {
+            Some(slot) => slot.1 = value,
+            None => self.attrs.push((name.to_string(), value)),
+        }
         self
     }
 
@@ -731,6 +751,61 @@ impl DatasetBuilder {
         self
     }
 
+    /// Compress with a plugin filter ([`PluginFilter`]), in the format the
+    /// libhdf5 plugin reads (h5py, hdf5plugin). Implies chunked storage.
+    /// Each filter needs its cargo feature (`lzf`, ...); writing fails with
+    /// `UnsupportedFilter` without it.
+    ///
+    /// [`PluginFilter`]: crate::chunked_write::PluginFilter
+    pub fn with_plugin_filter(&mut self, filter: crate::chunked_write::PluginFilter) -> &mut Self {
+        self.chunk_options.plugin = Some(filter);
+        self
+    }
+
+    /// Enable LZF compression (filter 32000) — h5py's built-in
+    /// `compression="lzf"`. Implies chunked storage; shuffle is applied
+    /// first unless `.without_shuffle()`. Requires the `lzf` cargo feature.
+    pub fn with_lzf(&mut self) -> &mut Self {
+        self.with_plugin_filter(crate::chunked_write::PluginFilter::Lzf)
+    }
+
+    /// Enable bitshuffle (filter 32008) with `compression` after the bit
+    /// transpose, in bitshuffle's default block size. Implies chunked
+    /// storage; no byte shuffle is added. Requires the `bitshuffle` cargo
+    /// feature.
+    pub fn with_bitshuffle(
+        &mut self,
+        compression: crate::chunked_write::BitshuffleCompression,
+    ) -> &mut Self {
+        self.with_plugin_filter(crate::chunked_write::PluginFilter::Bitshuffle {
+            block_size: 0,
+            compression,
+        })
+    }
+
+    /// Enable bzip2 (filter 307) at block size `level` (1-9). Implies
+    /// chunked storage; shuffle is applied first unless
+    /// `.without_shuffle()`. Requires the `bzip2` cargo feature.
+    pub fn with_bzip2(&mut self, level: u32) -> &mut Self {
+        self.with_plugin_filter(crate::chunked_write::PluginFilter::Bzip2 { level })
+    }
+
+    /// Enable Blosc (filter 32001) with `codec` at `level` (0-9) after
+    /// `shuffle`. Implies chunked storage; no extra HDF5 shuffle is added.
+    /// Requires the `blosc` cargo feature.
+    pub fn with_blosc(
+        &mut self,
+        codec: crate::chunked_write::BloscCodec,
+        level: u32,
+        shuffle: crate::chunked_write::BloscShuffle,
+    ) -> &mut Self {
+        self.with_plugin_filter(crate::chunked_write::PluginFilter::Blosc {
+            codec,
+            level,
+            shuffle,
+        })
+    }
+
     /// Enable Pcodec lossless numerical compression (private clawhdf5 filter
     /// ID 480).
     ///
@@ -848,34 +923,118 @@ impl DatasetBuilder {
 
 // ---- Group builder ----
 
-/// Builder for groups.
+/// One entry of a [`GroupBuilder`], kept in the order it was added (the
+/// order a group that tracks creation order lists its links in).
+pub(crate) enum GroupItem {
+    Dataset(Box<DatasetBuilder>),
+    Group(GroupBuilder),
+    /// A soft link: `name` resolves to whatever `target` names when read.
+    Soft {
+        name: String,
+        target: String,
+    },
+    /// An extra hard link to the object at `target` (a path in this file).
+    Hard {
+        name: String,
+        target: String,
+    },
+    /// An external link to `path` in the file `file`.
+    External {
+        name: String,
+        file: String,
+        path: String,
+    },
+}
+
+/// Builder for a group: its datasets, subgroups, links and attributes.
+///
+/// Names are paths relative to the group: `create_dataset("a/b/x")` creates
+/// the groups `a` and `a/b` as needed, as h5py does. A group added where a
+/// group of the same path already exists (added by another builder, or
+/// created as an intermediate group) is merged into it, like h5py's
+/// `require_group`; any other name used twice in a group is an error when the
+/// file is written. A path component must not be empty or `"."`.
 pub struct GroupBuilder {
     pub(crate) name: String,
-    pub(crate) datasets: Vec<DatasetBuilder>,
+    pub(crate) items: Vec<GroupItem>,
     pub(crate) attrs: Vec<(String, AttrValue)>,
-    /// (link_name, target_file, target_path)
+    /// Track (and index) link creation order; `None` follows the file's
-    pub(crate) external_links: Vec<(String, String, String)>,
+    /// default (`FileWriter::track_order`).
+    pub(crate) track_order: Option<bool>,
 }
 
 impl GroupBuilder {
     pub(crate) fn new(name: &str) -> Self {
         Self {
             name: name.to_string(),
-            datasets: Vec::new(),
+            items: Vec::new(),
             attrs: Vec::new(),
-            external_links: Vec::new(),
+            track_order: None,
         }
     }
 
+    /// Create a dataset in this group. `name` may be a relative path
+    /// (`"a/b/x"`); missing intermediate groups are created.
     pub fn create_dataset(&mut self, name: &str) -> &mut DatasetBuilder {
-        self.datasets.push(DatasetBuilder::new(name));
+        self.items
-        self.datasets.last_mut().unwrap()
+            .push(GroupItem::Dataset(Box::new(DatasetBuilder::new(name))));
+        match self.items.last_mut() {
+            Some(GroupItem::Dataset(d)) => d,
+            _ => unreachable!("just pushed a dataset"),
+        }
+    }
+
+    /// Start a subgroup of this group. Like `FileWriter::create_group`, the
+    /// builder is detached: fill it, then pass `finish()`'s result to
+    /// [`Self::add_group`]. `name` may be a relative path.
+    pub fn create_group(&self, name: &str) -> GroupBuilder {
+        GroupBuilder::new(name)
+    }
+
+    /// Add a finished subgroup to this group.
+    pub fn add_group(&mut self, group: FinishedGroup) -> &mut Self {
+        self.items.push(GroupItem::Group(group.group));
+        self
     }
 
     pub fn set_attr(&mut self, name: &str, value: AttrValue) {
         self.attrs.push((name.to_string(), value));
     }
 
+    /// Track the creation order of this group's links and attributes, and
+    /// index it, as h5py's `track_order=True` does: libhdf5 (and h5py) then
+    /// list the group's members, and its attributes, in the order they were
+    /// added rather than by name. libhdf5 numbers at most 65 535 attributes
+    /// on an object that tracks their order.
+    pub fn track_order(&mut self, track: bool) -> &mut Self {
+        self.track_order = Some(track);
+        self
+    }
+
+    /// Add a soft link `name` to the path `target` (absolute, or relative to
+    /// this group), like h5py's `grp[name] = h5py.SoftLink(target)`. The
+    /// target need not exist.
+    pub fn add_soft_link(&mut self, name: &str, target: &str) -> &mut Self {
+        self.items.push(GroupItem::Soft {
+            name: name.to_string(),
+            target: target.to_string(),
+        });
+        self
+    }
+
+    /// Add another hard link `name` to the group or dataset at `target`
+    /// (absolute, or relative to this group), like h5py's
+    /// `grp[name] = f[target]`. The target must be written in the same file;
+    /// its path may go through other hard links, but not through soft or
+    /// external links.
+    pub fn add_hard_link(&mut self, name: &str, target: &str) -> &mut Self {
+        self.items.push(GroupItem::Hard {
+            name: name.to_string(),
+            target: target.to_string(),
+        });
+        self
+    }
+
     /// Add an external link: a named pointer to an object in another HDF5 file.
     pub fn add_external_link(
         &mut self,
@@ -883,30 +1042,21 @@ impl GroupBuilder {
         target_file: &str,
         target_path: &str,
     ) -> &mut Self {
-        self.external_links.push((
+        self.items.push(GroupItem::External {
-            name.to_string(),
+            name: name.to_string(),
-            target_file.to_string(),
+            file: target_file.to_string(),
-            target_path.to_string(),
+            path: target_path.to_string(),
-        ));
+        });
         self
     }
 
     /// Consume the builder, returning a FinishedGroup to add to FileWriter.
     pub fn finish(self) -> FinishedGroup {
-        FinishedGroup {
+        FinishedGroup { group: self }
-            name: self.name,
-            datasets: self.datasets,
-            attrs: self.attrs,
-            external_links: self.external_links,
-        }
     }
 }
 
 /// A finished group ready for the file writer.
 pub struct FinishedGroup {
-    pub(crate) name: String,
+    pub(crate) group: GroupBuilder,
-    pub(crate) datasets: Vec<DatasetBuilder>,
-    pub(crate) attrs: Vec<(String, AttrValue)>,
-    /// (link_name, target_file, target_path)
-    pub(crate) external_links: Vec<(String, String, String)>,
 }

crates/clawhdf5-format/src/vl_data.rs

@@ -5,10 +5,13 @@
 //! `sequence_length(4 LE) + collection_address(offset_size LE) + object_index(4 LE)`.
 
 #[cfg(not(feature = "std"))]
-use alloc::{string::String, vec::Vec};
+use alloc::{borrow::Cow, collections::BTreeMap, format, string::String, vec, vec::Vec};
+#[cfg(feature = "std")]
+use std::{borrow::Cow, collections::BTreeMap};
 
+use crate::addr::to_usize;
 use crate::error::FormatError;
-use crate::global_heap::GlobalHeapCollection;
+use crate::global_heap::{GlobalHeapCollection, GlobalHeapIndex};
 
 /// A parsed variable-length element reference (global heap ID).
 #[derive(Debug, Clone)]
@@ -53,7 +56,7 @@ pub fn parse_vl_references(
 ) -> Result<Vec<VlElement>, FormatError> {
     let elem_size = 4 + offset_size as usize + 4; // length + address + index
     let total =
-        (num_elements as usize)
+        to_usize(num_elements)?
             .checked_mul(elem_size)
             .ok_or(FormatError::UnexpectedEof {
                 expected: usize::MAX,
@@ -66,7 +69,7 @@ pub fn parse_vl_references(
         });
     }
 
-    let mut elements = Vec::with_capacity(num_elements as usize);
+    let mut elements = Vec::with_capacity(to_usize(num_elements)?);
     let mut pos = 0;
 
     for _ in 0..num_elements {
@@ -109,7 +112,265 @@ fn is_undefined_address(addr: u64, offset_size: u8) -> bool {
     }
 }
 
+/// The size of one variable-length element in a file with `offset_size`-byte
+/// addresses: a sequence length (4), a global heap collection address and an
+/// object index (4). libhdf5 computes it this way rather than trusting the
+/// datatype message (`H5T_set_loc`).
+pub fn element_size(offset_size: u8) -> usize {
+    4 + offset_size as usize + 4
+}
+
+/// Refuse a variable-length datatype whose stored element size is not the
+/// one this file's offset size implies. Its elements would be laid out with
+/// a stride libhdf5 does not use, so every value after the first would be
+/// read from the wrong place.
+pub fn check_element_size(stored_size: u32, offset_size: u8) -> Result<(), FormatError> {
+    let expected = element_size(offset_size);
+    if stored_size as usize != expected {
+        return Err(FormatError::VlDataError(format!(
+            "variable-length datatype stores {stored_size}-byte elements; a file with \
+             {offset_size}-byte offsets uses {expected}"
+        )));
+    }
+    Ok(())
+}
+
+/// A collection's objects, located in the file data but not copied:
+/// `(index, offset, size)` of the first object with each index, sorted by
+/// index. Over a storage without the whole file in memory, also the
+/// collection's bytes (`(offset, bytes)`), read once when it is indexed.
+struct CachedCollection<'a> {
+    objects: Vec<(u16, usize, usize)>,
+    bytes: Option<(usize, Cow<'a, [u8]>)>,
+}
+
+impl<'a> CachedCollection<'a> {
+    fn new(index: GlobalHeapIndex, bytes: Option<(usize, Cow<'a, [u8]>)>) -> Self {
+        let mut objects: Vec<(u16, usize, usize)> = index
+            .objects
+            .iter()
+            .map(|o| (o.index, o.offset, o.size))
+            .collect();
+        // Stable, so the first object with a repeated index is kept.
+        objects.sort_by_key(|o| o.0);
+        objects.dedup_by_key(|o| o.0);
+        Self { objects, bytes }
+    }
+
+    /// What this entry costs to keep, in bytes (roughly).
+    fn cost(&self) -> usize {
+        let held = match &self.bytes {
+            Some((_, Cow::Owned(b))) => b.len(),
+            _ => 0,
+        };
+        64 + self.objects.len() * core::mem::size_of::<(u16, usize, usize)>() + held
+    }
+
+    fn get(&self, index: u32) -> Option<(usize, usize)> {
+        let index = u16::try_from(index).ok()?;
+        let i = self.objects.binary_search_by_key(&index, |o| o.0).ok()?;
+        Some((self.objects[i].1, self.objects[i].2))
+    }
+}
+
+/// How many bytes of collection indexes a [`VlResolver`] keeps before it
+/// drops them and starts again. Values are never copied into the cache, so
+/// this bounds what a read retains however many collections it visits.
+/// (Over a storage without the whole file in memory the collections' bytes
+/// are kept too, and count against this.)
+const CACHE_BUDGET: usize = 32 << 20;
+
+/// Resolves variable-length elements against a file's global heap, parsing
+/// each heap collection once however many elements point into it.
+///
+/// Values follow libhdf5: an element whose heap address is 0 is null (an
+/// empty string or sequence), and an element whose heap object is not
+/// exactly `length × base size` bytes is an error ("Expected global heap
+/// object size does not match"), not a truncated or padded value.
+///
+/// Memory stays bounded on hostile files: the cache holds where each
+/// object lies, not a copy of it, up to a fixed budget; and collections
+/// that overlap one another are refused (libhdf5 never writes them), so a
+/// file cannot make the resolver parse the same bytes as the objects of
+/// many collections.
+///
+/// The file is any [`Storage`](crate::storage::Storage) (`S`, a slice by default). Over one without
+/// the whole file in memory each collection is read once, when first used,
+/// and kept (within the budget above); [`Self::strings`],
+/// [`Self::string_bytes`] and [`Self::sequences`] work over any storage,
+/// [`Self::element`] and [`Self::string_element`], which borrow from the
+/// file, over a slice.
+pub struct VlResolver<'a, S: crate::storage::Storage + ?Sized = [u8]> {
+    file_data: &'a S,
+    offset_size: u8,
+    length_size: u8,
+    cache: BTreeMap<u64, CachedCollection<'a>>,
+    cached_bytes: usize,
+    budget: usize,
+    /// Start → end of every collection parsed so far (kept when the cache
+    /// is dropped, to check overlaps).
+    extents: BTreeMap<usize, usize>,
+}
+
+impl<'a> VlResolver<'a> {
+    /// A resolver over `file_data` (the file from its superblock on), with
+    /// the superblock's offset and length sizes.
+    pub fn new(file_data: &'a [u8], offset_size: u8, length_size: u8) -> Self {
+        Self::new_in(file_data, offset_size, length_size)
+    }
+
+    /// One element (the first [`element_size`](Self::element_size) bytes of
+    /// `elem`) of a variable-length sequence whose base type is `base_size`
+    /// bytes: its `length × base_size` bytes, or `None` for a null element
+    /// (heap address 0).
+    pub fn element(
+        &mut self,
+        elem: &[u8],
+        base_size: usize,
+    ) -> Result<Option<&'a [u8]>, FormatError> {
+        let vl = parse_vl_references(elem, 1, self.offset_size)?;
+        let vl = &vl[0];
+        if vl.collection_address == 0 {
+            return Ok(None);
+        }
+        let (start, size) = self.locate(vl)?;
+        let data = &self.file_data[start..start + size];
+        check_object_size(vl, data.len(), base_size)?;
+        Ok(Some(data))
+    }
+
+    /// One variable-length string element: its bytes up to the first NUL,
+    /// or `None` for a null element (h5dump prints it as `NULL`, h5py
+    /// returns it as empty).
+    pub fn string_element(&mut self, elem: &[u8]) -> Result<Option<&'a [u8]>, FormatError> {
+        Ok(self.element(elem, 1)?.map(cut_at_nul))
+    }
+}
+
+/// `data_len`, the size of `vl`'s heap object, against the `length ×
+/// base_size` bytes the element says it holds.
+fn check_object_size(vl: &VlElement, data_len: usize, base_size: usize) -> Result<(), FormatError> {
+    let expected = (vl.length as usize)
+        .checked_mul(base_size)
+        .ok_or_else(|| FormatError::Overflow("variable-length element size".into()))?;
+    if data_len != expected {
+        return Err(FormatError::VlDataError(format!(
+            "global heap object {} in the collection at {} holds {data_len} bytes; the element \
+             says {} × {base_size}",
+            vl.object_index, vl.collection_address, vl.length
+        )));
+    }
+    Ok(())
+}
+
+impl<'a, S: crate::storage::Storage + ?Sized> VlResolver<'a, S> {
+    /// [`VlResolver::new`] over any [`Storage`](crate::storage::Storage).
+    pub fn new_in(file_data: &'a S, offset_size: u8, length_size: u8) -> Self {
+        Self {
+            file_data,
+            offset_size,
+            length_size,
+            cache: BTreeMap::new(),
+            cached_bytes: 0,
+            budget: CACHE_BUDGET,
+            extents: BTreeMap::new(),
+        }
+    }
+
+    /// The size of one element in this file (see [`element_size`]).
+    pub fn element_size(&self) -> usize {
+        element_size(self.offset_size)
+    }
+
+    /// Split `raw` into elements; its length must be a whole number of them.
+    fn elements(&self, raw: &[u8]) -> Result<Vec<VlElement>, FormatError> {
+        let size = self.element_size();
+        if !raw.len().is_multiple_of(size) {
+            return Err(FormatError::VlDataError(format!(
+                "{} bytes is not a whole number of {size}-byte variable-length elements",
+                raw.len()
+            )));
+        }
+        parse_vl_references(raw, (raw.len() / size) as u64, self.offset_size)
+    }
+
+    /// The bytes of one element: `length × base_size` bytes from the heap,
+    /// or `None` for a null element.
+    fn resolve(&mut self, vl: &VlElement, base_size: usize) -> Result<Option<&[u8]>, FormatError> {
+        if vl.collection_address == 0 {
+            return Ok(None);
+        }
+        let data = self.object(vl)?;
+        check_object_size(vl, data.len(), base_size)?;
+        Ok(Some(data))
+    }
+
+    /// [`VlResolver::element`] over any storage: the element's bytes
+    /// (borrowed from the resolver's cache of heap collections, so they
+    /// live until the next call), or `None` for a null element.
+    pub fn element_in(
+        &mut self,
+        elem: &[u8],
+        base_size: usize,
+    ) -> Result<Option<&[u8]>, FormatError> {
+        let vl = parse_vl_references(elem, 1, self.offset_size)?;
+        self.resolve(&vl[0], base_size)
+    }
+
+    /// [`VlResolver::string_element`] over any storage (see
+    /// [`element_in`](Self::element_in)).
+    pub fn string_element_in(&mut self, elem: &[u8]) -> Result<Option<&[u8]>, FormatError> {
+        Ok(self.element_in(elem, 1)?.map(cut_at_nul))
+    }
+
+    /// The strings of the variable-length string elements in `raw`, as
+    /// bytes. A string ends at its first NUL, as libhdf5 returns it (it
+    /// converts each to a C string); a null element is empty.
+    pub fn string_bytes(&mut self, raw: &[u8]) -> Result<Vec<Vec<u8>>, FormatError> {
+        self.elements(raw)?
+            .iter()
+            .map(|vl| Ok(self.resolve(vl, 1)?.map(cut_at_nul).unwrap_or(&[]).to_vec()))
+            .collect()
+    }
+
+    /// The strings of the variable-length string elements in `raw`, decoded
+    /// as UTF-8 with invalid sequences replaced by U+FFFD (see
+    /// [`string_bytes`](Self::string_bytes) for the exact bytes).
+    pub fn strings(&mut self, raw: &[u8]) -> Result<Vec<String>, FormatError> {
+        Ok(self
+            .string_bytes(raw)?
+            .into_iter()
+            .map(|b| match String::from_utf8(b) {
+                Ok(s) => s,
+                Err(e) => String::from_utf8_lossy(e.as_bytes()).into_owned(),
+            })
+            .collect())
+    }
+
+    /// The sequences of the variable-length sequence elements in `raw`, each
+    /// as its `length × base_size` bytes in the base type's encoding.
+    pub fn sequences(&mut self, raw: &[u8], base_size: usize) -> Result<Vec<Vec<u8>>, FormatError> {
+        if base_size == 0 {
+            return Err(FormatError::VlDataError(
+                "variable-length sequence of a zero-size base type".into(),
+            ));
+        }
+        self.elements(raw)?
+            .iter()
+            .map(|vl| Ok(self.resolve(vl, base_size)?.unwrap_or(&[]).to_vec()))
+            .collect()
+    }
+}
+
+/// A string's bytes up to its first NUL.
+fn cut_at_nul(s: &[u8]) -> &[u8] {
+    &s[..s.iter().position(|&b| b == 0).unwrap_or(s.len())]
+}
+
 /// Resolve VL strings from raw data by looking up each element in the global heap.
+///
+/// Reads the first `num_elements` elements of `raw`. Strings end at their
+/// first NUL and invalid UTF-8 is replaced, as in [`VlResolver::strings`].
 pub fn read_vl_strings(
     file_data: &[u8],
     raw_data: &[u8],
@@ -117,35 +378,34 @@ pub fn read_vl_strings(
     offset_size: u8,
     length_size: u8,
 ) -> Result<Vec<String>, FormatError> {
-    let refs = parse_vl_references(raw_data, num_elements, offset_size)?;
+    read_vl_strings_in(file_data, raw_data, num_elements, offset_size, length_size)
-    let mut result = Vec::with_capacity(refs.len());
+}
 
-    for vl in &refs {
+/// [`read_vl_strings`] over any [`Storage`](crate::storage::Storage).
-        if vl.length == 0 && is_undefined_address(vl.collection_address, offset_size) {
+pub fn read_vl_strings_in<S: crate::storage::Storage + ?Sized>(
-            result.push(String::new());
+    file_data: &S,
-            continue;
+    raw_data: &[u8],
-        }
+    num_elements: u64,
-        if vl.length == 0 && vl.collection_address == 0 {
+    offset_size: u8,
-            result.push(String::new());
+    length_size: u8,
-            continue;
+) -> Result<Vec<String>, FormatError> {
-        }
+    let raw = first_elements(raw_data, num_elements, offset_size)?;
+    VlResolver::new_in(file_data, offset_size, length_size).strings(raw)
+}
 
-        let coll =
+/// The first `num_elements` elements of `raw`, or an error if it is shorter.
-            GlobalHeapCollection::parse(file_data, vl.collection_address as usize, length_size)?;
+fn first_elements(raw: &[u8], num_elements: u64, offset_size: u8) -> Result<&[u8], FormatError> {
-        let obj = coll.get_object(vl.object_index as u16).ok_or(
+    let total = usize::try_from(num_elements)
-            FormatError::GlobalHeapObjectNotFound {
+        .ok()
-                collection_address: vl.collection_address,
+        .and_then(|n| n.checked_mul(element_size(offset_size)))
-                index: vl.object_index as u16,
+        .ok_or(FormatError::UnexpectedEof {
-            },
+            expected: usize::MAX,
-        )?;
+            available: raw.len(),
-
+        })?;
-        // The object data is the raw string bytes
+    raw.get(..total).ok_or(FormatError::UnexpectedEof {
-        let len = (vl.length as usize).min(obj.data.len());
+        expected: total,
-        let s = String::from_utf8_lossy(&obj.data[..len]).into_owned();
+        available: raw.len(),
-        result.push(s);
+    })
-    }
-
-    Ok(result)
 }
 
 /// Resolve VL sequences from raw data, returning each element's bytes.
@@ -153,44 +413,139 @@ pub fn read_vl_strings(
 /// Each element is the sequence's full encoding — element count × base type
 /// size bytes, in the base type's byte order — so a sequence of `i32` yields
 /// four bytes per value. Decode it with the base type (e.g.
-/// [`crate::data_read::read_as_i64`]).
+/// [`crate::data_read::read_as_i64`]). This does not know the base type, so
+/// it returns each heap object whole; [`VlResolver::sequences`] also checks
+/// the object's size against the element's length.
 pub fn read_vl_bytes(
     file_data: &[u8],
     raw_data: &[u8],
     num_elements: u64,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<Vec<u8>>, FormatError> {
+    read_vl_bytes_in(file_data, raw_data, num_elements, offset_size, length_size)
+}
+
+/// [`read_vl_bytes`] over any [`Storage`](crate::storage::Storage).
+pub fn read_vl_bytes_in<S: crate::storage::Storage + ?Sized>(
+    file_data: &S,
+    raw_data: &[u8],
+    num_elements: u64,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<Vec<u8>>, FormatError> {
     let refs = parse_vl_references(raw_data, num_elements, offset_size)?;
+    let mut resolver = VlResolver::new_in(file_data, offset_size, length_size);
     let mut result = Vec::with_capacity(refs.len());
 
     for vl in &refs {
-        if vl.length == 0
+        // A heap address of 0 is a null element, as in VlResolver.
-            && (is_undefined_address(vl.collection_address, offset_size)
+        if vl.collection_address == 0 {
-                || vl.collection_address == 0)
-        {
             result.push(Vec::new());
             continue;
         }
-
-        let coll =
-            GlobalHeapCollection::parse(file_data, vl.collection_address as usize, length_size)?;
-        let obj = coll.get_object(vl.object_index as u16).ok_or(
-            FormatError::GlobalHeapObjectNotFound {
-                collection_address: vl.collection_address,
-                index: vl.object_index as u16,
-            },
-        )?;
-
         // The heap object holds the whole sequence. `vl.length` counts
         // elements, not bytes, so it is only the byte length when the base
         // type is one byte wide.
-        result.push(obj.data.clone());
+        let obj = resolver.object(vl)?;
+        result.push(obj.to_vec());
     }
 
     Ok(result)
 }
 
+impl<'a, S: crate::storage::Storage + ?Sized> VlResolver<'a, S> {
+    /// Where the heap object `vl` points to lies in the file, whatever its
+    /// size (`(offset, size)`); its collection is parsed on first use.
+    fn locate(&mut self, vl: &VlElement) -> Result<(usize, usize), FormatError> {
+        let addr = vl.collection_address;
+        // libhdf5 writes a null element with address 0, never the undefined
+        // address, and fails to read one ("addr undefined") even when its
+        // length is 0; we returned an empty value.
+        if is_undefined_address(addr, self.offset_size) {
+            return Err(FormatError::VlDataError(format!(
+                "variable-length element (length {}) has the undefined global heap address",
+                vl.length
+            )));
+        }
+        if !self.cache.contains_key(&addr) {
+            let offset = usize::try_from(addr).map_err(|_| FormatError::UnexpectedEof {
+                expected: usize::MAX,
+                available: crate::storage::len_usize(self.file_data),
+            })?;
+            let (bytes, base, index) =
+                GlobalHeapCollection::read_collection(self.file_data, addr, self.length_size)?;
+            // read_collection checked that the collection lies in the file.
+            let end = offset + to_usize(index.collection_size)?;
+            self.check_overlap(offset, end)?;
+            // With the whole file in memory the objects are sliced from it;
+            // otherwise the collection's bytes are kept.
+            let bytes = match self.file_data.as_contiguous() {
+                Some(_) => None,
+                None => Some((base, bytes)),
+            };
+            let coll = CachedCollection::new(index, bytes);
+            if self.cached_bytes.saturating_add(coll.cost()) > self.budget {
+                self.cache.clear();
+                self.cached_bytes = 0;
+            }
+            self.cached_bytes += coll.cost();
+            self.cache.insert(addr, coll);
+        }
+        self.cache[&addr]
+            .get(vl.object_index)
+            .ok_or(FormatError::GlobalHeapObjectNotFound {
+                collection_address: addr,
+                index: vl.object_index as u16,
+            })
+    }
+
+    /// The heap object `vl` points to, whatever its size; its collection is
+    /// parsed on first use.
+    fn object(&mut self, vl: &VlElement) -> Result<&[u8], FormatError> {
+        let (start, size) = self.locate(vl)?;
+        if let Some(all) = self.file_data.as_contiguous() {
+            return Ok(&all[start..start + size]);
+        }
+        match &self.cache[&vl.collection_address].bytes {
+            Some((base, bytes)) => Ok(&bytes[start - base..start - base + size]),
+            None => Err(FormatError::Storage(
+                "global heap collection bytes were not kept".into(),
+            )),
+        }
+    }
+
+    /// Record the collection at `start..end`, refusing one that overlaps a
+    /// collection already read. libhdf5 allocates each collection its own
+    /// block; overlapping ones only come from a crafted file, where they let
+    /// every byte be parsed again as the objects of each collection.
+    fn check_overlap(&mut self, start: usize, end: usize) -> Result<(), FormatError> {
+        if let Some(&known) = self.extents.get(&start) {
+            return if known == end {
+                Ok(())
+            } else {
+                Err(FormatError::VlDataError(format!(
+                    "global heap collection at {start} changed size"
+                )))
+            };
+        }
+        let before = self.extents.range(..start).next_back();
+        let after = self.extents.range(start..).next();
+        let clash = match (before, after) {
+            (Some((&s, &e)), _) if e > start => Some(s),
+            (_, Some((&s, _))) if s < end => Some(s),
+            _ => None,
+        };
+        if let Some(other) = clash {
+            return Err(FormatError::VlDataError(format!(
+                "global heap collection at {start} overlaps the one at {other}"
+            )));
+        }
+        self.extents.insert(start, end);
+        Ok(())
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -285,16 +640,53 @@ mod tests {
     }
 
     #[test]
-    fn null_vl_element_empty_string() {
+    fn an_undefined_heap_address_is_an_error_even_at_length_0() {
-        // length=0, address=undefined
+        // libhdf5 fails the read ("addr undefined"); h5py and libhdf5 write
-        let mut raw = Vec::new();
+        // a null element with address 0. We returned "".
-        raw.extend_from_slice(&0u32.to_le_bytes()); // length=0
+        let mut file_data = vec![0u8; 256];
-        raw.extend_from_slice(&u64::MAX.to_le_bytes()); // undefined address
+        build_gcol_at(&mut file_data, 64, &[(1, b"x")]);
-        raw.extend_from_slice(&0u32.to_le_bytes()); // index
+        for (os, undef) in [(8u8, u64::MAX), (4, 0xFFFF_FFFF)] {
+            for length in [0, 1] {
+                let mut raw = element(1, 64, 1, os);
+                raw.extend(element(length, undef, 1, os));
+                let mut r = VlResolver::new(&file_data, os, 8);
+                let e = r.string_bytes(&raw).unwrap_err().to_string();
+                assert!(e.contains("undefined"), "{e}");
+                assert!(r.sequences(&raw, 1).is_err());
+                assert!(r.string_element(&raw[raw.len() / 2..]).is_err());
+                let n = 2;
+                assert!(read_vl_strings(&file_data, &raw, n, os, 8).is_err());
+                assert!(read_vl_bytes(&file_data, &raw, n, os, 8).is_err());
+                // The defined element alone still reads.
+                assert_eq!(r.strings(&raw[..raw.len() / 2]).unwrap(), ["x"]);
+            }
+        }
+    }
 
-        let file_data = vec![0u8; 16];
+    #[test]
-        let strings = read_vl_strings(&file_data, &raw, 1, 8, 8).unwrap();
+    fn element_in_over_a_storage_matches_element_over_a_slice() {
-        assert_eq!(strings, vec![""]);
+        let mut file_data = vec![0u8; 512];
+        build_gcol_at(&mut file_data, 256, &[(1, b"Alice\0x"), (2, b"Bob")]);
+        let mut raw = build_vl_refs(&["Alice\0x", "Bob"], 256, 1, 8);
+        raw.extend(element(0, 0, 0, 8)); // null
+        raw.extend(element(9, 256, 1, 8)); // wrong length: an error
+        let storage = crate::storage::CountingStorage::new(file_data.clone());
+        let dynamic: &dyn crate::storage::Storage = &storage;
+        let mut slice = VlResolver::new(&file_data, 8, 8);
+        let mut any = VlResolver::new_in(dynamic, 8, 8);
+        for e in raw.chunks(16) {
+            let want = slice.element(e, 1).map(|o| o.map(<[u8]>::to_vec));
+            let got = any.element_in(e, 1).map(|o| o.map(<[u8]>::to_vec));
+            assert_eq!(format!("{want:?}"), format!("{got:?}"));
+            let want = slice.string_element(e).map(|o| o.map(<[u8]>::to_vec));
+            let got = any.string_element_in(e).map(|o| o.map(<[u8]>::to_vec));
+            assert_eq!(format!("{want:?}"), format!("{got:?}"));
+        }
+        assert_eq!(
+            any.string_element_in(&raw[..16]).unwrap(),
+            Some(&b"Alice"[..])
+        );
+        assert!(storage.reads() > 0);
     }
 
     #[test]
@@ -333,6 +725,127 @@ mod tests {
         assert_eq!(bytes, vec![vec![0xDE, 0xAD], vec![0xBE, 0xEF, 0xCA]]);
     }
 
+    fn element(length: u32, addr: u64, index: u32, offset_size: u8) -> Vec<u8> {
+        let mut raw = length.to_le_bytes().to_vec();
+        raw.extend_from_slice(&addr.to_le_bytes()[..offset_size as usize]);
+        raw.extend_from_slice(&index.to_le_bytes());
+        raw
+    }
+
+    #[test]
+    fn strings_end_at_the_first_nul() {
+        // libhdf5 hands each VL string over as a C string, so h5py sees
+        // "a\0b" as "a"; we used to return the NUL and what followed.
+        let mut file_data = vec![0u8; 512];
+        build_gcol_at(&mut file_data, 64, &[(1, b"a\0b"), (2, b"cd")]);
+        let mut raw = element(3, 64, 1, 8);
+        raw.extend(element(2, 64, 2, 8));
+        let mut r = VlResolver::new(&file_data, 8, 8);
+        assert_eq!(
+            r.string_bytes(&raw).unwrap(),
+            vec![b"a".to_vec(), b"cd".to_vec()]
+        );
+        assert_eq!(
+            read_vl_strings(&file_data, &raw, 2, 8, 8).unwrap(),
+            ["a", "cd"]
+        );
+    }
+
+    #[test]
+    fn a_heap_object_of_the_wrong_size_is_an_error() {
+        // libhdf5: "Expected global heap object size does not match". We
+        // used to return the object cut to the element's length.
+        let mut file_data = vec![0u8; 512];
+        build_gcol_at(&mut file_data, 64, &[(1, b"cdefgh"), (2, &[1, 0, 0, 0])]);
+        let mut r = VlResolver::new(&file_data, 8, 8);
+        assert!(r.string_bytes(&element(3, 64, 1, 8)).is_err());
+        assert!(r.string_bytes(&element(9, 64, 1, 8)).is_err());
+        assert!(read_vl_strings(&file_data, &element(3, 64, 1, 8), 1, 8, 8).is_err());
+        // A sequence of one i32 is 4 bytes; of two, 8.
+        assert_eq!(
+            r.sequences(&element(1, 64, 2, 8), 4).unwrap(),
+            vec![vec![1, 0, 0, 0]]
+        );
+        assert!(r.sequences(&element(2, 64, 2, 8), 4).is_err());
+        assert!(r.sequences(&element(1, 64, 2, 8), 0).is_err());
+    }
+
+    #[test]
+    fn address_zero_is_null_whatever_the_length() {
+        // libhdf5 treats a heap address of 0 as a null element.
+        let file_data = vec![0u8; 64];
+        let mut r = VlResolver::new(&file_data, 8, 8);
+        assert_eq!(
+            r.string_bytes(&element(5, 0, 1, 8)).unwrap(),
+            vec![Vec::<u8>::new()]
+        );
+        assert_eq!(
+            r.sequences(&element(5, 0, 1, 8), 4).unwrap(),
+            vec![Vec::<u8>::new()]
+        );
+    }
+
+    #[test]
+    fn four_byte_offsets_use_twelve_byte_elements() {
+        let mut file_data = vec![0u8; 512];
+        build_gcol_at(&mut file_data, 64, &[(1, b"one"), (2, b""), (3, b"three")]);
+        let mut raw = element(3, 64, 1, 4);
+        raw.extend(element(0, 64, 2, 4));
+        raw.extend(element(5, 64, 3, 4));
+        assert_eq!(raw.len(), 36);
+        let mut r = VlResolver::new(&file_data, 4, 8);
+        assert_eq!(r.element_size(), 12);
+        assert_eq!(r.strings(&raw).unwrap(), ["one", "", "three"]);
+        // Not a whole number of elements.
+        assert!(r.strings(&raw[..30]).is_err());
+    }
+
+    #[test]
+    fn the_cache_stays_within_its_budget_and_rereads_what_it_dropped() {
+        // Twenty collections of three objects each; a budget that holds
+        // about two of them. Reading every element twice must still return
+        // the right strings after the cache is dropped.
+        let mut file_data = vec![0u8; 64];
+        let mut raw = Vec::new();
+        for c in 0..20u64 {
+            let at = file_data.len();
+            let names: Vec<String> = (0..3).map(|i| format!("c{c}o{i}")).collect();
+            let objs: Vec<(u16, &[u8])> = names
+                .iter()
+                .enumerate()
+                .map(|(i, n)| (i as u16 + 1, n.as_bytes()))
+                .collect();
+            build_gcol_at(&mut file_data, at, &objs);
+            for (i, n) in names.iter().enumerate() {
+                raw.extend(element(n.len() as u32, at as u64, i as u32 + 1, 8));
+            }
+        }
+        raw.extend(raw.clone());
+        let mut r = VlResolver::new(&file_data, 8, 8);
+        let one = CachedCollection {
+            objects: vec![(0, 0, 0); 3],
+            bytes: None,
+        }
+        .cost();
+        r.budget = 2 * one + 1;
+        let want: Vec<String> = (0..2)
+            .flat_map(|_| (0..20).flat_map(|c| (0..3).map(move |i| format!("c{c}o{i}"))))
+            .collect();
+        for (k, chunk) in raw.chunks(16).enumerate() {
+            assert_eq!(r.strings(chunk).unwrap(), [want[k].clone()]);
+            assert!(r.cached_bytes <= r.budget);
+            assert!(r.cache.len() <= 2);
+        }
+    }
+
+    #[test]
+    fn element_size_is_checked_against_the_offset_size() {
+        assert!(check_element_size(16, 8).is_ok());
+        assert!(check_element_size(12, 4).is_ok());
+        assert!(check_element_size(16, 4).is_err());
+        assert!(check_element_size(524_304, 8).is_err());
+    }
+
     #[test]
     fn parse_vl_references_truncated_error() {
         let raw = vec![0u8; 10]; // too short for 1 element with offset_size=8

crates/clawhdf5-format/src/writer_tree.rs

@@ -0,0 +1,494 @@
+//! The group hierarchy `FileWriter` writes: builders flattened into a tree
+//! of groups, datasets and links, with path names expanded into
+//! intermediate groups, hard links resolved to objects, reference counts
+//! counted, and everything put in layout order.
+
+#[cfg(not(feature = "std"))]
+use alloc::{
+    collections::BTreeMap,
+    format,
+    string::{String, ToString},
+    vec,
+    vec::Vec,
+};
+#[cfg(feature = "std")]
+use std::collections::BTreeMap;
+
+use crate::error::FormatError;
+use crate::type_builders::{AttrValue, DatasetBuilder, GroupBuilder, GroupItem};
+
+/// Depth of the chain of unresolved hard links followed while resolving one
+/// hard-link target path (a bound on recursion; cycles are found exactly).
+const MAX_LINK_DEPTH: usize = 64;
+
+fn err(msg: String) -> FormatError {
+    FormatError::SerializationError(msg)
+}
+
+/// A link name must be one path component: not empty, not ".", and without
+/// '/' (a '/' separates components, so it cannot be part of a name).
+fn check_link_name(name: &str, path: &str) -> Result<(), FormatError> {
+    if name.is_empty() || name == "." || name.contains('/') {
+        return Err(err(format!(
+            "invalid object name {path:?}: every path component must be a \
+             non-empty name other than \".\""
+        )));
+    }
+    Ok(())
+}
+
+/// What a link in the final tree points at.
+#[derive(Debug, Clone, PartialEq)]
+pub(crate) enum LinkTo {
+    /// A group, by index into [`Tree::groups`] (layout order).
+    Group(usize),
+    /// A dataset, by index into [`Tree::datasets`] (layout order).
+    Dataset(usize),
+    Soft(String),
+    External {
+        file: String,
+        path: String,
+    },
+}
+
+pub(crate) struct Link {
+    pub(crate) name: String,
+    pub(crate) to: LinkTo,
+    /// Set when the group tracks creation order.
+    pub(crate) creation_order: Option<u64>,
+}
+
+pub(crate) struct Group {
+    pub(crate) attrs: Vec<(String, AttrValue)>,
+    /// Links in the order they are written.
+    pub(crate) links: Vec<Link>,
+    pub(crate) track_order: bool,
+    /// Number of hard links to this group (the root counts one for the
+    /// superblock's reference).
+    pub(crate) refcount: u32,
+}
+
+/// The flattened file: groups (root first) and datasets, both in the order
+/// they are laid out in the file.
+pub(crate) struct Tree {
+    pub(crate) groups: Vec<Group>,
+    pub(crate) datasets: Vec<(DatasetBuilder, u32)>,
+}
+
+// ---- construction ----
+
+enum Target {
+    Group(usize),
+    Dataset(usize),
+    Soft(String),
+    Hard(String),
+    External { file: String, path: String },
+}
+
+struct BuildGroup {
+    /// Full path, for messages.
+    path: String,
+    attrs: Vec<(String, AttrValue)>,
+    links: Vec<(String, Target)>,
+    by_name: BTreeMap<String, usize>,
+    track_order: Option<bool>,
+}
+
+struct Builder {
+    groups: Vec<BuildGroup>,
+    datasets: Vec<DatasetBuilder>,
+}
+
+fn join(parent: &str, name: &str) -> String {
+    if parent == "/" {
+        format!("/{name}")
+    } else {
+        format!("{parent}/{name}")
+    }
+}
+
+impl Builder {
+    fn new_group(&mut self, path: String) -> usize {
+        self.groups.push(BuildGroup {
+            path,
+            attrs: Vec::new(),
+            links: Vec::new(),
+            by_name: BTreeMap::new(),
+            track_order: None,
+        });
+        self.groups.len() - 1
+    }
+
+    /// Split `path` (relative to group `g`) into the group holding its last
+    /// component, creating missing intermediate groups, and that component.
+    fn parent_of<'p>(&mut self, g: usize, path: &'p str) -> Result<(usize, &'p str), FormatError> {
+        // An absolute path is accepted at the root only.
+        let rel = match path.strip_prefix('/') {
+            Some(rest) if g == 0 => rest,
+            Some(_) => {
+                return Err(err(format!(
+                    "invalid object name {path:?} in {}: absolute paths are accepted \
+                     only at the root",
+                    self.groups[g].path
+                )));
+            }
+            None => path,
+        };
+        let mut comps: Vec<&str> = rel.split('/').collect();
+        let last = comps.pop().unwrap_or("");
+        check_link_name(last, path)?;
+        let mut cur = g;
+        for c in comps {
+            check_link_name(c, path)?;
+            cur = match self.groups[cur].by_name.get(c).copied() {
+                Some(i) => match self.groups[cur].links[i].1 {
+                    Target::Group(child) => child,
+                    _ => {
+                        return Err(err(format!(
+                            "cannot create {path:?} in {}: {c:?} exists and is not a group",
+                            self.groups[g].path
+                        )));
+                    }
+                },
+                None => {
+                    let child = self.new_group(join(&self.groups[cur].path, c));
+                    self.push_link(cur, c, Target::Group(child))?;
+                    child
+                }
+            };
+        }
+        Ok((cur, last))
+    }
+
+    fn push_link(&mut self, g: usize, name: &str, to: Target) -> Result<(), FormatError> {
+        let grp = &mut self.groups[g];
+        if grp.by_name.contains_key(name) {
+            return Err(err(format!("{:?} already exists", join(&grp.path, name))));
+        }
+        grp.by_name.insert(name.to_string(), grp.links.len());
+        grp.links.push((name.to_string(), to));
+        Ok(())
+    }
+
+    /// Add `item` to group `g`.
+    fn add_item(&mut self, g: usize, item: GroupItem) -> Result<(), FormatError> {
+        match item {
+            GroupItem::Dataset(db) => {
+                let (parent, name) = self.parent_of(g, &db.name)?;
+                let name = name.to_string();
+                self.push_link(parent, &name, Target::Dataset(self.datasets.len()))?;
+                self.datasets.push(*db);
+            }
+            GroupItem::Group(gb) => self.add_group(g, gb)?,
+            GroupItem::Soft { name, target } => {
+                if target.is_empty() {
+                    return Err(err(format!("soft link {name:?} has an empty target")));
+                }
+                let (parent, last) = self.parent_of(g, &name)?;
+                self.push_link(parent, last, Target::Soft(target))?;
+            }
+            GroupItem::Hard { name, target } => {
+                let (parent, last) = self.parent_of(g, &name)?;
+                self.push_link(parent, last, Target::Hard(target))?;
+            }
+            GroupItem::External { name, file, path } => {
+                if file.is_empty() || path.is_empty() {
+                    return Err(err(format!(
+                        "external link {name:?} needs a file name and an object path"
+                    )));
+                }
+                let (parent, last) = self.parent_of(g, &name)?;
+                self.push_link(parent, last, Target::External { file, path })?;
+            }
+        }
+        Ok(())
+    }
+
+    /// Add the group `gb` (named by a path relative to group `g`), merging it
+    /// into a group already at that path.
+    fn add_group(&mut self, g: usize, gb: GroupBuilder) -> Result<(), FormatError> {
+        let (parent, last) = self.parent_of(g, &gb.name)?;
+        let idx = match self.groups[parent].by_name.get(last).copied() {
+            Some(i) => match self.groups[parent].links[i].1 {
+                Target::Group(child) => child,
+                _ => {
+                    return Err(err(format!(
+                        "{:?} already exists and is not a group",
+                        join(&self.groups[parent].path, last)
+                    )));
+                }
+            },
+            None => {
+                let child = self.new_group(join(&self.groups[parent].path, last));
+                self.push_link(parent, last, Target::Group(child))?;
+                child
+            }
+        };
+        self.merge_into(idx, gb)
+    }
+
+    /// Merge a builder's attributes, setting and items into group `idx`.
+    fn merge_into(&mut self, idx: usize, gb: GroupBuilder) -> Result<(), FormatError> {
+        // An attribute set again (by this builder or a merged one) takes the
+        // new value, as assigning `attrs[name]` in h5py does.
+        for (name, value) in gb.attrs {
+            let attrs = &mut self.groups[idx].attrs;
+            match attrs.iter_mut().find(|(n, _)| *n == name) {
+                Some(slot) => slot.1 = value,
+                None => attrs.push((name, value)),
+            }
+        }
+        if let Some(t) = gb.track_order {
+            match self.groups[idx].track_order {
+                Some(old) if old != t => {
+                    return Err(err(format!(
+                        "conflicting track_order settings for {}",
+                        self.groups[idx].path
+                    )));
+                }
+                _ => self.groups[idx].track_order = Some(t),
+            }
+        }
+        for item in gb.items {
+            self.add_item(idx, item)?;
+        }
+        Ok(())
+    }
+
+    /// The object a hard link's `target` path names, from group `from`.
+    ///
+    /// Hard links met on the way are resolved once and remembered in
+    /// `memo` (by group and link index), so a target that goes through
+    /// other hard links costs time linear in the links, not exponential; a
+    /// hard link met again while it is being resolved is a cycle.
+    fn resolve(
+        &self,
+        memo: &mut [Vec<Resolution>],
+        from: usize,
+        target: &str,
+        depth: usize,
+    ) -> Result<Obj, FormatError> {
+        if depth > MAX_LINK_DEPTH {
+            return Err(err(format!(
+                "hard link target {target:?}: more than {MAX_LINK_DEPTH} hard links \
+                 to follow"
+            )));
+        }
+        let (mut cur, rest) = match target.strip_prefix('/') {
+            Some(rest) => (0, rest),
+            None => (from, target),
+        };
+        if target.is_empty() {
+            return Err(err("a hard link needs a target path".to_string()));
+        }
+        let comps: Vec<&str> = rest
+            .split('/')
+            .filter(|c| !c.is_empty() && *c != ".")
+            .collect();
+        let mut obj = Obj::Group(cur);
+        for (i, c) in comps.iter().enumerate() {
+            let Obj::Group(g) = obj else {
+                return Err(err(format!(
+                    "hard link target {target:?}: {:?} is not a group",
+                    comps[..i].join("/")
+                )));
+            };
+            cur = g;
+            let grp = &self.groups[cur];
+            let Some(&li) = grp.by_name.get(*c) else {
+                return Err(err(format!(
+                    "hard link target {target:?} does not exist in the file"
+                )));
+            };
+            obj = match &grp.links[li].1 {
+                Target::Group(child) => Obj::Group(*child),
+                Target::Dataset(d) => Obj::Dataset(*d),
+                Target::Hard(p) => match memo[cur][li] {
+                    Resolution::Done(o) => o,
+                    Resolution::InProgress => {
+                        return Err(err(format!(
+                            "hard link target {target:?}: the hard link {:?} leads \
+                             back to itself (a cycle)",
+                            join(&grp.path, c)
+                        )));
+                    }
+                    Resolution::Todo => {
+                        memo[cur][li] = Resolution::InProgress;
+                        let o = self.resolve(memo, cur, p, depth + 1)?;
+                        memo[cur][li] = Resolution::Done(o);
+                        o
+                    }
+                },
+                Target::Soft(_) | Target::External { .. } => {
+                    return Err(err(format!(
+                        "hard link target {target:?} goes through a soft or external \
+                         link ({:?}); name the object by its hard-link path",
+                        join(&grp.path, c)
+                    )));
+                }
+            };
+        }
+        Ok(obj)
+    }
+}
+
+/// Where resolving one hard link has got to.
+#[derive(Clone, Copy)]
+enum Resolution {
+    Todo,
+    InProgress,
+    Done(Obj),
+}
+
+#[derive(Clone, Copy)]
+enum Obj {
+    Group(usize),
+    Dataset(usize),
+}
+
+/// Flatten the root group builder into a [`Tree`]. `default_track_order`
+/// applies to every group that does not set its own.
+pub(crate) fn build(root: GroupBuilder, default_track_order: bool) -> Result<Tree, FormatError> {
+    let mut b = Builder {
+        groups: Vec::new(),
+        datasets: Vec::new(),
+    };
+    b.new_group("/".to_string());
+    b.merge_into(0, root)?;
+
+    // Resolve hard links and count references.
+    let mut group_refs = vec![0u32; b.groups.len()];
+    let mut ds_refs = vec![0u32; b.datasets.len()];
+    group_refs[0] = 1; // the superblock's reference to the root
+    let mut memo: Vec<Vec<Resolution>> = b
+        .groups
+        .iter()
+        .map(|g| vec![Resolution::Todo; g.links.len()])
+        .collect();
+    let mut resolved: Vec<Vec<Option<Obj>>> = Vec::with_capacity(b.groups.len());
+    for (gi, g) in b.groups.iter().enumerate() {
+        let mut row = Vec::with_capacity(g.links.len());
+        for (li, (_, t)) in g.links.iter().enumerate() {
+            let obj = match t {
+                Target::Group(i) => Some(Obj::Group(*i)),
+                Target::Dataset(d) => Some(Obj::Dataset(*d)),
+                Target::Hard(p) => Some(match memo[gi][li] {
+                    Resolution::Done(o) => o,
+                    _ => {
+                        memo[gi][li] = Resolution::InProgress;
+                        let o = b.resolve(&mut memo, gi, p, 0)?;
+                        memo[gi][li] = Resolution::Done(o);
+                        o
+                    }
+                }),
+                Target::Soft(_) | Target::External { .. } => None,
+            };
+            match obj {
+                Some(Obj::Group(i)) => group_refs[i] += 1,
+                Some(Obj::Dataset(d)) => ds_refs[d] += 1,
+                None => {}
+            }
+            row.push(obj);
+        }
+        resolved.push(row);
+    }
+
+    // The order each group's links are written in: creation order when
+    // tracked; otherwise datasets, then groups, then other links (the order
+    // earlier versions wrote, so one-level files keep their layout).
+    let tracked: Vec<bool> = b
+        .groups
+        .iter()
+        .map(|g| g.track_order.unwrap_or(default_track_order))
+        .collect();
+    let link_order: Vec<Vec<usize>> = b
+        .groups
+        .iter()
+        .enumerate()
+        .map(|(gi, g)| {
+            let mut idx: Vec<usize> = (0..g.links.len()).collect();
+            if !tracked[gi] {
+                idx.sort_by_key(|&i| match g.links[i].1 {
+                    Target::Dataset(_) => 0,
+                    Target::Group(_) => 1,
+                    _ => 2,
+                });
+            }
+            idx
+        })
+        .collect();
+
+    // Layout order: groups depth-first from the root, following the links
+    // that created them; datasets group by group in that order.
+    let mut group_order = Vec::with_capacity(b.groups.len());
+    let mut stack = vec![0usize];
+    while let Some(g) = stack.pop() {
+        group_order.push(g);
+        let children: Vec<usize> = link_order[g]
+            .iter()
+            .filter_map(|&i| match b.groups[g].links[i].1 {
+                Target::Group(c) => Some(c),
+                _ => None,
+            })
+            .collect();
+        stack.extend(children.into_iter().rev());
+    }
+    let mut ds_order = Vec::with_capacity(b.datasets.len());
+    for &g in &group_order {
+        for &i in &link_order[g] {
+            if let Target::Dataset(d) = b.groups[g].links[i].1 {
+                ds_order.push(d);
+            }
+        }
+    }
+    let mut group_pos = vec![0usize; b.groups.len()];
+    for (pos, &g) in group_order.iter().enumerate() {
+        group_pos[g] = pos;
+    }
+    let mut ds_pos = vec![0usize; b.datasets.len()];
+    for (pos, &d) in ds_order.iter().enumerate() {
+        ds_pos[d] = pos;
+    }
+
+    let mut groups_by_id: Vec<Option<BuildGroup>> = b.groups.into_iter().map(Some).collect();
+    let mut groups = Vec::with_capacity(group_order.len());
+    for &g in &group_order {
+        let bg = groups_by_id[g].take().expect("each group is laid out once");
+        let mut targets: Vec<Option<(String, Target)>> = bg.links.into_iter().map(Some).collect();
+        let links = link_order[g]
+            .iter()
+            .map(|&i| {
+                let (name, t) = targets[i].take().expect("each link is written once");
+                let to = match (resolved[g][i], t) {
+                    (Some(Obj::Group(c)), _) => LinkTo::Group(group_pos[c]),
+                    (Some(Obj::Dataset(d)), _) => LinkTo::Dataset(ds_pos[d]),
+                    (None, Target::Soft(s)) => LinkTo::Soft(s),
+                    (None, Target::External { file, path }) => LinkTo::External { file, path },
+                    (None, _) => unreachable!("hard links are resolved"),
+                };
+                Link {
+                    name,
+                    to,
+                    creation_order: tracked[g].then_some(i as u64),
+                }
+            })
+            .collect();
+        groups.push(Group {
+            attrs: bg.attrs,
+            links,
+            track_order: tracked[g],
+            refcount: group_refs[g],
+        });
+    }
+    let mut ds_by_id: Vec<Option<DatasetBuilder>> = b.datasets.into_iter().map(Some).collect();
+    let datasets = ds_order
+        .iter()
+        .map(|&d| {
+            (
+                ds_by_id[d].take().expect("each dataset is laid out once"),
+                ds_refs[d],
+            )
+        })
+        .collect();
+    Ok(Tree { groups, datasets })
+}

crates/clawhdf5-format/tests/blosc2_alloc_bounds.rs

@@ -0,0 +1,641 @@
+//! Crafted Blosc2 frames and chunks cannot make the decoder allocate out of
+//! proportion to the HDF5 chunk it decodes.
+//!
+//! A frame's header, its offsets chunk and its chunk headers all declare
+//! sizes, and the decoder used to allocate what they declared: a 173-byte
+//! frame whose offsets chunk claimed 2 GiB was decoded in full before any
+//! check failed. Every allocation is now bounded by the output limit (the
+//! HDF5 chunk's size) and the input's length.
+//!
+//! Peak heap use is measured with a counting global allocator; the tests
+//! share it, so each holds `SERIAL` for its whole run.
+#![cfg(feature = "blosc2")]
+
+use std::alloc::{GlobalAlloc, Layout, System};
+use std::sync::Mutex;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+use clawhdf5_format::filters_blosc2::{blosc2_decompress, blosc2_decompress_chunk};
+
+struct Counting;
+
+static CURRENT: AtomicUsize = AtomicUsize::new(0);
+static PEAK: AtomicUsize = AtomicUsize::new(0);
+static SERIAL: Mutex<()> = Mutex::new(());
+
+unsafe impl GlobalAlloc for Counting {
+    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
+        let p = unsafe { System.alloc(layout) };
+        if !p.is_null() {
+            let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
+            PEAK.fetch_max(now, Ordering::Relaxed);
+        }
+        p
+    }
+
+    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
+        let p = unsafe { System.alloc_zeroed(layout) };
+        if !p.is_null() {
+            let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
+            PEAK.fetch_max(now, Ordering::Relaxed);
+        }
+        p
+    }
+
+    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
+        unsafe { System.dealloc(ptr, layout) };
+        CURRENT.fetch_sub(layout.size(), Ordering::Relaxed);
+    }
+}
+
+#[global_allocator]
+static ALLOC: Counting = Counting;
+
+/// Bytes allocated at the peak of `f`, above what was live when it started.
+fn peak_during<T>(f: impl FnOnce() -> T) -> (T, usize) {
+    let base = CURRENT.load(Ordering::Relaxed);
+    PEAK.store(base, Ordering::Relaxed);
+    let out = f();
+    (out, PEAK.load(Ordering::Relaxed).saturating_sub(base))
+}
+
+/// What decoding one HDF5 chunk of `limit` bytes from `input` may hold at
+/// once: the output, a few blocks of scratch (each no larger than the
+/// output), the offsets table, and the Zstandard decoder's state, which has
+/// a fixed ceiling: a window of at most 128 KiB (or twice the stream) and a
+/// block's table of sequences (up to 98,303 of 12 bytes, 1.2 MB).
+fn bound(limit: usize, input: &[u8]) -> usize {
+    6 * limit + 2 * input.len() + (2 << 20)
+}
+
+fn lock() -> std::sync::MutexGuard<'static, ()> {
+    SERIAL.lock().unwrap_or_else(|e| e.into_inner())
+}
+
+/// A 32-byte (extended) Blosc2 chunk header.
+fn chunk_header(ts: u8, nbytes: i32, blocksize: i32, cbytes: i32, special: u8) -> Vec<u8> {
+    let mut c = vec![5u8, 1, 0x05, ts];
+    for v in [nbytes, blocksize, cbytes] {
+        c.extend_from_slice(&v.to_le_bytes());
+    }
+    c.resize(32, 0);
+    c[31] = special << 4;
+    c
+}
+
+/// A chunk of `nbytes` bytes that repeats one value (special type 3).
+fn repeated(value: &[u8], nbytes: i32, blocksize: i32) -> Vec<u8> {
+    let mut c = chunk_header(
+        value.len() as u8,
+        nbytes,
+        blocksize,
+        32 + value.len() as i32,
+        3,
+    );
+    c.extend_from_slice(value);
+    c
+}
+
+/// A frame offset recording a special chunk of `kind` (1 zeros, 2 NaN).
+fn special_offset(kind: u8) -> [u8; 8] {
+    (((0x80 | kind) as i64) << 56).to_le_bytes()
+}
+
+/// A B2ND metalayer.
+fn nd_meta(shape: &[i64], chunks: &[i32], blocks: &[i32]) -> Vec<u8> {
+    let n = shape.len() as u8;
+    let mut m = vec![0x95, 0, n, 0x90 | n];
+    for s in shape {
+        m.push(0xd3);
+        m.extend_from_slice(&s.to_be_bytes());
+    }
+    for dims in [chunks, blocks] {
+        m.push(0x90 | n);
+        for d in dims {
+            m.push(0xd2);
+            m.extend_from_slice(&d.to_be_bytes());
+        }
+    }
+    m
+}
+
+/// A contiguous frame: header (with a `b2nd` metalayer if given), the data
+/// chunks, then the offsets chunk.
+fn frame(
+    meta: Option<&[u8]>,
+    nbytes: i64,
+    typesize: i32,
+    chunksize: i32,
+    data: &[u8],
+    offsets: &[u8],
+) -> Vec<u8> {
+    let mut h = vec![0u8; 91];
+    h[0] = 0x9e;
+    h[1] = 0xa8;
+    h[2..10].copy_from_slice(b"b2frame\0");
+    h[25] = 2;
+    match meta {
+        Some(m) => {
+            h.extend_from_slice(&[0xde, 0, 1, 0xa4]);
+            h.extend_from_slice(b"b2nd");
+            let at = h.len() as i32 + 5;
+            h.push(0xd2);
+            h.extend_from_slice(&at.to_be_bytes());
+            h.push(0xc6);
+            h.extend_from_slice(&(m.len() as u32).to_be_bytes());
+            h.extend_from_slice(m);
+        }
+        None => h.extend_from_slice(&[0xde, 0, 0]),
+    }
+    let header_len = h.len() as i32;
+    h[11..15].copy_from_slice(&header_len.to_be_bytes());
+    h[30..38].copy_from_slice(&nbytes.to_be_bytes());
+    h[39..47].copy_from_slice(&(data.len() as i64).to_be_bytes());
+    h[48..52].copy_from_slice(&typesize.to_be_bytes());
+    h[58..62].copy_from_slice(&chunksize.to_be_bytes());
+    h.extend_from_slice(data);
+    h.extend_from_slice(offsets);
+    let len = h.len() as u64;
+    h[16..24].copy_from_slice(&len.to_be_bytes());
+    h
+}
+
+/// The frame header's own sizes must not size the offsets chunk: a frame
+/// declaring 32 Mi chunks of 4 bytes, whose offsets chunk (40 bytes) says
+/// "one repeated offset, 256 MiB of them", made the decoder build all
+/// 256 MiB of offsets for a 1 MiB HDF5 chunk and then return 4 bytes.
+#[test]
+fn offsets_chunk_is_bounded_by_the_output_limit() {
+    let _g = lock();
+    let limit = 1 << 20;
+    let offsets_len: i32 = 256 << 20;
+    let nchunks = offsets_len as i64 / 8;
+    let offsets = repeated(&special_offset(1), offsets_len, 64 << 20);
+    let f = frame(None, nchunks * 4, 4, 4, &[], &offsets);
+    let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+    assert!(r.is_err(), "decoded {:?} bytes", r.map(|v| v.len()));
+    assert!(
+        peak <= bound(limit, &f),
+        "peak {peak} bytes for a {}-byte frame",
+        f.len()
+    );
+    // The same frame with a variable chunk size (0): the offsets chunk
+    // alone says how many chunks there are.
+    let f = frame(None, nchunks * 4, 4, 0, &[], &offsets);
+    let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+    assert!(r.is_err());
+    assert!(peak <= bound(limit, &f), "chunksize 0: peak {peak} bytes");
+}
+
+/// A legitimate frame of this shape (one chunk, its offset special) still
+/// decodes.
+#[test]
+fn small_frames_still_decode() {
+    let _g = lock();
+    let offsets = repeated(&special_offset(1), 8, 8);
+    let f = frame(None, 64, 4, 64, &[], &offsets);
+    assert_eq!(blosc2_decompress(&f, 64).unwrap(), vec![0; 64]);
+    let _ = blosc2_decompress_chunk;
+}
+
+/// A chunk that decodes to nothing kept its declared block size (up to
+/// 512 MiB) and allocated two scratch blocks of it: about 1 GiB for a
+/// 20-byte chunk.
+#[test]
+fn empty_chunk_does_not_allocate_its_block_size() {
+    let _g = lock();
+    let mut c = vec![5u8, 1, 0x01, 1];
+    for v in [0i32, 0x1FFF_F000, 20] {
+        c.extend_from_slice(&v.to_le_bytes());
+    }
+    c.resize(20, 0);
+    let (r, peak) = peak_during(|| blosc2_decompress_chunk(&c, 1 << 20));
+    assert_eq!(r.map(|v| v.len()).unwrap_or(0), 0);
+    assert!(
+        peak <= bound(0, &c),
+        "peak {peak} bytes for a 20-byte chunk"
+    );
+    // Inside a frame for a non-empty HDF5 chunk it is an error, not data.
+    let offsets = repeated(&0i64.to_le_bytes(), 8, 8);
+    let f = frame(None, 64, 4, 64, &c, &offsets);
+    let (r, peak) = peak_during(|| blosc2_decompress(&f, 64));
+    assert!(r.is_err(), "decoded {:?}", r.map(|v| v.len()));
+    assert!(peak <= bound(64, &f), "in a frame: peak {peak} bytes");
+}
+
+/// B2ND chunks were decoded whole, padding included, with up to 16x the
+/// HDF5 chunk size as their limit. Blocks are now placed as they are
+/// decoded, so the padding is never held.
+///
+/// Ten dimensions: nine of 3 split into blocks of 2 (padded to 4) and one
+/// of 4, so each chunk is 13x the array. One chunk, stored three ways: as a
+/// NaN chunk in the frame's offsets, as a repeated-value chunk, and as a
+/// chunk of stored (uncompressed) blocks.
+#[test]
+fn b2nd_padding_is_never_held() {
+    let _g = lock();
+    let ts = 4usize;
+    let mut shape = vec![3i64; 9];
+    shape.push(4);
+    let chunks: Vec<i32> = shape.iter().map(|&s| s as i32).collect();
+    let mut blocks = vec![2i32; 9];
+    blocks.push(4);
+    let meta = nd_meta(&shape, &chunks, &blocks);
+    let items: usize = shape.iter().product::<i64>() as usize;
+    let limit = items * ts;
+    let block_bytes = ts * blocks.iter().product::<i32>() as usize;
+    let ext_bytes = ts * 4usize.pow(9) * 4;
+    assert!(ext_bytes > 13 * limit);
+    let offsets = |off: [u8; 8]| repeated(&off, 8, 8);
+
+    let value = 1.5f32.to_le_bytes();
+    let stored = {
+        // Every block stored raw: block k holds the value k.
+        let mut c = chunk_header(4, ext_bytes as i32, block_bytes as i32, 0, 0);
+        c[2] = 0x02 | 0x10; // memcpyed, not split
+        c.truncate(16);
+        for k in 0..ext_bytes / block_bytes {
+            c.extend((k as f32).to_le_bytes().repeat(block_bytes / 4));
+        }
+        let n = c.len() as i32;
+        c[12..16].copy_from_slice(&n.to_le_bytes());
+        c
+    };
+    let cases: Vec<(&str, Vec<u8>)> = vec![
+        (
+            "NaN offset",
+            frame(
+                Some(&meta),
+                ext_bytes as i64,
+                4,
+                ext_bytes as i32,
+                &[],
+                &offsets(special_offset(2)),
+            ),
+        ),
+        (
+            "repeated value",
+            frame(
+                Some(&meta),
+                ext_bytes as i64,
+                4,
+                ext_bytes as i32,
+                &repeated(&value, ext_bytes as i32, block_bytes as i32),
+                &offsets(0i64.to_le_bytes()),
+            ),
+        ),
+        (
+            "stored blocks",
+            frame(
+                Some(&meta),
+                ext_bytes as i64,
+                4,
+                ext_bytes as i32,
+                &stored,
+                &offsets(0i64.to_le_bytes()),
+            ),
+        ),
+    ];
+    for (name, f) in cases {
+        let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+        let out = r.unwrap_or_else(|e| panic!("{name}: {e}"));
+        assert_eq!(out.len(), limit, "{name}");
+        match name {
+            "NaN offset" => assert!(
+                out.chunks(4)
+                    .all(|v| f32::from_le_bytes(v.try_into().unwrap()).is_nan())
+            ),
+            "repeated value" => assert!(out.chunks(4).all(|v| v == value)),
+            _ => {
+                // Element (i0..i9) lies in block (i0/2, .., i8/2), numbered
+                // in C order over a 2x..x2x1 grid of blocks.
+                let mut idx = [0usize; 10];
+                for (e, v) in out.chunks(4).enumerate() {
+                    let mut n = e;
+                    for d in (0..10).rev() {
+                        idx[d] = n % shape[d] as usize;
+                        n /= shape[d] as usize;
+                    }
+                    let k = idx[..9].iter().fold(0, |k, &i| k * 2 + i / 2);
+                    assert_eq!(
+                        f32::from_le_bytes(v.try_into().unwrap()),
+                        k as f32,
+                        "{name} {e}"
+                    );
+                }
+            }
+        }
+        assert!(
+            peak <= bound(limit, &f),
+            "{name}: peak {peak} bytes for a {limit}-byte chunk ({}-byte frame)",
+            f.len()
+        );
+    }
+}
+
+/// A B2ND chunk larger than the array (here 16x, the old cap) is refused,
+/// or at least never allocated.
+#[test]
+fn b2nd_chunk_larger_than_the_array_is_not_allocated() {
+    let _g = lock();
+    let limit = 1 << 20;
+    let c = 16 * limit as i32;
+    let meta = nd_meta(&[limit as i64], &[c], &[c]);
+    let offsets = repeated(&special_offset(1), 8, 8);
+    let f = frame(Some(&meta), c as i64, 1, c, &[], &offsets);
+    let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+    assert!(
+        peak <= bound(limit, &f),
+        "peak {peak} bytes ({:?})",
+        r.map(|v| v.len())
+    );
+}
+
+/// ruzstd reserves a frame's declared window (up to 100 MiB) before it
+/// decodes a frame with a decoder it has used before: a Blosc2 chunk of
+/// two 16-byte Zstandard streams, each declaring a 96 MiB window,
+/// allocated 96 MiB. c-blosc2 compresses each block with its size known,
+/// so its windows never exceed the block.
+#[test]
+fn zstd_window_is_bounded_by_the_output() {
+    let _g = lock();
+    let mut z = 0xfd2f_b528u32.to_le_bytes().to_vec();
+    // No single segment, no checksum; window 2^26 + 4/8 of it = 96 MiB.
+    z.extend_from_slice(&[0x00, (16 << 3) | 4]);
+    // One raw block, last, of 16 bytes.
+    let h = 1 | (16 << 3);
+    z.extend_from_slice(&[h as u8, (h >> 8) as u8, 0]);
+    z.extend_from_slice(&[7; 16]);
+    // Two blocks of 16 bytes, one stream each (not split), Zstandard
+    // (codec 4).
+    let chunk = |z: &[u8]| {
+        let mut c = vec![5u8, 1, 0x10 | (4 << 5), 1];
+        for v in [32i32, 16, 0] {
+            c.extend_from_slice(&v.to_le_bytes());
+        }
+        let first = 24 + 4 + z.len();
+        c.extend_from_slice(&24i32.to_le_bytes());
+        c.extend_from_slice(&(first as i32).to_le_bytes());
+        for _ in 0..2 {
+            c.extend_from_slice(&(z.len() as i32).to_le_bytes());
+            c.extend_from_slice(z);
+        }
+        let n = c.len() as i32;
+        c[12..16].copy_from_slice(&n.to_le_bytes());
+        c
+    };
+    let c = chunk(&z);
+    let (r, peak) = peak_during(|| blosc2_decompress_chunk(&c, 32));
+    assert!(peak <= bound(32, &c), "peak {peak} bytes ({r:?})");
+    assert!(r.is_err(), "{r:?}");
+    // The same streams with a window they can use read.
+    z[5] = 0;
+    assert_eq!(
+        blosc2_decompress_chunk(&chunk(&z), 32).unwrap(),
+        vec![7; 32]
+    );
+}
+
+/// xorshift64*: deterministic, so a failure reproduces.
+struct Rng(u64);
+
+impl Rng {
+    fn next(&mut self) -> u64 {
+        let mut x = self.0;
+        x ^= x >> 12;
+        x ^= x << 25;
+        x ^= x >> 27;
+        self.0 = x;
+        x.wrapping_mul(0x2545_F491_4F6C_DD1D)
+    }
+
+    fn below(&mut self, n: usize) -> usize {
+        (self.next() % n.max(1) as u64) as usize
+    }
+
+    /// A size that tends to the edges: small, a power of two, huge.
+    fn size(&mut self) -> i64 {
+        match self.below(6) {
+            0 => self.below(64) as i64,
+            1 => 1 << self.below(31),
+            2 => i32::MAX as i64 - self.below(4096) as i64,
+            3 => (1i64 << self.below(62)) + self.below(8) as i64,
+            4 => MAX_BLOCK - self.below(3) as i64,
+            _ => self.next() as i32 as i64,
+        }
+    }
+}
+
+const MAX_BLOCK: i64 = 536_866_816;
+
+/// One to four edits: bytes, or a size field written little-endian (chunk
+/// headers) or big-endian (frame headers), most often at a header's size
+/// fields.
+fn mutate(rng: &mut Rng, seed: &[u8], data_at: usize) -> Vec<u8> {
+    let mut v = seed.to_vec();
+    for _ in 0..1 + rng.below(4) {
+        let len = v.len();
+        if len < 16 {
+            v.push(rng.next() as u8);
+            continue;
+        }
+        match rng.below(8) {
+            0 => {
+                let i = rng.below(len);
+                v[i] ^= 1 << rng.below(8);
+            }
+            1 => {
+                let i = rng.below(len);
+                v[i] = rng.next() as u8;
+            }
+            2 => {
+                // Frame header: nbytes, cbytes (i64), typesize, chunksize.
+                let x = rng.size();
+                match rng.below(4) {
+                    0 if len >= 38 => v[30..38].copy_from_slice(&x.to_be_bytes()),
+                    1 if len >= 47 => v[39..47].copy_from_slice(&x.to_be_bytes()),
+                    2 if len >= 52 => v[48..52].copy_from_slice(&(x as i32).to_be_bytes()),
+                    _ if len >= 62 => v[58..62].copy_from_slice(&(x as i32).to_be_bytes()),
+                    _ => {}
+                }
+            }
+            3 | 4 => {
+                // A chunk header's nbytes, blocksize or cbytes: in the first
+                // data chunk, or anywhere (the offsets chunk comes last).
+                let at = if rng.below(2) == 0 && data_at + 16 <= len {
+                    data_at + 4 * (1 + rng.below(3))
+                } else {
+                    rng.below(len - 3)
+                };
+                let x = rng.size() as i32;
+                v[at..at + 4].copy_from_slice(&x.to_le_bytes());
+            }
+            5 => v.truncate(rng.below(len)),
+            6 => {
+                let at = rng.below(len);
+                v[at] = [0x10, 0x20, 0x30, 0x40, 0x05, 0x07, 0x02][rng.below(7)];
+            }
+            _ => {
+                let i = rng.below(len - 3);
+                let x = rng.size() as i32;
+                v[i..i + 4].copy_from_slice(&x.to_be_bytes());
+            }
+        }
+    }
+    v
+}
+
+/// Every fixture frame that decodes, with its decoded size.
+fn seeds() -> Vec<(Vec<u8>, usize)> {
+    let dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/blosc2");
+    let mut v = Vec::new();
+    for e in std::fs::read_dir(dir).unwrap() {
+        let p = e.unwrap().path();
+        if p.extension().is_some_and(|x| x == "b2f")
+            && let Ok(out) = std::fs::read(p.with_extension("out"))
+        {
+            v.push((std::fs::read(&p).unwrap(), out.len()));
+        }
+    }
+    v.sort();
+    assert!(v.len() >= 20, "fixtures missing");
+    v
+}
+
+fn header_len(frame: &[u8]) -> usize {
+    i32::from_be_bytes(frame[11..15].try_into().unwrap()) as usize
+}
+
+/// Mutated fixture frames, decoded with their HDF5 chunk size as the
+/// limit, and their first chunks on their own: whatever they declare, no
+/// decode holds more than a small multiple of the output and the input.
+#[test]
+fn fuzzed_frames_and_chunks_stay_within_the_allocation_bound() {
+    let _g = lock();
+    let seeds = seeds();
+    let mut rng = Rng(0xb2a1);
+    let mut worst = (0.0f64, String::new());
+    for i in 0..20_000 {
+        let (seed, limit) = &seeds[rng.below(seeds.len())];
+        let f = mutate(&mut rng, seed, header_len(seed));
+        let (r, peak) = peak_during(|| blosc2_decompress(&f, *limit));
+        if let Ok(out) = &r {
+            assert!(out.len() <= *limit, "iteration {i}: output past the limit");
+        }
+        assert!(
+            peak <= bound(*limit, &f),
+            "iteration {i}: peak {peak} bytes for a {limit}-byte chunk from {} bytes ({:?})",
+            f.len(),
+            r.map(|v| v.len())
+        );
+        let ratio = peak as f64 / bound(*limit, &f) as f64;
+        if ratio > worst.0 {
+            worst = (
+                ratio,
+                format!(
+                    "frame iteration {i}: peak {peak}, limit {limit}, input {}",
+                    f.len()
+                ),
+            );
+        }
+    }
+    for i in 0..20_000 {
+        let (seed, _) = &seeds[rng.below(seeds.len())];
+        let at = header_len(seed);
+        let chunk = &seed[at..];
+        let c = mutate(&mut rng, chunk, 0);
+        let limit = 1 << 16;
+        let (r, peak) = peak_during(|| blosc2_decompress_chunk(&c, limit));
+        assert!(
+            peak <= bound(limit, &c),
+            "chunk iteration {i}: peak {peak} bytes from {} bytes ({:?})",
+            c.len(),
+            r.map(|v| v.len())
+        );
+    }
+    eprintln!("worst peak / bound: {:.2} ({})", worst.0, worst.1);
+}
+
+/// Frames built from random header sizes, offsets chunks and B2ND shapes
+/// (chunk and block shapes that pad, special and repeated-value chunks).
+#[test]
+fn random_frames_stay_within_the_allocation_bound() {
+    let _g = lock();
+    let mut rng = Rng(0xb2a2);
+    for i in 0..5_000 {
+        let ts = [1usize, 2, 4, 8][rng.below(4)];
+        let ndim = 1 + rng.below(8);
+        let mut shape = Vec::new();
+        let mut chunks = Vec::new();
+        let mut blocks = Vec::new();
+        for _ in 0..ndim {
+            let s = 1 + rng.below(if ndim > 3 { 4 } else { 40 });
+            let c = if rng.below(8) == 0 {
+                s * (1 + rng.below(4))
+            } else {
+                1 + rng.below(s)
+            };
+            let b = 1 + rng.below(c);
+            shape.push(s as i64);
+            chunks.push(c as i32);
+            blocks.push(b as i32);
+        }
+        let items: usize = shape.iter().product::<i64>() as usize;
+        let limit = items * ts;
+        let meta = nd_meta(&shape, &chunks, &blocks);
+        let ext: usize = ts
+            * chunks
+                .iter()
+                .zip(&blocks)
+                .map(|(&c, &b)| (c as usize).div_ceil(b as usize) * b as usize)
+                .product::<usize>();
+        let nchunks: usize = shape
+            .iter()
+            .zip(&chunks)
+            .map(|(&s, &c)| (s as usize).div_ceil(c as usize))
+            .product();
+        let block_bytes = ts * blocks.iter().product::<i32>() as usize;
+        let chunksize = if rng.below(4) == 0 {
+            rng.size()
+        } else {
+            ext as i64
+        };
+        let nbytes = if rng.below(4) == 0 {
+            rng.size()
+        } else {
+            (nchunks * ext) as i64
+        };
+        let off_n = if rng.below(4) == 0 {
+            rng.size() as i32
+        } else {
+            8 * nchunks as i32
+        };
+        let (data, off) = match rng.below(3) {
+            0 => (Vec::new(), special_offset(1 + rng.below(2) as u8)),
+            _ => {
+                let bs = if rng.below(4) == 0 {
+                    rng.size() as i32
+                } else {
+                    block_bytes as i32
+                };
+                let value: Vec<u8> = (0..ts).map(|_| rng.next() as u8).collect();
+                let n = if rng.below(4) == 0 {
+                    rng.size() as i32
+                } else {
+                    ext as i32
+                };
+                (repeated(&value, n, bs), 0i64.to_le_bytes())
+            }
+        };
+        let offsets = repeated(&off, off_n, off_n.clamp(1, 8));
+        let meta = (rng.below(4) != 0).then_some(meta.as_slice());
+        let f = frame(meta, nbytes, ts as i32, chunksize as i32, &data, &offsets);
+        let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+        assert!(
+            peak <= bound(limit, &f),
+            "iteration {i}: peak {peak} bytes for a {limit}-byte chunk ({:?}, shape {shape:?} \
+             chunks {chunks:?} blocks {blocks:?})",
+            r.map(|v| v.len())
+        );
+    }
+}

crates/clawhdf5-format/tests/fixtures/blosc2/bytedelta.err

@@ -0,0 +1 @@
+filter 35