BENCHMARKS.md

@@ -484,6 +484,45 @@ explain the slower windows.
 
 ## Concurrent reads
 
+### Results after in-place chunk decoding (2026-09-26, tank, `c5334b1`)
+
+Same machine, files and commands, re-run after chunked reads started
+decoding into reusable per-thread buffers straight into the (typed) output,
+with the calling thread decoding alongside the pool. Load average 1.78 at
+the start; it rose to 6-9 during the runs (the clawhdf5 runs' own threads,
+and it stayed around 5-6 through the h5py runs, so something else was
+active). **This run was noisier than the previous one: h5py's own contiguous
+figures are about 40% lower than in the run below, and ours dropped
+similarly, so compare ratios within a run rather than MB/s across runs.**
+h5py was re-run in the same session.
+
+Each read decoding on its calling thread (`--decode-threads 1`, like h5py):
+
+| layout | mode | threads | clawhdf5 MB/s (eff) | h5py threads MB/s (eff) | h5py processes MB/s (eff) | vs h5py processes |
+|---|---|---:|---:|---:|---:|---:|
+| deflate | distinct | 1 | 670 (1.00) | 410 (1.00) | 397 (1.00) | 1.69x |
+| deflate | distinct | 4 | 2434 (0.91) | 406 (0.25) | 1470 (0.93) | 1.66x |
+| deflate | distinct | 8 | 3749 (0.70) | 406 (0.12) | 2398 (0.76) | 1.56x |
+| deflate | distinct | 16 | 4944 (0.46) | 390 (0.06) | 3135 (0.49) | 1.58x |
+| deflate | same | 1 | 211 (1.00) | 125 (1.00) | 124 (1.00) | 1.70x |
+| deflate | same | 16 | 1835 (0.54) | 122 (0.06) | 961 (0.48) | 1.91x |
+| contiguous | distinct | 1 | 6718 (1.00) | 5545 (1.00) | 5200 (1.00) | 1.29x |
+| contiguous | distinct | 16 | 11035 (0.10) | 4950 (0.06) | 10558 (0.13) | 1.05x |
+| contiguous | same | 1 | 14483 (1.00) | 2593 (1.00) | 2737 (1.00) | 5.29x |
+| contiguous | same | 16 | 132175 (0.57) | 2224 (0.05) | 14809 (0.34) | 8.93x |
+
+With the default rayon pool, deflate `distinct` reads 6143 MB/s from a single
+thread (15x h5py's 410 on one call) and 4556 MB/s at 16 threads (1.45x h5py
+processes); the other rows are within the noise of the table above.
+
+What changed: full reads of chunked datasets were 0.69x-0.76x of h5py
+processes at 16 threads in the run below, and are 1.58x here; with one
+thread they were 1.44x and are 1.69x. Minor page faults for the 16-thread
+run fell from about 4.6M to 0.2M (`/usr/bin/time -v`, provisional, loaded
+machine). clawhdf5 now reads faster than 16 h5py processes in every row of
+this benchmark except contiguous full reads at 16 threads, where both
+saturate memory bandwidth (1.05x).
+
 ### Results after the read fixes (2026-09-26, tank, `408f69e`)
 
 Same machine, files and commands as the first run below, re-run on an idle

CHANGELOG.md

@@ -2,6 +2,220 @@
 
 ## Unreleased
 
+### Chunked full reads (2026-09-26)
+- **Chunks are decoded straight into the output, into reused buffers.** A
+  full read of a chunked dataset faulted in about three times its size in
+  fresh pages: every chunk was decoded into a new buffer per filter stage
+  (the cached reader behind `read_*` decoded 128 chunks at a time before
+  placing any; the uncached one behind `MmapFile`, `LazyFile` and
+  `verify_provenance` decoded the whole dataset first), then assembled into
+  a byte buffer, which the typed readers copied once more. Now each chunk
+  is decoded into buffers the thread keeps between chunks and reads
+  (`clawhdf5_format::filters::DecodeScratch`,
+  `decompress_chunk_exact_with`: deflate inflates into a kept buffer with a
+  reset inflater, shuffle into the other one, Fletcher32 is checked in
+  place; other filters go through the registry as before) and copied
+  directly to its place in the output. Chunks still go into the file's
+  chunk cache when the whole dataset fits. Selection reads decode the
+  chunks they touch the same way.
+- **Typed full reads of chunked data skip the byte buffer.** `read_f32`,
+  `read_f64`, `read_i32`, `read_i64` and `read_u64` (on `File`, `MmapFile`
+  and `LazyFile`) of a chunked dataset stored as that type in native byte
+  order decode every chunk into the returned `Vec` (huge-page backed when
+  large, like the byte readers' output); other types and byte orders
+  convert as before. New public
+  `clawhdf5_format::data_read::read_chunked_native`.
+- **Reading threads no longer wait for a busy rayon pool.** A full read
+  handed its chunks to rayon and the calling thread slept until the pool had
+  decoded them, so with a small pool (2-4 threads) readers outside it queued
+  behind its workers. The calling thread now decodes too, and pool workers
+  join in only when free; a helper the pool starts after the read has
+  finished returns at once. A single read still spreads over the default
+  pool. This replaces the one-thread-pool special case below for full
+  reads. Chunks are placed from several threads only when the chunk index
+  puts them on the chunk grid at distinct places; a corrupt index is read
+  one chunk at a time, and the error reported is still the first failing
+  chunk's. New test `crates/clawhdf5/tests/busy_decode_pool.rs`.
+- **Fixed:** in a filtered dataset, a chunk stored with every filter skipped
+  (filter mask) and shorter than a chunk read with zeros in place of its
+  missing part through `File`'s `read_*`; it is now an error naming the
+  chunk, as `MmapFile`/`LazyFile` already made it.
+- New h5py comparison `crates/clawhdf5/tests/chunked_read_paths_interop.rs`:
+  every chunked read path (cached and uncached full reads, `MmapFile`,
+  `LazyFile`, small, strided and point selections, with and without the
+  `parallel` feature) for 1-8-byte integers and 2-8-byte floats in both
+  byte orders, through deflate, shuffle, Fletcher32, LZF, SZIP and Blosc,
+  with partial edge chunks, sparse datasets and fill values, and datasets
+  larger than the chunk cache.
+
+### Writer: large dense indexes (2026-09-26)
+- **`track_order` orders attributes too, as h5py's `track_order=True`
+  does.** It tracked link creation order only, so h5py listed a tracked
+  object's attributes by name. A tracking object's header now has the
+  attribute creation order tracked and indexed flags, an Attribute Info
+  message with the next creation order (also for inline attributes), and a
+  creation order on each inline attribute message; dense attribute storage
+  gets a creation-order index (B-tree type 9). `FileWriter::track_order` /
+  `FileBuilder::track_order` now apply to datasets' attributes as well, and
+  `DatasetBuilder::track_order` sets it per dataset. Groups and datasets
+  that track order are written differently from before; others are
+  unchanged. More than 65 535 attributes on a tracking object is an error
+  (libhdf5's creation order counter is 2 bytes). The reader
+  (`attribute::extract_attributes*`) lists a tracking object's attributes
+  in creation order. Test `track_order_lists_attributes_in_creation_order`
+  (h5py lists, reads and extends them in "r+" mode, including libhdf5's
+  move from inline to dense storage).
+- **No more 65 535-record limit on the writer's v2 B-trees.** Dense link
+  storage (name index and creation-order index), dense attribute storage
+  and the chunk index of datasets with more than one unlimited dimension
+  were written as a single leaf, so a group with more than 65 535 links, an
+  object with more than 65 535 dense attributes, or such a dataset with more
+  than 65 535 chunks was an error. The writer now builds internal nodes to
+  any depth (`clawhdf5_format::btree_v2_write`), with node capacities and
+  child-pointer widths from the same arithmetic as libhdf5's
+  `H5B2__hdr_init` (shared with the reader, `btree_v2::node_info`) and
+  libhdf5's node sizes (512 bytes for dense storage, 2048 for chunks).
+  Indexes that fit the old one-leaf layout are written byte for byte as
+  before. New tests `crates/clawhdf5/tests/deep_btree_interop.rs` (100 000
+  links, 70 000 attributes, 200 000 chunks; h5py, h5dump, clawhdf5, and
+  h5py "r+" edits) and `check_files_with_deep_btrees` (`h5rs check`).
+- **Links and attributes whose name hashes collide are found by name.** The
+  dense name indexes (a group's links: B-tree v2 type 5; an object's
+  attributes: type 8) are ordered by the name's lookup3 hash and, when two
+  hashes are equal, by the name itself, as libhdf5 compares them. The writer
+  broke ties by insertion order, so libhdf5 could not open one of two
+  colliding names (`"k69209"` and `"k155448"` share hash `0x3a0b13e6`;
+  collisions are likely from about 77 000 names). Regression test
+  `names_whose_hashes_collide_are_found_by_name` in
+  `crates/clawhdf5/tests/writer_groups_interop.rs`.
+
+### Blosc2 (2026-09-26)
+- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with
+  hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature
+  `blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`)
+  decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the
+  B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered
+  back into C order), Blosc2 chunks with their special values (zeros, NaN,
+  uninitialised, one repeated value), and the shuffle, bit-shuffle, delta
+  and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and
+  Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2
+  encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined
+  codecs and registered Blosc2 filters (e.g. bytedelta) are errors;
+  uninitialised chunks read as zeros. Tested against h5py 3.16 +
+  hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with
+  partial edge chunks; every integer width and f4/f8; datasets of zeros,
+  one value and NaN; Fletcher32 before Blosc2) and against frames from
+  python-blosc2 4.13.1 for what hdf5plugin never writes
+  (`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is
+  fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
+- **A crafted Blosc2 chunk cannot allocate more than a few times its HDF5
+  chunk size.** Found in review before release: the sizes a frame declares
+  sized the decoder's buffers. A 173-byte frame whose offsets chunk claimed
+  2 GiB was decoded in full for a 1 MiB chunk; an empty chunk allocated its
+  declared block size twice (about 1 GiB); a B2ND chunk was decoded whole
+  with its padding (up to 16x the chunk); and a Zstandard stream's declared
+  window (up to 100 MiB) was reserved as the decoder was reused, which also
+  affected Blosc 1 and bitshuffle with Zstandard. Now the offsets chunk is
+  capped at the chunk size, block sizes are clamped to the chunk, B2ND
+  blocks are placed as they are decoded (padding is never held), and the
+  Zstandard window is capped at twice the stream's output (at least
+  128 KiB). A B2ND chunk may no longer be larger than its array, which
+  hdf5-blosc2 never writes. `tests/blosc2_alloc_bounds.rs` measures peak
+  allocation for these frames and for 45,000 fuzzed ones: at most 6x the
+  chunk size, twice the input and 2 MiB of Zstandard state.
+
+### Remaining conformance errors (2026-09-26)
+Conformance on tank, `conformance/run.sh --no-fetch`: 598 of 697 files
+ok (575 before). Of the 5 our-errors left, 3 are corrupt data HDF5 2.0
+reads only through a bug (listed in `CONFORMANCE.md`), 2 are the Blosc2 and
+ZFP filters; the 2 mismatches are the known h5py big-endian VL bug. The
+five files whose cache image libhdf5 cannot load (`cve-2025-6269-*`,
+`cve-2025-6516`) count as ok because the library, like libhdf5, opens them
+and fails their objects (see below).
+- **Metadata cache images are read.** A file written with a metadata cache
+  image keeps its metadata cache entries in an image block the superblock
+  extension points at, and libhdf5 reads them in place of the file's own
+  bytes; in `h5clear_mdc_image.h5` the root group exists only there, and
+  every reader failed with `InvalidObjectHeaderVersion(0)`. `File`,
+  `MmapFile` and `LazyFile` (and `h5rs`) now apply the image at open
+  (`clawhdf5_format::superblock_ext::CacheImage`), with libhdf5's checks.
+  The file is not copied to do it: a mapped file gets the image's entries
+  written into a private copy-on-write mapping
+  (`clawhdf5_io::HDF5Read::private_copy`, `MAP_PRIVATE`), so only the pages
+  they land on are copied, and a buffer the opener owns (`File::from_bytes`,
+  `open_buffered`) is patched in place; files without an image are read
+  from the mapping exactly as before. (An interim version copied the whole
+  file onto the heap: 2 GB of memory to open a 1 GiB sparse file with an
+  image, and an abort for an 8 GiB one; `tests/cache_image_memory.rs`
+  guards it.) An image entry that runs past the end of file is refused
+  (libhdf5 checks only its start; the images it writes never do this). A
+  file whose image libhdf5 cannot load (`cve-2025-6269-*`, `cve-2025-6516`)
+  opens, as in libhdf5, and every object lookup fails with the image's
+  error (`File`, `MmapFile`; `LazyFile` reads the root group at open, so
+  its open fails). libhdf5 fails only its first metadata read and then
+  reads the file's own, possibly stale, bytes; those are never read here.
+  An interim version refused such a file at `File::open` while the
+  conformance probe reported it as libhdf5 does, so the gate counted five
+  files as agreeing with h5py that the library did not open; probe and
+  library now take the decision from the same
+  `superblock_ext::cache_image_state`.
+- **Every other opener applies the superblock extension and the cache
+  image too** (`superblock_ext::apply_cache_image_in_place`, writing into
+  the buffer each already owns): `clawhdf5_io`'s `NativeVol` (at `open`,
+  and on read for `from_bytes`), `AsyncHDF5File`, `MpiVol` (a minimal edit
+  through the same `vol::load_hdf5`; the `mpi-io` feature cannot be built
+  without an MPI installation, so it was not compiled), and the external
+  source files of a virtual dataset. They read a file with an image from
+  its own bytes — stale metadata, or none (`h5clear_mdc_image.h5` failed
+  with `InvalidObjectHeaderVersion(0)`) — and skipped the extension checks
+  `File::open` makes. These readers cannot open a file and fail each
+  object, so an image libhdf5 cannot load is refused with the image's
+  error.
+- **The superblock extension is decoded at open, as libhdf5 does:** a File
+  Space Info or Metadata Cache Image message libhdf5 cannot decode makes the
+  open fail (`cve-2020-10810`, `cve-2020-10812` were opened).
+  `FormatError::InvalidSuperblockExtension`, `InvalidCacheImage`.
+- **Dataset storage libhdf5 refuses at open is refused at open**
+  (`FormatError::InvalidDatasetStorage`, `data_read::check_dataset_storage`):
+  an element count times element size that overflows (`cve-2024-32624`
+  `/Dset_OBJREF` opened and reported its shape), contiguous storage past the
+  end of the file, compact data of the wrong size. An empty contiguous
+  dataset at a defined address, which clawhdf5 up to v2.7.0 wrote, still
+  opens.
+- **Wrong or missing data fixed:**
+  - a simple dataspace of rank 0 holds one element (it held 0;
+    `cve-2020-18494`), and contiguous storage larger than the dataset reads
+    (`cve-2024-32623`, `cve-2025-2309`; libhdf5 ignores the excess);
+  - scale-offset returned wrong values for ordinary h5py files — see
+    *Correctness* below; E-scale is refused, as in libhdf5; codes past the
+    end of the chunk stay an error (`cve-2025-2308`, where HDF5 2.0 reads
+    past its buffer);
+  - shuffle uses its own parameter as the element size, as libhdf5 does
+    (`cve-2025-44905`);
+  - an unfiltered chunk the index records at other than the chunk's size is
+    refused (it read with zeros for the missing bytes; `cve-2025-44904`);
+  - a v1 B-tree chunk index is read as libhdf5 reads it: each chunk is
+    looked up the way `H5B_find` / `H5D__btree_cmp3` / `H5D__btree_found`
+    look it up, and a chunk that lookup does not find reads as fill values.
+    A key with a non-zero element-size coordinate is found in a 1-D dataset
+    and not in one of rank 2 or more (`cve-2025-44905`
+    `/Shuffle_float_data_le`, which read the chunk's data where h5py reads
+    fill values); an interim fix refused every such key, including 1-D
+    files libhdf5 reads correctly.
+- **Refused as libhdf5 refuses them:** a v1 group with an empty link name
+  fails its listing (`FormatError::InvalidLinkName`; lookups still work,
+  `cve-2021-46244`); dataspaces with more than 32 dimensions, a rank on a
+  scalar or null dataspace, or a dimension over its maximum
+  (`FormatError::InvalidDataspace`).
+- `ObjectHeader::object_class` classifies a header as libhdf5 does (a
+  dataset needs a datatype *and* a dataspace).
+- Conformance harness: user-defined links were listed as objects by the
+  reference, unopenable objects were not deduplicated, nested array types
+  were hashed wrong (`tarray3.h5`), and the attributes of objects h5py
+  cannot open were compared; all fixed. `CONFORMANCE.md` lists the corrupt
+  objects HDF5 2.0 reads through a bug (`bad_nbit_parms_walk.h5` among
+  them: libhdf5's own test now requires that read to fail).
+
 ### Concurrent reads (2026-09-26)
 - **Full reads of chunked datasets scale with threads again when rayon's
   pool has one thread.** Each full read handed its chunks to rayon to
@@ -372,6 +586,7 @@
   missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
   clawhdf5)").
 - **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
+  (Blosc2 reads since the `blosc2` feature, see above.)
 - **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
   filter). HDF5 stores every chunk at the full chunk size, so a filter
   pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
@@ -787,6 +1002,28 @@
 - CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
 
 ### Correctness
+- **Scale-offset data read wrong values in every release that decoded it
+  (v2.2.0 to v2.7.0), silently, on ordinary h5py files** (fixed
+  2026-09-26). Of 1480 scale-offset datasets h5py writes across every
+  integer type (`i1` .. `u8`), `f4` and `f8`, both byte orders, with and
+  without a fill value, and `scaleoffset` from 0 to the full width, 332 did
+  not read as h5py reads them: **151 returned wrong values with no error**
+  and 181 failed to read. The common cause was a chunk libhdf5 stores at
+  full width (`minbits` equal to the type's width), which it does for any
+  full-width `scaleoffset` and on its own whenever a chunk's values span
+  most of the type's range: `scaleoffset=0` integer data with a wide range
+  (82 datasets, all wrong values), full-width `u4`/`i4`/`u8`/`i8` (51 wrong
+  values; the narrower types and the rest failed with "truncated minval" or
+  "implausible minbits"), and `f4` D-scale data with a large range (18,
+  wrong values). Such a chunk holds the elements as they are; they were
+  decoded as offsets from `minval`. Also fixed, found on crafted files: the
+  packed codes start at byte 21 whatever size the chunk records for
+  `minval` (`cve-2025-44905` `/Scale_offset_short_data_be`), and a chunk
+  with `minbits` 0 and a fill value is all fill values (it read as
+  `minval`). The whole matrix is now an interop test
+  (`crates/clawhdf5/tests/scaleoffset_interop.rs`, generated by h5py at
+  test time, every dataset compared); on v2.7.0's decoder it reports the
+  332. See `docs/known-issues.md`.
 - **Corrupt files libhdf5 refuses are now refused instead of read.** On the
   HDF Group's CVE reproducers, 18 objects that libhdf5 (HDF5 2.0, through
   h5py) refuses to open were read by clawhdf5, some as wrong data (a chunk

CLAUDE.md

@@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
 |-------|------|
 | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
 | `clawhdf5-io` | Read/write implementation |
-| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1) live in `clawhdf5-format`. No Blosc2 or ZFP. |
+| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
 | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
 | `clawhdf5` | Main facade crate |
 | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |

CONFORMANCE.md

@@ -13,15 +13,15 @@ fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
 
 | | |
 |---|---|
-| date | 2026-09-26 14:18 UTC |
+| date | 2026-09-26 17:10 UTC |
-| clawhdf5 commit | `73a01f1256fb9bf1b1e7601f755af9e8273cec4e` |
+| clawhdf5 commit | `d0e3beb3aa8290aae523ce280b4380e75484b6bc` |
 | machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
 | command | `conformance/run.sh --no-fetch --update-baseline` |
 | rustc | rustc 1.98.1 (48a229cea 2026-09-01) |
 | reference | h5py 3.16.0, HDF5 2.0.0, numpy 2.5.3, hdf5plugin 7.1.0, Python 3.14.4 |
 | h5dump | Version 1.14.6 (CVE corpus only) |
 | limits | 20 s timeout (SIGKILL), 4096 MiB address space, per process; 16 files in parallel |
-| runtime | 23 s probing + comparing (0 s fetch/build before it) |
+| runtime | 24 s probing + comparing (0 s fetch/build before it) |
 
 ## Results
 
@@ -36,16 +36,18 @@ A file's class is the first that applies:
 | corpus | files | ok | our-error | mismatch | h5py-cannot-read | panic | hang | crash | oom |
 |---|---|---|---|---|---|---|---|---|---|
 | NCAS-CMS_pyfive | 33 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
-| cve_hdf5 | 147 | 100 | 6 | 9 | 32 | 0 | 0 | 0 | 0 |
+| cve_hdf5 | 147 | 113 | 2 | 0 | 32 | 0 | 0 | 0 | 0 |
 | h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| hdf5 | 466 | 392 | 4 | 10 | 60 | 0 | 0 | 0 | 0 |
+| hdf5 | 466 | 403 | 2 | 1 | 60 | 0 | 0 | 0 | 0 |
 | netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
 | netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
 | usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
 | xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
-| **all** | **697** | **575** | **10** | **20** | **92** | **0** | **0** | **0** | **0** |
+| **all** | **697** | **599** | **4** | **2** | **92** | **0** | **0** | **0** | **0** |
 
-2 of the 20 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
+2 of the 2 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
+
+3 of the 4 our-errors are corrupt data that HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).
 
 Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):
 
@@ -70,26 +72,14 @@ Grouped by normalised error message. *files* counts files whose class this cause
 
 | files | objects | error | examples |
 |---:|---:|---|---|
-| 3 | 3 | `DataSizeMismatch { expected: N, actual: N }` | `cve_hdf5/cvefiles/cve-2020-18494.h5`, `cve_hdf5/cvefiles/cve-2024-32623.h5`, `cve_hdf5/cvefiles/cve-2025-2309.h5` |
+| 3 | 3 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `cve_hdf5/cvefiles/cve-2025-44904.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
-| 2 | 2 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
+| 1 | 1 | `UnsupportedFilter(N)` | `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zfp.h5` |
-| 2 | 2 | `UnsupportedFilter(N)` | `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc2.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zfp.h5` |
-| 1 | 1 | `UnexpectedEof { expected: N, available: N }` | `cve_hdf5/cvefiles/cve-2019-9151.h5` |
-| 1 | 1 | `MissingMessage(Dataspace)` | `cve_hdf5/cvefiles/cve-2024-33874.h5` |
-| 1 | 1 | `InvalidObjectHeaderVersion(N)` | `hdf5/tools/test/testfiles/h5clear_mdc_image.h5` |
 
 ## Mismatch root causes
 
 | files | objects | cause | examples |
 |---:|---:|---|---|
-| 13 | 14 | `missing-object` | `cve_hdf5/cvefiles/cve-2019-8397.h5`, `cve_hdf5/cvefiles/cve-2019-8398.h5`, `cve_hdf5/cvefiles/cve-2021-46243.h5` (+10 more) |
-| 2 | 6 | `extra-attr` | `cve_hdf5/cvefiles/cve-2018-17438`, `cve_hdf5/cvefiles/cve-2018-17439` |
 | 1 | 1 | `attr-values: ours=vlen(>u8) h5py=object layout=- filters=-` | `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` |
-| 1 | 4 | `extra-object` | `cve_hdf5/cvefiles/cve-2021-46244.h5` |
-| 1 | 1 | `values: ours=<f4 h5py=float32 layout=chunked filters=-` | `cve_hdf5/cvefiles/cve-2025-44904.h5` |
-| 1 | 1 | `values: ours=>i2 h5py=>i2 layout=chunked filters=[6]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
-| 1 | 1 | `values: ours=>f4 h5py=>f4 layout=chunked filters=[2]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
-| 1 | 1 | `values: ours=<f4 h5py=float32 layout=chunked filters=[2]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
-| 1 | 1 | `values: ours=((<i4)[6, 3])[4] h5py=(('<i4', (6, 3)), (4,)) layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tarray3.h5` |
 | 1 | 1 | `values: ours=vlen({r:>f4,i:>f4}8) h5py=object layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tcomplex_be.h5` |
 
 ## CVE corpus: clawhdf5 vs h5dump vs h5py
@@ -102,7 +92,7 @@ columns are.
 
 | tool | read | error | panic | crash | hang | oom |
 |---|---:|---:|---:|---:|---:|---:|
-| clawhdf5 | 140 | 7 | 0 | 0 | 0 | 0 |
+| clawhdf5 | 121 | 26 | 0 | 0 | 0 | 0 |
 | h5dump 1.14.6 | 16 | 129 | 0 | 2 | 0 | 0 |
 | h5py 3.16.0 / HDF5 2.0.0 | 115 | 31 | 0 | 1 | 0 | 0 |
 
@@ -156,27 +146,27 @@ columns are.
 | cvefiles/cve-2018-17435.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2018-17436 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2018-17437.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
-| cvefiles/cve-2018-17438 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | mismatch |
+| cvefiles/cve-2018-17438 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
-| cvefiles/cve-2018-17439 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | mismatch |
+| cvefiles/cve-2018-17439 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2019-8396.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
-| cvefiles/cve-2019-8397.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
+| cvefiles/cve-2019-8397.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/cve-2019-8398.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
+| cvefiles/cve-2019-8398.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/cve-2019-9151.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2019-9151.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | ok |
 | cvefiles/cve-2019-9152.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2020-10809 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2020-10810.h5 | error exit | open error | read 2 obj | h5py-cannot-read |
+| cvefiles/cve-2020-10810.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2020-10811.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
-| cvefiles/cve-2020-10812.h5 | error exit | open error | read 2 obj | h5py-cannot-read |
+| cvefiles/cve-2020-10812.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2020-18232.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
-| cvefiles/cve-2020-18494.h5 | ok | read 2 obj | read 2 obj, 1 errors | our-error |
+| cvefiles/cve-2020-18494.h5 | ok | read 2 obj | read 2 obj | ok |
 | cvefiles/cve-2021-36977.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2021-37501.h5 | error exit | read 18 obj, 1 errors | read 18 obj, 1 errors | ok |
 | cvefiles/cve-2021-45829.h5 | error exit | read 1 obj, 2 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2021-45830.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2021-45830.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2021-45833.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2021-46242.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
-| cvefiles/cve-2021-46243.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
+| cvefiles/cve-2021-46243.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
-| cvefiles/cve-2021-46244.h5 | error exit | read 2 obj, 1 errors | read 6 obj, 4 errors | mismatch |
+| cvefiles/cve-2021-46244.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2024-29157.h5 | error exit | read 4 obj, 7 errors | read 4 obj, 7 errors | ok |
 | cvefiles/cve-2024-29158.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2024-29159.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
@@ -201,50 +191,50 @@ columns are.
 | cvefiles/cve-2024-32615.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
 | cvefiles/cve-2024-32616.h5 | error exit | read 10 obj, 7 errors | read 10 obj, 6 errors | ok |
 | cvefiles/cve-2024-32617.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2024-32618.h5 | error exit | read 4 obj, 2 errors | read 3 obj, 1 errors | mismatch |
+| cvefiles/cve-2024-32618.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2024-32619.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
 | cvefiles/cve-2024-32620.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2024-32621.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2024-32622.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
-| cvefiles/cve-2024-32623.h5 | ok | read 6 obj | read 6 obj, 1 errors | our-error |
+| cvefiles/cve-2024-32623.h5 | ok | read 6 obj | read 6 obj | ok |
-| cvefiles/cve-2024-32624.h5 | error exit | read 6 obj, 1 errors | read 6 obj | ok |
+| cvefiles/cve-2024-32624.h5 | error exit | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
 | cvefiles/cve-2024-33873.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
-| cvefiles/cve-2024-33874.h5 | ok | read 6 obj, 1 errors | read 6 obj, 2 errors | our-error |
+| cvefiles/cve-2024-33874.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
 | cvefiles/cve-2024-33875.h5 | ok | read 2 obj | read 2 obj | ok |
 | cvefiles/cve-2024-33876.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
 | cvefiles/cve-2024-33877.h5 | error exit | read 8 obj, 1 errors | read 8 obj, 1 errors | ok |
 | cvefiles/cve-2025-2153.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
 | cvefiles/cve-2025-2308.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
-| cvefiles/cve-2025-2309.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | our-error |
+| cvefiles/cve-2025-2309.h5 | ok | read 6 obj, 1 errors | read 6 obj | ok |
 | cvefiles/cve-2025-2310.h5 | error exit | read 24 obj, 8 errors | read 24 obj, 8 errors | ok |
-| cvefiles/cve-2025-2912.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2912.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-2913.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2913.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-2914.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2914.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-2915.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2915.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-2923.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2923.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2025-2924.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-2925.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2025-2926.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2926.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | mismatch |
+| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
-| cvefiles/cve-2025-44905.h5 | error exit | read 25 obj, 3 errors | read 25 obj, 3 errors | mismatch |
+| cvefiles/cve-2025-44905.h5 | error exit | read 25 obj, 3 errors | read 25 obj, 3 errors | ok |
 | cvefiles/cve-2025-6269-1.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-6269-2.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-6269-3.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-6269-4.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2025-6270-1.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6270-1.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6270-2.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6270-2.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6270-3.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6270-3.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2025-6516.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2025-6750.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
-| cvefiles/cve-2025-6816.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6816.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6817.h5 | error exit | open error | read 1 obj | h5py-cannot-read |
+| cvefiles/cve-2025-6817.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6818.h5 | error exit | open error | read 1 obj | h5py-cannot-read |
+| cvefiles/cve-2025-6818.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-6856.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6856.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2025-6857.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2025-6858.h5 | SIGSEGV | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6858.h5 | SIGSEGV | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2025-7067.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
-| cvefiles/cve-2025-7068.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-7068.h5 | error exit | open error | open error | h5py-cannot-read |
-| cvefiles/cve-2025-7069.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-7069.h5 | error exit | open error | open error | h5py-cannot-read |
 | cvefiles/cve-2026-26200.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
 | cvefiles/cve-2026-34734.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
 | cvefiles/cve-2026-92627.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
@@ -275,6 +265,11 @@ columns are.
 - **Types h5py widens.** Where h5py reads a type into a numpy type of a different size
   (FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not
   compared (shape and presence still are): dataset file type size 1 -> numpy float16 (2) (15x), attr file type size 1 -> numpy float16 (2) (15x), dataset file type size 2 -> numpy float32 (4) (2x), dataset file type size 8 -> numpy float128 (16) (1x), dataset file type size 12 -> numpy float128 (16) (1x), attr file type size 2 -> numpy float32 (4) (1x), dataset file type size 2 -> numpy >f4 (4) (1x), attr file type size 2 -> numpy >f4 (4) (1x).
+- **Corrupt data HDF5 2.0 reads through a bug.** clawhdf5 refuses these objects; h5py 3.16 /
+  HDF5 2.0 returns values for them that the file does not hold:
+  - `cve_hdf5/cvefiles/cve-2025-2308.h5` `/Scale_offset_long_long_data_le`: scale-offset codes run past the end of the chunk: HDF5 2.0 reads past its buffer; libhdf5's develop branch refuses the chunk ("Buffer too short").
+  - `cve_hdf5/cvefiles/cve-2025-44904.h5` `/Scale_offset_float_data_le`: unfiltered chunks of 38 and 37 bytes for 48-byte chunks: HDF5 2.0 fills the rest with whatever its buffer held; libhdf5's develop branch refuses them ("incorrect chunk size returned from index for unfiltered chunk").
+  - `hdf5/test/testfiles/bad_nbit_parms_walk.h5` `/Nbit_int_data_le`: an N-Bit parameter list one value short: HDF5 2.0 reads past the list; libhdf5's own test (`test_filter_bad_params`, test/dsets.c) now requires the read to fail.
 - **References** are compared by presence only (`R`), not by target.
 
 ## Objects h5py fails on but clawhdf5 reads
@@ -282,7 +277,6 @@ columns are.
 - 19 x `OSError: Can't synchronously read data (no appropriate function for conversion path)`
 - 1 x `TypeError: unhandled dtype kind M (dtype('…'))`
 - 1 x `TypeError: No NumPy equivalent for TypeTimeID exists`
-- 1 x `KeyError: "…"`
 - 1 x `ValueError: Insufficient precision in available types to represent (N, N, N, N, N)`
 
 ## Reproduce

README.md

@@ -668,7 +668,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
 ├── Core HDF5
 │   ├── clawhdf5-format      — Binary parser/writer (no_std-capable), shared type definitions
 │   ├── clawhdf5-io          — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
-│   ├── clawhdf5-filters     — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format
+│   ├── clawhdf5-filters     — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc/Blosc2 filters live in clawhdf5-format
 │   ├── clawhdf5-derive      — Proc macros
 │   ├── clawhdf5             — High-level API
 │   ├── clawhdf5-netcdf4     — NetCDF-4 support
@@ -779,13 +779,15 @@ stores keep their setting. Opt out with `float16 = false` or
 | `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) |
 | `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
 | `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
-| `plugin-filters` | no | All four above |
+| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
+| `plugin-filters` | no | All five above |
 
-Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with
+ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`,
-`UnsupportedFilter`, whose message names the filter. Any other filter can be
+whose message names the filter. clawhdf5 cannot write Blosc2. Any other
-supplied at run time with `filter_registry::register_filter` (a decoder
+filter can be supplied at run time with `filter_registry::register_filter` (a
-closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`)
+decoder closure, or a `FilterCodec` that also encodes). The facade
-forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write
+(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and
+`plugin-filters`. Write
 with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
 and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
 `crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard

conformance/baseline.json

@@ -1,15 +1,15 @@
 {
  "comment": "conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
- "commit": "73a01f1256fb9bf1b1e7601f755af9e8273cec4e",
+ "commit": "d0e3beb3aa8290aae523ce280b4380e75484b6bc",
- "date": "2026-09-26 14:18 UTC",
+ "date": "2026-09-26 17:10 UTC",
  "reference": "h5py 3.16.0 / HDF5 2.0.0",
  "files": 697,
- "ok": 575,
+ "ok": 599,
  "counts": {
   "h5py-cannot-read": 92,
-  "mismatch": 20,
+  "mismatch": 2,
-  "ok": 575,
+  "ok": 599,
-  "our-error": 10
+  "our-error": 4
  },
  "per_corpus": {
   "NCAS-CMS_pyfive": {
@@ -18,18 +18,17 @@
   },
   "cve_hdf5": {
    "h5py-cannot-read": 32,
-   "mismatch": 9,
+   "ok": 113,
-   "ok": 100,
+   "our-error": 2
-   "our-error": 6
   },
   "h5py_data": {
    "ok": 4
   },
   "hdf5": {
    "h5py-cannot-read": 60,
-   "mismatch": 10,
+   "mismatch": 1,
-   "ok": 392,
+   "ok": 403,
-   "our-error": 4
+   "our-error": 2
   },
   "netcdf-c": {
    "ok": 20
@@ -117,14 +116,22 @@
   "cve_hdf5/cvefiles/cve-2018-17434.h5",
   "cve_hdf5/cvefiles/cve-2018-17435.h5",
   "cve_hdf5/cvefiles/cve-2018-17437.h5",
+  "cve_hdf5/cvefiles/cve-2018-17438",
+  "cve_hdf5/cvefiles/cve-2018-17439",
   "cve_hdf5/cvefiles/cve-2019-8396.h5",
+  "cve_hdf5/cvefiles/cve-2019-8397.h5",
+  "cve_hdf5/cvefiles/cve-2019-8398.h5",
+  "cve_hdf5/cvefiles/cve-2019-9151.h5",
   "cve_hdf5/cvefiles/cve-2019-9152.h5",
   "cve_hdf5/cvefiles/cve-2020-10811.h5",
   "cve_hdf5/cvefiles/cve-2020-18232.h5",
+  "cve_hdf5/cvefiles/cve-2020-18494.h5",
   "cve_hdf5/cvefiles/cve-2021-36977.h5",
   "cve_hdf5/cvefiles/cve-2021-37501.h5",
   "cve_hdf5/cvefiles/cve-2021-45829.h5",
   "cve_hdf5/cvefiles/cve-2021-45833.h5",
+  "cve_hdf5/cvefiles/cve-2021-46243.h5",
+  "cve_hdf5/cvefiles/cve-2021-46244.h5",
   "cve_hdf5/cvefiles/cve-2024-29157.h5",
   "cve_hdf5/cvefiles/cve-2024-29158.h5",
   "cve_hdf5/cvefiles/cve-2024-29159.h5",
@@ -148,18 +155,23 @@
   "cve_hdf5/cvefiles/cve-2024-32615.h5",
   "cve_hdf5/cvefiles/cve-2024-32616.h5",
   "cve_hdf5/cvefiles/cve-2024-32617.h5",
+  "cve_hdf5/cvefiles/cve-2024-32618.h5",
   "cve_hdf5/cvefiles/cve-2024-32619.h5",
   "cve_hdf5/cvefiles/cve-2024-32620.h5",
   "cve_hdf5/cvefiles/cve-2024-32621.h5",
   "cve_hdf5/cvefiles/cve-2024-32622.h5",
+  "cve_hdf5/cvefiles/cve-2024-32623.h5",
   "cve_hdf5/cvefiles/cve-2024-32624.h5",
   "cve_hdf5/cvefiles/cve-2024-33873.h5",
+  "cve_hdf5/cvefiles/cve-2024-33874.h5",
   "cve_hdf5/cvefiles/cve-2024-33875.h5",
   "cve_hdf5/cvefiles/cve-2024-33876.h5",
   "cve_hdf5/cvefiles/cve-2024-33877.h5",
+  "cve_hdf5/cvefiles/cve-2025-2309.h5",
   "cve_hdf5/cvefiles/cve-2025-2310.h5",
   "cve_hdf5/cvefiles/cve-2025-2924.h5",
   "cve_hdf5/cvefiles/cve-2025-2925.h5",
+  "cve_hdf5/cvefiles/cve-2025-44905.h5",
   "cve_hdf5/cvefiles/cve-2025-6269-1.h5",
   "cve_hdf5/cvefiles/cve-2025-6269-2.h5",
   "cve_hdf5/cvefiles/cve-2025-6269-3.h5",
@@ -183,6 +195,7 @@
   "h5py_data/vlen_string_s390x.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bitgroom.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc2.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bshuf.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bzip2.h5",
   "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_granularbr.h5",
@@ -264,6 +277,7 @@
   "hdf5/test/testfiles/tmtimeo.h5",
   "hdf5/test/testfiles/tnullspace.h5",
   "hdf5/test/testfiles/tsizeslheap.h5",
+  "hdf5/tools/test/testfiles/bigendian/tall.h5",
   "hdf5/tools/test/testfiles/bigendian/tdset2.h5",
   "hdf5/tools/test/testfiles/binfp64.h5",
   "hdf5/tools/test/testfiles/binin16.h5",
@@ -284,6 +298,7 @@
   "hdf5/tools/test/testfiles/h5clear_fsm_persist_noclose.h5",
   "hdf5/tools/test/testfiles/h5clear_fsm_persist_user_equal.h5",
   "hdf5/tools/test/testfiles/h5clear_fsm_persist_user_less.h5",
+  "hdf5/tools/test/testfiles/h5clear_mdc_image.h5",
   "hdf5/tools/test/testfiles/h5clear_sec2_v0.h5",
   "hdf5/tools/test/testfiles/h5clear_sec2_v2.h5",
   "hdf5/tools/test/testfiles/h5copy_extlinks_src.h5",
@@ -337,6 +352,7 @@
   "hdf5/tools/test/testfiles/h5diff_softlinks.h5",
   "hdf5/tools/test/testfiles/h5diff_strings1.h5",
   "hdf5/tools/test/testfiles/h5diff_strings2.h5",
+  "hdf5/tools/test/testfiles/h5diff_types.h5",
   "hdf5/tools/test/testfiles/h5fc_edge_v3.h5",
   "hdf5/tools/test/testfiles/h5fc_err_level.h5",
   "hdf5/tools/test/testfiles/h5fc_ext1_f.h5",
@@ -414,9 +430,11 @@
   "hdf5/tools/test/testfiles/tCVE_2018_11206_fill_new.h5",
   "hdf5/tools/test/testfiles/tCVE_2018_11206_fill_old.h5",
   "hdf5/tools/test/testfiles/taindices.h5",
+  "hdf5/tools/test/testfiles/tall.h5",
   "hdf5/tools/test/testfiles/tarray1.h5",
   "hdf5/tools/test/testfiles/tarray1_big.h5",
   "hdf5/tools/test/testfiles/tarray2.h5",
+  "hdf5/tools/test/testfiles/tarray3.h5",
   "hdf5/tools/test/testfiles/tarray4.h5",
   "hdf5/tools/test/testfiles/tarray5.h5",
   "hdf5/tools/test/testfiles/tarray8.h5",
@@ -449,6 +467,7 @@
   "hdf5/tools/test/testfiles/textlinksrc.h5",
   "hdf5/tools/test/testfiles/textlinktar.h5",
   "hdf5/tools/test/testfiles/textpfe.h5",
+  "hdf5/tools/test/testfiles/tfcontents1.h5",
   "hdf5/tools/test/testfiles/tfcontents2.h5",
   "hdf5/tools/test/testfiles/tfilters.h5",
   "hdf5/tools/test/testfiles/tfloat16.h5",
@@ -505,6 +524,7 @@
   "hdf5/tools/test/testfiles/tstr3.h5",
   "hdf5/tools/test/testfiles/tudfilter.h5",
   "hdf5/tools/test/testfiles/tudfilter2.h5",
+  "hdf5/tools/test/testfiles/tudlink.h5",
   "hdf5/tools/test/testfiles/tvldtypes1.h5",
   "hdf5/tools/test/testfiles/tvldtypes2.h5",
   "hdf5/tools/test/testfiles/tvldtypes3.h5",
@@ -513,6 +533,8 @@
   "hdf5/tools/test/testfiles/tvlenstr_array.h5",
   "hdf5/tools/test/testfiles/tvlstr.h5",
   "hdf5/tools/test/testfiles/tvms.h5",
+  "hdf5/tools/test/testfiles/twithub.h5",
+  "hdf5/tools/test/testfiles/twithub513.h5",
   "hdf5/tools/test/testfiles/txtfp32.h5",
   "hdf5/tools/test/testfiles/txtfp64.h5",
   "hdf5/tools/test/testfiles/txtin16.h5",
@@ -558,6 +580,7 @@
   "hdf5/tools/test/testfiles/xml/tenum.h5",
   "hdf5/tools/test/testfiles/xml/test35.nc",
   "hdf5/tools/test/testfiles/xml/tloop2.h5",
+  "hdf5/tools/test/testfiles/xml/tmany.h5",
   "hdf5/tools/test/testfiles/xml/tname-amp.h5",
   "hdf5/tools/test/testfiles/xml/tname-apos.h5",
   "hdf5/tools/test/testfiles/xml/tname-gt.h5",

conformance/compare.py

@@ -146,7 +146,13 @@ for rel in files:
             if a.get("kind") != b.get("kind") and "error" not in b and "error" not in a:
                 issues.append(("mismatch", f"{p}: kind {a.get('kind')} vs ours {b.get('kind')}", "kind", b))
                 ok = False
+            # h5py could not open the object at all: it read none of its
+            # attributes or links, so there is nothing to compare ours with
+            # (the object's own error is compared above and below).
+            ref_unopened = a.get("kind") == "unknown" and "error" in a
             for k in ("error", "list_error", "attrs_error"):
+                if ref_unopened and k != "error":
+                    continue
                 if k in b and k not in a:
                     issues.append(("our-error", f"{p}: {k}: {b[k]}", b[k], b))
                     ok = False
@@ -164,7 +170,7 @@ for rel in files:
                     issues.append(("mismatch", f"{p}: values differ (h5py {a.get('dtype')} vs ours {b.get('dtype')})", "values", b | {"ref_head": a.get("head"), "ref_dtype": a.get("dtype")}))
                     ok = False
             ra, oa = a.get("attrs") or {}, b.get("attrs") or {}
-            if "attrs_error" not in b and "attrs_error" not in a:
+            if "attrs_error" not in b and "attrs_error" not in a and not ref_unopened:
                 for an in sorted(set(ra) | set(oa)):
                     x, y = ra.get(an), oa.get(an)
                     if x is None:

conformance/probe/src/main.rs

@@ -31,7 +31,7 @@ use clawhdf5_format::filter_pipeline::FilterPipeline;
 use clawhdf5_format::group_v1::{self, GroupEntry};
 use clawhdf5_format::group_v2;
 use clawhdf5_format::message_type::MessageType;
-use clawhdf5_format::object_header::ObjectHeader;
+use clawhdf5_format::object_header::{ObjectClass, ObjectHeader};
 use clawhdf5_format::signature;
 use clawhdf5_format::superblock::Superblock;
 use clawhdf5_format::symbol_table::SymbolTableMessage;
@@ -308,18 +308,20 @@ impl<'a> Ctx<'a> {
             )
             .map_err(e)?;
         }
-        let (shape, n) = Self::shape(&ds);
-        rec.insert("shape".into(), shape);
-        if n.saturating_mul(dt.type_size() as u64) > MAX_BYTES {
-            rec.insert("skipped".into(), Value::String("too large".into()));
-            return Ok(());
-        }
         let lm = h
             .messages
             .iter()
             .find(|m| m.msg_type == MessageType::DataLayout)
             .ok_or("MissingMessage(DataLayout)")?;
         let dl = DataLayout::parse(&lm.data, self.os, self.ls).map_err(e)?;
+        // What libhdf5 checks when it opens the dataset (as File::dataset).
+        data_read::check_dataset_storage(&dl, &ds, &dt, self.data.len() as u64).map_err(e)?;
+        let (shape, n) = Self::shape(&ds);
+        rec.insert("shape".into(), shape);
+        if n.saturating_mul(dt.type_size() as u64) > MAX_BYTES {
+            rec.insert("skipped".into(), Value::String("too large".into()));
+            return Ok(());
+        }
         rec.insert(
             "layout".into(),
             Value::String(
@@ -657,6 +659,20 @@ fn is_group(h: &ObjectHeader) -> bool {
     })
 }
 
+/// The probe's kind for an object header: libhdf5's object class
+/// ([`ObjectHeader::object_class`]: group, then dataset — a datatype *and* a
+/// dataspace — then named datatype), which is what h5py opens the object as.
+/// The root group, and a header with only link messages, count as groups.
+fn kind_of(h: &ObjectHeader, is_root: bool) -> &'static str {
+    match h.object_class() {
+        Some(ObjectClass::Group) => "group",
+        Some(ObjectClass::Dataset) => "dataset",
+        _ if is_root || is_group(h) => "group",
+        Some(ObjectClass::NamedDatatype) => "datatype",
+        None => "unknown",
+    }
+}
+
 fn main() {
     install_hook();
     let path = std::env::args().nth(1).expect("usage: probe <file>");
@@ -696,6 +712,42 @@ fn main() {
             return;
         }
     };
+    // libhdf5 decodes the superblock extension at open (an error refuses
+    // the file), and loads a metadata cache image over the file's own
+    // metadata. It loads the image only when it first reads metadata — the
+    // root group — so a file whose image it cannot load still opens and
+    // that read fails. The library decides all three cases with the same
+    // `cache_image_state`: `File` and `MmapFile` open such a file and fail
+    // every object lookup with the image's error, which is what the probe
+    // records here (on the root group, where libhdf5 reports it).
+    use clawhdf5_format::superblock_ext::{self, CacheImageState};
+    let state = match guarded(|| superblock_ext::cache_image_state(hdf5, &sb).map_err(e)) {
+        Ok(x) => x,
+        Err(msg) => {
+            top.insert("open_error".into(), Value::String(msg));
+            println!("{}", Value::Object(top));
+            return;
+        }
+    };
+    let mut image_error = None;
+    let view = match state {
+        CacheImageState::Absent => None,
+        CacheImageState::Unloadable(err) => {
+            image_error = Some(e(err));
+            None
+        }
+        CacheImageState::Loaded(image) => {
+            let mut v = hdf5.to_vec();
+            match image.block(hdf5).and_then(|b| image.apply(b, &mut v)) {
+                Ok(()) => Some(v),
+                Err(err) => {
+                    image_error = Some(e(err));
+                    None
+                }
+            }
+        }
+    };
+    let hdf5: &[u8] = view.as_deref().unwrap_or(hdf5);
     top.insert("superblock_version".into(), json!(sb.version));
     let ctx = Ctx {
         data: hdf5,
@@ -723,6 +775,9 @@ fn main() {
         let mut rec = Map::new();
         rec.insert("path".into(), Value::String(p.clone()));
         let r = guarded(|| {
+            if let Some(msg) = &image_error {
+                return Err(msg.clone());
+            }
             let h = ctx.header(addr)?;
             Ok(h)
         });
@@ -735,23 +790,7 @@ fn main() {
                 continue;
             }
         };
-        let is_ds = h
+        let kind = kind_of(&h, addr == sb.root_group_address);
-            .messages
-            .iter()
-            .any(|m| m.msg_type == MessageType::DataLayout);
-        let kind = if is_ds {
-            "dataset"
-        } else if is_group(&h) || addr == sb.root_group_address {
-            "group"
-        } else if h
-            .messages
-            .iter()
-            .any(|m| m.msg_type == MessageType::Datatype)
-        {
-            "datatype"
-        } else {
-            "unknown"
-        };
         rec.insert("kind".into(), Value::String(kind.into()));
         if kind == "dataset"
             && let Err(msg) = guarded(|| ctx.read_dataset(&h, &mut rec))
@@ -867,6 +906,42 @@ mod tests {
         assert!(ieee_layout(&f32le));
     }
 
+    #[test]
+    fn kind_follows_libhdf5_object_class() {
+        use clawhdf5_format::object_header::HeaderMessage;
+        let header = |types: &[MessageType]| ObjectHeader {
+            version: 2,
+            messages: types
+                .iter()
+                .map(|&msg_type| HeaderMessage {
+                    msg_type,
+                    size: 0,
+                    flags: 0,
+                    creation_order: None,
+                    data: Vec::new(),
+                })
+                .collect(),
+            reference_count: None,
+            flags: 0,
+            access_time: None,
+            modification_time: None,
+            change_time: None,
+            birth_time: None,
+        };
+        use MessageType::*;
+        // cve-2024-33874 `/Dset1`: a datatype and a layout but no dataspace
+        // is a named datatype to libhdf5 (h5py opens it as one).
+        assert_eq!(kind_of(&header(&[Datatype, DataLayout]), false), "datatype");
+        assert_eq!(
+            kind_of(&header(&[Datatype, Dataspace, DataLayout]), false),
+            "dataset"
+        );
+        assert_eq!(kind_of(&header(&[SymbolTable]), false), "group");
+        assert_eq!(kind_of(&header(&[Link]), false), "group");
+        assert_eq!(kind_of(&header(&[]), true), "group");
+        assert_eq!(kind_of(&header(&[]), false), "unknown");
+    }
+
     #[test]
     fn partial_precision_int_is_shifted_and_sign_extended() {
         let dt = Datatype::FixedPoint {

conformance/ref.py

@@ -111,8 +111,11 @@ def note_conversion(tid, dt, rec):
 
 
 def hash_values(arr, dt, rec):
-    if dt.subdtype is not None:
+    # h5py expands an HDF5 array element type into trailing array dims, a
-        # h5py expands an HDF5 array element type into trailing array dims
+    # nested array type (an array of arrays) into all of them. Converting the
+    # expanded array back to the inner subarray type would broadcast every
+    # element into a whole subarray, so strip every level.
+    while dt.subdtype is not None:
         dt = dt.subdtype[0]
     arr = np.asarray(arr, dtype=dt)
     if simple(dt):
@@ -173,9 +176,13 @@ def main(path):
         return
     objects = []
     seen = set()
-    stack = [("/", None)]
+    # Objects h5py cannot open have no ObjectID to deduplicate by; they are
+    # deduplicated by the address their hard link points at instead, as the
+    # probe deduplicates every object by header address.
+    seen_unopenable = set()
+    stack = [("/", None, None)]
     while stack:
-        p, obj = stack.pop()
+        p, obj, link_addr = stack.pop()
         if len(objects) >= MAX_OBJECTS:
             top["truncated"] = True
             break
@@ -185,6 +192,10 @@ def main(path):
                 obj = f[p]
             key = hash(obj.id)  # h5py ObjectID hash = (fileno, object address/token)
         except Exception as e:  # noqa: BLE001
+            if link_addr is not None:
+                if link_addr in seen_unopenable:
+                    continue
+                seen_unopenable.add(link_addr)
             rec["kind"] = "unknown"
             rec["error"] = err(e)
             objects.append(rec)
@@ -232,15 +243,18 @@ def main(path):
                 base = "" if p == "/" else p
                 kids = []
                 for n in names:
+                    # The link's own type: `obj.get(n, getlink=True)` reports
+                    # a user-defined link (type 64-255) as a HardLink.
                     try:
-                        link = obj.get(n, getlink=True)
+                        info = obj.id.links.get_info(n.encode("utf-8", "surrogateescape"))
                     except Exception:  # noqa: BLE001
-                        link = None
+                        info = None
-                    if link is not None and not isinstance(link, h5py.HardLink):
+                    if info is not None and info.type != h5py.h5l.TYPE_HARD:
                         continue
-                    kids.append(f"{base}/{n}")
+                    addr = info.u if info is not None else None
-                for k in reversed(kids):
+                    kids.append((f"{base}/{n}", addr))
-                    stack.append((k, None))
+                for k, addr in reversed(kids):
+                    stack.append((k, None, addr))
             except Exception as e:  # noqa: BLE001
                 rec["list_error"] = err(e)
         objects.append(rec)

conformance/report.py

@@ -98,13 +98,37 @@ def is_h5py_be_vlen(i):
             and ">" in (i.get("ours_dtype") or ""))
 
 
+# Objects the reference (h5py 3.16 / HDF5 2.0) reads only because of an
+# HDF5 2.0 bug, and that clawhdf5 refuses: each one reads past a buffer or
+# returns bytes the file does not hold, and libhdf5's develop branch refuses all
+# three. (file, object) -> why. Checked 2026-09-26 against HDF5 2.0.0
+# and HDFGroup/hdf5 develop sources; see docs/known-issues.md.
+LIBHDF5_BUGS = {
+    ("cve_hdf5/cvefiles/cve-2025-2308.h5", "/Scale_offset_long_long_data_le"):
+        "scale-offset codes run past the end of the chunk: HDF5 2.0 reads past its buffer; "
+        "libhdf5's develop branch refuses the chunk (\"Buffer too short\")",
+    ("cve_hdf5/cvefiles/cve-2025-44904.h5", "/Scale_offset_float_data_le"):
+        "unfiltered chunks of 38 and 37 bytes for 48-byte chunks: HDF5 2.0 fills the rest with "
+        "whatever its buffer held; libhdf5's develop branch refuses them (\"incorrect chunk size returned "
+        "from index for unfiltered chunk\")",
+    ("hdf5/test/testfiles/bad_nbit_parms_walk.h5", "/Nbit_int_data_le"):
+        "an N-Bit parameter list one value short: HDF5 2.0 reads past the list; libhdf5's own "
+        "test (`test_filter_bad_params`, test/dsets.c) now requires the read to fail",
+}
+
+
+def is_libhdf5_bug(rel, i):
+    return i["kind"] == "our-error" and any(
+        f == rel and i["detail"].startswith(obj + ":") for (f, obj) in LIBHDF5_BUGS)
+
+
 known = collections.defaultdict(list)
 for r in rows:
-    if r["class"] != "mismatch":
-        continue
     iss = issues.get(r["file"], [])
-    if iss and all(is_h5py_be_vlen(i) for i in iss):
+    if r["class"] == "mismatch" and iss and all(is_h5py_be_vlen(i) for i in iss):
         known["h5py-be-vlen"].append(r["file"])
+    if r["class"] == "our-error" and iss and all(is_libhdf5_bug(r["file"], i) for i in iss):
+        known["libhdf5-2.0"].append(r["file"])
 
 
 # --- the CVE corpus: clawhdf5 vs h5dump vs h5py ------------------------------
@@ -230,9 +254,13 @@ for c in sorted(by_corpus):
     w(f"| {c} | {sum(cnt.values())} | " + " | ".join(str(cnt.get(k, 0)) for k in CLASSES) + " |")
 w(f"| **all** | **{len(rows)}** | " + " | ".join(f"**{total.get(k, 0)}**" for k in CLASSES) + " |")
 w("")
-n_known = sum(len(v) for v in known.values())
+if known["h5py-be-vlen"]:
-if n_known:
+    w(f"{len(known['h5py-be-vlen'])} of the {total.get('mismatch', 0)} mismatches are a known h5py bug, "
-    w(f"{n_known} of the {total.get('mismatch', 0)} mismatches are a known h5py bug, not ours (see *Known not-our-bug*).")
+      "not ours (see *Known not-our-bug*).")
+    w("")
+if known["libhdf5-2.0"]:
+    w(f"{len(known['libhdf5-2.0'])} of the {total.get('our-error', 0)} our-errors are corrupt data that "
+      "HDF5 2.0 reads only through a bug and clawhdf5 refuses (see *Known not-our-bug*).")
     w("")
 w("Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):")
 w("")
@@ -311,6 +339,11 @@ if res["incomparable"]:
     w("  (FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not")
     w("  compared (shape and presence still are): "
       + ", ".join(f"{k} ({n}x)" for k, n in res["incomparable"]) + ".")
+w("- **Corrupt data HDF5 2.0 reads through a bug.** clawhdf5 refuses these objects; h5py 3.16 /")
+w("  HDF5 2.0 returns values for them that the file does not hold:")
+for (f, obj), why in sorted(LIBHDF5_BUGS.items()):
+    here = "" if f in known["libhdf5-2.0"] else " (not an our-error in this run)"
+    w(f"  - `{f}` `{obj}`: {why}{here}.")
 w("- **References** are compared by presence only (`R`), not by target.")
 w("")
 if res.get("ref_only_errors"):

crates/clawhdf5-format/Cargo.toml

@@ -76,8 +76,10 @@ bitshuffle = ["lz4_flex", "ruzstd"]
 bzip2 = ["dep:bzip2", "std"]
 # Blosc 1 (32001) with its BloscLZ, LZ4, Snappy, Zlib and Zstandard codecs.
 blosc = ["lz4_flex", "ruzstd", "snap", "deflate", "std"]
+# Blosc2 (32026), read-only: frames, B2ND arrays, and the Blosc codecs above.
+blosc2 = ["blosc"]
 # Every plugin filter above.
-plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc"]
+plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2"]
 
 [[bench]]
 name = "parallel_decompress_bench"

crates/clawhdf5-format/src/attribute.rs

@@ -450,6 +450,8 @@ fn extract_attributes_with(
     on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
 ) -> Result<Vec<AttributeMessage>, FormatError> {
     let mut attrs = Vec::new();
+    // Each attribute's creation order, where the file records one.
+    let mut orders: Vec<u32> = Vec::new();
 
     // Collect compact attributes (inline in OH)
     for msg in &header.messages {
@@ -479,7 +481,10 @@ fn extract_attributes_with(
             };
             let attr = attr.and_then(|a| check_in_header(a, header));
             match attr {
-                Ok(attr) => attrs.push(attr),
+                Ok(attr) => {
+                    attrs.push(attr);
+                    orders.push(msg.creation_order.map_or(0, u32::from));
+                }
                 Err(e) => on_error(e)?,
             }
         }
@@ -487,20 +492,30 @@ fn extract_attributes_with(
 
     // Check for dense attributes via AttributeInfo message
     let attr_info = find_attribute_info(header, offset_size)?;
-    if let Some(info) = attr_info
+    if let Some(info) = &attr_info
         && let Some(fh_addr) = info.fractal_heap_address
     {
         extract_dense_attributes(
             file_data,
-            &info,
+            info,
             fh_addr,
             offset_size,
             length_size,
             &mut attrs,
+            &mut orders,
             on_error,
         )?;
     }
 
+    // An object that tracks attribute creation order lists its attributes
+    // in that order (h5py's `track_order=True`), as libhdf5 does; otherwise
+    // they come in storage order.
+    if attr_info.is_some_and(|i| i.max_creation_index.is_some()) {
+        let mut paired: Vec<(u32, AttributeMessage)> = orders.into_iter().zip(attrs).collect();
+        paired.sort_by_key(|(o, _)| *o);
+        attrs = paired.into_iter().map(|(_, a)| a).collect();
+    }
+
     Ok(attrs)
 }
 
@@ -518,7 +533,9 @@ fn find_attribute_info(
     Ok(None)
 }
 
-/// Extract attributes from dense storage (fractal heap + B-tree v2).
+/// Extract attributes from dense storage (fractal heap + B-tree v2), and
+/// each one's creation order into `orders`.
+#[allow(clippy::too_many_arguments)]
 fn extract_dense_attributes(
     file_data: &[u8],
     attr_info: &AttributeInfoMessage,
@@ -526,6 +543,7 @@ fn extract_dense_attributes(
     offset_size: u8,
     length_size: u8,
     attrs: &mut Vec<AttributeMessage>,
+    orders: &mut Vec<u32>,
     on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
 ) -> Result<(), FormatError> {
     // Parse fractal heap
@@ -561,7 +579,14 @@ fn extract_dense_attributes(
                 AttributeMessage::parse_in_file(&attr_data, file_data, offset_size, length_size)
             });
         match attr {
-            Ok(attr) => attrs.push(attr),
+            Ok(attr) => {
+                attrs.push(attr);
+                let order = record
+                    .data
+                    .get(id_len + 1..id_len + 5)
+                    .map_or(0, |b| u32::from_le_bytes([b[0], b[1], b[2], b[3]]));
+                orders.push(order);
+            }
             Err(e) => on_error(e)?,
         }
     }

crates/clawhdf5-format/src/btree_v2.rs

@@ -71,7 +71,7 @@ fn ensure_len(data: &[u8], pos: usize, needed: usize) -> Result<(), FormatError>
 
 /// Compute the number of bytes needed to represent a count, using variable-width encoding.
 /// B-tree v2 uses this for the number of records fields in internal nodes.
-fn bytes_for_max_records(max_nrec: u64) -> usize {
+pub(crate) fn bytes_for_max_records(max_nrec: u64) -> usize {
     if max_nrec == 0 {
         return 1;
     }
@@ -163,7 +163,7 @@ impl BTreeV2Header {
 /// Compute maximum records per node for a given depth level.
 /// leaf: (node_size - overhead) / record_size
 /// internal: depends on pointers
-fn max_records_leaf(node_size: u32, record_size: u16) -> u64 {
+pub(crate) fn max_records_leaf(node_size: u32, record_size: u16) -> u64 {
     // Leaf overhead: signature(4) + version(1) + type(1) + checksum(4) = 10
     let overhead = 10u32;
     if node_size <= overhead || record_size == 0 {
@@ -418,10 +418,7 @@ fn collect_internal_records(
 }
 
 /// Most records a subtree whose root is at `depth` can hold (libhdf5's
-/// `cum_max_nrec`): a leaf holds `max_leaf_nrec`; an internal node at depth
+/// `cum_max_nrec`). See [`node_info`].
-/// `d` holds `max_nrec(d)` records and `max_nrec(d) + 1` subtrees of depth
-/// `d - 1`, where `max_nrec(d)` is what fits in a node once each record is
-/// paired with a child pointer of the width depth `d` needs.
 fn cum_max_records(
     node_size: u32,
     record_size: u16,
@@ -429,24 +426,82 @@ fn cum_max_records(
     max_leaf_nrec: u64,
     depth: u16,
 ) -> u64 {
+    node_info_from_leaf(node_size, record_size, offset_size, max_leaf_nrec, depth)
+        .last()
+        .map_or(max_leaf_nrec, |n| n.cum_max_nrec)
+}
+
+/// Capacity of a B-tree v2 node at one depth, as libhdf5 computes it
+/// (`H5B2__hdr_init`'s `node_info`).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) struct NodeInfo {
+    /// Most records one node at this depth holds.
+    pub(crate) max_nrec: u64,
+    /// Most records a subtree rooted at this depth holds.
+    pub(crate) cum_max_nrec: u64,
+    /// Bytes a subtree's total record count takes in a pointer to a node
+    /// at this depth (0 for a leaf, whose count is its own).
+    pub(crate) cum_max_nrec_size: usize,
+}
+
+/// Node capacities for depths `0..=depth` (entry `d` for depth `d`): a leaf
+/// holds `max_nrec(0)` records; an internal node at depth `d` holds
+/// `max_nrec(d)` records and `max_nrec(d) + 1` subtrees of depth `d - 1`,
+/// where `max_nrec(d)` is what fits in a node once each record is paired
+/// with a child pointer of the width depth `d` needs (address, the child's
+/// record count in the width a *leaf's* maximum needs, and below the first
+/// internal level the child subtree's total in the width its maximum
+/// needs), with one pointer more than records.
+pub(crate) fn node_info(
+    node_size: u32,
+    record_size: u16,
+    offset_size: u8,
+    depth: u16,
+) -> Vec<NodeInfo> {
+    let max_leaf = max_records_leaf(node_size, record_size);
+    node_info_from_leaf(node_size, record_size, offset_size, max_leaf, depth)
+}
+
+fn node_info_from_leaf(
+    node_size: u32,
+    record_size: u16,
+    offset_size: u8,
+    max_leaf_nrec: u64,
+    depth: u16,
+) -> Vec<NodeInfo> {
     // Internal node overhead: signature(4) + version(1) + type(1) + checksum(4).
     const PREFIX: u64 = 10;
     let nrec_width = bytes_for_max_records(max_leaf_nrec) as u64;
-    let mut cum = max_leaf_nrec;
+    let mut info = Vec::with_capacity(usize::from(depth) + 1);
-    let mut cum_width = 0u64;
+    info.push(NodeInfo {
+        max_nrec: max_leaf_nrec,
+        cum_max_nrec: max_leaf_nrec,
+        cum_max_nrec_size: 0,
+    });
     for d in 1..=depth {
-        let ptr = u64::from(offset_size) + nrec_width + if d > 1 { cum_width } else { 0 };
+        let below = info[usize::from(d) - 1];
+        let ptr = u64::from(offset_size)
+            + nrec_width
+            + if d > 1 {
+                below.cum_max_nrec_size as u64
+            } else {
+                0
+            };
         let max_nrec = u64::from(node_size)
             .saturating_sub(PREFIX)
             .saturating_sub(ptr)
             / (u64::from(record_size) + ptr).max(1);
-        cum = max_nrec
+        let cum = max_nrec
             .saturating_add(1)
-            .saturating_mul(cum)
+            .saturating_mul(below.cum_max_nrec)
             .saturating_add(max_nrec);
-        cum_width = bytes_for_max_records(cum) as u64;
+        info.push(NodeInfo {
+            max_nrec,
+            cum_max_nrec: cum,
+            cum_max_nrec_size: bytes_for_max_records(cum),
+        });
     }
-    cum
+    info
 }
 
 #[cfg(test)]

crates/clawhdf5-format/src/btree_v2_write.rs

@@ -0,0 +1,396 @@
+//! Writing version-2 B-trees: a header (`BTHD`) and its nodes, leaves
+//! (`BTLF`) and, for more records than one leaf holds, internal nodes
+//! (`BTIN`) to any depth.
+//!
+//! Node capacities come from [`crate::btree_v2::node_info`], the arithmetic
+//! libhdf5 uses (`H5B2__hdr_init`) and the reader decodes pointers with, so
+//! the pointer widths the writer encodes are the ones every reader expects.
+
+#[cfg(not(feature = "std"))]
+use alloc::{format, vec, vec::Vec};
+
+use crate::btree_v2::{NodeInfo, bytes_for_max_records, node_info};
+use crate::checksum::jenkins_lookup3;
+use crate::error::FormatError;
+
+/// How a B-tree is laid out: its record type and node geometry, as the
+/// header records them.
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct BTreeV2Params {
+    /// Record type (5: link names, 6: link creation order, 8: attribute
+    /// names, 9: attribute creation order, 10/11: chunks).
+    pub(crate) tree_type: u8,
+    /// Bytes per node.
+    pub(crate) node_size: u32,
+    /// Bytes per record.
+    pub(crate) record_size: u16,
+    /// Split and merge percentages. The writer fills nodes itself; these
+    /// only tell libhdf5 when to split and merge as it modifies the tree.
+    pub(crate) split_percent: u8,
+    pub(crate) merge_percent: u8,
+}
+
+/// Size of a B-tree v2 header.
+pub(crate) fn header_size(offset_size: u8, length_size: u8) -> usize {
+    4 + 1 + 1 + 4 + 2 + 2 + 1 + 1 + offset_size as usize + 2 + length_size as usize + 4
+}
+
+/// Deepest tree the writer builds. Even at the smallest fan-out libhdf5's
+/// arithmetic allows, a few levels hold more records than any file could.
+const MAX_WRITE_DEPTH: u16 = 32;
+
+/// Write a B-tree v2 holding `records` (`record_size` bytes each,
+/// concatenated, already in the tree's key order) at `addr`: the header,
+/// then its nodes, each `node_size` bytes. No records gives a header with
+/// an undefined root.
+///
+/// The tree is as shallow as the node size allows: a single leaf when the
+/// records fit one, otherwise internal nodes above leaves. Records are
+/// spread evenly over each node's children, so every node but the root is
+/// at least about half full (above libhdf5's merge threshold, which is below
+/// half), and each node holds at most its depth's maximum.
+pub(crate) fn build_btree_v2(
+    p: BTreeV2Params,
+    records: &[u8],
+    addr: u64,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<u8>, FormatError> {
+    let rs = usize::from(p.record_size);
+    if rs == 0 || !records.len().is_multiple_of(rs) {
+        return Err(FormatError::SerializationError(format!(
+            "B-tree v2 records are {} bytes, not a multiple of the record size {rs}",
+            records.len()
+        )));
+    }
+    let n = (records.len() / rs) as u64;
+    let hdr_len = header_size(offset_size, length_size);
+
+    // The shallowest depth whose subtree can hold every record.
+    let mut info = node_info(p.node_size, p.record_size, offset_size, 0);
+    let max_leaf = info[0].max_nrec;
+    if max_leaf == 0 || max_leaf > u64::from(u16::MAX) {
+        return Err(FormatError::SerializationError(format!(
+            "a {}-byte B-tree v2 node holds {max_leaf} {}-byte records; \
+             a node holds 1 to 65535",
+            p.node_size, p.record_size
+        )));
+    }
+    let mut depth = 0u16;
+    while info[usize::from(depth)].cum_max_nrec < n {
+        depth += 1;
+        if depth > MAX_WRITE_DEPTH {
+            return Err(FormatError::SerializationError(format!(
+                "{n} records do not fit a B-tree v2 of {}-byte nodes",
+                p.node_size
+            )));
+        }
+        info = node_info(p.node_size, p.record_size, offset_size, depth);
+        let max = info[usize::from(depth)].max_nrec;
+        if max == 0 || max > u64::from(u16::MAX) {
+            return Err(FormatError::SerializationError(format!(
+                "a {}-byte B-tree v2 internal node holds {max} records; \
+                 a node holds 1 to 65535",
+                p.node_size
+            )));
+        }
+    }
+
+    let mut w = TreeWriter {
+        p,
+        records,
+        info: &info,
+        nrec_width: bytes_for_max_records(max_leaf),
+        offset_size,
+        first_node: addr + hdr_len as u64,
+        nodes: Vec::new(),
+    };
+    let root = (n > 0).then(|| w.node(depth, 0, n as usize)).transpose()?;
+
+    let mut out = Vec::with_capacity(hdr_len + w.nodes.len() * p.node_size as usize);
+    out.extend_from_slice(b"BTHD");
+    out.push(0); // version
+    out.push(p.tree_type);
+    out.extend_from_slice(&p.node_size.to_le_bytes());
+    out.extend_from_slice(&p.record_size.to_le_bytes());
+    out.extend_from_slice(&depth.to_le_bytes());
+    out.push(p.split_percent);
+    out.push(p.merge_percent);
+    match root {
+        Some(r) => push_uint(&mut out, r.addr, offset_size as usize),
+        None => out.extend(core::iter::repeat_n(0xFF, offset_size as usize)),
+    }
+    let root_nrec = root.map_or(0, |r| r.nrec);
+    out.extend_from_slice(&(root_nrec as u16).to_le_bytes());
+    push_uint(&mut out, n, length_size as usize);
+    let sum = jenkins_lookup3(&out);
+    out.extend_from_slice(&sum.to_le_bytes());
+    debug_assert_eq!(out.len(), hdr_len);
+    for node in &w.nodes {
+        out.extend_from_slice(node);
+    }
+    Ok(out)
+}
+
+/// A written node, as its parent points at it.
+#[derive(Debug, Clone, Copy)]
+struct NodeRef {
+    addr: u64,
+    /// Records in the node itself.
+    nrec: u64,
+    /// Records in the subtree it roots.
+    all_nrec: u64,
+}
+
+struct TreeWriter<'a> {
+    p: BTreeV2Params,
+    records: &'a [u8],
+    info: &'a [NodeInfo],
+    /// Width of a child's record count: what a leaf's maximum needs.
+    nrec_width: usize,
+    offset_size: u8,
+    /// Address of the first node (right after the header).
+    first_node: u64,
+    /// Nodes in file order (children before their parent).
+    nodes: Vec<Vec<u8>>,
+}
+
+impl TreeWriter<'_> {
+    fn record(&self, i: usize) -> &[u8] {
+        let rs = usize::from(self.p.record_size);
+        &self.records[i * rs..(i + 1) * rs]
+    }
+
+    fn push_node(&mut self, mut node: Vec<u8>) -> u64 {
+        // The checksum covers the node up to it, not the padding after.
+        let sum = jenkins_lookup3(&node);
+        node.extend_from_slice(&sum.to_le_bytes());
+        debug_assert!(node.len() <= self.p.node_size as usize);
+        node.resize(self.p.node_size as usize, 0);
+        let addr = self.first_node + self.nodes.len() as u64 * u64::from(self.p.node_size);
+        self.nodes.push(node);
+        addr
+    }
+
+    /// Write the subtree of `depth` holding records `first..first + n`.
+    fn node(&mut self, depth: u16, first: usize, n: usize) -> Result<NodeRef, FormatError> {
+        let rs = usize::from(self.p.record_size);
+        let mut node = Vec::with_capacity(self.p.node_size as usize);
+        if depth == 0 {
+            debug_assert!(n as u64 <= self.info[0].max_nrec);
+            node.extend_from_slice(b"BTLF");
+            node.push(0); // version
+            node.push(self.p.tree_type);
+            node.extend_from_slice(&self.records[first * rs..(first + n) * rs]);
+            let addr = self.push_node(node);
+            return Ok(NodeRef {
+                addr,
+                nrec: n as u64,
+                all_nrec: n as u64,
+            });
+        }
+
+        // As few children as hold the records, at least two, with the
+        // records spread evenly: `k` children and `k - 1` records between
+        // them.
+        let below = self.info[usize::from(depth) - 1].cum_max_nrec;
+        let k = (n as u64 + 1).div_ceil(below + 1).max(2);
+        let max = self.info[usize::from(depth)].max_nrec;
+        if k - 1 > max || (n as u64) < k - 1 + k {
+            return Err(FormatError::SerializationError(format!(
+                "cannot spread {n} B-tree v2 records over {k} children at depth {depth}"
+            )));
+        }
+        let k = k as usize;
+        let in_children = n - (k - 1);
+        let (base, extra) = (in_children / k, in_children % k);
+
+        let mut children = Vec::with_capacity(k);
+        let mut separators = Vec::with_capacity(k - 1);
+        let mut next = first;
+        for c in 0..k {
+            let m = base + usize::from(c < extra);
+            children.push(self.node(depth - 1, next, m)?);
+            next += m;
+            if c + 1 < k {
+                separators.push(next);
+                next += 1;
+            }
+        }
+        debug_assert_eq!(next, first + n);
+
+        node.extend_from_slice(b"BTIN");
+        node.push(0); // version
+        node.push(self.p.tree_type);
+        for &s in &separators {
+            node.extend_from_slice(self.record(s));
+        }
+        let total_width = if depth > 1 {
+            self.info[usize::from(depth) - 1].cum_max_nrec_size
+        } else {
+            0
+        };
+        for c in &children {
+            push_uint(&mut node, c.addr, self.offset_size as usize);
+            push_uint(&mut node, c.nrec, self.nrec_width);
+            if depth > 1 {
+                push_uint(&mut node, c.all_nrec, total_width);
+            }
+        }
+        let addr = self.push_node(node);
+        Ok(NodeRef {
+            addr,
+            nrec: (k - 1) as u64,
+            all_nrec: n as u64,
+        })
+    }
+}
+
+/// Append `v` as a `width`-byte little-endian integer.
+fn push_uint(buf: &mut Vec<u8>, v: u64, width: usize) {
+    let bytes = v.to_le_bytes();
+    buf.extend_from_slice(&bytes[..width.min(8)]);
+    buf.extend(vec![0u8; width.saturating_sub(8)]);
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+
+    fn params(node_size: u32, record_size: u16) -> BTreeV2Params {
+        BTreeV2Params {
+            tree_type: 5,
+            node_size,
+            record_size,
+            split_percent: 100,
+            merge_percent: 40,
+        }
+    }
+
+    /// `n` 11-byte records: a big-endian counter, so byte order is key order.
+    fn records(n: usize, rs: usize) -> Vec<u8> {
+        let mut out = Vec::with_capacity(n * rs);
+        for i in 0..n {
+            let mut r = vec![0u8; rs];
+            r[..8].copy_from_slice(&(i as u64).to_be_bytes());
+            out.extend_from_slice(&r);
+        }
+        out
+    }
+
+    fn roundtrip(node_size: u32, rs: u16, n: usize, os: u8, ls: u8) -> BTreeV2Header {
+        let recs = records(n, usize::from(rs));
+        let base = 4096u64;
+        let tree = build_btree_v2(params(node_size, rs), &recs, base, os, ls).unwrap();
+        let mut file = vec![0u8; base as usize];
+        file.extend_from_slice(&tree);
+        let hdr = BTreeV2Header::parse(&file, base as usize, os, ls).unwrap();
+        assert_eq!(hdr.total_records, n as u64);
+        let got = collect_btree_v2_records(&file, &hdr, os, ls).unwrap();
+        assert_eq!(got.len(), n);
+        let flat: Vec<u8> = got.into_iter().flat_map(|r| r.data).collect();
+        assert_eq!(flat, recs, "node {node_size} rs {rs} n {n}");
+        hdr
+    }
+
+    #[test]
+    fn one_leaf_then_deeper_trees_read_back_in_order() {
+        // 512-byte nodes of 11-byte records: 45 per leaf, 1149 at depth 1,
+        // 26 449 at depth 2.
+        let info = node_info(512, 11, 8, 3);
+        assert_eq!(
+            info.iter().map(|i| i.cum_max_nrec).collect::<Vec<_>>(),
+            [45, 1149, 26_449, 608_349]
+        );
+        for (n, depth) in [
+            (0, 0),
+            (1, 0),
+            (45, 0),
+            (46, 1),
+            (1149, 1),
+            (1150, 2),
+            (26_449, 2),
+            (26_450, 3),
+            (100_000, 3),
+        ] {
+            let hdr = roundtrip(512, 11, n, 8, 8);
+            assert_eq!(hdr.depth, depth, "{n} records");
+        }
+    }
+
+    #[test]
+    fn pointer_widths_follow_the_offset_and_length_sizes() {
+        for (os, ls) in [(4, 4), (8, 4), (4, 8), (2, 2)] {
+            roundtrip(512, 11, 5000, os, ls);
+        }
+        // Wide counts: a leaf of 2048 bytes / 9-byte records (226, one byte)
+        // and deeper subtree totals of three bytes.
+        roundtrip(2048, 9, 300_000, 8, 8);
+    }
+
+    #[test]
+    fn every_node_is_within_its_capacity_and_above_the_merge_threshold() {
+        let rs = 17u16;
+        let n = 70_000usize;
+        let info = node_info(512, rs, 8, 3);
+        let recs = records(n, usize::from(rs));
+        let tree = build_btree_v2(params(512, rs), &recs, 0, 8, 8).unwrap();
+        let hdr_len = header_size(8, 8);
+        let nodes = (tree.len() - hdr_len) / 512;
+        for i in 0..nodes {
+            let node = &tree[hdr_len + i * 512..hdr_len + (i + 1) * 512];
+            let sig = &node[..4];
+            if sig == b"BTLF" {
+                continue; // counts checked through the parents below
+            }
+            assert_eq!(sig, b"BTIN");
+        }
+        // Walk from the header: each child's count within [40%, 100%].
+        let hdr = BTreeV2Header::parse(&tree, 0, 8, 8).unwrap();
+        assert_eq!(hdr.depth, 3);
+        assert!(u64::from(hdr.num_records_in_root) <= info[3].max_nrec);
+        fn walk(tree: &[u8], addr: usize, nrec: usize, depth: usize, info: &[NodeInfo], rs: usize) {
+            if depth == 0 {
+                return;
+            }
+            let nrec_w = bytes_for_max_records(info[0].max_nrec);
+            let tot_w = if depth > 1 {
+                info[depth - 1].cum_max_nrec_size
+            } else {
+                0
+            };
+            let mut pos = addr + 6 + nrec * rs;
+            for _ in 0..=nrec {
+                let a = u64::from_le_bytes(tree[pos..pos + 8].try_into().unwrap()) as usize;
+                pos += 8;
+                let mut c = 0usize;
+                for b in 0..nrec_w {
+                    c |= usize::from(tree[pos + b]) << (8 * b);
+                }
+                pos += nrec_w + tot_w;
+                let max = info[depth - 1].max_nrec as usize;
+                assert!(c <= max && c * 100 > max * 40, "{c} of {max}");
+                walk(tree, a, c, depth - 1, info, rs);
+            }
+        }
+        walk(
+            &tree,
+            hdr.root_node_address as usize,
+            usize::from(hdr.num_records_in_root),
+            3,
+            &info,
+            usize::from(rs),
+        );
+        assert!(nodes > 0);
+    }
+
+    #[test]
+    fn a_node_too_small_or_too_big_is_an_error() {
+        assert!(build_btree_v2(params(16, 11), &records(1, 11), 0, 8, 8).is_err());
+        // A leaf with room for more than 65 535 records.
+        assert!(build_btree_v2(params(1 << 20, 11), &records(1, 11), 0, 8, 8).is_err());
+        // Records that are not whole.
+        assert!(build_btree_v2(params(512, 11), &[0u8; 12], 0, 8, 8).is_err());
+    }
+}

crates/clawhdf5-format/src/chunked_write.rs

@@ -6,6 +6,7 @@ extern crate alloc;
 #[cfg(not(feature = "std"))]
 use alloc::{format, vec, vec::Vec};
 
+use crate::btree_v2_write::{BTreeV2Params, build_btree_v2};
 use crate::checksum::jenkins_lookup3;
 use crate::chunk_cache::{CACHE_LINE_SIZE, align_to_cache_line};
 use crate::chunk_grid::ChunkGrid;
@@ -1105,11 +1106,12 @@ const BT2_CHUNK_FILTERED: u8 = 11;
 ///
 /// `records` are `(scaled coordinates, chunk)` in lexicographic order of the
 /// coordinates, which is the order the library's comparator
-/// (`H5VM_vector_cmp_u`) keeps them in. The tree is a single leaf: the
+/// (`H5VM_vector_cmp_u`) keeps them in. Up to 65 535 chunks go in a single
-/// library's 2048-byte node when the records fit, otherwise a leaf node
+/// leaf: the library's 2048-byte node when the records fit, otherwise a leaf
-/// sized to hold them all (the root's record count is 16-bit, so at most
+/// node sized to hold them all (the layout the writer has always used, kept
-/// 65535 chunks). Returns the bytes and the node size the layout message
+/// so those files do not change). More chunks get the library's 2048-byte
-/// must record.
+/// nodes with internal nodes above the leaves. Returns the bytes and the
+/// node size the layout message must record.
 fn build_btree_v2_chunk_index_at(
     rank: usize,
     records: &[(Vec<u64>, &WrittenChunk)],
@@ -1119,73 +1121,49 @@ fn build_btree_v2_chunk_index_at(
     base_address: u64,
 ) -> Result<(Vec<u8>, u32), FormatError> {
     let os = offset_size as usize;
-    let nrec = u16::try_from(records.len()).map_err(|_| {
-        FormatError::ChunkedReadError(
-            "more than 65535 chunks with more than one unlimited dimension: \
-             use larger chunks"
-                .into(),
-        )
-    })?;
     let chunk_size_bytes = has_filters.then(|| {
         let slots: Vec<Option<WrittenChunk>> =
             records.iter().map(|(_, c)| Some((*c).clone())).collect();
         filtered_chunk_size_len(&slots)
     });
     let record_size = os + chunk_size_bytes.map_or(0, |n| n + 4) + 8 * rank;
+    let record_size_u16 = u16::try_from(record_size)
+        .map_err(|_| FormatError::Overflow("B-tree v2 record size".into()))?;
+    let node_size = if records.len() <= usize::from(u16::MAX) {
         // Leaf: signature, version, type, records, checksum.
         let leaf_len = 4 + 1 + 1 + records.len() * record_size + 4;
-    let node_size = u32::try_from(leaf_len)
+        u32::try_from(leaf_len)
             .map_err(|_| FormatError::Overflow("B-tree v2 leaf size".into()))?
-        .max(BT2_NODE_SIZE);
+            .max(BT2_NODE_SIZE)
+    } else {
+        BT2_NODE_SIZE
+    };
     let tree_type = if has_filters {
         BT2_CHUNK_FILTERED
     } else {
         BT2_CHUNK_UNFILTERED
     };
 
-    let hdr_len = 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1 + os + 2 + length_size as usize + 4;
+    let mut flat = Vec::with_capacity(records.len() * record_size);
-    let leaf_address = base_address + hdr_len as u64;
-
-    let mut out = Vec::with_capacity(hdr_len + node_size as usize);
-    out.extend_from_slice(b"BTHD");
-    out.push(0); // version
-    out.push(tree_type);
-    out.extend_from_slice(&node_size.to_le_bytes());
-    out.extend_from_slice(&(record_size as u16).to_le_bytes());
-    out.extend_from_slice(&0u16.to_le_bytes()); // depth
-    out.push(BT2_SPLIT_PERCENT);
-    out.push(BT2_MERGE_PERCENT);
-    if records.is_empty() {
-        out.extend(core::iter::repeat_n(0xFF, os));
-    } else {
-        push_addr(&mut out, leaf_address, offset_size);
-    }
-    out.extend_from_slice(&nrec.to_le_bytes());
-    match length_size {
-        4 => out.extend_from_slice(&(records.len() as u32).to_le_bytes()),
-        _ => out.extend_from_slice(&(records.len() as u64).to_le_bytes()),
-    }
-    let sum = jenkins_lookup3(&out);
-    out.extend_from_slice(&sum.to_le_bytes());
-    debug_assert_eq!(out.len(), hdr_len);
-    if records.is_empty() {
-        return Ok((out, node_size));
-    }
-
-    let leaf_start = out.len();
-    out.extend_from_slice(b"BTLF");
-    out.push(0); // version
-    out.push(tree_type);
     for (scaled, chunk) in records {
-        push_index_element(&mut out, Some(chunk), offset_size, chunk_size_bytes);
+        push_index_element(&mut flat, Some(chunk), offset_size, chunk_size_bytes);
         for &c in scaled {
-            out.extend_from_slice(&c.to_le_bytes());
+            flat.extend_from_slice(&c.to_le_bytes());
         }
     }
-    let sum = jenkins_lookup3(&out[leaf_start..]);
+    let out = build_btree_v2(
-    out.extend_from_slice(&sum.to_le_bytes());
+        BTreeV2Params {
-    // The library reads whole nodes; pad the leaf out to the node size.
+            tree_type,
-    out.resize(leaf_start + node_size as usize, 0);
+            node_size,
+            record_size: record_size_u16,
+            split_percent: BT2_SPLIT_PERCENT,
+            merge_percent: BT2_MERGE_PERCENT,
+        },
+        &flat,
+        base_address,
+        offset_size,
+        length_size,
+    )?;
     Ok((out, node_size))
 }
 

crates/clawhdf5-format/src/data_read.rs

@@ -32,6 +32,80 @@ fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatErr
     Ok(())
 }
 
+/// The storage checks libhdf5 makes when it opens a dataset, before any
+/// data is read (`H5D__contig_check`, `H5D__compact_init`), so a dataset
+/// they refuse fails to open, as in libhdf5, instead of opening and
+/// reporting a shape nothing can be read from:
+///
+/// - the element count times the element size must not overflow 64 bits
+///   ("size of dataset's storage overflowed" — `cve-2024-32624`
+///   `/Dset_OBJREF`, 2^62 references of 8 bytes);
+/// - contiguous storage at a defined address must end within the file's
+///   `file_len` bytes (the HDF5 data up to the end of file the superblock
+///   records);
+/// - compact data must be exactly the dataset's size.
+///
+/// Deliberately not refused, unlike libhdf5: an empty contiguous dataset at
+/// a defined address (libhdf5's overflow test `addr + 0 <= addr` refuses
+/// it), which clawhdf5 up to v2.7.0 wrote. Chunked and virtual layouts are
+/// checked when their data is read.
+pub fn check_dataset_storage(
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    file_len: u64,
+) -> Result<(), FormatError> {
+    if !matches!(
+        layout,
+        DataLayout::Contiguous { .. } | DataLayout::Compact { .. }
+    ) {
+        return Ok(());
+    }
+    const OVERFLOWED: &str = "size of dataset's storage overflowed";
+    let n = dataspace
+        .checked_num_elements()
+        .map_err(|_| FormatError::InvalidDatasetStorage(OVERFLOWED))?;
+    let data_size = n
+        .checked_mul(u64::from(datatype.type_size()))
+        .ok_or(FormatError::InvalidDatasetStorage(OVERFLOWED))?;
+    match layout {
+        DataLayout::Contiguous {
+            address: Some(address),
+            ..
+        } if address
+            .checked_add(data_size)
+            .is_none_or(|end| end > file_len) =>
+        {
+            Err(FormatError::InvalidDatasetStorage(
+                "invalid dataset size, likely file corruption",
+            ))
+        }
+        DataLayout::Compact { data } if data.len() as u64 != data_size => {
+            Err(FormatError::InvalidDatasetStorage(
+                "bad value from dataset header - size of compact dataset's data buffer \
+                 doesn't match size of dataset data",
+            ))
+        }
+        _ => Ok(()),
+    }
+}
+
+/// How many bytes to read from a contiguous dataset's storage of
+/// `storage_size` bytes (the layout message's size) holding `needed` bytes
+/// of elements. libhdf5 reads the elements' bytes from the start of the
+/// storage and ignores storage past them (`H5D__contig_check` checks only
+/// that the elements fit in the file), so a larger storage reads; one too
+/// small to hold the elements is an error.
+pub fn contiguous_read_len(storage_size: u64, needed: usize) -> Result<usize, FormatError> {
+    if storage_size < needed as u64 {
+        return Err(FormatError::DataSizeMismatch {
+            expected: needed,
+            actual: usize::try_from(storage_size).unwrap_or(usize::MAX),
+        });
+    }
+    Ok(needed)
+}
+
 /// Zero-copy read of contiguous raw data, returning a borrowed slice.
 ///
 /// For contiguous layouts, returns a direct `&[u8]` slice into `file_data`.
@@ -55,13 +129,7 @@ pub fn read_raw_data_zerocopy<'a>(
         DataLayout::Contiguous { address, size } => {
             let addr = address.ok_or(FormatError::NoDataAllocated)?;
             let addr = addr as usize;
-            let sz = *size as usize;
+            let sz = contiguous_read_len(*size, expected_size)?;
-            if sz != expected_size {
-                return Err(FormatError::DataSizeMismatch {
-                    expected: expected_size,
-                    actual: sz,
-                });
-            }
             ensure_len(file_data, addr, sz)?;
             Ok(Some(&file_data[addr..addr + sz]))
         }
@@ -172,13 +240,7 @@ fn read_raw_data_full_impl(
         DataLayout::Contiguous { address, size } => {
             let addr = address.ok_or(FormatError::NoDataAllocated)?;
             let addr = addr as usize;
-            let sz = *size as usize;
+            let sz = contiguous_read_len(*size, expected_size)?;
-            if sz != expected_size {
-                return Err(FormatError::DataSizeMismatch {
-                    expected: expected_size,
-                    actual: sz,
-                });
-            }
             ensure_len(file_data, addr, sz)?;
             let mut out = crate::bulk_alloc::vec_for_bulk(sz);
             out.extend_from_slice(&file_data[addr..addr + sz]);
@@ -756,6 +818,120 @@ pub fn read_selection_native<T: NativeElement>(
     crate::gather::gather::<T>(raw, dims, elem_size, selection).map(Some)
 }
 
+/// The bytes of a slice of [`NativeElement`]s.
+#[cfg(feature = "std")]
+fn bytes_of_mut<T: NativeElement>(values: &mut [T]) -> &mut [u8] {
+    // SAFETY: `T: NativeElement` has no padding and every bit pattern is a
+    // valid value, so its storage may be viewed, and written, as bytes; the
+    // byte slice covers exactly the values' storage and borrows it
+    // exclusively for its lifetime.
+    unsafe {
+        core::slice::from_raw_parts_mut(
+            values.as_mut_ptr().cast::<u8>(),
+            core::mem::size_of_val(values),
+        )
+    }
+}
+
+/// `count` zeroed values of `T`, from zeroed pages where the allocator can
+/// (see [`crate::chunked_read::alloc_output`]) and backed by huge pages when
+/// large. A size taken from the file surfaces as an error, not an abort.
+#[cfg(feature = "std")]
+fn alloc_zeroed_values<T: NativeElement>(count: usize) -> Result<Vec<T>, FormatError> {
+    if count == 0 || core::mem::size_of::<T>() == 0 {
+        return Ok(Vec::new());
+    }
+    let failed = || {
+        FormatError::Overflow(format!(
+            "cannot allocate {count} values of {} bytes for dataset output",
+            core::mem::size_of::<T>()
+        ))
+    };
+    let layout = core::alloc::Layout::array::<T>(count).map_err(|_| failed())?;
+    // SAFETY: `layout` has non-zero size (count > 0, T not zero-sized).
+    let ptr = unsafe { std::alloc::alloc_zeroed(layout) };
+    if ptr.is_null() {
+        return Err(failed());
+    }
+    crate::bulk_alloc::advise_huge_pages(ptr, layout.size());
+    // SAFETY: allocated by the global allocator with the layout of
+    // `[T; count]`, which is what `Vec<T>` with capacity `count` frees; all
+    // bytes are zero, a valid `T` (`NativeElement`: any bit pattern is).
+    Ok(unsafe { Vec::from_raw_parts(ptr.cast::<T>(), count, count) })
+}
+
+/// Read a whole chunked dataset that stores `T` natively
+/// ([`NativeElement::is_native`]) straight into a `Vec<T>`: each chunk is
+/// decoded and copied to its place in the typed output, with no byte buffer
+/// to convert from afterwards. Unallocated chunks read as the dataset's fill
+/// value, as [`crate::fill_value::read_full_with_fill`] makes them.
+///
+/// `Ok(None)` when this does not apply — the datatype is not `T`'s native
+/// representation (another type, another byte order: the caller converts
+/// through the byte readers and the `read_as_*` functions), the layout is
+/// not chunked, no storage is allocated, or the data lives in external
+/// files. `cache` is the file's chunk cache, used as
+/// [`crate::chunked_read::read_chunked_data_cached`] uses it.
+#[cfg(feature = "std")]
+#[allow(clippy::too_many_arguments)]
+pub fn read_chunked_native<T: NativeElement>(
+    messages: &[crate::object_header::HeaderMessage],
+    file_data: &[u8],
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    cache: Option<&ChunkCache>,
+) -> Result<Option<Vec<T>>, FormatError> {
+    use crate::fill_value;
+    use crate::message_type::MessageType;
+
+    if !T::is_native(datatype)
+        || !matches!(layout, DataLayout::Chunked { .. })
+        || !fill_value::has_storage(layout)
+        || messages
+            .iter()
+            .any(|m| m.msg_type == MessageType::ExternalDataFiles)
+    {
+        return Ok(None);
+    }
+    let size = core::mem::size_of::<T>();
+    let mut values = crate::chunked_read::read_chunked_full(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        pipeline,
+        offset_size,
+        length_size,
+        cache,
+        |total_bytes| {
+            if !total_bytes.is_multiple_of(size) {
+                return Err(FormatError::DataSizeMismatch {
+                    expected: total_bytes.next_multiple_of(size),
+                    actual: total_bytes,
+                });
+            }
+            alloc_zeroed_values::<T>(total_bytes / size)
+        },
+        |values| bytes_of_mut(values),
+    )?;
+    let fill = fill_value::dataset_fill_value_in(file_data, messages, offset_size, length_size)?;
+    fill_value::apply_to_unallocated_chunks(
+        bytes_of_mut(&mut values),
+        file_data,
+        layout,
+        dataspace,
+        size,
+        fill.as_deref(),
+        offset_size,
+        length_size,
+    )?;
+    Ok(Some(values))
+}
+
 /// Convert raw bytes to `f64` values.
 pub fn read_as_f64(raw: &[u8], datatype: &Datatype) -> Result<Vec<f64>, FormatError> {
     // Array datatypes read as a flat sequence of their base elements, and
@@ -2632,6 +2808,71 @@ mod tests {
         assert_eq!(result.unwrap(), &[1.5f32, 2.5, 3.5]);
     }
 
+    /// libhdf5 reads a contiguous dataset's elements from the start of its
+    /// storage and ignores storage past them (cve-2024-32623's scalar
+    /// `/Dset1` has 240 bytes of storage for one 4-byte element). Storage too
+    /// small for the elements is still an error.
+    #[test]
+    fn contiguous_storage_larger_than_the_elements_reads() {
+        let dt = make_f64_le_type();
+        let ds = make_simple_dataspace(&[2]);
+        let mut file_data = vec![0u8; 64];
+        file_data[..8].copy_from_slice(&1.5f64.to_le_bytes());
+        file_data[8..16].copy_from_slice(&2.5f64.to_le_bytes());
+        file_data[16..24].copy_from_slice(&9.0f64.to_le_bytes());
+        let layout = DataLayout::Contiguous {
+            address: Some(0),
+            size: 40,
+        };
+        let raw = read_raw_data(&file_data, &layout, &ds, &dt).unwrap();
+        assert_eq!(raw, file_data[..16]);
+        let zc = read_raw_data_zerocopy(&file_data, &layout, &ds, &dt).unwrap();
+        assert_eq!(zc, Some(&file_data[..16]));
+        let small = DataLayout::Contiguous {
+            address: Some(0),
+            size: 8,
+        };
+        assert!(matches!(
+            read_raw_data(&file_data, &small, &ds, &dt),
+            Err(FormatError::DataSizeMismatch { .. })
+        ));
+    }
+
+    /// `H5D__contig_check` / `H5D__compact_init`, run when a dataset opens.
+    #[test]
+    fn dataset_storage_checks_at_open() {
+        let dt = make_f64_le_type();
+        let contiguous = |address| DataLayout::Contiguous { address, size: 0 };
+        // cve-2024-32624 `/Dset_OBJREF`: 2^62 + 2 elements of 8 bytes.
+        let huge = make_simple_dataspace(&[(1 << 62) + 2]);
+        assert_eq!(
+            check_dataset_storage(&contiguous(None), &huge, &dt, 1 << 20),
+            Err(FormatError::InvalidDatasetStorage(
+                "size of dataset's storage overflowed"
+            ))
+        );
+        let ds = make_simple_dataspace(&[4]);
+        assert!(check_dataset_storage(&contiguous(Some(100)), &ds, &dt, 132).is_ok());
+        assert!(matches!(
+            check_dataset_storage(&contiguous(Some(100)), &ds, &dt, 131),
+            Err(FormatError::InvalidDatasetStorage(_))
+        ));
+        assert!(matches!(
+            check_dataset_storage(&contiguous(Some(u64::MAX - 8)), &ds, &dt, u64::MAX),
+            Err(FormatError::InvalidDatasetStorage(_))
+        ));
+        // Not allocated, and (unlike libhdf5) empty at a defined address.
+        assert!(check_dataset_storage(&contiguous(None), &ds, &dt, 0).is_ok());
+        let empty = make_simple_dataspace(&[0]);
+        assert!(check_dataset_storage(&contiguous(Some(64)), &empty, &dt, 64).is_ok());
+        let compact = |n: usize| DataLayout::Compact { data: vec![0; n] };
+        assert!(check_dataset_storage(&compact(32), &ds, &dt, 0).is_ok());
+        assert!(matches!(
+            check_dataset_storage(&compact(24), &ds, &dt, 0),
+            Err(FormatError::InvalidDatasetStorage(_))
+        ));
+    }
+
     #[test]
     fn zerocopy_size_mismatch() {
         let dt = make_f64_le_type();

crates/clawhdf5-format/src/dataspace.rs

@@ -7,6 +7,9 @@ use alloc::vec::Vec;
 
 use crate::error::FormatError;
 
+/// Most dimensions a dataspace can have (`H5S_MAX_RANK`).
+pub const MAX_RANK: u8 = 32;
+
 /// Type of dataspace.
 #[derive(Debug, Clone, PartialEq)]
 pub enum DataspaceType {
@@ -67,6 +70,12 @@ impl Dataspace {
         let version = data[0];
         let rank = data[1];
         let flags = data[2];
+        // H5O__sdspace_decode's checks.
+        if rank > MAX_RANK {
+            return Err(FormatError::InvalidDataspace(
+                "simple dataspace dimensionality is too large",
+            ));
+        }
 
         let (space_type, header_size) = match version {
             1 => {
@@ -88,6 +97,11 @@ impl Dataspace {
                     2 => DataspaceType::Null,
                     _ => return Err(FormatError::InvalidDataspaceType(type_byte)),
                 };
+                if st != DataspaceType::Simple && rank > 0 {
+                    return Err(FormatError::InvalidDataspace(
+                        "invalid rank for scalar or NULL dataspace",
+                    ));
+                }
                 (st, 4usize)
             }
             _ => return Err(FormatError::InvalidDataspaceVersion(version)),
@@ -107,8 +121,13 @@ impl Dataspace {
         // Read max dimensions if flags bit 0 is set
         let max_dimensions = if flags & 0x01 != 0 {
             let mut max_dims = Vec::with_capacity(rank as usize);
-            for _ in 0..rank {
+            for &dim in &dimensions {
                 let val = read_length(data, pos, length_size)?;
+                if dim > val {
+                    return Err(FormatError::InvalidDataspace(
+                        "dataspace dimension size is greater than its maximum size",
+                    ));
+                }
                 max_dims.push(val);
                 pos += ls;
             }
@@ -176,7 +195,6 @@ impl Dataspace {
         match self.space_type {
             DataspaceType::Null => Ok(0),
             DataspaceType::Scalar => Ok(1),
-            DataspaceType::Simple if self.dimensions.is_empty() => Ok(0),
             DataspaceType::Simple => self
                 .dimensions
                 .iter()
@@ -195,18 +213,14 @@ impl Dataspace {
         match self.space_type {
             DataspaceType::Null => 0,
             DataspaceType::Scalar => 1,
-            DataspaceType::Simple => {
+            // A simple dataspace of rank 0 holds one element, as in libhdf5
-                if self.dimensions.is_empty() {
+            // (the product of no dimensions). Saturate rather than wrap: a
-                    0
+            // wrapped product could under-size a buffer. Size-critical
-                } else {
+            // callers use `checked_num_elements`.
-                    // Saturate rather than wrap: a wrapped product could
+            DataspaceType::Simple => self
-                    // under-size a buffer. Size-critical callers use
+                .dimensions
-                    // `checked_num_elements`.
-                    self.dimensions
                 .iter()
-                        .fold(1u64, |acc, &d| acc.saturating_mul(d))
+                .fold(1u64, |acc, &d| acc.saturating_mul(d)),
-                }
-            }
         }
     }
 }
@@ -352,4 +366,44 @@ mod tests {
         let ds = Dataspace::parse(&data, 8).unwrap();
         assert_eq!(ds.max_dimensions, Some(vec![10]));
     }
+
+    /// A simple dataspace of rank 0 (cve-2020-18494's `/dset1`) holds one
+    /// element in libhdf5, which h5py reads as shape `()`. It was 0.
+    #[test]
+    fn simple_rank_zero_holds_one_element() {
+        let data = build_v2_dataspace(0, 0, 1, &[], None);
+        let ds = Dataspace::parse(&data, 8).unwrap();
+        assert_eq!(ds.space_type, DataspaceType::Simple);
+        assert_eq!(ds.num_elements(), 1);
+        assert_eq!(ds.checked_num_elements().unwrap(), 1);
+    }
+
+    /// `H5O__sdspace_decode`'s checks.
+    #[test]
+    fn refuses_what_libhdf5_refuses() {
+        let too_many = build_v2_dataspace(33, 0, 1, &[1; 33], None);
+        assert!(matches!(
+            Dataspace::parse(&too_many, 8),
+            Err(FormatError::InvalidDataspace(_))
+        ));
+        let scalar_with_rank = build_v2_dataspace(1, 0, 0, &[4], None);
+        assert!(matches!(
+            Dataspace::parse(&scalar_with_rank, 8),
+            Err(FormatError::InvalidDataspace(_))
+        ));
+        let null_with_rank = build_v2_dataspace(1, 0, 2, &[4], None);
+        assert!(matches!(
+            Dataspace::parse(&null_with_rank, 8),
+            Err(FormatError::InvalidDataspace(_))
+        ));
+        let over_max = build_v1_dataspace(2, 0x01, &[5, 20], Some(&[10, 10]));
+        assert!(matches!(
+            Dataspace::parse(&over_max, 8),
+            Err(FormatError::InvalidDataspace(_))
+        ));
+        // 32 dimensions, and a size equal to the maximum or unlimited, are fine.
+        assert!(Dataspace::parse(&build_v2_dataspace(32, 0, 1, &[1; 32], None), 8).is_ok());
+        let at_max = build_v1_dataspace(2, 0x01, &[10, 20], Some(&[10, u64::MAX]));
+        assert!(Dataspace::parse(&at_max, 8).is_ok());
+    }
 }

crates/clawhdf5-format/src/error.rs

@@ -223,6 +223,26 @@ pub enum FormatError {
         /// The file's actual length in bytes.
         actual_len: u64,
     },
+    /// A link libhdf5 refuses to list: a symbol-table entry with an empty
+    /// name ("invalid link name"). Listing the group fails, as in libhdf5.
+    InvalidLinkName,
+    /// A dataspace message libhdf5 refuses to decode (the reason is
+    /// libhdf5's own error text): more than 32 dimensions, a rank on a
+    /// scalar or null dataspace, a dimension larger than its maximum.
+    InvalidDataspace(&'static str),
+    /// A dataset whose storage libhdf5 refuses when it opens the dataset
+    /// (the reason is libhdf5's own error text): an element count times
+    /// element size that overflows, contiguous storage past the end of the
+    /// file, compact data of the wrong size.
+    InvalidDatasetStorage(&'static str),
+    /// A superblock extension message libhdf5 refuses to decode when it
+    /// opens the file (the reason is libhdf5's own error text): a File Space
+    /// Info message that runs off its end or has a bad page size, a metadata
+    /// cache image outside the file, …
+    InvalidSuperblockExtension(&'static str),
+    /// A metadata cache image block libhdf5 refuses to load (the reason is
+    /// libhdf5's own error text).
+    InvalidCacheImage(&'static str),
 }
 
 impl fmt::Display for FormatError {
@@ -494,6 +514,21 @@ impl fmt::Display for FormatError {
                      but the file is {actual_len} bytes"
                 )
             }
+            FormatError::InvalidLinkName => {
+                write!(f, "invalid link name: a group entry has an empty name")
+            }
+            FormatError::InvalidDataspace(why) => {
+                write!(f, "invalid dataspace: {why}")
+            }
+            FormatError::InvalidDatasetStorage(why) => {
+                write!(f, "invalid dataset storage: {why}")
+            }
+            FormatError::InvalidSuperblockExtension(why) => {
+                write!(f, "invalid superblock extension: {why}")
+            }
+            FormatError::InvalidCacheImage(why) => {
+                write!(f, "invalid metadata cache image: {why}")
+            }
         }
     }
 }

crates/clawhdf5-format/src/file_writer.rs

@@ -7,6 +7,7 @@
 use alloc::{format, vec, vec::Vec};
 
 use crate::attribute::AttributeMessage;
+use crate::btree_v2_write::{BTreeV2Params, build_btree_v2};
 use crate::chunked_write::{
     ChunkOptions, PrecompressedChunks, build_chunked_data_from_precompressed, precompress_chunks,
 };
@@ -74,14 +75,57 @@ const DENSE_LINK_THRESHOLD: usize = 8;
 
 // ---- OH builders ----
 
+/// An object's attributes as its header stores them: inline Attribute
+/// messages, or (`dense`) the Attribute Info message of dense storage; with
+/// `track_order`, their creation order tracked and indexed.
+#[derive(Clone, Copy)]
+pub(crate) struct AttrStorage<'a> {
+    pub(crate) attrs: &'a [AttributeMessage],
+    pub(crate) dense: Option<&'a DenseAttrBlob>,
+    pub(crate) track_order: bool,
+}
+
+impl AttrStorage<'_> {
+    /// Add the attribute messages to the header being built. Tracking
+    /// creation order, as libhdf5 does it: the header's flags say so, an
+    /// Attribute Info message is written even for inline attributes (it
+    /// holds the next creation order), and each inline attribute's message
+    /// carries its creation order.
+    fn add_to(&self, w: &mut ObjectHeaderWriter) {
+        if self.track_order {
+            w.track_attr_order();
+        }
+        if let Some(blob) = self.dense {
+            w.add_message(MessageType::AttributeInfo, blob.attr_info_message.clone());
+            return;
+        }
+        if self.track_order {
+            w.add_message(
+                MessageType::AttributeInfo,
+                serialize_attribute_info(
+                    u64::MAX,
+                    u64::MAX,
+                    Some((self.attrs.len() as u16, u64::MAX)),
+                ),
+            );
+        }
+        for (i, attr) in self.attrs.iter().enumerate() {
+            w.add_message_with_order(
+                MessageType::Attribute,
+                attr.serialize(LENGTH_SIZE),
+                i as u16,
+            );
+        }
+    }
+}
+
 #[allow(clippy::too_many_arguments)]
 pub(crate) fn build_chunked_dataset_oh(
     dt: &Datatype,
     ds: &Dataspace,
     layout_message: &[u8],
     pipeline_message: Option<&[u8]>,
-    attrs: &[AttributeMessage],
+    attrs: AttrStorage<'_>,
-    dense_blob: Option<&DenseAttrBlob>,
     fill_message: &[u8],
     refcount: u32,
 ) -> Result<Vec<u8>, FormatError> {
@@ -93,13 +137,7 @@ pub(crate) fn build_chunked_dataset_oh(
     if let Some(pm) = pipeline_message {
         w.add_message(MessageType::FilterPipeline, pm.to_vec());
     }
-    if let Some(blob) = dense_blob {
+    attrs.add_to(&mut w);
-        w.add_message(MessageType::AttributeInfo, blob.attr_info_message.clone());
-    } else {
-        for attr in attrs {
-            w.add_message(MessageType::Attribute, attr.serialize(LENGTH_SIZE));
-        }
-    }
     add_refcount(&mut w, refcount);
     w.serialize()
 }
@@ -110,8 +148,7 @@ pub(crate) fn build_dataset_oh(
     ds: &Dataspace,
     data_addr: u64,
     data_size: u64,
-    attrs: &[AttributeMessage],
+    attrs: AttrStorage<'_>,
-    dense_blob: Option<&DenseAttrBlob>,
     fill_message: &[u8],
     refcount: u32,
 ) -> Result<Vec<u8>, FormatError> {
@@ -131,13 +168,7 @@ pub(crate) fn build_dataset_oh(
     dl.extend_from_slice(&data_addr.to_le_bytes());
     dl.extend_from_slice(&data_size.to_le_bytes());
     w.add_message(MessageType::DataLayout, dl);
-    if let Some(blob) = dense_blob {
+    attrs.add_to(&mut w);
-        w.add_message(MessageType::AttributeInfo, blob.attr_info_message.clone());
-    } else {
-        for attr in attrs {
-            w.add_message(MessageType::Attribute, attr.serialize(LENGTH_SIZE));
-        }
-    }
     add_refcount(&mut w, refcount);
     w.serialize()
 }
@@ -147,8 +178,7 @@ pub(crate) fn build_compact_dataset_oh(
     dt: &Datatype,
     ds: &Dataspace,
     data: &[u8],
-    attrs: &[AttributeMessage],
+    attrs: AttrStorage<'_>,
-    dense_blob: Option<&DenseAttrBlob>,
     fill_message: &[u8],
     refcount: u32,
 ) -> Result<Vec<u8>, FormatError> {
@@ -163,13 +193,7 @@ pub(crate) fn build_compact_dataset_oh(
     dl.extend_from_slice(&(data.len() as u16).to_le_bytes());
     dl.extend_from_slice(data);
     w.add_message(MessageType::DataLayout, dl);
-    if let Some(blob) = dense_blob {
+    attrs.add_to(&mut w);
-        w.add_message(MessageType::AttributeInfo, blob.attr_info_message.clone());
-    } else {
-        for attr in attrs {
-            w.add_message(MessageType::Attribute, attr.serialize(LENGTH_SIZE));
-        }
-    }
     add_refcount(&mut w, refcount);
     w.serialize()
 }
@@ -181,8 +205,7 @@ pub(crate) fn build_group_oh(
     links: &[LinkMessage],
     link_info: &[u8],
     dense_links: bool,
-    attrs: &[AttributeMessage],
+    attrs: AttrStorage<'_>,
-    dense_blob: Option<&DenseAttrBlob>,
     refcount: u32,
 ) -> Result<Vec<u8>, FormatError> {
     let mut w = ObjectHeaderWriter::new();
@@ -197,13 +220,7 @@ pub(crate) fn build_group_oh(
             w.add_message(MessageType::Link, link.serialize(OFFSET_SIZE));
         }
     }
-    if let Some(blob) = dense_blob {
+    attrs.add_to(&mut w);
-        w.add_message(MessageType::AttributeInfo, blob.attr_info_message.clone());
-    } else {
-        for attr in attrs {
-            w.add_message(MessageType::Attribute, attr.serialize(LENGTH_SIZE));
-        }
-    }
     add_refcount(&mut w, refcount);
     w.serialize()
 }
@@ -890,11 +907,31 @@ fn write_frhp(p: WriteFrhp) -> Vec<u8> {
     frhp
 }
 
+/// libhdf5 numbers the attributes of an object that tracks their creation
+/// order with a 2-byte counter.
+fn check_tracked_attr_count(track_order: bool, n: usize) -> Result<(), FormatError> {
+    if track_order && n > usize::from(u16::MAX) {
+        return Err(FormatError::SerializationError(format!(
+            "{n} attributes on one object with creation order tracked: libhdf5 \
+             numbers at most {} (set fewer, or turn off track_order)",
+            u16::MAX
+        )));
+    }
+    Ok(())
+}
+
 /// Build dense attribute storage for a set of attributes.
+///
+/// With `track_order` the Attribute Info message tracks creation order (an
+/// attribute's creation order is its position in `attrs`) and a type-9
+/// creation-order index follows the name index, as libhdf5 writes for h5py's
+/// `track_order=True`. libhdf5 numbers at most 65 535 attributes.
 pub(crate) fn build_dense_attrs(
     attrs: &[AttributeMessage],
     base_address: u64,
+    track_order: bool,
 ) -> Result<DenseAttrBlob, FormatError> {
+    check_tracked_attr_count(track_order, attrs.len())?;
     // Dense attrs use v3 attribute messages (adds character set encoding byte).
     let serialized: Vec<Vec<u8>> = attrs.iter().map(|a| a.serialize_v3(LENGTH_SIZE)).collect();
 
@@ -910,31 +947,55 @@ pub(crate) fn build_dense_attrs(
     let heap_id_length = heap.heap_id_length;
     let heap_ids = &heap.heap_ids;
 
-    // Build B-tree v2 type 8 records (17 bytes each)
+    // Build B-tree v2 type 8 records (17 bytes each), in the index's key
+    // order: libhdf5 compares the name hash, then — for names whose hashes
+    // collide — the names themselves (`strcmp`).
     let record_size: u16 = heap_id_length + 1 + 4 + 4;
-    let mut records: Vec<(u32, u32, Vec<u8>)> = Vec::with_capacity(attrs.len());
+    let mut order: Vec<usize> = (0..attrs.len()).collect();
-    for (i, heap_id) in heap_ids.iter().enumerate() {
+    order.sort_by(|&a, &b| {
+        name_hashes[a]
+            .cmp(&name_hashes[b])
+            .then_with(|| attrs[a].name.as_bytes().cmp(attrs[b].name.as_bytes()))
+    });
+    let records: Vec<Vec<u8>> = order
+        .into_iter()
+        .map(|i| {
             let mut rec = Vec::with_capacity(record_size as usize);
-        rec.extend_from_slice(heap_id);
+            rec.extend_from_slice(&heap_ids[i]);
             rec.push(0); // msg_flags
             rec.extend_from_slice(&(i as u32).to_le_bytes()); // creation_order
             rec.extend_from_slice(&name_hashes[i].to_le_bytes()); // hash
-        records.push((name_hashes[i], i as u32, rec));
+            rec
-    }
+        })
-    records.sort_by(|a, b| a.0.cmp(&b.0).then(a.1.cmp(&b.1)));
+        .collect();
-
-    let records: Vec<Vec<u8>> = records.into_iter().map(|(_, _, rec)| rec).collect();
     let bthd_addr = btree_addr;
     let mut blob = heap.blob;
-    blob.extend_from_slice(&single_leaf_v2_btree(
+    blob.extend_from_slice(&dense_v2_btree(8, record_size, &records, bthd_addr)?);
-        8,
-        record_size,
-        &records,
-        bthd_addr,
-        "attributes on one object",
-    )?);
 
-    let attr_info = serialize_attribute_info(frhp_addr, bthd_addr);
+    let order = if track_order {
+        // Type 9 records: heap ID, message flags, creation order (the key).
+        let records: Vec<Vec<u8>> = heap_ids
+            .iter()
+            .enumerate()
+            .map(|(i, heap_id)| {
+                let mut rec = heap_id.clone();
+                rec.push(0); // msg_flags
+                rec.extend_from_slice(&(i as u32).to_le_bytes());
+                rec
+            })
+            .collect();
+        let corder_addr = base_address + blob.len() as u64;
+        blob.extend_from_slice(&dense_v2_btree(
+            9,
+            heap_id_length + 1 + 4,
+            &records,
+            corder_addr,
+        )?);
+        Some((attrs.len() as u16, corder_addr))
+    } else {
+        None
+    };
+    let attr_info = serialize_attribute_info(frhp_addr, bthd_addr, order);
 
     Ok(DenseAttrBlob {
         attr_info_message: attr_info,
@@ -953,69 +1014,57 @@ pub(crate) struct DenseLinkBlob {
     pub(crate) blob: Vec<u8>,
 }
 
-/// A v2 B-tree of `btree_type` holding `records` (already in key order) in a
+/// libhdf5's node size for the dense link and attribute indexes it creates
-/// single leaf, laid out at `addr`: the header, then the leaf. `what` names
+/// (`H5G_NAME_BT2_NODE_SIZE`, `H5A_NAME_BT2_NODE_SIZE`, and the
-/// the records in the error for too many ("links in one group").
+/// creation-order indexes'), with their split and merge percentages.
-fn single_leaf_v2_btree(
+const DENSE_BT2_NODE_SIZE: u32 = 512;
+const DENSE_BT2_SPLIT_PERCENT: u8 = 100;
+const DENSE_BT2_MERGE_PERCENT: u8 = 40;
+
+/// A dense-storage v2 B-tree of `btree_type` holding `records` (already in
+/// key order), laid out at `addr`: the header, then its nodes.
+///
+/// Up to 65 535 records go in one leaf node sized to hold them (the layout
+/// the writer has always used, kept so those files do not change). A leaf's
+/// record count is a 2-byte field, and libhdf5 sizes a leaf's capacity from
+/// the node size: a node with room for more than 65 535 records makes it
+/// overflow that count when it adds one, so the node is capped at a full
+/// leaf. More records get libhdf5's own 512-byte nodes, with internal nodes
+/// above the leaves.
+fn dense_v2_btree(
     btree_type: u8,
     record_size: u16,
     records: &[Vec<u8>],
     addr: u64,
-    what: &str,
 ) -> Result<Vec<u8>, FormatError> {
-    let os = OFFSET_SIZE as usize;
+    let rs = usize::from(record_size);
-    let ls = LENGTH_SIZE as usize;
+    let n = records.len();
-    // The root node's record count is a 2-byte field; more records need
+    let node_size = if n <= usize::from(u16::MAX) {
-    // internal nodes, which the writer does not build.
+        let btlf_size = 4 + 1 + 1 + n * rs + 4;
-    let num_records = u16::try_from(records.len()).map_err(|_| {
+        let max_node = 4 + 1 + 1 + usize::from(u16::MAX) * rs + 4;
-        FormatError::SerializationError(format!(
+        u32::try_from(btlf_size.next_power_of_two().max(512).min(max_node))
-            "{} {what}: at most {} can be written \
+            .map_err(|_| FormatError::Overflow("B-tree v2 node size".into()))?
-             (a deeper B-tree index is not implemented)",
+    } else {
-            records.len(),
+        DENSE_BT2_NODE_SIZE
-            u16::MAX
+    };
-        ))
+    let flat: Vec<u8> = records
-    })?;
+        .iter()
-    let bthd_size = 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1 + os + 2 + ls + 4;
+        .inspect(|r| debug_assert_eq!(r.len(), rs))
-    let btlf_size = 4 + 1 + 1 + (records.len() * record_size as usize) + 4;
+        .flat_map(|r| r.iter().copied())
-    // libhdf5 sizes a leaf's capacity from the node size, and a leaf's
+        .collect();
-    // record count is a 2-byte field: a node with room for more than
+    build_btree_v2(
-    // 65 535 records makes it overflow that count when it adds one (the
+        BTreeV2Params {
-    // group can then no longer be listed). Cap the node at a full leaf.
+            tree_type: btree_type,
-    let max_node = btlf_size - records.len() * record_size as usize
+            node_size,
-        + usize::from(u16::MAX) * record_size as usize;
+            record_size,
-    let node_size = btlf_size.next_power_of_two().max(512).min(max_node) as u32;
+            split_percent: DENSE_BT2_SPLIT_PERCENT,
-    let btlf_addr = addr + bthd_size as u64;
+            merge_percent: DENSE_BT2_MERGE_PERCENT,
-
+        },
-    let mut out = Vec::with_capacity(bthd_size + node_size as usize);
+        &flat,
-    out.extend_from_slice(b"BTHD");
+        addr,
-    out.push(0); // version
+        OFFSET_SIZE,
-    out.push(btree_type);
+        LENGTH_SIZE,
-    out.extend_from_slice(&node_size.to_le_bytes());
+    )
-    out.extend_from_slice(&record_size.to_le_bytes());
-    out.extend_from_slice(&0u16.to_le_bytes()); // depth = 0 (single leaf)
-    out.push(100); // split_percent
-    out.push(40); // merge_percent
-    write_offset(&mut out, btlf_addr, OFFSET_SIZE);
-    out.extend_from_slice(&num_records.to_le_bytes());
-    write_length(&mut out, records.len() as u64, LENGTH_SIZE);
-    let checksum = crate::checksum::jenkins_lookup3(&out);
-    out.extend_from_slice(&checksum.to_le_bytes());
-    debug_assert_eq!(out.len(), bthd_size);
-
-    let mut btlf = Vec::with_capacity(node_size as usize);
-    btlf.extend_from_slice(b"BTLF");
-    btlf.push(0); // version
-    btlf.push(btree_type);
-    for rec in records {
-        debug_assert_eq!(rec.len(), record_size as usize);
-        btlf.extend_from_slice(rec);
-    }
-    // The checksum follows the records, not the end of the node.
-    let checksum = crate::checksum::jenkins_lookup3(&btlf);
-    btlf.extend_from_slice(&checksum.to_le_bytes());
-    btlf.resize(node_size as usize, 0);
-    out.extend_from_slice(&btlf);
-    Ok(out)
 }
 
 /// Build dense link storage for a group's links, laid out at `base_address`.
@@ -1039,14 +1088,18 @@ pub(crate) fn build_dense_links(
     let heap = build_single_block_fractal_heap(&serialized, base_address, 32, 7)?;
     let heap_id_length = heap.heap_id_length;
 
-    // Type 5 records: hash(4) + heap_id. The B-tree's key is the name hash,
+    // Type 5 records: hash(4) + heap_id. The B-tree's key is the name hash
-    // so records are sorted by (hash, order).
+    // and, for names whose hashes collide, the name (libhdf5 compares them
+    // with `strcmp`): records out of that order are not found by name.
     let mut by_name: Vec<(u32, usize)> = links
         .iter()
         .enumerate()
         .map(|(i, l)| (crate::checksum::jenkins_lookup3(l.name.as_bytes()), i))
         .collect();
-    by_name.sort_unstable();
+    by_name.sort_unstable_by(|&(ha, a), &(hb, b)| {
+        ha.cmp(&hb)
+            .then_with(|| links[a].name.as_bytes().cmp(links[b].name.as_bytes()))
+    });
     let name_records: Vec<Vec<u8>> = by_name
         .iter()
         .map(|&(hash, i)| {
@@ -1057,12 +1110,11 @@ pub(crate) fn build_dense_links(
         .collect();
     let name_bt_addr = heap.btree_addr;
     let mut blob = heap.blob;
-    blob.extend_from_slice(&single_leaf_v2_btree(
+    blob.extend_from_slice(&dense_v2_btree(
         5,
         4 + heap_id_length,
         &name_records,
         name_bt_addr,
-        "links in one group",
     )?);
 
     let link_info_message = if track_order {
@@ -1082,12 +1134,11 @@ pub(crate) fn build_dense_links(
             })
             .collect();
         let order_bt_addr = base_address + blob.len() as u64;
-        blob.extend_from_slice(&single_leaf_v2_btree(
+        blob.extend_from_slice(&dense_v2_btree(
             6,
             8 + heap_id_length,
             &order_records,
             order_bt_addr,
-            "links in one group",
         )?);
         let next_order = by_order.last().map_or(0, |&(o, _)| o + 1);
         serialize_link_info(
@@ -1147,12 +1198,25 @@ fn encode_managed_id(offset: u64, length: u64, max_heap_size: u16, id_length: u1
     id
 }
 
-fn serialize_attribute_info(fh_addr: u64, btree_name_addr: u64) -> Vec<u8> {
+/// Serialize an Attribute Info message (version 0). `order` — the next
+/// creation order to assign and the creation-order index's address — is
+/// present when creation order is tracked and indexed.
+fn serialize_attribute_info(
+    fh_addr: u64,
+    btree_name_addr: u64,
+    order: Option<(u16, u64)>,
+) -> Vec<u8> {
     let mut data = Vec::new();
     data.push(0); // version
-    data.push(0x00); // flags
+    data.push(if order.is_some() { 0x03 } else { 0x00 }); // flags: tracked, indexed
+    if let Some((next, _)) = order {
+        data.extend_from_slice(&next.to_le_bytes());
+    }
     data.extend_from_slice(&fh_addr.to_le_bytes());
     data.extend_from_slice(&btree_name_addr.to_le_bytes());
+    if let Some((_, corder_addr)) = order {
+        data.extend_from_slice(&corder_addr.to_le_bytes());
+    }
     data
 }
 
@@ -1227,8 +1291,7 @@ pub(crate) fn build_vds_dataset_oh(
     dt: &Datatype,
     ds: &Dataspace,
     global_heap_addr: u64,
-    attrs: &[AttributeMessage],
+    attrs: AttrStorage<'_>,
-    dense_blob: Option<&DenseAttrBlob>,
     fill_message: &[u8],
     refcount: u32,
 ) -> Result<Vec<u8>, FormatError> {
@@ -1243,13 +1306,7 @@ pub(crate) fn build_vds_dataset_oh(
     dl.extend_from_slice(&global_heap_addr.to_le_bytes());
     dl.extend_from_slice(&1u32.to_le_bytes()); // object index 1 in the collection
     w.add_message(MessageType::DataLayout, dl);
-    if let Some(blob) = dense_blob {
+    attrs.add_to(&mut w);
-        w.add_message(MessageType::AttributeInfo, blob.attr_info_message.clone());
-    } else {
-        for attr in attrs {
-            w.add_message(MessageType::Attribute, attr.serialize(LENGTH_SIZE));
-        }
-    }
     add_refcount(&mut w, refcount);
     w.serialize()
 }
@@ -1284,7 +1341,8 @@ fn write_undef_offset(buf: &mut Vec<u8>, offset_size: u8) {
 pub struct FileWriter {
     /// The root group's contents (its name is unused).
     root: GroupBuilder,
-    /// Default for groups that do not call [`GroupBuilder::track_order`].
+    /// Default for groups and datasets that do not set their own
+    /// `track_order`.
     track_order: bool,
     /// Global alignment threshold: datasets with raw data >= this many bytes
     /// will have their data aligned to `alignment_bytes`.
@@ -1319,11 +1377,18 @@ struct DsFlat {
     virtual_sources: Option<Vec<VdsMapping>>,
     /// Number of hard links to the dataset.
     refcount: u32,
+    /// Track (and index) attribute creation order.
+    track_order: bool,
 }
 
 /// Convert a DatasetBuilder into a DsFlat, handling VDS (which does not
 /// require a `data` field).
-fn flatten_ds(db: DatasetBuilder, refcount: u32) -> Result<DsFlat, FormatError> {
+fn flatten_ds(
+    db: DatasetBuilder,
+    refcount: u32,
+    default_track_order: bool,
+) -> Result<DsFlat, FormatError> {
+    let track_order = db.track_order.unwrap_or(default_track_order);
     let dt = db.datatype.ok_or(FormatError::DatasetMissingData)?;
     let shape = db.shape.ok_or(FormatError::DatasetMissingShape)?;
     let is_vds = db.virtual_sources.is_some();
@@ -1373,6 +1438,7 @@ fn flatten_ds(db: DatasetBuilder, refcount: u32) -> Result<DsFlat, FormatError>
         alignment: db.alignment,
         virtual_sources: db.virtual_sources,
         refcount,
+        track_order,
     })
 }
 
@@ -1429,10 +1495,12 @@ impl FileWriter {
         self
     }
 
-    /// Track (and index) link creation order in every group that does not
+    /// Track (and index) creation order — of links and attributes in every
-    /// set its own [`GroupBuilder::track_order`], the root included — as
+    /// group that does not set its own [`GroupBuilder::track_order`], the
-    /// h5py's `track_order=True`: libhdf5 then lists members in the order
+    /// root included, and of attributes on every dataset that does not set
-    /// they were added. Off by default (members are listed by name).
+    /// its own [`DatasetBuilder::track_order`] — as h5py's
+    /// `track_order=True`: libhdf5 then lists members and attributes in the
+    /// order they were added. Off by default (they are listed by name).
     pub fn track_order(&mut self, track: bool) -> &mut Self {
         self.track_order = track;
         self
@@ -1503,7 +1571,7 @@ impl FileWriter {
         let all_ds: Vec<DsFlat> = tree
             .datasets
             .into_iter()
-            .map(|(db, refcount)| flatten_ds(db, refcount))
+            .map(|(db, refcount)| flatten_ds(db, refcount, self.track_order))
             .collect::<Result<_, _>>()?;
         let groups: Vec<GrpFlat> = tree
             .groups
@@ -1520,6 +1588,15 @@ impl FileWriter {
             })
             .collect();
 
+        // Refuse up front what dense storage would refuse after the work.
+        let tracked = groups
+            .iter()
+            .map(|g| (g.track_order, g.attrs.len()))
+            .chain(all_ds.iter().map(|d| (d.track_order, d.attrs.len())));
+        for (track, n) in tracked {
+            check_tracked_attr_count(track, n)?;
+        }
+
         // Every datatype must have an on-disk encoding before anything is laid
         // out: `Datatype::serialize` itself cannot report a failure.
         let group_attrs = groups.iter().flat_map(|g| &g.attrs);
@@ -1574,7 +1651,7 @@ impl FileWriter {
             .map(|(gi, g)| {
                 let dummy_links = g.link_messages(&[], &[]);
                 let attr_blob = group_dense[gi]
-                    .then(|| build_dense_attrs(&g.attrs, 0))
+                    .then(|| build_dense_attrs(&g.attrs, 0, g.track_order))
                     .transpose()?;
                 let li = if group_links_dense[gi] {
                     serialize_link_info(
@@ -1590,8 +1667,11 @@ impl FileWriter {
                     &dummy_links,
                     &li,
                     group_links_dense[gi],
-                    &g.attrs,
+                    AttrStorage {
-                    attr_blob.as_ref(),
+                        attrs: &g.attrs,
+                        dense: attr_blob.as_ref(),
+                        track_order: g.track_order,
+                    },
                     g.refcount,
                 )
                 .map(|oh| oh.len())
@@ -1610,7 +1690,7 @@ impl FileWriter {
         let mut dummy_cursor = 0u64;
         for (i, d) in all_ds.iter().enumerate() {
             let dense_blob = ds_dense[i]
-                .then(|| build_dense_attrs(&d.attrs, 0))
+                .then(|| build_dense_attrs(&d.attrs, 0, d.track_order))
                 .transpose()?;
             if is_vds[i] {
                 // VDS: dummy OH with address 0 to get the OH size. The global
@@ -1619,8 +1699,11 @@ impl FileWriter {
                     &d.dt,
                     &d.ds,
                     0, // dummy address
-                    &d.attrs,
+                    AttrStorage {
-                    dense_blob.as_ref(),
+                        attrs: &d.attrs,
+                        dense: dense_blob.as_ref(),
+                        track_order: d.track_order,
+                    },
                     &d.fill_message,
                     d.refcount,
                 )?;
@@ -1659,8 +1742,11 @@ impl FileWriter {
                     &d.ds,
                     &result.layout_message,
                     result.pipeline_message.as_deref(),
-                    &d.attrs,
+                    AttrStorage {
-                    dense_blob.as_ref(),
+                        attrs: &d.attrs,
+                        dense: dense_blob.as_ref(),
+                        track_order: d.track_order,
+                    },
                     &d.fill_message,
                     d.refcount,
                 )?;
@@ -1674,8 +1760,11 @@ impl FileWriter {
                     &d.dt,
                     &d.ds,
                     &d.raw,
-                    &d.attrs,
+                    AttrStorage {
-                    dense_blob.as_ref(),
+                        attrs: &d.attrs,
+                        dense: dense_blob.as_ref(),
+                        track_order: d.track_order,
+                    },
                     &d.fill_message,
                     d.refcount,
                 )?;
@@ -1690,8 +1779,11 @@ impl FileWriter {
                     &d.ds,
                     0,
                     d.raw.len() as u64,
-                    &d.attrs,
+                    AttrStorage {
-                    dense_blob.as_ref(),
+                        attrs: &d.attrs,
+                        dense: dense_blob.as_ref(),
+                        track_order: d.track_order,
+                    },
                     &d.fill_message,
                     d.refcount,
                 )?;
@@ -1735,7 +1827,7 @@ impl FileWriter {
                 group_link_blob_addrs.push(None);
             }
             if group_dense[gi] {
-                let blob = build_dense_attrs(&g.attrs, cursor2 as u64)?;
+                let blob = build_dense_attrs(&g.attrs, cursor2 as u64, g.track_order)?;
                 cursor2 += blob.blob.len();
                 group_dense_blobs.push(Some(blob));
             } else {
@@ -1752,7 +1844,8 @@ impl FileWriter {
                 let addr = cursor2 as u64;
                 cursor2 += sz;
                 if ds_dense[i] {
-                    let blob = build_dense_attrs(&all_ds[i].attrs, cursor2 as u64)?;
+                    let blob =
+                        build_dense_attrs(&all_ds[i].attrs, cursor2 as u64, all_ds[i].track_order)?;
                     cursor2 += blob.blob.len();
                     ds_dense_blobs.push(Some(blob));
                 } else {
@@ -1776,8 +1869,11 @@ impl FileWriter {
                     &d.dt,
                     &d.ds,
                     heap_addr,
-                    &d.attrs,
+                    AttrStorage {
-                    ds_dense_blobs[i].as_ref(),
+                        attrs: &d.attrs,
+                        dense: ds_dense_blobs[i].as_ref(),
+                        track_order: d.track_order,
+                    },
                     &d.fill_message,
                     d.refcount,
                 )?;
@@ -1804,8 +1900,11 @@ impl FileWriter {
                     &d.ds,
                     &result.layout_message,
                     result.pipeline_message.as_deref(),
-                    &d.attrs,
+                    AttrStorage {
-                    ds_dense_blobs[i].as_ref(),
+                        attrs: &d.attrs,
+                        dense: ds_dense_blobs[i].as_ref(),
+                        track_order: d.track_order,
+                    },
                     &d.fill_message,
                     d.refcount,
                 )?;
@@ -1820,8 +1919,11 @@ impl FileWriter {
                     &d.dt,
                     &d.ds,
                     &d.raw,
-                    &d.attrs,
+                    AttrStorage {
-                    ds_dense_blobs[i].as_ref(),
+                        attrs: &d.attrs,
+                        dense: ds_dense_blobs[i].as_ref(),
+                        track_order: d.track_order,
+                    },
                     &d.fill_message,
                     d.refcount,
                 )?;
@@ -1846,8 +1948,11 @@ impl FileWriter {
                     &d.ds,
                     cursor2 as u64,
                     d.raw.len() as u64,
-                    &d.attrs,
+                    AttrStorage {
-                    ds_dense_blobs[i].as_ref(),
+                        attrs: &d.attrs,
+                        dense: ds_dense_blobs[i].as_ref(),
+                        track_order: d.track_order,
+                    },
                     &d.fill_message,
                     d.refcount,
                 )?;
@@ -1915,8 +2020,11 @@ impl FileWriter {
                 &links,
                 &li,
                 link_blob.is_some(),
-                &g.attrs,
+                AttrStorage {
-                group_dense_blobs[gi].as_ref(),
+                    attrs: &g.attrs,
+                    dense: group_dense_blobs[gi].as_ref(),
+                    track_order: g.track_order,
+                },
                 g.refcount,
             )?;
             debug_assert_eq!(oh.len(), group_oh_sizes[gi]);
@@ -2147,6 +2255,92 @@ mod tests {
         assert_eq!(read_dataset_f64(&bytes, "data"), vec![1.0, 2.0, 3.0]);
     }
 
+    /// Attribute names of the object at `path`, in the order the reader
+    /// lists them.
+    fn attr_names(bytes: &[u8], path: &str) -> Vec<String> {
+        let sig = signature::find_signature(bytes).unwrap();
+        let sb = Superblock::parse(bytes, sig).unwrap();
+        let addr = if path == "/" {
+            sb.root_group_address
+        } else {
+            resolve_path_any(bytes, &sb, path).unwrap()
+        };
+        let hdr =
+            ObjectHeader::parse(bytes, addr as usize, sb.offset_size, sb.length_size).unwrap();
+        crate::attribute::extract_attributes_full(bytes, &hdr, sb.offset_size, sb.length_size)
+            .unwrap()
+            .into_iter()
+            .map(|a| a.name)
+            .collect()
+    }
+
+    #[test]
+    fn tracked_attributes_are_read_in_creation_order() {
+        let set = |names: &[String]| -> Vec<(String, AttrValue)> {
+            names
+                .iter()
+                .enumerate()
+                .map(|(i, n)| (n.clone(), AttrValue::I64(i as i64)))
+                .collect()
+        };
+        let compact: Vec<String> = ["zeta", "alpha", "mid"].map(String::from).to_vec();
+        let dense: Vec<String> = (0..30).rev().map(|i| format!("a{i:02}")).collect();
+        let mut fw = FileWriter::new();
+        fw.track_order(true);
+        for (n, v) in set(&compact) {
+            fw.set_root_attr(&n, v);
+        }
+        let ds = fw.create_dataset("dense");
+        ds.with_i32_data(&[1]);
+        for (n, v) in set(&dense) {
+            ds.set_attr(&n, v);
+        }
+        let ds = fw.create_dataset("untracked");
+        ds.with_i32_data(&[1]).track_order(false);
+        for (n, v) in set(&dense) {
+            ds.set_attr(&n, v);
+        }
+        let mut g = fw.create_group("g");
+        g.track_order(false);
+        for (n, v) in set(&compact) {
+            g.set_attr(&n, v);
+        }
+        fw.add_group(g.finish());
+        let bytes = fw.finish().unwrap();
+        assert_eq!(attr_names(&bytes, "/"), compact);
+        assert_eq!(attr_names(&bytes, "dense"), dense);
+        // Without tracking: storage order (inline: as added; dense: hash).
+        assert_eq!(attr_names(&bytes, "g"), compact);
+        let mut by_hash = dense.clone();
+        by_hash.sort_by_key(|n| crate::checksum::jenkins_lookup3(n.as_bytes()));
+        assert_eq!(attr_names(&bytes, "untracked"), by_hash);
+    }
+
+    #[test]
+    fn too_many_tracked_attributes_is_an_error() {
+        // libhdf5 numbers at most 65 535 attributes on an object that
+        // tracks their creation order (a 2-byte field). (`set_attr` looks
+        // for an earlier value, so 65 536 of them through the builder take
+        // a while; build the messages directly.)
+        let attrs: Vec<AttributeMessage> = (0..65_536)
+            .map(|i| build_attr_message(&format!("a{i}"), &AttrValue::I64(i)))
+            .collect();
+        let err = build_dense_attrs(&attrs, 0, true)
+            .err()
+            .unwrap()
+            .to_string();
+        assert!(err.contains("65536 attributes on one object"), "{err}");
+        assert!(build_dense_attrs(&attrs[1..], 0, true).is_ok());
+        assert!(build_dense_attrs(&attrs, 0, false).is_ok());
+        let mut fw = FileWriter::new();
+        let ds = fw.create_dataset("x");
+        ds.with_i32_data(&[1]).track_order(true);
+        for i in 0..20 {
+            ds.set_attr(&format!("a{i}"), AttrValue::I64(i));
+        }
+        assert!(fw.finish().is_ok());
+    }
+
     #[test]
     fn dense_attrs_root_group_self_roundtrip() {
         let mut fw = FileWriter::new();

crates/clawhdf5-format/src/filter_registry.rs

@@ -5,7 +5,8 @@
 //! * **Built-in filters** — a static table of the filters compiled into this
 //!   build: the HDF5 standard filters (deflate, shuffle, Fletcher32, szip,
 //!   N-Bit, scale-offset) and the plugin filters whose cargo features are
-//!   enabled (LZ4, Zstandard, pcodec, LZF, bitshuffle, bzip2, blosc).
+//!   enabled (LZ4, Zstandard, pcodec, LZF, bitshuffle, bzip2, blosc,
+//!   blosc2).
 //!   [`builtin_filters`] lists them.
 //! * **Registered filters** (`std` only) — codecs the application supplies
 //!   for any other ID with [`register_filter`] (a [`FilterCodec`], or just a
@@ -166,7 +167,7 @@ pub fn known_filter(id: u16) -> Option<(&'static str, Option<&'static str>)> {
         32019 => ("JPEG", None),
         32022 => ("BitGroom", None),
         32023 => ("Granular BitRound", None),
-        32026 => ("Blosc2", None),
+        32026 => ("Blosc2", Some("blosc2")),
         _ => return None,
     })
 }
@@ -452,12 +453,12 @@ pub(crate) mod tests {
     #[test]
     fn unsupported_filter_error_names_the_filter() {
         let msg = FormatError::UnsupportedFilter(32026).to_string();
+        assert!(msg.contains("Blosc2") && msg.contains("`blosc2`"), "{msg}");
+        let msg = FormatError::UnsupportedFilter(32013).to_string();
         assert!(
-            msg.contains("Blosc2") && msg.contains("not implemented"),
+            msg.contains("ZFP") && msg.contains("not implemented"),
             "{msg}"
         );
-        let msg = FormatError::UnsupportedFilter(32013).to_string();
-        assert!(msg.contains("ZFP"), "{msg}");
         let msg = FormatError::UnsupportedFilter(32000).to_string();
         assert!(msg.contains("LZF") && msg.contains("`lzf`"), "{msg}");
         assert_eq!(

crates/clawhdf5-format/src/filters.rs

@@ -148,6 +148,183 @@ pub fn decompress_chunk_exact(
     Ok(data)
 }
 
+/// Buffers a chunk decoder keeps between chunks, so decoding a dataset's
+/// chunks one after another reuses the same memory instead of allocating
+/// (and faulting in) fresh buffers for every chunk and every filter stage.
+///
+/// Use one per thread with [`decompress_chunk_exact_with`]. Buffers larger
+/// than [`DecodeScratch::RETAIN_BYTES`] are released by
+/// [`DecodeScratch::trim`], so a scratch kept for a long time (a
+/// thread-local, say) does not hold on to a huge chunk's memory.
+#[derive(Default)]
+pub struct DecodeScratch {
+    a: Vec<u8>,
+    b: Vec<u8>,
+    #[cfg(feature = "deflate")]
+    inflater: Option<flate2::Decompress>,
+}
+
+impl core::fmt::Debug for DecodeScratch {
+    fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
+        f.debug_struct("DecodeScratch")
+            .field("a_capacity", &self.a.capacity())
+            .field("b_capacity", &self.b.capacity())
+            .finish_non_exhaustive()
+    }
+}
+
+/// Which buffer holds the data between two filter stages.
+#[derive(Clone, Copy)]
+enum Stage {
+    /// `compressed[..len]`: still (a prefix of) the stored bytes.
+    Stored(usize),
+    A,
+    B,
+}
+
+impl DecodeScratch {
+    /// Largest buffer [`trim`](Self::trim) keeps (1 MiB): enough for common
+    /// chunk sizes (a 256 x 256 `f32` chunk is 256 KiB) while bounding what
+    /// every long-lived thread (rayon's workers never exit) holds on to, at
+    /// two buffers each.
+    pub const RETAIN_BYTES: usize = 1 << 20;
+
+    /// An empty scratch; buffers are allocated on first use.
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    /// Release any buffer larger than [`Self::RETAIN_BYTES`].
+    pub fn trim(&mut self) {
+        for buf in [&mut self.a, &mut self.b] {
+            if buf.capacity() > Self::RETAIN_BYTES {
+                *buf = Vec::new();
+            }
+        }
+    }
+}
+
+/// [`decompress_chunk_exact`] into reusable buffers: the decoded chunk is
+/// returned as a slice of `scratch` (or of `compressed`, when every filter
+/// that was applied only appended a checksum), valid until `scratch` is used
+/// again.
+///
+/// Deflate, shuffle and Fletcher32 — h5py's and libhdf5's usual pipeline —
+/// decode without allocating once `scratch` has grown to the chunk size: the
+/// inflater writes into a kept buffer (and its state is reset, not
+/// rebuilt), shuffle interleaves into the other buffer, and Fletcher32 checks
+/// the checksum and drops it in place. Every other filter goes through the
+/// filter registry as [`decompress_chunk_masked`] does, and its output
+/// replaces a scratch buffer. The result is byte for byte what
+/// [`decompress_chunk_exact`] returns, with the same errors.
+pub fn decompress_chunk_exact_with<'s>(
+    compressed: &'s [u8],
+    pipeline: &FilterPipeline,
+    chunk_size: usize,
+    element_size: u32,
+    filter_mask: u32,
+    coords: &[u64],
+    scratch: &'s mut DecodeScratch,
+) -> Result<&'s [u8], FormatError> {
+    // Same per-stage bounds as `decompress_chunk_masked`.
+    let mut bounds = [0usize; 32];
+    let mut bounds_vec = Vec::new();
+    let bounds: &mut [usize] = if pipeline.filters.len() <= bounds.len() {
+        &mut bounds[..pipeline.filters.len()]
+    } else {
+        bounds_vec.resize(pipeline.filters.len(), 0);
+        &mut bounds_vec
+    };
+    let mut size = chunk_size;
+    for (i, filter) in pipeline.filters.iter().enumerate() {
+        bounds[i] = size;
+        if !filter_skipped(filter_mask, i) {
+            size = filter_output_bound(filter.filter_id, size);
+        }
+    }
+
+    let mut stage = Stage::Stored(compressed.len());
+    for (i, filter) in pipeline.filters.iter().enumerate().rev() {
+        if filter_skipped(filter_mask, i) {
+            continue;
+        }
+        let ctx = FilterContext {
+            filter,
+            element_size: element_size as usize,
+            max_output: bounds[i],
+        };
+        // The stage's input, and the buffer its output goes to (the one
+        // not holding the input).
+        let (input, out): (&[u8], &mut Vec<u8>) = match stage {
+            Stage::Stored(len) => (&compressed[..len], &mut scratch.a),
+            Stage::A => (&scratch.a, &mut scratch.b),
+            Stage::B => (&scratch.b, &mut scratch.a),
+        };
+        let next = match stage {
+            Stage::Stored(_) | Stage::B => Stage::A,
+            Stage::A => Stage::B,
+        };
+        match filter.filter_id {
+            // Built in and never overridable (`register_filter` refuses
+            // built-in IDs), so the registry would pick exactly these.
+            FILTER_FLETCHER32 => {
+                // Check and drop the checksum where the data is.
+                let payload = fletcher32_payload(input)?;
+                stage = match stage {
+                    Stage::Stored(_) => Stage::Stored(payload),
+                    Stage::A => {
+                        scratch.a.truncate(payload);
+                        Stage::A
+                    }
+                    Stage::B => {
+                        scratch.b.truncate(payload);
+                        Stage::B
+                    }
+                };
+                continue;
+            }
+            FILTER_SHUFFLE => shuffle_decompress_into(input, ctx.element_size, out),
+            #[cfg(all(
+                feature = "deflate",
+                not(all(target_os = "macos", feature = "system-zlib-decompress"))
+            ))]
+            FILTER_DEFLATE => {
+                let limit = if ctx.max_output != 0 {
+                    ctx.max_output
+                } else {
+                    MAX_DECOMPRESS_SIZE
+                };
+                let size_hint = if ctx.max_output != 0 {
+                    ctx.max_output
+                } else {
+                    input.len().saturating_mul(4).min(1 << 20)
+                };
+                let inflater = scratch
+                    .inflater
+                    .get_or_insert_with(|| flate2::Decompress::new(true));
+                inflater.reset(true);
+                inflate_bounded_into(inflater, input, size_hint, limit, out)
+                    .map_err(FormatError::DecompressionError)?;
+            }
+            _ => *out = filter_registry::decode(input, &ctx)?,
+        }
+        stage = next;
+    }
+
+    let data: &[u8] = match stage {
+        Stage::Stored(len) => &compressed[..len],
+        Stage::A => &scratch.a,
+        Stage::B => &scratch.b,
+    };
+    if chunk_size != 0 && data.len() != chunk_size {
+        return Err(FormatError::ChunkedReadError(format!(
+            "chunk at {coords:?} decoded to {} bytes, expected {chunk_size}",
+            data.len()
+        )));
+    }
+    Ok(data)
+}
+
 /// Apply a filter pipeline to compress a chunk.
 /// Filters are applied in FORWARD order for compression.
 pub fn compress_chunk(
@@ -183,7 +360,7 @@ pub(crate) static BUILTIN_FILTERS: &[BuiltinFilter] = &[
     BuiltinFilter {
         id: FILTER_SHUFFLE,
         name: "shuffle",
-        decode: |d, c| shuffle_decompress(d, c.element_size),
+        decode: |d, c| shuffle_decompress(d, shuffle_type_size(c.client_data(), c.element_size)?),
         encode: Some(|d, c| shuffle_compress(d, c.element_size)),
     },
     BuiltinFilter {
@@ -278,25 +455,15 @@ pub(crate) static BUILTIN_FILTERS: &[BuiltinFilter] = &[
         },
         encode: None,
     },
+    #[cfg(feature = "blosc2")]
+    BuiltinFilter {
+        id: crate::filter_pipeline::FILTER_BLOSC2,
+        name: "blosc2",
+        decode: crate::filters_blosc2::blosc2_decode,
+        encode: None,
+    },
 ];
 
-/// Decode the HDF5 scale-offset filter (id 6).
-///
-/// Supports all three scale-offset variants:
-/// - `H5Z_SO_FLOAT_DSCALE` (0): `value = minval + code / 10^D`
-/// - `H5Z_SO_FLOAT_ESCALE` (1): `value = minval + code * 2^E`
-/// - `H5Z_SO_INT` (2): `value = minval + code`
-///
-/// Compressed buffer layout: `minbits` (u32 LE) · `minval_width` (1 byte)
-/// · `minval` (`minval_width` bytes) · 8 reserved bytes · MSB-first packed
-/// codes (`nelmts * minbits` bits). The all-ones code is reserved for the
-/// defined fill value.
-///
-/// `cd` is the `H5Zscaleoffset.c` parameter block: `[0]`=scale type,
-/// `[1]`=scale factor (decimal digits D for D-scale, binary exponent E for
-/// E-scale, interpreted as i32 for negative exponents), `[2]`=element count,
-/// `[4]`=element size, `[5]`=signed flag, `[6]`=byte order (1 = big-endian),
-/// `[7]`=fill defined, `[8..]`=fill value bits.
 /// `f64::powi` equivalent that works under `no_std` (no libm/std available).
 /// Exponentiation by squaring, matching `powi`'s semantics for negative
 /// exponents via reciprocal.
@@ -318,6 +485,32 @@ fn powi_f64(base: f64, mut exp: i32) -> f64 {
     if neg { 1.0 / result } else { result }
 }
 
+/// Decode the HDF5 scale-offset filter (id 6) as libhdf5 does
+/// (`H5Z__filter_scaleoffset`, reverse direction).
+///
+/// - Integers (`H5Z_SO_INT`): `value = minval + code`.
+/// - Floats, D-scale (`H5Z_SO_FLOAT_DSCALE`): `value = code / 10^D + min`.
+/// - Floats, E-scale: refused, as libhdf5 refuses it ("E-scaling method not
+///   supported"); no library writes it.
+///
+/// Compressed buffer layout: `minbits` (u32 LE) · the size of `minval` in
+/// bytes (1 byte; libhdf5 uses at most 8 of them) · `minval` · packed codes
+/// at byte 21, whatever the stored size of `minval` (`buf_offset` is fixed)
+/// · MSB-first, `minbits` bits per element. With a fill value defined, the
+/// all-ones code of `minbits` bits is the fill value — for `minbits == 0`
+/// that is every element. `minbits` equal to the element's full width means
+/// the elements are stored as they are (in little-endian order), and an
+/// integer scale factor of the full width means the filter left the chunk
+/// untouched.
+///
+/// `cd` is the `H5Zscaleoffset.c` parameter block: `[0]` scale type, `[1]`
+/// scale factor, `[2]` element count, `[3]` class (0 integer, 1 float),
+/// `[4]` element size, `[5]` signed, `[6]` byte order (1 = big-endian), `[7]`
+/// fill defined, `[8..]` fill value bits.
+///
+/// Packed data too short for its codes is an error (libhdf5 2.0 read past
+/// the end of the chunk buffer — `cve-2025-2308` — and later releases
+/// refuse it: "Buffer too short").
 fn scaleoffset_decompress(
     data: &[u8],
     cd: &[u32],
@@ -326,74 +519,101 @@ fn scaleoffset_decompress(
     const H5Z_SO_FLOAT_DSCALE: u32 = 0;
     const H5Z_SO_FLOAT_ESCALE: u32 = 1;
     const H5Z_SO_INT: u32 = 2;
+    /// Where the packed codes start (`buf_offset` in `H5Zscaleoffset.c`).
+    const BUF_OFFSET: usize = 21;
+    let err = |why: &str| FormatError::ChunkedReadError(format!("scale-offset: {why}"));
     if cd.len() < 8 {
-        return Err(FormatError::ChunkedReadError(
+        return Err(err("missing filter client data"));
-            "scale-offset: missing filter client data".into(),
-        ));
     }
     let scale_type = cd[0];
-    let is_float = scale_type == H5Z_SO_FLOAT_DSCALE || scale_type == H5Z_SO_FLOAT_ESCALE;
+    let is_float = match cd[3] {
-    if scale_type != H5Z_SO_INT && !is_float {
+        0 => false,
-        return Err(FormatError::UnsupportedFilter(FILTER_SCALEOFFSET));
+        1 => true,
+        _ => return Err(err("cannot use C integer datatype for cast")),
+    };
+    if is_float && scale_type != H5Z_SO_FLOAT_DSCALE && scale_type != H5Z_SO_FLOAT_ESCALE
+        || !is_float && scale_type != H5Z_SO_INT
+    {
+        return Err(err("invalid scale type"));
+    }
+    if scale_type == H5Z_SO_FLOAT_ESCALE {
+        return Err(err("E-scaling method not supported"));
     }
     let nelmts = cd[2] as usize;
     let elem_size = cd[4] as usize;
-    if elem_size == 0 || elem_size > 8 || (is_float && elem_size != 4 && elem_size != 8) {
+    let size_ok = if is_float {
-        return Err(FormatError::ChunkedReadError(
+        matches!(elem_size, 4 | 8)
-            "scale-offset: unsupported element size".into(),
+    } else {
-        ));
+        matches!(elem_size, 1 | 2 | 4 | 8)
+    };
+    if !size_ok {
+        return Err(err("cannot use C integer datatype for cast"));
+    }
+    let full_bits = elem_size * 8;
+    // An integer's scale factor is the number of bits kept; all of them
+    // means the filter stored the chunk as it was.
+    if !is_float && (cd[1] as i32).max(0) as usize > full_bits {
+        return Err(err("minimum number of bits exceeds maximum"));
+    }
+    if !is_float && cd[1] as i32 == full_bits as i32 {
+        return Ok(data.to_vec());
     }
     // The decoded output must match the chunk's uncompressed size; reject an
     // element count that would over-allocate (e.g. minbits == 0 with a huge
     // nelmts and no packed payload to bound it).
     let out_bytes = nelmts
         .checked_mul(elem_size)
-        .ok_or_else(|| FormatError::ChunkedReadError("scale-offset: size overflow".into()))?;
+        .ok_or_else(|| err("size overflow"))?;
     if expected_bytes != 0 && out_bytes > expected_bytes {
-        return Err(FormatError::ChunkedReadError(
+        return Err(err("element count exceeds chunk size"));
-            "scale-offset: element count exceeds chunk size".into(),
-        ));
     }
     let signed = cd[5] == 1;
     let big_endian = cd[6] == 1;
     let fill_defined = cd[7] == 1;
 
-    // --- header: minbits, then minval, then 8 reserved bytes ---
+    // --- header: minbits, then the size of minval and minval ---
     if data.len() < 5 {
-        return Err(FormatError::ChunkedReadError(
+        return Err(err("buffer too short"));
-            "scale-offset: truncated header".into(),
-        ));
     }
     let minbits = u32::from_le_bytes([data[0], data[1], data[2], data[3]]) as usize;
-    let minval_width = data[4] as usize;
+    if minbits > full_bits {
-    let minval_end = 5 + minval_width;
+        return Err(err("minimum number of bits exceeds size of type"));
-    if data.len() < minval_end {
-        return Err(FormatError::ChunkedReadError(
-            "scale-offset: truncated minval".into(),
-        ));
     }
-    let minval_bytes = &data[5..minval_end];
+    let minval_size = usize::from(data[4]).min(8);
+    let minval_bytes = data
+        .get(5..5 + minval_size)
+        .ok_or_else(|| err("buffer too short"))?;
+    let minval = minval_bytes
+        .iter()
+        .rev()
+        .fold(0u64, |acc, &b| (acc << 8) | u64::from(b));
 
-    // --- unpack the per-element codes (MSB-first), shared by both variants ---
+    // Full precision: the elements follow as they were, little-endian.
-    if minbits > 64 {
+    if minbits == full_bits {
-        return Err(FormatError::ChunkedReadError(
+        let raw = data
-            "scale-offset: implausible minbits".into(),
+            .get(BUF_OFFSET..)
-        ));
+            .and_then(|d| d.get(..out_bytes))
+            .ok_or_else(|| err("buffer too short"))?;
+        let mut out = raw.to_vec();
+        if big_endian {
+            for e in out.chunks_exact_mut(elem_size) {
+                e.reverse();
             }
+        }
+        return Ok(out);
+    }
+
+    // --- unpack the per-element codes (MSB-first) ---
     let codes: Vec<u64> = if minbits == 0 {
-        // No packed payload: every element equals minval.
+        // No packed payload: every code is 0.
         vec![0u64; nelmts]
     } else {
-        let packed = data.get(minval_end + 8..).ok_or_else(|| {
+        let packed = data.get(BUF_OFFSET..).unwrap_or(&[]);
-            FormatError::ChunkedReadError("scale-offset: truncated packed data".into())
-        })?;
         let need_bits = nelmts
             .checked_mul(minbits)
-            .ok_or_else(|| FormatError::ChunkedReadError("scale-offset: size overflow".into()))?;
+            .ok_or_else(|| err("size overflow"))?;
         if packed.len() * 8 < need_bits {
-            return Err(FormatError::ChunkedReadError(
+            return Err(err("packed data too short"));
-                "scale-offset: packed data too short".into(),
-            ));
         }
         let mut out = Vec::with_capacity(nelmts);
         let mut bitpos = 0usize;
@@ -408,35 +628,28 @@ fn scaleoffset_decompress(
         }
         out
     };
-    // The fill code (all ones) only exists when there are bits to pack.
+    // With a fill value defined, the all-ones code of `minbits` bits (0
-    let has_fill_code = fill_defined && minbits > 0 && minbits < 64;
+    // when minbits is 0) stands for it. minbits < 64 here.
-    // Computed for all 1..=64 widths; `1 << 64` would overflow, so saturate.
+    let fill_code: u64 = (1u64 << minbits) - 1;
-    let fill_code: u64 = if minbits == 0 {
+    let fill_bits = || {
-        0
+        let lo = u64::from(*cd.get(8).unwrap_or(&0));
-    } else if minbits >= 64 {
+        let hi = u64::from(*cd.get(9).unwrap_or(&0));
-        u64::MAX
+        lo | (hi << 32)
-    } else {
-        (1u64 << minbits) - 1
     };
 
     if is_float {
-        let is_escale = scale_type == H5Z_SO_FLOAT_ESCALE;
         let scale_factor = cd[1] as i32;
-        let minval = read_le_float(minval_bytes, elem_size);
+        let minval = bits_to_float(minval, elem_size);
         let fill_value = if fill_defined {
-            let lo = *cd.get(8).unwrap_or(&0) as u64;
+            bits_to_float(fill_bits(), elem_size)
-            let hi = *cd.get(9).unwrap_or(&0) as u64;
-            bits_to_float(lo | (hi << 32), elem_size)
         } else {
             0.0
         };
         let values: Vec<f64> = codes
             .iter()
             .map(|&code| {
-                if has_fill_code && code == fill_code {
+                if fill_defined && code == fill_code {
                     fill_value
-                } else if is_escale {
-                    minval + code as f64 * powi_f64(2.0, scale_factor)
                 } else if elem_size == 4 {
                     // H5Z_scaleoffset_modify_3/4 for `float`: the code is
                     // read as an `int` and everything is single precision,
@@ -456,21 +669,19 @@ fn scaleoffset_decompress(
             .collect();
         Ok(write_floats(&values, elem_size, big_endian))
     } else {
-        let minval = read_le_int(minval_bytes, signed);
         let fill_value: i64 = if fill_defined {
-            let lo = *cd.get(8).unwrap_or(&0) as u64;
+            sign_extend(fill_bits(), elem_size, signed)
-            let hi = *cd.get(9).unwrap_or(&0) as u64;
-            sign_extend(lo | (hi << 32), elem_size, signed)
         } else {
             0
         };
         let values: Vec<i64> = codes
             .iter()
             .map(|&code| {
-                if has_fill_code && code == fill_code {
+                if fill_defined && code == fill_code {
                     fill_value
                 } else {
-                    minval.wrapping_add(code as i64)
+                    // `(type)(buf[i] + minval)`: wraps at the element width.
+                    (code.wrapping_add(minval)) as i64
                 }
             })
             .collect();
@@ -478,21 +689,6 @@ fn scaleoffset_decompress(
     }
 }
 
-/// Read a little-endian float of `size` bytes (4 = f32, otherwise f64) as f64.
-fn read_le_float(bytes: &[u8], size: usize) -> f64 {
-    if size == 4 {
-        let mut b = [0u8; 4];
-        let n = bytes.len().min(4);
-        b[..n].copy_from_slice(&bytes[..n]);
-        f32::from_le_bytes(b) as f64
-    } else {
-        let mut b = [0u8; 8];
-        let n = bytes.len().min(8);
-        b[..n].copy_from_slice(&bytes[..n]);
-        f64::from_le_bytes(b)
-    }
-}
-
 /// Interpret the low bits of `raw` as an IEEE float of `size` bytes.
 fn bits_to_float(raw: u64, size: usize) -> f64 {
     if size == 4 {
@@ -525,16 +721,6 @@ fn write_floats(values: &[f64], elem_size: usize, big_endian: bool) -> Vec<u8> {
     out
 }
 
-/// Read a little-endian integer of `bytes.len()` bytes, sign-extending when
-/// `signed`. Used for the scale-offset `minval` field.
-fn read_le_int(bytes: &[u8], signed: bool) -> i64 {
-    let mut raw: u64 = 0;
-    for (i, &b) in bytes.iter().enumerate().take(8) {
-        raw |= (b as u64) << (i * 8);
-    }
-    sign_extend(raw, bytes.len().min(8), signed)
-}
-
 /// Interpret the low `size` bytes of `raw` as a (possibly signed) integer.
 fn sign_extend(raw: u64, size: usize, signed: bool) -> i64 {
     if size == 0 || size >= 8 {
@@ -892,33 +1078,55 @@ pub(crate) fn inflate_bounded(
     size_hint: usize,
     limit: usize,
 ) -> Result<Vec<u8>, String> {
-    use flate2::{Decompress, FlushDecompress, Status};
+    let mut out = Vec::new();
+    inflate_bounded_into(
+        &mut flate2::Decompress::new(true),
+        data,
+        size_hint,
+        limit,
+        &mut out,
+    )?;
+    Ok(out)
+}
+
+/// [`inflate_bounded`] with a fresh or reset `inflater`, into `out`: its
+/// contents are replaced and its allocation reused.
+#[cfg(feature = "deflate")]
+fn inflate_bounded_into(
+    inflater: &mut flate2::Decompress,
+    data: &[u8],
+    size_hint: usize,
+    limit: usize,
+    out: &mut Vec<u8>,
+) -> Result<(), String> {
+    use flate2::{FlushDecompress, Status};
 
     // One byte of headroom past the limit distinguishes an over-size stream
     // from one that legitimately ends exactly at the limit.
     let max_capacity = limit.saturating_add(1);
-    let mut out = Vec::new();
+    // A kept buffer may already be larger than `max_capacity`; the decoder
-    out.try_reserve_exact(size_hint.clamp(1, max_capacity))
+    // can then write past the limit, which the check below still refuses.
+    out.clear();
+    let want = size_hint.clamp(1, max_capacity);
+    out.try_reserve_exact(want)
         .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
 
-    let mut inflater = Decompress::new(true);
     loop {
         let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
         let status = inflater
-            .decompress_vec(
+            .decompress_vec(&data[in_before as usize..], out, FlushDecompress::Finish)
-                &data[in_before as usize..],
-                &mut out,
-                FlushDecompress::Finish,
-            )
             .map_err(|e| format!("deflate: {e}"))?;
         if out.len() > limit {
             return Err("deflate: output exceeds size limit".into());
         }
         match status {
-            Status::StreamEnd => return Ok(out),
+            Status::StreamEnd => return Ok(()),
             Status::Ok | Status::BufError if out.len() == out.capacity() => {
                 // Out of room: double, up to the limit.
-                let grow = out.capacity().min(max_capacity - out.capacity()).max(1);
+                let grow = out
+                    .capacity()
+                    .min(max_capacity.saturating_sub(out.capacity()))
+                    .max(1);
                 out.try_reserve_exact(grow)
                     .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
             }
@@ -1216,17 +1424,48 @@ fn zstd_compress(data: &[u8], level: u32) -> Result<Vec<u8>, FormatError> {
 /// Unshuffle (decompress direction): reconstruct interleaved element bytes.
 /// On disk: all byte-0s of each element together, then all byte-1s, etc.
 /// Output: elements in natural order.
+/// The element size the shuffle filter works with: its parameter, as
+/// libhdf5 uses it (`H5Z__filter_shuffle`), not the dataset's element size.
+/// They are the same in every file a library wrote; a corrupt parameter
+/// larger than the chunk makes libhdf5 leave the chunk as it is, and so
+/// does [`shuffle_decompress`] (`cve-2025-44905`'s `Shuffle_float_data_be`).
+/// A zero parameter is an error ("invalid shuffle parameters"); a pipeline
+/// without the parameter (never written by libhdf5) uses the element size.
+fn shuffle_type_size(cd: &[u32], element_size: usize) -> Result<usize, FormatError> {
+    match cd {
+        [] => Ok(element_size),
+        [0] | [_, _, ..] => Err(FormatError::FilterError(
+            "invalid shuffle parameters".into(),
+        )),
+        [size] => Ok(*size as usize),
+    }
+}
+
 fn shuffle_decompress(data: &[u8], element_size: usize) -> Result<Vec<u8>, FormatError> {
+    let mut result = Vec::new();
+    shuffle_decompress_into(data, element_size, &mut result);
+    Ok(result)
+}
+
+/// [`shuffle_decompress`] into `result`, replacing its contents and reusing
+/// its allocation.
+fn shuffle_decompress_into(data: &[u8], element_size: usize, result: &mut Vec<u8>) {
     if element_size <= 1 {
-        return Ok(data.to_vec());
+        result.clear();
+        result.extend_from_slice(data);
+        return;
     }
     // Like libhdf5, only whole elements are shuffled; trailing bytes (e.g. a
     // Fletcher32 checksum appended before the shuffle) are stored as-is.
     let whole = data.len() - data.len() % element_size;
     let (data, tail) = data.split_at(whole);
     let num_elements = data.len() / element_size;
-    let mut result = vec![0u8; whole];
+    // Every byte of `result[..whole]` is overwritten below, so a reused
-    result.reserve_exact(tail.len());
+    // buffer keeps its old bytes instead of being zeroed first; only growth
+    // is zero-filled.
+    result.truncate(whole);
+    result.reserve_exact(whole + tail.len() - result.len());
+    result.resize(whole, 0);
 
     // The shuffled stream is `element_size` byte planes of `num_elements`
     // bytes each; un-shuffling interleaves them. This is on the read path of
@@ -1245,10 +1484,10 @@ fn shuffle_decompress(data: &[u8], element_size: usize) -> Result<Vec<u8>, Forma
         }
     }
     match element_size {
-        2 => interleave::<2>(data, num_elements, &mut result),
+        2 => interleave::<2>(data, num_elements, result),
-        4 => interleave::<4>(data, num_elements, &mut result),
+        4 => interleave::<4>(data, num_elements, result),
-        8 => interleave::<8>(data, num_elements, &mut result),
+        8 => interleave::<8>(data, num_elements, result),
-        16 => interleave::<16>(data, num_elements, &mut result),
+        16 => interleave::<16>(data, num_elements, result),
         _ => {
             for (i, element) in result.chunks_exact_mut(element_size).enumerate() {
                 for (j, byte) in element.iter_mut().enumerate() {
@@ -1258,8 +1497,6 @@ fn shuffle_decompress(data: &[u8], element_size: usize) -> Result<Vec<u8>, Forma
         }
     }
     result.extend_from_slice(tail);
-
-    Ok(result)
 }
 
 /// Shuffle (compress direction): group bytes by position within each element.
@@ -1411,6 +1648,12 @@ fn fletcher32_compute(data: &[u8]) -> u32 {
 /// Verify Fletcher32 checksum and strip it from the data.
 /// The last 4 bytes are the stored checksum.
 fn fletcher32_verify(data: &[u8]) -> Result<Vec<u8>, FormatError> {
+    fletcher32_payload(data).map(|len| data[..len].to_vec())
+}
+
+/// Verify the Fletcher32 checksum that ends `data`; the length of the data
+/// before it.
+fn fletcher32_payload(data: &[u8]) -> Result<usize, FormatError> {
     if data.len() < 4 {
         return Err(FormatError::FilterError(
             "fletcher32: data too short for checksum".into(),
@@ -1430,7 +1673,7 @@ fn fletcher32_verify(data: &[u8]) -> Result<Vec<u8>, FormatError> {
             computed,
         });
     }
-    Ok(payload.to_vec())
+    Ok(payload.len())
 }
 
 /// Append Fletcher32 checksum to data.
@@ -1545,6 +1788,113 @@ fn pcodec_decompress(
 #[cfg(test)]
 mod tests {
 
+    /// `decompress_chunk_exact_with` returns exactly what
+    /// `decompress_chunk_exact` returns — data or error — for every pipeline
+    /// shape, filter mask and chunk size, with one scratch reused across all
+    /// of them in an order that grows, shrinks and swaps its buffers.
+    #[test]
+    fn decode_with_scratch_matches_the_allocating_decoder() {
+        let f = |filter_id: u16, client_data: Vec<u32>| FilterDescription {
+            filter_id,
+            name: None,
+            flags: 0,
+            client_data,
+        };
+        let mut pipelines = vec![
+            vec![f(FILTER_SHUFFLE, vec![4])],
+            vec![f(FILTER_FLETCHER32, vec![])],
+            // NetCDF-4's order: the checksum is taken before shuffle.
+            vec![f(FILTER_FLETCHER32, vec![]), f(FILTER_SHUFFLE, vec![4])],
+        ];
+        #[cfg(feature = "deflate")]
+        pipelines.extend([
+            vec![f(FILTER_DEFLATE, vec![4])],
+            vec![f(FILTER_SHUFFLE, vec![4]), f(FILTER_DEFLATE, vec![4])],
+            // h5py's order with `fletcher32=True`: checksum last.
+            vec![
+                f(FILTER_SHUFFLE, vec![4]),
+                f(FILTER_DEFLATE, vec![4]),
+                f(FILTER_FLETCHER32, vec![]),
+            ],
+            vec![
+                f(FILTER_FLETCHER32, vec![]),
+                f(FILTER_SHUFFLE, vec![4]),
+                f(FILTER_DEFLATE, vec![1]),
+            ],
+        ]);
+        #[cfg(feature = "lzf")]
+        pipelines.push(vec![
+            f(FILTER_SHUFFLE, vec![4]),
+            f(crate::filter_pipeline::FILTER_LZF, vec![]),
+        ]);
+
+        let mut scratch = DecodeScratch::new();
+        for elements in [1usize, 7, 4096, 3, 65536, 100] {
+            let data: Vec<u8> = (0..elements as u32)
+                .flat_map(|i| (i.wrapping_mul(2654435761) >> (i % 13)).to_le_bytes())
+                .collect();
+            for filters in &pipelines {
+                let pipeline = FilterPipeline {
+                    version: 2,
+                    filters: filters.clone(),
+                };
+                let n = filters.len() as u32;
+                for mask in 0..(1u32 << n) {
+                    // Encode only the filters the mask says were applied.
+                    let mut stored = data.clone();
+                    for (i, filter) in filters.iter().enumerate() {
+                        if mask & (1 << i) == 0 {
+                            let ctx = FilterContext {
+                                filter,
+                                element_size: 4,
+                                max_output: 0,
+                            };
+                            stored = filter_registry::encode(&stored, &ctx).unwrap();
+                        }
+                    }
+                    let mut cases = vec![(stored.clone(), data.len())];
+                    // Corrupt: last byte flipped, truncated, wrong size.
+                    let mut flipped = stored.clone();
+                    *flipped.last_mut().unwrap() ^= 0x5a;
+                    cases.push((flipped, data.len()));
+                    cases.push((stored[..stored.len() / 2].to_vec(), data.len()));
+                    cases.push((stored.clone(), data.len() + 4));
+                    cases.push((stored.clone(), 0));
+                    for (bytes, size) in cases {
+                        let want = decompress_chunk_exact(&bytes, &pipeline, size, 4, mask, &[3]);
+                        let got = decompress_chunk_exact_with(
+                            &bytes,
+                            &pipeline,
+                            size,
+                            4,
+                            mask,
+                            &[3],
+                            &mut scratch,
+                        )
+                        .map(<[u8]>::to_vec);
+                        match (&want, &got) {
+                            (Ok(w), Ok(g)) => assert_eq!(w, g, "{filters:?} mask {mask}"),
+                            (Err(w), Err(g)) => {
+                                assert_eq!(w.to_string(), g.to_string(), "{filters:?}")
+                            }
+                            _ => panic!("{filters:?} mask {mask} size {size}: {want:?} vs {got:?}"),
+                        }
+                    }
+                }
+            }
+        }
+        // Long-lived scratch gives back a huge chunk's buffers.
+        let big = vec![0u8; DecodeScratch::RETAIN_BYTES + 8];
+        let shuffle = FilterPipeline {
+            version: 2,
+            filters: vec![f(FILTER_SHUFFLE, vec![4])],
+        };
+        decompress_chunk_exact_with(&big, &shuffle, big.len(), 4, 0, &[0], &mut scratch).unwrap();
+        scratch.trim();
+        assert!(scratch.a.capacity() <= DecodeScratch::RETAIN_BYTES);
+        assert!(scratch.b.capacity() <= DecodeScratch::RETAIN_BYTES);
+    }
+
     /// A chunk whose pipeline decodes to fewer bytes than the chunk holds is
     /// an error naming the chunk, never a short buffer the reader pads.
     #[test]
@@ -2434,48 +2784,125 @@ mod tests {
         }
     }
 
-    fn as_f64(bytes: &[u8]) -> Vec<f64> {
+    /// E-scale: libhdf5 refuses it on read and write ("E-scaling method not
-        bytes
+    /// supported"); it was decoded here, never checked against anything.
-            .as_chunks::<8>()
+    #[test]
-            .0
+    fn scaleoffset_float_escale_is_refused_as_in_libhdf5() {
-            .iter()
+        let cd = [1u32, 1, 4, 1, 8, 0, 0, 0];
-            .map(|c| f64::from_le_bytes(*c))
+        let mut raw = vec![2, 0, 0, 0, 8];
-            .collect()
+        raw.extend_from_slice(&[0; 16]);
+        raw.push(0x1B);
+        assert!(scaleoffset_decompress(&raw, &cd, 0).is_err());
     }
 
+    /// A scale type that does not match the class is refused, as libhdf5
+    /// refuses it ("invalid scale type").
     #[test]
-    fn scaleoffset_float_escale_e1() {
+    fn scaleoffset_scale_type_must_match_the_class() {
-        // f64 [0.0, 2.0, 4.0, 6.0], E=1 (×2^1=2), fill_defined=0.
+        let mut raw = vec![2, 0, 0, 0, 8];
-        // cd: scale_type=1, E=1, nelmts=4, elem_size=8.
+        raw.extend_from_slice(&[0; 16]);
-        let cd = [1u32, 1, 4, 0, 8, 0, 0, 0];
+        raw.push(0x1B);
-        let raw: &[u8] = &[
+        assert!(scaleoffset_decompress(&raw, &[0, 0, 4, 0, 4, 1, 0, 0], 0).is_err());
-            2, 0, 0, 0, // minbits=2
+        assert!(scaleoffset_decompress(&raw, &[2, 0, 4, 1, 4, 1, 0, 0], 0).is_err());
-            8, // minval_width=8
+        assert!(scaleoffset_decompress(&raw, &[2, 0, 4, 7, 4, 1, 0, 0], 0).is_err());
-            0, 0, 0, 0, 0, 0, 0, 0, // minval=0.0f64
-            0, 0, 0, 0, 0, 0, 0, 0,    // 8 reserved bytes
-            0x1B, // packed codes: 00 01 10 11 MSB-first
-        ];
-        let got = as_f64(&scaleoffset_decompress(raw, &cd, 0).unwrap());
-        assert_eq!(got, vec![0.0, 2.0, 4.0, 6.0]);
     }
 
+    /// `cve-2025-44905` `/Scale_offset_short_data_be`, chunk (4, 0): the
+    /// stored size of `minval` is 0. libhdf5 reads a `minval` of 0 and the
+    /// packed codes from byte 21 regardless; we read them from byte 13
+    /// (5 + size + 8), so the values differed from h5py's.
     #[test]
-    fn scaleoffset_float_escale_neg_exp() {
+    fn scaleoffset_codes_start_at_byte_21_whatever_the_minval_size() {
-        // f64 [0.0, 0.5, 1.0, 1.5], E=-1 (×2^-1=0.5), fill_defined=0.
+        // big-endian i16, 12 elements, fill -2 (cd 65534), minbits 3.
-        // cd[1] = 0xFFFF_FFFF which casts to i32 = -1.
+        let mut cd = vec![2u32, 0, 12, 0, 2, 1, 1, 1, 65534];
-        let cd = [1u32, 0xFFFF_FFFF, 4, 0, 8, 0, 0, 0];
+        cd.resize(20, 0);
-        let raw: &[u8] = &[
+        let raw = unhex("0300000000d20e00000000000034000000000000000400000000");
-            2, 0, 0, 0, // minbits=2
+        let got = scaleoffset_decompress(&raw, &cd, 24).unwrap();
-            8, // minval_width=8
+        // Codes of 3 bits from byte 21 (04 00 00 00 00): 0, 1, 0, ...; h5py
-            0, 0, 0, 0, 0, 0, 0, 0, // minval=0.0f64
+        // reads the chunk's first row as 0, 1, 0.
-            0, 0, 0, 0, 0, 0, 0, 0,    // 8 reserved bytes
+        let want: Vec<i16> = vec![0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
-            0x1B, // packed codes: 00 01 10 11 MSB-first
+        let want: Vec<u8> = want.iter().flat_map(|v| v.to_be_bytes()).collect();
-        ];
+        assert_eq!(got, want);
-        let got = as_f64(&scaleoffset_decompress(raw, &cd, 0).unwrap());
-        let exp = [0.0f64, 0.5, 1.0, 1.5];
-        for (g, e) in got.iter().zip(exp.iter()) {
-            assert!((g - e).abs() < 1e-9, "got {g} expected {e}");
     }
+
+    /// With a fill value defined, libhdf5 compares each code with the
+    /// all-ones code of `minbits` bits — which for `minbits == 0` is 0, so a
+    /// chunk with no packed codes reads as all fill values (the compressor
+    /// writes that for a chunk of nothing but fill values). It read as
+    /// `minval` here.
+    #[test]
+    fn scaleoffset_minbits_zero_with_a_fill_value_is_all_fill() {
+        let mut cd = vec![2u32, 0, 3, 0, 4, 1, 0, 1, (-7i32) as u32];
+        cd.resize(20, 0);
+        let mut raw = vec![0, 0, 0, 0, 8];
+        raw.extend_from_slice(&5i64.to_le_bytes());
+        raw.extend_from_slice(&[0; 8]);
+        assert_eq!(
+            scaleoffset_decompress(&raw, &cd, 12).unwrap(),
+            i32_le(&[-7, -7, -7])
+        );
+        // Without a fill value every element is minval.
+        cd[7] = 0;
+        assert_eq!(
+            scaleoffset_decompress(&raw, &cd, 12).unwrap(),
+            i32_le(&[5, 5, 5])
+        );
+    }
+
+    /// `minbits` of the full width stores the elements as they are
+    /// (little-endian), without `minval`; a full-width integer scale factor
+    /// means the filter left the chunk untouched.
+    #[test]
+    fn scaleoffset_full_width_is_stored_as_is() {
+        let mut cd = vec![2u32, 0, 2, 0, 2, 1, 1, 0];
+        cd.resize(20, 0);
+        let mut raw = vec![16, 0, 0, 0, 8];
+        raw.extend_from_slice(&100i64.to_le_bytes());
+        raw.extend_from_slice(&[0; 8]);
+        raw.extend_from_slice(&[0x34, 0x12, 0xfe, 0xff]);
+        assert_eq!(
+            scaleoffset_decompress(&raw, &cd, 4).unwrap(),
+            [0x12, 0x34, 0xff, 0xfe]
+        );
+        cd[1] = 16;
+        assert_eq!(
+            scaleoffset_decompress(&[1, 2, 3, 4], &cd, 4).unwrap(),
+            [1, 2, 3, 4]
+        );
+        cd[1] = 17;
+        assert!(scaleoffset_decompress(&[1, 2, 3, 4], &cd, 4).is_err());
+        // minbits wider than the type.
+        cd[1] = 0;
+        raw[0] = 17;
+        assert!(scaleoffset_decompress(&raw, &cd, 4).is_err());
+    }
+
+    /// The shuffle filter uses its own parameter as the element size, as
+    /// libhdf5 does; a parameter larger than the chunk leaves the chunk as
+    /// it is (`cve-2025-44905` `/Shuffle_float_data_be`, whose parameter is
+    /// 4261347332: h5py and h5dump read the stored bytes unshuffled).
+    #[test]
+    fn shuffle_uses_its_parameter() {
+        let data: Vec<u8> = (0..16).collect();
+        let shuffled = shuffle_compress(&data, 4).unwrap();
+        let pipeline = |cd: Vec<u32>| FilterPipeline {
+            version: 2,
+            filters: vec![one_filter(FILTER_SHUFFLE, cd)],
+        };
+        // The dataset's element size says 2; the parameter says 4.
+        assert_eq!(
+            decompress_chunk(&shuffled, &pipeline(vec![4]), 16, 2).unwrap(),
+            data
+        );
+        assert_eq!(
+            decompress_chunk(&shuffled, &pipeline(vec![4_261_347_332]), 16, 4).unwrap(),
+            shuffled
+        );
+        assert!(decompress_chunk(&shuffled, &pipeline(vec![0]), 16, 4).is_err());
+        assert_eq!(
+            decompress_chunk(&shuffled, &pipeline(vec![]), 16, 4).unwrap(),
+            data
+        );
     }
 
     // --- N-Bit (filter id 5) --------------------------------------------------

crates/clawhdf5-format/src/filters_bitshuffle.rs

@@ -218,12 +218,20 @@ pub(crate) fn bitshuffle_decode(
 }
 
 /// Decode Zstandard frames into exactly `dst`, failing if they hold more.
+///
+/// ruzstd reserves a frame's declared window (by default up to 100 MiB)
+/// before decoding it, so the window is capped at what the output could
+/// need: twice `dst` (window sizes are rounded up), and at least 128 KiB.
+/// The encoders behind these filters (c-blosc, c-blosc2, bitshuffle)
+/// compress each block in one call with its size known, so libzstd's
+/// window never exceeds the block.
 #[cfg(any(feature = "bitshuffle", feature = "blosc"))]
 pub(crate) fn zstd_decode_into(
     decoder: &mut ruzstd::decoding::FrameDecoder,
     frames: &[u8],
     dst: &mut [u8],
 ) -> Result<usize, FormatError> {
+    decoder.set_max_window_size((2 * dst.len()).max(1 << 17) as u64);
     decoder
         .decode_all(frames, dst)
         .map_err(|e| FormatError::DecompressionError(format!("zstd: {e}")))

crates/clawhdf5-format/src/filters_blosc.rs

@@ -49,7 +49,7 @@ fn le32(b: &[u8], at: usize) -> Result<usize, FormatError> {
 
 /// The codec inside a Blosc frame (flags bits 5-7).
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
-enum Codec {
+pub(crate) enum Codec {
     BloscLz,
     Lz4,
     Snappy,
@@ -58,7 +58,7 @@ enum Codec {
 }
 
 impl Codec {
-    fn from_flags(flags: u8) -> Result<Codec, FormatError> {
+    pub(crate) fn from_flags(flags: u8) -> Result<Codec, FormatError> {
         match flags >> 5 {
             0 => Ok(Codec::BloscLz),
             1 => Ok(Codec::Lz4),
@@ -71,7 +71,7 @@ impl Codec {
 }
 
 /// Decode one codec stream into exactly `dst`.
-fn decode_stream(
+pub(crate) fn decode_stream(
     codec: Codec,
     src: &[u8],
     dst: &mut [u8],

crates/clawhdf5-format/src/group_v1.rs

@@ -21,12 +21,35 @@ pub struct GroupEntry {
     pub cache_type: u32,
 }
 
-/// Given a SymbolTableMessage, resolve all group children.
+/// Given a SymbolTableMessage, resolve all group children: the group's
+/// listing.
+///
+/// An entry with an empty name fails the listing with
+/// [`FormatError::InvalidLinkName`], as it fails libhdf5's link iteration
+/// (`H5G__ent_to_link`: "invalid link name"). Looking a name up
+/// ([`resolve_path`], and the path resolution in
+/// [`crate::group_v2::resolve_path_any`]) still works in such a group, as it
+/// does in libhdf5.
 pub fn resolve_v1_group_entries(
     file_data: &[u8],
     sym_table_msg: &SymbolTableMessage,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    let entries = v1_group_entries(file_data, sym_table_msg, offset_size, length_size)?;
+    if entries.iter().any(|e| e.name.is_empty()) {
+        return Err(FormatError::InvalidLinkName);
+    }
+    Ok(entries)
+}
+
+/// Every entry of a v1 group, empty names included — for looking a name up,
+/// which never matches an empty name.
+pub(crate) fn v1_group_entries(
+    file_data: &[u8],
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<Vec<GroupEntry>, FormatError> {
     // Parse local heap
     let heap = LocalHeap::parse(
@@ -207,8 +230,7 @@ pub fn resolve_path(
     let mut current_sym_table = root_sym_table.clone();
 
     for (i, component) in components.iter().enumerate() {
-        let entries =
+        let entries = v1_group_entries(file_data, &current_sym_table, offset_size, length_size)?;
-            resolve_v1_group_entries(file_data, &current_sym_table, offset_size, length_size)?;
 
         let found = entries.iter().find(|e| e.name == *component);
         match found {
@@ -425,6 +447,22 @@ mod tests {
         assert_eq!(entries[1].object_header_address, 0x2000);
     }
 
+    /// cve-2021-46244 `/BAG_root`: a symbol-table entry with an empty name.
+    /// libhdf5 fails the group's listing ("invalid link name"); a lookup of
+    /// the other names still works.
+    #[test]
+    fn empty_entry_name_fails_the_listing_not_a_lookup() {
+        let (file, msg) = build_synthetic_group(&[("", 0x1000, 0), ("elevation", 0x2000, 0)], 8, 8);
+        assert_eq!(
+            resolve_v1_group_entries(&file, &msg, 8, 8).unwrap_err(),
+            FormatError::InvalidLinkName
+        );
+        assert_eq!(
+            resolve_path(&file, &msg, "elevation", 8, 8).unwrap(),
+            0x2000
+        );
+    }
+
     #[test]
     fn resolve_path_single_level() {
         let (file, msg) =

crates/clawhdf5-format/src/group_v2.rs

@@ -450,7 +450,9 @@ fn resolve_group_entries(
             .find(|m| m.msg_type == MessageType::SymbolTable)
             .ok_or_else(|| FormatError::PathNotFound(String::from("no symbol table message")))?;
         let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
-        group_v1::resolve_v1_group_entries(file_data, &stm, offset_size, length_size)
+        // A lookup: an entry with an empty name (which fails a listing) is
+        // skipped by the name comparison, as in libhdf5.
+        group_v1::v1_group_entries(file_data, &stm, offset_size, length_size)
     } else if is_v2_group(object_header) {
         resolve_v2_group_entries(file_data, object_header, offset_size, length_size)
     } else {

crates/clawhdf5-format/src/lib.rs

@@ -61,6 +61,7 @@ pub mod attribute;
 pub mod attribute_info;
 pub mod btree_v1;
 pub mod btree_v2;
+mod btree_v2_write;
 mod bulk_alloc;
 pub mod checksum;
 pub mod chunk_cache;
@@ -86,6 +87,8 @@ pub mod filters;
 mod filters_bitshuffle;
 #[cfg(feature = "blosc")]
 pub mod filters_blosc;
+#[cfg(feature = "blosc2")]
+pub mod filters_blosc2;
 #[cfg(feature = "bzip2")]
 mod filters_bzip2;
 #[cfg(feature = "lzf")]
@@ -118,6 +121,7 @@ pub mod selection;
 pub mod shared_message;
 pub mod signature;
 pub mod superblock;
+pub mod superblock_ext;
 pub mod symbol_table;
 #[cfg(all(
     test,

crates/clawhdf5-format/src/object_header.rs

@@ -75,7 +75,40 @@ fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
     })
 }
 
+/// The kind of object an object header describes.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ObjectClass {
+    /// A group: the header has a Symbol Table or a Link Info message.
+    Group,
+    /// A dataset: the header has a Datatype and a Dataspace message.
+    Dataset,
+    /// A committed (named) datatype: a Datatype message, no Dataspace.
+    NamedDatatype,
+}
+
 impl ObjectHeader {
+    /// The kind of object this header describes, decided as libhdf5 decides
+    /// it (`H5O__obj_class_real`): group first (a Symbol Table or Link Info
+    /// message), then dataset (a Datatype *and* a Dataspace message — not a
+    /// Data Layout message), then named datatype (a Datatype message).
+    /// `None` when none applies; libhdf5 then cannot open the object
+    /// ("unable to determine object type").
+    ///
+    /// A header with a Datatype and a Data Layout message but no Dataspace
+    /// is a named datatype to libhdf5, not a dataset.
+    pub fn object_class(&self) -> Option<ObjectClass> {
+        let has = |t: MessageType| self.messages.iter().any(|m| m.msg_type == t);
+        if has(MessageType::SymbolTable) || has(MessageType::LinkInfo) {
+            Some(ObjectClass::Group)
+        } else if has(MessageType::Datatype) && has(MessageType::Dataspace) {
+            Some(ObjectClass::Dataset)
+        } else if has(MessageType::Datatype) {
+            Some(ObjectClass::NamedDatatype)
+        } else {
+            None
+        }
+    }
+
     /// Parse an object header at the given offset in the data buffer.
     ///
     /// `offset_size` and `length_size` come from the superblock.
@@ -662,6 +695,54 @@ fn check_message(
 mod tests {
     use super::*;
 
+    fn header_with(types: &[MessageType]) -> ObjectHeader {
+        ObjectHeader {
+            version: 2,
+            messages: types
+                .iter()
+                .map(|&msg_type| HeaderMessage {
+                    msg_type,
+                    size: 0,
+                    flags: 0,
+                    creation_order: None,
+                    data: Vec::new(),
+                })
+                .collect(),
+            reference_count: None,
+            flags: 0,
+            access_time: None,
+            modification_time: None,
+            change_time: None,
+            birth_time: None,
+        }
+    }
+
+    #[test]
+    fn object_class_follows_libhdf5() {
+        use MessageType::*;
+        let class = |t: &[MessageType]| header_with(t).object_class();
+        assert_eq!(
+            class(&[Datatype, Dataspace, DataLayout]),
+            Some(ObjectClass::Dataset)
+        );
+        // A Data Layout message does not make a dataset without a dataspace
+        // (cve-2024-33874 `/Dset1`: h5py opens it as a named datatype).
+        assert_eq!(
+            class(&[Datatype, DataLayout]),
+            Some(ObjectClass::NamedDatatype)
+        );
+        assert_eq!(class(&[Datatype]), Some(ObjectClass::NamedDatatype));
+        // Group messages win over dataset messages.
+        assert_eq!(
+            class(&[Datatype, Dataspace, SymbolTable]),
+            Some(ObjectClass::Group)
+        );
+        assert_eq!(class(&[LinkInfo]), Some(ObjectClass::Group));
+        // Link messages alone are not a group; nothing is not an object.
+        assert_eq!(class(&[Link]), None);
+        assert_eq!(class(&[]), None);
+    }
+
     // Helper: build a v1 object header with given messages
     fn build_v1_header(
         messages: &[(u16, &[u8], u8)], // (type, data, flags)

crates/clawhdf5-format/src/object_header_writer.rs

@@ -12,9 +12,16 @@ use crate::message_type::MessageType;
 /// its size truncated to 16 bits produced files libhdf5 refuses.
 pub const MAX_MESSAGE_SIZE: usize = u16::MAX as usize;
 
+/// Object header flags: attribute creation order tracked (each message
+/// then carries a 2-byte creation order) and indexed.
+const OHDR_ATTR_CRT_ORDER_TRACKED: u8 = 0x04;
+const OHDR_ATTR_CRT_ORDER_INDEXED: u8 = 0x08;
+
 /// Writer for v2 object headers with proper checksums.
 pub struct ObjectHeaderWriter {
-    messages: Vec<(MessageType, Vec<u8>, u8)>, // (type, data, msg_flags)
+    messages: Vec<(MessageType, Vec<u8>, u8, u16)>, // (type, data, msg_flags, creation order)
+    /// Attribute creation order tracked and indexed.
+    attr_order: bool,
 }
 
 impl ObjectHeaderWriter {
@@ -22,17 +29,33 @@ impl ObjectHeaderWriter {
     pub fn new() -> Self {
         Self {
             messages: Vec::new(),
+            attr_order: false,
         }
     }
 
+    /// Track and index attribute creation order, as libhdf5 does for an
+    /// object created with `H5P_CRT_ORDER_TRACKED | H5P_CRT_ORDER_INDEXED`
+    /// (h5py's `track_order=True`): the header's flags say so, and every
+    /// message carries a creation order (an attribute's own; 0 for the
+    /// others). libhdf5 reads the setting back from these flags.
+    pub fn track_attr_order(&mut self) {
+        self.attr_order = true;
+    }
+
     /// Add a message to the header with default flags (0).
     pub fn add_message(&mut self, msg_type: MessageType, data: Vec<u8>) {
-        self.messages.push((msg_type, data, 0));
+        self.messages.push((msg_type, data, 0, 0));
     }
 
     /// Add a message with specific flags.
     pub fn add_message_with_flags(&mut self, msg_type: MessageType, data: Vec<u8>, flags: u8) {
-        self.messages.push((msg_type, data, flags));
+        self.messages.push((msg_type, data, flags, 0));
+    }
+
+    /// Add a message with its creation order, which is written only when
+    /// attribute creation order is tracked ([`Self::track_attr_order`]).
+    pub fn add_message_with_order(&mut self, msg_type: MessageType, data: Vec<u8>, order: u16) {
+        self.messages.push((msg_type, data, 0, order));
     }
 
     /// Serialize the complete v2 object header (OHDR + messages + checksum).
@@ -41,10 +64,10 @@ impl ObjectHeaderWriter {
     /// than [`MAX_MESSAGE_SIZE`] (e.g. an attribute over ~64 KiB, which would
     /// need dense attribute storage), rather than writing a corrupt header.
     pub fn serialize(&self) -> Result<Vec<u8>, FormatError> {
-        if let Some((msg_type, data, _)) = self
+        if let Some((msg_type, data, _, _)) = self
             .messages
             .iter()
-            .find(|(_, data, _)| data.len() > MAX_MESSAGE_SIZE)
+            .find(|(_, data, _, _)| data.len() > MAX_MESSAGE_SIZE)
         {
             return Err(FormatError::SerializationError(format!(
                 "{msg_type:?} message is {} bytes; an object header message holds at most \
@@ -52,11 +75,13 @@ impl ObjectHeaderWriter {
                 data.len()
             )));
         }
-        // Calculate total message bytes: each message has type(1) + size(2) + flags(1) + data
+        // Calculate total message bytes: each message has type(1) + size(2) +
+        // flags(1) [+ creation order(2)] + data
+        let msg_header = if self.attr_order { 6 } else { 4 };
         let msg_bytes_total: usize = self
             .messages
             .iter()
-            .map(|(_, data, _)| 4 + data.len())
+            .map(|(_, data, _, _)| msg_header + data.len())
             .sum();
 
         // Determine chunk size field width based on msg_bytes_total
@@ -68,6 +93,12 @@ impl ObjectHeaderWriter {
             (0x02u8, 4)
         };
 
+        let flags = if self.attr_order {
+            flags | OHDR_ATTR_CRT_ORDER_TRACKED | OHDR_ATTR_CRT_ORDER_INDEXED
+        } else {
+            flags
+        };
+
         let mut buf = Vec::new();
 
         // OHDR signature
@@ -85,7 +116,7 @@ impl ObjectHeaderWriter {
         }
 
         // Messages
-        for (msg_type, data, msg_flags) in &self.messages {
+        for (msg_type, data, msg_flags, order) in &self.messages {
             let type_id = msg_type.to_u16();
             assert!(
                 type_id <= 255,
@@ -94,6 +125,9 @@ impl ObjectHeaderWriter {
             buf.push(type_id as u8); // type (1 byte in v2)
             buf.extend_from_slice(&(data.len() as u16).to_le_bytes()); // size (2 bytes)
             buf.push(*msg_flags); // flags
+            if self.attr_order {
+                buf.extend_from_slice(&order.to_le_bytes()); // creation order
+            }
             buf.extend_from_slice(data);
         }
 
@@ -193,6 +227,21 @@ mod tests {
         assert_eq!(hdr.messages.len(), 0);
     }
 
+    #[test]
+    fn tracked_attribute_order_is_in_the_flags_and_every_message() {
+        let mut writer = ObjectHeaderWriter::new();
+        writer.track_attr_order();
+        writer.add_message(MessageType::Dataspace, vec![1, 2, 3, 4]);
+        writer.add_message_with_order(MessageType::Attribute, vec![5, 6], 7);
+        let bytes = writer.serialize().unwrap();
+        assert_eq!(bytes[5] & 0x0C, 0x0C);
+        let hdr = ObjectHeader::parse(&bytes, 0, 8, 8).unwrap();
+        assert_eq!(hdr.messages.len(), 2);
+        assert_eq!(hdr.messages[0].creation_order, Some(0));
+        assert_eq!(hdr.messages[1].creation_order, Some(7));
+        assert_eq!(hdr.messages[1].data, vec![5, 6]);
+    }
+
     #[test]
     fn two_messages_roundtrip() {
         let mut writer = ObjectHeaderWriter::new();

crates/clawhdf5-format/src/parallel_read.rs

@@ -41,6 +41,132 @@ pub fn pool_can_parallelise() -> bool {
     rayon::current_num_threads() > 1
 }
 
+/// How many rayon workers [`run_with_helpers`] should ask to help with
+/// `items` work items, given that the calling thread works too: the pool's
+/// other threads (all of them when the caller is not one), at most one per
+/// item beyond the caller's first.
+pub(crate) fn helper_count(items: usize) -> usize {
+    let pool = rayon::current_num_threads();
+    // A one-thread pool means "decode on the calling thread" (the setting
+    // benchmarks use to compare with h5py, where each call decodes on its
+    // caller): no helper, so one read never uses two cores.
+    if pool <= 1 {
+        return 0;
+    }
+    let others = if rayon::current_thread_index().is_some() {
+        pool.saturating_sub(1)
+    } else {
+        pool
+    };
+    others.min(items.saturating_sub(1))
+}
+
+/// Run `body` on the calling thread and on up to `helpers` rayon workers at
+/// once, returning when the caller's call has finished and every worker that
+/// started one has too. `body` shares its work out itself (typically by
+/// claiming items from an atomic counter until none are left).
+///
+/// The caller never waits for a worker to *become* free: helpers are queued
+/// on the pool, and one that only gets to run after the caller has finished
+/// returns without calling `body`. So a busy or small pool can only fail to
+/// speed a read up, never hold it back — with `par_iter`, the calling thread
+/// (not a pool worker) handed all the work to the pool and slept, and N
+/// threads reading through a 2-worker pool decoded on 2 cores.
+///
+/// A panic in `body`, on any thread, is resumed on the caller once every
+/// helper that started has stopped.
+pub(crate) fn run_with_helpers(helpers: usize, body: &(dyn Fn() + Sync)) {
+    use std::panic::{AssertUnwindSafe, catch_unwind, resume_unwind};
+    use std::sync::{Arc, Condvar, Mutex, PoisonError};
+
+    if helpers == 0 {
+        body();
+        return;
+    }
+
+    type Body = dyn Fn() + Sync + 'static;
+    struct Shared {
+        /// `body`, its lifetime erased. Only dereferenced by a helper that
+        /// registered in `state` while it was open (see below).
+        body: *const Body,
+        /// (closed, helpers inside `body`).
+        state: Mutex<(bool, usize)>,
+        idle: Condvar,
+        panic: Mutex<Option<Box<dyn core::any::Any + Send>>>,
+    }
+    // SAFETY: `body` points to a `Sync` closure, so calling it from other
+    // threads is allowed; the pointer is only used under the protocol below,
+    // which keeps it from outliving the closure.
+    unsafe impl Send for Shared {}
+    unsafe impl Sync for Shared {}
+
+    fn help(shared: &Shared) {
+        {
+            let mut state = shared.state.lock().unwrap_or_else(PoisonError::into_inner);
+            if state.0 {
+                return;
+            }
+            state.1 += 1;
+        }
+        // SAFETY: registered while open, so the caller of `run_with_helpers`
+        // is still inside it (it closes, then waits until no helper is
+        // registered, before returning), and `body` is alive.
+        let body = unsafe { &*shared.body };
+        if let Err(payload) = catch_unwind(AssertUnwindSafe(body)) {
+            shared
+                .panic
+                .lock()
+                .unwrap_or_else(PoisonError::into_inner)
+                .get_or_insert(payload);
+        }
+        let mut state = shared.state.lock().unwrap_or_else(PoisonError::into_inner);
+        state.1 -= 1;
+        if state.1 == 0 {
+            shared.idle.notify_all();
+        }
+    }
+
+    let body_ptr: *const (dyn Fn() + Sync + '_) = body;
+    // SAFETY: only the lifetime changes (same fat-pointer layout). The
+    // pointer is dereferenced only while this function is running: see
+    // `help` and the wait below.
+    let body_ptr: *const Body = unsafe { core::mem::transmute(body_ptr) };
+    let shared = Arc::new(Shared {
+        body: body_ptr,
+        state: Mutex::new((false, 0)),
+        idle: Condvar::new(),
+        panic: Mutex::new(None),
+    });
+    for _ in 0..helpers {
+        let shared = Arc::clone(&shared);
+        rayon::spawn(move || help(&shared));
+    }
+    let caller = catch_unwind(AssertUnwindSafe(body));
+    {
+        // Close, then wait for the helpers inside `body`; later ones return
+        // at once. This must happen even if `body` panicked on this thread.
+        let mut state = shared.state.lock().unwrap_or_else(PoisonError::into_inner);
+        state.0 = true;
+        while state.1 > 0 {
+            state = shared
+                .idle
+                .wait(state)
+                .unwrap_or_else(PoisonError::into_inner);
+        }
+    }
+    if let Err(payload) = caller {
+        resume_unwind(payload);
+    }
+    let helper_panic = shared
+        .panic
+        .lock()
+        .unwrap_or_else(PoisonError::into_inner)
+        .take();
+    if let Some(payload) = helper_panic {
+        resume_unwind(payload);
+    }
+}
+
 /// Decompress chunks in parallel using lane-partitioned assignment.
 ///
 /// Instead of naive `par_iter`, chunks are deterministically assigned to lanes
@@ -258,6 +384,60 @@ mod tests {
         (file, infos)
     }
 
+    /// Every item is processed exactly once, whatever mix of caller and
+    /// helpers ends up doing it.
+    #[test]
+    fn run_with_helpers_shares_all_work() {
+        use core::sync::atomic::{AtomicUsize, Ordering};
+        for helpers in [0, 1, 3, 16] {
+            let n = 1000;
+            let next = AtomicUsize::new(0);
+            let done: Vec<AtomicUsize> = (0..n).map(|_| AtomicUsize::new(0)).collect();
+            run_with_helpers(helpers, &|| {
+                loop {
+                    let i = next.fetch_add(1, Ordering::Relaxed);
+                    if i >= n {
+                        break;
+                    }
+                    done[i].fetch_add(1, Ordering::Relaxed);
+                }
+            });
+            assert!(done.iter().all(|d| d.load(Ordering::Relaxed) == 1));
+        }
+    }
+
+    /// A panic in the shared body reaches the caller whichever thread it
+    /// happened on, and only after the helpers inside the body have left it
+    /// (they borrow the caller's stack).
+    #[test]
+    fn run_with_helpers_propagates_panics() {
+        use core::sync::atomic::{AtomicUsize, Ordering};
+        use std::panic::{AssertUnwindSafe, catch_unwind};
+        let caller = std::thread::current().id();
+        for panic_on_caller in [true, false] {
+            let inside = AtomicUsize::new(0);
+            let calls = AtomicUsize::new(0);
+            let result = catch_unwind(AssertUnwindSafe(|| {
+                run_with_helpers(4, &|| {
+                    inside.fetch_add(1, Ordering::SeqCst);
+                    calls.fetch_add(1, Ordering::SeqCst);
+                    let on_caller = std::thread::current().id() == caller;
+                    std::thread::sleep(std::time::Duration::from_millis(20));
+                    inside.fetch_sub(1, Ordering::SeqCst);
+                    if on_caller == panic_on_caller {
+                        panic!("boom");
+                    }
+                });
+            }));
+            // A helper may never have run (the pool was slow to start it),
+            // in which case nothing panicked when `panic_on_caller` is false.
+            if panic_on_caller || calls.load(Ordering::SeqCst) > 1 {
+                assert!(result.is_err());
+            }
+            assert_eq!(inside.load(Ordering::SeqCst), 0);
+        }
+    }
+
     /// Every parallel decoder refuses a chunk that decodes short, naming it.
     #[test]
     fn short_decoded_chunk_is_an_error() {

crates/clawhdf5-format/src/partial_read.rs

@@ -18,13 +18,13 @@ use alloc::{format, vec, vec::Vec};
 #[cfg(feature = "std")]
 use std::string as alloc_or_std;
 
-use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks};
+use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks_for_read};
 use crate::data_layout::DataLayout;
 use crate::data_read::extract_selection_from_buffer;
 use crate::dataspace::Dataspace;
 use crate::error::FormatError;
 use crate::filter_pipeline::FilterPipeline;
-use crate::filters::{all_filters_skipped, decompress_chunk_exact};
+use crate::filters::{all_filters_skipped, decompress_chunk_exact_with};
 use crate::selection::Selection;
 
 /// The smallest axis-aligned box containing every selected element, as
@@ -294,17 +294,20 @@ pub fn read_selection(
             btree_address: Some(_),
             ..
         } => {
-            let (chunks, chunk_dims) = list_chunks(
+            let (chunks, chunk_dims) = list_chunks_for_read(
                 file_data,
                 layout,
                 dataspace,
                 elem_size,
+                pipeline,
                 offset_size,
                 length_size,
             )?;
             let rank = dims.len();
             let chunk_shape: Vec<u64> = chunk_dims.iter().map(|&d| d as u64).collect();
             let chunk_bytes = crate::chunked_read::checked_chunk_byte_len(&chunk_dims, elem_size)?;
+            // Chunks are decoded into this thread's reusable buffers.
+            crate::chunked_read::with_scratch(|scratch| -> Result<(), FormatError> {
                 for chunk in &chunks {
                     if chunk.offsets.len() < rank || chunk.address == u64::MAX {
                         continue;
@@ -328,18 +331,17 @@ pub fn read_selection(
                         })?;
                     // Mirrors the full-read path: filter-mask bit i set means
                     // filter i was not applied to this chunk.
-                let decoded;
                     let data: &[u8] = match pipeline {
                         Some(pl) if !all_filters_skipped(pl, chunk.filter_mask) => {
-                        decoded = decompress_chunk_exact(
+                            decompress_chunk_exact_with(
                                 raw,
                                 pl,
                                 chunk_bytes,
                                 elem_size as u32,
                                 chunk.filter_mask,
                                 &chunk.offsets[..rank],
-                        )?;
+                                scratch,
-                        &decoded
+                            )?
                         }
                         _ => raw,
                     };
@@ -353,6 +355,8 @@ pub fn read_selection(
                         elem_size,
                     );
                 }
+                Ok(())
+            })?;
         }
         _ => return Ok(None),
     }

crates/clawhdf5-format/src/superblock_ext.rs

@@ -0,0 +1,812 @@
+//! The superblock extension of a version 2 or 3 superblock, and the
+//! metadata cache image it can point to.
+//!
+//! libhdf5 reads the extension when it opens a file (`H5F__super_read`) and
+//! decodes the messages that configure the file: v1 B-tree "K" values, File
+//! Space Info, and the Metadata Cache Image. A message that does not decode
+//! makes the file fail to open, so [`read_superblock_extension`] decodes and
+//! checks them the way libhdf5 does.
+//!
+//! A metadata cache image (written with `H5Pset_mdc_image_config`) is a
+//! block holding serialized metadata cache entries — object headers, B-tree
+//! nodes, heaps — each with its file address. libhdf5 loads it into its
+//! cache before it reads any other metadata (`H5C__load_cache_image`,
+//! `H5C__reconstruct_cache_contents`), and the entries take the place of
+//! the file's bytes at their addresses: the file itself may hold stale or
+//! no metadata there (in `h5clear_mdc_image.h5` the root group's header is
+//! only in the image). [`CacheImage::apply`] does the same with bytes: it
+//! writes every entry at its address, so every parser reads what libhdf5
+//! reads. It writes into whatever the opener gives it — a private
+//! copy-on-write mapping of the file, or a buffer the opener owns — so the
+//! file is never copied whole.
+
+#[cfg(not(feature = "std"))]
+use alloc::{collections::BTreeSet, vec::Vec};
+#[cfg(feature = "std")]
+use std::collections::BTreeSet;
+
+use crate::error::FormatError;
+use crate::message_type::MessageType;
+use crate::object_header::ObjectHeader;
+use crate::superblock::Superblock;
+
+/// Message type of the File Space Info message.
+const MSG_FSINFO: u16 = 0x0017;
+/// Message type of the Metadata Cache Image message.
+const MSG_MDCI: u16 = 0x0018;
+/// Header message flag: the library did not know the message when it wrote
+/// it back (`H5O_MSG_FLAG_WAS_UNKNOWN`); libhdf5 then ignores its contents.
+const MSG_FLAG_WAS_UNKNOWN: u8 = 0x20;
+
+/// `H5F_FILE_SPACE_PAGE_SIZE_MIN` / `_MAX`.
+const PAGE_SIZE_MIN: u64 = 512;
+const PAGE_SIZE_MAX: u64 = 1024 * 1024 * 1024;
+/// libhdf5's default file space page size, used for a version 0 message.
+const PAGE_SIZE_DEFAULT: u64 = 4096;
+/// Free-space managers whose addresses a persisting version 1 File Space
+/// Info message lists (`H5F_MEM_PAGE_SUPER` .. `H5F_MEM_PAGE_NTYPES`), and
+/// a version 0 one (`H5FD_MEM_SUPER` .. `H5FD_MEM_NTYPES`).
+const FSM_ADDRS_V1: usize = 12;
+const FSM_ADDRS_V0: usize = 6;
+
+/// Metadata cache image block limits (`H5Cimage.c`, `H5ACprivate.h`).
+const MDCI_SIGNATURE: &[u8; 4] = b"MDCI";
+const MDCI_HAVE_RESIZE_STATUS: u8 = 0x01;
+const MDCI_ENTRY_IS_FD_PARENT: u8 = 0x04;
+const MDCI_ENTRY_IS_FD_CHILD: u8 = 0x08;
+/// `H5AC_NTYPES`: entry type ids are below this.
+const MDCI_NTYPES: u8 = 30;
+/// `H5C_RING_NTYPES`.
+const MDCI_RING_NTYPES: u8 = 6;
+/// `H5AC__CACHE_IMAGE__ENTRY_AGEOUT__MAX`.
+const MDCI_AGE_MAX: u8 = 100;
+
+/// A decoded File Space Info message (0x0017), mapped to version 1 as
+/// libhdf5 maps a version 0 one.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct FileSpaceInfo {
+    /// Message version as stored (0 or 1).
+    pub version: u8,
+    /// File space strategy (`H5F_fspace_strategy_t`).
+    pub strategy: u8,
+    /// Whether free space is persisted.
+    pub persist: bool,
+    /// Free-space section threshold.
+    pub threshold: u64,
+    /// File space page size.
+    pub page_size: u64,
+}
+
+/// Where a metadata cache image block is (Metadata Cache Image message,
+/// 0x0018).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct CacheImageLocation {
+    /// Address of the image block.
+    pub address: u64,
+    /// Length of the image block in bytes.
+    pub length: u64,
+}
+
+/// The messages of a superblock extension that libhdf5 decodes at open.
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct SuperblockExtension {
+    /// v1 B-tree "K" values (chunk index, symbol table node, symbol table
+    /// leaf), when the extension overrides the defaults.
+    pub btree_k: Option<(u16, u16, u16)>,
+    /// The File Space Info message.
+    pub file_space_info: Option<FileSpaceInfo>,
+    /// The metadata cache image, when the file has one.
+    pub cache_image: Option<CacheImageLocation>,
+}
+
+fn ext_err(why: &'static str) -> FormatError {
+    FormatError::InvalidSuperblockExtension(why)
+}
+
+const RAN_OFF: &str = "ran off end of input buffer while decoding";
+
+/// A little-endian cursor over one message or block, failing with
+/// `overrun` when it runs off the end.
+struct Cursor<'a> {
+    data: &'a [u8],
+    pos: usize,
+    overrun: FormatError,
+}
+
+impl<'a> Cursor<'a> {
+    fn new(data: &'a [u8], overrun: FormatError) -> Self {
+        Cursor {
+            data,
+            pos: 0,
+            overrun,
+        }
+    }
+
+    fn take(&mut self, n: usize) -> Result<&'a [u8], FormatError> {
+        let end = self
+            .pos
+            .checked_add(n)
+            .filter(|&e| e <= self.data.len())
+            .ok_or_else(|| self.overrun.clone())?;
+        let s = &self.data[self.pos..end];
+        self.pos = end;
+        Ok(s)
+    }
+
+    fn u8(&mut self) -> Result<u8, FormatError> {
+        Ok(self.take(1)?[0])
+    }
+
+    fn uint(&mut self, width: u8) -> Result<u64, FormatError> {
+        let b = self.take(width as usize)?;
+        Ok(b.iter()
+            .rev()
+            .fold(0u64, |acc, &x| (acc << 8) | u64::from(x)))
+    }
+
+    /// An address of `width` bytes; `None` when undefined (all ones).
+    fn addr(&mut self, width: u8) -> Result<Option<u64>, FormatError> {
+        let v = self.uint(width)?;
+        let undef = if width >= 8 {
+            u64::MAX
+        } else {
+            (1u64 << (8 * u32::from(width))) - 1
+        };
+        Ok((v != undef).then_some(v))
+    }
+}
+
+/// Decode and check the superblock extension of `sb`, as libhdf5 does when
+/// it opens the file. `data` is the file from the superblock on, up to the
+/// end of file the superblock records (its end is libhdf5's "eoa").
+///
+/// Returns `Ok(None)` for a superblock without an extension (versions 0
+/// and 1 have none). A message libhdf5 fails to decode, or a cache image
+/// that does not lie inside the file, is an error: libhdf5 refuses to open
+/// such a file (`cve-2020-10810`: a File Space Info message too short for
+/// the free-space manager addresses it announces; `cve-2020-10812`: a cache
+/// image past the end of the file).
+pub fn read_superblock_extension(
+    data: &[u8],
+    sb: &Superblock,
+) -> Result<Option<SuperblockExtension>, FormatError> {
+    let os = sb.offset_size;
+    let ls = sb.length_size;
+    let undef = if os >= 8 {
+        u64::MAX
+    } else {
+        (1u64 << (8 * u32::from(os))) - 1
+    };
+    let Some(addr) = sb.superblock_extension_address.filter(|&a| a != undef) else {
+        return Ok(None);
+    };
+    let addr = usize::try_from(addr).map_err(|_| ext_err("address out of range"))?;
+    let header = ObjectHeader::parse(data, addr, os, ls)?;
+    let eoa = data.len() as u64;
+
+    let mut ext = SuperblockExtension::default();
+    for msg in &header.messages {
+        match msg.msg_type {
+            MessageType::BTreeKValues => {
+                let mut c = Cursor::new(&msg.data, ext_err(RAN_OFF));
+                if c.u8()? != 0 {
+                    return Err(ext_err("bad version number for v1 B-tree 'K' message"));
+                }
+                let chunk = c.uint(2)? as u16;
+                let snode = c.uint(2)? as u16;
+                let leaf = c.uint(2)? as u16;
+                ext.btree_k = Some((chunk, snode, leaf));
+            }
+            MessageType::Unknown(MSG_FSINFO) if msg.flags & MSG_FLAG_WAS_UNKNOWN == 0 => {
+                ext.file_space_info = Some(decode_fsinfo(&msg.data, os, ls)?);
+            }
+            MessageType::Unknown(MSG_MDCI) => {
+                let mut c = Cursor::new(&msg.data, ext_err(RAN_OFF));
+                if c.u8()? != 0 {
+                    return Err(ext_err(
+                        "bad version number for metadata cache image message",
+                    ));
+                }
+                let address = c.addr(os)?;
+                let length = c.uint(ls)?;
+                let Some(address) = address else {
+                    return Err(ext_err("metadata cache image address is undefined"));
+                };
+                if address.checked_add(length).is_none_or(|end| end > eoa) {
+                    return Err(ext_err(
+                        "metadata cache image: address plus size exceeds file eoa",
+                    ));
+                }
+                ext.cache_image = Some(CacheImageLocation { address, length });
+            }
+            _ => {}
+        }
+    }
+    Ok(Some(ext))
+}
+
+/// `H5O__fsinfo_decode` plus the checks `H5F__super_read` makes on it.
+fn decode_fsinfo(data: &[u8], os: u8, ls: u8) -> Result<FileSpaceInfo, FormatError> {
+    let mut c = Cursor::new(data, ext_err(RAN_OFF));
+    let version = c.u8()?;
+    let info = if version == 0 {
+        let old_strategy = c.u8()?;
+        let threshold = c.uint(ls)?;
+        // H5F_file_space_type_t: 1 ALL_PERSIST, 2 ALL, 3 AGGR_VFD, 4 VFD.
+        let (strategy, persist) = match old_strategy {
+            1 => {
+                for _ in 0..FSM_ADDRS_V0 {
+                    c.addr(os)?;
+                }
+                (0, true)
+            }
+            2 => (0, false),
+            3 => (2, false),
+            4 => (3, false),
+            _ => return Err(ext_err("invalid file space strategy")),
+        };
+        FileSpaceInfo {
+            version,
+            strategy,
+            persist,
+            threshold,
+            page_size: PAGE_SIZE_DEFAULT,
+        }
+    } else {
+        if version > 1 {
+            return Err(ext_err("File space info message's version out of bounds"));
+        }
+        let strategy = c.u8()?;
+        let persist = c.u8()? != 0;
+        let threshold = c.uint(ls)?;
+        let page_size = c.uint(ls)?;
+        if page_size == 0 || page_size > PAGE_SIZE_MAX {
+            return Err(ext_err("invalid page size in file space info"));
+        }
+        c.uint(2)?; // page end metadata threshold
+        c.addr(os)?; // EOA before the free-space managers
+        if persist {
+            for _ in 0..FSM_ADDRS_V1 {
+                c.addr(os)?;
+            }
+        }
+        FileSpaceInfo {
+            version,
+            strategy,
+            persist,
+            threshold,
+            page_size,
+        }
+    };
+    if info.page_size < PAGE_SIZE_MIN {
+        return Err(ext_err("file space page size too small"));
+    }
+    Ok(info)
+}
+
+/// One entry of a metadata cache image: `len` bytes at `image_offset` in
+/// the image block, belonging at file address `address`.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct ImageEntry {
+    address: u64,
+    image_offset: usize,
+    len: usize,
+}
+
+/// A decoded metadata cache image: where its block is, and the entries it
+/// holds. [`CacheImage::apply`] writes the entries over a file's bytes.
+///
+/// Only the entry list is kept, never a copy of the file: an opener that
+/// maps the file applies the image to a private copy-on-write mapping, so
+/// only the pages the entries land on are copied.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct CacheImage {
+    location: CacheImageLocation,
+    entries: Vec<ImageEntry>,
+}
+
+/// What an opener must do about a file's metadata cache image.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum CacheImageState {
+    /// The file has no image: its bytes are its metadata.
+    Absent,
+    /// The file has an image that loads: apply it with [`CacheImage::apply`].
+    Loaded(CacheImage),
+    /// The file has an image libhdf5 fails to load. libhdf5 still opens the
+    /// file (the image loads at the first metadata read), and that read
+    /// fails with this error.
+    Unloadable(FormatError),
+}
+
+impl CacheImage {
+    /// Decode the metadata cache image at `location` in `data` (the file
+    /// from the superblock on, up to its recorded end of file). The image is
+    /// checked as libhdf5 checks it (`H5C__decode_cache_image_header`,
+    /// `H5C__reconstruct_cache_entry`): signature and version, the image
+    /// length it records, entry types, rings and ages in range, entry
+    /// addresses inside the file and not repeated, flush-dependency parents
+    /// already in the cache.
+    ///
+    /// One check is stricter than libhdf5's: an entry must end inside the
+    /// file. libhdf5 checks only that it starts there, and serves the rest
+    /// from the image; the images libhdf5 writes never do this (every entry
+    /// lies below the image block, which is written last), and the bytes an
+    /// entry would put past the end of file have nowhere to go in a view of
+    /// the file.
+    ///
+    /// libhdf5 does not verify the block's trailing checksum when it loads
+    /// an image, so neither does this.
+    pub fn decode(
+        data: &[u8],
+        location: CacheImageLocation,
+        sb: &Superblock,
+    ) -> Result<Self, FormatError> {
+        let (offset_size, length_size) = (sb.offset_size, sb.length_size);
+        let bad = FormatError::InvalidCacheImage;
+        let block = image_block(data, location)?;
+        let eoa = data.len() as u64;
+        let mut c = Cursor::new(block, bad(RAN_OFF));
+
+        // Header: signature, version, flags, image data length, entry count.
+        if c.take(4)? != MDCI_SIGNATURE {
+            return Err(bad("bad metadata cache image header signature"));
+        }
+        if c.u8()? != 0 {
+            return Err(bad("bad metadata cache image version"));
+        }
+        if c.u8()? & MDCI_HAVE_RESIZE_STATUS != 0 {
+            return Err(bad("MDC resize status not yet supported"));
+        }
+        if c.uint(length_size)? != location.length {
+            return Err(bad("bad metadata cache image data length"));
+        }
+        let n_entries = c.uint(4)?;
+        if n_entries == 0 {
+            return Err(bad("bad metadata cache entry count"));
+        }
+
+        let mut entries = Vec::new();
+        // What is in libhdf5's cache when it loads the image: the superblock
+        // and the superblock extension's object header (read to find the
+        // image). Each entry's flush-dependency parents are looked up in the
+        // cache as the entry is inserted (`H5C__reconstruct_cache_contents`
+        // searches the index inside the loop that inserts the entries, in
+        // HDF5 1.14.6 and 2.0.0 alike), so a parent must be one of those or
+        // an earlier entry.
+        let mut cached = BTreeSet::new();
+        cached.insert(0);
+        if let Some(ext) = sb.superblock_extension_address {
+            cached.insert(ext);
+        }
+        let mut seen = BTreeSet::new();
+        for _ in 0..n_entries {
+            let type_id = c.u8()?;
+            if type_id >= MDCI_NTYPES {
+                return Err(bad("type id is out of valid range"));
+            }
+            let flags = c.u8()?;
+            if c.u8()? >= MDCI_RING_NTYPES {
+                return Err(bad("ring is out of valid range"));
+            }
+            if c.u8()? > MDCI_AGE_MAX {
+                return Err(bad("entry age is out of policy range"));
+            }
+            let children = c.uint(2)?;
+            // libhdf5 checks the parent flag against the child count only in
+            // debug builds (release builds refuse any entry with children);
+            // the image format's own rule is checked here.
+            if (flags & MDCI_ENTRY_IS_FD_PARENT != 0) != (children > 0) {
+                return Err(bad("flush dependency parent flag and child count disagree"));
+            }
+            c.uint(2)?; // dirty dependency children: reset for a read-only open
+            let parents = c.uint(2)?;
+            if (flags & MDCI_ENTRY_IS_FD_CHILD != 0) != (parents > 0) {
+                return Err(bad("flush dependency child flag and parent count disagree"));
+            }
+            c.uint(4)?; // LRU rank
+            let address = c
+                .addr(offset_size)?
+                .filter(|&a| a < eoa)
+                .ok_or(bad("invalid entry address range"))?;
+            let size = c.uint(length_size)?;
+            if size == 0 {
+                return Err(bad("invalid entry size"));
+            }
+            for _ in 0..parents {
+                let parent = c
+                    .addr(offset_size)?
+                    .ok_or(bad("invalid flush dependency parent offset"))?;
+                if !seen.contains(&parent) && !cached.contains(&parent) {
+                    return Err(bad("fd parent not in cache"));
+                }
+            }
+            let len = usize::try_from(size).map_err(|_| bad(RAN_OFF))?;
+            let image_offset = c.pos;
+            c.take(len)?;
+            if address.checked_add(size).is_none_or(|end| end > eoa) {
+                return Err(bad("entry extends past the end of file"));
+            }
+            if !seen.insert(address) {
+                return Err(bad("duplicate addresses in cache"));
+            }
+            entries.push(ImageEntry {
+                address,
+                image_offset,
+                len,
+            });
+        }
+        Ok(CacheImage { location, entries })
+    }
+
+    /// Where the image block is.
+    pub fn location(&self) -> CacheImageLocation {
+        self.location
+    }
+
+    /// The number of entries in the image.
+    pub fn len(&self) -> usize {
+        self.entries.len()
+    }
+
+    /// Whether the image has no entries (a decoded image always has some).
+    pub fn is_empty(&self) -> bool {
+        self.entries.is_empty()
+    }
+
+    /// The file ranges (address, length) the image's entries replace.
+    pub fn entry_ranges(&self) -> impl Iterator<Item = (u64, usize)> + '_ {
+        self.entries.iter().map(|e| (e.address, e.len))
+    }
+
+    /// The image block in `data`, the bytes [`Self::decode`] read it from.
+    pub fn block<'a>(&self, data: &'a [u8]) -> Result<&'a [u8], FormatError> {
+        image_block(data, self.location)
+    }
+
+    /// Write every entry over `dst`, the file's bytes from the superblock
+    /// on (as long as the `data` the image was decoded from), taking the
+    /// entries from `block` (the image block, see [`Self::block`]). `block`
+    /// must not alias `dst`: an entry may land on the block itself.
+    pub fn apply(&self, block: &[u8], dst: &mut [u8]) -> Result<(), FormatError> {
+        let short = || FormatError::InvalidCacheImage("image applied to the wrong file");
+        for e in &self.entries {
+            let src = block
+                .get(e.image_offset..e.image_offset + e.len)
+                .ok_or_else(short)?;
+            let at = usize::try_from(e.address).map_err(|_| short())?;
+            dst.get_mut(at..at + e.len)
+                .ok_or_else(short)?
+                .copy_from_slice(src);
+        }
+        Ok(())
+    }
+}
+
+fn image_block(data: &[u8], location: CacheImageLocation) -> Result<&[u8], FormatError> {
+    let bad = FormatError::InvalidCacheImage;
+    let start = usize::try_from(location.address).map_err(|_| bad("address out of range"))?;
+    let len = usize::try_from(location.length).map_err(|_| bad("length out of range"))?;
+    start
+        .checked_add(len)
+        .and_then(|end| data.get(start..end))
+        .ok_or(bad("image block extends past the end of the file"))
+}
+
+/// What an opener must do before reading a file's metadata: check the
+/// superblock extension ([`read_superblock_extension`]; an error means
+/// libhdf5 refuses to open the file) and decode any metadata cache image
+/// ([`CacheImage::decode`]). `data` is the file from the superblock on, up
+/// to its recorded end of file.
+pub fn cache_image_state(data: &[u8], sb: &Superblock) -> Result<CacheImageState, FormatError> {
+    match read_superblock_extension(data, sb)? {
+        Some(SuperblockExtension {
+            cache_image: Some(location),
+            ..
+        }) => Ok(match CacheImage::decode(data, location, sb) {
+            Ok(image) => CacheImageState::Loaded(image),
+            Err(e) => CacheImageState::Unloadable(e),
+        }),
+        _ => Ok(CacheImageState::Absent),
+    }
+}
+
+/// [`cache_image_state`] for a reader that holds the file's bytes in a
+/// buffer of its own: check the superblock extension and write any cache
+/// image over `data` in place (only the image block is copied). An image
+/// libhdf5 cannot load is an error here: such a reader has no way to open
+/// the file and fail each object instead.
+pub fn apply_cache_image_in_place(data: &mut [u8], sb: &Superblock) -> Result<(), FormatError> {
+    match cache_image_state(data, sb)? {
+        CacheImageState::Absent => Ok(()),
+        CacheImageState::Unloadable(e) => Err(e),
+        CacheImageState::Loaded(image) => {
+            let block = image.block(data)?.to_vec();
+            image.apply(&block, data)
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The file's bytes with the image at `loc` applied.
+    fn apply_cache_image(
+        data: &[u8],
+        loc: CacheImageLocation,
+        sb: &Superblock,
+    ) -> Result<Vec<u8>, FormatError> {
+        let image = CacheImage::decode(data, loc, sb)?;
+        let mut out = data.to_vec();
+        image.apply(image.block(data)?, &mut out)?;
+        Ok(out)
+    }
+
+    fn sb_v2(ext: u64) -> Superblock {
+        Superblock {
+            version: 2,
+            offset_size: 8,
+            length_size: 8,
+            base_address: 0,
+            eof_address: 0,
+            root_group_address: 0,
+            group_leaf_node_k: None,
+            group_internal_node_k: None,
+            indexed_storage_internal_node_k: None,
+            free_space_address: None,
+            driver_info_address: None,
+            consistency_flags: 0,
+            superblock_extension_address: Some(ext),
+            checksum: None,
+            page_size: None,
+        }
+    }
+
+    /// A file whose superblock extension (a version 1 object header at 48)
+    /// holds the given messages, padded to `len` bytes.
+    fn file_with_ext(messages: &[(u16, &[u8])], len: usize) -> Vec<u8> {
+        let mut body = Vec::new();
+        for (t, d) in messages {
+            let padded = d.len().div_ceil(8) * 8;
+            body.extend_from_slice(&t.to_le_bytes());
+            body.extend_from_slice(&(padded as u16).to_le_bytes());
+            body.extend_from_slice(&[0x14, 0, 0, 0]);
+            body.extend_from_slice(d);
+            body.resize(body.len() + padded - d.len(), 0);
+        }
+        let mut f = vec![0u8; 48];
+        f.push(1);
+        f.push(0);
+        f.extend_from_slice(&(messages.len() as u16).to_le_bytes());
+        f.extend_from_slice(&1u32.to_le_bytes());
+        f.extend_from_slice(&(body.len() as u32).to_le_bytes());
+        f.extend_from_slice(&[0; 4]);
+        f.extend_from_slice(&body);
+        f.resize(len, 0);
+        f
+    }
+
+    fn fsinfo_v1(page_size: u64, persist: bool, n_addrs: usize) -> Vec<u8> {
+        let mut m = vec![1, 1, u8::from(persist)];
+        m.extend_from_slice(&1u64.to_le_bytes());
+        m.extend_from_slice(&page_size.to_le_bytes());
+        m.extend_from_slice(&0u16.to_le_bytes());
+        m.extend_from_slice(&u64::MAX.to_le_bytes());
+        for _ in 0..n_addrs {
+            m.extend_from_slice(&u64::MAX.to_le_bytes());
+        }
+        m
+    }
+
+    fn mdci(address: u64, length: u64) -> Vec<u8> {
+        let mut m = vec![0];
+        m.extend_from_slice(&address.to_le_bytes());
+        m.extend_from_slice(&length.to_le_bytes());
+        m
+    }
+
+    #[test]
+    fn no_extension() {
+        assert_eq!(
+            read_superblock_extension(&[0; 64], &sb_v2(u64::MAX)).unwrap(),
+            None
+        );
+    }
+
+    #[test]
+    fn file_space_info_as_libhdf5_decodes_it() {
+        // What FileWriter::with_page_size writes.
+        let f = file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, false, 0))], 256);
+        let ext = read_superblock_extension(&f, &sb_v2(48)).unwrap().unwrap();
+        assert_eq!(ext.file_space_info.unwrap().page_size, 4096);
+        let f = file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, true, 12))], 512);
+        assert!(read_superblock_extension(&f, &sb_v2(48)).is_ok());
+
+        let refused = |m: Vec<u8>| {
+            let f = file_with_ext(&[(MSG_FSINFO, &m)], 512);
+            read_superblock_extension(&f, &sb_v2(48)).unwrap_err()
+        };
+        // Persisting, but too short for the manager addresses.
+        let mut short = fsinfo_v1(4096, true, 12);
+        short.truncate(short.len() - 8);
+        assert_eq!(refused(short), ext_err(RAN_OFF));
+        assert!(matches!(
+            refused(fsinfo_v1(256, false, 0)),
+            FormatError::InvalidSuperblockExtension(_)
+        ));
+        assert!(matches!(
+            refused(fsinfo_v1(0, false, 0)),
+            FormatError::InvalidSuperblockExtension(_)
+        ));
+        let mut v2 = fsinfo_v1(4096, false, 0);
+        v2[0] = 2;
+        assert!(matches!(
+            refused(v2),
+            FormatError::InvalidSuperblockExtension(_)
+        ));
+        // cve-2020-10810: version 0, strategy ALL_PERSIST, and a message of
+        // 32 bytes that cannot hold the six addresses that follow.
+        let mut v0 = vec![0u8, 1];
+        v0.extend_from_slice(&[0, 1, 0, 0, 0, 0, 0, 0]);
+        v0.resize(32, 0xff);
+        assert_eq!(refused(v0), ext_err(RAN_OFF));
+        // A version 0 message without persistence is fine.
+        let mut v0 = vec![0u8, 2];
+        v0.extend_from_slice(&[0; 8]);
+        let f = file_with_ext(&[(MSG_FSINFO, &v0)], 256);
+        assert!(read_superblock_extension(&f, &sb_v2(48)).is_ok());
+    }
+
+    #[test]
+    fn cache_image_location_must_be_inside_the_file() {
+        let f = file_with_ext(&[(MSG_MDCI, &mdci(128, 64))], 192);
+        let ext = read_superblock_extension(&f, &sb_v2(48)).unwrap().unwrap();
+        assert_eq!(
+            ext.cache_image,
+            Some(CacheImageLocation {
+                address: 128,
+                length: 64
+            })
+        );
+        // cve-2020-10812: 256 MiB at 0x10100 in a 2565-byte file.
+        let f = file_with_ext(&[(MSG_MDCI, &mdci(0x10100, 0x1000_0000))], 2565);
+        assert!(matches!(
+            read_superblock_extension(&f, &sb_v2(48)),
+            Err(FormatError::InvalidSuperblockExtension(_))
+        ));
+        let f = file_with_ext(&[(MSG_MDCI, &mdci(u64::MAX, 8))], 256);
+        assert!(read_superblock_extension(&f, &sb_v2(48)).is_err());
+    }
+
+    /// A cache image block with `entries` of (address, bytes).
+    fn image(entries: &[(u64, &[u8])]) -> Vec<u8> {
+        let with_deps: Vec<_> = entries.iter().map(|&(a, b)| (a, b, 0, None)).collect();
+        image_with_deps(&with_deps)
+    }
+
+    /// A cache image block with `entries` of (address, bytes, flush
+    /// dependency children, flush dependency parent).
+    fn image_with_deps(entries: &[(u64, &[u8], u16, Option<u64>)]) -> Vec<u8> {
+        let mut b = Vec::new();
+        b.extend_from_slice(MDCI_SIGNATURE);
+        b.push(0);
+        b.push(0);
+        b.extend_from_slice(&0u64.to_le_bytes()); // length, patched below
+        b.extend_from_slice(&(entries.len() as u32).to_le_bytes());
+        for &(addr, bytes, children, parent) in entries {
+            let mut flags = 0x02; // in LRU
+            if children > 0 {
+                flags |= MDCI_ENTRY_IS_FD_PARENT;
+            }
+            if parent.is_some() {
+                flags |= MDCI_ENTRY_IS_FD_CHILD;
+            }
+            b.extend_from_slice(&[5, flags, 1, 0]); // type, flags, ring, age
+            b.extend_from_slice(&children.to_le_bytes());
+            b.extend_from_slice(&0u16.to_le_bytes()); // dirty children
+            b.extend_from_slice(&u16::from(parent.is_some()).to_le_bytes());
+            b.extend_from_slice(&0i32.to_le_bytes());
+            b.extend_from_slice(&addr.to_le_bytes());
+            b.extend_from_slice(&(bytes.len() as u64).to_le_bytes());
+            if let Some(p) = parent {
+                b.extend_from_slice(&p.to_le_bytes());
+            }
+            b.extend_from_slice(bytes);
+        }
+        b.extend_from_slice(&[0; 4]); // checksum (not verified, as in libhdf5)
+        let n = b.len() as u64;
+        b[6..14].copy_from_slice(&n.to_le_bytes());
+        b
+    }
+
+    #[test]
+    fn cache_image_entries_replace_the_file_bytes() {
+        let img = image(&[(16, b"HEADER"), (40, b"NODE")]);
+        let mut f = vec![0u8; 64];
+        let at = f.len() as u64;
+        f.extend_from_slice(&img);
+        let loc = CacheImageLocation {
+            address: at,
+            length: img.len() as u64,
+        };
+        let out = apply_cache_image(&f, loc, &sb_v2(u64::MAX)).unwrap();
+        assert_eq!(out.len(), f.len());
+        assert_eq!(&out[16..22], b"HEADER");
+        assert_eq!(&out[40..44], b"NODE");
+        assert_eq!(&out[..16], &f[..16]);
+
+        let bad = |img: Vec<u8>| {
+            let mut f = vec![0u8; 64];
+            f.extend_from_slice(&img);
+            let loc = CacheImageLocation {
+                address: 64,
+                length: img.len() as u64,
+            };
+            apply_cache_image(&f, loc, &sb_v2(u64::MAX)).unwrap_err()
+        };
+        let mut sig = image(&[(16, b"x")]);
+        sig[0] = b'X';
+        assert!(matches!(bad(sig), FormatError::InvalidCacheImage(_)));
+        assert!(matches!(
+            bad(image(&[(16, b"a"), (16, b"b")])),
+            FormatError::InvalidCacheImage("duplicate addresses in cache")
+        ));
+        assert!(matches!(
+            bad(image(&[(1 << 20, b"far")])),
+            FormatError::InvalidCacheImage("invalid entry address range")
+        ));
+        let mut len = image(&[(16, b"x")]);
+        len[6] ^= 1;
+        assert!(matches!(bad(len), FormatError::InvalidCacheImage(_)));
+        // An entry that starts inside the file (64 bytes, then a 60-byte
+        // image) but runs past its end.
+        assert!(matches!(
+            bad(image(&[(123, b"8 bytes!")])),
+            FormatError::InvalidCacheImage("entry extends past the end of file")
+        ));
+        let mut cut = image(&[(16, b"abcdef")]);
+        let n = cut.len() as u64 - 8;
+        cut.truncate(cut.len() - 8);
+        cut[6..14].copy_from_slice(&n.to_le_bytes());
+        assert!(matches!(bad(cut), FormatError::InvalidCacheImage(_)));
+    }
+
+    /// libhdf5 resolves an entry's flush-dependency parents as it inserts
+    /// the entry (`H5C__reconstruct_cache_contents`): a parent must be an
+    /// earlier entry, or the superblock or its extension's object header,
+    /// which are cached before the image loads. A parent listed after its
+    /// child fails ("fd parent not in cache?!?").
+    #[test]
+    fn flush_dependency_parents_must_already_be_cached() {
+        let load = |img: Vec<u8>| {
+            let mut f = vec![0u8; 64];
+            f.extend_from_slice(&img);
+            let loc = CacheImageLocation {
+                address: 64,
+                length: img.len() as u64,
+            };
+            apply_cache_image(&f, loc, &sb_v2(48))
+        };
+        // Parent first, as libhdf5 writes images.
+        assert!(
+            load(image_with_deps(&[
+                (16, b"P", 1, None),
+                (40, b"C", 0, Some(16))
+            ]))
+            .is_ok()
+        );
+        // Child first: libhdf5 does not find the parent.
+        assert_eq!(
+            load(image_with_deps(&[
+                (40, b"C", 0, Some(16)),
+                (16, b"P", 1, None)
+            ]))
+            .unwrap_err(),
+            FormatError::InvalidCacheImage("fd parent not in cache")
+        );
+        // The superblock extension's header (at 48 here) is in the cache.
+        assert!(load(image_with_deps(&[(40, b"C", 0, Some(48))])).is_ok());
+        // An entry cannot be its own parent.
+        assert!(load(image_with_deps(&[(40, b"C", 1, Some(40))])).is_err());
+    }
+}

crates/clawhdf5-format/src/type_builders.rs

@@ -503,6 +503,9 @@ pub struct DatasetBuilder {
     /// `data` field is ignored; instead the global heap blob is built from
     /// these mappings and a VDS layout message is emitted.
     pub(crate) virtual_sources: Option<Vec<VdsMapping>>,
+    /// Track (and index) attribute creation order; `None` follows the
+    /// file's default (`FileWriter::track_order`).
+    pub(crate) track_order: Option<bool>,
     #[cfg(feature = "provenance")]
     pub(crate) provenance: Option<ProvenanceConfig>,
 }
@@ -522,11 +525,22 @@ impl DatasetBuilder {
             compact: false,
             alignment: 0,
             virtual_sources: None,
+            track_order: None,
             #[cfg(feature = "provenance")]
             provenance: None,
         }
     }
 
+    /// Track the creation order of this dataset's attributes, and index it,
+    /// as h5py's `create_dataset(..., track_order=True)` does: libhdf5 (and
+    /// h5py) then list the attributes in the order they were set rather
+    /// than by name. libhdf5 numbers at most 65 535 attributes on an object
+    /// that tracks their order; more is an error when the file is written.
+    pub fn track_order(&mut self, track: bool) -> &mut Self {
+        self.track_order = Some(track);
+        self
+    }
+
     pub fn with_f64_data(&mut self, data: &[f64]) -> &mut Self {
         self.datatype = Some(make_f64_type());
         let mut b = Vec::with_capacity(data.len() * 8);
@@ -986,10 +1000,11 @@ impl GroupBuilder {
         self.attrs.push((name.to_string(), value));
     }
 
-    /// Track the creation order of this group's links, and index it, as
+    /// Track the creation order of this group's links and attributes, and
-    /// h5py's `track_order=True` does: libhdf5 (and h5py) then list the
+    /// index it, as h5py's `track_order=True` does: libhdf5 (and h5py) then
-    /// group's members in the order they were added rather than by name.
+    /// list the group's members, and its attributes, in the order they were
-    /// Applies to links only, not to attributes.
+    /// added rather than by name. libhdf5 numbers at most 65 535 attributes
+    /// on an object that tracks their order.
     pub fn track_order(&mut self, track: bool) -> &mut Self {
         self.track_order = Some(track);
         self

crates/clawhdf5-format/src/vds.rs

@@ -803,7 +803,11 @@ impl<'a, 'r> Sources<'a, 'r> {
             let resolver = self.resolver.ok_or_else(|| {
                 vds_err("external-file virtual dataset sources require a file resolver")
             })?;
-            self.cached_file = Some((String::from(name), resolver(name)?));
+            let mut bytes = resolver(name)?;
+            if let Some(b) = bytes.as_mut() {
+                load_source_file(b)?;
+            }
+            self.cached_file = Some((String::from(name), bytes));
         }
         // An external file is handed over whole; its addresses are relative
         // to its superblock, so skip any user block.
@@ -851,6 +855,23 @@ impl<'a, 'r> Sources<'a, 'r> {
     }
 }
 
+/// Check an external source file's superblock extension as libhdf5 does
+/// when it opens the file, and write any metadata cache image over its
+/// metadata in place: libhdf5 reads the image's entries instead of the
+/// file's own, possibly stale, bytes (`crate::superblock_ext`). A source
+/// file whose image cannot be loaded is an error, as other corrupt source
+/// files are here.
+fn load_source_file(whole: &mut [u8]) -> Result<(), FormatError> {
+    let base = crate::signature::find_signature(whole)?;
+    let sb = crate::superblock::Superblock::parse(&whole[base..], 0)?;
+    // The end of file the superblock records; a truncated source file is
+    // read as before, up to its length.
+    let end = sb
+        .data_end(base as u64, whole.len() as u64)
+        .map_or(whole.len(), |e| base + e as usize);
+    crate::superblock_ext::apply_cache_image_in_place(&mut whole[base..end], &sb)
+}
+
 /// Whether elements of `dt` contain addresses into their own file:
 /// variable-length data (global-heap IDs) or references.
 fn holds_file_addresses(dt: &Datatype) -> bool {

crates/clawhdf5-format/tests/blosc2_alloc_bounds.rs

@@ -0,0 +1,641 @@
+//! Crafted Blosc2 frames and chunks cannot make the decoder allocate out of
+//! proportion to the HDF5 chunk it decodes.
+//!
+//! A frame's header, its offsets chunk and its chunk headers all declare
+//! sizes, and the decoder used to allocate what they declared: a 173-byte
+//! frame whose offsets chunk claimed 2 GiB was decoded in full before any
+//! check failed. Every allocation is now bounded by the output limit (the
+//! HDF5 chunk's size) and the input's length.
+//!
+//! Peak heap use is measured with a counting global allocator; the tests
+//! share it, so each holds `SERIAL` for its whole run.
+#![cfg(feature = "blosc2")]
+
+use std::alloc::{GlobalAlloc, Layout, System};
+use std::sync::Mutex;
+use std::sync::atomic::{AtomicUsize, Ordering};
+
+use clawhdf5_format::filters_blosc2::{blosc2_decompress, blosc2_decompress_chunk};
+
+struct Counting;
+
+static CURRENT: AtomicUsize = AtomicUsize::new(0);
+static PEAK: AtomicUsize = AtomicUsize::new(0);
+static SERIAL: Mutex<()> = Mutex::new(());
+
+unsafe impl GlobalAlloc for Counting {
+    unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
+        let p = unsafe { System.alloc(layout) };
+        if !p.is_null() {
+            let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
+            PEAK.fetch_max(now, Ordering::Relaxed);
+        }
+        p
+    }
+
+    unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
+        let p = unsafe { System.alloc_zeroed(layout) };
+        if !p.is_null() {
+            let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
+            PEAK.fetch_max(now, Ordering::Relaxed);
+        }
+        p
+    }
+
+    unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
+        unsafe { System.dealloc(ptr, layout) };
+        CURRENT.fetch_sub(layout.size(), Ordering::Relaxed);
+    }
+}
+
+#[global_allocator]
+static ALLOC: Counting = Counting;
+
+/// Bytes allocated at the peak of `f`, above what was live when it started.
+fn peak_during<T>(f: impl FnOnce() -> T) -> (T, usize) {
+    let base = CURRENT.load(Ordering::Relaxed);
+    PEAK.store(base, Ordering::Relaxed);
+    let out = f();
+    (out, PEAK.load(Ordering::Relaxed).saturating_sub(base))
+}
+
+/// What decoding one HDF5 chunk of `limit` bytes from `input` may hold at
+/// once: the output, a few blocks of scratch (each no larger than the
+/// output), the offsets table, and the Zstandard decoder's state, which has
+/// a fixed ceiling: a window of at most 128 KiB (or twice the stream) and a
+/// block's table of sequences (up to 98,303 of 12 bytes, 1.2 MB).
+fn bound(limit: usize, input: &[u8]) -> usize {
+    6 * limit + 2 * input.len() + (2 << 20)
+}
+
+fn lock() -> std::sync::MutexGuard<'static, ()> {
+    SERIAL.lock().unwrap_or_else(|e| e.into_inner())
+}
+
+/// A 32-byte (extended) Blosc2 chunk header.
+fn chunk_header(ts: u8, nbytes: i32, blocksize: i32, cbytes: i32, special: u8) -> Vec<u8> {
+    let mut c = vec![5u8, 1, 0x05, ts];
+    for v in [nbytes, blocksize, cbytes] {
+        c.extend_from_slice(&v.to_le_bytes());
+    }
+    c.resize(32, 0);
+    c[31] = special << 4;
+    c
+}
+
+/// A chunk of `nbytes` bytes that repeats one value (special type 3).
+fn repeated(value: &[u8], nbytes: i32, blocksize: i32) -> Vec<u8> {
+    let mut c = chunk_header(
+        value.len() as u8,
+        nbytes,
+        blocksize,
+        32 + value.len() as i32,
+        3,
+    );
+    c.extend_from_slice(value);
+    c
+}
+
+/// A frame offset recording a special chunk of `kind` (1 zeros, 2 NaN).
+fn special_offset(kind: u8) -> [u8; 8] {
+    (((0x80 | kind) as i64) << 56).to_le_bytes()
+}
+
+/// A B2ND metalayer.
+fn nd_meta(shape: &[i64], chunks: &[i32], blocks: &[i32]) -> Vec<u8> {
+    let n = shape.len() as u8;
+    let mut m = vec![0x95, 0, n, 0x90 | n];
+    for s in shape {
+        m.push(0xd3);
+        m.extend_from_slice(&s.to_be_bytes());
+    }
+    for dims in [chunks, blocks] {
+        m.push(0x90 | n);
+        for d in dims {
+            m.push(0xd2);
+            m.extend_from_slice(&d.to_be_bytes());
+        }
+    }
+    m
+}
+
+/// A contiguous frame: header (with a `b2nd` metalayer if given), the data
+/// chunks, then the offsets chunk.
+fn frame(
+    meta: Option<&[u8]>,
+    nbytes: i64,
+    typesize: i32,
+    chunksize: i32,
+    data: &[u8],
+    offsets: &[u8],
+) -> Vec<u8> {
+    let mut h = vec![0u8; 91];
+    h[0] = 0x9e;
+    h[1] = 0xa8;
+    h[2..10].copy_from_slice(b"b2frame\0");
+    h[25] = 2;
+    match meta {
+        Some(m) => {
+            h.extend_from_slice(&[0xde, 0, 1, 0xa4]);
+            h.extend_from_slice(b"b2nd");
+            let at = h.len() as i32 + 5;
+            h.push(0xd2);
+            h.extend_from_slice(&at.to_be_bytes());
+            h.push(0xc6);
+            h.extend_from_slice(&(m.len() as u32).to_be_bytes());
+            h.extend_from_slice(m);
+        }
+        None => h.extend_from_slice(&[0xde, 0, 0]),
+    }
+    let header_len = h.len() as i32;
+    h[11..15].copy_from_slice(&header_len.to_be_bytes());
+    h[30..38].copy_from_slice(&nbytes.to_be_bytes());
+    h[39..47].copy_from_slice(&(data.len() as i64).to_be_bytes());
+    h[48..52].copy_from_slice(&typesize.to_be_bytes());
+    h[58..62].copy_from_slice(&chunksize.to_be_bytes());
+    h.extend_from_slice(data);
+    h.extend_from_slice(offsets);
+    let len = h.len() as u64;
+    h[16..24].copy_from_slice(&len.to_be_bytes());
+    h
+}
+
+/// The frame header's own sizes must not size the offsets chunk: a frame
+/// declaring 32 Mi chunks of 4 bytes, whose offsets chunk (40 bytes) says
+/// "one repeated offset, 256 MiB of them", made the decoder build all
+/// 256 MiB of offsets for a 1 MiB HDF5 chunk and then return 4 bytes.
+#[test]
+fn offsets_chunk_is_bounded_by_the_output_limit() {
+    let _g = lock();
+    let limit = 1 << 20;
+    let offsets_len: i32 = 256 << 20;
+    let nchunks = offsets_len as i64 / 8;
+    let offsets = repeated(&special_offset(1), offsets_len, 64 << 20);
+    let f = frame(None, nchunks * 4, 4, 4, &[], &offsets);
+    let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+    assert!(r.is_err(), "decoded {:?} bytes", r.map(|v| v.len()));
+    assert!(
+        peak <= bound(limit, &f),
+        "peak {peak} bytes for a {}-byte frame",
+        f.len()
+    );
+    // The same frame with a variable chunk size (0): the offsets chunk
+    // alone says how many chunks there are.
+    let f = frame(None, nchunks * 4, 4, 0, &[], &offsets);
+    let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+    assert!(r.is_err());
+    assert!(peak <= bound(limit, &f), "chunksize 0: peak {peak} bytes");
+}
+
+/// A legitimate frame of this shape (one chunk, its offset special) still
+/// decodes.
+#[test]
+fn small_frames_still_decode() {
+    let _g = lock();
+    let offsets = repeated(&special_offset(1), 8, 8);
+    let f = frame(None, 64, 4, 64, &[], &offsets);
+    assert_eq!(blosc2_decompress(&f, 64).unwrap(), vec![0; 64]);
+    let _ = blosc2_decompress_chunk;
+}
+
+/// A chunk that decodes to nothing kept its declared block size (up to
+/// 512 MiB) and allocated two scratch blocks of it: about 1 GiB for a
+/// 20-byte chunk.
+#[test]
+fn empty_chunk_does_not_allocate_its_block_size() {
+    let _g = lock();
+    let mut c = vec![5u8, 1, 0x01, 1];
+    for v in [0i32, 0x1FFF_F000, 20] {
+        c.extend_from_slice(&v.to_le_bytes());
+    }
+    c.resize(20, 0);
+    let (r, peak) = peak_during(|| blosc2_decompress_chunk(&c, 1 << 20));
+    assert_eq!(r.map(|v| v.len()).unwrap_or(0), 0);
+    assert!(
+        peak <= bound(0, &c),
+        "peak {peak} bytes for a 20-byte chunk"
+    );
+    // Inside a frame for a non-empty HDF5 chunk it is an error, not data.
+    let offsets = repeated(&0i64.to_le_bytes(), 8, 8);
+    let f = frame(None, 64, 4, 64, &c, &offsets);
+    let (r, peak) = peak_during(|| blosc2_decompress(&f, 64));
+    assert!(r.is_err(), "decoded {:?}", r.map(|v| v.len()));
+    assert!(peak <= bound(64, &f), "in a frame: peak {peak} bytes");
+}
+
+/// B2ND chunks were decoded whole, padding included, with up to 16x the
+/// HDF5 chunk size as their limit. Blocks are now placed as they are
+/// decoded, so the padding is never held.
+///
+/// Ten dimensions: nine of 3 split into blocks of 2 (padded to 4) and one
+/// of 4, so each chunk is 13x the array. One chunk, stored three ways: as a
+/// NaN chunk in the frame's offsets, as a repeated-value chunk, and as a
+/// chunk of stored (uncompressed) blocks.
+#[test]
+fn b2nd_padding_is_never_held() {
+    let _g = lock();
+    let ts = 4usize;
+    let mut shape = vec![3i64; 9];
+    shape.push(4);
+    let chunks: Vec<i32> = shape.iter().map(|&s| s as i32).collect();
+    let mut blocks = vec![2i32; 9];
+    blocks.push(4);
+    let meta = nd_meta(&shape, &chunks, &blocks);
+    let items: usize = shape.iter().product::<i64>() as usize;
+    let limit = items * ts;
+    let block_bytes = ts * blocks.iter().product::<i32>() as usize;
+    let ext_bytes = ts * 4usize.pow(9) * 4;
+    assert!(ext_bytes > 13 * limit);
+    let offsets = |off: [u8; 8]| repeated(&off, 8, 8);
+
+    let value = 1.5f32.to_le_bytes();
+    let stored = {
+        // Every block stored raw: block k holds the value k.
+        let mut c = chunk_header(4, ext_bytes as i32, block_bytes as i32, 0, 0);
+        c[2] = 0x02 | 0x10; // memcpyed, not split
+        c.truncate(16);
+        for k in 0..ext_bytes / block_bytes {
+            c.extend((k as f32).to_le_bytes().repeat(block_bytes / 4));
+        }
+        let n = c.len() as i32;
+        c[12..16].copy_from_slice(&n.to_le_bytes());
+        c
+    };
+    let cases: Vec<(&str, Vec<u8>)> = vec![
+        (
+            "NaN offset",
+            frame(
+                Some(&meta),
+                ext_bytes as i64,
+                4,
+                ext_bytes as i32,
+                &[],
+                &offsets(special_offset(2)),
+            ),
+        ),
+        (
+            "repeated value",
+            frame(
+                Some(&meta),
+                ext_bytes as i64,
+                4,
+                ext_bytes as i32,
+                &repeated(&value, ext_bytes as i32, block_bytes as i32),
+                &offsets(0i64.to_le_bytes()),
+            ),
+        ),
+        (
+            "stored blocks",
+            frame(
+                Some(&meta),
+                ext_bytes as i64,
+                4,
+                ext_bytes as i32,
+                &stored,
+                &offsets(0i64.to_le_bytes()),
+            ),
+        ),
+    ];
+    for (name, f) in cases {
+        let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+        let out = r.unwrap_or_else(|e| panic!("{name}: {e}"));
+        assert_eq!(out.len(), limit, "{name}");
+        match name {
+            "NaN offset" => assert!(
+                out.chunks(4)
+                    .all(|v| f32::from_le_bytes(v.try_into().unwrap()).is_nan())
+            ),
+            "repeated value" => assert!(out.chunks(4).all(|v| v == value)),
+            _ => {
+                // Element (i0..i9) lies in block (i0/2, .., i8/2), numbered
+                // in C order over a 2x..x2x1 grid of blocks.
+                let mut idx = [0usize; 10];
+                for (e, v) in out.chunks(4).enumerate() {
+                    let mut n = e;
+                    for d in (0..10).rev() {
+                        idx[d] = n % shape[d] as usize;
+                        n /= shape[d] as usize;
+                    }
+                    let k = idx[..9].iter().fold(0, |k, &i| k * 2 + i / 2);
+                    assert_eq!(
+                        f32::from_le_bytes(v.try_into().unwrap()),
+                        k as f32,
+                        "{name} {e}"
+                    );
+                }
+            }
+        }
+        assert!(
+            peak <= bound(limit, &f),
+            "{name}: peak {peak} bytes for a {limit}-byte chunk ({}-byte frame)",
+            f.len()
+        );
+    }
+}
+
+/// A B2ND chunk larger than the array (here 16x, the old cap) is refused,
+/// or at least never allocated.
+#[test]
+fn b2nd_chunk_larger_than_the_array_is_not_allocated() {
+    let _g = lock();
+    let limit = 1 << 20;
+    let c = 16 * limit as i32;
+    let meta = nd_meta(&[limit as i64], &[c], &[c]);
+    let offsets = repeated(&special_offset(1), 8, 8);
+    let f = frame(Some(&meta), c as i64, 1, c, &[], &offsets);
+    let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+    assert!(
+        peak <= bound(limit, &f),
+        "peak {peak} bytes ({:?})",
+        r.map(|v| v.len())
+    );
+}
+
+/// ruzstd reserves a frame's declared window (up to 100 MiB) before it
+/// decodes a frame with a decoder it has used before: a Blosc2 chunk of
+/// two 16-byte Zstandard streams, each declaring a 96 MiB window,
+/// allocated 96 MiB. c-blosc2 compresses each block with its size known,
+/// so its windows never exceed the block.
+#[test]
+fn zstd_window_is_bounded_by_the_output() {
+    let _g = lock();
+    let mut z = 0xfd2f_b528u32.to_le_bytes().to_vec();
+    // No single segment, no checksum; window 2^26 + 4/8 of it = 96 MiB.
+    z.extend_from_slice(&[0x00, (16 << 3) | 4]);
+    // One raw block, last, of 16 bytes.
+    let h = 1 | (16 << 3);
+    z.extend_from_slice(&[h as u8, (h >> 8) as u8, 0]);
+    z.extend_from_slice(&[7; 16]);
+    // Two blocks of 16 bytes, one stream each (not split), Zstandard
+    // (codec 4).
+    let chunk = |z: &[u8]| {
+        let mut c = vec![5u8, 1, 0x10 | (4 << 5), 1];
+        for v in [32i32, 16, 0] {
+            c.extend_from_slice(&v.to_le_bytes());
+        }
+        let first = 24 + 4 + z.len();
+        c.extend_from_slice(&24i32.to_le_bytes());
+        c.extend_from_slice(&(first as i32).to_le_bytes());
+        for _ in 0..2 {
+            c.extend_from_slice(&(z.len() as i32).to_le_bytes());
+            c.extend_from_slice(z);
+        }
+        let n = c.len() as i32;
+        c[12..16].copy_from_slice(&n.to_le_bytes());
+        c
+    };
+    let c = chunk(&z);
+    let (r, peak) = peak_during(|| blosc2_decompress_chunk(&c, 32));
+    assert!(peak <= bound(32, &c), "peak {peak} bytes ({r:?})");
+    assert!(r.is_err(), "{r:?}");
+    // The same streams with a window they can use read.
+    z[5] = 0;
+    assert_eq!(
+        blosc2_decompress_chunk(&chunk(&z), 32).unwrap(),
+        vec![7; 32]
+    );
+}
+
+/// xorshift64*: deterministic, so a failure reproduces.
+struct Rng(u64);
+
+impl Rng {
+    fn next(&mut self) -> u64 {
+        let mut x = self.0;
+        x ^= x >> 12;
+        x ^= x << 25;
+        x ^= x >> 27;
+        self.0 = x;
+        x.wrapping_mul(0x2545_F491_4F6C_DD1D)
+    }
+
+    fn below(&mut self, n: usize) -> usize {
+        (self.next() % n.max(1) as u64) as usize
+    }
+
+    /// A size that tends to the edges: small, a power of two, huge.
+    fn size(&mut self) -> i64 {
+        match self.below(6) {
+            0 => self.below(64) as i64,
+            1 => 1 << self.below(31),
+            2 => i32::MAX as i64 - self.below(4096) as i64,
+            3 => (1i64 << self.below(62)) + self.below(8) as i64,
+            4 => MAX_BLOCK - self.below(3) as i64,
+            _ => self.next() as i32 as i64,
+        }
+    }
+}
+
+const MAX_BLOCK: i64 = 536_866_816;
+
+/// One to four edits: bytes, or a size field written little-endian (chunk
+/// headers) or big-endian (frame headers), most often at a header's size
+/// fields.
+fn mutate(rng: &mut Rng, seed: &[u8], data_at: usize) -> Vec<u8> {
+    let mut v = seed.to_vec();
+    for _ in 0..1 + rng.below(4) {
+        let len = v.len();
+        if len < 16 {
+            v.push(rng.next() as u8);
+            continue;
+        }
+        match rng.below(8) {
+            0 => {
+                let i = rng.below(len);
+                v[i] ^= 1 << rng.below(8);
+            }
+            1 => {
+                let i = rng.below(len);
+                v[i] = rng.next() as u8;
+            }
+            2 => {
+                // Frame header: nbytes, cbytes (i64), typesize, chunksize.
+                let x = rng.size();
+                match rng.below(4) {
+                    0 if len >= 38 => v[30..38].copy_from_slice(&x.to_be_bytes()),
+                    1 if len >= 47 => v[39..47].copy_from_slice(&x.to_be_bytes()),
+                    2 if len >= 52 => v[48..52].copy_from_slice(&(x as i32).to_be_bytes()),
+                    _ if len >= 62 => v[58..62].copy_from_slice(&(x as i32).to_be_bytes()),
+                    _ => {}
+                }
+            }
+            3 | 4 => {
+                // A chunk header's nbytes, blocksize or cbytes: in the first
+                // data chunk, or anywhere (the offsets chunk comes last).
+                let at = if rng.below(2) == 0 && data_at + 16 <= len {
+                    data_at + 4 * (1 + rng.below(3))
+                } else {
+                    rng.below(len - 3)
+                };
+                let x = rng.size() as i32;
+                v[at..at + 4].copy_from_slice(&x.to_le_bytes());
+            }
+            5 => v.truncate(rng.below(len)),
+            6 => {
+                let at = rng.below(len);
+                v[at] = [0x10, 0x20, 0x30, 0x40, 0x05, 0x07, 0x02][rng.below(7)];
+            }
+            _ => {
+                let i = rng.below(len - 3);
+                let x = rng.size() as i32;
+                v[i..i + 4].copy_from_slice(&x.to_be_bytes());
+            }
+        }
+    }
+    v
+}
+
+/// Every fixture frame that decodes, with its decoded size.
+fn seeds() -> Vec<(Vec<u8>, usize)> {
+    let dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/blosc2");
+    let mut v = Vec::new();
+    for e in std::fs::read_dir(dir).unwrap() {
+        let p = e.unwrap().path();
+        if p.extension().is_some_and(|x| x == "b2f")
+            && let Ok(out) = std::fs::read(p.with_extension("out"))
+        {
+            v.push((std::fs::read(&p).unwrap(), out.len()));
+        }
+    }
+    v.sort();
+    assert!(v.len() >= 20, "fixtures missing");
+    v
+}
+
+fn header_len(frame: &[u8]) -> usize {
+    i32::from_be_bytes(frame[11..15].try_into().unwrap()) as usize
+}
+
+/// Mutated fixture frames, decoded with their HDF5 chunk size as the
+/// limit, and their first chunks on their own: whatever they declare, no
+/// decode holds more than a small multiple of the output and the input.
+#[test]
+fn fuzzed_frames_and_chunks_stay_within_the_allocation_bound() {
+    let _g = lock();
+    let seeds = seeds();
+    let mut rng = Rng(0xb2a1);
+    let mut worst = (0.0f64, String::new());
+    for i in 0..20_000 {
+        let (seed, limit) = &seeds[rng.below(seeds.len())];
+        let f = mutate(&mut rng, seed, header_len(seed));
+        let (r, peak) = peak_during(|| blosc2_decompress(&f, *limit));
+        if let Ok(out) = &r {
+            assert!(out.len() <= *limit, "iteration {i}: output past the limit");
+        }
+        assert!(
+            peak <= bound(*limit, &f),
+            "iteration {i}: peak {peak} bytes for a {limit}-byte chunk from {} bytes ({:?})",
+            f.len(),
+            r.map(|v| v.len())
+        );
+        let ratio = peak as f64 / bound(*limit, &f) as f64;
+        if ratio > worst.0 {
+            worst = (
+                ratio,
+                format!(
+                    "frame iteration {i}: peak {peak}, limit {limit}, input {}",
+                    f.len()
+                ),
+            );
+        }
+    }
+    for i in 0..20_000 {
+        let (seed, _) = &seeds[rng.below(seeds.len())];
+        let at = header_len(seed);
+        let chunk = &seed[at..];
+        let c = mutate(&mut rng, chunk, 0);
+        let limit = 1 << 16;
+        let (r, peak) = peak_during(|| blosc2_decompress_chunk(&c, limit));
+        assert!(
+            peak <= bound(limit, &c),
+            "chunk iteration {i}: peak {peak} bytes from {} bytes ({:?})",
+            c.len(),
+            r.map(|v| v.len())
+        );
+    }
+    eprintln!("worst peak / bound: {:.2} ({})", worst.0, worst.1);
+}
+
+/// Frames built from random header sizes, offsets chunks and B2ND shapes
+/// (chunk and block shapes that pad, special and repeated-value chunks).
+#[test]
+fn random_frames_stay_within_the_allocation_bound() {
+    let _g = lock();
+    let mut rng = Rng(0xb2a2);
+    for i in 0..5_000 {
+        let ts = [1usize, 2, 4, 8][rng.below(4)];
+        let ndim = 1 + rng.below(8);
+        let mut shape = Vec::new();
+        let mut chunks = Vec::new();
+        let mut blocks = Vec::new();
+        for _ in 0..ndim {
+            let s = 1 + rng.below(if ndim > 3 { 4 } else { 40 });
+            let c = if rng.below(8) == 0 {
+                s * (1 + rng.below(4))
+            } else {
+                1 + rng.below(s)
+            };
+            let b = 1 + rng.below(c);
+            shape.push(s as i64);
+            chunks.push(c as i32);
+            blocks.push(b as i32);
+        }
+        let items: usize = shape.iter().product::<i64>() as usize;
+        let limit = items * ts;
+        let meta = nd_meta(&shape, &chunks, &blocks);
+        let ext: usize = ts
+            * chunks
+                .iter()
+                .zip(&blocks)
+                .map(|(&c, &b)| (c as usize).div_ceil(b as usize) * b as usize)
+                .product::<usize>();
+        let nchunks: usize = shape
+            .iter()
+            .zip(&chunks)
+            .map(|(&s, &c)| (s as usize).div_ceil(c as usize))
+            .product();
+        let block_bytes = ts * blocks.iter().product::<i32>() as usize;
+        let chunksize = if rng.below(4) == 0 {
+            rng.size()
+        } else {
+            ext as i64
+        };
+        let nbytes = if rng.below(4) == 0 {
+            rng.size()
+        } else {
+            (nchunks * ext) as i64
+        };
+        let off_n = if rng.below(4) == 0 {
+            rng.size() as i32
+        } else {
+            8 * nchunks as i32
+        };
+        let (data, off) = match rng.below(3) {
+            0 => (Vec::new(), special_offset(1 + rng.below(2) as u8)),
+            _ => {
+                let bs = if rng.below(4) == 0 {
+                    rng.size() as i32
+                } else {
+                    block_bytes as i32
+                };
+                let value: Vec<u8> = (0..ts).map(|_| rng.next() as u8).collect();
+                let n = if rng.below(4) == 0 {
+                    rng.size() as i32
+                } else {
+                    ext as i32
+                };
+                (repeated(&value, n, bs), 0i64.to_le_bytes())
+            }
+        };
+        let offsets = repeated(&off, off_n, off_n.clamp(1, 8));
+        let meta = (rng.below(4) != 0).then_some(meta.as_slice());
+        let f = frame(meta, nbytes, ts as i32, chunksize as i32, &data, &offsets);
+        let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
+        assert!(
+            peak <= bound(limit, &f),
+            "iteration {i}: peak {peak} bytes for a {limit}-byte chunk ({:?}, shape {shape:?} \
+             chunks {chunks:?} blocks {blocks:?})",
+            r.map(|v| v.len())
+        );
+    }
+}

crates/clawhdf5-format/tests/fixtures/blosc2/bytedelta.err

@@ -0,0 +1 @@
+filter 35

crates/clawhdf5-format/tests/fixtures/blosc2/generate.py

@@ -0,0 +1,129 @@
+"""Generate the Blosc2 frames the `filters_blosc2` unit tests decode.
+
+Each case is `<name>.b2f` (a Blosc2 contiguous frame, what the HDF5 Blosc2
+filter stores per chunk) and `<name>.out` (what decoding it must give: the
+first chunk of a plain frame, or the whole array in C order for a B2ND
+frame), or `<name>.err` (a frame clawhdf5 must refuse; the file holds a word
+the error message must contain).
+
+These cover what files written by h5py + hdf5plugin never contain, but a
+Blosc2 frame may: special chunks (repeated value, NaN, uninitialised), the
+delta filter over many blocks and odd type sizes, bit shuffle of blocks that
+are not a multiple of 8 elements, shuffle with a byte-group size, forced
+stream splitting, multi-chunk B2ND arrays with padded edge chunks and a
+chunk of zeros, and features clawhdf5 refuses (dictionaries, registered
+filters).
+
+Written with python-blosc2 4.13.1 (c-blosc2 3.3.4) in a scratch venv
+(`pip install blosc2`). Re-run only to regenerate:
+
+    python generate.py <this directory>
+"""
+import os
+import sys
+
+import blosc2
+import numpy as np
+
+out = sys.argv[1]
+
+
+def save(name, frame, expected):
+    with open(os.path.join(out, name + ".b2f"), "wb") as f:
+        f.write(frame)
+    with open(os.path.join(out, name + ".out"), "wb") as f:
+        f.write(expected)
+
+
+def save_err(name, frame, word):
+    with open(os.path.join(out, name + ".b2f"), "wb") as f:
+        f.write(frame)
+    with open(os.path.join(out, name + ".err"), "w") as f:
+        f.write(word)
+
+
+def plain(data, **cparams):
+    """A one-chunk super-chunk frame of `data`, as hdf5-blosc2 writes."""
+    data = np.ascontiguousarray(data)
+    cp = blosc2.CParams(typesize=data.dtype.itemsize, **cparams)
+    sc = blosc2.SChunk(chunksize=data.nbytes, cparams=cp)
+    sc.append_data(data)
+    return sc.to_cframe(), data.tobytes()
+
+
+def special(nitems, dtype, kind, value=None):
+    dt = np.dtype(dtype)
+    sc = blosc2.SChunk(chunksize=nitems * dt.itemsize,
+                       cparams=blosc2.CParams(typesize=dt.itemsize))
+    sc.fill_special(nitems, kind, value)
+    return sc.to_cframe()
+
+
+# Special chunks. A repeated value stays in the frame as a 33+ byte chunk;
+# NaN and uninitialised chunks become special offsets.
+save("value_i4", special(300, "<i4", blosc2.SpecialValue.VALUE, 123456),
+     np.full(300, 123456, "<i4").tobytes())
+save("value_f8", special(250, "<f8", blosc2.SpecialValue.VALUE, -2.5),
+     np.full(250, -2.5, "<f8").tobytes())
+save("nan_f4", special(500, "<f4", blosc2.SpecialValue.NAN),
+     np.full(500, np.nan, "<f4").tobytes())
+save("nan_f8", special(300, "<f8", blosc2.SpecialValue.NAN),
+     np.full(300, np.nan, "<f8").tobytes())
+save("zero_u2", special(2000, "<u2", blosc2.SpecialValue.ZERO), bytes(4000))
+# Uninitialised values: libhdf5 would hand back whatever memory it had;
+# clawhdf5 returns zeros.
+save("uninit_i8", special(64, "<i8", blosc2.SpecialValue.UNINIT), bytes(512))
+
+rng = np.random.default_rng(11)
+ramp = lambda n, dt: ((np.arange(n) * 7) % 1000 + rng.integers(0, 3, n)).astype(dt)
+# Slowly varying: what the delta filter is for (noise would be stored raw).
+smooth = lambda n, dt: (np.arange(n) // 3 + 1000).astype(dt)
+
+# Delta over many blocks, for type sizes 1, 2, 4, 8, 3 (bytes) and 16 (u64
+# pairs).
+for dt, n, codec in [("<u1", 2000, blosc2.Codec.LZ4), ("<i2", 1000, blosc2.Codec.BLOSCLZ),
+                     ("<i4", 700, blosc2.Codec.LZ4), ("<u8", 400, blosc2.Codec.BLOSCLZ)]:
+    save(f"delta_{np.dtype(dt).name}_{codec.name.lower()}",
+         *plain(smooth(n, dt), codec=codec, blocksize=256,
+                filters=[blosc2.Filter.DELTA], filters_meta=[0]))
+rec3 = np.frombuffer(smooth(3 * 300, "<u1").tobytes(), dtype="V3")
+save("delta_v3", *plain(rec3, blocksize=300, filters=[blosc2.Filter.DELTA], filters_meta=[0]))
+rec16 = np.frombuffer(smooth(2 * 200, "<u8").tobytes(), dtype="V16")
+save("delta_shuffle_v16", *plain(rec16, blocksize=512,
+                                 filters=[blosc2.Filter.DELTA, blosc2.Filter.SHUFFLE],
+                                 filters_meta=[0, 0]))
+
+# Bit shuffle of blocks whose element count is not a multiple of 8 (44-byte
+# blocks of 4-byte elements: 8 transposed, 3 copied).
+save("bitshuffle_odd_blocks", *plain(ramp(500, "<i4"), codec=blosc2.Codec.ZSTD, blocksize=44,
+                                     filters=[blosc2.Filter.BITSHUFFLE], filters_meta=[0]))
+# Shuffle in groups of 2 bytes of an 8-byte type (filters_meta).
+save("shuffle_meta2", *plain(ramp(500, "<i8"), codec=blosc2.Codec.ZLIB,
+                             filters=[blosc2.Filter.SHUFFLE], filters_meta=[2]))
+# Streams split per byte, and never split.
+save("always_split", *plain(ramp(1000, "<f4"), codec=blosc2.Codec.LZ4HC,
+                            splitmode=blosc2.SplitMode.ALWAYS_SPLIT))
+save("never_split", *plain(ramp(1500, "<u2"), codec=blosc2.Codec.ZSTD,
+                           splitmode=blosc2.SplitMode.NEVER_SPLIT))
+
+# B2ND arrays of several chunks whose edge chunks and blocks are padded, and
+# one whose middle chunk is all zeros (a special offset).
+for name, shape, chunks, blocks, dt in [
+    ("b2nd_2d", (37, 29), (10, 16), (4, 6), "<i4"),
+    ("b2nd_3d", (9, 10, 7), (4, 5, 3), (3, 2, 2), "<f4"),
+    ("b2nd_4d", (5, 7, 5, 6), (3, 2, 5, 4), (2, 2, 3, 3), "<u2"),
+]:
+    a = ramp(int(np.prod(shape)), dt).reshape(shape)
+    arr = blosc2.asarray(a, chunks=chunks, blocks=blocks)
+    save(name, arr.to_cframe(), a.tobytes())
+a = ramp(60 * 20, "<i2").reshape(60, 20)
+a[20:40, :] = 0
+arr = blosc2.asarray(a, chunks=(20, 20), blocks=(8, 16))
+save("b2nd_zero_chunk", arr.to_cframe(), a.tobytes())
+
+# Refused: a dictionary, and a registered filter (bytedelta).
+frame, _ = plain(ramp(4000, "<i4"), codec=blosc2.Codec.ZSTD, use_dict=True, blocksize=2048)
+save_err("zstd_dict", frame, "dictionar")
+frame, _ = plain(ramp(500, "<i4"), filters=[blosc2.Filter.SHUFFLE, blosc2.Filter.BYTEDELTA],
+                 filters_meta=[0, 4])
+save_err("bytedelta", frame, "filter 35")

crates/clawhdf5-format/tests/fixtures/blosc2/zstd_dict.err

@@ -0,0 +1 @@
+dictionar

crates/clawhdf5-io/src/async_read.rs

@@ -288,6 +288,12 @@ impl AsyncHDF5File {
         // superblock records, as libhdf5 does.
         let end = superblock.data_end(user_block as u64, whole_len)?;
         data.truncate(end as usize);
+        // Check the superblock extension as libhdf5 does at open, and write
+        // any metadata cache image over the file's metadata (libhdf5 reads
+        // the image's entries instead of the file's own, possibly stale,
+        // bytes). An image libhdf5 cannot load is refused: this reader has
+        // no way to open the file and fail each object instead.
+        clawhdf5_format::superblock_ext::apply_cache_image_in_place(&mut data, &superblock)?;
         Ok(Self { data, superblock })
     }
 
@@ -430,6 +436,43 @@ mod tests {
         fw.finish().unwrap()
     }
 
+    /// libhdf5's `h5clear_mdc_image.h5` (see the clawhdf5 crate's
+    /// `tests/metadata_cache_image.rs`): the root group's header exists
+    /// only in the file's metadata cache image.
+    fn cache_image_fixture() -> Vec<u8> {
+        std::fs::read(concat!(
+            env!("CARGO_MANIFEST_DIR"),
+            "/../clawhdf5/tests/fixtures/h5clear_mdc_image.h5"
+        ))
+        .unwrap()
+    }
+
+    #[tokio::test]
+    async fn reads_through_a_metadata_cache_image() {
+        let bytes = cache_image_fixture();
+        let file = AsyncHDF5File::from_bytes(bytes.clone()).unwrap();
+        let info = file.read_dataset_raw("DSET").await.unwrap();
+        assert_eq!(info.shape, [50, 100]);
+        let values: Vec<i32> = info
+            .raw
+            .as_chunks::<4>()
+            .0
+            .iter()
+            .map(|&b| i32::from_le_bytes(b))
+            .collect();
+        let expected: Vec<i32> = (0..50).flat_map(|i| (0..100).map(move |j| i * j)).collect();
+        assert_eq!(values, expected);
+
+        // An image libhdf5 cannot load is refused at open.
+        let mut bad = bytes;
+        let at = bad.windows(4).position(|w| w == b"MDCI").unwrap();
+        bad[at] = b'X';
+        assert!(matches!(
+            AsyncHDF5File::from_bytes(bad),
+            Err(AsyncHDF5Error::Format(FormatError::InvalidCacheImage(_)))
+        ));
+    }
+
     // --- AsyncMemoryReader tests ---
 
     #[tokio::test]

crates/clawhdf5-io/src/lib.rs

@@ -41,6 +41,65 @@ pub trait HDF5Read {
     fn is_empty(&self) -> bool {
         self.as_bytes().is_empty()
     }
+
+    /// A private, writable copy of [`Self::as_bytes`]: writes to it stay in
+    /// this process and never reach the underlying storage.
+    ///
+    /// Readers use it to lay a file's metadata cache image over the file's
+    /// own metadata. The default copies the bytes; a memory-mapped reader
+    /// returns a copy-on-write mapping instead, so only the pages written to
+    /// are copied and the rest stay shared with the page cache.
+    fn private_copy(&self) -> io::Result<PrivateCopy> {
+        Ok(PrivateCopy::Owned(self.as_bytes().to_vec()))
+    }
+}
+
+/// A private, writable copy of a file's bytes (see
+/// [`HDF5Read::private_copy`]).
+pub enum PrivateCopy {
+    /// The bytes copied onto the heap.
+    Owned(Vec<u8>),
+    /// A copy-on-write mapping of the file: pages are copied only when
+    /// written to.
+    #[cfg(feature = "mmap")]
+    Mapped(memmap2::MmapMut),
+}
+
+impl PrivateCopy {
+    /// Whether this is a copy-on-write mapping rather than a heap copy.
+    pub fn is_mapped(&self) -> bool {
+        !matches!(self, PrivateCopy::Owned(_))
+    }
+}
+
+impl std::fmt::Debug for PrivateCopy {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.debug_struct("PrivateCopy")
+            .field("len", &self.len())
+            .field("mapped", &self.is_mapped())
+            .finish()
+    }
+}
+
+impl std::ops::Deref for PrivateCopy {
+    type Target = [u8];
+    fn deref(&self) -> &[u8] {
+        match self {
+            PrivateCopy::Owned(v) => v,
+            #[cfg(feature = "mmap")]
+            PrivateCopy::Mapped(m) => m,
+        }
+    }
+}
+
+impl std::ops::DerefMut for PrivateCopy {
+    fn deref_mut(&mut self) -> &mut [u8] {
+        match self {
+            PrivateCopy::Owned(v) => v,
+            #[cfg(feature = "mmap")]
+            PrivateCopy::Mapped(m) => m,
+        }
+    }
 }
 
 /// Read-write access to HDF5 data.

crates/clawhdf5-io/src/mmap.rs

@@ -87,6 +87,19 @@ impl HDF5Read for MmapReader {
     fn as_bytes(&self) -> &[u8] {
         &self.mmap
     }
+
+    /// A private copy-on-write mapping of the file (`MAP_PRIVATE`): only the
+    /// pages written to are copied.
+    fn private_copy(&self) -> io::Result<crate::PrivateCopy> {
+        if self.mmap.is_empty() {
+            return Ok(crate::PrivateCopy::Owned(Vec::new()));
+        }
+        // SAFETY: as for `open`: the caller keeps the file from being
+        // modified while the mapping is alive. Writes to a private mapping
+        // never reach the file.
+        let map = unsafe { memmap2::MmapOptions::new().map_copy(&self._file)? };
+        Ok(crate::PrivateCopy::Mapped(map))
+    }
 }
 
 /// Writable memory-mapped file for read-write HDF5 access.
@@ -218,6 +231,22 @@ mod tests {
         fs::remove_file(&path).ok();
     }
 
+    #[test]
+    fn private_copy_is_a_copy_on_write_mapping() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("cow.bin");
+        fs::write(&path, [1u8, 2, 3, 4]).unwrap();
+        let reader = MmapReader::open(&path).unwrap();
+        let mut copy = reader.private_copy().unwrap();
+        assert!(copy.is_mapped());
+        copy[1] = 99;
+        assert_eq!(&copy[..], &[1, 99, 3, 4]);
+        // Neither the reader's mapping nor the file sees the write.
+        assert_eq!(reader.as_bytes(), &[1, 2, 3, 4]);
+        drop(copy);
+        assert_eq!(fs::read(&path).unwrap(), [1, 2, 3, 4]);
+    }
+
     #[test]
     fn mmap_reader_read_at() {
         let dir = std::env::temp_dir();

crates/clawhdf5-io/src/mpi_vol.rs

@@ -191,7 +191,10 @@ fn mpi_collective_read(vol: &MpiVol, location: &str, path: &str) -> Result<Vec<u
     let mut len_buf = [0usize; 1];
 
     if rank == 0 {
-        let file = std::fs::read(location).map_err(VolError::Io)?;
+        let mut file = std::fs::read(location).map_err(VolError::Io)?;
+        // Checked as libhdf5 checks a file at open, with any metadata cache
+        // image written over the metadata (see `vol::load_hdf5`).
+        crate::vol::load_hdf5(&mut file)?;
         // From the superblock to the recorded end of file; truncated files
         // are refused.
         let (bytes, sb) = crate::vol::hdf5_view(&file)?;

crates/clawhdf5-io/src/prefetch.rs

@@ -195,6 +195,10 @@ impl<R: HDF5Read> HDF5Read for PrefetchReader<R> {
     fn as_bytes(&self) -> &[u8] {
         self.inner.as_bytes()
     }
+
+    fn private_copy(&self) -> std::io::Result<crate::PrivateCopy> {
+        self.inner.private_copy()
+    }
 }
 
 // ---------------------------------------------------------------------------

crates/clawhdf5-io/src/vol.rs

@@ -208,6 +208,9 @@ pub trait VirtualObjectLayer: Send + Sync {
 pub struct NativeVol {
     data: Option<Vec<u8>>,
     location: Option<String>,
+    /// Why the bytes given to [`NativeVol::from_bytes`] cannot be read
+    /// (what `open` would have refused them for).
+    load_error: Option<String>,
 }
 
 /// The HDF5 bytes of a whole file and its superblock: from the superblock
@@ -228,12 +231,35 @@ pub(crate) fn hdf5_view(
     Ok((&data[..end as usize], sb))
 }
 
+/// Check a whole file as libhdf5 does when it opens it ([`hdf5_view`], and
+/// the superblock extension), and write any metadata cache image over the
+/// file's metadata in place: libhdf5 reads the image's entries instead of
+/// the file's own bytes at their addresses, which may be stale
+/// (`clawhdf5_format::superblock_ext`). A file whose image libhdf5 cannot
+/// load is refused: a connector that reads whole datasets has no way to
+/// open the file and fail each object instead.
+pub(crate) fn load_hdf5(whole: &mut [u8]) -> Result<(), VolError> {
+    use clawhdf5_format::superblock_ext::apply_cache_image_in_place;
+    let err = |e: clawhdf5_format::error::FormatError| VolError::DataError(e.to_string());
+    let (len, sb) = {
+        let (data, sb) = hdf5_view(whole)?;
+        (data.len(), sb)
+    };
+    // hdf5_view's bytes start at the superblock, after any user block.
+    let base = clawhdf5_format::signature::split_user_block(whole)
+        .map_err(err)?
+        .0
+        .len();
+    apply_cache_image_in_place(&mut whole[base..base + len], &sb).map_err(err)
+}
+
 impl NativeVol {
     /// Create a new native VOL connector.
     pub fn new() -> Self {
         Self {
             data: None,
             location: None,
+            load_error: None,
         }
     }
 
@@ -245,10 +271,12 @@ impl NativeVol {
     }
 
     /// Create a native VOL connector from bytes already in memory.
-    pub fn from_bytes(data: Vec<u8>) -> Self {
+    pub fn from_bytes(mut data: Vec<u8>) -> Self {
+        let load_error = load_hdf5(&mut data).err().map(|e| e.to_string());
         Self {
             data: Some(data),
             location: Some("<memory>".into()),
+            load_error,
         }
     }
 
@@ -281,10 +309,12 @@ impl VirtualObjectLayer for NativeVol {
     }
 
     fn open(&mut self, location: &str) -> Result<(), VolError> {
-        let data = std::fs::read(location)?;
+        let mut data = std::fs::read(location)?;
-        // Refuse a truncated file at open, as libhdf5 does.
+        // Refuse a truncated file at open, as libhdf5 does, and load any
-        hdf5_view(&data)?;
+        // metadata cache image.
+        load_hdf5(&mut data)?;
         self.data = Some(data);
+        self.load_error = None;
         self.location = Some(location.to_string());
         Ok(())
     }
@@ -299,6 +329,9 @@ impl VirtualObjectLayer for NativeVol {
         let data = self.data.as_ref().ok_or_else(|| {
             VolError::Io(io::Error::new(io::ErrorKind::NotConnected, "file not open"))
         })?;
+        if let Some(e) = &self.load_error {
+            return Err(VolError::DataError(e.clone()));
+        }
 
         use clawhdf5_format::{
             data_layout::DataLayout, data_read::read_raw_data_full, dataspace::Dataspace,
@@ -434,6 +467,43 @@ mod tests {
         assert_eq!(raw.len(), 24);
     }
 
+    /// libhdf5's `h5clear_mdc_image.h5` (see the clawhdf5 crate's
+    /// `tests/metadata_cache_image.rs`): the root group's header exists
+    /// only in the file's metadata cache image, so reading the file's own
+    /// bytes finds zeros there.
+    #[test]
+    fn native_vol_reads_through_a_metadata_cache_image() {
+        let path = concat!(
+            env!("CARGO_MANIFEST_DIR"),
+            "/../clawhdf5/tests/fixtures/h5clear_mdc_image.h5"
+        );
+        let expected: Vec<u8> = (0..50)
+            .flat_map(|i| (0..100).map(move |j| i * j))
+            .flat_map(i32::to_le_bytes)
+            .collect();
+        let vol = NativeVol::open_path(path).unwrap();
+        assert_eq!(vol.read_dataset("DSET").unwrap(), expected);
+        let bytes = std::fs::read(path).unwrap();
+        let vol = NativeVol::from_bytes(bytes.clone());
+        assert_eq!(vol.read_dataset("DSET").unwrap(), expected);
+
+        // An image libhdf5 cannot load is refused, not read around.
+        let mut bad = bytes;
+        let at = bad.windows(4).position(|w| w == b"MDCI").unwrap();
+        bad[at] = b'X';
+        let err = NativeVol::from_bytes(bad.clone())
+            .read_dataset("DSET")
+            .unwrap_err();
+        assert!(err.to_string().contains("cache image"), "{err}");
+        let dir = tempfile::tempdir().unwrap();
+        let bad_path = dir.path().join("bad_image.h5");
+        std::fs::write(&bad_path, &bad).unwrap();
+        let err = NativeVol::open_path(bad_path.to_str().unwrap())
+            .err()
+            .unwrap();
+        assert!(err.to_string().contains("cache image"), "{err}");
+    }
+
     #[test]
     fn vol_error_display() {
         let err = VolError::Unsupported("read_dataset".into());

crates/clawhdf5-tools/src/h5.rs

@@ -234,6 +234,11 @@ impl H5 {
     }
 
     pub fn header(&self, addr: u64) -> Result<ObjectHeader> {
+        // A metadata cache image libhdf5 cannot load: the file opens, and
+        // every object fails (its bytes may hold stale metadata).
+        if let Some(e) = self.file.cache_image_error() {
+            return Err(Error::at(addr, format!("metadata cache image: {e}")));
+        }
         let off = usize::try_from(addr).map_err(|_| Error::at(addr, "address out of range"))?;
         ObjectHeader::parse(self.data(), off, self.os(), self.ls())
             .map_err(|e| Error::at(addr, format!("object header: {e}")))

crates/clawhdf5-tools/tests/h5rs_interop.rs

@@ -940,11 +940,17 @@ fn write_nested_links(dir: &Path) -> Vec<String> {
         g.create_dataset(&format!("n{i:02}")).with_i32_data(&[i]);
     }
     g.add_hard_link("back", "/a/b/c");
+    // Attribute creation order tracked too: dense, with a type-9 index.
+    for i in (0..12).rev() {
+        g.set_attr(&format!("attr{i:02}"), AttrValue::I64(i));
+    }
     b.add_group(g.finish());
     let mut g = b.create_group("compact_ordered");
     g.track_order(true);
     g.create_dataset("z").with_i32_data(&[1]);
     g.create_dataset("a").with_i32_data(&[2]);
+    g.set_attr("zz", AttrValue::I64(1));
+    g.set_attr("aa", AttrValue::I64(2));
     b.add_group(g.finish());
     let nested = dir.join("nested.h5");
     b.write(&nested).unwrap();
@@ -1024,3 +1030,75 @@ fn check_files_with_big_dense_storage() {
     assert_eq!(code(&o), 0, "{p}:\n{}", stdout(&o));
     assert!(stdout(&o).contains("no problems found"), "{}", stdout(&o));
 }
+
+/// `(type, depth)` of every v2 B-tree header in a file written with 8-byte
+/// offsets and lengths (found by signature and checksum).
+fn btree_v2_depths(data: &[u8]) -> Vec<(u8, u16)> {
+    const LEN: usize = 4 + 1 + 1 + 4 + 2 + 2 + 1 + 1 + 8 + 2 + 8;
+    let mut out = Vec::new();
+    for at in 0..data.len().saturating_sub(LEN + 4) {
+        if &data[at..at + 4] != b"BTHD" {
+            continue;
+        }
+        let stored = u32::from_le_bytes(data[at + LEN..at + LEN + 4].try_into().unwrap());
+        if jenkins_lookup3(&data[at..at + LEN]) == stored {
+            out.push((
+                data[at + 5],
+                u16::from_le_bytes([data[at + 12], data[at + 13]]),
+            ));
+        }
+    }
+    out
+}
+
+#[test]
+fn check_files_with_deep_btrees() {
+    // Dense indexes and a chunk index too big for one leaf: the writer then
+    // builds internal nodes, whose child pointers carry record counts in
+    // widths derived from the node size. `check` reads every record through
+    // them and compares the count with the header's.
+    use clawhdf5::{AttrValue, FileBuilder};
+    const U: u64 = u64::MAX;
+    let dir = tempfile::tempdir().unwrap();
+    let mut b = FileBuilder::new();
+    let x = b.create_dataset("x");
+    x.with_i32_data(&[7]);
+    for i in 0..70_000 {
+        x.set_attr(&format!("attr_{i}"), AttrValue::I64(i));
+    }
+    let mut g = b.create_group("g");
+    g.track_order(true);
+    for i in 0..100_000 {
+        g.add_hard_link(&format!("k{i}"), "/x");
+    }
+    b.add_group(g.finish());
+    let p = dir.path().join("deep.h5").to_string_lossy().into_owned();
+    b.write(&p).unwrap();
+    let o = h5rs(&["check", &p]);
+    assert_eq!(code(&o), 0, "{p}:\n{}", stdout(&o));
+    assert!(stdout(&o).contains("no problems found"), "{}", stdout(&o));
+    let mut depths = btree_v2_depths(&std::fs::read(&p).unwrap());
+    depths.sort();
+    assert_eq!(depths, [(5, 3), (6, 3), (8, 3)]);
+
+    let mut b = FileBuilder::new();
+    b.create_dataset("d")
+        .with_i32_data(&(0..200_000).collect::<Vec<i32>>())
+        .with_shape(&[400, 500])
+        .with_chunks(&[1, 1])
+        .with_maxshape(&[U, U]);
+    b.create_dataset("z")
+        .with_i32_data(&(0..70_000).collect::<Vec<i32>>())
+        .with_shape(&[70, 1000])
+        .with_chunks(&[1, 1])
+        .with_maxshape(&[U, U])
+        .with_deflate(1);
+    let p = dir.path().join("chunks.h5").to_string_lossy().into_owned();
+    b.write(&p).unwrap();
+    let o = h5rs(&["check", "--data", &p]);
+    assert_eq!(code(&o), 0, "{p}:\n{}", stdout(&o));
+    assert!(stdout(&o).contains("no problems found"), "{}", stdout(&o));
+    let mut depths = btree_v2_depths(&std::fs::read(&p).unwrap());
+    depths.sort();
+    assert_eq!(depths, [(10, 2), (11, 2)]);
+}

crates/clawhdf5/Cargo.toml

@@ -47,8 +47,9 @@ lzf = ["clawhdf5-format/lzf"]
 bitshuffle = ["clawhdf5-format/bitshuffle"]
 bzip2 = ["clawhdf5-format/bzip2"]
 blosc = ["clawhdf5-format/blosc"]
+blosc2 = ["clawhdf5-format/blosc2"]
 # Every plugin filter.
-plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc"]
+plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2"]
 # Dataset::verify_provenance() — recompute a dataset's SHA-256 and compare
 # against its stored _provenance_sha256 attribute. On by default, matching
 # clawhdf5-format's own default-on `provenance` feature.

crates/clawhdf5/src/cache_image.rs

@@ -0,0 +1,85 @@
+//! Metadata cache images, as the file openers apply them.
+//!
+//! A file written with a metadata cache image keeps metadata cache entries
+//! (object headers, B-tree nodes, heaps) in an image block, and libhdf5
+//! reads those entries in place of the file's own bytes at their addresses
+//! (see `clawhdf5_format::superblock_ext`). The metadata parsers read one
+//! contiguous byte slice, so the image has to be laid over the file's bytes
+//! — without copying the file:
+//!
+//! - an opener that holds the file in a buffer it owns writes the entries
+//!   into that buffer (only the image block is copied, as libhdf5 copies
+//!   it);
+//! - an opener that maps the file ([`File::open`](crate::File::open),
+//!   [`MmapFile`](crate::MmapFile), [`LazyFile::open_mmap`]
+//!   (crate::LazyFile::open_mmap)) writes them into a private copy-on-write
+//!   mapping of the file ([`clawhdf5_io::HDF5Read::private_copy`]): only the
+//!   pages the entries land on are copied, and the rest of the file stays
+//!   shared with the page cache;
+//! - a file without an image is read from the original bytes, as before.
+
+use clawhdf5_format::error::FormatError;
+use clawhdf5_format::superblock::Superblock;
+use clawhdf5_format::superblock_ext::{self, CacheImageState};
+use clawhdf5_io::PrivateCopy;
+
+use crate::error::Error;
+
+/// A file's metadata, as an opener must read it.
+pub(crate) enum ImageView {
+    /// Read the opener's bytes: the file has no cache image, or it was
+    /// written into a buffer the opener owns.
+    Plain,
+    /// The file with its cache image written in: a private copy of the
+    /// whole file (the HDF5 data at the same offsets as in the file).
+    Patched(PrivateCopy),
+    /// The file has a cache image libhdf5 cannot load.
+    Unloadable(FormatError),
+}
+
+/// Check the superblock extension of the file whose bytes are `whole` (the
+/// HDF5 data in `base..end`) and lay any cache image over a private copy
+/// of the file made by `copy`. An error means libhdf5 refuses the file.
+pub(crate) fn private_view(
+    whole: &[u8],
+    base: usize,
+    end: usize,
+    sb: &Superblock,
+    copy: impl FnOnce() -> std::io::Result<PrivateCopy>,
+) -> Result<ImageView, Error> {
+    let data = &whole[base..end];
+    Ok(match superblock_ext::cache_image_state(data, sb)? {
+        CacheImageState::Absent => ImageView::Plain,
+        CacheImageState::Unloadable(e) => ImageView::Unloadable(e),
+        CacheImageState::Loaded(image) => {
+            let mut view = copy().map_err(Error::Io)?;
+            let dst =
+                view.get_mut(base..end)
+                    .ok_or(Error::Format(FormatError::InvalidCacheImage(
+                        "the file changed while it was opened",
+                    )))?;
+            image.apply(image.block(data)?, dst)?;
+            ImageView::Patched(view)
+        }
+    })
+}
+
+/// [`private_view`] for a file held in `whole`, a buffer the opener owns:
+/// the image is written into it in place.
+pub(crate) fn in_place(
+    whole: &mut [u8],
+    base: usize,
+    end: usize,
+    sb: &Superblock,
+) -> Result<ImageView, Error> {
+    let data = &mut whole[base..end];
+    Ok(match superblock_ext::cache_image_state(data, sb)? {
+        CacheImageState::Absent => ImageView::Plain,
+        CacheImageState::Unloadable(e) => ImageView::Unloadable(e),
+        CacheImageState::Loaded(image) => {
+            let block = image.block(data)?.to_vec();
+            image.apply(&block, data)?;
+            ImageView::Plain
+        }
+    })
+}

crates/clawhdf5/src/lazy.rs

@@ -46,6 +46,13 @@ pub struct LazyFile<R: HDF5Read> {
     /// End of the HDF5 data (`Superblock::data_end`, absolute).
     end: usize,
     superblock: Superblock,
+    /// A file that holds a metadata cache image, with the image written in:
+    /// the reader's [`HDF5Read::private_copy`] of the whole file (a
+    /// copy-on-write mapping for [`clawhdf5_io::MmapReader`], so only the
+    /// pages the image's entries land on are copied; see
+    /// `crate::cache_image`). `None` for a file without an image, read
+    /// straight from the reader.
+    patched: Option<clawhdf5_io::PrivateCopy>,
     root_header: ObjectHeader,
     /// Cache of parsed object headers, keyed by address.
     header_cache: RefCell<HashMap<u64, ObjectHeader>>,
@@ -82,7 +89,24 @@ impl<R: HDF5Read> LazyFile<R> {
         let superblock = Superblock::parse(data, 0)?;
         // Refuse a truncated file; read nothing past the recorded end of file.
         let end = base + superblock.data_end(base as u64, whole_len)? as usize;
-        let data = &reader.as_bytes()[base..end];
+        // Decode the superblock extension as libhdf5 does at open, and load
+        // a metadata cache image over the file's metadata.
+        let view =
+            crate::cache_image::private_view(reader.as_bytes(), base, end, &superblock, || {
+                reader.private_copy()
+            })?;
+        let patched = match view {
+            crate::cache_image::ImageView::Plain => None,
+            crate::cache_image::ImageView::Patched(p) => Some(p),
+            // libhdf5 opens such a file and fails its first metadata read;
+            // a LazyFile reads the root group's header at open, so the open
+            // is that read.
+            crate::cache_image::ImageView::Unloadable(e) => return Err(e.into()),
+        };
+        let data = match &patched {
+            Some(p) => &p[base..end],
+            None => &reader.as_bytes()[base..end],
+        };
         let root_header = ObjectHeader::parse(
             data,
             superblock.root_group_address as usize,
@@ -94,6 +118,7 @@ impl<R: HDF5Read> LazyFile<R> {
             base,
             end,
             superblock,
+            patched,
             root_header,
             header_cache: RefCell::new(HashMap::new()),
         })
@@ -111,7 +136,10 @@ impl<R: HDF5Read> LazyFile<R> {
     }
 
     fn hdf5_bytes(&self) -> &[u8] {
-        &self.reader.as_bytes()[self.base..self.end]
+        match &self.patched {
+            Some(p) => &p[self.base..self.end],
+            None => &self.reader.as_bytes()[self.base..self.end],
+        }
     }
 
     /// Returns a reference to the parsed superblock.
@@ -135,10 +163,11 @@ impl<R: HDF5Read> LazyFile<R> {
         if !has_message(&hdr, MessageType::DataLayout) {
             return Err(Error::NotADataset(path.to_string()));
         }
-        Ok(LazyDataset {
+        LazyDataset {
             file: self,
             header: hdr,
-        })
+        }
+        .check_open()
     }
 
     /// Resolve a path and return a `LazyGroup` handle.
@@ -284,10 +313,11 @@ impl<'f, R: HDF5Read> LazyGroup<'f, R> {
         if !has_message(&hdr, MessageType::DataLayout) {
             return Err(Error::NotADataset(name.to_string()));
         }
-        Ok(LazyDataset {
+        LazyDataset {
             file: self.file,
             header: hdr,
-        })
+        }
+        .check_open()
     }
 
     /// Get a subgroup within this group by name.
@@ -326,6 +356,24 @@ pub struct LazyDataset<'f, R: HDF5Read> {
 }
 
 impl<'f, R: HDF5Read> LazyDataset<'f, R> {
+    /// libhdf5's storage checks when it opens a dataset
+    /// ([`data_read::check_dataset_storage`]): a dataset whose element count
+    /// times element size overflows, or whose contiguous storage runs past
+    /// the end of the file, fails to open. A datatype, dataspace or layout
+    /// that does not decode is left for the read to report, as before (the
+    /// dataset still opens, and its attributes can be read).
+    fn check_open(self) -> Result<Self, Error> {
+        let decoded = (|| -> Result<_, Error> {
+            let data = self.required_payload(MessageType::Dataspace)?;
+            let ds = Dataspace::parse(&data, self.file.length_size())?;
+            Ok((self.datatype()?, ds, self.data_layout()?))
+        })();
+        if let Ok((dt, ds, dl)) = decoded {
+            data_read::check_dataset_storage(&dl, &ds, &dt, self.file.hdf5_bytes().len() as u64)?;
+        }
+        Ok(self)
+    }
+
     /// Returns the shape (dimensions) of the dataset.
     pub fn shape(&self) -> Result<Vec<u64>, Error> {
         let ds = self.dataspace()?;
@@ -349,6 +397,9 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
 
     /// Read all data as `f64` values.
     pub fn read_f64(&self) -> Result<Vec<f64>, Error> {
+        if let Some(values) = self.read_chunked_native::<f64>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_f64(&raw, &dt)?)
@@ -396,6 +447,9 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
 
     /// Read all data as `f32` values.
     pub fn read_f32(&self) -> Result<Vec<f32>, Error> {
+        if let Some(values) = self.read_chunked_native::<f32>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_f32(&raw, &dt)?)
@@ -403,6 +457,9 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
 
     /// Read all data as `i32` values.
     pub fn read_i32(&self) -> Result<Vec<i32>, Error> {
+        if let Some(values) = self.read_chunked_native::<i32>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_i32(&raw, &dt)?)
@@ -410,6 +467,9 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
 
     /// Read all data as `i64` values.
     pub fn read_i64(&self) -> Result<Vec<i64>, Error> {
+        if let Some(values) = self.read_chunked_native::<i64>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_i64(&raw, &dt)?)
@@ -417,6 +477,9 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
 
     /// Read all data as `u64` values.
     pub fn read_u64(&self) -> Result<Vec<u64>, Error> {
+        if let Some(values) = self.read_chunked_native::<u64>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_u64(&raw, &dt)?)
@@ -550,6 +613,33 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
             .transpose()
     }
 
+    /// A chunked dataset that stores `T` natively, decoded straight into a
+    /// `Vec<T>` (no byte buffer to convert); `None` for any other dataset
+    /// (see [`data_read::read_chunked_native`]).
+    fn read_chunked_native<T: data_read::NativeElement>(&self) -> Result<Option<Vec<T>>, Error> {
+        let dl = self.data_layout()?;
+        if !matches!(dl, DataLayout::Chunked { .. }) {
+            return Ok(None);
+        }
+        let dt = self.datatype()?;
+        if !T::is_native(&dt) {
+            return Ok(None);
+        }
+        let ds = self.dataspace()?;
+        let pipeline = self.filter_pipeline()?;
+        Ok(data_read::read_chunked_native::<T>(
+            &self.header.messages,
+            self.file.hdf5_bytes(),
+            &dl,
+            &ds,
+            &dt,
+            pipeline.as_ref(),
+            self.file.offset_size(),
+            self.file.length_size(),
+            None,
+        )?)
+    }
+
     fn read_raw(&self) -> Result<Vec<u8>, Error> {
         let dt = self.datatype()?;
         let ds = self.dataspace()?;

crates/clawhdf5/src/lib.rs

@@ -24,6 +24,7 @@
 //! builder.write("output.h5").unwrap();
 //! ```
 
+mod cache_image;
 pub mod error;
 pub mod lazy;
 #[cfg(feature = "mmap")]

crates/clawhdf5/src/mmap_file.rs

@@ -38,6 +38,14 @@ pub struct MmapFile {
     /// End of the HDF5 data (`Superblock::data_end`, absolute).
     end: usize,
     superblock: Superblock,
+    /// A file that holds a metadata cache image, with the image written in:
+    /// a private copy-on-write mapping of the whole file, so only the pages
+    /// the image's entries land on are copied (see `crate::cache_image`).
+    /// `None` for a file without an image, read straight from the mapping.
+    patched: Option<clawhdf5_io::PrivateCopy>,
+    /// The file has a metadata cache image libhdf5 cannot load: every
+    /// object lookup fails with this error (see `File`).
+    image_error: Option<FormatError>,
 }
 
 impl MmapFile {
@@ -50,18 +58,34 @@ impl MmapFile {
         let superblock = Superblock::parse(data, 0)?;
         // Refuse a truncated file; read nothing past the recorded end of file.
         let end = base + superblock.data_end(base as u64, whole_len)? as usize;
+        // Decode the superblock extension as libhdf5 does at open, and load
+        // a metadata cache image over the file's metadata.
+        let view =
+            crate::cache_image::private_view(reader.as_bytes(), base, end, &superblock, || {
+                clawhdf5_io::HDF5Read::private_copy(&reader)
+            })?;
+        let (patched, image_error) = match view {
+            crate::cache_image::ImageView::Plain => (None, None),
+            crate::cache_image::ImageView::Patched(p) => (Some(p), None),
+            crate::cache_image::ImageView::Unloadable(e) => (None, Some(e)),
+        };
         Ok(Self {
             reader,
             base,
             end,
             superblock,
+            patched,
+            image_error,
         })
     }
 
     /// The file's bytes from the superblock on — the space HDF5 addresses
     /// index into.
     fn hdf5_bytes(&self) -> &[u8] {
-        &self.reader.as_bytes()[self.base..self.end]
+        match &self.patched {
+            Some(p) => &p[self.base..self.end],
+            None => &self.reader.as_bytes()[self.base..self.end],
+        }
     }
 
     /// Size of the user block before the superblock (0 for most files).
@@ -79,21 +103,22 @@ impl MmapFile {
 
     /// Resolve a path and return a `MmapDataset` handle.
     pub fn dataset(&self, path: &str) -> Result<MmapDataset<'_>, Error> {
-        let data = self.hdf5_bytes();
+        let data = self.meta()?;
         let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
         let hdr = self.parse_header(addr)?;
         if !has_message(&hdr, MessageType::DataLayout) {
             return Err(Error::NotADataset(path.to_string()));
         }
-        Ok(MmapDataset {
+        MmapDataset {
             file: self,
             header: hdr,
-        })
+        }
+        .check_open()
     }
 
     /// Resolve a path and return a `MmapGroup` handle.
     pub fn group(&self, path: &str) -> Result<MmapGroup<'_>, Error> {
-        let data = self.hdf5_bytes();
+        let data = self.meta()?;
         let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
         Ok(MmapGroup {
             file: self,
@@ -108,14 +133,29 @@ impl MmapFile {
         self.hdf5_bytes()
     }
 
+    /// The error of a metadata cache image libhdf5 cannot load, when the
+    /// file has one (see [`crate::File::cache_image_error`]).
+    pub fn cache_image_error(&self) -> Option<&FormatError> {
+        self.image_error.as_ref()
+    }
+
     /// Returns a reference to the parsed superblock.
     pub fn superblock(&self) -> &Superblock {
         &self.superblock
     }
 
+    /// The bytes to read metadata from; fails for a file whose cache image
+    /// cannot be loaded.
+    fn meta(&self) -> Result<&[u8], FormatError> {
+        match &self.image_error {
+            Some(e) => Err(e.clone()),
+            None => Ok(self.hdf5_bytes()),
+        }
+    }
+
     fn parse_header(&self, address: u64) -> Result<ObjectHeader, FormatError> {
         ObjectHeader::parse(
-            self.hdf5_bytes(),
+            self.meta()?,
             address as usize,
             self.superblock.offset_size,
             self.superblock.length_size,
@@ -211,10 +251,11 @@ impl<'f> MmapGroup<'f> {
         if !has_message(&hdr, MessageType::DataLayout) {
             return Err(Error::NotADataset(name.to_string()));
         }
-        Ok(MmapDataset {
+        MmapDataset {
             file: self.file,
             header: hdr,
-        })
+        }
+        .check_open()
     }
 
     /// Get a subgroup within this group by name.
@@ -235,7 +276,7 @@ impl<'f> MmapGroup<'f> {
     /// [`group_v2::resolve_group_children`]); dangling, external and
     /// user-defined links are left out.
     fn children(&self) -> Result<Vec<GroupEntry>, Error> {
-        let data = self.file.hdf5_bytes();
+        let data = self.file.meta()?;
         group_v2::resolve_group_children(data, &self.file.superblock, self.address)
             .map_err(Error::Format)
     }
@@ -253,6 +294,24 @@ pub struct MmapDataset<'f> {
 }
 
 impl<'f> MmapDataset<'f> {
+    /// libhdf5's storage checks when it opens a dataset
+    /// ([`data_read::check_dataset_storage`]): a dataset whose element count
+    /// times element size overflows, or whose contiguous storage runs past
+    /// the end of the file, fails to open. A datatype, dataspace or layout
+    /// that does not decode is left for the read to report, as before (the
+    /// dataset still opens, and its attributes can be read).
+    fn check_open(self) -> Result<Self, Error> {
+        let decoded = (|| -> Result<_, Error> {
+            let data = self.required_payload(MessageType::Dataspace)?;
+            let ds = Dataspace::parse(&data, self.file.length_size())?;
+            Ok((self.datatype()?, ds, self.data_layout()?))
+        })();
+        if let Ok((dt, ds, dl)) = decoded {
+            data_read::check_dataset_storage(&dl, &ds, &dt, self.file.hdf5_bytes().len() as u64)?;
+        }
+        Ok(self)
+    }
+
     /// Returns the shape (dimensions) of the dataset.
     pub fn shape(&self) -> Result<Vec<u64>, Error> {
         let ds = self.dataspace()?;
@@ -276,6 +335,9 @@ impl<'f> MmapDataset<'f> {
 
     /// Read all data as `f64` values.
     pub fn read_f64(&self) -> Result<Vec<f64>, Error> {
+        if let Some(values) = self.read_chunked_native::<f64>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_f64(&raw, &dt)?)
@@ -310,6 +372,9 @@ impl<'f> MmapDataset<'f> {
 
     /// Read all data as `f32` values.
     pub fn read_f32(&self) -> Result<Vec<f32>, Error> {
+        if let Some(values) = self.read_chunked_native::<f32>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_f32(&raw, &dt)?)
@@ -317,6 +382,9 @@ impl<'f> MmapDataset<'f> {
 
     /// Read all data as `i32` values.
     pub fn read_i32(&self) -> Result<Vec<i32>, Error> {
+        if let Some(values) = self.read_chunked_native::<i32>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_i32(&raw, &dt)?)
@@ -324,6 +392,9 @@ impl<'f> MmapDataset<'f> {
 
     /// Read all data as `i64` values.
     pub fn read_i64(&self) -> Result<Vec<i64>, Error> {
+        if let Some(values) = self.read_chunked_native::<i64>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_i64(&raw, &dt)?)
@@ -331,6 +402,9 @@ impl<'f> MmapDataset<'f> {
 
     /// Read all data as `u64` values.
     pub fn read_u64(&self) -> Result<Vec<u64>, Error> {
+        if let Some(values) = self.read_chunked_native::<u64>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         let dt = self.datatype()?;
         Ok(data_read::read_as_u64(&raw, &dt)?)
@@ -390,13 +464,7 @@ impl<'f> MmapDataset<'f> {
         match &dl {
             DataLayout::Contiguous { address, size } => {
                 let addr = address.ok_or(Error::Format(FormatError::NoDataAllocated))?;
-                let sz = *size as usize;
+                let sz = clawhdf5_format::data_read::contiguous_read_len(*size, expected)?;
-                if sz != expected {
-                    return Err(Error::Format(FormatError::DataSizeMismatch {
-                        expected,
-                        actual: sz,
-                    }));
-                }
                 let data = self.file.hdf5_bytes();
                 let a = addr as usize;
                 if a + sz > data.len() {
@@ -496,6 +564,33 @@ impl<'f> MmapDataset<'f> {
             .transpose()
     }
 
+    /// A chunked dataset that stores `T` natively, decoded straight into a
+    /// `Vec<T>` (no byte buffer to convert); `None` for any other dataset
+    /// (see [`data_read::read_chunked_native`]).
+    fn read_chunked_native<T: data_read::NativeElement>(&self) -> Result<Option<Vec<T>>, Error> {
+        let dl = self.data_layout()?;
+        if !matches!(dl, DataLayout::Chunked { .. }) {
+            return Ok(None);
+        }
+        let dt = self.datatype()?;
+        if !T::is_native(&dt) {
+            return Ok(None);
+        }
+        let ds = self.dataspace()?;
+        let pipeline = self.filter_pipeline()?;
+        Ok(data_read::read_chunked_native::<T>(
+            &self.header.messages,
+            self.file.hdf5_bytes(),
+            &dl,
+            &ds,
+            &dt,
+            pipeline.as_ref(),
+            self.file.offset_size(),
+            self.file.length_size(),
+            None,
+        )?)
+    }
+
     fn read_raw(&self) -> Result<Vec<u8>, Error> {
         let dt = self.datatype()?;
         let ds = self.dataspace()?;

crates/clawhdf5/src/reader.rs

@@ -21,6 +21,7 @@ use clawhdf5_format::object_header::ObjectHeader;
 use clawhdf5_format::signature;
 use clawhdf5_format::superblock::Superblock;
 
+use crate::cache_image::{self, ImageView};
 use crate::error::Error;
 use crate::types::{AttrValue, DType, classify_datatype, read_attrs};
 
@@ -55,12 +56,24 @@ struct FileData {
     base: usize,
     /// End of the HDF5 data in the file (`Superblock::data_end`, absolute).
     end: usize,
+    /// A mapped file that holds a metadata cache image, with the image
+    /// written in: a private copy-on-write mapping of the whole file, so
+    /// only the pages the image's entries land on are copied (see
+    /// `crate::cache_image`). `None` for every other file: a file without
+    /// an image is read straight from the mapping, and an owned buffer has
+    /// the image written into it in place.
+    patched: Option<clawhdf5_io::PrivateCopy>,
+    /// The file has a metadata cache image libhdf5 cannot load. libhdf5
+    /// opens such a file and fails its first metadata read (the image loads
+    /// then); every object lookup here fails with this error, and no
+    /// metadata is read from the file's own, possibly stale, bytes.
+    image_error: Option<FormatError>,
 }
 
 impl FileData {
     /// Locate the superblock and parse it. A truncated file is refused, and
     /// bytes past the recorded end of file are not read, as in libhdf5.
-    fn new(backing: Backing) -> Result<(Self, Superblock), Error> {
+    fn new(mut backing: Backing) -> Result<(Self, Superblock), Error> {
         let whole = backing.whole_file();
         let (user_block, hdf5) = signature::split_user_block(whole)?;
         let base = user_block.len();
@@ -68,16 +81,54 @@ impl FileData {
         let end = superblock.data_end(base as u64, whole.len() as u64)?;
         // data_end is at most the file length (less the user block).
         let end = base + end as usize;
-        Ok((Self { backing, base, end }, superblock))
+        // libhdf5 decodes the superblock extension at open (a message it
+        // cannot decode fails the open) and loads a metadata cache image
+        // over the file's own metadata.
+        let view = match &mut backing {
+            Backing::Owned(v) => cache_image::in_place(v, base, end, &superblock)?,
+            #[cfg(feature = "mmap")]
+            Backing::Mmap(r) => {
+                cache_image::private_view(r.as_bytes(), base, end, &superblock, || {
+                    clawhdf5_io::HDF5Read::private_copy(r)
+                })?
+            }
+        };
+        let (patched, image_error) = match view {
+            ImageView::Plain => (None, None),
+            ImageView::Patched(p) => (Some(p), None),
+            ImageView::Unloadable(e) => (None, Some(e)),
+        };
+        Ok((
+            Self {
+                backing,
+                base,
+                end,
+                patched,
+                image_error,
+            },
+            superblock,
+        ))
     }
 
     fn as_bytes(&self) -> &[u8] {
-        &self.backing.whole_file()[self.base..self.end]
+        match &self.patched {
+            Some(p) => &p[self.base..self.end],
+            None => &self.backing.whole_file()[self.base..self.end],
+        }
     }
 
     fn len(&self) -> usize {
         self.as_bytes().len()
     }
+
+    /// The bytes to read metadata from; fails for a file whose cache image
+    /// cannot be loaded (see [`Self::image_error`]).
+    fn meta(&self) -> Result<&[u8], FormatError> {
+        match &self.image_error {
+            Some(e) => Err(e.clone()),
+            None => Ok(self.as_bytes()),
+        }
+    }
 }
 
 // ---------------------------------------------------------------------------
@@ -164,16 +215,17 @@ impl File {
     ///
     /// The path uses `/` separators (e.g., `"group1/values"`).
     pub fn dataset(&self, path: &str) -> Result<Dataset<'_>, Error> {
-        let data = self.data.as_bytes();
+        let data = self.data.meta()?;
         let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
         let hdr = self.parse_header(addr)?;
         if !has_message(&hdr, MessageType::DataLayout) {
             return Err(Error::NotADataset(path.to_string()));
         }
-        Ok(Dataset {
+        Dataset {
             file: self,
             header: hdr,
-        })
+        }
+        .check_open()
     }
 
     /// A `Dataset` handle for the object header at `address` (an address
@@ -186,10 +238,11 @@ impl File {
         if !has_message(&hdr, MessageType::DataLayout) {
             return Err(Error::NotADataset(format!("object at address {address}")));
         }
-        Ok(Dataset {
+        Dataset {
             file: self,
             header: hdr,
-        })
+        }
+        .check_open()
     }
 
     /// Resolve a path and return a `Group` handle.
@@ -197,7 +250,7 @@ impl File {
     /// The path uses `/` separators (e.g., `"sensors"`).
     /// Use `"/"` or `""` for the root group.
     pub fn group(&self, path: &str) -> Result<Group<'_>, Error> {
-        let data = self.data.as_bytes();
+        let data = self.data.meta()?;
         let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
         Ok(Group {
             file: self,
@@ -253,11 +306,23 @@ impl File {
 
     /// Returns the file's bytes from the superblock on (after any user
     /// block). Every HDF5 address in the file indexes this slice, so it is
-    /// what the `clawhdf5_format` parsers expect as `file_data`.
+    /// what the `clawhdf5_format` parsers expect as `file_data`. For a file
+    /// with a metadata cache image these are the bytes with the image
+    /// applied; when the image cannot be loaded they are the file's own
+    /// bytes, whose metadata may be stale (every object lookup fails then).
     pub fn as_bytes(&self) -> &[u8] {
         self.data.as_bytes()
     }
 
+    /// The error of a metadata cache image libhdf5 cannot load, when the
+    /// file has one. Such a file opens, as in libhdf5, and every object
+    /// lookup fails with this error; code that parses [`Self::as_bytes`]
+    /// itself should check it first, since those bytes then hold the
+    /// file's own, possibly stale, metadata.
+    pub fn cache_image_error(&self) -> Option<&FormatError> {
+        self.data.image_error.as_ref()
+    }
+
     /// Size of the user block before the superblock (0 for most files).
     /// Matches h5py's `File.userblock_size`.
     pub fn user_block_size(&self) -> u64 {
@@ -302,7 +367,7 @@ impl File {
         raw: &[u8],
     ) -> Result<Vec<Vec<u8>>, Error> {
         crate::vlen::decode_string_bytes(
-            self.as_bytes(),
+            self.data.meta()?,
             datatype,
             raw,
             self.offset_size(),
@@ -320,7 +385,7 @@ impl File {
         raw: &[u8],
     ) -> Result<Vec<Vec<T>>, Error> {
         crate::vlen::decode_vlen(
-            self.as_bytes(),
+            self.data.meta()?,
             datatype,
             raw,
             self.offset_size(),
@@ -330,7 +395,7 @@ impl File {
 
     fn parse_header(&self, address: u64) -> Result<ObjectHeader, FormatError> {
         ObjectHeader::parse(
-            self.data.as_bytes(),
+            self.data.meta()?,
             address as usize,
             self.superblock.offset_size,
             self.superblock.length_size,
@@ -427,10 +492,11 @@ impl<'f> Group<'f> {
         if !has_message(&hdr, MessageType::DataLayout) {
             return Err(Error::NotADataset(name.to_string()));
         }
-        Ok(Dataset {
+        Dataset {
             file: self.file,
             header: hdr,
-        })
+        }
+        .check_open()
     }
 
     /// Get a subgroup within this group by name.
@@ -451,7 +517,7 @@ impl<'f> Group<'f> {
     /// [`group_v2::resolve_group_children`]); dangling, external and
     /// user-defined links are left out.
     fn children(&self) -> Result<Vec<GroupEntry>, Error> {
-        let data = self.file.data.as_bytes();
+        let data = self.file.data.meta()?;
         group_v2::resolve_group_children(data, &self.file.superblock, self.address)
             .map_err(Error::Format)
     }
@@ -469,6 +535,24 @@ pub struct Dataset<'f> {
 }
 
 impl<'f> Dataset<'f> {
+    /// libhdf5's storage checks when it opens a dataset
+    /// ([`data_read::check_dataset_storage`]): a dataset whose element count
+    /// times element size overflows, or whose contiguous storage runs past
+    /// the end of the file, fails to open. A datatype, dataspace or layout
+    /// that does not decode is left for the read to report, as before (the
+    /// dataset still opens, and its attributes can be read).
+    fn check_open(self) -> Result<Self, Error> {
+        let decoded = (|| -> Result<_, Error> {
+            let data = self.required_payload(MessageType::Dataspace)?;
+            let ds = Dataspace::parse(&data, self.file.length_size())?;
+            Ok((self.datatype()?, ds, self.data_layout()?))
+        })();
+        if let Ok((dt, ds, dl)) = decoded {
+            data_read::check_dataset_storage(&dl, &ds, &dt, self.file.data.len() as u64)?;
+        }
+        Ok(self)
+    }
+
     /// Returns the shape (dimensions) of the dataset.
     pub fn shape(&self) -> Result<Vec<u64>, Error> {
         let ds = self.dataspace()?;
@@ -507,6 +591,9 @@ impl<'f> Dataset<'f> {
         if let Ok(Some(bytes)) = self.read_raw_ref() {
             return Ok(data_read::read_as_f64(bytes, &dt)?);
         }
+        if let Some(values) = self.read_chunked_native::<f64>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         Ok(data_read::read_as_f64(&raw, &dt)?)
     }
@@ -524,6 +611,9 @@ impl<'f> Dataset<'f> {
         if let Ok(Some(bytes)) = self.read_raw_ref() {
             return Ok(data_read::read_as_f32(bytes, &dt)?);
         }
+        if let Some(values) = self.read_chunked_native::<f32>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         Ok(data_read::read_as_f32(&raw, &dt)?)
     }
@@ -536,6 +626,9 @@ impl<'f> Dataset<'f> {
         if let Ok(Some(bytes)) = self.read_raw_ref() {
             return Ok(data_read::read_as_i32(bytes, &dt)?);
         }
+        if let Some(values) = self.read_chunked_native::<i32>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         Ok(data_read::read_as_i32(&raw, &dt)?)
     }
@@ -548,6 +641,9 @@ impl<'f> Dataset<'f> {
         if let Ok(Some(bytes)) = self.read_raw_ref() {
             return Ok(data_read::read_as_i64(bytes, &dt)?);
         }
+        if let Some(values) = self.read_chunked_native::<i64>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         Ok(data_read::read_as_i64(&raw, &dt)?)
     }
@@ -560,6 +656,9 @@ impl<'f> Dataset<'f> {
         if let Ok(Some(bytes)) = self.read_raw_ref() {
             return Ok(data_read::read_as_u64(bytes, &dt)?);
         }
+        if let Some(values) = self.read_chunked_native::<u64>()? {
+            return Ok(values);
+        }
         let raw = self.read_raw()?;
         Ok(data_read::read_as_u64(&raw, &dt)?)
     }
@@ -1050,6 +1149,34 @@ impl<'f> Dataset<'f> {
             .transpose()
     }
 
+    /// A chunked dataset that stores `T` natively, decoded straight into a
+    /// `Vec<T>` through the file's chunk cache (no byte buffer to convert);
+    /// `None` for any other dataset (see
+    /// [`data_read::read_chunked_native`]).
+    fn read_chunked_native<T: data_read::NativeElement>(&self) -> Result<Option<Vec<T>>, Error> {
+        let dl = self.data_layout()?;
+        if !matches!(dl, DataLayout::Chunked { .. }) {
+            return Ok(None);
+        }
+        let dt = self.datatype()?;
+        if !T::is_native(&dt) {
+            return Ok(None);
+        }
+        let ds = self.dataspace()?;
+        let pipeline = self.filter_pipeline()?;
+        Ok(data_read::read_chunked_native::<T>(
+            &self.header.messages,
+            self.file.data.as_bytes(),
+            &dl,
+            &ds,
+            &dt,
+            pipeline.as_ref(),
+            self.file.offset_size(),
+            self.file.length_size(),
+            Some(&self.file.chunk_cache),
+        )?)
+    }
+
     fn read_raw(&self) -> Result<Vec<u8>, Error> {
         let dt = self.datatype()?;
         let ds = self.dataspace()?;
@@ -1240,3 +1367,44 @@ mod sibling_file_name_tests {
         }
     }
 }
+
+#[cfg(all(test, feature = "mmap"))]
+mod zero_copy_tests {
+    use super::*;
+
+    /// Where `File::open` reads metadata from: `Some(true)` for the file's
+    /// own mapping, `Some(false)` for a private copy-on-write mapping.
+    fn reads_from_the_mapping(f: &File) -> Option<bool> {
+        let Backing::Mmap(r) = &f.data.backing else {
+            return None;
+        };
+        let mapped = r.as_bytes()[f.data.base..].as_ptr();
+        match &f.data.patched {
+            None => Some(std::ptr::eq(f.as_bytes().as_ptr(), mapped)),
+            Some(p) => {
+                assert!(p.is_mapped(), "the image went into a heap copy of the file");
+                Some(false)
+            }
+        }
+    }
+
+    #[test]
+    fn a_file_without_a_cache_image_is_read_from_the_mapping() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("plain.h5");
+        let mut b = crate::FileBuilder::new();
+        b.create_dataset("d").with_f64_data(&[1.0, 2.0]);
+        b.write(&path).unwrap();
+        let f = File::open(&path).unwrap();
+        assert_eq!(reads_from_the_mapping(&f), Some(true));
+        assert_eq!(f.dataset("d").unwrap().read_f64().unwrap(), [1.0, 2.0]);
+    }
+
+    #[test]
+    fn a_cache_image_goes_into_a_copy_on_write_mapping() {
+        let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
+            .join("tests/fixtures/h5clear_mdc_image.h5");
+        let f = File::open(path).unwrap();
+        assert_eq!(reads_from_the_mapping(&f), Some(false));
+    }
+}

crates/clawhdf5/src/writer.rs

@@ -88,9 +88,11 @@ impl FileBuilder {
         self
     }
 
-    /// Track link creation order in every group that does not set its own
+    /// Track the creation order of links and attributes in every group, and
-    /// (`GroupBuilder::track_order`), as h5py's `track_order=True`: libhdf5
+    /// of attributes on every dataset, that does not set its own
-    /// then lists members in the order they were added.
+    /// (`GroupBuilder::track_order`, `DatasetBuilder::track_order`), as
+    /// h5py's `track_order=True`: libhdf5 then lists members and attributes
+    /// in the order they were added.
     pub fn track_order(&mut self, track: bool) -> &mut Self {
         self.writer.track_order(track);
         self

crates/clawhdf5/tests/busy_decode_pool.rs

@@ -0,0 +1,112 @@
+//! Full reads of chunked datasets must not wait for a busy rayon pool of
+//! any size.
+//!
+//! A full read handed its chunks to the rayon pool (`par_iter`) and the
+//! calling thread — not a pool worker — slept until the pool had decoded
+//! them. With a small pool (2-4 threads) and more reading threads than
+//! workers, every reader queued behind the same few workers
+//! (`docs/known-issues.md`, "Concurrent and contiguous read performance").
+//! Now the calling thread decodes too, and pool workers only help when they
+//! are free. The test keeps both workers of a two-thread pool busy and
+//! requires reads to finish anyway, with the right values.
+//!
+//! One test in its own binary: it configures the process-wide rayon pool.
+
+#![cfg(feature = "parallel")]
+
+use std::sync::mpsc;
+use std::time::Duration;
+
+use clawhdf5::{File, FileBuilder};
+
+const N: usize = 4096; // 64 chunks of 64 elements
+
+fn values() -> Vec<f64> {
+    (0..N).map(|i| i as f64 * 0.25 - 7.0).collect()
+}
+
+fn build() -> File {
+    let mut b = FileBuilder::new();
+    b.create_dataset("data")
+        .with_f64_data(&values())
+        .with_shape(&[N as u64])
+        .with_chunks(&[64])
+        .with_deflate(1)
+        .with_provenance("test-suite", "2026-09-26T00:00:00Z", None);
+    File::from_bytes(b.finish().unwrap()).unwrap()
+}
+
+/// Run `f` on a fresh thread; `None` if it has not finished within `limit`.
+fn finishes_within<T: Send + 'static>(
+    limit: Duration,
+    f: impl FnOnce() -> T + Send + 'static,
+) -> Option<T> {
+    let (tx, rx) = mpsc::channel();
+    std::thread::spawn(move || {
+        let _ = tx.send(f());
+    });
+    rx.recv_timeout(limit).ok()
+}
+
+#[test]
+fn full_reads_do_not_wait_for_a_busy_small_pool() {
+    const WORKERS: usize = 2;
+    rayon::ThreadPoolBuilder::new()
+        .num_threads(WORKERS)
+        .build_global()
+        .expect("this test binary configures the global pool first");
+
+    // Built first: the writer compresses on the pool too.
+    let file = std::sync::Arc::new(build());
+
+    // Occupy every worker of the pool until the reads are done.
+    let (started_tx, started_rx) = mpsc::channel();
+    let (release_tx, release_rx) = mpsc::channel::<()>();
+    let release_rx = std::sync::Arc::new(std::sync::Mutex::new(release_rx));
+    for _ in 0..WORKERS {
+        let (started_tx, release_rx) = (started_tx.clone(), release_rx.clone());
+        rayon::spawn(move || {
+            started_tx.send(()).unwrap();
+            let _ = release_rx.lock().unwrap().recv();
+        });
+    }
+    for _ in 0..WORKERS {
+        started_rx.recv().unwrap();
+    }
+
+    let limit = Duration::from_secs(20);
+    // Several readers at once, as in the `concurrent_read` benchmark: the
+    // cached full read (`read_*`), the typed one and the uncached reader
+    // behind `verify_provenance`.
+    let readers: Vec<_> = (0..4)
+        .map(|_| {
+            let file = std::sync::Arc::clone(&file);
+            std::thread::spawn(move || {
+                finishes_within(limit, move || {
+                    let ds = file.dataset("data").unwrap();
+                    (
+                        ds.read_f64().unwrap(),
+                        ds.read_f32().unwrap(),
+                        ds.verify_provenance().unwrap(),
+                    )
+                })
+            })
+        })
+        .collect();
+    let results: Vec<_> = readers.into_iter().map(|h| h.join().unwrap()).collect();
+    // Free the workers before asserting, so a failure does not hang the
+    // blocked reader threads forever.
+    for _ in 0..WORKERS {
+        release_tx.send(()).unwrap();
+    }
+
+    let want = values();
+    let want_f32: Vec<f32> = want.iter().map(|&v| v as f32).collect();
+    for result in results {
+        let (f64s, f32s, verified) =
+            result.expect("a full read waited for the busy two-thread rayon pool");
+        assert_eq!(f64s, want);
+        assert_eq!(f32s, want_f32);
+        assert_eq!(verified, clawhdf5::provenance::VerifyResult::Ok);
+    }
+}

crates/clawhdf5/tests/cache_image_memory.rs

@@ -0,0 +1,139 @@
+//! Opening a file with a metadata cache image must not copy the file.
+//!
+//! The image's entries are laid over the file's bytes in a private
+//! copy-on-write mapping (`File::open`, `MmapFile::open`,
+//! `LazyFile::open_mmap`), so only the pages they land on are copied. The
+//! first implementation copied the whole file onto the heap at open: a
+//! 1 GiB file that takes a few KB on disk needed 2 GB of memory, and an
+//! 8 GiB one aborted the process. Here libhdf5 itself (the library h5py
+//! bundles, through ctypes: `H5Pset_mdc_image_config`) adds an image to a
+//! 1 GiB sparse file, and the process's resident memory must stay far below
+//! the file's size while each opener lists the file and reads its small
+//! dataset.
+//!
+//! One test in its own binary, so no other test's allocations land in the
+//! measurement. Linux only (it reads `VmRSS` from `/proc/self/status`).
+//! Skipped when python3 with h5py is unavailable, unless
+//! `CLAWHDF5_REQUIRE_INTEROP=1`.
+
+#![cfg(target_os = "linux")]
+
+use std::path::Path;
+use std::process::Command;
+
+use clawhdf5::{File, LazyFile, MmapFile};
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn python_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+fn rss_bytes() -> u64 {
+    let status = std::fs::read_to_string("/proc/self/status").unwrap();
+    let line = status.lines().find(|l| l.starts_with("VmRSS:")).unwrap();
+    let kb: u64 = line.split_whitespace().nth(1).unwrap().parse().unwrap();
+    kb * 1024
+}
+
+/// A 1 GiB sparse file: `/big`, 2^27 `f8` with only its last element
+/// written, `/small` = 0..10, with a metadata cache image added by libhdf5.
+fn make_file(path: &Path) {
+    let script = format!(
+        r#"
+import ctypes, glob, os, h5py, numpy as np
+path = "{path}"
+libs = glob.glob(os.path.join(os.path.dirname(h5py.__file__), os.pardir, "h5py.libs", "libhdf5-*.so*"))
+assert libs, "no libhdf5 bundled with h5py"
+lib = ctypes.CDLL(libs[0])
+class Cfg(ctypes.Structure):
+    _fields_ = [("version", ctypes.c_int), ("generate_image", ctypes.c_bool),
+                ("save_resize_status", ctypes.c_bool), ("entry_ageout", ctypes.c_int)]
+with h5py.File(path, "w", libver="latest") as f:
+    d = f.create_dataset("big", shape=(2**27,), dtype="f8")
+    d[-1] = 7.5
+    f.create_dataset("small", data=np.arange(10, dtype="<i4"))
+fapl = h5py.h5p.create(h5py.h5p.FILE_ACCESS)
+fapl.set_libver_bounds(h5py.h5f.LIBVER_LATEST, h5py.h5f.LIBVER_LATEST)
+cfg = Cfg(1, True, False, -1)
+assert lib.H5Pset_mdc_image_config(ctypes.c_int64(fapl.id), ctypes.byref(cfg)) >= 0
+f = h5py.File(h5py.h5f.open(path.encode(), h5py.h5f.ACC_RDWR, fapl=fapl))
+f["small"][()]; f["big"].shape
+f.close()
+assert os.path.getsize(path) >= 2**30
+with open(path, "rb") as fh:
+    fh.seek(-(1 << 20), 2)
+    assert b"MDCI" in fh.read(), "libhdf5 wrote no cache image"
+with h5py.File(path, "r") as f:
+    assert list(f["small"][()]) == list(range(10))
+"#,
+        path = path.display()
+    );
+    let out = Command::new(python())
+        .args(["-c", &script])
+        .output()
+        .expect("failed to run python");
+    assert!(
+        out.status.success(),
+        "python failed:\n{}",
+        String::from_utf8_lossy(&out.stderr)
+    );
+}
+
+#[test]
+fn a_cache_image_does_not_copy_the_file() {
+    if !python_available() {
+        assert!(
+            !std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1"),
+            "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+        );
+        eprintln!("SKIP: python3 with h5py not available");
+        return;
+    }
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("sparse_image.h5");
+    make_file(&path);
+
+    const LIMIT: u64 = 256 << 20;
+    let small: Vec<i32> = (0..10).collect();
+    let before = rss_bytes();
+    {
+        let f = File::open(&path).unwrap();
+        let mut names = f.root().datasets().unwrap();
+        names.sort();
+        assert_eq!(names, ["big", "small"]);
+        assert_eq!(f.dataset("small").unwrap().read_i32().unwrap(), small);
+        assert_eq!(f.dataset("big").unwrap().shape().unwrap(), [1 << 27]);
+        let grew = rss_bytes().saturating_sub(before);
+        assert!(
+            grew < LIMIT,
+            "File::open: resident memory grew {grew} bytes"
+        );
+    }
+    let before = rss_bytes();
+    {
+        let f = MmapFile::open(&path).unwrap();
+        assert_eq!(f.dataset("small").unwrap().read_i32().unwrap(), small);
+        let grew = rss_bytes().saturating_sub(before);
+        assert!(
+            grew < LIMIT,
+            "MmapFile::open: resident memory grew {grew} bytes"
+        );
+    }
+    let before = rss_bytes();
+    {
+        let f = LazyFile::open_mmap(&path).unwrap();
+        assert_eq!(f.dataset("small").unwrap().read_i32().unwrap(), small);
+        let grew = rss_bytes().saturating_sub(before);
+        assert!(
+            grew < LIMIT,
+            "LazyFile::open_mmap: resident memory grew {grew} bytes"
+        );
+    }
+}

crates/clawhdf5/tests/chunk_index_interop.rs

@@ -559,20 +559,6 @@ fn we_write_btree_v2_for_several_unlimited_dims() {
     check_we_write(&cases);
 }
 
-/// A single-leaf B-tree has a 16-bit record count; beyond it the writer
-/// refuses rather than writing a tree libhdf5 would misread.
-#[test]
-fn btree_v2_index_past_one_leaf_is_refused() {
-    let mut b = FileBuilder::new();
-    b.create_dataset("d")
-        .with_i32_data(&vec![0i32; 70_000])
-        .with_shape(&[70_000, 1])
-        .with_chunks(&[1, 1])
-        .with_maxshape(&[u64::MAX, u64::MAX]);
-    let dir = tempfile::tempdir().unwrap();
-    assert!(b.write(dir.path().join("too_many.h5")).is_err());
-}
-
 /// A maxshape equal to the shape cannot grow, so it needs no chunks: the
 /// dataset stays contiguous (as h5py makes it) unless chunks are requested.
 #[test]

crates/clawhdf5/tests/chunked_read_paths_interop.rs

@@ -0,0 +1,361 @@
+//! Every full and selection read path of chunked datasets against h5py.
+//!
+//! h5py (libhdf5) writes chunked datasets of every numeric type the typed
+//! readers cover, in both byte orders, through deflate, shuffle,
+//! Fletcher32, LZF, SZIP and Blosc, in 1-3 dimensional shapes whose chunks
+//! do not divide them (partial edge chunks), plus sparse datasets whose
+//! unwritten chunks read as a fill value. Next to each it stores the values
+//! as contiguous `f64`, and checks that h5py reads the chunked dataset back
+//! as those values.
+//!
+//! clawhdf5 must read every chunked dataset as those values through every
+//! reader: `File` (the chunk-cached reader, twice so the second read can hit
+//! the cache, and the typed readers that decode straight into their output),
+//! `File::from_bytes`, `MmapFile`, `LazyFile`, and the selection readers
+//! (a small hyperslab, a strided one covering most of the dataset, points).
+//! With `--features parallel` the same reads decode chunks on several
+//! threads. A filter this build does not include must be an error, never
+//! data.
+//!
+//! Skipped when python3 with h5py is unavailable, unless
+//! `CLAWHDF5_REQUIRE_INTEROP=1`.
+
+use std::process::Command;
+
+use clawhdf5::{File, LazyFile, MmapFile, Selection};
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+/// Whether the interop test can run; panics instead of skipping when
+/// `CLAWHDF5_REQUIRE_INTEROP=1`.
+fn have_python() -> bool {
+    let ok = Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false);
+    if ok {
+        return true;
+    }
+    assert!(
+        !interop_required(),
+        "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+    );
+    eprintln!("SKIP: python3 with h5py not available");
+    false
+}
+
+/// Writes `c/<name>` (chunked) and `e/<name>` (the expected values, `f64`,
+/// contiguous) for every case; prints one line per case:
+/// `name filter_ids(comma-separated or -)`.
+const GENERATE: &str = r#"
+import sys
+import numpy as np, h5py
+try:
+    import hdf5plugin
+except ImportError:
+    hdf5plugin = None
+path = sys.argv[1]
+
+dtypes = []
+for code in ['i1', 'u1', 'i2', 'u2', 'i4', 'u4', 'i8', 'u8', 'f2', 'f4', 'f8']:
+    orders = ['|'] if code[1] == '1' else ['<', '>']
+    dtypes += [np.dtype(o + code) for o in orders]
+
+filters = {
+    'none': {},
+    'gzip': dict(compression='gzip', compression_opts=4),
+    'shuffle_gzip': dict(shuffle=True, compression='gzip', compression_opts=1),
+    # h5py puts the checksum last (applied after deflate).
+    'shuffle_gzip_fletcher': dict(shuffle=True, compression='gzip', fletcher32=True),
+    'fletcher': dict(fletcher32=True),
+    'shuffle_lzf': dict(shuffle=True, compression='lzf'),
+}
+if h5py.h5z.filter_avail(h5py.h5z.FILTER_SZIP):
+    filters['szip'] = dict(compression='szip', compression_opts=('nn', 8))
+if hdf5plugin is not None:
+    filters['blosc'] = dict(**hdf5plugin.Blosc(cname='lz4', clevel=5,
+                                                shuffle=hdf5plugin.Blosc.SHUFFLE))
+
+shapes = [((37, 23), (8, 5)), ((101,), (16,)), ((9, 10, 11), (4, 3, 5))]
+
+def values(n, dt):
+    i = np.arange(n, dtype=np.int64)
+    if dt.kind == 'f':
+        v = ((i * 7 + 3) % 1000) / 8.0 - 60.0
+    elif dt.kind == 'i':
+        v = (i * 7 + 3) % 200 - 100
+    else:
+        v = (i * 7 + 3) % 250
+    return v.astype(dt)
+
+def filter_ids(dset):
+    plist = dset.id.get_create_plist()
+    ids = [str(plist.get_filter(k)[0]) for k in range(plist.get_nfilters())]
+    return ','.join(ids) or '-'
+
+with h5py.File(path, 'w') as f:
+    n = 0
+    for dt in dtypes:
+        for fname, fopts in filters.items():
+            for s, (shape, chunks) in enumerate(shapes):
+                name = f"{dt.str.replace('|', 'x').replace('<', 'le').replace('>', 'be')}_{fname}_{s}"
+                data = values(int(np.prod(shape)), dt).reshape(shape)
+                try:
+                    d = f.create_dataset('c/' + name, data=data, chunks=chunks, **fopts)
+                except (ValueError, TypeError) as e:
+                    continue  # a filter that refuses this type
+                f.create_dataset('e/' + name, data=data.astype('<f8'))
+                assert np.array_equal(d[()], data), name
+                print(name, filter_ids(d))
+                n += 1
+        # Larger than the file's 16 MiB chunk cache, so `File` decodes into
+        # its reusable buffers instead of caching chunks.
+        if dt.str in ('<f4', '>i4'):
+            name = f"{dt.str.replace('<', 'le').replace('>', 'be')}_large"
+            shape = (2600, 2048)
+            data = values(shape[0] * shape[1], dt).reshape(shape)
+            d = f.create_dataset('c/' + name, data=data, chunks=(256, 256),
+                                 shuffle=True, compression='gzip', compression_opts=1)
+            f.create_dataset('e/' + name, data=data.astype('<f8'))
+            assert np.array_equal(d[()], data), name
+            print(name, filter_ids(d))
+        # Sparse: only some chunks written; the rest read as the fill value
+        # (non-default, and default 0).
+        for fill in [None, 42]:
+            for fname in ['none', 'shuffle_gzip']:
+                name = f"{dt.str.replace('|', 'x').replace('<', 'le').replace('>', 'be')}_sparse_{fname}_{fill}"
+                shape, chunks = (37, 23), (8, 5)
+                kw = dict(filters[fname])
+                if fill is not None:
+                    kw['fillvalue'] = np.array(fill, dtype=dt)
+                d = f.create_dataset('c/' + name, shape=shape, dtype=dt, chunks=chunks, **kw)
+                full = values(37 * 23, dt).reshape(shape)
+                d[3:17, 4:12] = full[3:17, 4:12]
+                d[30:, 20:] = full[30:, 20:]
+                expect = d[()]
+                want = np.full(shape, 0 if fill is None else fill, dtype=dt)
+                want[3:17, 4:12] = full[3:17, 4:12]
+                want[30:, 20:] = full[30:, 20:]
+                assert np.array_equal(expect, want), name
+                f.create_dataset('e/' + name, data=want.astype('<f8'))
+                print(name, filter_ids(d))
+"#;
+
+/// A filter this build can decode.
+fn filters_available(ids: &str) -> bool {
+    ids == "-"
+        || ids.split(',').all(|id| {
+            clawhdf5_format::filter_registry::is_filter_available(id.parse().expect("filter id"))
+        })
+}
+
+/// The expected values converted as libhdf5 converts them for each typed
+/// reader (every case's values are exact in `f32` and within `i32`).
+struct Expected {
+    f64s: Vec<f64>,
+}
+
+impl Expected {
+    fn f32s(&self) -> Vec<f32> {
+        self.f64s.iter().map(|&v| v as f32).collect()
+    }
+    /// Truncation toward zero; negative values read as unsigned are 0.
+    fn i32s(&self) -> Vec<i32> {
+        self.f64s.iter().map(|&v| v as i32).collect()
+    }
+    fn i64s(&self) -> Vec<i64> {
+        self.f64s.iter().map(|&v| v as i64).collect()
+    }
+    fn u64s(&self) -> Vec<u64> {
+        self.f64s.iter().map(|&v| v as u64).collect()
+    }
+    fn select(&self, idx: &[usize]) -> Expected {
+        Expected {
+            f64s: idx.iter().map(|&i| self.f64s[i]).collect(),
+        }
+    }
+}
+
+/// The typed readers' results for one dataset, compared with `want`.
+macro_rules! check_typed {
+    ($ds:expr, $want:expr, $name:expr, $path:expr) => {{
+        let (ds, want, name, path) = (&$ds, &$want, $name, $path);
+        assert_eq!(ds.read_f64().unwrap(), want.f64s, "{name} {path} f64");
+        assert_eq!(ds.read_f32().unwrap(), want.f32s(), "{name} {path} f32");
+        assert_eq!(ds.read_i32().unwrap(), want.i32s(), "{name} {path} i32");
+        assert_eq!(ds.read_i64().unwrap(), want.i64s(), "{name} {path} i64");
+        assert_eq!(ds.read_u64().unwrap(), want.u64s(), "{name} {path} u64");
+    }};
+}
+
+/// Row-major indices of the elements `sel` picks from a dataset of `dims`.
+fn selected(sel: &Selection, dims: &[u64]) -> Vec<usize> {
+    let strides: Vec<usize> = (0..dims.len())
+        .map(|d| dims[d + 1..].iter().product::<u64>() as usize)
+        .collect();
+    match sel {
+        Selection::Hyperslab {
+            start,
+            stride,
+            count,
+            block,
+        } => {
+            // Per dimension, the selected coordinates in order.
+            let axes: Vec<Vec<u64>> = (0..dims.len())
+                .map(|d| {
+                    (0..count[d])
+                        .flat_map(|c| (0..block[d]).map(move |b| start[d] + c * stride[d] + b))
+                        .collect()
+                })
+                .collect();
+            let mut out = vec![0usize];
+            for (d, axis) in axes.iter().enumerate() {
+                let stride = strides[d];
+                out = out
+                    .iter()
+                    .flat_map(|&base| axis.iter().map(move |&x| base + x as usize * stride))
+                    .collect();
+            }
+            out
+        }
+        Selection::Points(points) => points
+            .iter()
+            .map(|p| p.iter().zip(&strides).map(|(&x, &s)| x as usize * s).sum())
+            .collect(),
+        _ => unreachable!(),
+    }
+}
+
+/// A small box (read through the partial-read path), a strided selection
+/// covering most of the dataset (read in full, then selected), and points.
+fn selections(dims: &[u64]) -> Vec<Selection> {
+    let rank = dims.len();
+    let small = Selection::Hyperslab {
+        start: dims.iter().map(|&d| d / 3).collect(),
+        stride: vec![1; rank],
+        count: dims.iter().map(|&d| (d / 4).max(1)).collect(),
+        block: vec![1; rank],
+    };
+    let strided = Selection::Hyperslab {
+        start: vec![0; rank],
+        stride: vec![2; rank],
+        count: dims.iter().map(|&d| d.div_ceil(2)).collect(),
+        block: vec![1; rank],
+    };
+    let points = Selection::Points(vec![
+        vec![0; rank],
+        dims.iter().map(|&d| d - 1).collect(),
+        dims.iter().map(|&d| d / 2).collect(),
+        dims.iter().map(|&d| (d * 2) / 3).collect(),
+    ]);
+    vec![small, strided, points]
+}
+
+#[test]
+fn chunked_reads_match_h5py_on_every_path() {
+    if !have_python() {
+        return;
+    }
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("chunked_paths.h5");
+    let out = Command::new(python())
+        .arg("-c")
+        .arg(GENERATE)
+        .arg(&path)
+        .output()
+        .expect("run python");
+    assert!(
+        out.status.success(),
+        "generator failed: {}",
+        String::from_utf8_lossy(&out.stderr)
+    );
+    let cases: Vec<(String, String)> = String::from_utf8(out.stdout)
+        .unwrap()
+        .lines()
+        .map(|l| {
+            let (name, ids) = l.split_once(' ').unwrap();
+            (name.to_string(), ids.to_string())
+        })
+        .collect();
+    assert!(cases.len() > 300, "only {} cases", cases.len());
+    if interop_required() {
+        // The generator must have covered the plugin filters too.
+        for f in ["_szip_", "_blosc_", "_shuffle_lzf_", "_sparse_"] {
+            assert!(cases.iter().any(|(n, _)| n.contains(f)), "no {f} case");
+        }
+    }
+
+    let file = File::open(&path).unwrap();
+    let owned = File::from_bytes(std::fs::read(&path).unwrap()).unwrap();
+    let mmap = MmapFile::open(&path).unwrap();
+    let lazy = LazyFile::open_mmap(&path).unwrap();
+    let (mut checked, mut refused) = (0, 0);
+    for (name, ids) in &cases {
+        let chunked = format!("c/{name}");
+        let want = Expected {
+            f64s: file
+                .dataset(&format!("e/{name}"))
+                .unwrap()
+                .read_f64()
+                .unwrap(),
+        };
+        let ds = file.dataset(&chunked).unwrap();
+        if !filters_available(ids) {
+            // Unsupported filter: an error from every reader, never wrong
+            // data. (An optional filter, as Blosc is, may have declined every
+            // chunk — they are then stored as-is and read fine.)
+            let results = [
+                ds.read_f64(),
+                owned.dataset(&chunked).unwrap().read_f64(),
+                mmap.dataset(&chunked).unwrap().read_f64(),
+                lazy.dataset(&chunked).unwrap().read_f64(),
+            ];
+            let errors = results.iter().filter(|r| r.is_err()).count();
+            assert!(errors == 0 || errors == results.len(), "{name}");
+            for values in results.into_iter().flatten() {
+                assert_eq!(values, want.f64s, "{name}");
+            }
+            if errors > 0 {
+                assert!(ds.read_f32().is_err(), "{name}");
+                refused += 1;
+                continue;
+            }
+        }
+        // Twice: the second read of a small dataset comes from the cache.
+        check_typed!(ds, want, name, "File");
+        check_typed!(ds, want, name, "File (cached)");
+        check_typed!(owned.dataset(&chunked).unwrap(), want, name, "from_bytes");
+        check_typed!(mmap.dataset(&chunked).unwrap(), want, name, "MmapFile");
+        check_typed!(lazy.dataset(&chunked).unwrap(), want, name, "LazyFile");
+
+        let dims = ds.shape().unwrap();
+        for sel in selections(&dims) {
+            let want = want.select(&selected(&sel, &dims));
+            assert_eq!(
+                ds.read_f64_selection(&sel).unwrap(),
+                want.f64s,
+                "{name} {sel:?}"
+            );
+            assert_eq!(
+                ds.read_f32_selection(&sel).unwrap(),
+                want.f32s(),
+                "{name} {sel:?}"
+            );
+            assert_eq!(
+                ds.read_i64_selection(&sel).unwrap(),
+                want.i64s(),
+                "{name} {sel:?}"
+            );
+        }
+        checked += 1;
+    }
+    eprintln!("{checked} datasets read on every path, {refused} refused (filter not built in)");
+    assert!(checked > 300);
+}