docs: changelog for the review follow-up fixes

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

.gitea/workflows/conformance.yml

@@ -0,0 +1,56 @@
+name: Conformance
+# Nightly: read every file of the pinned public HDF5 corpora with clawhdf5 and
+# with h5py/libhdf5 and compare (conformance/run.sh; CONFORMANCE.md explains
+# the method). Fails on any panic, hang, crash or out-of-memory in clawhdf5,
+# and when the ok count drops below conformance/baseline.json or a file the
+# baseline lists as ok stops being ok. The report is printed into the job log;
+# nothing is uploaded (artifact actions are JavaScript, which rust:latest
+# cannot run — see CLAUDE.md).
+on:
+  schedule:
+    - cron: "17 3 * * *"
+  workflow_dispatch:
+jobs:
+  conformance:
+    runs-on: ubuntu-latest
+    container: rust:latest
+    timeout-minutes: 60
+    env:
+      CARGO_NET_RETRY: "10"
+    steps:
+      # Plain git, not actions/checkout (a JavaScript action; see ci.yml).
+      - name: Check out
+        run: |
+          git init -q .
+          git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
+          for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
+          git checkout -q FETCH_HEAD
+      - name: Install h5py, h5dump and the probe's codec libraries
+        # hdf5-tools: h5dump for the CVE-corpus comparison. libaec-dev and
+        # pkg-config: the probe builds clawhdf5-format with `szip` (the core
+        # crates' default build needs neither).
+        run: |
+          apt-get update
+          apt-get install -y --no-install-recommends python3 python3-venv hdf5-tools libaec-dev pkg-config
+          python3 -m venv /opt/conformance
+          /opt/conformance/bin/pip install --no-cache-dir -r conformance/requirements.txt
+          /opt/conformance/bin/python -c "import h5py, hdf5plugin; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'hdf5plugin', hdf5plugin.version)"
+          h5dump --version
+      - name: Probe unit tests
+        run: cargo test --release --manifest-path conformance/probe/Cargo.toml
+        env:
+          CARGO_TARGET_DIR: conformance/.cache/target
+      - name: Sweep
+        # The corpora come from GitHub (pinned commits, conformance/corpus.txt),
+        # so this job needs a runner that reaches github.com.
+        env:
+          CLAWHDF5_PYTHON: /opt/conformance/bin/python
+        run: bash conformance/run.sh
+      - name: Report
+        if: always()
+        run: |
+          if [ -f CONFORMANCE.md ]; then cat CONFORMANCE.md; else echo "no report was generated"; fi
+          if [ -f conformance/.cache/results/summary.md ]; then
+            echo; echo "---- per-file detail (conformance/.cache/results/summary.md) ----"
+            cat conformance/.cache/results/summary.md
+          fi

CHANGELOG.md

@@ -197,6 +197,23 @@
   takes `--f32`; it had kept printing "f32" after the default changed.
 
 ### Interop
+- **Conformance sweep in the repo** (`conformance/`, report in
+  `CONFORMANCE.md`). `conformance/run.sh` fetches eight public HDF5 corpora
+  pinned by commit (libhdf5's test files, the HDF Group's CVE reproducers,
+  pyfive, netcdf-c, netcdf4-python, h5wasm, h5py, xarray-data) into a
+  gitignored cache, reads every file with clawhdf5 and with h5py/libhdf5 (and
+  the CVE files with h5dump) under a timeout and memory limit, compares them
+  object by object and regenerates the report — about 30 s once the corpus is
+  cached. A nightly Gitea job (`.gitea/workflows/conformance.yml`) runs it and
+  fails on any panic, hang, crash or out-of-memory, or when a file in
+  `conformance/baseline.json` stops reading identically. First report, on
+  42b81d9: 467 of 697 files identical to h5py, 123 our-error, 15 mismatch
+  (2 of them an h5py bug), 92 that libhdf5 cannot read, no panics, hangs or
+  crashes. Compared with the ad-hoc audit sweep, the probe now compares
+  N-Bit floats (and integers with a bit offset) as the values libhdf5
+  converts them to rather than raw file bytes — 8 files that were reported as
+  mismatches read identically — and the reference side no longer flips
+  between runs when libhdf5 aborts while freeing h5py objects.
 - `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
   libhdf5.** The float datatype encoder hard-coded the sign bit's position to
   63, correct only for `f64`; libhdf5 validates it and refused the dataset. It
@@ -236,6 +253,12 @@
   - A pipeline with Fletcher32 ahead of the compressor (h5py
     `set_fletcher32()` then `set_deflate()`) no longer fails with "deflate:
     output exceeds size limit".
+- `clawhdf5-format`: **HDF5 1.4/1.6-era files are readable.** Data Layout
+  message versions 1 and 2 (compact, contiguous, and chunked through the
+  version-1 B-tree) failed with `InvalidLayoutVersion` — 84 of the 686 files in
+  the 2026-09-25 audit sweep, 205 datasets. They now read as libhdf5 does;
+  checked byte for byte against h5py on HDF5's own test files
+  (`tests/legacy_format_interop.rs`).
 
 ### Storage
 - `clawhdf5-format`: **half-precision datasets.**
@@ -273,6 +296,90 @@
 - CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
 
 ### Correctness
+- `clawhdf5-format` VDS: variable-length and reference data from a source in
+  another file is refused. Those elements are global-heap IDs and object
+  addresses in the source file; copied into the virtual dataset they would
+  be decoded against the wrong file and name another object.
+- `clawhdf5-agent`: a store whose `/meta` has an attribute that cannot be
+  decoded fails to open (`MemoryError::Schema`). With `attrs()` now leaving
+  unreadable attributes out, it would otherwise have opened with defaults in
+  place of its settings (`float16`, `compression`, the WAL mark, ...).
+- `clawhdf5-format` reader: an old-style group whose local heap has a free
+  list pointing outside the heap was listed with names read from the broken
+  heap (garbage names on `cve-2021-36977.h5` once its user block was
+  applied). libhdf5 refuses such a heap ("bad heap free list"); so do we now,
+  with `FormatError::InvalidLocalHeapFreeList`. As in libhdf5 the free list
+  is checked when the first name is read (`LocalHeap::validate_free_list`,
+  new), so an empty group with a damaged heap still lists as empty.
+- **Files with a user block** (`h5py.File(..., userblock_size=N)`, `h5jam`;
+  the superblock at 512, 1024, …) could not be read: every address in the
+  file is relative to the superblock, but it was applied from byte 0
+  (`InvalidObjectHeaderVersion` on the root group). `File` (mmap, buffered,
+  `from_bytes`), `MmapFile`, `LazyFile`, `AsyncHDF5File`, the VOL readers,
+  the HNSW loader and external VDS sources now view the file from the
+  superblock on, using the signature's position as the base address as
+  libhdf5 does; `user_block_size()` reports the user block (h5py's
+  `userblock_size`), and `as_bytes()` returns the bytes from the superblock
+  on. **Breaking (format crate):** `Superblock::parse` refuses a non-zero
+  signature offset with `FormatError::UserBlockNotStripped`, since the
+  addresses it returns would be applied to the wrong bytes; pass the slice
+  from `signature::split_user_block` (new) and parse at offset 0.
+- `clawhdf5-format` reader: version-1 shared messages (HDF5 1.6-era files,
+  e.g. a dataset using a committed datatype in libhdf5's `tcompound.h5`)
+  read the heap-offset field of the embedded symbol-table entry as the
+  target address and failed with `InvalidObjectHeaderVersion`. The address
+  is now read after it, as libhdf5 does. **Breaking (format crate):**
+  `shared_message::parse_shared_ref` takes `length_size`. A reference whose
+  target header has no message of the referenced type is now
+  `FormatError::SharedMessageTargetMissing` instead of returning the first
+  other message found there (which decoded as garbage).
+- `clawhdf5-format` reader: array members of version-1 compound datatypes
+  (HDF5 1.6-era files, e.g. libhdf5's `tcompound.h5`) were read as a single
+  element: a `[4] i32` member came back as one `i32`, with the wrong size.
+  The legacy per-member dimension fields are now decoded into an array type,
+  as libhdf5 does; more than four dimensions, or a zero-sized one, is an
+  error.
+- `clawhdf5-format` virtual datasets (VDS), checked against HDF5 2.0 through
+  h5py (`crates/clawhdf5/tests/vds_interop.rs`):
+  - **Wrong data:** elements no mapping supplies — unmapped regions, and
+    mappings whose source file or dataset is missing — read as 0 instead of
+    the virtual dataset's fill value (e.g. h5py `fillvalue=-1`). Assembly moved
+    to the new `vds` module: `vds::read_virtual_dataset` takes the fill value
+    and a resolver that can refuse a name (`VdsFileResolver`), and `File`
+    passes the dataset's fill value. A missing source *dataset* read as an
+    error; it is fill now, as in libhdf5. Source datasets are read with their
+    own fill value for unallocated chunks, and a source whose datatype differs
+    from the virtual dataset's is an error (libhdf5 converts; we do not).
+    `File` now refuses a source name that leaves the virtual file's directory
+    (`../x.h5`, absolute paths), or any external source of a `File::from_bytes`
+    file, with an error — these used to read as fill.
+    **Behaviour change:** the raw-read API (`read_raw_data_full*`), which has
+    no fill value, now returns an error for a virtual dataset with unmapped
+    elements instead of zeros.
+  - Unlimited and printf-style mappings are supported (all 7 VDS files in the
+    libhdf5 test set are such mappings, e.g. Eiger/Percival detector layouts).
+    `%b` in a source file or dataset name is the block number and `%%` a
+    literal `%` (other `%` sequences are an error, as in libhdf5); block `j`
+    is read from the source named with `j`, probing from 0 up to the first
+    missing source. Unlimited source/virtual selections cover as much as the
+    source's current extent fills, including a partial last block. As
+    libhdf5 does on `H5Dget_space`, the extent is recomputed from the sources
+    present (default "last available" view, printf gap 0) —
+    `vds::virtual_dataset_extent`, used by `Dataset::shape()` — so e.g.
+    `vds-eiger.h5` is `[5, 10, 10]`, not its stored `[20, 10, 10]`. A source
+    stored in the other byte order is byte-swapped (libhdf5 converts);
+    other type conversions remain an error.
+  - Hyperslab selection versions 1 and 2 were refused ("only version-3
+    hyperslab selections are supported"). Version 1 is what libhdf5 writes for
+    every VDS created with the default format bounds (h5py's default), so
+    those could not be read at all; version 2 is its encoding of an unlimited
+    selection. Both are decoded now, as are irregular hyperslabs (a union of
+    blocks, read in row-major order as libhdf5 iterates them).
+    `SerializedSelection` exposes the raw form, including unlimited counts.
+  - The version-1 mapping list HDF5 2.0 writes (low version bound 2.0) was
+    misparsed: each entry's flags byte was read as the start of the source
+    file name, and names shared with an earlier entry (stored as that entry's
+    index) were not followed. Now decoded as `H5D__virtual_load_layout` does.
 - `clawhdf5-format` reader — **values returned wrong with no error:**
   - Fixed Array and Extensible Array chunk indexes were laid out by the
     dataset's current shape instead of its max shape (23 libhdf5 test files,
@@ -294,11 +401,77 @@
   - Two threads reading two chunked datasets through one `File` could get each
     other's chunks (the shared chunk cache was switched between datasets
     across separate lock acquisitions). The cache is now keyed by dataset.
+  - Compound datatype version 1 members with legacy array dimensions (HDF5
+    before 1.4, which had no array class) were read as a single scalar at
+    the member's offset; they are now array members, as in libhdf5
+    (`tarrold.h5`, `tcompound.h5`). Only reachable once layout versions 1/2
+    were readable, since the files that use it are that old.
+- `clawhdf5-format` reader — errors on valid files: a version-1 shared
+  message (a committed datatype in HDF5 1.4/1.6-era files) was read as if the
+  object header address followed the reserved bytes; it follows a link-name
+  offset (the reference is an old-style symbol table entry), so the reader
+  followed the name offset and failed with `InvalidObjectHeaderVersion`
+  (`tcompound.h5`). New `shared_message::parse_shared_ref_sized` takes the
+  superblock's length size; `parse_shared_ref` assumes it equals the offset
+  size.
 - `clawhdf5-format` reader — errors on valid files: enum and bool datasets
   through the numeric readers; the "don't filter partial edge chunks" layout
   flag; Fletcher32 ahead of deflate (NetCDF-4's order). Unknown-message flags
   follow libhdf5 (`tbogus.h5`): "fail if unknown" is refused, "fail if unknown
   and writing" is ignored by a reader.
+- `clawhdf5-format` reader — dense groups and attributes (links or
+  attributes kept in a fractal heap indexed by a v2 B-tree):
+  - A link heap larger than the root indirect block's direct rows (512 KiB
+    with libhdf5's defaults: a few thousand long link names, or ~20 000 short
+    ones) could not be listed: child indirect blocks were given the wrong
+    number of rows, so every link stored in one was unreachable.
+  - v2 B-trees of depth 3 or more (a dense group of ~22 000+ links) were
+    misparsed: internal-node child pointers were read with widths from an
+    estimate instead of libhdf5's per-depth record capacities, and the
+    listing failed. The same B-tree code indexes dense attributes, shared
+    messages and chunks.
+  - Fractal-heap "huge" objects (larger than the heap's managed-object
+    limit, 4 KiB by default — e.g. an 8 KiB dense attribute or a link with a
+    very long name) and "tiny" objects are now read; the ID type was taken
+    from the wrong bits (6-7, the version, instead of 4-5), so a huge object
+    failed and took every attribute on its object down with it (NetCDF-4
+    files such as netcdf4-python's `issue671.nc`). Huge objects are found
+    directly from the ID or through the huge-object v2 B-tree, filtered or
+    not.
+  - Heaps with an I/O filter pipeline (a group created with a filter on its
+    creation property list compresses its link heap) are now read: the
+    header's pipeline was skipped with the wrong size, so its checksum was
+    looked for in the wrong place, and filtered direct blocks were read raw.
+  - A user-defined link (link class 65-255, e.g. 187 in libhdf5's
+    `tall.h5`/`tudlink.h5`) made its whole group unlistable. Such links
+    cannot be followed without the application that registered the class, so
+    they are now left out of `datasets()`/`groups()` and path lookup, as h5py
+    leaves out links it cannot open; reserved link types are still an error.
+- `clawhdf5` — soft links are listed, as h5py lists them: `datasets()` and
+  `groups()` on `Group`/`MmapGroup`/`LazyGroup` include each soft link under
+  its own name as the kind of object it resolves to, and `dataset(name)` /
+  `group(name)` open through it. Relative targets resolve from the group
+  holding the link. Dangling or cyclic soft links, external links and
+  user-defined links are left out (h5py lists their names but cannot open
+  them). Previously soft links were missing from the listings, and in
+  old-style (symbol table) groups a soft link made the listing fail. New
+  `group_v2::resolve_group_children` / `resolve_path_from` and
+  `group_v1::v1_soft_links` in `clawhdf5-format`.
+- `clawhdf5` — one unreadable attribute no longer fails `attrs()` for every
+  attribute on its object: it is left out of the map, and the new
+  `attrs_with_errors()` (on every group and dataset handle) returns the map
+  plus one error per attribute left out. Returned values are always complete.
+  An error in the attribute index itself (attribute info message, dense heap
+  header or B-tree) still fails the call. `clawhdf5-format` gains
+  `attribute::extract_attributes_tolerant`; `extract_attributes_full` stays
+  strict.
+- `clawhdf5-format` reader — files with shared object header messages
+  (SOHM, `H5Pset_shared_mesg_index`): a datatype, dataspace, filter pipeline
+  or attribute stored in the file's SOHM heap failed with "invalid shared
+  message version: 2" — only shared fill values loaded the SOHM table — so
+  such files' datasets and attributes could not be read.
+  `shared_message::resolve_shared_message` now loads the table when a
+  reference needs it (36 cases of the audit's read matrix).
 - `clawhdf5-format` writer — **files libhdf5 rejects or reads wrong:**
   - Extensible Array (one unlimited dimension): chunks from index 244 on were
     written but never indexed and read as 0, by libhdf5 and by us.

CONFORMANCE.md

@@ -0,0 +1,300 @@
+# clawhdf5 conformance report
+
+Every HDF5 file of eight public corpora (pinned by commit) is read twice — by
+clawhdf5 (`conformance/probe`, the same `clawhdf5-format` calls the facade
+makes) and by h5py/libhdf5 (`conformance/ref.py`) — and the two readings are
+compared object by object: the set of hard-linked objects, each dataset's and
+attribute's shape, and a SHA-256 of its values in a canonical encoding. The
+CVE corpus is also run through `h5dump`. Each side runs under a timeout and an
+address-space limit, so a hang, crash or runaway allocation is recorded, not
+fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.
+
+## Run
+
+| | |
+|---|---|
+| date | 2026-09-26 03:46 UTC |
+| clawhdf5 commit | `10d1029ead524e2fe64c2cd7f61b28067d9e449c` |
+| machine | `tank`: AMD Ryzen 7 7800X3D 8-Core Processor, 16 CPUs, 61 GiB, Linux 7.0.0-34-generic x86_64 |
+| command | `conformance/run.sh --no-fetch --update-baseline` |
+| rustc | rustc 1.98.1 (48a229cea 2026-09-01) |
+| reference | h5py 3.16.0, HDF5 2.0.0, numpy 2.5.3, hdf5plugin 7.1.0, Python 3.14.4 |
+| h5dump | Version 1.14.6 (CVE corpus only) |
+| limits | 20 s timeout (SIGKILL), 4096 MiB address space, per process; 16 files in parallel |
+| runtime | 22 s probing + comparing (0 s fetch/build before it) |
+
+## Results
+
+A file's class is the first that applies:
+
+- **panic / hang / crash / oom** — clawhdf5 panicked (caught per object or not), hit the timeout, died on a signal, or failed an allocation. The CI gate fails on any of these.
+- **h5py-cannot-read** — libhdf5 could not open the file (or itself crashed or hung). Nothing to compare against; most are the deliberately malformed CVE reproducers.
+- **our-error** — clawhdf5 returned an error for something h5py reads.
+- **mismatch** — both read it, but the shapes, values, object set or attribute set differ.
+- **ok** — every object h5py reads, clawhdf5 reads identically.
+
+| corpus | files | ok | our-error | mismatch | h5py-cannot-read | panic | hang | crash | oom |
+|---|---|---|---|---|---|---|---|---|---|
+| NCAS-CMS_pyfive | 33 | 32 | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
+| cve_hdf5 | 147 | 100 | 6 | 9 | 32 | 0 | 0 | 0 | 0 |
+| h5py_data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| hdf5 | 466 | 386 | 8 | 12 | 60 | 0 | 0 | 0 | 0 |
+| netcdf-c | 20 | 20 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| netcdf4-python | 18 | 18 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| usnistgov_h5wasm | 5 | 5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| xarray-data | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
+| **all** | **697** | **569** | **14** | **22** | **92** | **0** | **0** | **0** | **0** |
+
+2 of the 22 mismatches are a known h5py bug, not ours (see *Known not-our-bug*).
+
+Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):
+
+| corpus | source | commit |
+|---|---|---|
+| hdf5 | https://github.com/HDFGroup/hdf5 | `a3cf1ea82cc7` |
+| cve_hdf5 | https://github.com/HDFGroup/cve_hdf5 | `3fd1f5ae3869` |
+| netcdf-c | https://github.com/Unidata/netcdf-c | `beb7b9585273` |
+| NCAS-CMS_pyfive | https://github.com/NCAS-CMS/pyfive | `8cf07b874913` |
+| usnistgov_h5wasm | https://github.com/usnistgov/h5wasm | `02f6336527d2` |
+| netcdf4-python | https://github.com/Unidata/netcdf4-python | `6e67576d39ae` |
+| xarray-data | https://github.com/pydata/xarray-data | `a35297e9da2c` |
+| h5py_data | https://github.com/h5py/h5py (`h5py/tests/data_files`) | `b2f0347c4200` |
+
+## Panics, hangs, crashes, out-of-memory
+
+None.
+
+## Our-error root causes
+
+Grouped by normalised error message. *files* counts files whose class this cause affects.
+
+| files | objects | error | examples |
+|---:|---:|---|---|
+| 6 | 6 | `UnsupportedFilter(N)` | `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_blosc2.h5`, `hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bshuf.h5` (+3 more) |
+| 3 | 3 | `DataSizeMismatch { expected: N, actual: N }` | `cve_hdf5/cvefiles/cve-2020-18494.h5`, `cve_hdf5/cvefiles/cve-2024-32623.h5`, `cve_hdf5/cvefiles/cve-2025-2309.h5` |
+| 2 | 2 | `ChunkedReadError("…")` | `cve_hdf5/cvefiles/cve-2025-2308.h5`, `hdf5/test/testfiles/bad_nbit_parms_walk.h5` |
+| 1 | 1 | `UnexpectedEof { expected: N, available: N }` | `cve_hdf5/cvefiles/cve-2019-9151.h5` |
+| 1 | 1 | `MissingMessage(Dataspace)` | `cve_hdf5/cvefiles/cve-2024-33874.h5` |
+| 1 | 1 | `InvalidObjectHeaderVersion(N)` | `hdf5/tools/test/testfiles/h5clear_mdc_image.h5` |
+
+## Mismatch root causes
+
+| files | objects | cause | examples |
+|---:|---:|---|---|
+| 13 | 14 | `missing-object` | `cve_hdf5/cvefiles/cve-2019-8397.h5`, `cve_hdf5/cvefiles/cve-2019-8398.h5`, `cve_hdf5/cvefiles/cve-2021-46243.h5` (+10 more) |
+| 3 | 7 | `extra-attr` | `cve_hdf5/cvefiles/cve-2018-17438`, `cve_hdf5/cvefiles/cve-2018-17439`, `cve_hdf5/cvefiles/cve-2024-33874.h5` |
+| 3 | 6 | `extra-object` | `cve_hdf5/cvefiles/cve-2021-46244.h5`, `hdf5/tools/test/testfiles/h5clear_fsm_persist_less.h5`, `hdf5/tools/test/testfiles/h5stat_err_refcount.h5` |
+| 1 | 1 | `attr-values: ours=vlen(>u8) h5py=object layout=- filters=-` | `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5` |
+| 1 | 1 | `values: ours=<f4 h5py=float32 layout=chunked filters=-` | `cve_hdf5/cvefiles/cve-2025-44904.h5` |
+| 1 | 1 | `values: ours=>i2 h5py=>i2 layout=chunked filters=[6]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
+| 1 | 1 | `values: ours=>f4 h5py=>f4 layout=chunked filters=[2]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
+| 1 | 1 | `values: ours=<f4 h5py=float32 layout=chunked filters=[2]` | `cve_hdf5/cvefiles/cve-2025-44905.h5` |
+| 1 | 1 | `values: ours=((<i4)[6, 3])[4] h5py=(('<i4', (6, 3)), (4,)) layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tarray3.h5` |
+| 1 | 1 | `values: ours=vlen({r:>f4,i:>f4}8) h5py=object layout=contiguous filters=-` | `hdf5/tools/test/testfiles/tcomplex_be.h5` |
+
+## CVE corpus: clawhdf5 vs h5dump vs h5py
+
+The 147 files of [HDFGroup/cve_hdf5](https://github.com/HDFGroup/cve_hdf5) — reproducers for
+published libhdf5 CVEs and fuzzer finds. *read* = produced output (possibly with per-object
+errors), *error* = refused cleanly. h5dump exits non-zero on any error anywhere in a file, so
+its read/error split is not comparable with the other two rows; the panic, crash, hang and oom
+columns are.
+
+| tool | read | error | panic | crash | hang | oom |
+|---|---:|---:|---:|---:|---:|---:|
+| clawhdf5 | 142 | 5 | 0 | 0 | 0 | 0 |
+| h5dump 1.14.6 | 16 | 129 | 0 | 2 | 0 | 0 |
+| h5py 3.16.0 / HDF5 2.0.0 | 115 | 31 | 0 | 1 | 0 | 0 |
+
+<details><summary>Per-file outcomes</summary>
+
+| file | h5dump | h5py | clawhdf5 | class |
+|---|---|---|---|---|
+| cvefiles/cve-2016-4330.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2016-4331.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
+| cvefiles/cve-2016-4332-mtime-new.h5 | error exit | read 25 obj, 1 errors | read 25 obj | ok |
+| cvefiles/cve-2016-4332-mtime.h5 | error exit | read 4 obj, 3 errors | read 4 obj | ok |
+| cvefiles/cve-2016-4332-stab.h5 | error exit | open error | read 65 obj | h5py-cannot-read |
+| cvefiles/cve-2016-4333.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2017-17505.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2017-17506.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2017-17507.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2017-17508.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
+| cvefiles/cve-2017-17509.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2018-11202.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2018-11203.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2018-11204.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
+| cvefiles/cve-2018-11205.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
+| cvefiles/cve-2018-11206-new.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-11206-old.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-11207.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2018-13866.h5 | error exit | open error | open error | h5py-cannot-read |
+| cvefiles/cve-2018-13867.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2018-13868.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-13869.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2018-13870.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2018-13871.h5 | error exit | read 2 obj | read 2 obj | ok |
+| cvefiles/cve-2018-13872.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2018-13873.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
+| cvefiles/cve-2018-13874.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2018-13875.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-13876.h5 | error exit | open error | read 2 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2018-14031.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-14033.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-14034.h5 | error exit | read 1 obj, 2 errors | read 1 obj | ok |
+| cvefiles/cve-2018-14035.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-14460.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
+| cvefiles/cve-2018-15671.h5 | ok | read 1 obj | read 1 obj | ok |
+| cvefiles/cve-2018-15672.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2018-16438.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
+| cvefiles/cve-2018-17233.h5 | error exit | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
+| cvefiles/cve-2018-17234.h5 | error exit | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
+| cvefiles/cve-2018-17237.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2018-17432.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-17433 | error exit | open error | open error | h5py-cannot-read |
+| cvefiles/cve-2018-17434.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-17435.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-17436 | error exit | open error | open error | h5py-cannot-read |
+| cvefiles/cve-2018-17437.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2018-17438 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | mismatch |
+| cvefiles/cve-2018-17439 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | mismatch |
+| cvefiles/cve-2019-8396.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
+| cvefiles/cve-2019-8397.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
+| cvefiles/cve-2019-8398.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
+| cvefiles/cve-2019-9151.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 2 errors | our-error |
+| cvefiles/cve-2019-9152.h5 | error exit | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2020-10809 | error exit | open error | open error | h5py-cannot-read |
+| cvefiles/cve-2020-10810.h5 | error exit | open error | read 2 obj | h5py-cannot-read |
+| cvefiles/cve-2020-10811.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | ok |
+| cvefiles/cve-2020-10812.h5 | error exit | open error | read 2 obj | h5py-cannot-read |
+| cvefiles/cve-2020-18232.h5 | error exit | read 3 obj, 2 errors | read 3 obj, 2 errors | ok |
+| cvefiles/cve-2020-18494.h5 | ok | read 2 obj | read 2 obj, 1 errors | our-error |
+| cvefiles/cve-2021-36977.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2021-37501.h5 | error exit | read 18 obj, 1 errors | read 18 obj, 1 errors | ok |
+| cvefiles/cve-2021-45829.h5 | error exit | read 1 obj, 2 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2021-45830.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2021-45833.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2021-46242.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2021-46243.h5 | error exit | read 3 obj, 2 errors | read 2 obj, 1 errors | mismatch |
+| cvefiles/cve-2021-46244.h5 | error exit | read 2 obj, 1 errors | read 6 obj, 3 errors | mismatch |
+| cvefiles/cve-2024-29157.h5 | error exit | read 4 obj, 7 errors | read 4 obj, 7 errors | ok |
+| cvefiles/cve-2024-29158.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2024-29159.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2024-29160.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
+| cvefiles/cve-2024-29161.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 2 errors | ok |
+| cvefiles/cve-2024-29162.h5 | error exit | read 17 obj, 4 errors | read 17 obj, 3 errors | ok |
+| cvefiles/cve-2024-29163.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
+| cvefiles/cve-2024-29164.h5 | ok | read 3 obj | read 3 obj | ok |
+| cvefiles/cve-2024-29165.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2024-29166.h5 | error exit | read 17 obj, 2 errors | read 17 obj | ok |
+| cvefiles/cve-2024-32605.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | ok |
+| cvefiles/cve-2024-32606.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2024-32607-1.h5 | ok | read 10 obj | read 10 obj | ok |
+| cvefiles/cve-2024-32607-2.h5 | error exit | read 9 obj, 1 errors | read 9 obj, 1 errors | ok |
+| cvefiles/cve-2024-32608.h5 | error exit | read 6 obj, 1 errors | read 6 obj | ok |
+| cvefiles/cve-2024-32609.h5 | error exit | SIGSEGV | read 3 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2024-32610.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/cve-2024-32611.h5 | ok | read 6 obj | read 6 obj | ok |
+| cvefiles/cve-2024-32612.h5 | ok | read 3 obj | read 3 obj | ok |
+| cvefiles/cve-2024-32613.h5 | error exit | read 7 obj, 1 errors | read 7 obj, 1 errors | ok |
+| cvefiles/cve-2024-32614.h5 | error exit | read 25 obj, 2 errors | read 25 obj, 1 errors | ok |
+| cvefiles/cve-2024-32615.h5 | error exit | read 4 obj, 1 errors | read 4 obj, 1 errors | ok |
+| cvefiles/cve-2024-32616.h5 | error exit | read 10 obj, 7 errors | read 10 obj, 5 errors | ok |
+| cvefiles/cve-2024-32617.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2024-32618.h5 | error exit | read 4 obj, 2 errors | read 3 obj | mismatch |
+| cvefiles/cve-2024-32619.h5 | error exit | read 3 obj, 2 errors | read 3 obj | ok |
+| cvefiles/cve-2024-32620.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2024-32621.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2024-32622.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2024-32623.h5 | ok | read 6 obj | read 6 obj, 1 errors | our-error |
+| cvefiles/cve-2024-32624.h5 | error exit | read 6 obj, 1 errors | read 6 obj | ok |
+| cvefiles/cve-2024-33873.h5 | error exit | read 4 obj, 1 errors | read 4 obj | ok |
+| cvefiles/cve-2024-33874.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | our-error |
+| cvefiles/cve-2024-33875.h5 | ok | read 2 obj | read 2 obj | ok |
+| cvefiles/cve-2024-33876.h5 | ok | read 3 obj, 1 errors | read 3 obj, 1 errors | ok |
+| cvefiles/cve-2024-33877.h5 | error exit | read 8 obj, 1 errors | read 8 obj, 1 errors | ok |
+| cvefiles/cve-2025-2153.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2308.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 2 errors | our-error |
+| cvefiles/cve-2025-2309.h5 | ok | read 6 obj, 1 errors | read 6 obj, 1 errors | our-error |
+| cvefiles/cve-2025-2310.h5 | error exit | read 24 obj, 8 errors | read 24 obj, 8 errors | ok |
+| cvefiles/cve-2025-2912.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2913.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2914.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2915.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2923.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-2924.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2025-2925.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2025-2926.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-44904.h5 | error exit | read 25 obj, 1 errors | read 25 obj, 1 errors | mismatch |
+| cvefiles/cve-2025-44905.h5 | error exit | read 25 obj, 3 errors | read 25 obj, 3 errors | mismatch |
+| cvefiles/cve-2025-6269-1.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2025-6269-2.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2025-6269-3.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2025-6269-4.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2025-6270-1.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6270-2.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6270-3.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6516.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2025-6750.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6816.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6817.h5 | error exit | open error | read 1 obj | h5py-cannot-read |
+| cvefiles/cve-2025-6818.h5 | error exit | open error | read 1 obj | h5py-cannot-read |
+| cvefiles/cve-2025-6856.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-6857.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2025-6858.h5 | SIGSEGV | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-7067.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2025-7068.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2025-7069.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| cvefiles/cve-2026-26200.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| cvefiles/cve-2026-34734.h5 | error exit | read 2 obj, 1 errors | read 2 obj | ok |
+| cvefiles/cve-2026-92627.h5 | error exit | read 2 obj, 1 errors | read 2 obj, 1 errors | ok |
+| cvefiles/unknown-1.h5 | error exit | read 11 obj, 1 errors | read 11 obj, 1 errors | ok |
+| fuzzerfiles/gh-4431-poc-03.h5 | error exit | read 1 obj | read 1 obj | ok |
+| fuzzerfiles/gh-4432-poc-05.h5 | SIGSEGV | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| fuzzerfiles/gh-4433-poc-08.h5 | error exit | read 1 obj, 1 errors | read 1 obj | ok |
+| fuzzerfiles/gh-4434-poc-09.h5 | error exit | open error | read 1 obj, 1 errors | h5py-cannot-read |
+| fuzzerfiles/gh-4435-poc-10.h5 | error exit | read 1 obj, 1 errors | read 1 obj, 1 errors | ok |
+| fuzzerfiles/gh-4585.h5 | error exit | open error | open error | h5py-cannot-read |
+| fuzzerfiles/gh_2649_flawed.h5 | error exit | read 9 obj, 1 errors | read 9 obj, 1 errors | ok |
+| fuzzerfiles/gh_2649_plain_model.h5 | ok | read 10 obj | read 10 obj | ok |
+
+</details>
+
+## Known not-our-bug
+
+- **h5py big-endian variable-length sequences.** h5py returns the elements of a VL sequence
+  whose base type is big-endian with the file's big-endian bytes but a native (little-endian)
+  numpy dtype, so the values it reports are byte-swapped garbage; `h5dump` prints the values
+  clawhdf5 reads. Reproducer: `h5py.vlen_dtype(np.dtype('>f4'))` dataset holding `[1.0, 2.0]`
+  reads back in h5py as `[4.6e-41, 9.0e-44]`. Affected here: `NCAS-CMS_pyfive/tests/data/attr_datatypes.hdf5`, `hdf5/tools/test/testfiles/tcomplex_be.h5`.
+- **Non-IEEE floats and partial-precision integers (N-Bit).** libhdf5 converts a float whose
+  bit layout is not IEEE (e.g. `H5Tset_precision` for the N-Bit filter) or an integer with a
+  bit offset / reduced precision into the plain numpy type of the same size. The probe
+  compares such values as converted numbers, not raw file bytes (before 2026-09-25 it compared
+  raw bytes, which reported every N-Bit float dataset as a mismatch).
+- **Types h5py widens.** Where h5py reads a type into a numpy type of a different size
+  (FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not
+  compared (shape and presence still are): dataset file type size 1 -> numpy float16 (2) (15x), attr file type size 1 -> numpy float16 (2) (15x), dataset file type size 2 -> numpy float32 (4) (2x), dataset file type size 8 -> numpy float128 (16) (1x), dataset file type size 12 -> numpy float128 (16) (1x), attr file type size 2 -> numpy float32 (4) (1x), dataset file type size 2 -> numpy >f4 (4) (1x), attr file type size 2 -> numpy >f4 (4) (1x).
+- **References** are compared by presence only (`R`), not by target.
+
+## Objects h5py fails on but clawhdf5 reads
+
+- 19 x `KeyError: '…'`
+- 19 x `OSError: Can't synchronously read data (no appropriate function for conversion path)`
+- 1 x `TypeError: unhandled dtype kind M (dtype('…'))`
+- 1 x `OSError: Can't synchronously read data (bad coordinate offset)`
+- 1 x `TypeError: No NumPy equivalent for TypeTimeID exists`
+- 1 x `KeyError: "…"`
+- 1 x `ValueError: Insufficient precision in available types to represent (N, N, N, N, N)`
+
+## Reproduce
+
+```sh
+# needs: Rust, python3 with h5py numpy hdf5plugin (conformance/requirements.txt), h5dump (hdf5-tools), git
+CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh
+```
+
+The corpus (about 450 MB of sparse checkouts) is cached in `conformance/.cache/`; results for
+every file, both sides' raw JSON and stderr, are in `conformance/.cache/results/`.
+`conformance/baseline.json` holds the ok files the nightly CI job (`.gitea/workflows/conformance.yml`)
+must keep; `conformance/run.sh --update-baseline` rewrites it.

conformance/.gitignore

@@ -0,0 +1,3 @@
+/.cache/
+# pin the probe's dependencies (the workspace lock is not committed)
+!/probe/Cargo.lock

conformance/README.md

@@ -0,0 +1,39 @@
+# Conformance sweep
+
+Reads every HDF5 file of eight public corpora with clawhdf5 and with
+h5py/libhdf5, compares the two readings object by object, and writes
+[`CONFORMANCE.md`](../CONFORMANCE.md).
+
+```sh
+CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh      # ~30 s once the corpus is cached
+conformance/run.sh --update-baseline                             # after an intended change in results
+```
+
+Needs Rust, `git`, `h5dump` (Debian/Ubuntu `hdf5-tools`), `libaec` (for the
+probe's `szip` feature; `libaec-dev`), and a Python with the packages in
+`requirements.txt`. The first run downloads about 450 MB of sparse checkouts.
+
+| file | role |
+|---|---|
+| `corpus.txt` | the corpora: git URL, pinned commit, swept root, sparse-checkout patterns |
+| `fetch-corpus.sh` | shallow, sparse, blob-filtered checkout of each pinned commit into `.cache/src/` (gitignored); no-op when already there |
+| `list_files.py` | which files are probed (HDF5/netCDF-4 extensions minus netCDF classic, plus the CVE reproducers) |
+| `probe/` | the clawhdf5 side: a standalone crate (outside the workspace, so `cargo test --workspace` never builds it) that walks a file with `clawhdf5-format` and prints canonical JSON |
+| `ref.py` | the h5py side: the same JSON from h5py |
+| `run_one.sh` | runs both sides on one file (and `h5dump` on the CVE corpus) under a timeout and an address-space limit |
+| `compare.py` | classifies each file (ok / our-error / mismatch / h5py-cannot-read / panic / hang / crash / oom) and groups root causes |
+| `report.py` | writes `CONFORMANCE.md` |
+| `check.py` | the gate: fails on any panic/hang/crash/oom, on an ok count below `baseline.json`, or on a baseline-ok file that is no longer ok |
+| `baseline.json` | the ok files the gate holds the line on |
+| `requirements.txt` | pinned h5py / numpy / hdf5plugin / netCDF4 |
+
+Results for every file (both sides' JSON and stderr, `results.csv`,
+`results.json`, `summary.md`) are left in `.cache/results/`.
+
+The nightly job is `.gitea/workflows/conformance.yml`; it prints the report
+into the job log.
+
+The canonical value encoding both sides hash is documented at the top of
+`probe/src/main.rs`. Values are compared as libhdf5 presents them: a float
+with a non-IEEE bit layout (N-Bit) or an integer with a bit offset is compared
+as the converted number, not as raw file bytes.

conformance/baseline.json

@@ -0,0 +1,618 @@
+{
+ "comment": "conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
+ "commit": "10d1029ead524e2fe64c2cd7f61b28067d9e449c",
+ "date": "2026-09-26 03:46 UTC",
+ "reference": "h5py 3.16.0 / HDF5 2.0.0",
+ "files": 697,
+ "ok": 569,
+ "counts": {
+  "h5py-cannot-read": 92,
+  "mismatch": 22,
+  "ok": 569,
+  "our-error": 14
+ },
+ "per_corpus": {
+  "NCAS-CMS_pyfive": {
+   "mismatch": 1,
+   "ok": 32
+  },
+  "cve_hdf5": {
+   "h5py-cannot-read": 32,
+   "mismatch": 9,
+   "ok": 100,
+   "our-error": 6
+  },
+  "h5py_data": {
+   "ok": 4
+  },
+  "hdf5": {
+   "h5py-cannot-read": 60,
+   "mismatch": 12,
+   "ok": 386,
+   "our-error": 8
+  },
+  "netcdf-c": {
+   "ok": 20
+  },
+  "netcdf4-python": {
+   "ok": 18
+  },
+  "usnistgov_h5wasm": {
+   "ok": 5
+  },
+  "xarray-data": {
+   "ok": 4
+  }
+ },
+ "ok_files": [
+  "NCAS-CMS_pyfive/tests/compact.hdf5",
+  "NCAS-CMS_pyfive/tests/data/btreev2.hdf5",
+  "NCAS-CMS_pyfive/tests/data/chunked.hdf5",
+  "NCAS-CMS_pyfive/tests/data/cmip_bad_eg.nc",
+  "NCAS-CMS_pyfive/tests/data/compressed.hdf5",
+  "NCAS-CMS_pyfive/tests/data/compressed_v1.hdf5",
+  "NCAS-CMS_pyfive/tests/data/dataset_datatypes.hdf5",
+  "NCAS-CMS_pyfive/tests/data/dataset_multidim.hdf5",
+  "NCAS-CMS_pyfive/tests/data/dim_scales.hdf5",
+  "NCAS-CMS_pyfive/tests/data/earliest.hdf5",
+  "NCAS-CMS_pyfive/tests/data/enum_h5variable.hdf5",
+  "NCAS-CMS_pyfive/tests/data/enum_variable.hdf5",
+  "NCAS-CMS_pyfive/tests/data/enum_variable.nc",
+  "NCAS-CMS_pyfive/tests/data/enums_from_netcdf.nc",
+  "NCAS-CMS_pyfive/tests/data/fillvalue_earliest.hdf5",
+  "NCAS-CMS_pyfive/tests/data/fillvalue_latest.hdf5",
+  "NCAS-CMS_pyfive/tests/data/filter_pipeline_v2.hdf5",
+  "NCAS-CMS_pyfive/tests/data/fletcher32.hdf5",
+  "NCAS-CMS_pyfive/tests/data/fractal_heap_no_mci_rlat.nc",
+  "NCAS-CMS_pyfive/tests/data/groups.hdf5",
+  "NCAS-CMS_pyfive/tests/data/h5netcdf_test.hdf5",
+  "NCAS-CMS_pyfive/tests/data/issue23_A.nc",
+  "NCAS-CMS_pyfive/tests/data/issue23_A_contiguous.nc",
+  "NCAS-CMS_pyfive/tests/data/issue23_B.nc",
+  "NCAS-CMS_pyfive/tests/data/latest.hdf5",
+  "NCAS-CMS_pyfive/tests/data/netcdf4_classic.nc",
+  "NCAS-CMS_pyfive/tests/data/new_style_groups.hdf5",
+  "NCAS-CMS_pyfive/tests/data/noy_AERmonZ_UKESM1-0-LL_piControl_r1i1p1f2_gnz_200001-200012.nc",
+  "NCAS-CMS_pyfive/tests/data/references.hdf5",
+  "NCAS-CMS_pyfive/tests/data/resizable.hdf5",
+  "NCAS-CMS_pyfive/tests/opaque_datetime.hdf5",
+  "NCAS-CMS_pyfive/tests/opaque_fixed.hdf5",
+  "cve_hdf5/cvefiles/cve-2016-4330.h5",
+  "cve_hdf5/cvefiles/cve-2016-4331.h5",
+  "cve_hdf5/cvefiles/cve-2016-4332-mtime-new.h5",
+  "cve_hdf5/cvefiles/cve-2016-4332-mtime.h5",
+  "cve_hdf5/cvefiles/cve-2016-4333.h5",
+  "cve_hdf5/cvefiles/cve-2017-17505.h5",
+  "cve_hdf5/cvefiles/cve-2017-17506.h5",
+  "cve_hdf5/cvefiles/cve-2017-17507.h5",
+  "cve_hdf5/cvefiles/cve-2017-17508.h5",
+  "cve_hdf5/cvefiles/cve-2017-17509.h5",
+  "cve_hdf5/cvefiles/cve-2018-11202.h5",
+  "cve_hdf5/cvefiles/cve-2018-11203.h5",
+  "cve_hdf5/cvefiles/cve-2018-11204.h5",
+  "cve_hdf5/cvefiles/cve-2018-11205.h5",
+  "cve_hdf5/cvefiles/cve-2018-11206-new.h5",
+  "cve_hdf5/cvefiles/cve-2018-11206-old.h5",
+  "cve_hdf5/cvefiles/cve-2018-11207.h5",
+  "cve_hdf5/cvefiles/cve-2018-13867.h5",
+  "cve_hdf5/cvefiles/cve-2018-13868.h5",
+  "cve_hdf5/cvefiles/cve-2018-13869.h5",
+  "cve_hdf5/cvefiles/cve-2018-13870.h5",
+  "cve_hdf5/cvefiles/cve-2018-13871.h5",
+  "cve_hdf5/cvefiles/cve-2018-13872.h5",
+  "cve_hdf5/cvefiles/cve-2018-13873.h5",
+  "cve_hdf5/cvefiles/cve-2018-13875.h5",
+  "cve_hdf5/cvefiles/cve-2018-14031.h5",
+  "cve_hdf5/cvefiles/cve-2018-14033.h5",
+  "cve_hdf5/cvefiles/cve-2018-14034.h5",
+  "cve_hdf5/cvefiles/cve-2018-14035.h5",
+  "cve_hdf5/cvefiles/cve-2018-14460.h5",
+  "cve_hdf5/cvefiles/cve-2018-15671.h5",
+  "cve_hdf5/cvefiles/cve-2018-15672.h5",
+  "cve_hdf5/cvefiles/cve-2018-16438.h5",
+  "cve_hdf5/cvefiles/cve-2018-17233.h5",
+  "cve_hdf5/cvefiles/cve-2018-17234.h5",
+  "cve_hdf5/cvefiles/cve-2018-17237.h5",
+  "cve_hdf5/cvefiles/cve-2018-17432.h5",
+  "cve_hdf5/cvefiles/cve-2018-17434.h5",
+  "cve_hdf5/cvefiles/cve-2018-17435.h5",
+  "cve_hdf5/cvefiles/cve-2018-17437.h5",
+  "cve_hdf5/cvefiles/cve-2019-8396.h5",
+  "cve_hdf5/cvefiles/cve-2019-9152.h5",
+  "cve_hdf5/cvefiles/cve-2020-10811.h5",
+  "cve_hdf5/cvefiles/cve-2020-18232.h5",
+  "cve_hdf5/cvefiles/cve-2021-36977.h5",
+  "cve_hdf5/cvefiles/cve-2021-37501.h5",
+  "cve_hdf5/cvefiles/cve-2021-45829.h5",
+  "cve_hdf5/cvefiles/cve-2021-45833.h5",
+  "cve_hdf5/cvefiles/cve-2024-29157.h5",
+  "cve_hdf5/cvefiles/cve-2024-29158.h5",
+  "cve_hdf5/cvefiles/cve-2024-29159.h5",
+  "cve_hdf5/cvefiles/cve-2024-29160.h5",
+  "cve_hdf5/cvefiles/cve-2024-29161.h5",
+  "cve_hdf5/cvefiles/cve-2024-29162.h5",
+  "cve_hdf5/cvefiles/cve-2024-29163.h5",
+  "cve_hdf5/cvefiles/cve-2024-29164.h5",
+  "cve_hdf5/cvefiles/cve-2024-29165.h5",
+  "cve_hdf5/cvefiles/cve-2024-29166.h5",
+  "cve_hdf5/cvefiles/cve-2024-32605.h5",
+  "cve_hdf5/cvefiles/cve-2024-32606.h5",
+  "cve_hdf5/cvefiles/cve-2024-32607-1.h5",
+  "cve_hdf5/cvefiles/cve-2024-32607-2.h5",
+  "cve_hdf5/cvefiles/cve-2024-32608.h5",
+  "cve_hdf5/cvefiles/cve-2024-32610.h5",
+  "cve_hdf5/cvefiles/cve-2024-32611.h5",
+  "cve_hdf5/cvefiles/cve-2024-32612.h5",
+  "cve_hdf5/cvefiles/cve-2024-32613.h5",
+  "cve_hdf5/cvefiles/cve-2024-32614.h5",
+  "cve_hdf5/cvefiles/cve-2024-32615.h5",
+  "cve_hdf5/cvefiles/cve-2024-32616.h5",
+  "cve_hdf5/cvefiles/cve-2024-32617.h5",
+  "cve_hdf5/cvefiles/cve-2024-32619.h5",
+  "cve_hdf5/cvefiles/cve-2024-32620.h5",
+  "cve_hdf5/cvefiles/cve-2024-32621.h5",
+  "cve_hdf5/cvefiles/cve-2024-32622.h5",
+  "cve_hdf5/cvefiles/cve-2024-32624.h5",
+  "cve_hdf5/cvefiles/cve-2024-33873.h5",
+  "cve_hdf5/cvefiles/cve-2024-33875.h5",
+  "cve_hdf5/cvefiles/cve-2024-33876.h5",
+  "cve_hdf5/cvefiles/cve-2024-33877.h5",
+  "cve_hdf5/cvefiles/cve-2025-2310.h5",
+  "cve_hdf5/cvefiles/cve-2025-2924.h5",
+  "cve_hdf5/cvefiles/cve-2025-2925.h5",
+  "cve_hdf5/cvefiles/cve-2025-6269-1.h5",
+  "cve_hdf5/cvefiles/cve-2025-6269-2.h5",
+  "cve_hdf5/cvefiles/cve-2025-6269-3.h5",
+  "cve_hdf5/cvefiles/cve-2025-6269-4.h5",
+  "cve_hdf5/cvefiles/cve-2025-6516.h5",
+  "cve_hdf5/cvefiles/cve-2025-6857.h5",
+  "cve_hdf5/cvefiles/cve-2025-7067.h5",
+  "cve_hdf5/cvefiles/cve-2026-26200.h5",
+  "cve_hdf5/cvefiles/cve-2026-34734.h5",
+  "cve_hdf5/cvefiles/cve-2026-92627.h5",
+  "cve_hdf5/cvefiles/unknown-1.h5",
+  "cve_hdf5/fuzzerfiles/gh-4431-poc-03.h5",
+  "cve_hdf5/fuzzerfiles/gh-4432-poc-05.h5",
+  "cve_hdf5/fuzzerfiles/gh-4433-poc-08.h5",
+  "cve_hdf5/fuzzerfiles/gh-4435-poc-10.h5",
+  "cve_hdf5/fuzzerfiles/gh_2649_flawed.h5",
+  "cve_hdf5/fuzzerfiles/gh_2649_plain_model.h5",
+  "h5py_data/compound-dtype-complex.h5",
+  "h5py_data/vlen_string_dset.h5",
+  "h5py_data/vlen_string_dset_utc.h5",
+  "h5py_data/vlen_string_s390x.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_bitgroom.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_granularbr.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_jpeg.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_lz4.h5",
+  "hdf5/HDF5Examples/C/H5FLT/tfiles/h5ex_d_zstd.h5",
+  "hdf5/HDF5Examples/C/H5G/16/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/C/H5G/16/h5ex_g_traverse.h5",
+  "hdf5/HDF5Examples/C/H5G/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/C/H5G/h5ex_g_traverse.h5",
+  "hdf5/HDF5Examples/C/H5G/h5ex_g_visit.h5",
+  "hdf5/HDF5Examples/FORTRAN/H5G/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/FORTRAN/H5G/h5ex_g_traverse.h5",
+  "hdf5/HDF5Examples/FORTRAN/H5G/h5ex_g_visit.h5",
+  "hdf5/HDF5Examples/JAVA/H5G/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/JAVA/H5G/h5ex_g_visit.h5",
+  "hdf5/HDF5Examples/JAVA/compat/H5G/110/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/JAVA/compat/H5G/110/h5ex_g_visit.h5",
+  "hdf5/HDF5Examples/JAVA/compat/H5G/h5ex_g_iterate.h5",
+  "hdf5/HDF5Examples/JAVA/compat/H5G/h5ex_g_visit.h5",
+  "hdf5/c++/test/th5s.h5",
+  "hdf5/hl/test/testfiles/test_ds_be.h5",
+  "hdf5/hl/test/testfiles/test_ds_be_new_ref-32bit.h5",
+  "hdf5/hl/test/testfiles/test_ds_be_new_ref.h5",
+  "hdf5/hl/test/testfiles/test_ds_le.h5",
+  "hdf5/hl/test/testfiles/test_ds_le_new_ref.h5",
+  "hdf5/hl/test/testfiles/test_ld.h5",
+  "hdf5/hl/test/testfiles/test_table_be.h5",
+  "hdf5/hl/test/testfiles/test_table_cray.h5",
+  "hdf5/hl/test/testfiles/test_table_le.h5",
+  "hdf5/test/testfiles/aggr.h5",
+  "hdf5/test/testfiles/bad_chunk_ndims.h5",
+  "hdf5/test/testfiles/bad_compound.h5",
+  "hdf5/test/testfiles/bad_offset.h5",
+  "hdf5/test/testfiles/be_data.h5",
+  "hdf5/test/testfiles/be_extlink1.h5",
+  "hdf5/test/testfiles/be_extlink2.h5",
+  "hdf5/test/testfiles/btree_idx_1_6.h5",
+  "hdf5/test/testfiles/btree_idx_1_8.h5",
+  "hdf5/test/testfiles/charsets.h5",
+  "hdf5/test/testfiles/corrupt_stab_msg.h5",
+  "hdf5/test/testfiles/deflate.h5",
+  "hdf5/test/testfiles/file_image_core_test.h5",
+  "hdf5/test/testfiles/filespace_1_6.h5",
+  "hdf5/test/testfiles/filespace_1_8.h5",
+  "hdf5/test/testfiles/fill18.h5",
+  "hdf5/test/testfiles/fill_old.h5",
+  "hdf5/test/testfiles/filter_error.h5",
+  "hdf5/test/testfiles/fsm_aggr_nopersist.h5",
+  "hdf5/test/testfiles/fsm_aggr_persist.h5",
+  "hdf5/test/testfiles/group_old.h5",
+  "hdf5/test/testfiles/h5fc_ext1_f.h5",
+  "hdf5/test/testfiles/h5fc_ext1_i.h5",
+  "hdf5/test/testfiles/h5fc_ext2_if.h5",
+  "hdf5/test/testfiles/h5fc_ext2_sf.h5",
+  "hdf5/test/testfiles/h5fc_ext3_isf.h5",
+  "hdf5/test/testfiles/h5fc_ext_none.h5",
+  "hdf5/test/testfiles/le_data.h5",
+  "hdf5/test/testfiles/le_extlink1.h5",
+  "hdf5/test/testfiles/le_extlink2.h5",
+  "hdf5/test/testfiles/memleak_H5O_dtype_decode_helper_H5Odtype.h5",
+  "hdf5/test/testfiles/mergemsg.h5",
+  "hdf5/test/testfiles/noencoder.h5",
+  "hdf5/test/testfiles/none.h5",
+  "hdf5/test/testfiles/paged_nopersist.h5",
+  "hdf5/test/testfiles/paged_persist.h5",
+  "hdf5/test/testfiles/specmetaread.h5",
+  "hdf5/test/testfiles/tarrold.h5",
+  "hdf5/test/testfiles/tbad_msg_count.h5",
+  "hdf5/test/testfiles/tbogus.h5",
+  "hdf5/test/testfiles/test_filters_be.h5",
+  "hdf5/test/testfiles/test_filters_le.h5",
+  "hdf5/test/testfiles/th5s.h5",
+  "hdf5/test/testfiles/tlayouto.h5",
+  "hdf5/test/testfiles/tmisc38a.h5",
+  "hdf5/test/testfiles/tmisc38b.h5",
+  "hdf5/test/testfiles/tmtimen.h5",
+  "hdf5/test/testfiles/tmtimeo.h5",
+  "hdf5/test/testfiles/tnullspace.h5",
+  "hdf5/test/testfiles/tsizeslheap.h5",
+  "hdf5/tools/test/testfiles/bigendian/tdset2.h5",
+  "hdf5/tools/test/testfiles/binfp64.h5",
+  "hdf5/tools/test/testfiles/binin16.h5",
+  "hdf5/tools/test/testfiles/binin32.h5",
+  "hdf5/tools/test/testfiles/binin8.h5",
+  "hdf5/tools/test/testfiles/binin8w.h5",
+  "hdf5/tools/test/testfiles/binuin16.h5",
+  "hdf5/tools/test/testfiles/binuin32.h5",
+  "hdf5/tools/test/testfiles/bounds_latest_latest.h5",
+  "hdf5/tools/test/testfiles/charsets.h5",
+  "hdf5/tools/test/testfiles/compounds_array_vlen1.h5",
+  "hdf5/tools/test/testfiles/compounds_array_vlen2.h5",
+  "hdf5/tools/test/testfiles/err_attr_dspace.h5",
+  "hdf5/tools/test/testfiles/file_space.h5",
+  "hdf5/tools/test/testfiles/filter_fail.h5",
+  "hdf5/tools/test/testfiles/h5clear_fsm_persist_equal.h5",
+  "hdf5/tools/test/testfiles/h5clear_fsm_persist_noclose.h5",
+  "hdf5/tools/test/testfiles/h5clear_fsm_persist_user_equal.h5",
+  "hdf5/tools/test/testfiles/h5clear_fsm_persist_user_less.h5",
+  "hdf5/tools/test/testfiles/h5clear_sec2_v0.h5",
+  "hdf5/tools/test/testfiles/h5clear_sec2_v2.h5",
+  "hdf5/tools/test/testfiles/h5copy_extlinks_src.h5",
+  "hdf5/tools/test/testfiles/h5copy_extlinks_trg.h5",
+  "hdf5/tools/test/testfiles/h5copy_ref.h5",
+  "hdf5/tools/test/testfiles/h5copytst.h5",
+  "hdf5/tools/test/testfiles/h5copytst_new.h5",
+  "hdf5/tools/test/testfiles/h5diff_attr1.h5",
+  "hdf5/tools/test/testfiles/h5diff_attr2.h5",
+  "hdf5/tools/test/testfiles/h5diff_attr3.h5",
+  "hdf5/tools/test/testfiles/h5diff_attr_v_level1.h5",
+  "hdf5/tools/test/testfiles/h5diff_attr_v_level2.h5",
+  "hdf5/tools/test/testfiles/h5diff_basic1.h5",
+  "hdf5/tools/test/testfiles/h5diff_basic2.h5",
+  "hdf5/tools/test/testfiles/h5diff_comp_vl_strs.h5",
+  "hdf5/tools/test/testfiles/h5diff_danglelinks1.h5",
+  "hdf5/tools/test/testfiles/h5diff_danglelinks2.h5",
+  "hdf5/tools/test/testfiles/h5diff_dset1.h5",
+  "hdf5/tools/test/testfiles/h5diff_dset2.h5",
+  "hdf5/tools/test/testfiles/h5diff_dset3.h5",
+  "hdf5/tools/test/testfiles/h5diff_dset_zero_dim_size1.h5",
+  "hdf5/tools/test/testfiles/h5diff_dset_zero_dim_size2.h5",
+  "hdf5/tools/test/testfiles/h5diff_dtypes.h5",
+  "hdf5/tools/test/testfiles/h5diff_empty.h5",
+  "hdf5/tools/test/testfiles/h5diff_enum_invalid_values.h5",
+  "hdf5/tools/test/testfiles/h5diff_eps1.h5",
+  "hdf5/tools/test/testfiles/h5diff_eps2.h5",
+  "hdf5/tools/test/testfiles/h5diff_exclude1-1.h5",
+  "hdf5/tools/test/testfiles/h5diff_exclude1-2.h5",
+  "hdf5/tools/test/testfiles/h5diff_exclude2-1.h5",
+  "hdf5/tools/test/testfiles/h5diff_exclude2-2.h5",
+  "hdf5/tools/test/testfiles/h5diff_exclude3-1.h5",
+  "hdf5/tools/test/testfiles/h5diff_exclude3-2.h5",
+  "hdf5/tools/test/testfiles/h5diff_ext2softlink_src.h5",
+  "hdf5/tools/test/testfiles/h5diff_ext2softlink_trg.h5",
+  "hdf5/tools/test/testfiles/h5diff_extlink_src.h5",
+  "hdf5/tools/test/testfiles/h5diff_extlink_trg.h5",
+  "hdf5/tools/test/testfiles/h5diff_grp_recurse1.h5",
+  "hdf5/tools/test/testfiles/h5diff_grp_recurse2.h5",
+  "hdf5/tools/test/testfiles/h5diff_grp_recurse_ext1.h5",
+  "hdf5/tools/test/testfiles/h5diff_grp_recurse_ext2-1.h5",
+  "hdf5/tools/test/testfiles/h5diff_grp_recurse_ext2-2.h5",
+  "hdf5/tools/test/testfiles/h5diff_grp_recurse_ext2-3.h5",
+  "hdf5/tools/test/testfiles/h5diff_hyper1.h5",
+  "hdf5/tools/test/testfiles/h5diff_hyper2.h5",
+  "hdf5/tools/test/testfiles/h5diff_linked_softlink.h5",
+  "hdf5/tools/test/testfiles/h5diff_links.h5",
+  "hdf5/tools/test/testfiles/h5diff_onion_dset_1d.h5",
+  "hdf5/tools/test/testfiles/h5diff_onion_dset_ext.h5",
+  "hdf5/tools/test/testfiles/h5diff_onion_objs.h5",
+  "hdf5/tools/test/testfiles/h5diff_softlinks.h5",
+  "hdf5/tools/test/testfiles/h5diff_strings1.h5",
+  "hdf5/tools/test/testfiles/h5diff_strings2.h5",
+  "hdf5/tools/test/testfiles/h5fc_edge_v3.h5",
+  "hdf5/tools/test/testfiles/h5fc_err_level.h5",
+  "hdf5/tools/test/testfiles/h5fc_ext1_f.h5",
+  "hdf5/tools/test/testfiles/h5fc_ext1_i.h5",
+  "hdf5/tools/test/testfiles/h5fc_ext1_s.h5",
+  "hdf5/tools/test/testfiles/h5fc_ext2_if.h5",
+  "hdf5/tools/test/testfiles/h5fc_ext2_is.h5",
+  "hdf5/tools/test/testfiles/h5fc_ext2_sf.h5",
+  "hdf5/tools/test/testfiles/h5fc_ext3_isf.h5",
+  "hdf5/tools/test/testfiles/h5fc_ext_none.h5",
+  "hdf5/tools/test/testfiles/h5fc_non_v3.h5",
+  "hdf5/tools/test/testfiles/h5repack_CVE-2018-14460.h5",
+  "hdf5/tools/test/testfiles/h5repack_CVE-2018-17432.h5",
+  "hdf5/tools/test/testfiles/h5repack_aggr.h5",
+  "hdf5/tools/test/testfiles/h5repack_attr.h5",
+  "hdf5/tools/test/testfiles/h5repack_attr_refs.h5",
+  "hdf5/tools/test/testfiles/h5repack_deflate.h5",
+  "hdf5/tools/test/testfiles/h5repack_early.h5",
+  "hdf5/tools/test/testfiles/h5repack_ext.h5",
+  "hdf5/tools/test/testfiles/h5repack_f32le.h5",
+  "hdf5/tools/test/testfiles/h5repack_f32le_ex.h5",
+  "hdf5/tools/test/testfiles/h5repack_fill.h5",
+  "hdf5/tools/test/testfiles/h5repack_filters.h5",
+  "hdf5/tools/test/testfiles/h5repack_fletcher.h5",
+  "hdf5/tools/test/testfiles/h5repack_fsm_aggr_nopersist.h5",
+  "hdf5/tools/test/testfiles/h5repack_fsm_aggr_persist.h5",
+  "hdf5/tools/test/testfiles/h5repack_hlink.h5",
+  "hdf5/tools/test/testfiles/h5repack_int32le_1d.h5",
+  "hdf5/tools/test/testfiles/h5repack_int32le_1d_ex.h5",
+  "hdf5/tools/test/testfiles/h5repack_int32le_2d.h5",
+  "hdf5/tools/test/testfiles/h5repack_int32le_2d_ex.h5",
+  "hdf5/tools/test/testfiles/h5repack_int32le_3d.h5",
+  "hdf5/tools/test/testfiles/h5repack_int32le_3d_ex.h5",
+  "hdf5/tools/test/testfiles/h5repack_layout.UD.h5",
+  "hdf5/tools/test/testfiles/h5repack_layout.h5",
+  "hdf5/tools/test/testfiles/h5repack_layout2.h5",
+  "hdf5/tools/test/testfiles/h5repack_layout3.h5",
+  "hdf5/tools/test/testfiles/h5repack_layouto.h5",
+  "hdf5/tools/test/testfiles/h5repack_named_dtypes.h5",
+  "hdf5/tools/test/testfiles/h5repack_nbit.h5",
+  "hdf5/tools/test/testfiles/h5repack_nested_8bit_enum.h5",
+  "hdf5/tools/test/testfiles/h5repack_nested_8bit_enum_deflated.h5",
+  "hdf5/tools/test/testfiles/h5repack_none.h5",
+  "hdf5/tools/test/testfiles/h5repack_objs.h5",
+  "hdf5/tools/test/testfiles/h5repack_paged_nopersist.h5",
+  "hdf5/tools/test/testfiles/h5repack_paged_persist.h5",
+  "hdf5/tools/test/testfiles/h5repack_refs.h5",
+  "hdf5/tools/test/testfiles/h5repack_shuffle.h5",
+  "hdf5/tools/test/testfiles/h5repack_soffset.h5",
+  "hdf5/tools/test/testfiles/h5repack_szip.h5",
+  "hdf5/tools/test/testfiles/h5repack_uint8be.h5",
+  "hdf5/tools/test/testfiles/h5repack_uint8be_ex.h5",
+  "hdf5/tools/test/testfiles/h5stat_err_old_fill.h5",
+  "hdf5/tools/test/testfiles/h5stat_err_old_layout.h5",
+  "hdf5/tools/test/testfiles/h5stat_filters.h5",
+  "hdf5/tools/test/testfiles/h5stat_idx.h5",
+  "hdf5/tools/test/testfiles/h5stat_newgrat.h5",
+  "hdf5/tools/test/testfiles/h5stat_threshold.h5",
+  "hdf5/tools/test/testfiles/h5stat_tsohm.h5",
+  "hdf5/tools/test/testfiles/mod_h5clear_mdc_image.h5",
+  "hdf5/tools/test/testfiles/non_comparables1.h5",
+  "hdf5/tools/test/testfiles/non_comparables2.h5",
+  "hdf5/tools/test/testfiles/old_h5fc_ext1_f.h5",
+  "hdf5/tools/test/testfiles/old_h5fc_ext1_i.h5",
+  "hdf5/tools/test/testfiles/old_h5fc_ext1_s.h5",
+  "hdf5/tools/test/testfiles/old_h5fc_ext2_if.h5",
+  "hdf5/tools/test/testfiles/old_h5fc_ext2_is.h5",
+  "hdf5/tools/test/testfiles/old_h5fc_ext2_sf.h5",
+  "hdf5/tools/test/testfiles/old_h5fc_ext3_isf.h5",
+  "hdf5/tools/test/testfiles/old_h5fc_ext_none.h5",
+  "hdf5/tools/test/testfiles/packedbits.h5",
+  "hdf5/tools/test/testfiles/t128bit_float.h5",
+  "hdf5/tools/test/testfiles/tCVE-2021-37501_attr_decode.h5",
+  "hdf5/tools/test/testfiles/tCVE_2018_11206_fill_new.h5",
+  "hdf5/tools/test/testfiles/tCVE_2018_11206_fill_old.h5",
+  "hdf5/tools/test/testfiles/taindices.h5",
+  "hdf5/tools/test/testfiles/tarray1.h5",
+  "hdf5/tools/test/testfiles/tarray1_big.h5",
+  "hdf5/tools/test/testfiles/tarray2.h5",
+  "hdf5/tools/test/testfiles/tarray4.h5",
+  "hdf5/tools/test/testfiles/tarray5.h5",
+  "hdf5/tools/test/testfiles/tarray8.h5",
+  "hdf5/tools/test/testfiles/tattr.h5",
+  "hdf5/tools/test/testfiles/tattr2.h5",
+  "hdf5/tools/test/testfiles/tattr4_be.h5",
+  "hdf5/tools/test/testfiles/tattrintsize.h5",
+  "hdf5/tools/test/testfiles/tattrreg.h5",
+  "hdf5/tools/test/testfiles/tbfloat16.h5",
+  "hdf5/tools/test/testfiles/tbfloat16_be.h5",
+  "hdf5/tools/test/testfiles/tbigdims.h5",
+  "hdf5/tools/test/testfiles/tbinary.h5",
+  "hdf5/tools/test/testfiles/tbitnopaque.h5",
+  "hdf5/tools/test/testfiles/tchar.h5",
+  "hdf5/tools/test/testfiles/tcmpdattrintsize.h5",
+  "hdf5/tools/test/testfiles/tcmpdintarray.h5",
+  "hdf5/tools/test/testfiles/tcmpdints.h5",
+  "hdf5/tools/test/testfiles/tcmpdintsize.h5",
+  "hdf5/tools/test/testfiles/tcomplex.h5",
+  "hdf5/tools/test/testfiles/tcompound.h5",
+  "hdf5/tools/test/testfiles/tcompound_complex.h5",
+  "hdf5/tools/test/testfiles/tcompound_complex2.h5",
+  "hdf5/tools/test/testfiles/tdatareg.h5",
+  "hdf5/tools/test/testfiles/tdset.h5",
+  "hdf5/tools/test/testfiles/tdset2.h5",
+  "hdf5/tools/test/testfiles/tdset_idx.h5",
+  "hdf5/tools/test/testfiles/tempty.h5",
+  "hdf5/tools/test/testfiles/textlink.h5",
+  "hdf5/tools/test/testfiles/textlinkfar.h5",
+  "hdf5/tools/test/testfiles/textlinksrc.h5",
+  "hdf5/tools/test/testfiles/textlinktar.h5",
+  "hdf5/tools/test/testfiles/textpfe.h5",
+  "hdf5/tools/test/testfiles/tfcontents2.h5",
+  "hdf5/tools/test/testfiles/tfilters.h5",
+  "hdf5/tools/test/testfiles/tfloat16.h5",
+  "hdf5/tools/test/testfiles/tfloat16_be.h5",
+  "hdf5/tools/test/testfiles/tfloat4.h5",
+  "hdf5/tools/test/testfiles/tfloat6.h5",
+  "hdf5/tools/test/testfiles/tfloat8.h5",
+  "hdf5/tools/test/testfiles/tfloatsattrs.h5",
+  "hdf5/tools/test/testfiles/tfpformat.h5",
+  "hdf5/tools/test/testfiles/tfvalues.h5",
+  "hdf5/tools/test/testfiles/tgroup.h5",
+  "hdf5/tools/test/testfiles/tgrp_comments.h5",
+  "hdf5/tools/test/testfiles/tgrpnullspace.h5",
+  "hdf5/tools/test/testfiles/thlink.h5",
+  "hdf5/tools/test/testfiles/thyperslab.h5",
+  "hdf5/tools/test/testfiles/tintascii.h5",
+  "hdf5/tools/test/testfiles/tints4dims.h5",
+  "hdf5/tools/test/testfiles/tintsattrs.h5",
+  "hdf5/tools/test/testfiles/tintsnodata.h5",
+  "hdf5/tools/test/testfiles/tlarge_objname.h5",
+  "hdf5/tools/test/testfiles/tldouble.h5",
+  "hdf5/tools/test/testfiles/tldouble_scalar.h5",
+  "hdf5/tools/test/testfiles/tlonglinks.h5",
+  "hdf5/tools/test/testfiles/tloop.h5",
+  "hdf5/tools/test/testfiles/tnamed_dtype_attr.h5",
+  "hdf5/tools/test/testfiles/tnestedcmpddt.h5",
+  "hdf5/tools/test/testfiles/tnestedcomp.h5",
+  "hdf5/tools/test/testfiles/tno-subset.h5",
+  "hdf5/tools/test/testfiles/tnullspace.h5",
+  "hdf5/tools/test/testfiles/torderattr.h5",
+  "hdf5/tools/test/testfiles/tordergr.h5",
+  "hdf5/tools/test/testfiles/trefer_attr.h5",
+  "hdf5/tools/test/testfiles/trefer_compat.h5",
+  "hdf5/tools/test/testfiles/trefer_ext1.h5",
+  "hdf5/tools/test/testfiles/trefer_ext2.h5",
+  "hdf5/tools/test/testfiles/trefer_grp.h5",
+  "hdf5/tools/test/testfiles/trefer_obj.h5",
+  "hdf5/tools/test/testfiles/trefer_obj_del.h5",
+  "hdf5/tools/test/testfiles/trefer_param.h5",
+  "hdf5/tools/test/testfiles/trefer_reg.h5",
+  "hdf5/tools/test/testfiles/trefer_reg_1d.h5",
+  "hdf5/tools/test/testfiles/tsaf.h5",
+  "hdf5/tools/test/testfiles/tscalarattrintsize.h5",
+  "hdf5/tools/test/testfiles/tscalarintattrsize.h5",
+  "hdf5/tools/test/testfiles/tscalarintsize.h5",
+  "hdf5/tools/test/testfiles/tscalarstring.h5",
+  "hdf5/tools/test/testfiles/tslink.h5",
+  "hdf5/tools/test/testfiles/tsoftlinks.h5",
+  "hdf5/tools/test/testfiles/tst_onion_dset_1d.h5",
+  "hdf5/tools/test/testfiles/tst_onion_dset_ext.h5",
+  "hdf5/tools/test/testfiles/tst_onion_objs.h5",
+  "hdf5/tools/test/testfiles/tstr.h5",
+  "hdf5/tools/test/testfiles/tstr2.h5",
+  "hdf5/tools/test/testfiles/tstr3.h5",
+  "hdf5/tools/test/testfiles/tudfilter.h5",
+  "hdf5/tools/test/testfiles/tudfilter2.h5",
+  "hdf5/tools/test/testfiles/tvldtypes1.h5",
+  "hdf5/tools/test/testfiles/tvldtypes2.h5",
+  "hdf5/tools/test/testfiles/tvldtypes3.h5",
+  "hdf5/tools/test/testfiles/tvldtypes4.h5",
+  "hdf5/tools/test/testfiles/tvldtypes5.h5",
+  "hdf5/tools/test/testfiles/tvlenstr_array.h5",
+  "hdf5/tools/test/testfiles/tvlstr.h5",
+  "hdf5/tools/test/testfiles/tvms.h5",
+  "hdf5/tools/test/testfiles/txtfp32.h5",
+  "hdf5/tools/test/testfiles/txtfp64.h5",
+  "hdf5/tools/test/testfiles/txtin16.h5",
+  "hdf5/tools/test/testfiles/txtin32.h5",
+  "hdf5/tools/test/testfiles/txtin8.h5",
+  "hdf5/tools/test/testfiles/txtstr.h5",
+  "hdf5/tools/test/testfiles/txtuin16.h5",
+  "hdf5/tools/test/testfiles/txtuin32.h5",
+  "hdf5/tools/test/testfiles/vds/1_a.h5",
+  "hdf5/tools/test/testfiles/vds/1_b.h5",
+  "hdf5/tools/test/testfiles/vds/1_c.h5",
+  "hdf5/tools/test/testfiles/vds/1_d.h5",
+  "hdf5/tools/test/testfiles/vds/1_e.h5",
+  "hdf5/tools/test/testfiles/vds/1_f.h5",
+  "hdf5/tools/test/testfiles/vds/1_vds.h5",
+  "hdf5/tools/test/testfiles/vds/2_a.h5",
+  "hdf5/tools/test/testfiles/vds/2_b.h5",
+  "hdf5/tools/test/testfiles/vds/2_c.h5",
+  "hdf5/tools/test/testfiles/vds/2_d.h5",
+  "hdf5/tools/test/testfiles/vds/2_e.h5",
+  "hdf5/tools/test/testfiles/vds/2_vds.h5",
+  "hdf5/tools/test/testfiles/vds/3_1_vds.h5",
+  "hdf5/tools/test/testfiles/vds/3_2_vds.h5",
+  "hdf5/tools/test/testfiles/vds/4_0.h5",
+  "hdf5/tools/test/testfiles/vds/4_1.h5",
+  "hdf5/tools/test/testfiles/vds/4_2.h5",
+  "hdf5/tools/test/testfiles/vds/4_vds.h5",
+  "hdf5/tools/test/testfiles/vds/5_a.h5",
+  "hdf5/tools/test/testfiles/vds/5_b.h5",
+  "hdf5/tools/test/testfiles/vds/5_c.h5",
+  "hdf5/tools/test/testfiles/vds/5_vds.h5",
+  "hdf5/tools/test/testfiles/vds/a.h5",
+  "hdf5/tools/test/testfiles/vds/b.h5",
+  "hdf5/tools/test/testfiles/vds/c.h5",
+  "hdf5/tools/test/testfiles/vds/d.h5",
+  "hdf5/tools/test/testfiles/vds/f-0.h5",
+  "hdf5/tools/test/testfiles/vds/f-3.h5",
+  "hdf5/tools/test/testfiles/vds/vds-eiger.h5",
+  "hdf5/tools/test/testfiles/vds/vds-percival-unlim-maxmin.h5",
+  "hdf5/tools/test/testfiles/xml/tbitfields.h5",
+  "hdf5/tools/test/testfiles/xml/tcompound2.h5",
+  "hdf5/tools/test/testfiles/xml/tdset2.h5",
+  "hdf5/tools/test/testfiles/xml/tenum.h5",
+  "hdf5/tools/test/testfiles/xml/test35.nc",
+  "hdf5/tools/test/testfiles/xml/tloop2.h5",
+  "hdf5/tools/test/testfiles/xml/tname-amp.h5",
+  "hdf5/tools/test/testfiles/xml/tname-apos.h5",
+  "hdf5/tools/test/testfiles/xml/tname-gt.h5",
+  "hdf5/tools/test/testfiles/xml/tname-lt.h5",
+  "hdf5/tools/test/testfiles/xml/tname-quot.h5",
+  "hdf5/tools/test/testfiles/xml/tname-sp.h5",
+  "hdf5/tools/test/testfiles/xml/tnodata.h5",
+  "hdf5/tools/test/testfiles/xml/tobjref.h5",
+  "hdf5/tools/test/testfiles/xml/topaque.h5",
+  "hdf5/tools/test/testfiles/xml/tref-escapes-at.h5",
+  "hdf5/tools/test/testfiles/xml/tref-escapes.h5",
+  "hdf5/tools/test/testfiles/xml/tref.h5",
+  "hdf5/tools/test/testfiles/xml/tstring-at.h5",
+  "hdf5/tools/test/testfiles/xml/tstring.h5",
+  "hdf5/tools/test/testfiles/zerodim.h5",
+  "netcdf-c/h5_test/ref_tst_h_compounds.h5",
+  "netcdf-c/h5_test/ref_tst_h_compounds2.h5",
+  "netcdf-c/nc_test4/ref_hdf5_compat1.nc",
+  "netcdf-c/nc_test4/ref_hdf5_compat2.nc",
+  "netcdf-c/nc_test4/ref_hdf5_compat3.nc",
+  "netcdf-c/nc_test4/ref_szip.h5",
+  "netcdf-c/nc_test4/ref_tst_compounds.nc",
+  "netcdf-c/nc_test4/ref_tst_dims.nc",
+  "netcdf-c/nc_test4/ref_tst_interops4.nc",
+  "netcdf-c/nc_test4/ref_tst_xplatform2_1.nc",
+  "netcdf-c/nc_test4/ref_tst_xplatform2_2.nc",
+  "netcdf-c/nc_test4/tdset.h5",
+  "netcdf-c/ncdump/ref_nc_test_netcdf4_4_0.nc",
+  "netcdf-c/ncdump/ref_no_ncproperty.nc",
+  "netcdf-c/ncdump/ref_provenance_v1.nc",
+  "netcdf-c/ncdump/ref_test_corrupt_magic.nc",
+  "netcdf-c/ncdump/ref_tst_compounds2.nc",
+  "netcdf-c/ncdump/ref_tst_compounds3.nc",
+  "netcdf-c/ncdump/ref_tst_compounds4.nc",
+  "netcdf-c/ncdump/ref_tst_irish_rover.nc",
+  "netcdf4-python/examples/data/prmsl.2000.nc",
+  "netcdf4-python/examples/data/prmsl.2001.nc",
+  "netcdf4-python/examples/data/prmsl.2002.nc",
+  "netcdf4-python/examples/data/prmsl.2003.nc",
+  "netcdf4-python/examples/data/prmsl.2004.nc",
+  "netcdf4-python/examples/data/prmsl.2005.nc",
+  "netcdf4-python/examples/data/prmsl.2006.nc",
+  "netcdf4-python/examples/data/prmsl.2007.nc",
+  "netcdf4-python/examples/data/prmsl.2008.nc",
+  "netcdf4-python/examples/data/prmsl.2009.nc",
+  "netcdf4-python/examples/data/prmsl.2010.nc",
+  "netcdf4-python/examples/data/prmsl.2011.nc",
+  "netcdf4-python/examples/data/rtofs_glo_3dz_f006_6hrly_reg3.nc",
+  "netcdf4-python/test/20171025_2056.Cloud_Top_Height.nc",
+  "netcdf4-python/test/issue1152.nc",
+  "netcdf4-python/test/issue671.nc",
+  "netcdf4-python/test/issue672.nc",
+  "netcdf4-python/test/test_gold.nc",
+  "usnistgov_h5wasm/test/array.h5",
+  "usnistgov_h5wasm/test/compressed.h5",
+  "usnistgov_h5wasm/test/empty.h5",
+  "usnistgov_h5wasm/test/float16.h5",
+  "usnistgov_h5wasm/test/vlen.h5",
+  "xarray-data/ROMS_example.nc",
+  "xarray-data/basin_mask.nc",
+  "xarray-data/imerghh_730.hdf5",
+  "xarray-data/precipitation.nc4"
+ ]
+}

conformance/check.py

@@ -0,0 +1,88 @@
+#!/usr/bin/env python3
+"""check.py <results_dir> <baseline.json> [--update]
+
+The conformance gate. Fails (exit 1) when
+  * clawhdf5 panicked, hung, crashed or ran out of memory on any file, or
+  * the ok count fell below the baseline's, or
+  * a file the baseline lists as ok is no longer ok (even if another file
+    became ok and the total held).
+New ok files are reported so the baseline can be raised (--update rewrites it
+from the results).
+"""
+import json
+import os
+import sys
+
+FATAL = ("panic", "hang", "crash", "oom")
+
+
+def main():
+    args = [a for a in sys.argv[1:] if not a.startswith("--")]
+    update = "--update" in sys.argv
+    res_dir, base_path = args
+    res = json.load(open(os.path.join(res_dir, "results.json")))
+    rows = res["rows"]
+    counts = {}
+    per_corpus = {}
+    for r in rows:
+        counts[r["class"]] = counts.get(r["class"], 0) + 1
+        pc = per_corpus.setdefault(r["corpus"], {})
+        pc[r["class"]] = pc.get(r["class"], 0) + 1
+    ok_files = sorted(r["file"] for r in rows if r["class"] == "ok")
+
+    if update:
+        meta = {}
+        mp = os.path.join(res_dir, "report-meta.json")
+        if os.path.exists(mp):
+            meta = json.load(open(mp))
+        base = {
+            "comment": "conformance/run.sh fails if the ok count drops below `ok` or a file in `ok_files` stops being ok. "
+                       "Regenerate with `conformance/run.sh --update-baseline` after an intended change.",
+            "commit": meta.get("commit", ""),
+            "date": meta.get("date", ""),
+            "reference": meta.get("reference", ""),
+            "files": len(rows),
+            "ok": len(ok_files),
+            "counts": dict(sorted(counts.items())),
+            "per_corpus": {k: dict(sorted(v.items())) for k, v in sorted(per_corpus.items())},
+            "ok_files": ok_files,
+        }
+        with open(base_path, "w") as fh:
+            json.dump(base, fh, indent=1)
+            fh.write("\n")
+        print(f"baseline updated: {len(ok_files)} ok of {len(rows)} files -> {base_path}")
+        return 0
+
+    base = json.load(open(base_path))
+    failures = []
+    fatal = [r for r in rows if r["class"] in FATAL]
+    for r in fatal:
+        failures.append(f"{r['class']}: {r['file']}: {r['ours_detail'][:200]}")
+    if len(ok_files) < base["ok"]:
+        failures.append(f"ok count dropped: {len(ok_files)} < baseline {base['ok']}")
+    now_ok = set(ok_files)
+    by_file = {r["file"]: r for r in rows}
+    for f in base["ok_files"]:
+        if f not in now_ok:
+            r = by_file.get(f)
+            why = f"now {r['class']}: {(r['ours_detail'] or r['first_issue'])[:200]}" if r else "no longer in the corpus"
+            failures.append(f"regressed: {f}: {why}")
+    gained = sorted(now_ok - set(base["ok_files"]))
+
+    print(f"conformance: {len(ok_files)} ok of {len(rows)} files (baseline {base['ok']} of {base['files']}); "
+          + ", ".join(f"{k} {v}" for k, v in sorted(counts.items())))
+    if gained:
+        print(f"{len(gained)} file(s) newly ok — raise the baseline with `conformance/run.sh --update-baseline`:")
+        for f in gained:
+            print(f"  + {f}")
+    if failures:
+        print(f"CONFORMANCE GATE FAILED ({len(failures)}):")
+        for f in failures:
+            print(f"  - {f}")
+        return 1
+    print("conformance gate passed")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

conformance/compare.py

@@ -0,0 +1,289 @@
+#!/usr/bin/env python3
+"""compare.py <results_dir>: classify each file and group failures by root cause.
+
+Writes <results_dir>/results.csv, results.json and summary.md.
+File classes (first match wins):
+  hang, oom, crash, panic       ours: timeout / allocation failure / signal / any panic (caught or not)
+  h5py-cannot-read              libhdf5/h5py failed to open the file (or crashed/hung)
+  our-error                     we fail to open, list, or read something h5py reads
+  mismatch                      we read something with different shape/values, or a different object set
+  ok
+"""
+import collections
+import csv
+import json
+import os
+import re
+import sys
+
+R = sys.argv[1]
+RUNS = os.path.join(R, "runs")
+
+
+def load(d, name):
+    rc_p = os.path.join(d, name + ".rc")
+    if not os.path.exists(rc_p):
+        return None
+    rc = int(open(rc_p).read().strip() or -1)
+    err = open(os.path.join(d, name + ".err"), errors="replace").read()
+    js = None
+    try:
+        js = json.load(open(os.path.join(d, name + ".json")))
+    except Exception:  # noqa: BLE001
+        pass
+    return {"rc": rc, "err": err, "json": js}
+
+
+def proc_status(p):
+    """-> (status, detail)"""
+    if p is None:
+        return "missing", ""
+    rc, err = p["rc"], p["err"]
+    first_panic = next((ln for ln in err.splitlines() if ln.startswith("PANIC:") or "panicked at" in ln), "")
+    if rc == 0 and p["json"] is not None:
+        return "ok", ""
+    if rc == 137 or rc == 124:
+        return "hang", f"timeout ({os.environ.get('TMO', '20')} s)"
+    if "memory allocation of" in err or "MemoryError" in err or "std::bad_alloc" in err:
+        m = re.search(r"memory allocation of \d+ bytes failed", err)
+        return "oom", m.group(0) if m else "allocation failure"
+    if "overflowed its stack" in err:
+        return "crash", "stack overflow"
+    if rc == 101:
+        return "panic", first_panic or (err.strip().splitlines() or [""])[-1]
+    if rc in (134, 139, 136, 135, 132) or rc > 128:
+        sig = {134: "SIGABRT", 139: "SIGSEGV", 136: "SIGFPE", 135: "SIGBUS", 132: "SIGILL"}.get(rc, f"signal {rc - 128}")
+        tail = [ln for ln in err.strip().splitlines() if ln.strip()][-1:]
+        return "crash", f"{sig}: {tail[0][:200] if tail else ''}"
+    tail = [ln for ln in err.strip().splitlines() if ln.strip()][-1:]
+    return "crash", f"rc={rc}: {tail[0][:200] if tail else ''}"
+
+
+def norm(msg):
+    m = msg.split("\n")[0]
+    m = re.sub(r"0x[0-9a-fA-F]+", "X", m)
+    m = re.sub(r'"[^"]*"', '"…"', m)
+    m = re.sub(r"'[^']*'", "'…'", m)
+    m = re.sub(r"\d+", "N", m)
+    return m[:160]
+
+
+def panic_head(msg):
+    """First line + first clawhdf5 frame of a PANIC record."""
+    lines = msg.split("\n")
+    frame = next((ln.strip() for ln in lines[1:] if "clawhdf5_format" in ln), "")
+    return lines[0][:300], frame[:300]
+
+
+def eq_shape(a, b):
+    return a == b
+
+
+rows = []
+issues_by_file = {}
+root_causes = collections.defaultdict(lambda: {"files": set(), "count": 0, "examples": []})
+mismatch_causes = collections.defaultdict(lambda: {"files": set(), "count": 0, "examples": []})
+panics = []
+ref_only_errors = collections.Counter()
+incomparable = collections.Counter()
+
+
+def add(bucket, key, file, example):
+    b = bucket[key]
+    b["count"] += 1
+    if file not in b["files"] and len(b["examples"]) < 6:
+        b["examples"].append(example)
+    b["files"].add(file)
+
+
+files = [ln.strip() for ln in open(os.path.join(R, "files.txt")) if ln.strip()]
+for rel in files:
+    d = os.path.join(RUNS, rel.replace("/", "__"))
+    corpus = rel.split("/")[0]
+    ours, ref = load(d, "ours"), load(d, "ref")
+    h5dump = load(d, "h5dump")
+    os_, od = proc_status(ours)
+    rs, rd = proc_status(ref)
+    oj = ours["json"] if ours else None
+    rj = ref["json"] if ref else None
+    issues = []  # (kind, detail)
+    caught_panics = []
+
+    def scan_err(path, what, msg):
+        if msg.startswith("PANIC:"):
+            caught_panics.append((path, what, msg))
+
+    if oj:
+        for o in oj.get("objects", []):
+            for k in ("error", "attrs_error", "list_error"):
+                if k in o:
+                    scan_err(o["path"], k, o[k])
+            for an, av in (o.get("attrs") or {}).items():
+                if "error" in av:
+                    scan_err(o["path"], f"attr {an}", av["error"])
+        if oj.get("open_error", "").startswith("PANIC:"):
+            caught_panics.append(("<open>", "open", oj["open_error"]))
+
+    ref_open_fail = rs != "ok" or (rj is not None and "open_error" in rj)
+    ours_open_err = oj.get("open_error") if oj else None
+    n_obj = n_ok = 0
+    if os_ == "ok" and rj and not ref_open_fail and not ours_open_err:
+        ro = {x["path"]: x for x in rj.get("objects", [])}
+        oo = {x["path"]: x for x in oj.get("objects", [])}
+        our_list_errors = [x for x in oo.values() if "list_error" in x]
+        for p in sorted(set(ro) | set(oo)):
+            a, b = ro.get(p), oo.get(p)
+            n_obj += 1
+            if a is None:
+                issues.append(("mismatch", f"extra object {p} (kind={b.get('kind')})", "extra-object", b))
+                continue
+            if b is None:
+                if our_list_errors:
+                    continue  # accounted for by the list_error
+                issues.append(("mismatch", f"missing object {p} (kind={a.get('kind')})", "missing-object", a))
+                continue
+            ok = True
+            if a.get("kind") != b.get("kind") and "error" not in b and "error" not in a:
+                issues.append(("mismatch", f"{p}: kind {a.get('kind')} vs ours {b.get('kind')}", "kind", b))
+                ok = False
+            for k in ("error", "list_error", "attrs_error"):
+                if k in b and k not in a:
+                    issues.append(("our-error", f"{p}: {k}: {b[k]}", b[k], b))
+                    ok = False
+                elif k in a and k not in b and k == "error":
+                    ref_only_errors[norm(a[k])] += 1
+            if a.get("kind") == "dataset" and "error" not in a and "error" not in b:
+                if "skipped" in a or "skipped" in b:
+                    pass
+                elif a.get("converted"):
+                    incomparable[f"dataset {a['converted']}"] += 1
+                elif a.get("shape") != b.get("shape"):
+                    issues.append(("mismatch", f"{p}: shape {a.get('shape')} vs ours {b.get('shape')}", "shape", b))
+                    ok = False
+                elif a.get("hash") != b.get("hash"):
+                    issues.append(("mismatch", f"{p}: values differ (h5py {a.get('dtype')} vs ours {b.get('dtype')})", "values", b | {"ref_head": a.get("head"), "ref_dtype": a.get("dtype")}))
+                    ok = False
+            ra, oa = a.get("attrs") or {}, b.get("attrs") or {}
+            if "attrs_error" not in b and "attrs_error" not in a:
+                for an in sorted(set(ra) | set(oa)):
+                    x, y = ra.get(an), oa.get(an)
+                    if x is None:
+                        issues.append(("mismatch", f"{p}@{an}: extra attribute", "extra-attr", y or {}))
+                    elif y is None:
+                        issues.append(("mismatch", f"{p}@{an}: missing attribute", "missing-attr", x))
+                    elif "error" in y and "error" not in x:
+                        issues.append(("our-error", f"{p}@{an}: {y['error']}", y["error"], y))
+                    elif "error" in x:
+                        continue
+                    elif x.get("converted"):
+                        incomparable[f"attr {x['converted']}"] += 1
+                    elif x.get("shape") != y.get("shape"):
+                        issues.append(("mismatch", f"{p}@{an}: attr shape {x.get('shape')} vs ours {y.get('shape')}", "attr-shape", y | {"ref_dtype": x.get("dtype")}))
+                    elif x.get("hash") != y.get("hash"):
+                        issues.append(("mismatch", f"{p}@{an}: attr values differ (h5py {x.get('dtype')} vs ours {y.get('dtype')})", "attr-values", y | {"ref_head": x.get("head"), "ref_dtype": x.get("dtype")}))
+            if ok:
+                n_ok += 1
+
+    # classify
+    if os_ in ("hang", "oom", "crash", "panic"):
+        cls = os_
+    elif caught_panics:
+        cls = "panic"
+    elif ref_open_fail:
+        cls = "h5py-cannot-read"
+    elif ours_open_err:
+        cls = "our-error"
+        issues.append(("our-error", f"open: {ours_open_err}", ours_open_err, {}))
+    elif any(i[0] == "our-error" for i in issues):
+        cls = "our-error"
+    elif issues:
+        cls = "mismatch"
+    else:
+        cls = "ok"
+
+    if os_ in ("hang", "oom", "crash", "panic") or caught_panics:
+        panics.append({
+            "file": rel, "class": cls, "detail": od,
+            "stderr": (ours["err"] if ours else "")[:3000],
+            "caught": [(p, w, m[:2500]) for p, w, m in caught_panics[:3]],
+            "n_caught": len(caught_panics),
+        })
+    for kind, detail, key, rec in issues:
+        if kind == "our-error":
+            add(root_causes, norm(key), rel, detail[:300])
+        else:
+            if key in ("values", "attr-values", "shape", "attr-shape"):
+                mk = f"{key}: ours={rec.get('dtype')} h5py={rec.get('ref_dtype')} layout={rec.get('layout','-')} filters={rec.get('filters','-')}"
+            else:
+                mk = key
+            add(mismatch_causes, mk, rel, detail[:300] + (f" | ref_head={rec.get('ref_head')} our_head={rec.get('head')}" if rec.get("ref_head") else ""))
+    ref_detail = rd if rs != "ok" else ((rj or {}).get("open_error") or "")
+    h5d = ""
+    if h5dump:
+        rc = h5dump["rc"]
+        h5d = {0: "ok", 1: "error", 137: "hang", 124: "hang", 134: "SIGABRT", 139: "SIGSEGV", 136: "SIGFPE", 135: "SIGBUS"}.get(rc, f"rc={rc}")
+        if "memory allocation" in h5dump["err"] or "Cannot allocate" in h5dump["err"]:
+            h5d += "(oom)"
+    rows.append({
+        "file": rel, "corpus": corpus, "class": cls,
+        "ours": os_ if os_ != "ok" else ("open-error" if ours_open_err else ("panic" if caught_panics else "ok")),
+        "ours_detail": (od or ours_open_err or (caught_panics[0][2].split("\n")[0] if caught_panics else ""))[:300],
+        "ref": rs if rs != "ok" else ("open-error" if (rj or {}).get("open_error") else "ok"),
+        "ref_detail": ref_detail[:300],
+        "h5dump_1_14_6": h5d,
+        "h5dump_detail": ([ln for ln in h5dump["err"].splitlines() if ln.strip()][-1:] or [""])[0][:200] if h5dump else "",
+        "objects": n_obj, "objects_ok": n_ok,
+        "issues": len(issues), "first_issue": issues[0][1][:300] if issues else "",
+        "superblock": (oj or {}).get("superblock_version", ""),
+    })
+    # the first issues of each file, for report.py's known-cause matching
+    issues_by_file[rel] = [
+        {"kind": k, "key": key, "detail": det[:300], "ours_dtype": rec.get("dtype"), "ref_dtype": rec.get("ref_dtype")}
+        for k, det, key, rec in issues[:50]
+    ]
+
+with open(os.path.join(R, "results.csv"), "w", newline="") as fh:
+    w = csv.DictWriter(fh, fieldnames=list(rows[0].keys()))
+    w.writeheader()
+    w.writerows(rows)
+
+
+def ser(b):
+    return {k: {"files": len(v["files"]), "count": v["count"], "examples": v["examples"], "file_list": sorted(v["files"])} for k, v in sorted(b.items(), key=lambda kv: -len(kv[1]["files"]))}
+
+
+json.dump({"rows": rows, "issues": issues_by_file, "root_causes": ser(root_causes), "mismatch_causes": ser(mismatch_causes),
+           "panics": panics, "incomparable": incomparable.most_common(), "ref_only_errors": ref_only_errors.most_common()},
+          open(os.path.join(R, "results.json"), "w"), indent=1)
+
+classes = ["ok", "our-error", "mismatch", "h5py-cannot-read", "hang", "panic", "crash", "oom"]
+by_corpus = collections.defaultdict(collections.Counter)
+for r in rows:
+    by_corpus[r["corpus"]][r["class"]] += 1
+    by_corpus["ALL"][r["class"]] += 1
+lines = ["# Conformance sweep summary", "", "| corpus | files | " + " | ".join(classes) + " |", "|---" * (len(classes) + 2) + "|"]
+for c in sorted(by_corpus, key=lambda k: (k == "ALL", k)):
+    cnt = by_corpus[c]
+    lines.append(f"| {c} | {sum(cnt.values())} | " + " | ".join(str(cnt.get(k, 0)) for k in classes) + " |")
+lines += ["", "## Panics / hangs / crashes / OOM", ""]
+for p in panics:
+    lines.append(f"- **{p['file']}** [{p['class']}] {p['detail']}")
+    for path, what, m in p["caught"][:1]:
+        lines.append("  ```\n  " + f"{path} ({what}): " + m.replace("\n", "\n  ")[:1500] + "\n  ```")
+    if not p["caught"] and p["stderr"]:
+        lines.append("  ```\n  " + p["stderr"].strip()[:1500].replace("\n", "\n  ") + "\n  ```")
+lines += ["", "## Our-error root causes (files affected)", ""]
+for k, v in ser(root_causes).items():
+    lines.append(f"- [{v['files']} files, {v['count']} objs] `{k}`")
+    for ex in v["examples"][:3]:
+        lines.append(f"    - {ex}")
+lines += ["", "## Mismatch root causes", ""]
+for k, v in ser(mismatch_causes).items():
+    lines.append(f"- [{v['files']} files, {v['count']} objs] `{k}`")
+    for ex in v["examples"][:3]:
+        lines.append(f"    - {ex}")
+lines += ["", "## Objects h5py fails on but we read (top)", ""]
+for k, n in ref_only_errors.most_common(15):
+    lines.append(f"- {n} x `{k}`")
+open(os.path.join(R, "summary.md"), "w").write("\n".join(lines) + "\n")
+print("\n".join(lines[:4 + len(by_corpus)]))

conformance/corpus.txt

@@ -0,0 +1,19 @@
+# Conformance corpora, pinned by commit. fetch-corpus.sh reads this file.
+#
+# name  git-url  commit  root  [sparse-checkout patterns...]
+#
+# `root` is the directory inside the checkout that is swept ("." = all of it).
+# Patterns are git non-cone sparse-checkout patterns; none = whole repository.
+# Every file under <root> with an HDF5/netCDF-4 extension is probed; for
+# cve_hdf5 the extension-less files in cvefiles/ and fuzzerfiles/ are too.
+# Licences: each corpus keeps its upstream licence; nothing here is committed
+# to this repository — the files are downloaded into the gitignored cache.
+hdf5              https://github.com/HDFGroup/hdf5.git            a3cf1ea82cc7a66e50029a688121e1b105a7ce88  .  *.h5 *.he5 *.nc *.hdf5 *.h5f
+cve_hdf5          https://github.com/HDFGroup/cve_hdf5.git        3fd1f5ae3869e01b8ae02b41d7108de7ffb1a374  .
+netcdf-c          https://github.com/Unidata/netcdf-c.git         beb7b9585273c1548386231a59b809d906359033  .  /nc_test4/*.nc /ncdump/*.nc /nc_test4/*.h5 /ncdump/*.h5 /h5_test/*.h5 /hdf5_test/*.h5
+NCAS-CMS_pyfive   https://github.com/NCAS-CMS/pyfive.git          8cf07b8749133f41c5e30b8a4c604486f687fe74  .  *.h5 *.hdf5 *.hdf *.nc *.he5
+usnistgov_h5wasm  https://github.com/usnistgov/h5wasm.git         02f6336527d2812783fcedabfbf42127ec8d06d2  .  *.h5 *.hdf5 *.hdf *.nc *.he5
+netcdf4-python    https://github.com/Unidata/netcdf4-python.git   6e67576d39aef8091fb20bd767b4f1a52ddc1bec  .  *.nc *.h5
+xarray-data       https://github.com/pydata/xarray-data.git       a35297e9da2cc99c811014f0c8a4297345a5c28d  .  /basin_mask.nc /precipitation.nc4 /imerghh_730.hdf5 /eraint_uvz.nc /ROMS_example.nc /tiny.nc
+# h5py 3.16.0 (tag 3.16.0), its test data files.
+h5py_data         https://github.com/h5py/h5py.git                b2f0347c4200333acd89b43733f1caa0c115162f  h5py/tests/data_files  /h5py/tests/data_files/*

conformance/fetch-corpus.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# fetch-corpus.sh [cache_dir]
+#
+# Download the corpora pinned in conformance/corpus.txt into the (gitignored)
+# cache: <cache>/src/<name> is a shallow, sparse, blob-filtered checkout of the
+# pinned commit and <cache>/corpus/<name> links to the swept root inside it.
+# A corpus already checked out at its pinned commit is left alone, so a second
+# run costs nothing and needs no network.
+set -euo pipefail
+HERE="$(cd "$(dirname "$0")" && pwd)"
+CACHE="${1:-${CONFORMANCE_CACHE:-$HERE/.cache}}"
+mkdir -p "$CACHE/src" "$CACHE/corpus"
+CACHE="$(cd "$CACHE" && pwd)"
+
+retry() { local i; for i in 1 2 3 4; do "$@" && return 0; sleep $((i * 5)); done; return 1; }
+
+grep -v '^[[:space:]]*\(#\|$\)' "$HERE/corpus.txt" | while read -r name url commit root patterns; do
+  src="$CACHE/src/$name"
+  if [ -d "$src/.git" ] && [ "$(git -C "$src" rev-parse HEAD 2>/dev/null)" = "$commit" ]; then
+    echo "cached   $name @ ${commit:0:12}"
+  else
+    echo "fetching $name @ ${commit:0:12} from $url"
+    rm -rf "$src"
+    git init -q "$src"
+    git -C "$src" remote add origin "$url"
+    git -C "$src" config advice.detachedHead false
+    if [ -n "$patterns" ]; then
+      git -C "$src" config core.sparseCheckout true
+      # no-cone patterns (globs); `set -f` keeps the shell from expanding them
+      (set -f; printf '%s\n' $patterns) > "$src/.git/info/sparse-checkout"
+    fi
+    retry git -C "$src" fetch -q --depth 1 --filter=blob:none origin "$commit"
+    retry git -C "$src" checkout -q FETCH_HEAD
+    got="$(git -C "$src" rev-parse HEAD)"
+    [ "$got" = "$commit" ] || { echo "error: $name checked out $got, expected $commit" >&2; exit 1; }
+  fi
+  ln -sfn "$src/$root" "$CACHE/corpus/$name"
+done
+echo "corpus ready in $CACHE/corpus"

conformance/list_files.py

@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+"""list_files.py <corpus_dir>: print the files the sweep probes, one per line,
+as <corpus>/<path> in byte order.
+
+  * every file named *.h5 *.hdf5 *.he5 *.nc *.nc4 *.hdf *.h5f in each corpus,
+    except netCDF classic / 64-bit-offset / CDF5 files (magic "CDF"): they are
+    not HDF5, so neither side can read them and they say nothing;
+  * plus, for cve_hdf5, every file in cvefiles/ and fuzzerfiles/ except
+    .md/.c sources — the reproducers are mostly extension-less, and they are
+    kept whatever their bytes look like (that is their point).
+"""
+import os
+import sys
+
+EXTS = (".h5", ".hdf5", ".he5", ".nc", ".nc4", ".hdf", ".h5f")
+
+
+def walk(top):
+    for dirpath, dirnames, filenames in os.walk(top):
+        dirnames[:] = [d for d in dirnames if d != ".git"]
+        for fn in filenames:
+            p = os.path.join(dirpath, fn)
+            if os.path.isfile(p) and not os.path.islink(p):
+                yield os.path.relpath(p, top)
+
+
+def main(root):
+    out = set()
+    for corpus in sorted(os.listdir(root)):
+        top = os.path.join(root, corpus)
+        if not os.path.isdir(top):
+            continue
+        for rel in walk(top):
+            path = os.path.join(top, rel)
+            if rel.lower().endswith(EXTS):
+                with open(path, "rb") as fh:
+                    if fh.read(3) == b"CDF":
+                        continue
+                out.add(f"{corpus}/{rel}")
+            elif corpus == "cve_hdf5" and rel.split(os.sep)[0] in ("cvefiles", "fuzzerfiles") \
+                    and not rel.endswith((".md", ".c")):
+                out.add(f"{corpus}/{rel}")
+    for f in sorted(out, key=lambda s: s.encode()):
+        print(f)
+
+
+if __name__ == "__main__":
+    main(sys.argv[1])

conformance/probe/Cargo.lock

@@ -0,0 +1,458 @@
+# This file is automatically @generated by Cargo.
+# It is not intended for manual editing.
+version = 4
+
+[[package]]
+name = "adler2"
+version = "2.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
+
+[[package]]
+name = "better_io"
+version = "0.2.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ef0a3155e943e341e557863e69a708999c94ede624e37865c8e2a91b94efa78f"
+
+[[package]]
+name = "block-buffer"
+version = "0.10.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
+dependencies = [
+ "generic-array",
+]
+
+[[package]]
+name = "byteorder"
+version = "1.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
+
+[[package]]
+name = "cc"
+version = "1.5.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f360145194ee8e21db5ee7f3fcd4fe52210864c75c985dae33218202c8bbe040"
+dependencies = [
+ "find-msvc-tools",
+ "jobserver",
+ "libc",
+ "shlex",
+]
+
+[[package]]
+name = "cfg-if"
+version = "1.0.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
+
+[[package]]
+name = "clawhdf5-format"
+version = "2.7.0"
+dependencies = [
+ "byteorder",
+ "flate2",
+ "libaec-sys",
+ "lz4_flex",
+ "pco",
+ "portable-atomic",
+ "sha2",
+ "zstd",
+]
+
+[[package]]
+name = "conformance-probe"
+version = "0.1.0"
+dependencies = [
+ "clawhdf5-format",
+ "serde_json",
+ "sha2",
+]
+
+[[package]]
+name = "cpufeatures"
+version = "0.2.17"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
+dependencies = [
+ "libc",
+]
+
+[[package]]
+name = "crc32fast"
+version = "1.5.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "01a7799fd6b852db0e61728dde9a204c423b44d689dbd432522543614b490e78"
+dependencies = [
+ "cfg-if",
+]
+
+[[package]]
+name = "crunchy"
+version = "0.2.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
+
+[[package]]
+name = "crypto-common"
+version = "0.1.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
+dependencies = [
+ "generic-array",
+ "typenum",
+]
+
+[[package]]
+name = "digest"
+version = "0.10.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
+dependencies = [
+ "block-buffer",
+ "crypto-common",
+]
+
+[[package]]
+name = "dtype_dispatch"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ab23e69df104e2fd85ee63a533a22d2132ef5975dc6b36f9f3e5a7305e4a8ed7"
+
+[[package]]
+name = "find-msvc-tools"
+version = "0.1.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "aedcfb3409746eddb02b9e19ebda1c3394f759a152e48ee875a0844d1b955484"
+
+[[package]]
+name = "flate2"
+version = "1.1.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6e634e2e0ebac1ee034020da1ca582e17ffe4e0f5e985823721e168928136dcb"
+dependencies = [
+ "crc32fast",
+ "miniz_oxide",
+ "zlib-rs",
+]
+
+[[package]]
+name = "generic-array"
+version = "0.14.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
+dependencies = [
+ "typenum",
+ "version_check",
+]
+
+[[package]]
+name = "getrandom"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099"
+dependencies = [
+ "cfg-if",
+ "libc",
+ "r-efi",
+]
+
+[[package]]
+name = "half"
+version = "2.7.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b"
+dependencies = [
+ "cfg-if",
+ "crunchy",
+ "zerocopy",
+]
+
+[[package]]
+name = "itoa"
+version = "1.0.18"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682"
+
+[[package]]
+name = "jobserver"
+version = "0.1.35"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3"
+dependencies = [
+ "getrandom",
+ "libc",
+]
+
+[[package]]
+name = "libaec-sys"
+version = "0.1.0"
+dependencies = [
+ "pkg-config",
+]
+
+[[package]]
+name = "libc"
+version = "0.2.189"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
+
+[[package]]
+name = "lz4_flex"
+version = "0.11.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "373f5eceeeab7925e0c1098212f2fbc4d416adec9d35051a6ab251e824c1854a"
+dependencies = [
+ "twox-hash",
+]
+
+[[package]]
+name = "memchr"
+version = "2.8.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
+
+[[package]]
+name = "miniz_oxide"
+version = "0.9.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b63fbc4a50860e98e7b2aa7804ded1db5cbc3aff9193adaff57a6931bf7c4b4c"
+dependencies = [
+ "adler2",
+ "simd-adler32",
+]
+
+[[package]]
+name = "pco"
+version = "1.0.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "386342cad4c6e97f081568e5d910ea7d871314c843aa8fc564f2a6b64cab9456"
+dependencies = [
+ "better_io",
+ "dtype_dispatch",
+ "half",
+ "rand_xoshiro",
+]
+
+[[package]]
+name = "pkg-config"
+version = "0.3.34"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
+
+[[package]]
+name = "portable-atomic"
+version = "1.15.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "05c8b63e8d9609db387f0324918f81d68fe27748f084ef092fb35954d0539a85"
+
+[[package]]
+name = "proc-macro2"
+version = "1.0.107"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9"
+dependencies = [
+ "unicode-ident",
+]
+
+[[package]]
+name = "quote"
+version = "1.0.47"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001"
+dependencies = [
+ "proc-macro2",
+]
+
+[[package]]
+name = "r-efi"
+version = "6.0.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf"
+
+[[package]]
+name = "rand_core"
+version = "0.6.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
+
+[[package]]
+name = "rand_xoshiro"
+version = "0.6.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6f97cdb2a36ed4183de61b2f824cc45c9f1037f28afe0a322e9fff4c108b5aaa"
+dependencies = [
+ "rand_core",
+]
+
+[[package]]
+name = "serde"
+version = "1.0.229"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba"
+dependencies = [
+ "serde_core",
+]
+
+[[package]]
+name = "serde_core"
+version = "1.0.229"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48"
+dependencies = [
+ "serde_derive",
+]
+
+[[package]]
+name = "serde_derive"
+version = "1.0.229"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 3.0.6",
+]
+
+[[package]]
+name = "serde_json"
+version = "1.0.151"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14"
+dependencies = [
+ "itoa",
+ "memchr",
+ "serde",
+ "serde_core",
+ "zmij",
+]
+
+[[package]]
+name = "sha2"
+version = "0.10.9"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
+dependencies = [
+ "cfg-if",
+ "cpufeatures",
+ "digest",
+]
+
+[[package]]
+name = "shlex"
+version = "2.0.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba"
+
+[[package]]
+name = "simd-adler32"
+version = "0.3.10"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
+
+[[package]]
+name = "syn"
+version = "2.0.119"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "unicode-ident",
+]
+
+[[package]]
+name = "syn"
+version = "3.0.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "unicode-ident",
+]
+
+[[package]]
+name = "twox-hash"
+version = "2.1.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5283634e518fe9e82c7b20520bb4bc209009fd16c82077c802f8111ecbb0117a"
+
+[[package]]
+name = "typenum"
+version = "1.20.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20"
+
+[[package]]
+name = "unicode-ident"
+version = "1.0.26"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954"
+
+[[package]]
+name = "version_check"
+version = "0.9.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
+
+[[package]]
+name = "zerocopy"
+version = "0.8.59"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6df92bf3d9227be3d53173901ddbffac2babc27ae50f397776ffd6dc33f800cb"
+dependencies = [
+ "zerocopy-derive",
+]
+
+[[package]]
+name = "zerocopy-derive"
+version = "0.8.59"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ac4f328cf2f05d084e496c3e9c3f33ed0a183656a16e1fcec4d464d8373aec82"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.119",
+]
+
+[[package]]
+name = "zlib-rs"
+version = "0.6.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b268e58e7c693d7c271f93ffc4ba3b380412554231c85bf61ca7af91042a4112"
+
+[[package]]
+name = "zmij"
+version = "1.0.23"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b"
+
+[[package]]
+name = "zstd"
+version = "0.13.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a"
+dependencies = [
+ "zstd-safe",
+]
+
+[[package]]
+name = "zstd-safe"
+version = "7.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "64d80649ab6db9d9f6f9c80a40becd948eda4714a0a5ac8c4d157a32231c7882"
+dependencies = [
+ "zstd-sys",
+]
+
+[[package]]
+name = "zstd-sys"
+version = "2.1.0+zstd.1.5.7"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0ef0a8027ec3ee71300ab3bcbcd0393f434aa72b91ca6d635a39941deae8eea0"
+dependencies = [
+ "cc",
+ "pkg-config",
+]

conformance/probe/Cargo.toml

@@ -0,0 +1,25 @@
+[package]
+name = "conformance-probe"
+version = "0.1.0"
+edition = "2024"
+rust-version = "1.92"
+publish = false
+description = "Walks an HDF5 file with clawhdf5-format and prints a canonical JSON description (see conformance/README.md)"
+
+# Deliberately outside the main workspace: `cargo test --workspace` never
+# builds it, and it links the optional C codecs (zstd, libaec) that the core
+# crates' default build must not.
+[workspace]
+
+[dependencies]
+clawhdf5-format = { path = "../../crates/clawhdf5-format", features = ["lz4", "zstd", "szip", "pcodec"] }
+serde_json = "1"
+sha2 = "0.10"
+
+[profile.release]
+# Keep panics catchable (the probe records them per object) and turn integer
+# overflow into a reported panic instead of silent wraparound.
+debug = 1
+overflow-checks = true
+debug-assertions = true
+panic = "unwind"

conformance/probe/src/main.rs

@@ -0,0 +1,898 @@
+//! Conformance probe: walks an HDF5 file with clawhdf5-format (the same calls
+//! the `clawhdf5` facade makes) and prints a canonical JSON description:
+//! every hard-linked object (sorted-name DFS, deduplicated by header address),
+//! and for each dataset / attribute its shape plus the SHA-256 of its values
+//! in a canonical encoding shared with `ref.py`.
+//!
+//! Canonical value encoding (per element, concatenated, row-major):
+//!   int / float / bitfield / enum / time : element bytes, little-endian
+//!   non-IEEE-layout float (e.g. N-Bit)   : the IEEE float of the same size it converts to
+//!   int with bit offset / short precision: the full-width integer it converts to
+//!   opaque                               : raw bytes
+//!   compound                             : members in declaration order (padding dropped)
+//!   array                                : base elements row-major
+//!   string (fixed or VL)                 : b'S' + u32le len + bytes (cut at first NUL, trailing spaces stripped)
+//!   VL sequence                          : b'V' + u32le count + base elements
+//!   reference                            : b'R' (payload not compared)
+//!
+//! Every object is processed inside catch_unwind; a caught panic is recorded
+//! with its message, location and the clawhdf5 frames of its backtrace.
+
+use std::cell::RefCell;
+use std::collections::{HashMap, HashSet};
+use std::panic::{self, AssertUnwindSafe};
+use std::rc::Rc;
+
+use clawhdf5_format::attribute::extract_attributes_full;
+use clawhdf5_format::data_layout::DataLayout;
+use clawhdf5_format::data_read;
+use clawhdf5_format::dataspace::{Dataspace, DataspaceType};
+use clawhdf5_format::datatype::{Datatype, DatatypeByteOrder};
+use clawhdf5_format::filter_pipeline::FilterPipeline;
+use clawhdf5_format::global_heap::GlobalHeapCollection;
+use clawhdf5_format::group_v1::{self, GroupEntry};
+use clawhdf5_format::group_v2;
+use clawhdf5_format::message_type::MessageType;
+use clawhdf5_format::object_header::ObjectHeader;
+use clawhdf5_format::signature;
+use clawhdf5_format::superblock::Superblock;
+use clawhdf5_format::symbol_table::SymbolTableMessage;
+use serde_json::{Map, Value, json};
+use sha2::{Digest, Sha256};
+
+const MAX_BYTES: u64 = 200 * 1024 * 1024;
+const MAX_OBJECTS: usize = 200_000;
+
+thread_local! {
+    static LAST_PANIC: RefCell<Option<String>> = const { RefCell::new(None) };
+}
+
+fn install_hook() {
+    panic::set_hook(Box::new(|info| {
+        let msg = if let Some(s) = info.payload().downcast_ref::<&str>() {
+            s.to_string()
+        } else if let Some(s) = info.payload().downcast_ref::<String>() {
+            s.clone()
+        } else {
+            "<non-string panic>".into()
+        };
+        let loc = info
+            .location()
+            .map(|l| format!("{}:{}", l.file(), l.line()))
+            .unwrap_or_default();
+        let bt = std::backtrace::Backtrace::force_capture().to_string();
+        // keep only frames from clawhdf5 code
+        let mut frames = Vec::new();
+        let lines: Vec<&str> = bt.lines().collect();
+        for (i, l) in lines.iter().enumerate() {
+            let t = l.trim();
+            if t.contains("clawhdf5_format::") || t.contains("conformance_probe::") {
+                let at = lines
+                    .get(i + 1)
+                    .map(|n| n.trim())
+                    .filter(|n| n.starts_with("at "))
+                    .map(|n| {
+                        let n = n.trim_start_matches("at ");
+                        match n.find("/crates/") {
+                            Some(p) => n[p + 1..].to_string(),
+                            None => n.to_string(),
+                        }
+                    })
+                    .unwrap_or_default();
+                let name = t.split_once(": ").map(|x| x.1).unwrap_or(t);
+                frames.push(format!("{name} ({at})"));
+                if frames.len() >= 12 {
+                    break;
+                }
+            }
+        }
+        let full = format!("PANIC: {msg} @ {loc}\n  {}", frames.join("\n  "));
+        eprintln!("{full}");
+        LAST_PANIC.with(|p| *p.borrow_mut() = Some(full));
+    }));
+}
+
+/// Run `f`, turning a panic into Err("PANIC: ...").
+fn guarded<T>(f: impl FnOnce() -> Result<T, String>) -> Result<T, String> {
+    match panic::catch_unwind(AssertUnwindSafe(f)) {
+        Ok(r) => r,
+        Err(_) => Err(LAST_PANIC
+            .with(|p| p.borrow_mut().take())
+            .unwrap_or_else(|| "PANIC: <unknown>".into())),
+    }
+}
+
+fn e<E: std::fmt::Debug>(x: E) -> String {
+    format!("{x:?}")
+}
+
+struct Ctx<'a> {
+    data: &'a [u8],
+    os: u8,
+    ls: u8,
+    base_dir: std::path::PathBuf,
+    heaps: RefCell<HashMap<u64, Result<Rc<GlobalHeapCollection>, String>>>,
+}
+
+impl<'a> Ctx<'a> {
+    fn header(&self, addr: u64) -> Result<ObjectHeader, String> {
+        ObjectHeader::parse(self.data, addr as usize, self.os, self.ls).map_err(e)
+    }
+
+    fn payload(&self, h: &ObjectHeader, t: MessageType) -> Result<Option<Vec<u8>>, String> {
+        match h.messages.iter().find(|m| m.msg_type == t) {
+            None => Ok(None),
+            Some(m) => {
+                clawhdf5_format::shared_message::message_data(self.data, m, self.os, self.ls)
+                    .map(|c| Some(c.into_owned()))
+                    .map_err(e)
+            }
+        }
+    }
+
+    fn heap_obj(&self, addr: u64, idx: u32) -> Result<Vec<u8>, String> {
+        let coll = {
+            let mut cache = self.heaps.borrow_mut();
+            cache
+                .entry(addr)
+                .or_insert_with(|| {
+                    GlobalHeapCollection::parse(self.data, addr as usize, self.ls)
+                        .map(Rc::new)
+                        .map_err(e)
+                })
+                .clone()?
+        };
+        coll.get_object(idx as u16)
+            .map(|o| o.data.clone())
+            .ok_or_else(|| {
+                format!("GlobalHeapObjectNotFound {{ collection_address: {addr}, index: {idx} }}")
+            })
+    }
+
+    fn read_offset(&self, b: &[u8]) -> u64 {
+        let mut v = 0u64;
+        for (i, x) in b.iter().take(self.os as usize).enumerate() {
+            v |= (*x as u64) << (8 * i);
+        }
+        v
+    }
+
+    fn canon(&self, dt: &Datatype, b: &[u8], out: &mut Vec<u8>) -> Result<(), String> {
+        let size = dt.type_size() as usize;
+        if b.len() < size {
+            return Err(format!(
+                "canon: element slice {} < type size {size}",
+                b.len()
+            ));
+        }
+        match dt {
+            Datatype::FloatingPoint { .. } if !ieee_layout(dt) => {
+                canon_custom_float(dt, &b[..size], out)?
+            }
+            Datatype::FixedPoint { .. } if partial_int(dt) => {
+                canon_partial_int(dt, &b[..size], out)?
+            }
+            Datatype::FixedPoint { byte_order, .. }
+            | Datatype::BitField { byte_order, .. }
+            | Datatype::FloatingPoint { byte_order, .. } => match byte_order {
+                DatatypeByteOrder::LittleEndian => out.extend_from_slice(&b[..size]),
+                DatatypeByteOrder::BigEndian => out.extend(b[..size].iter().rev()),
+                DatatypeByteOrder::Vax => return Err("canon: VAX byte order".into()),
+            },
+            Datatype::Time { .. } | Datatype::Opaque { .. } => out.extend_from_slice(&b[..size]),
+            Datatype::String { .. } => canon_str(&b[..size], out),
+            Datatype::Compound { members, .. } => {
+                for m in members {
+                    let off = m.byte_offset as usize;
+                    let ms = m.datatype.type_size() as usize;
+                    if off.checked_add(ms).is_none_or(|end| end > size) {
+                        return Err(format!("canon: member {} out of bounds", m.name));
+                    }
+                    self.canon(&m.datatype, &b[off..off + ms], out)?;
+                }
+            }
+            Datatype::Reference { .. } => out.push(b'R'),
+            Datatype::Enumeration { base_type, .. } => self.canon(base_type, b, out)?,
+            Datatype::Array {
+                base_type,
+                dimensions,
+            } => {
+                let n: usize = dimensions.iter().map(|d| *d as usize).product();
+                let bs = base_type.type_size() as usize;
+                for i in 0..n {
+                    self.canon(base_type, &b[i * bs..], out)?;
+                }
+            }
+            Datatype::VariableLength {
+                is_string,
+                base_type,
+                ..
+            } => {
+                let len = u32::from_le_bytes([b[0], b[1], b[2], b[3]]) as usize;
+                let addr = self.read_offset(&b[4..]);
+                let idx_off = 4 + self.os as usize;
+                let idx = u32::from_le_bytes([
+                    b[idx_off],
+                    b[idx_off + 1],
+                    b[idx_off + 2],
+                    b[idx_off + 3],
+                ]);
+                let obj = if len == 0 || addr == 0 || addr == u64::MAX >> (64 - 8 * self.os as u32)
+                {
+                    Vec::new()
+                } else {
+                    self.heap_obj(addr, idx)?
+                };
+                if *is_string {
+                    let l = len.min(obj.len());
+                    canon_str(&obj[..l], out);
+                } else {
+                    let bs = base_type.type_size() as usize;
+                    if bs == 0 {
+                        return Err("canon: VL base size 0".into());
+                    }
+                    let need = len.checked_mul(bs).ok_or("canon: VL overflow")?;
+                    if len > 0 && obj.len() < need {
+                        return Err(format!("canon: VL object {} < {need}", obj.len()));
+                    }
+                    out.push(b'V');
+                    out.extend_from_slice(&(len as u32).to_le_bytes());
+                    for i in 0..len {
+                        self.canon(base_type, &obj[i * bs..], out)?;
+                    }
+                }
+            }
+        }
+        Ok(())
+    }
+
+    /// Returns (shape json, n_elements)
+    fn shape(ds: &Dataspace) -> (Value, u64) {
+        match ds.space_type {
+            DataspaceType::Null => (Value::String("null".into()), 0),
+            DataspaceType::Scalar => (json!([]), 1),
+            DataspaceType::Simple => {
+                let n = ds.dimensions.iter().fold(1u64, |a, d| a.saturating_mul(*d));
+                (json!(ds.dimensions), n)
+            }
+        }
+    }
+
+    fn hash_values(
+        &self,
+        dt: &Datatype,
+        raw: &[u8],
+        n: u64,
+        rec: &mut Map<String, Value>,
+    ) -> Result<(), String> {
+        let size = dt.type_size() as usize;
+        let need = (n as usize).checked_mul(size).ok_or("n*size overflow")?;
+        if raw.len() != need {
+            return Err(format!(
+                "raw length {} != n_elements {n} * type_size {size}",
+                raw.len()
+            ));
+        }
+        let mut canon = Vec::with_capacity(need);
+        for i in 0..n as usize {
+            self.canon(dt, &raw[i * size..(i + 1) * size], &mut canon)?;
+        }
+        let h = Sha256::digest(&canon);
+        rec.insert("hash".into(), Value::String(hex(&h)));
+        rec.insert(
+            "head".into(),
+            Value::String(hex(&canon[..canon.len().min(48)])),
+        );
+        Ok(())
+    }
+
+    /// VDS source files resolve next to the virtual file; like the library,
+    /// refuse absolute paths and `..`.
+    fn vds_resolver(
+        &self,
+    ) -> impl Fn(&str) -> Result<Option<Vec<u8>>, clawhdf5_format::error::FormatError> + use<> {
+        let base = self.base_dir.clone();
+        move |name: &str| {
+            use clawhdf5_format::error::FormatError;
+            let p = std::path::Path::new(name);
+            if p.is_absolute()
+                || p.components()
+                    .any(|c| matches!(c, std::path::Component::ParentDir))
+            {
+                return Err(FormatError::ChunkedReadError(format!("refused {name}")));
+            }
+            match std::fs::read(base.join(p)) {
+                Ok(b) => Ok(Some(b)),
+                Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(None),
+                Err(err) => Err(FormatError::ChunkedReadError(err.to_string())),
+            }
+        }
+    }
+
+    fn read_dataset(&self, h: &ObjectHeader, rec: &mut Map<String, Value>) -> Result<(), String> {
+        let dtb = self
+            .payload(h, MessageType::Datatype)?
+            .ok_or("MissingMessage(Datatype)")?;
+        let (dt, _) = Datatype::parse(&dtb).map_err(e)?;
+        rec.insert("dtype".into(), Value::String(dtype_str(&dt)));
+        let dsb = self
+            .payload(h, MessageType::Dataspace)?
+            .ok_or("MissingMessage(Dataspace)")?;
+        let mut ds = Dataspace::parse(&dsb, self.ls).map_err(e)?;
+        // A virtual dataset's extent can come from its sources (unlimited /
+        // printf mappings), as h5py reports it, rather than the stored one.
+        if let Some(lm) = h
+            .messages
+            .iter()
+            .find(|m| m.msg_type == MessageType::DataLayout)
+            && let Ok(dl @ DataLayout::Virtual { .. }) =
+                DataLayout::parse(&lm.data, self.os, self.ls)
+        {
+            let resolver = self.vds_resolver();
+            ds.dimensions = clawhdf5_format::vds::virtual_dataset_extent(
+                self.data,
+                &dl,
+                &ds,
+                self.os,
+                self.ls,
+                Some(&resolver),
+            )
+            .map_err(e)?;
+        }
+        let (shape, n) = Self::shape(&ds);
+        rec.insert("shape".into(), shape);
+        if n.saturating_mul(dt.type_size() as u64) > MAX_BYTES {
+            rec.insert("skipped".into(), Value::String("too large".into()));
+            return Ok(());
+        }
+        let lm = h
+            .messages
+            .iter()
+            .find(|m| m.msg_type == MessageType::DataLayout)
+            .ok_or("MissingMessage(DataLayout)")?;
+        let dl = DataLayout::parse(&lm.data, self.os, self.ls).map_err(e)?;
+        rec.insert(
+            "layout".into(),
+            Value::String(
+                match &dl {
+                    DataLayout::Compact { .. } => "compact",
+                    DataLayout::Contiguous { .. } => "contiguous",
+                    DataLayout::Chunked { .. } => "chunked",
+                    DataLayout::Virtual { .. } => "virtual",
+                }
+                .into(),
+            ),
+        );
+        let pipeline = match self.payload(h, MessageType::FilterPipeline)? {
+            Some(p) => Some(FilterPipeline::parse(&p).map_err(e)?),
+            None => None,
+        };
+        if let Some(p) = &pipeline {
+            rec.insert(
+                "filters".into(),
+                json!(p.filters.iter().map(|f| f.filter_id).collect::<Vec<_>>()),
+            );
+        }
+        let raw = if matches!(dl, DataLayout::Virtual { .. }) {
+            let resolver = self.vds_resolver();
+            let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
+                self.data,
+                &h.messages,
+                self.os,
+                self.ls,
+            )
+            .map_err(e)?;
+            clawhdf5_format::vds::read_virtual_dataset(
+                self.data,
+                &dl,
+                &ds,
+                &dt,
+                fill.as_deref(),
+                self.os,
+                self.ls,
+                Some(&resolver),
+            )
+            .map_err(e)?
+            .data
+        } else {
+            let cache = clawhdf5_format::chunk_cache::ChunkCache::new();
+            clawhdf5_format::fill_value::read_full_with_fill::<clawhdf5_format::error::FormatError>(
+                &h.messages,
+                self.data,
+                &dl,
+                &ds,
+                dt.type_size() as usize,
+                self.os,
+                self.ls,
+                || {
+                    data_read::read_raw_data_cached(
+                        self.data,
+                        &dl,
+                        &ds,
+                        &dt,
+                        pipeline.as_ref(),
+                        self.os,
+                        self.ls,
+                        &cache,
+                    )
+                },
+            )
+            .map_err(e)?
+        };
+        self.hash_values(&dt, &raw, n, rec)
+    }
+
+    fn attrs(&self, h: &ObjectHeader) -> Result<Map<String, Value>, String> {
+        let msgs = extract_attributes_full(self.data, h, self.os, self.ls).map_err(e)?;
+        let mut out = Map::new();
+        for a in &msgs {
+            let r = guarded(|| {
+                let mut rec = Map::new();
+                rec.insert("dtype".into(), Value::String(dtype_str(&a.datatype)));
+                let (shape, n) = Self::shape(&a.dataspace);
+                rec.insert("shape".into(), shape);
+                self.hash_values(&a.datatype, &a.raw_data, n, &mut rec)?;
+                Ok(rec)
+            });
+            let v = match r {
+                Ok(rec) => Value::Object(rec),
+                Err(msg) => json!({ "error": msg }),
+            };
+            out.insert(a.name.clone(), v);
+        }
+        Ok(out)
+    }
+
+    fn entries(&self, h: &ObjectHeader) -> Result<Vec<GroupEntry>, String> {
+        let v1 = h
+            .messages
+            .iter()
+            .find(|m| m.msg_type == MessageType::SymbolTable);
+        if let Some(m) = v1 {
+            let stm = SymbolTableMessage::parse(&m.data, self.os).map_err(e)?;
+            group_v1::resolve_v1_group_entries(self.data, &stm, self.os, self.ls).map_err(e)
+        } else if h
+            .messages
+            .iter()
+            .any(|m| m.msg_type == MessageType::LinkInfo || m.msg_type == MessageType::Link)
+        {
+            group_v2::resolve_v2_group_entries(self.data, h, self.os, self.ls).map_err(e)
+        } else {
+            Ok(Vec::new())
+        }
+    }
+}
+
+/// Element bytes as an unsigned integer (at most 16 bytes), honouring byte order.
+fn element_bits(b: &[u8], byte_order: &DatatypeByteOrder) -> Result<u128, String> {
+    if b.len() > 16 {
+        return Err(format!("canon: {}-byte numeric element", b.len()));
+    }
+    let mut v = 0u128;
+    match byte_order {
+        DatatypeByteOrder::LittleEndian => {
+            for (i, x) in b.iter().enumerate() {
+                v |= u128::from(*x) << (8 * i);
+            }
+        }
+        DatatypeByteOrder::BigEndian => {
+            for x in b {
+                v = (v << 8) | u128::from(*x);
+            }
+        }
+        DatatypeByteOrder::Vax => return Err("canon: VAX byte order".into()),
+    }
+    Ok(v)
+}
+
+fn field(v: u128, pos: u32, len: u32) -> u128 {
+    if len == 0 || pos >= 128 {
+        return 0;
+    }
+    let v = v >> pos;
+    if len >= 128 {
+        v
+    } else {
+        v & ((1u128 << len) - 1)
+    }
+}
+
+/// True when a float's bit fields are exactly IEEE 754 binary16/32/64 for its
+/// size. h5py hands back such a type's bytes untouched; any other layout (an
+/// N-Bit `H5Tset_precision` float, say) is *converted* by libhdf5 into the
+/// numpy float of the same size, so comparing raw bytes would be meaningless.
+fn ieee_layout(dt: &Datatype) -> bool {
+    let Datatype::FloatingPoint {
+        size,
+        bit_offset,
+        bit_precision,
+        exponent_location,
+        exponent_size,
+        mantissa_location,
+        mantissa_size,
+        exponent_bias,
+        ..
+    } = dt
+    else {
+        return true;
+    };
+    let std = match size {
+        2 => (16, 10, 5, 10, 15),
+        4 => (32, 23, 8, 23, 127),
+        8 => (64, 52, 11, 52, 1023),
+        _ => return true, // no same-size numpy float to convert to: compare raw
+    };
+    *bit_offset == 0
+        && (
+            *bit_precision,
+            *exponent_location,
+            *exponent_size,
+            *mantissa_size,
+            *exponent_bias,
+        ) == (std.0, std.1, std.2, std.3, std.4)
+        && *mantissa_location == 0
+}
+
+/// Canonicalise a non-IEEE-layout float the way libhdf5's float->float
+/// conversion presents it to h5py: as the IEEE float of the same size.
+/// Assumes the implied-leading-one normalisation and the sign bit at the top
+/// of the precision (what `H5Tset_precision` produces; the parser does not
+/// keep either field).
+fn canon_custom_float(dt: &Datatype, b: &[u8], out: &mut Vec<u8>) -> Result<(), String> {
+    let Datatype::FloatingPoint {
+        size,
+        byte_order,
+        bit_offset,
+        bit_precision,
+        exponent_location,
+        exponent_size,
+        mantissa_location,
+        mantissa_size,
+        exponent_bias,
+    } = dt
+    else {
+        unreachable!()
+    };
+    let (esize, msize) = (u32::from(*exponent_size), u32::from(*mantissa_size));
+    if esize == 0 || esize > 30 || msize > 64 {
+        return Err(format!("canon: unsupported float layout e{esize} m{msize}"));
+    }
+    let v = element_bits(b, byte_order)?;
+    let sign_pos = (u32::from(*bit_offset) + u32::from(*bit_precision)).saturating_sub(1);
+    let neg = field(v, sign_pos, 1) == 1;
+    let e = field(v, u32::from(*exponent_location), esize) as i64;
+    let m = field(v, u32::from(*mantissa_location), msize);
+    let emax = (1i64 << esize) - 1;
+    let bias = i64::from(*exponent_bias);
+    let mag = if e == emax {
+        if m == 0 { f64::INFINITY } else { f64::NAN }
+    } else if e == 0 {
+        (m as f64) * 2f64.powi((1 - bias - msize as i64) as i32)
+    } else {
+        ((1u128 << msize) as f64 + m as f64) * 2f64.powi((e - bias - msize as i64) as i32)
+    };
+    let x = if neg { -mag } else { mag };
+    match size {
+        2 => out
+            .extend_from_slice(&clawhdf5_format::float16::f32_to_f16_bits(x as f32).to_le_bytes()),
+        4 => out.extend_from_slice(&(x as f32).to_le_bytes()),
+        8 => out.extend_from_slice(&x.to_le_bytes()),
+        _ => unreachable!("ieee_layout keeps other sizes raw"),
+    }
+    Ok(())
+}
+
+/// Integers stored with a bit offset or reduced precision (N-Bit): libhdf5
+/// converts them to the full-width integer of the same size, shifting the
+/// value down and sign-extending from the top precision bit.
+fn canon_partial_int(dt: &Datatype, b: &[u8], out: &mut Vec<u8>) -> Result<(), String> {
+    let Datatype::FixedPoint {
+        size,
+        byte_order,
+        signed,
+        bit_offset,
+        bit_precision,
+    } = dt
+    else {
+        unreachable!()
+    };
+    let prec = u32::from(*bit_precision);
+    let v = element_bits(b, byte_order)?;
+    let mut x = field(v, u32::from(*bit_offset), prec);
+    if *signed && prec > 0 && prec < 128 && field(x, prec - 1, 1) == 1 {
+        x |= !0u128 << prec;
+    }
+    out.extend_from_slice(&x.to_le_bytes()[..*size as usize]);
+    Ok(())
+}
+
+fn partial_int(dt: &Datatype) -> bool {
+    matches!(dt, Datatype::FixedPoint { size, bit_offset, bit_precision, .. }
+        if *bit_offset != 0 || u32::from(*bit_precision) != size * 8)
+}
+
+fn canon_str(b: &[u8], out: &mut Vec<u8>) {
+    let cut = b.iter().position(|&c| c == 0).unwrap_or(b.len());
+    let mut s = &b[..cut];
+    while let [rest @ .., b' '] = s {
+        s = rest;
+    }
+    out.push(b'S');
+    out.extend_from_slice(&(s.len() as u32).to_le_bytes());
+    out.extend_from_slice(s);
+}
+
+fn hex(b: &[u8]) -> String {
+    b.iter().map(|x| format!("{x:02x}")).collect()
+}
+
+fn dtype_str(dt: &Datatype) -> String {
+    match dt {
+        Datatype::FixedPoint {
+            size,
+            signed,
+            byte_order,
+            ..
+        } => {
+            format!(
+                "{}{}{}",
+                bo(byte_order),
+                if *signed { "i" } else { "u" },
+                size
+            )
+        }
+        Datatype::FloatingPoint {
+            size, byte_order, ..
+        } => format!("{}f{}", bo(byte_order), size),
+        Datatype::BitField {
+            size, byte_order, ..
+        } => format!("{}b{}", bo(byte_order), size),
+        Datatype::Time { size, .. } => format!("time{size}"),
+        Datatype::String { size, .. } => format!("S{size}"),
+        Datatype::Opaque { size, .. } => format!("V{size}"),
+        Datatype::Compound { size, members } => format!(
+            "{{{}}}{size}",
+            members
+                .iter()
+                .map(|m| format!("{}:{}", m.name, dtype_str(&m.datatype)))
+                .collect::<Vec<_>>()
+                .join(",")
+        ),
+        Datatype::Reference { ref_type, .. } => format!("ref({ref_type:?})"),
+        Datatype::Enumeration { base_type, .. } => format!("enum({})", dtype_str(base_type)),
+        Datatype::VariableLength {
+            is_string: true, ..
+        } => "vlstr".into(),
+        Datatype::VariableLength { base_type, .. } => format!("vlen({})", dtype_str(base_type)),
+        Datatype::Array {
+            base_type,
+            dimensions,
+        } => format!("({}){dimensions:?}", dtype_str(base_type)),
+    }
+}
+
+fn bo(b: &DatatypeByteOrder) -> &'static str {
+    match b {
+        DatatypeByteOrder::LittleEndian => "<",
+        DatatypeByteOrder::BigEndian => ">",
+        DatatypeByteOrder::Vax => "vax",
+    }
+}
+
+fn is_group(h: &ObjectHeader) -> bool {
+    h.messages.iter().any(|m| {
+        matches!(
+            m.msg_type,
+            MessageType::LinkInfo | MessageType::Link | MessageType::SymbolTable
+        )
+    })
+}
+
+fn main() {
+    install_hook();
+    let path = std::env::args().nth(1).expect("usage: probe <file>");
+    let mut top = Map::new();
+    top.insert("file".into(), Value::String(path.clone()));
+    let data = match std::fs::read(&path) {
+        Ok(d) => d,
+        Err(err) => {
+            top.insert("open_error".into(), Value::String(format!("Io({err})")));
+            println!("{}", Value::Object(top));
+            return;
+        }
+    };
+    // Every address is relative to the superblock: look at the file from
+    // there on (past any user block), as libhdf5 does.
+    let hdf5: &[u8] = match signature::find_signature(&data) {
+        Ok(off) => &data[off..],
+        Err(_) => &data,
+    };
+    let sb = guarded(|| Superblock::parse(hdf5, 0).map_err(e));
+    let sb = match sb {
+        Ok(sb) => sb,
+        Err(msg) => {
+            top.insert("open_error".into(), Value::String(msg));
+            println!("{}", Value::Object(top));
+            return;
+        }
+    };
+    top.insert("superblock_version".into(), json!(sb.version));
+    let ctx = Ctx {
+        data: hdf5,
+        os: sb.offset_size,
+        ls: sb.length_size,
+        base_dir: std::path::Path::new(&path)
+            .parent()
+            .map(|p| p.to_path_buf())
+            .unwrap_or_default(),
+        heaps: RefCell::new(HashMap::new()),
+    };
+    let mut objects: Vec<Value> = Vec::new();
+    let mut visited = HashSet::new();
+    let mut soft_v1 = 0u64;
+    // explicit DFS stack: (address, path)
+    let mut stack: Vec<(u64, String)> = vec![(sb.root_group_address, "/".to_string())];
+    while let Some((addr, p)) = stack.pop() {
+        if objects.len() >= MAX_OBJECTS {
+            top.insert("truncated".into(), json!(true));
+            break;
+        }
+        if !visited.insert(addr) {
+            continue;
+        }
+        let mut rec = Map::new();
+        rec.insert("path".into(), Value::String(p.clone()));
+        let r = guarded(|| {
+            let h = ctx.header(addr)?;
+            Ok(h)
+        });
+        let h = match r {
+            Ok(h) => h,
+            Err(msg) => {
+                rec.insert("kind".into(), Value::String("unknown".into()));
+                rec.insert("error".into(), Value::String(msg));
+                objects.push(Value::Object(rec));
+                continue;
+            }
+        };
+        let is_ds = h
+            .messages
+            .iter()
+            .any(|m| m.msg_type == MessageType::DataLayout);
+        let kind = if is_ds {
+            "dataset"
+        } else if is_group(&h) || addr == sb.root_group_address {
+            "group"
+        } else if h
+            .messages
+            .iter()
+            .any(|m| m.msg_type == MessageType::Datatype)
+        {
+            "datatype"
+        } else {
+            "unknown"
+        };
+        rec.insert("kind".into(), Value::String(kind.into()));
+        if kind == "dataset"
+            && let Err(msg) = guarded(|| ctx.read_dataset(&h, &mut rec))
+        {
+            rec.insert("error".into(), Value::String(msg));
+        }
+        if kind != "datatype" {
+            match guarded(|| ctx.attrs(&h)) {
+                Ok(m) => {
+                    rec.insert("attrs".into(), Value::Object(m));
+                }
+                Err(msg) => {
+                    rec.insert("attrs_error".into(), Value::String(msg));
+                }
+            }
+        }
+        if kind == "group" {
+            match guarded(|| ctx.entries(&h)) {
+                Ok(mut ents) => {
+                    ents.retain(|en| {
+                        if en.cache_type == 2 {
+                            soft_v1 += 1;
+                            false
+                        } else {
+                            true
+                        }
+                    });
+                    ents.sort_by(|a, b| a.name.cmp(&b.name));
+                    let base = if p == "/" { String::new() } else { p.clone() };
+                    for en in ents.into_iter().rev() {
+                        stack.push((en.object_header_address, format!("{base}/{}", en.name)));
+                    }
+                }
+                Err(msg) => {
+                    rec.insert("list_error".into(), Value::String(msg));
+                }
+            }
+        }
+        objects.push(Value::Object(rec));
+    }
+    if soft_v1 > 0 {
+        top.insert("v1_soft_link_entries".into(), json!(soft_v1));
+    }
+    top.insert("objects".into(), Value::Array(objects));
+    println!("{}", Value::Object(top));
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    /// The N-Bit float of libhdf5's `test/testfiles/le_data.h5`
+    /// (`Nbit_float_data_le`): offset 7, precision 20, sign bit 26, exponent
+    /// 20+6 (bias 31), mantissa 7+13.
+    fn nbit_f32(byte_order: DatatypeByteOrder) -> Datatype {
+        Datatype::FloatingPoint {
+            size: 4,
+            byte_order,
+            bit_offset: 7,
+            bit_precision: 20,
+            exponent_location: 20,
+            exponent_size: 6,
+            mantissa_location: 7,
+            mantissa_size: 13,
+            exponent_bias: 31,
+        }
+    }
+
+    fn canon_one(dt: &Datatype, bytes: &[u8]) -> Vec<u8> {
+        let mut out = Vec::new();
+        canon_custom_float(dt, bytes, &mut out).unwrap();
+        out
+    }
+
+    #[test]
+    fn nbit_float_canonicalises_to_the_value_libhdf5_returns() {
+        let le = nbit_f32(DatatypeByteOrder::LittleEndian);
+        let be = nbit_f32(DatatypeByteOrder::BigEndian);
+        assert!(!ieee_layout(&le));
+        // 1.0: exponent = bias, mantissa 0
+        let one: u32 = 31 << 20;
+        assert_eq!(canon_one(&le, &one.to_le_bytes()), 1.0f32.to_le_bytes());
+        assert_eq!(canon_one(&be, &one.to_be_bytes()), 1.0f32.to_le_bytes());
+        // -2.1999512 (h5py's reading of the file's -2.2): sign, e = 32, m = 819
+        let v: u32 = (1 << 26) | (32 << 20) | (819 << 7);
+        assert_eq!(
+            canon_one(&le, &v.to_le_bytes()),
+            (-2.199_951_2f32).to_le_bytes()
+        );
+        assert_eq!(canon_one(&le, &[0; 4]), 0.0f32.to_le_bytes());
+    }
+
+    #[test]
+    fn ieee_floats_keep_their_raw_bytes() {
+        let f32le = Datatype::FloatingPoint {
+            size: 4,
+            byte_order: DatatypeByteOrder::LittleEndian,
+            bit_offset: 0,
+            bit_precision: 32,
+            exponent_location: 23,
+            exponent_size: 8,
+            mantissa_location: 0,
+            mantissa_size: 23,
+            exponent_bias: 127,
+        };
+        assert!(ieee_layout(&f32le));
+    }
+
+    #[test]
+    fn partial_precision_int_is_shifted_and_sign_extended() {
+        let dt = Datatype::FixedPoint {
+            size: 4,
+            byte_order: DatatypeByteOrder::BigEndian,
+            signed: true,
+            bit_offset: 4,
+            bit_precision: 17,
+        };
+        assert!(partial_int(&dt));
+        let stored = (((-5i32) as u32) & 0x1_FFFF) << 4;
+        let mut out = Vec::new();
+        canon_partial_int(&dt, &stored.to_be_bytes(), &mut out).unwrap();
+        assert_eq!(out, (-5i32).to_le_bytes());
+    }
+}

conformance/ref.py

@@ -0,0 +1,259 @@
+#!/usr/bin/env python3
+"""Reference probe: same JSON as the Rust `conformance-probe`, produced with h5py.
+
+Walk: iterative DFS from '/', children in sorted (UTF-8 byte) name order, hard
+links only, each object once (first path wins, deduplicated by object identity).
+Canonical value encoding: see harness/src/main.rs.
+"""
+import hashlib
+import json
+import os
+import struct
+import sys
+
+import numpy as np
+import h5py
+
+try:
+    import hdf5plugin  # noqa: F401  registers blosc/lz4/zstd/bzip2/... filters
+except Exception:  # pragma: no cover
+    pass
+
+MAX_BYTES = 200 * 1024 * 1024
+MAX_OBJECTS = 200_000
+
+
+def canon_str(b, out):
+    if isinstance(b, str):
+        b = b.encode("utf-8", "surrogateescape")
+    b = bytes(b)
+    cut = b.find(b"\x00")
+    if cut >= 0:
+        b = b[:cut]
+    b = b.rstrip(b" ")
+    out += b"S" + struct.pack("<I", len(b)) + b
+
+
+def simple(dt):
+    if dt.fields:
+        return all(simple(dt.fields[n][0]) for n in dt.names)
+    if dt.subdtype:
+        return simple(dt.subdtype[0])
+    return dt.kind in "iufcbV"
+
+
+def packed(dt):
+    if dt.fields:
+        return np.dtype([(n, packed(dt.fields[n][0])) for n in dt.names])
+    if dt.subdtype:
+        base, shape = dt.subdtype
+        return np.dtype((packed(base), shape))
+    if dt.kind in "iufcb":
+        return dt.newbyteorder("<")
+    return dt
+
+
+def canon_el(dt, val, out):
+    if dt.fields:
+        for n in dt.names:
+            canon_el(dt.fields[n][0], val[n], out)
+        return
+    if dt.subdtype:
+        base, _ = dt.subdtype
+        for x in np.asarray(val).reshape(-1):
+            canon_el(base, x, out)
+        return
+    k = dt.kind
+    if k in "iufcb":
+        out += np.asarray(val, dtype=dt).astype(dt.newbyteorder("<")).tobytes()
+    elif k == "V":
+        out += np.asarray(val, dtype=dt).tobytes()
+    elif k == "S":
+        canon_str(val, out)
+    elif k == "O":
+        if h5py.check_string_dtype(dt) is not None:
+            canon_str(val if val is not None else b"", out)
+        elif h5py.check_ref_dtype(dt) is not None:
+            out += b"R"
+        else:
+            base = h5py.check_vlen_dtype(dt)
+            if base is None:
+                raise TypeError(f"unhandled object dtype {dt!r}")
+            arr = np.asarray(val if val is not None else [], dtype=base).reshape(-1)
+            out += b"V" + struct.pack("<I", arr.shape[0])
+            if simple(base):
+                out += arr.astype(packed(base)).tobytes()
+            else:
+                for x in arr:
+                    canon_el(base, x, out)
+    elif k == "U":
+        canon_str(str(val), out)
+    else:
+        raise TypeError(f"unhandled dtype kind {k} ({dt!r})")
+
+
+def has_obj(dt):
+    if dt.fields:
+        return any(has_obj(dt.fields[n][0]) for n in dt.names)
+    if dt.subdtype:
+        return has_obj(dt.subdtype[0])
+    return dt.kind == "O"
+
+
+def note_conversion(tid, dt, rec):
+    """h5py converts some file types (FP8, bfloat16, x87 long double, ...) to a
+    different-sized numpy type; then value bytes are not comparable."""
+    try:
+        if not has_obj(dt) and tid.get_size() != dt.itemsize:
+            rec["converted"] = f"file type size {tid.get_size()} -> numpy {dt} ({dt.itemsize})"
+    except Exception:  # noqa: BLE001
+        pass
+
+
+def hash_values(arr, dt, rec):
+    if dt.subdtype is not None:
+        # h5py expands an HDF5 array element type into trailing array dims
+        dt = dt.subdtype[0]
+    arr = np.asarray(arr, dtype=dt)
+    if simple(dt):
+        c = np.ascontiguousarray(arr).astype(packed(dt)).tobytes()
+    else:
+        out = bytearray()
+        for x in arr.reshape(-1):
+            canon_el(dt, x, out)
+        c = bytes(out)
+    rec["hash"] = hashlib.sha256(c).hexdigest()
+    rec["head"] = c[:48].hex()
+
+
+def err(e):
+    s = f"{type(e).__name__}: {e}"
+    return s.splitlines()[0][:400] if s else type(e).__name__
+
+
+def shape_of(s):
+    return "null" if s is None else list(s)
+
+
+def n_bytes(shape, tid):
+    n = 1
+    for d in shape or ():
+        n *= d
+    return n * tid.get_size()
+
+
+def read_attrs(obj):
+    out = {}
+    names = sorted(obj.attrs.keys(), key=lambda s: s.encode("utf-8", "surrogateescape"))
+    for name in names:
+        rec = {}
+        try:
+            aid = obj.attrs.get_id(name)
+            rec["dtype"] = str(aid.dtype)
+            rec["shape"] = shape_of(aid.shape)
+            note_conversion(aid.get_type(), aid.dtype, rec)
+            if aid.shape is None:
+                hash_values(np.empty((0,), dtype=aid.dtype), aid.dtype, rec)
+            else:
+                val = obj.attrs[name]
+                hash_values(val, aid.dtype, rec)
+        except Exception as e:  # noqa: BLE001
+            rec = {"error": err(e)}
+        out[name] = rec
+    return out
+
+
+def main(path):
+    top = {"file": path}
+    try:
+        f = h5py.File(path, "r")
+    except Exception as e:  # noqa: BLE001
+        top["open_error"] = err(e)
+        print(json.dumps(top))
+        return
+    objects = []
+    seen = set()
+    stack = [("/", None)]
+    while stack:
+        p, obj = stack.pop()
+        if len(objects) >= MAX_OBJECTS:
+            top["truncated"] = True
+            break
+        rec = {"path": p}
+        try:
+            if obj is None:
+                obj = f[p]
+            key = hash(obj.id)  # h5py ObjectID hash = (fileno, object address/token)
+        except Exception as e:  # noqa: BLE001
+            rec["kind"] = "unknown"
+            rec["error"] = err(e)
+            objects.append(rec)
+            continue
+        if key in seen:
+            continue
+        seen.add(key)
+        if isinstance(obj, h5py.Dataset):
+            kind = "dataset"
+        elif isinstance(obj, h5py.Group):
+            kind = "group"
+        elif isinstance(obj, h5py.Datatype):
+            kind = "datatype"
+        else:
+            kind = "unknown"
+        rec["kind"] = kind
+        if kind == "dataset":
+            try:
+                dt = obj.dtype
+                rec["dtype"] = str(dt)
+                rec["shape"] = shape_of(obj.shape)
+                note_conversion(obj.id.get_type(), dt, rec)
+                if obj.shape is None:
+                    hash_values(np.empty((0,), dtype=dt), dt, rec)
+                elif n_bytes(obj.shape, obj.id.get_type()) > MAX_BYTES:
+                    rec["skipped"] = "too large"
+                else:
+                    arr = np.empty(obj.shape, dtype=dt)
+                    if arr.size:
+                        try:
+                            obj.read_direct(arr)
+                        except Exception:  # noqa: BLE001
+                            arr = obj[()]
+                    hash_values(arr, dt, rec)
+            except Exception as e:  # noqa: BLE001
+                rec["error"] = err(e)
+        if kind != "datatype":
+            try:
+                rec["attrs"] = read_attrs(obj)
+            except Exception as e:  # noqa: BLE001
+                rec["attrs_error"] = err(e)
+        if kind == "group":
+            try:
+                names = sorted(obj.keys(), key=lambda s: s.encode("utf-8", "surrogateescape"))
+                base = "" if p == "/" else p
+                kids = []
+                for n in names:
+                    try:
+                        link = obj.get(n, getlink=True)
+                    except Exception:  # noqa: BLE001
+                        link = None
+                    if link is not None and not isinstance(link, h5py.HardLink):
+                        continue
+                    kids.append(f"{base}/{n}")
+                for k in reversed(kids):
+                    stack.append((k, None))
+            except Exception as e:  # noqa: BLE001
+                rec["list_error"] = err(e)
+        objects.append(rec)
+    top["objects"] = objects
+    print(json.dumps(top), flush=True)
+    # Exit without tearing down the h5py objects: freeing them for some files
+    # that hold references (hdf5's h5repack_attr_refs.h5, cve-2024-32623.h5)
+    # makes libhdf5 2.0 abort with "free(): chunks in smallbin corrupted"
+    # about half the time. That happens after the reading is done, so it says
+    # nothing about what h5py read, but it flipped those files between ok and
+    # h5py-cannot-read from one run to the next.
+    os._exit(0)
+
+
+if __name__ == "__main__":
+    main(sys.argv[1])

conformance/report.py

@@ -0,0 +1,335 @@
+#!/usr/bin/env python3
+"""report.py <results_dir> <CONFORMANCE.md> <corpus_dir>
+
+Render the sweep's results (compare.py's results.json plus the raw per-side
+runs) as CONFORMANCE.md, and write <results_dir>/report-meta.json (commit,
+date, versions) for check.py --update.
+"""
+import collections
+import datetime
+import json
+import os
+import platform
+
+import subprocess
+import sys
+
+import h5py
+import numpy
+
+try:
+    import hdf5plugin
+    HDF5PLUGIN = hdf5plugin.version
+except Exception:  # noqa: BLE001
+    HDF5PLUGIN = "not installed"
+
+R, OUT_MD, CORPUS = sys.argv[1], sys.argv[2], sys.argv[3]
+HERE = os.path.dirname(os.path.abspath(__file__))
+ROOT = os.path.dirname(HERE)
+CLASSES = ["ok", "our-error", "mismatch", "h5py-cannot-read", "panic", "hang", "crash", "oom"]
+
+
+def sh(*cmd, cwd=ROOT):
+    try:
+        return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=30).stdout.strip()
+    except Exception:  # noqa: BLE001
+        return ""
+
+
+def cpu_model():
+    try:
+        for ln in open("/proc/cpuinfo"):
+            if ln.startswith(("model name", "Model")):
+                return ln.split(":", 1)[1].strip()
+    except OSError:
+        pass
+    return platform.processor() or "unknown"
+
+
+def mem_gib():
+    try:
+        for ln in open("/proc/meminfo"):
+            if ln.startswith("MemTotal:"):
+                return f"{int(ln.split()[1]) / 1048576:.0f} GiB"
+    except OSError:
+        pass
+    return "?"
+
+
+res = json.load(open(os.path.join(R, "results.json")))
+meta_run = json.load(open(os.path.join(R, "meta.json"))) if os.path.exists(os.path.join(R, "meta.json")) else {}
+rows = res["rows"]
+issues = res.get("issues", {})
+
+# safe.directory: a checkout owned by another user (a container) is still ours to read
+commit = sh("git", "-c", "safe.directory=*", "rev-parse", "HEAD") or os.environ.get("GITHUB_SHA", "unknown")
+lib_dirty = sh("git", "-c", "safe.directory=*", "status", "--porcelain", "--", "crates", "Cargo.toml")
+h5dump_v = sh("h5dump", "--version").replace("h5dump: ", "")
+meta = {
+    "date": datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d %H:%M UTC"),
+    "commit": commit + (" (library sources modified)" if lib_dirty else ""),
+    "reference": f"h5py {h5py.__version__} / HDF5 {h5py.version.hdf5_version}",
+}
+json.dump(meta, open(os.path.join(R, "report-meta.json"), "w"), indent=1)
+
+pins = []
+for ln in open(os.path.join(HERE, "corpus.txt")):
+    if ln.strip() and not ln.lstrip().startswith("#"):
+        name, url, rev, root, *_ = ln.split()
+        pins.append((name, url, rev, root))
+
+by_corpus = collections.defaultdict(collections.Counter)
+for r in rows:
+    by_corpus[r["corpus"]][r["class"]] += 1
+total = collections.Counter(r["class"] for r in rows)
+
+
+def ex_list(files, n=3):
+    s = ", ".join(f"`{f}`" for f in files[:n])
+    return s + (f" (+{len(files) - n} more)" if len(files) > n else "")
+
+
+# --- known causes that are not clawhdf5 bugs --------------------------------
+def is_h5py_be_vlen(i):
+    """h5py returns the elements of a VL sequence of a big-endian base type
+    with their file (big-endian) bytes but a native-endian dtype."""
+    return (i["kind"] == "mismatch" and i["key"] in ("values", "attr-values")
+            and (i.get("ref_dtype") == "object") and (i.get("ours_dtype") or "").startswith("vlen(")
+            and ">" in (i.get("ours_dtype") or ""))
+
+
+known = collections.defaultdict(list)
+for r in rows:
+    if r["class"] != "mismatch":
+        continue
+    iss = issues.get(r["file"], [])
+    if iss and all(is_h5py_be_vlen(i) for i in iss):
+        known["h5py-be-vlen"].append(r["file"])
+
+
+# --- the CVE corpus: clawhdf5 vs h5dump vs h5py ------------------------------
+def side(run, name):
+    p = os.path.join(R, "runs", run, name)
+    if not os.path.exists(p + ".rc"):
+        return None
+    rc = int(open(p + ".rc").read().strip() or -1)
+    err = open(p + ".err", errors="replace").read()
+    try:
+        j = json.load(open(p + ".json"))
+    except Exception:  # noqa: BLE001
+        j = None
+    return rc, err, j
+
+
+def outcome(s, rust=False):
+    """-> (bucket, text). bucket in read / error / panic / crash / hang / oom."""
+    if s is None:
+        return "missing", "not run"
+    rc, err, j = s
+    if rc in (137, 124):
+        return "hang", "hang (killed at timeout)"
+    if "memory allocation of" in err or "MemoryError" in err or "bad_alloc" in err or "Cannot allocate" in err:
+        return "oom", "out of memory"
+    if rust and (rc == 101 or "PANIC:" in err):
+        return "panic", "panic"
+    if "overflowed its stack" in err:
+        return "crash", "stack overflow"
+    if rc == 139:
+        return "crash", "SIGSEGV"
+    if rc == 134:
+        return "crash", "SIGABRT" + (" (heap corruption)" if ("corrupted" in err or "free()" in err) else "")
+    if rc > 128:
+        return "crash", f"signal {rc - 128}"
+    if j is None:
+        return ("error", "error exit") if rc in (0, 1) else ("crash", f"exit {rc}")
+    if "open_error" in j:
+        return "error", "open error"
+    objs = j.get("objects", [])
+    ne = sum(1 for o in objs for k in ("error", "attrs_error", "list_error") if k in o)
+    ne += sum(1 for o in objs for a in (o.get("attrs") or {}).values() if "error" in a)
+    return "read", f"read {len(objs)} obj" + (f", {ne} errors" if ne else "")
+
+
+def h5dump_outcome(s):
+    if s is None:
+        return "missing", "not run"
+    rc, err, _ = s
+    if rc in (137, 124):
+        return "hang", "hang (killed at timeout)"
+    if "memory allocation" in err or "Cannot allocate" in err:
+        return "oom", "out of memory"
+    if rc == 139:
+        return "crash", "SIGSEGV"
+    if rc == 134:
+        return "crash", "SIGABRT" + (" (heap corruption)" if ("corrupted" in err or "free()" in err) else "")
+    if rc > 128:
+        return "crash", f"signal {rc - 128}"
+    return ("read", "ok") if rc == 0 else ("error", "error exit")
+
+
+cve_rows = []
+buckets = {"clawhdf5": collections.Counter(), "h5dump": collections.Counter(), "h5py": collections.Counter()}
+ours_panic = {r["file"] for r in rows if r["class"] == "panic"}
+for r in rows:
+    if r["corpus"] != "cve_hdf5":
+        continue
+    run = r["file"].replace("/", "__")
+    o = outcome(side(run, "ours"), rust=True)
+    if o[0] == "read" and r["file"] in ours_panic:
+        o = ("panic", "caught panic")
+    p = outcome(side(run, "ref"))
+    d = h5dump_outcome(side(run, "h5dump"))
+    buckets["clawhdf5"][o[0]] += 1
+    buckets["h5py"][p[0]] += 1
+    buckets["h5dump"][d[0]] += 1
+    cve_rows.append((r["file"].split("/", 1)[1], d[1], p[1], o[1], r["class"]))
+
+# --- render -----------------------------------------------------------------
+L = []
+w = L.append
+w("# clawhdf5 conformance report")
+w("")
+w("Every HDF5 file of eight public corpora (pinned by commit) is read twice — by")
+w("clawhdf5 (`conformance/probe`, the same `clawhdf5-format` calls the facade")
+w("makes) and by h5py/libhdf5 (`conformance/ref.py`) — and the two readings are")
+w("compared object by object: the set of hard-linked objects, each dataset's and")
+w("attribute's shape, and a SHA-256 of its values in a canonical encoding. The")
+w("CVE corpus is also run through `h5dump`. Each side runs under a timeout and an")
+w("address-space limit, so a hang, crash or runaway allocation is recorded, not")
+w("fatal. This file is generated by `conformance/run.sh`; do not edit it by hand.")
+w("")
+w("## Run")
+w("")
+w("| | |")
+w("|---|---|")
+w(f"| date | {meta['date']} |")
+w(f"| clawhdf5 commit | `{meta['commit']}` |")
+w(f"| machine | `{platform.node()}`: {cpu_model()}, {os.cpu_count()} CPUs, {mem_gib()}, {platform.system()} {platform.release()} {platform.machine()} |")
+w(f"| command | `{os.environ.get('CONFORMANCE_CMD', 'conformance/run.sh')}` |")
+w(f"| rustc | {sh('rustc', '-V')} |")
+w(f"| reference | h5py {h5py.__version__}, HDF5 {h5py.version.hdf5_version}, numpy {numpy.__version__}, hdf5plugin {HDF5PLUGIN}, Python {platform.python_version()} |")
+w(f"| h5dump | {h5dump_v} (CVE corpus only) |")
+if meta_run:
+    w(f"| limits | {meta_run.get('timeout_s')} s timeout (SIGKILL), {int(meta_run.get('mem_kb', 0)) // 1024} MiB address space, per process; {meta_run.get('jobs')} files in parallel |")
+    w(f"| runtime | {meta_run.get('probe_seconds')} s probing + comparing ({meta_run.get('build_seconds')} s fetch/build before it) |")
+w("")
+w("## Results")
+w("")
+w("A file's class is the first that applies:")
+w("")
+w("- **panic / hang / crash / oom** — clawhdf5 panicked (caught per object or not), hit the timeout, died on a signal, or failed an allocation. The CI gate fails on any of these.")
+w("- **h5py-cannot-read** — libhdf5 could not open the file (or itself crashed or hung). Nothing to compare against; most are the deliberately malformed CVE reproducers.")
+w("- **our-error** — clawhdf5 returned an error for something h5py reads.")
+w("- **mismatch** — both read it, but the shapes, values, object set or attribute set differ.")
+w("- **ok** — every object h5py reads, clawhdf5 reads identically.")
+w("")
+w("| corpus | files | " + " | ".join(CLASSES) + " |")
+w("|---" * (len(CLASSES) + 2) + "|")
+for c in sorted(by_corpus):
+    cnt = by_corpus[c]
+    w(f"| {c} | {sum(cnt.values())} | " + " | ".join(str(cnt.get(k, 0)) for k in CLASSES) + " |")
+w(f"| **all** | **{len(rows)}** | " + " | ".join(f"**{total.get(k, 0)}**" for k in CLASSES) + " |")
+w("")
+n_known = sum(len(v) for v in known.values())
+if n_known:
+    w(f"{n_known} of the {total.get('mismatch', 0)} mismatches are a known h5py bug, not ours (see *Known not-our-bug*).")
+    w("")
+w("Corpora (fetched by `conformance/fetch-corpus.sh` into the gitignored `conformance/.cache/`):")
+w("")
+w("| corpus | source | commit |")
+w("|---|---|---|")
+for name, url, rev, root in pins:
+    w(f"| {name} | {url.removesuffix('.git')}" + ("" if root == "." else f" (`{root}`)") + f" | `{rev[:12]}` |")
+w("")
+
+w("## Panics, hangs, crashes, out-of-memory")
+w("")
+if not res["panics"]:
+    w("None.")
+else:
+    for p in res["panics"]:
+        w(f"- `{p['file']}` [{p['class']}] {p['detail']}")
+w("")
+
+w("## Our-error root causes")
+w("")
+w("Grouped by normalised error message. *files* counts files whose class this cause affects.")
+w("")
+w("| files | objects | error | examples |")
+w("|---:|---:|---|---|")
+for k, v in res["root_causes"].items():
+    w(f"| {v['files']} | {v['count']} | `{k.replace('|', '/')}` | {ex_list(v['file_list'])} |")
+w("")
+w("## Mismatch root causes")
+w("")
+w("| files | objects | cause | examples |")
+w("|---:|---:|---|---|")
+for k, v in res["mismatch_causes"].items():
+    w(f"| {v['files']} | {v['count']} | `{k.replace('|', '/')}` | {ex_list(v['file_list'])} |")
+w("")
+
+w("## CVE corpus: clawhdf5 vs h5dump vs h5py")
+w("")
+w(f"The {len(cve_rows)} files of [HDFGroup/cve_hdf5](https://github.com/HDFGroup/cve_hdf5) — reproducers for")
+w("published libhdf5 CVEs and fuzzer finds. *read* = produced output (possibly with per-object")
+w("errors), *error* = refused cleanly. h5dump exits non-zero on any error anywhere in a file, so")
+w("its read/error split is not comparable with the other two rows; the panic, crash, hang and oom")
+w("columns are.")
+w("")
+w("| tool | read | error | panic | crash | hang | oom |")
+w("|---|---:|---:|---:|---:|---:|---:|")
+for tool, label in (("clawhdf5", "clawhdf5"), ("h5dump", f"h5dump {h5dump_v.split()[-1] if h5dump_v else ''}"),
+                    ("h5py", f"h5py {h5py.__version__} / HDF5 {h5py.version.hdf5_version}")):
+    b = buckets[tool]
+    w(f"| {label} | " + " | ".join(str(b.get(k, 0)) for k in ("read", "error", "panic", "crash", "hang", "oom")) + " |")
+w("")
+w("<details><summary>Per-file outcomes</summary>")
+w("")
+w("| file | h5dump | h5py | clawhdf5 | class |")
+w("|---|---|---|---|---|")
+for f, d, p, o, cls in cve_rows:
+    w(f"| {f} | {d} | {p} | {o} | {cls} |")
+w("")
+w("</details>")
+w("")
+
+w("## Known not-our-bug")
+w("")
+w("- **h5py big-endian variable-length sequences.** h5py returns the elements of a VL sequence")
+w("  whose base type is big-endian with the file's big-endian bytes but a native (little-endian)")
+w("  numpy dtype, so the values it reports are byte-swapped garbage; `h5dump` prints the values")
+w("  clawhdf5 reads. Reproducer: `h5py.vlen_dtype(np.dtype('>f4'))` dataset holding `[1.0, 2.0]`")
+w("  reads back in h5py as `[4.6e-41, 9.0e-44]`. Affected here: "
+  + (ex_list(sorted(known["h5py-be-vlen"]), 10) if known["h5py-be-vlen"] else "none") + ".")
+w("- **Non-IEEE floats and partial-precision integers (N-Bit).** libhdf5 converts a float whose")
+w("  bit layout is not IEEE (e.g. `H5Tset_precision` for the N-Bit filter) or an integer with a")
+w("  bit offset / reduced precision into the plain numpy type of the same size. The probe")
+w("  compares such values as converted numbers, not raw file bytes (before 2026-09-25 it compared")
+w("  raw bytes, which reported every N-Bit float dataset as a mismatch).")
+if res["incomparable"]:
+    w("- **Types h5py widens.** Where h5py reads a type into a numpy type of a different size")
+    w("  (FP8 -> float16, bfloat16 -> float32, x87 long double -> float128) the values are not")
+    w("  compared (shape and presence still are): "
+      + ", ".join(f"{k} ({n}x)" for k, n in res["incomparable"]) + ".")
+w("- **References** are compared by presence only (`R`), not by target.")
+w("")
+if res.get("ref_only_errors"):
+    w("## Objects h5py fails on but clawhdf5 reads")
+    w("")
+    for k, n in res["ref_only_errors"][:15]:
+        w(f"- {n} x `{k}`")
+    w("")
+w("## Reproduce")
+w("")
+w("```sh")
+w("# needs: Rust, python3 with h5py numpy hdf5plugin (conformance/requirements.txt), h5dump (hdf5-tools), git")
+w("CLAWHDF5_PYTHON=/path/to/venv/bin/python conformance/run.sh")
+w("```")
+w("")
+w("The corpus (about 450 MB of sparse checkouts) is cached in `conformance/.cache/`; results for")
+w("every file, both sides' raw JSON and stderr, are in `conformance/.cache/results/`.")
+w("`conformance/baseline.json` holds the ok files the nightly CI job (`.gitea/workflows/conformance.yml`)")
+w("must keep; `conformance/run.sh --update-baseline` rewrites it.")
+
+with open(OUT_MD, "w") as fh:
+    fh.write("\n".join(L) + "\n")

conformance/requirements.txt

@@ -0,0 +1,6 @@
+# The reference side of the conformance sweep. Pinned so the nightly job and a
+# local run compare against the same libhdf5 (h5py wheels bundle it).
+h5py==3.16.0
+numpy==2.5.3
+hdf5plugin==7.1.0
+netCDF4==1.7.4

conformance/run.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+# conformance/run.sh — the clawhdf5 conformance sweep, end to end.
+#
+#   fetch the pinned corpora (cached) -> build the probe -> probe every file
+#   with clawhdf5 and with h5py (and h5dump for the CVE corpus), each under a
+#   timeout and a memory limit -> compare -> write CONFORMANCE.md -> check the
+#   result against conformance/baseline.json.
+#
+# Usage: conformance/run.sh [--no-fetch] [--no-report] [--update-baseline]
+#
+# Environment:
+#   CLAWHDF5_PYTHON    python with h5py, numpy, hdf5plugin (default: repo .venv, then python3)
+#   CONFORMANCE_CACHE  corpus / build / results cache (default: conformance/.cache)
+#   CONFORMANCE_OUT    results directory (default: $CONFORMANCE_CACHE/results)
+#   CONFORMANCE_REPORT report path (default: CONFORMANCE.md at the repo root)
+#   JOBS               parallel files (default: nproc)
+#   CONFORMANCE_PROBE  use this prebuilt probe binary instead of building one
+#   TMO / MEM_KB       per-process timeout in seconds (20) / address-space limit in KiB (4 GiB)
+#
+# Exit status: 0 = gate passed; 1 = a panic/hang/crash/oom in clawhdf5, or the
+# ok count fell below the baseline, or a baseline-ok file regressed; 2 = setup error.
+set -euo pipefail
+HERE="$(cd "$(dirname "$0")" && pwd)"
+ROOT="$(cd "$HERE/.." && pwd)"
+FETCH=1 REPORT=1 UPDATE=0
+for a in "$@"; do
+  case "$a" in
+    --no-fetch) FETCH=0 ;;
+    --no-report) REPORT=0 ;;
+    --update-baseline) UPDATE=1 ;;
+    -h|--help) sed -n '2,23p' "$0"; exit 0 ;;
+    *) echo "unknown argument: $a" >&2; exit 2 ;;
+  esac
+done
+
+export PATH="$HOME/.cargo/bin:$PATH"
+CACHE="${CONFORMANCE_CACHE:-$HERE/.cache}"
+mkdir -p "$CACHE"; CACHE="$(cd "$CACHE" && pwd)"
+OUT="${CONFORMANCE_OUT:-$CACHE/results}"
+REPORT_PATH="${CONFORMANCE_REPORT:-$ROOT/CONFORMANCE.md}"
+JOBS="${JOBS:-$(nproc 2>/dev/null || echo 4)}"
+if [ -n "${CLAWHDF5_PYTHON:-}" ]; then PY="$CLAWHDF5_PYTHON"
+elif [ -x "$ROOT/.venv/bin/python" ]; then PY="$ROOT/.venv/bin/python"
+else PY="$(command -v python3)"; fi
+export PY TMO="${TMO:-20}" MEM_KB="${MEM_KB:-4194304}"
+command -v h5dump >/dev/null || { echo "error: h5dump not found (install hdf5-tools)" >&2; exit 2; }
+"$PY" -c 'import h5py, numpy, hdf5plugin' || { echo "error: $PY lacks h5py/numpy/hdf5plugin" >&2; exit 2; }
+
+t0=$(date +%s)
+[ "$FETCH" = 1 ] && bash "$HERE/fetch-corpus.sh" "$CACHE"
+C="$CACHE/corpus"
+[ -d "$C" ] || { echo "error: no corpus in $C (run without --no-fetch)" >&2; exit 2; }
+
+if [ -n "${CONFORMANCE_PROBE:-}" ]; then
+  export PROBE="$CONFORMANCE_PROBE"   # a prebuilt probe, e.g. an older one for a before/after
+else
+  echo "== building the probe"
+  CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-$CACHE/target}" \
+    cargo build -q --release --manifest-path "$HERE/probe/Cargo.toml"
+  export PROBE="${CARGO_TARGET_DIR:-$CACHE/target}/release/conformance-probe"
+fi
+t1=$(date +%s)
+
+rm -rf "$OUT"; mkdir -p "$OUT"
+"$PY" "$HERE/list_files.py" "$C" > "$OUT/files.txt"
+echo "== probing $(wc -l <"$OUT/files.txt") files, $JOBS at a time (timeout ${TMO}s, limit $((MEM_KB / 1024)) MiB)"
+export C OUT HERE
+# The shell's "Segmentation fault (core dumped)" notices go to probe.log; the
+# signals themselves are recorded in each side's .rc.
+xargs -a "$OUT/files.txt" -d '\n' -P "$JOBS" -I{} bash -c '
+  f="$1"; d="$OUT/runs/${f//\//__}"
+  case "$f" in cve_hdf5/*) export WITH_H5DUMP=1 ;; esac
+  "$HERE/run_one.sh" "$C/$f" "$d"' _ {} 2>"$OUT/probe.log"
+echo "== comparing"
+"$PY" "$HERE/compare.py" "$OUT" >/dev/null
+t2=$(date +%s)
+cat > "$OUT/meta.json" <<EOF
+{"build_seconds": $((t1 - t0)), "probe_seconds": $((t2 - t1)), "jobs": $JOBS, "timeout_s": $TMO, "mem_kb": $MEM_KB}
+EOF
+export CONFORMANCE_CMD="${CONFORMANCE_CMD:-conformance/run.sh${*:+ $*}}"
+if [ "$REPORT" = 1 ]; then
+  "$PY" "$HERE/report.py" "$OUT" "$REPORT_PATH" "$C"
+  echo "== wrote $REPORT_PATH"
+fi
+if [ "$UPDATE" = 1 ]; then
+  "$PY" "$HERE/check.py" "$OUT" "$HERE/baseline.json" --update
+fi
+"$PY" "$HERE/check.py" "$OUT" "$HERE/baseline.json"

conformance/run_one.sh

@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+# run_one.sh <file> <outdir>
+#
+# Probe one file with clawhdf5 (PROBE) and with h5py (PY ref.py), and with
+# h5dump too when WITH_H5DUMP is set. Each side runs under a timeout (TMO
+# seconds, SIGKILL) and an address-space limit (MEM_KB), with core dumps off.
+# Writes <outdir>/<side>.{json,err,rc}; rc 137 = killed by the timeout.
+set -u
+f="$1"; out="$2"; mkdir -p "$out"
+HERE="$(cd "$(dirname "$0")" && pwd)"
+: "${PROBE:?PROBE must name the conformance-probe binary}"
+: "${PY:?PY must name a python with h5py}"
+TMO="${TMO:-20}"
+MEM_KB="${MEM_KB:-4194304}"
+run() { # name cmd...
+  local name=$1; shift
+  ( ulimit -v "$MEM_KB"; ulimit -c 0; RUST_BACKTRACE=1 exec timeout -s KILL "$TMO" "$@" ) \
+      >"$out/$name.json" 2>"$out/$name.err"
+  echo $? >"$out/$name.rc"
+}
+run ours "$PROBE" "$f"
+run ref "$PY" "$HERE/ref.py" "$f"
+if [ -n "${WITH_H5DUMP:-}" ]; then
+  run h5dump h5dump "$f"
+  : >"$out/h5dump.json" # h5dump's text dump is not compared, only its exit status
+fi
+exit 0

crates/clawhdf5-agent/src/schema.rs

@@ -477,10 +477,34 @@ fn write_string_dataset(
     }
 }
 
+/// `/meta`'s attributes, failing if any of them cannot be read.
+///
+/// `Group::attrs` leaves out an attribute it cannot decode. For the store's
+/// settings that would silently fall back to defaults (e.g. `float16`, the
+/// WAL mark), so an unreadable attribute is an error here, as it was before
+/// `attrs` became tolerant.
+fn meta_attrs(
+    file: &clawhdf5::File,
+) -> Result<std::collections::HashMap<String, AttrValue>, MemoryError> {
+    let meta = file
+        .group("meta")
+        .map_err(|e| MemoryError::Schema(format!("missing /meta group: {e}")))?;
+    let (attrs, errors) = meta
+        .attrs_with_errors()
+        .map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
+    if let Some(e) = errors.first() {
+        return Err(MemoryError::Schema(format!(
+            "cannot read /meta attrs: {} unreadable, first: {e}",
+            errors.len()
+        )));
+    }
+    Ok(attrs)
+}
+
 /// Validate an HDF5 file has the correct schema and load all data.
 /// Read the checkpoint's [`WalMark`] from `/meta`, if it has one.
 pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
-    let attrs = file.group("meta").ok()?.attrs().ok()?;
+    let attrs = meta_attrs(file).ok()?;
     let len = match attrs.get(WAL_APPLIED_LEN_ATTR)? {
         AttrValue::I64(v) => u64::try_from(*v).ok()?,
         _ => return None,
@@ -498,10 +522,7 @@ pub fn read_signature(
     file: &clawhdf5::File,
 ) -> Result<Option<crate::signing::StoredSignature>, MemoryError> {
     use crate::signing::{Manifest, StoredSignature, from_hex};
-    let attrs = file
-        .group("meta")
-        .and_then(|g| g.attrs())
-        .map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
+    let attrs = meta_attrs(file)?;
     let version = match attrs.get(SIG_VERSION_ATTR) {
         None => return Ok(None),
         Some(AttrValue::I64(v)) => *v,
@@ -552,18 +573,14 @@ pub fn read_signature(
 
 /// Read the checkpoint bookkeeping from `/meta`.
 pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
-    let ann_generation = file
-        .group("meta")
-        .ok()
-        .and_then(|g| g.attrs().ok())
-        .and_then(|attrs| match attrs.get(ANN_GENERATION_ATTR) {
-            Some(AttrValue::I64(v)) => Some(*v as u64),
-            _ => None,
-        });
-    let signed = file
-        .group("meta")
-        .and_then(|g| g.attrs())
-        .is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
+    let ann_generation =
+        meta_attrs(file)
+            .ok()
+            .and_then(|attrs| match attrs.get(ANN_GENERATION_ATTR) {
+                Some(AttrValue::I64(v)) => Some(*v as u64),
+                _ => None,
+            });
+    let signed = meta_attrs(file).is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
     CheckpointMeta {
         wal_applied: read_wal_mark(file),
         ann_generation,
@@ -575,12 +592,7 @@ pub fn validate_and_load(
     file: &clawhdf5::File,
 ) -> Result<(MemoryConfig, MemoryCache, SessionCache, KnowledgeCache), MemoryError> {
     // Read /meta group attributes
-    let meta = file
-        .group("meta")
-        .map_err(|e| MemoryError::Schema(format!("missing /meta group: {e}")))?;
-    let attrs = meta
-        .attrs()
-        .map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
+    let attrs = meta_attrs(file)?;
 
     let schema_version = match attrs.get("schema_version") {
         Some(AttrValue::String(s)) => s.clone(),

crates/clawhdf5-agent/tests/float16_store.rs

@@ -258,3 +258,43 @@ fn an_existing_f32_store_stays_f32() {
     assert_eq!(&values[..before.1.len()], before.1.as_slice());
     assert_eq!(&values[before.1.len()..], odd.as_slice());
 }
+
+/// `Group::attrs` leaves out an attribute it cannot decode. A store whose
+/// `float16` setting is unreadable must not open as `float16 = false` (or with
+/// any other default in place of a setting it has): it is an error.
+#[test]
+fn unreadable_meta_attribute_fails_open_instead_of_defaulting() {
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("store.h5");
+    {
+        let mut m = HDF5Memory::create(config(&dir, "store.h5", true)).unwrap();
+        m.save(entry(1)).unwrap();
+        m.flush_wal().unwrap();
+    }
+    assert!(HDF5Memory::open_read_only(&path).is_ok());
+
+    // Give the `float16` attribute message an unknown version (the name is
+    // at +8 in a version-1 message and +9 in a version-3 one).
+    let mut bytes = std::fs::read(&path).unwrap();
+    let name = b"float16\0";
+    let mut hit = false;
+    let positions: Vec<usize> = (9..bytes.len() - name.len())
+        .filter(|&p| &bytes[p..p + name.len()] == name)
+        .collect();
+    for pos in positions {
+        for (back, version) in [(8, 1u8), (9, 3u8)] {
+            if bytes[pos - back] == version {
+                bytes[pos - back] = 0x7f;
+                hit = true;
+            }
+        }
+    }
+    assert!(hit, "float16 attribute message not found");
+    std::fs::write(&path, &bytes).unwrap();
+
+    match HDF5Memory::open_read_only(&path) {
+        Err(MemoryError::Schema(msg)) => assert!(msg.contains("/meta"), "{msg}"),
+        Err(e) => panic!("unexpected error: {e}"),
+        Ok(_) => panic!("store opened with an unreadable float16 setting"),
+    }
+}

crates/clawhdf5-ann/src/hnsw.rs

@@ -13,7 +13,7 @@ use clawhdf5_format::filter_pipeline::FilterPipeline;
 use clawhdf5_format::group_v2::resolve_path_any;
 use clawhdf5_format::message_type::MessageType;
 use clawhdf5_format::object_header::ObjectHeader;
-use clawhdf5_format::signature::find_signature;
+use clawhdf5_format::signature::split_user_block;
 use clawhdf5_format::superblock::Superblock;
 use clawhdf5_io::FileWriter as IoFileWriter;
 
@@ -861,8 +861,9 @@ impl HnswIndex {
     /// The HDF5 data must contain the `/ann/vectors`, `/ann/graph_layer_*`,
     /// and `/ann/config` datasets as produced by [`to_hdf5_bytes`].
     pub fn load_from_hdf5(data: &[u8]) -> Result<Self, FormatError> {
-        let sig_offset = find_signature(data)?;
-        let sb = Superblock::parse(data, sig_offset)?;
+        // Addresses are relative to the superblock: skip any user block.
+        let (_, data) = split_user_block(data)?;
+        let sb = Superblock::parse(data, 0)?;
 
         // Read config dataset and its attributes
         let config_attrs = read_dataset_attrs(data, &sb, "ann/config")?;

crates/clawhdf5-format/src/attribute.rs

@@ -97,7 +97,7 @@ impl AttributeMessage {
             return Ok(Cow::Borrowed(bytes));
         }
         let (file_data, offset_size) = file.ok_or(FormatError::UnresolvedSharedMessage)?;
-        let shared_ref = shared_message::parse_shared_ref(bytes, offset_size)?;
+        let shared_ref = shared_message::parse_shared_ref_sized(bytes, offset_size, length_size)?;
         shared_message::resolve_shared_message(
             file_data,
             &shared_ref,
@@ -394,42 +394,80 @@ pub fn find_attribute<'a>(
 ///
 /// Use this instead of `extract_attributes` when reading files that may use dense storage
 /// (e.g., objects with many attributes, typically >8).
+///
+/// Fails if any attribute cannot be read; see [`extract_attributes_tolerant`]
+/// to read the others.
 pub fn extract_attributes_full(
     file_data: &[u8],
     header: &ObjectHeader,
     offset_size: u8,
     length_size: u8,
+) -> Result<Vec<AttributeMessage>, FormatError> {
+    extract_attributes_with(file_data, header, offset_size, length_size, &mut Err)
+}
+
+/// Like [`extract_attributes_full`], but an attribute that cannot be read
+/// (a corrupt or unsupported attribute message, or a heap object that cannot
+/// be located) is left out and its error returned alongside the attributes
+/// that could be read, instead of failing them all.
+///
+/// Errors in the structures that index the attributes (the Attribute Info
+/// message, the dense-storage heap header or B-tree) still fail the call:
+/// then it is unknown which attributes exist at all.
+pub fn extract_attributes_tolerant(
+    file_data: &[u8],
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<(Vec<AttributeMessage>, Vec<FormatError>), FormatError> {
+    let mut errors = Vec::new();
+    let attrs = extract_attributes_with(file_data, header, offset_size, length_size, &mut |e| {
+        errors.push(e);
+        Ok(())
+    })?;
+    Ok((attrs, errors))
+}
+
+/// Read every attribute; each one that fails goes to `on_error`, which
+/// either stops the read (returns the error) or skips that attribute.
+fn extract_attributes_with(
+    file_data: &[u8],
+    header: &ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
+    on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
 ) -> Result<Vec<AttributeMessage>, FormatError> {
     let mut attrs = Vec::new();
 
     // Collect compact attributes (inline in OH)
     for msg in &header.messages {
         if msg.msg_type == MessageType::Attribute {
-            if shared_message::is_shared(msg.flags) {
+            let attr = if shared_message::is_shared(msg.flags) {
                 // Shared attribute: resolve the reference to get actual attribute data
-                let shared_ref = shared_message::parse_shared_ref(&msg.data, offset_size)?;
-                let resolved_data = shared_message::resolve_shared_message(
-                    file_data,
-                    &shared_ref,
-                    MessageType::Attribute,
-                    offset_size,
-                    length_size,
-                )?;
-                let attr = AttributeMessage::parse_in_file(
-                    &resolved_data,
-                    file_data,
-                    offset_size,
-                    length_size,
-                )?;
-                attrs.push(attr);
+                shared_message::parse_shared_ref_sized(&msg.data, offset_size, length_size)
+                    .and_then(|shared_ref| {
+                        shared_message::resolve_shared_message(
+                            file_data,
+                            &shared_ref,
+                            MessageType::Attribute,
+                            offset_size,
+                            length_size,
+                        )
+                    })
+                    .and_then(|resolved| {
+                        AttributeMessage::parse_in_file(
+                            &resolved,
+                            file_data,
+                            offset_size,
+                            length_size,
+                        )
+                    })
             } else {
-                let attr = AttributeMessage::parse_in_file(
-                    &msg.data,
-                    file_data,
-                    offset_size,
-                    length_size,
-                )?;
-                attrs.push(attr);
+                AttributeMessage::parse_in_file(&msg.data, file_data, offset_size, length_size)
+            };
+            match attr {
+                Ok(attr) => attrs.push(attr),
+                Err(e) => on_error(e)?,
             }
         }
     }
@@ -439,9 +477,15 @@ pub fn extract_attributes_full(
     if let Some(info) = attr_info
         && let Some(fh_addr) = info.fractal_heap_address
     {
-        let dense_attrs =
-            extract_dense_attributes(file_data, &info, fh_addr, offset_size, length_size)?;
-        attrs.extend(dense_attrs);
+        extract_dense_attributes(
+            file_data,
+            &info,
+            fh_addr,
+            offset_size,
+            length_size,
+            &mut attrs,
+            on_error,
+        )?;
     }
 
     Ok(attrs)
@@ -468,7 +512,9 @@ fn extract_dense_attributes(
     fh_addr: u64,
     offset_size: u8,
     length_size: u8,
-) -> Result<Vec<AttributeMessage>, FormatError> {
+    attrs: &mut Vec<AttributeMessage>,
+    on_error: &mut dyn FnMut(FormatError) -> Result<(), FormatError>,
+) -> Result<(), FormatError> {
     // Parse fractal heap
     let fh = FractalHeapHeader::parse(file_data, fh_addr as usize, offset_size, length_size)?;
 
@@ -482,28 +528,32 @@ fn extract_dense_attributes(
     let btree_hdr = BTreeV2Header::parse(file_data, btree_addr as usize, offset_size, length_size)?;
     let records = collect_btree_v2_records(file_data, &btree_hdr, offset_size, length_size)?;
 
-    let mut attrs = Vec::new();
     for record in &records {
         // Per HDF5 spec, both type 8 and type 9 records start with heap_id:
         //   Type 8: heap_id(8) + msg_flags(1) + creation_order(4) + hash(4)
         //   Type 9: heap_id(8) + msg_flags(1) + creation_order(4)
-        let id_offset = 0;
-
-        if record.data.len() < id_offset + fh.heap_id_length as usize {
+        let id_len = fh.heap_id_length as usize;
+        let Some(id_bytes) = record.data.get(..id_len) else {
+            on_error(FormatError::UnexpectedEof {
+                expected: id_len,
+                available: record.data.len(),
+            })?;
             continue;
-        }
-        let id_bytes = &record.data[id_offset..id_offset + fh.heap_id_length as usize];
-
-        // Read attribute message from fractal heap
-        let attr_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
+        };
 
         // The data in the heap is a complete attribute message
-        let attr =
-            AttributeMessage::parse_in_file(&attr_data, file_data, offset_size, length_size)?;
-        attrs.push(attr);
+        let attr = fh
+            .read_managed_object(file_data, id_bytes, offset_size)
+            .and_then(|attr_data| {
+                AttributeMessage::parse_in_file(&attr_data, file_data, offset_size, length_size)
+            });
+        match attr {
+            Ok(attr) => attrs.push(attr),
+            Err(e) => on_error(e)?,
+        }
     }
 
-    Ok(attrs)
+    Ok(())
 }
 
 #[cfg(test)]

crates/clawhdf5-format/src/btree_v2.rs

@@ -323,39 +323,21 @@ fn collect_internal_records(
     let records_start = pos;
     pos += records_total;
 
-    // Compute sizes for child pointers
-    // max_records at child depth - for variable-width nrec encoding
+    // Child pointer layout, as libhdf5 computes it (H5B2__hdr_init): the
+    // child's record count is always encoded in the width needed for a
+    // *leaf's* maximum, and — below the first internal level — the child
+    // subtree's total record count in the width needed for the most records
+    // a subtree of that depth can hold.
     let child_depth = depth - 1;
-    let max_nrec_child = if child_depth == 0 {
-        max_leaf_nrec
-    } else {
-        // For internal nodes at child_depth, the true max_nrec depends on the
-        // node size, record size, and the recursive width of child pointer
-        // entries (which themselves depend on max_nrec at deeper levels).
-        // Computing the exact value requires iterating from the leaf level
-        // upward, as described in the HDF5 spec (III.A.2 "Computing the Size
-        // of B-tree Nodes").
-        //
-        // We use `max_leaf_nrec * 2` as a conservative upper bound. This
-        // over-estimates the nrec encoding width, which means we may read
-        // slightly more bytes per child pointer than strictly necessary, but
-        // never fewer. The over-read bytes are harmless because we only
-        // decode `num_records` entries (the actual count from the node header).
-        //
-        // Known limitation: for very deep trees (depth > 3) with small record
-        // sizes, the true max could exceed this estimate, causing us to
-        // under-allocate the nrec encoding width and misparse child pointers.
-        // In practice, HDF5 B-tree v2 depths rarely exceed 2-3.
-        max_leaf_nrec * 2
-    };
-    let nrec_width = bytes_for_max_records(max_nrec_child);
-
-    // Total records in subtree width (only if depth > 1)
+    let nrec_width = bytes_for_max_records(max_leaf_nrec);
     let total_nrec_width = if depth > 1 {
-        // Width to hold total records in a subtree
-        // We compute max possible total records at this subtree depth
-        let max_total = header_max_total_records(max_leaf_nrec, depth - 1);
-        bytes_for_max_records(max_total)
+        bytes_for_max_records(cum_max_records(
+            node_size,
+            record_size,
+            offset_size,
+            max_leaf_nrec,
+            child_depth,
+        ))
     } else {
         0
     };
@@ -435,14 +417,36 @@ fn collect_internal_records(
     Ok(())
 }
 
-/// Estimate maximum total records at a given depth (for variable-width encoding).
-fn header_max_total_records(max_leaf_nrec: u64, depth: u16) -> u64 {
-    // Conservative: branching factor * max_leaf at each level
-    let mut total = max_leaf_nrec;
-    for _ in 0..depth {
-        total = total.saturating_mul(max_leaf_nrec.max(2));
+/// Most records a subtree whose root is at `depth` can hold (libhdf5's
+/// `cum_max_nrec`): a leaf holds `max_leaf_nrec`; an internal node at depth
+/// `d` holds `max_nrec(d)` records and `max_nrec(d) + 1` subtrees of depth
+/// `d - 1`, where `max_nrec(d)` is what fits in a node once each record is
+/// paired with a child pointer of the width depth `d` needs.
+fn cum_max_records(
+    node_size: u32,
+    record_size: u16,
+    offset_size: u8,
+    max_leaf_nrec: u64,
+    depth: u16,
+) -> u64 {
+    // Internal node overhead: signature(4) + version(1) + type(1) + checksum(4).
+    const PREFIX: u64 = 10;
+    let nrec_width = bytes_for_max_records(max_leaf_nrec) as u64;
+    let mut cum = max_leaf_nrec;
+    let mut cum_width = 0u64;
+    for d in 1..=depth {
+        let ptr = u64::from(offset_size) + nrec_width + if d > 1 { cum_width } else { 0 };
+        let max_nrec = u64::from(node_size)
+            .saturating_sub(PREFIX)
+            .saturating_sub(ptr)
+            / (u64::from(record_size) + ptr).max(1);
+        cum = max_nrec
+            .saturating_add(1)
+            .saturating_mul(cum)
+            .saturating_add(max_nrec);
+        cum_width = bytes_for_max_records(cum) as u64;
     }
-    total
+    cum
 }
 
 #[cfg(test)]
@@ -512,9 +516,15 @@ mod tests {
         child_nrec: u64,
     ) -> Vec<u8> {
         let max_leaf = max_records_leaf(node_size, record_size);
-        let nrec_width = bytes_for_max_records(if depth == 1 { max_leaf } else { max_leaf * 2 });
+        let nrec_width = bytes_for_max_records(max_leaf);
         let total_width = if depth > 1 {
-            bytes_for_max_records(header_max_total_records(max_leaf, depth - 1))
+            bytes_for_max_records(cum_max_records(
+                node_size,
+                record_size,
+                8,
+                max_leaf,
+                depth - 1,
+            ))
         } else {
             0
         };
@@ -673,4 +683,18 @@ mod tests {
         let records = collect_btree_v2_records(&header, &hdr, 8, 8).unwrap();
         assert!(records.is_empty());
     }
+
+    #[test]
+    fn subtree_capacity_matches_libhdf5() {
+        // A link-name index (11-byte records, 512-byte nodes, 8-byte
+        // addresses): libhdf5's H5B2__hdr_init gives 45 records per leaf,
+        // then cum_max_nrec 1 149 at depth 1 and 26 449 at depth 2 — two
+        // bytes of subtree count in a depth-3 root's child pointers, where
+        // leaf_max^3 = 91 125 would need three.
+        let leaf = max_records_leaf(512, 11);
+        assert_eq!(leaf, 45);
+        assert_eq!(cum_max_records(512, 11, 8, leaf, 0), 45);
+        assert_eq!(cum_max_records(512, 11, 8, leaf, 1), 1_149);
+        assert_eq!(cum_max_records(512, 11, 8, leaf, 2), 26_449);
+    }
 }

crates/clawhdf5-format/src/data_layout.rs

@@ -1,7 +1,7 @@
 //! HDF5 Data Layout message parsing (message type 0x0008).
 
 #[cfg(not(feature = "std"))]
-use alloc::{string::String, vec::Vec};
+use alloc::{format, string::String, vec::Vec};
 
 #[cfg(feature = "std")]
 use std::string::String;
@@ -45,7 +45,9 @@ pub enum DataLayout {
         chunk_dimensions: Vec<u32>,
         /// B-tree address, or `None` if undefined.
         btree_address: Option<u64>,
-        /// Layout version (3 or 4).
+        /// Layout version (3 or 4). Version 1/2 messages (HDF5 1.4/1.6-era)
+        /// use the same version-1 B-tree chunk index as version 3 and are
+        /// reported as 3.
         version: u8,
         /// Chunk index type (v4 only).
         chunk_index_type: Option<u8>,
@@ -72,21 +74,33 @@ pub enum DataLayout {
     },
 }
 
+/// Version-1 VDS mapping flag: the source file name is stored by an earlier
+/// entry, whose index follows in place of the name.
+const VDS_SOURCE_FILE_SHARED: u8 = 0x01;
+/// Version-1 VDS mapping flag: likewise for the source dataset name.
+const VDS_SOURCE_DSET_SHARED: u8 = 0x02;
+/// Version-1 VDS mapping flag: the source is in the virtual file itself
+/// (`"."`); no file name is stored.
+const VDS_SOURCE_SAME_FILE: u8 = 0x04;
+const VDS_ALL_FLAGS: u8 = VDS_SOURCE_FILE_SHARED | VDS_SOURCE_DSET_SHARED | VDS_SOURCE_SAME_FILE;
+
 /// Parse VDS mappings from global-heap object data.
 ///
 /// The global-heap block holding a VDS mapping list is laid out as
-/// (reverse-engineered and validated against HDF5 2.0):
+/// (`H5D__virtual_store_layout` / `H5D__virtual_load_layout` in libhdf5):
 ///
 /// ```text
 /// version(1) · nused(length_size, LE) · entry[nused] · checksum(4)
 /// ```
 ///
 /// Each entry is:
-/// - source file name — a null-terminated string in **block version 0**; in
-///   **block version 1** a same-file reference is encoded as a single `0x04`
-///   marker byte (the source file is the virtual file itself) in place of the
-///   name;
-/// - source dataset name (null-terminated string);
+/// - **block version 1 only:** a flags byte. `0x04`: the source is in the
+///   virtual file itself and no file name is stored; `0x01`/`0x02`: the
+///   source file/dataset name is that of an earlier entry, whose index
+///   (`length_size` bytes) is stored instead of the name. libhdf5 2.0 writes
+///   version 1 when the file's low version bound is 2.0 and it saves space;
+/// - source file name (null-terminated string, unless flagged above);
+/// - source dataset name (null-terminated string, unless flagged above);
 /// - source selection (serialized `H5S` dataspace selection — self-describing
 ///   in length);
 /// - virtual selection (serialized `H5S` dataspace selection).
@@ -112,7 +126,7 @@ pub fn parse_vds_mappings(
     // `nused` is untrusted; don't pre-allocate from it. Each entry consumes at
     // least a few bytes, so the loop is naturally bounded by the heap data and
     // a bogus `nused` simply errors out on the first short read.
-    let mut mappings = Vec::new();
+    let mut mappings: Vec<VdsMapping> = Vec::new();
     // Reads one self-describing selection at `pos`, returning its raw bytes and
     // advancing past it — bounds-checked so a corrupt selection can't overrun.
     let read_selection = |heap_data: &[u8], pos: &mut usize| -> Result<Vec<u8>, FormatError> {
@@ -132,17 +146,57 @@ pub fn parse_vds_mappings(
         Ok(bytes)
     };
 
-    for _ in 0..nused {
-        // Source file name (with the version-1 same-file marker handled).
-        let source_file = if version >= 1 && heap_data.get(pos) == Some(&0x04) {
+    if version > 1 {
+        return Err(FormatError::ChunkedReadError(
+            "unsupported VDS mapping block version".into(),
+        ));
+    }
+    for i in 0..nused {
+        // Version 1 prefixes each entry with a flags byte; a name may then be
+        // omitted (same file) or replaced by the index of an earlier entry
+        // holding the same name (`H5D__virtual_load_layout`).
+        let flags = if version >= 1 {
+            let f = *heap_data.get(pos).ok_or(FormatError::UnexpectedEof {
+                expected: pos + 1,
+                available: heap_data.len(),
+            })?;
             pos += 1;
+            if f & !VDS_ALL_FLAGS != 0 {
+                return Err(FormatError::ChunkedReadError(
+                    "unknown VDS mapping flags".into(),
+                ));
+            }
+            f
+        } else {
+            0
+        };
+        // Index of an earlier entry, for a shared name.
+        let earlier = |pos: &mut usize| -> Result<usize, FormatError> {
+            let idx = read_length(heap_data, *pos, length_size)?;
+            *pos += ls;
+            if idx >= i {
+                return Err(FormatError::ChunkedReadError(
+                    "VDS mapping shares a name with a later entry".into(),
+                ));
+            }
+            Ok(idx as usize)
+        };
+
+        let source_file = if flags & VDS_SOURCE_SAME_FILE != 0 {
             String::from(".")
+        } else if flags & VDS_SOURCE_FILE_SHARED != 0 {
+            let idx = earlier(&mut pos)?;
+            mappings[idx].source_file.clone()
         } else {
             read_null_terminated_string(heap_data, &mut pos)?
         };
 
-        // Source dataset name.
-        let source_dataset = read_null_terminated_string(heap_data, &mut pos)?;
+        let source_dataset = if flags & VDS_SOURCE_DSET_SHARED != 0 {
+            let idx = earlier(&mut pos)?;
+            mappings[idx].source_dataset.clone()
+        } else {
+            read_null_terminated_string(heap_data, &mut pos)?
+        };
 
         // Source selection, then virtual selection (both self-describing length).
         let source_selection = read_selection(heap_data, &mut pos)?;
@@ -261,6 +315,7 @@ impl DataLayout {
         let layout_class = data[1];
 
         match version {
+            1 | 2 => Self::parse_v1_v2(data, offset_size),
             3 => Self::parse_v3(data, layout_class, offset_size, length_size),
             // v5 (emitted by HDF5 1.14+/2.0 with `libver=latest`) uses the same
             // message structure as v4 — only the version number was bumped.
@@ -269,6 +324,87 @@ impl DataLayout {
         }
     }
 
+    /// Layout message versions 1 and 2 (HDF5 before 1.6.3):
+    ///
+    /// ```text
+    /// version(1) · dimensionality(1) · layout class(1) · reserved(5)
+    /// · address(offset_size)      — contiguous and chunked only
+    /// · dimension sizes(4 × dimensionality)
+    /// · compact data size(4) · compact raw data — compact only
+    /// ```
+    ///
+    /// The dimension sizes are the dataset's (contiguous/compact) or the
+    /// chunk's (chunked) extent plus a trailing element-size dimension, as in
+    /// version 3's chunked form. libhdf5 ignores them for contiguous storage
+    /// and sizes the data from the dataspace; the product of the stored
+    /// dimensions is that same size, and a disagreement (a dimension that was
+    /// truncated to 32 bits) is caught by the reader's size check rather than
+    /// returning wrong data.
+    fn parse_v1_v2(data: &[u8], offset_size: u8) -> Result<DataLayout, FormatError> {
+        ensure_len(data, 0, 8)?;
+        let dimensionality = data[1] as usize;
+        let layout_class = data[2];
+        // H5O_LAYOUT_NDIMS: 32 dataspace dimensions + the element-size one.
+        if dimensionality > 33 {
+            return Err(FormatError::Overflow(format!(
+                "data layout dimensionality {dimensionality} exceeds 33"
+            )));
+        }
+        let mut p = 8;
+        let os = offset_size as usize;
+        let address = match layout_class {
+            1 | 2 => {
+                ensure_len(data, p, os)?;
+                let a = if is_undefined(data, p, offset_size) {
+                    None
+                } else {
+                    Some(read_offset(data, p, offset_size)?)
+                };
+                p += os;
+                a
+            }
+            0 => None,
+            _ => return Err(FormatError::InvalidLayoutClass(layout_class)),
+        };
+        ensure_len(data, p, dimensionality * 4)?;
+        let dims: Vec<u32> = data[p..p + dimensionality * 4]
+            .as_chunks::<4>()
+            .0
+            .iter()
+            .map(|c| u32::from_le_bytes(*c))
+            .collect();
+        p += dimensionality * 4;
+        match layout_class {
+            0 => {
+                ensure_len(data, p, 4)?;
+                let size =
+                    u32::from_le_bytes([data[p], data[p + 1], data[p + 2], data[p + 3]]) as usize;
+                ensure_len(data, p + 4, size)?;
+                Ok(DataLayout::Compact {
+                    data: data[p + 4..p + 4 + size].to_vec(),
+                })
+            }
+            1 => {
+                let size = dims
+                    .iter()
+                    .try_fold(1u64, |acc, &d| acc.checked_mul(d as u64))
+                    .ok_or_else(|| {
+                        FormatError::Overflow(format!("contiguous layout size {dims:?}"))
+                    })?;
+                Ok(DataLayout::Contiguous { address, size })
+            }
+            _ => Ok(DataLayout::Chunked {
+                chunk_dimensions: dims,
+                btree_address: address,
+                version: 3,
+                chunk_index_type: None,
+                single_chunk_filtered_size: None,
+                single_chunk_filter_mask: None,
+                dont_filter_partial_edge_chunks: false,
+            }),
+        }
+    }
+
     fn parse_v3(
         data: &[u8],
         layout_class: u8,
@@ -546,6 +682,108 @@ impl DataLayout {
 mod tests {
     use super::*;
 
+    /// Version 1/2 header: version, dimensionality, class, reserved(5).
+    fn v1v2_header(version: u8, ndims: u8, class: u8) -> Vec<u8> {
+        vec![version, ndims, class, 0, 0, 0, 0, 0]
+    }
+
+    #[test]
+    fn v2_compact() {
+        let mut buf = v1v2_header(2, 2, 0);
+        // dims (3 elements of 2 bytes) — no address for compact
+        buf.extend_from_slice(&3u32.to_le_bytes());
+        buf.extend_from_slice(&2u32.to_le_bytes());
+        buf.extend_from_slice(&6u32.to_le_bytes()); // compact size (u32 in v1/v2)
+        buf.extend_from_slice(&[1, 0, 2, 0, 3, 0]);
+        assert_eq!(
+            DataLayout::parse(&buf, 8, 8).unwrap(),
+            DataLayout::Compact {
+                data: vec![1, 0, 2, 0, 3, 0]
+            }
+        );
+    }
+
+    #[test]
+    fn v1_contiguous_size_from_dimensions() {
+        let mut buf = v1v2_header(1, 3, 1);
+        buf.extend_from_slice(&0x800u32.to_le_bytes()); // 4-byte address
+        for d in [10u32, 20, 4] {
+            buf.extend_from_slice(&d.to_le_bytes());
+        }
+        assert_eq!(
+            DataLayout::parse(&buf, 4, 4).unwrap(),
+            DataLayout::Contiguous {
+                address: Some(0x800),
+                size: 800,
+            }
+        );
+    }
+
+    #[test]
+    fn v1_contiguous_undefined_address() {
+        let mut buf = v1v2_header(1, 2, 1);
+        buf.extend_from_slice(&[0xFF; 8]);
+        buf.extend_from_slice(&5u32.to_le_bytes());
+        buf.extend_from_slice(&8u32.to_le_bytes());
+        assert_eq!(
+            DataLayout::parse(&buf, 8, 8).unwrap(),
+            DataLayout::Contiguous {
+                address: None,
+                size: 40,
+            }
+        );
+    }
+
+    #[test]
+    fn v1_chunked_maps_to_btree_v1_index() {
+        let mut buf = v1v2_header(1, 3, 2);
+        buf.extend_from_slice(&0x1234u64.to_le_bytes());
+        for d in [50u32, 50, 4] {
+            buf.extend_from_slice(&d.to_le_bytes());
+        }
+        assert_eq!(
+            DataLayout::parse(&buf, 8, 8).unwrap(),
+            DataLayout::Chunked {
+                chunk_dimensions: vec![50, 50, 4],
+                btree_address: Some(0x1234),
+                version: 3,
+                chunk_index_type: None,
+                single_chunk_filtered_size: None,
+                single_chunk_filter_mask: None,
+                dont_filter_partial_edge_chunks: false,
+            }
+        );
+    }
+
+    #[test]
+    fn v1v2_rejects_bad_class_dimensionality_and_truncation() {
+        assert_eq!(
+            DataLayout::parse(&v1v2_header(1, 1, 3), 8, 8).unwrap_err(),
+            FormatError::InvalidLayoutClass(3)
+        );
+        assert!(matches!(
+            DataLayout::parse(&v1v2_header(2, 34, 1), 8, 8).unwrap_err(),
+            FormatError::Overflow(_)
+        ));
+        // Chunked, dims cut short.
+        let mut buf = v1v2_header(1, 2, 2);
+        buf.extend_from_slice(&0x10u64.to_le_bytes());
+        buf.extend_from_slice(&7u32.to_le_bytes());
+        assert!(matches!(
+            DataLayout::parse(&buf, 8, 8).unwrap_err(),
+            FormatError::UnexpectedEof { .. }
+        ));
+        // Compact, raw data shorter than its declared size.
+        let mut buf = v1v2_header(2, 1, 0);
+        buf.extend_from_slice(&4u32.to_le_bytes());
+        buf.extend_from_slice(&100u32.to_le_bytes());
+        buf.extend_from_slice(&[0; 4]);
+        assert!(matches!(
+            DataLayout::parse(&buf, 8, 8).unwrap_err(),
+            FormatError::UnexpectedEof { .. }
+        ));
+    }
+
     #[test]
     fn v3_compact() {
         let mut buf = vec![3u8, 0]; // version=3, class=0 (compact)
@@ -849,6 +1087,62 @@ mod tests {
         assert_eq!(v1.iter_linear_1d(8).unwrap(), vec![4, 5, 6, 7]);
     }
 
+    #[test]
+    fn parse_vds_mappings_v1_shared_names() {
+        // Written by HDF5 2.0 (h5py, libver=("v200", "v200")) for three
+        // mappings from `a_rather_long_source_file.h5:a_rather_long_dataset_name`
+        // and one from the same file: the entries carry flags 0x00, 0x03, 0x03
+        // and 0x06, so names after the first are stored as entry indices.
+        let blob: &[u8] = &[
+            0x01, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x61, 0x5f, 0x72, 0x61,
+            0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x73, 0x6f, 0x75, 0x72,
+            0x63, 0x65, 0x5f, 0x66, 0x69, 0x6c, 0x65, 0x2e, 0x68, 0x35, 0x00, 0x61, 0x5f, 0x72,
+            0x61, 0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x64, 0x61, 0x74,
+            0x61, 0x73, 0x65, 0x74, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x00, 0x02, 0x00, 0x00, 0x00,
+            0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00,
+            0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02,
+            0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00,
+            0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03,
+            0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x04, 0x00, 0x01, 0x00, 0x01,
+            0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02,
+            0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
+            0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00,
+            0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x00, 0x01, 0x00,
+            0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02, 0x00,
+            0x00, 0x00, 0x02, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00,
+            0x01, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02,
+            0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00,
+            0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00,
+            0x00, 0x01, 0x02, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01,
+            0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x8e, 0xa7, 0xea, 0x7a,
+        ];
+        let mappings = parse_vds_mappings(blob, 8).unwrap();
+        let names: Vec<(&str, &str)> = mappings
+            .iter()
+            .map(|m| (m.source_file.as_str(), m.source_dataset.as_str()))
+            .collect();
+        let (file, dset) = ("a_rather_long_source_file.h5", "a_rather_long_dataset_name");
+        assert_eq!(
+            names,
+            vec![(file, dset), (file, dset), (file, dset), (".", dset)]
+        );
+    }
+
+    #[test]
+    fn parse_vds_mappings_v1_forward_reference_is_error() {
+        // Entry 0 claiming to share entry 0's file name must not index past
+        // the entries decoded so far.
+        let mut blob = vec![0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x01];
+        blob.extend_from_slice(&[0u8; 8]);
+        blob.extend_from_slice(b"d\0");
+        assert!(parse_vds_mappings(&blob, 8).is_err());
+        // Unknown flag bits are refused.
+        let blob = [0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x08, b'd', 0];
+        assert!(parse_vds_mappings(&blob, 8).is_err());
+    }
+
     #[test]
     fn parse_vds_mappings_external_v0() {
         // Block version 0 with an explicit (external) source file name.

crates/clawhdf5-format/src/data_read.rs

@@ -191,14 +191,9 @@ fn read_raw_data_full_impl(
             offset_size,
             length_size,
         ),
-        DataLayout::Virtual {
-            global_heap_address,
-            global_heap_index,
-            ..
-        } => read_virtual_data(
+        DataLayout::Virtual { .. } => read_virtual_data(
             file_data,
-            *global_heap_address,
-            *global_heap_index,
+            layout,
             dataspace,
             datatype,
             offset_size,
@@ -465,158 +460,54 @@ pub fn read_raw_data_selection(
     }
 }
 
-/// Assemble a **Virtual Dataset (VDS)** from its source mappings.
+/// Assemble a **Virtual Dataset (VDS)** through the raw-read API, which has no
+/// access to the dataset's fill value message.
 ///
-/// Supports virtual datasets of any rank. Same-file sources are read directly;
-/// **external-file** sources are read through the caller-supplied `resolver`,
-/// which maps a stored source file name to that file's bytes. Each mapping's
-/// selected source elements are scattered into the virtual buffer at the
-/// positions given by the virtual selection (both enumerated in row-major
-/// order, as HDF5 pairs them). Unmapped regions are left at the zero fill value.
-///
-/// A mapping whose external source file the resolver cannot supply (`None`) is
-/// skipped, leaving its region at fill — matching HDF5's tolerance of missing
-/// sources. An external source with no resolver at all is a hard error.
+/// Delegates to [`crate::vds::read_virtual_dataset`]. Because the fill value
+/// is unknown here, a virtual dataset with any element no mapping supplies
+/// (an unmapped region, or a missing source file or dataset) is an error
+/// rather than a guess at the fill value; so is one whose extent libhdf5
+/// would report differently from the stored dataspace (unlimited mappings).
+/// Use [`crate::vds::read_virtual_dataset`] to read those.
 #[allow(clippy::too_many_arguments)]
 fn read_virtual_data(
     file_data: &[u8],
-    global_heap_address: Option<u64>,
-    global_heap_index: u32,
+    layout: &DataLayout,
     dataspace: &Dataspace,
     datatype: &Datatype,
     offset_size: u8,
     length_size: u8,
     resolver: Option<&VdsSourceResolver>,
 ) -> Result<Vec<u8>, FormatError> {
-    use crate::data_layout::parse_vds_mappings;
-    use crate::global_heap::GlobalHeapCollection;
-    use crate::selection::Selection;
-
-    let elem_size = datatype.type_size() as usize;
-    let mut out = crate::chunked_read::alloc_output(crate::chunked_read::checked_byte_len(
-        dataspace.checked_num_elements()?,
-        elem_size,
-    )?)?;
-
-    let virtual_dims = &dataspace.dimensions;
-
-    let addr = global_heap_address.ok_or_else(|| {
-        FormatError::ChunkedReadError("virtual dataset has no mapping global heap".into())
-    })?;
-    let coll = GlobalHeapCollection::parse(file_data, addr as usize, length_size)?;
-    let obj =
-        coll.get_object(global_heap_index as u16)
-            .ok_or(FormatError::GlobalHeapObjectNotFound {
-                collection_address: addr,
-                index: global_heap_index as u16,
-            })?;
-    let mappings = parse_vds_mappings(&obj.data, length_size)?;
-
-    for m in &mappings {
-        let same_file = m.source_file.is_empty() || m.source_file == ".";
-
-        // Resolve the bytes of the file holding this source dataset.
-        let external;
-        let src_file_data: &[u8] = if same_file {
-            file_data
-        } else {
-            let r = resolver.ok_or_else(|| {
-                FormatError::ChunkedReadError(
-                    "external-file virtual dataset sources require a file resolver".into(),
-                )
-            })?;
-            match r(&m.source_file) {
-                Some(bytes) => {
-                    external = bytes;
-                    &external
-                }
-                // Source file unavailable: leave this region at fill value.
-                None => continue,
-            }
-        };
-
-        let (vsel, _) = Selection::decode_serialized(&m.virtual_selection)?;
-        let (ssel, _) = Selection::decode_serialized(&m.source_selection)?;
-
-        let (src_raw, src_dims) =
-            read_named_dataset_raw(src_file_data, &m.source_dataset, offset_size, length_size)?;
-
-        let vidx = vsel.iter_linear(virtual_dims)?;
-        let sidx = ssel.iter_linear(&src_dims)?;
-        if vidx.len() != sidx.len() {
-            return Err(FormatError::ChunkedReadError(
-                "virtual/source selection element counts differ".into(),
-            ));
-        }
-
-        for (&v, &s) in vidx.iter().zip(sidx.iter()) {
-            let (vo, so) = (v as usize * elem_size, s as usize * elem_size);
-            if vo + elem_size > out.len() || so + elem_size > src_raw.len() {
-                return Err(FormatError::ChunkedReadError(
-                    "virtual dataset selection out of bounds".into(),
-                ));
-            }
-            out[vo..vo + elem_size].copy_from_slice(&src_raw[so..so + elem_size]);
-        }
-    }
-
-    Ok(out)
-}
-
-/// Read a named dataset's raw (decoded) bytes and its dimensions, navigating
-/// from the superblock. Used to pull VDS source datasets out of the same file.
-fn read_named_dataset_raw(
-    file_data: &[u8],
-    path: &str,
-    _offset_size: u8,
-    _length_size: u8,
-) -> Result<(Vec<u8>, Vec<u64>), FormatError> {
-    use crate::filter_pipeline::FilterPipeline;
-    use crate::group_v2::resolve_path_any;
-    use crate::message_type::MessageType;
-    use crate::object_header::ObjectHeader;
-    use crate::signature::find_signature;
-    use crate::superblock::Superblock;
-
-    let sig = find_signature(file_data)?;
-    let sb = Superblock::parse(file_data, sig)?;
-    let addr = resolve_path_any(file_data, &sb, path)?;
-    let hdr = ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size)?;
-
-    let find = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t);
-    let ds_msg = find(MessageType::Dataspace)
-        .ok_or_else(|| FormatError::ChunkedReadError("VDS source has no dataspace".into()))?;
-    let dataspace = Dataspace::parse(&ds_msg.data, sb.length_size)?;
-    let dt_msg = find(MessageType::Datatype)
-        .ok_or_else(|| FormatError::ChunkedReadError("VDS source has no datatype".into()))?;
-    let (datatype, _) = Datatype::parse(&dt_msg.data)?;
-    let dl_msg = find(MessageType::DataLayout)
-        .ok_or_else(|| FormatError::ChunkedReadError("VDS source has no data layout".into()))?;
-    let layout = DataLayout::parse(&dl_msg.data, sb.offset_size, sb.length_size)?;
-    // A virtual dataset whose source is itself another virtual dataset could
-    // form a cycle (A -> B -> A) and recurse into a stack overflow. Nested
-    // virtual sources are exotic and unsupported, so stop here cleanly.
-    if matches!(layout, DataLayout::Virtual { .. }) {
+    let wrapped =
+        resolver.map(|r| move |name: &str| -> Result<Option<Vec<u8>>, FormatError> { Ok(r(name)) });
+    let wrapped_ref = wrapped.as_ref().map(|w| w as &crate::vds::VdsFileResolver);
+    let v = crate::vds::read_virtual_dataset(
+        file_data,
+        layout,
+        dataspace,
+        datatype,
+        None,
+        offset_size,
+        length_size,
+        wrapped_ref,
+    )?;
+    if v.dims != dataspace.dimensions {
         return Err(FormatError::ChunkedReadError(
-            "virtual dataset source is itself virtual (unsupported)".into(),
+            "virtual dataset extent differs from its stored dataspace; \
+             read it with vds::read_virtual_dataset"
+                .into(),
         ));
     }
-    let pipeline = find(MessageType::FilterPipeline)
-        .map(|m| FilterPipeline::parse(&m.data))
-        .transpose()?;
-
-    let raw = read_raw_data_full(
-        file_data,
-        &layout,
-        &dataspace,
-        &datatype,
-        pipeline.as_ref(),
-        sb.offset_size,
-        sb.length_size,
-    )?;
-    Ok((raw, dataspace.dimensions.clone()))
+    if v.unmapped > 0 {
+        return Err(FormatError::ChunkedReadError(
+            "virtual dataset has elements no source supplies, which read as its \
+             fill value; read it with vds::read_virtual_dataset and the fill value"
+                .into(),
+        ));
+    }
+    Ok(v.data)
 }
-
 /// Extract selected elements from a full dataset buffer.
 pub fn extract_selection_from_buffer(
     full_data: &[u8],

crates/clawhdf5-format/src/datatype.rs

@@ -423,13 +423,44 @@ impl Datatype {
                         ensure_len(data, pos, 4)?;
                         let byte_offset = LittleEndian::read_u32(&data[pos..pos + 4]) as u64;
                         pos += 4;
+                        // v1 members can be fixed-size arrays of the member
+                        // type (libhdf5 builds an array type from these
+                        // fields; the permutation is ignored, as libhdf5
+                        // does). Skipping them read a `[4] i32` member as
+                        // one `i32`.
+                        let mut array_dims = Vec::new();
                         if version == 1 {
                             ensure_len(data, pos, 28)?;
+                            let ndims = data[pos] as usize;
+                            // libhdf5 refuses more than four dimensions and,
+                            // when building the array type, a zero-sized one.
+                            let zero_dim = (0..ndims.min(4)).any(|j| {
+                                let at = pos + 12 + 4 * j;
+                                LittleEndian::read_u32(&data[at..at + 4]) == 0
+                            });
+                            if ndims > 4 || zero_dim {
+                                return Err(FormatError::InvalidDatatypeVersion {
+                                    class: class_id,
+                                    version,
+                                });
+                            }
+                            array_dims = (0..ndims)
+                                .map(|j| {
+                                    let at = pos + 12 + 4 * j;
+                                    LittleEndian::read_u32(&data[at..at + 4])
+                                })
+                                .collect();
                             pos += 28;
                         }
-                        let (member_dt, consumed) =
+                        let (mut member_dt, consumed) =
                             Self::parse_with_depth(&data[pos..], depth + 1)?;
                         pos += consumed;
+                        if !array_dims.is_empty() {
+                            member_dt = Datatype::Array {
+                                base_type: Box::new(member_dt),
+                                dimensions: array_dims,
+                            };
+                        }
                         members.push(CompoundMember {
                             name,
                             byte_offset,
@@ -1336,6 +1367,64 @@ mod tests {
         assert_xyid_compound(dt);
     }
 
+    #[test]
+    fn test_compound_v1_member_array_fields() {
+        // HDF5 1.6 wrote array members of a v1 compound through the legacy
+        // per-member fields (as in libhdf5's tools/test/testfiles/
+        // tcompound.h5 `type2`: `int_array` [4] i32, `float_array` [5][6]
+        // f32). They used to be skipped, reading each member as a scalar.
+        let i32le: [u8; 12] = [
+            0x10, 0x08, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x00,
+        ];
+        let mut b = vec![0x16, 0x02, 0x00, 0x00, 0x88, 0x00, 0x00, 0x00];
+        for (name, offset, dims) in [
+            (&b"int_array"[..], 0u32, &[4u32][..]),
+            (&b"xy"[..], 16, &[5u32, 6][..]),
+        ] {
+            let mut padded = name.to_vec();
+            padded.resize((name.len() + 1 + 7) & !7, 0);
+            b.extend_from_slice(&padded);
+            b.extend_from_slice(&offset.to_le_bytes());
+            b.push(dims.len() as u8);
+            b.extend_from_slice(&[0u8; 3 + 4 + 4]); // reserved, permutation, reserved
+            for j in 0..4 {
+                b.extend_from_slice(&dims.get(j).copied().unwrap_or(0).to_le_bytes());
+            }
+            b.extend_from_slice(&i32le);
+        }
+        let (dt, consumed) = Datatype::parse(&b).unwrap();
+        assert_eq!(consumed, b.len());
+        let Datatype::Compound { members, .. } = dt else {
+            panic!("expected Compound, got {dt:?}");
+        };
+        let got: Vec<(&str, u64, u32, Option<Vec<u32>>)> = members
+            .iter()
+            .map(|m| {
+                let dims = match &m.datatype {
+                    Datatype::Array { dimensions, .. } => Some(dimensions.clone()),
+                    _ => None,
+                };
+                (m.name.as_str(), m.byte_offset, m.datatype.type_size(), dims)
+            })
+            .collect();
+        assert_eq!(
+            got,
+            vec![
+                ("int_array", 0, 16, Some(vec![4])),
+                ("xy", 16, 120, Some(vec![5, 6])),
+            ]
+        );
+
+        // More than four dimensions cannot be encoded, and libhdf5 refuses a
+        // zero-sized dimension (a fuzzed tcompound.h5, cve-2024-32616.h5).
+        let mut bad = b.clone();
+        bad[8 + 16 + 4] = 5;
+        assert!(Datatype::parse(&bad).is_err());
+        let mut bad = b.clone();
+        bad[8 + 16 + 4] = 2; // [4, 0]
+        assert!(Datatype::parse(&bad).is_err());
+    }
+
     #[test]
     fn test_compound_v1_truncated_is_error_not_panic() {
         let bytes = compound_v1_bytes();

crates/clawhdf5-format/src/error.rs

@@ -80,6 +80,9 @@ pub enum FormatError {
     InvalidLocalHeapSignature,
     /// Invalid local heap version.
     InvalidLocalHeapVersion(u8),
+    /// A local heap's free list points outside its data segment (libhdf5:
+    /// "bad heap free list").
+    InvalidLocalHeapFreeList,
     /// Invalid B-tree v1 signature.
     InvalidBTreeSignature,
     /// Invalid B-tree node type.
@@ -117,6 +120,14 @@ pub enum FormatError {
     /// A message is marked shared but was parsed without access to the file,
     /// so the reference to the real message could not be followed.
     UnresolvedSharedMessage,
+    /// A shared-message reference points at an object header that holds no
+    /// (unshared) message of the referenced type (raw message type id).
+    SharedMessageTargetMissing(u16),
+    /// A superblock was parsed at a non-zero offset of the buffer (the file
+    /// has a user block of this many bytes). HDF5 addresses are relative to
+    /// the superblock, so the buffer must start there: see
+    /// `signature::split_user_block`.
+    UserBlockNotStripped(u64),
     /// A selection does not fit the dataset it was applied to (wrong rank, or
     /// it reaches past a dimension's extent).
     SelectionOutOfBounds(String),
@@ -270,6 +281,9 @@ impl fmt::Display for FormatError {
             FormatError::InvalidLocalHeapSignature => {
                 write!(f, "invalid local heap signature")
             }
+            FormatError::InvalidLocalHeapFreeList => {
+                write!(f, "bad local heap free list")
+            }
             FormatError::InvalidLocalHeapVersion(v) => {
                 write!(f, "invalid local heap version: {v}")
             }
@@ -339,6 +353,16 @@ impl fmt::Display for FormatError {
             FormatError::SelectionOutOfBounds(msg) => {
                 write!(f, "selection out of bounds: {msg}")
             }
+            FormatError::UserBlockNotStripped(n) => write!(
+                f,
+                "file has a {n}-byte user block: parse the bytes from the superblock on \
+                 (signature::split_user_block)"
+            ),
+            FormatError::SharedMessageTargetMissing(t) => write!(
+                f,
+                "shared message reference points at an object header with no message of type \
+                 {t:#06x}"
+            ),
             FormatError::UnresolvedSharedMessage => write!(
                 f,
                 "message is shared but no file data was available to resolve it"

crates/clawhdf5-format/src/file_writer.rs

@@ -1947,6 +1947,12 @@ mod tests {
             root_block_address: 0,
             current_rows_in_root_indirect_block: 0,
             managed_objects_count: 0,
+            huge_btree_address: u64::MAX,
+            filter_pipeline: None,
+            root_direct_block_filtered_size: 0,
+            root_direct_block_filter_mask: 0,
+            offset_size: 8,
+            length_size: 8,
         };
         let (off, len) = fh.decode_managed_id(&id).unwrap();
         assert_eq!(off, 100);

crates/clawhdf5-format/src/fractal_heap.rs

@@ -1,12 +1,14 @@
 //! HDF5 Fractal Heap parsing for v2 group link storage.
 
 #[cfg(not(feature = "std"))]
-use alloc::vec::Vec;
+use alloc::{format, vec::Vec};
 
 #[cfg(feature = "checksum")]
 use byteorder::{ByteOrder, LittleEndian};
 
+use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
 use crate::error::FormatError;
+use crate::filter_pipeline::FilterPipeline;
 
 /// Parsed fractal heap header (signature "FRHP").
 #[derive(Debug, Clone)]
@@ -33,6 +35,23 @@ pub struct FractalHeapHeader {
     pub current_rows_in_root_indirect_block: u16,
     /// Total number of managed objects.
     pub managed_objects_count: u64,
+    /// Address of the v2 B-tree indexing "huge" objects (undefined address
+    /// when the heap has none). Huge objects are those larger than
+    /// `max_managed_object_size`; they live outside the heap's blocks.
+    pub huge_btree_address: u64,
+    /// The heap's I/O filter pipeline, if it has one. It applies to managed
+    /// direct blocks and to huge objects.
+    pub filter_pipeline: Option<FilterPipeline>,
+    /// Stored (filtered) size of the root direct block; meaningful only when
+    /// the heap is filtered and its root is a direct block.
+    pub root_direct_block_filtered_size: u64,
+    /// Filter mask of the root direct block (bit *i* set = filter *i*
+    /// skipped); meaningful only when the heap is filtered.
+    pub root_direct_block_filter_mask: u32,
+    /// Size of addresses in the file ("Size of Offsets").
+    pub offset_size: u8,
+    /// Size of lengths in the file ("Size of Lengths").
+    pub length_size: u8,
 }
 
 fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
@@ -79,6 +98,38 @@ fn is_undefined(val: u64, offset_size: u8) -> bool {
     }
 }
 
+/// Little-endian unsigned integer of up to 8 bytes.
+fn le_uint(bytes: &[u8]) -> u64 {
+    bytes
+        .iter()
+        .take(8)
+        .enumerate()
+        .fold(0u64, |acc, (i, &b)| acc | (u64::from(b) << (i * 8)))
+}
+
+fn heap_error(msg: &str) -> FormatError {
+    FormatError::ChunkedReadError(format!("fractal heap: {msg}"))
+}
+
+/// Heap ID type, from bits 4-5 of an ID's first byte (libhdf5's
+/// `H5HF_ID_TYPE_MASK`, 0x30); bits 6-7 are the ID version, which must be 0.
+const HEAP_ID_MANAGED: u8 = 0;
+const HEAP_ID_HUGE: u8 = 1;
+const HEAP_ID_TINY: u8 = 2;
+
+/// The type (0 managed, 1 huge, 2 tiny) of a heap ID from its first byte,
+/// refusing an ID version other than 0.
+fn heap_id_type(first: u8) -> Result<u8, FormatError> {
+    if first >> 6 != 0 {
+        return Err(heap_error("unsupported heap ID version"));
+    }
+    Ok((first >> 4) & 0x03)
+}
+
+/// v2 B-tree record types indexing a heap's huge objects.
+const BTREE_HUGE_INDIRECT: u8 = 1;
+const BTREE_HUGE_INDIRECT_FILTERED: u8 = 2;
+
 impl FractalHeapHeader {
     /// Parse a fractal heap header at the given offset.
     pub fn parse(
@@ -122,11 +173,17 @@ impl FractalHeapHeader {
         ]);
         pos += 4;
 
-        // Skip several fixed fields: next_huge_object_id(ls), btree_huge_objects_address(os),
-        // free_space_managed_blocks(ls), managed_block_free_space_manager_address(os),
+        // next_huge_object_id (length_size)
+        ensure_len(file_data, pos, ls)?;
+        pos += ls;
+        // btree_huge_objects_address (offset_size)
+        let huge_btree_address = read_offset(file_data, pos, offset_size)?;
+        pos += os;
+
+        // Skip: free_space_managed_blocks(ls), managed_block_free_space_manager_address(os),
         // managed_space_in_heap(ls), allocated_managed_space_in_heap(ls),
         // direct_block_allocation_iterator_offset(ls)
-        let skip_size = 5 * ls + 2 * os;
+        let skip_size = 4 * ls + os;
         ensure_len(file_data, pos, skip_size)?;
         pos += skip_size;
 
@@ -134,14 +191,9 @@ impl FractalHeapHeader {
         let managed_objects_count = read_offset(file_data, pos, length_size)?;
         pos += ls;
 
-        // huge_objects_size (length_size)
-        pos += ls;
-        // huge_objects_count (length_size)
-        pos += ls;
-        // tiny_objects_size (length_size)
-        pos += ls;
-        // tiny_objects_count (length_size)
-        pos += ls;
+        // huge_objects_size, huge_objects_count, tiny_objects_size,
+        // tiny_objects_count (length_size each)
+        pos += 4 * ls;
 
         // table_width (2)
         ensure_len(file_data, pos, 2)?;
@@ -175,16 +227,28 @@ impl FractalHeapHeader {
         ensure_len(file_data, pos, 2)?;
         let current_rows_in_root_indirect_block =
             u16::from_le_bytes([file_data[pos], file_data[pos + 1]]);
-        #[allow(unused_variables, unused_mut, unused_assignments)]
-        let mut pos = pos + 2;
+        pos += 2;
 
-        // Skip IO filter encoded info if present
+        // With I/O filters: root direct block's filtered size (length_size),
+        // its filter mask (4), then the encoded filter pipeline message.
+        let mut filter_pipeline = None;
+        let mut root_direct_block_filtered_size = 0;
+        let mut root_direct_block_filter_mask = 0;
         if io_filter_encoded_length > 0 {
-            // root_block_filter_info_size (length_size) + filter_mask (4)
-            #[allow(unused_assignments)]
-            {
-                pos += ls + 4;
-            }
+            root_direct_block_filtered_size = read_offset(file_data, pos, length_size)?;
+            pos += ls;
+            ensure_len(file_data, pos, 4)?;
+            root_direct_block_filter_mask = u32::from_le_bytes([
+                file_data[pos],
+                file_data[pos + 1],
+                file_data[pos + 2],
+                file_data[pos + 3],
+            ]);
+            pos += 4;
+            let n = io_filter_encoded_length as usize;
+            ensure_len(file_data, pos, n)?;
+            filter_pipeline = Some(FilterPipeline::parse(&file_data[pos..pos + n])?);
+            pos += n;
         }
 
         // Validate header checksum
@@ -200,6 +264,8 @@ impl FractalHeapHeader {
                 });
             }
         }
+        #[cfg(not(feature = "checksum"))]
+        let _ = pos;
 
         Ok(FractalHeapHeader {
             heap_id_length,
@@ -213,13 +279,19 @@ impl FractalHeapHeader {
             root_block_address,
             current_rows_in_root_indirect_block,
             managed_objects_count,
+            huge_btree_address,
+            filter_pipeline,
+            root_direct_block_filtered_size,
+            root_direct_block_filter_mask,
+            offset_size,
+            length_size,
         })
     }
 
     /// Decode a managed heap ID into (offset_in_heap, object_length).
     ///
     /// The heap ID layout for managed objects (type 0):
-    /// - Byte 0: bits 6-7 = type (0), bits 4-5 = version (0), bits 0-3 = reserved
+    /// - Byte 0: bits 6-7 = version (0), bits 4-5 = type (0), bits 0-3 = reserved
     /// - Bytes 1+: offset (max_heap_size bits, LE) then length (remaining bits, LE)
     pub fn decode_managed_id(&self, id_bytes: &[u8]) -> Result<(u64, u64), FormatError> {
         if id_bytes.is_empty() {
@@ -229,8 +301,8 @@ impl FractalHeapHeader {
             });
         }
 
-        let id_type = (id_bytes[0] >> 6) & 0x03;
-        if id_type != 0 {
+        let id_type = heap_id_type(id_bytes[0])?;
+        if id_type != HEAP_ID_MANAGED {
             return Err(FormatError::InvalidHeapIdType(id_type));
         }
 
@@ -269,12 +341,183 @@ impl FractalHeapHeader {
         Ok((heap_offset, length_val))
     }
 
-    /// Read a managed object from the heap given its raw heap ID bytes.
+    /// Read any object from the heap given its raw heap ID bytes: managed
+    /// (stored in the heap's blocks), huge (stored outside them, found
+    /// directly from the ID or through the huge-object v2 B-tree, optionally
+    /// filtered) or tiny (stored in the ID itself).
+    ///
+    /// Despite its name this accepts every ID type; `offset_size` must match
+    /// the one the header was parsed with.
     pub fn read_managed_object(
         &self,
         file_data: &[u8],
         id_bytes: &[u8],
         offset_size: u8,
+    ) -> Result<Vec<u8>, FormatError> {
+        let Some(&first) = id_bytes.first() else {
+            return Err(FormatError::UnexpectedEof {
+                expected: 1,
+                available: 0,
+            });
+        };
+        match heap_id_type(first)? {
+            HEAP_ID_MANAGED => self.read_heap_managed(file_data, id_bytes, offset_size),
+            HEAP_ID_HUGE => self.read_huge_object(file_data, id_bytes),
+            HEAP_ID_TINY => self.read_tiny_object(id_bytes),
+            other => Err(FormatError::InvalidHeapIdType(other)),
+        }
+    }
+
+    /// Whether a huge object's ID holds its address and length directly
+    /// (libhdf5 does this when they fit in the ID), rather than a key into
+    /// the huge-object B-tree.
+    fn huge_ids_direct(&self) -> bool {
+        let room = usize::from(self.heap_id_length).saturating_sub(1);
+        let os = usize::from(self.offset_size);
+        let ls = usize::from(self.length_size);
+        if self.filter_pipeline.is_some() {
+            room >= os + ls + 4 + ls
+        } else {
+            room >= os + ls
+        }
+    }
+
+    /// Read a huge object (heap ID type 1).
+    fn read_huge_object(&self, file_data: &[u8], id: &[u8]) -> Result<Vec<u8>, FormatError> {
+        let os = usize::from(self.offset_size);
+        let ls = usize::from(self.length_size);
+        // (address, stored length, filter mask, decoded length); the last two
+        // only matter for a filtered heap.
+        let (addr, stored_len, mask, mem_len) = if self.huge_ids_direct() {
+            let body = &id[1..];
+            let need = if self.filter_pipeline.is_some() {
+                os + ls + 4 + ls
+            } else {
+                os + ls
+            };
+            ensure_len(body, 0, need)?;
+            let addr = le_uint(&body[..os]);
+            let len = le_uint(&body[os..os + ls]);
+            if self.filter_pipeline.is_some() {
+                let mask = u32::from_le_bytes([
+                    body[os + ls],
+                    body[os + ls + 1],
+                    body[os + ls + 2],
+                    body[os + ls + 3],
+                ]);
+                let mem = le_uint(&body[os + ls + 4..os + ls + 4 + ls]);
+                (addr, len, mask, mem)
+            } else {
+                (addr, len, 0, len)
+            }
+        } else {
+            let key_len = (usize::from(self.heap_id_length).saturating_sub(1)).min(8);
+            ensure_len(id, 1, key_len)?;
+            let key = le_uint(&id[1..1 + key_len]);
+            self.find_huge_record(file_data, key)?
+        };
+
+        let start = usize::try_from(addr).map_err(|_| heap_error("huge object address"))?;
+        let len = usize::try_from(stored_len).map_err(|_| heap_error("huge object length"))?;
+        ensure_len(file_data, start, len)?;
+        let stored = &file_data[start..start + len];
+        match &self.filter_pipeline {
+            None => Ok(stored.to_vec()),
+            Some(pipeline) => {
+                let mem = usize::try_from(mem_len).map_err(|_| heap_error("huge object size"))?;
+                let out = crate::filters::decompress_chunk_masked(stored, pipeline, mem, 1, mask)?;
+                if out.len() != mem {
+                    return Err(heap_error("filtered huge object decoded to the wrong size"));
+                }
+                Ok(out)
+            }
+        }
+    }
+
+    /// Look up huge object `key` in the huge-object v2 B-tree, returning
+    /// (address, stored length, filter mask, decoded length).
+    fn find_huge_record(
+        &self,
+        file_data: &[u8],
+        key: u64,
+    ) -> Result<(u64, u64, u32, u64), FormatError> {
+        if is_undefined(self.huge_btree_address, self.offset_size) {
+            return Err(heap_error(
+                "huge object ID but the heap has no huge-object index",
+            ));
+        }
+        let hdr = BTreeV2Header::parse(
+            file_data,
+            self.huge_btree_address as usize,
+            self.offset_size,
+            self.length_size,
+        )?;
+        let os = usize::from(self.offset_size);
+        let ls = usize::from(self.length_size);
+        let filtered = self.filter_pipeline.is_some();
+        let (expected_type, rec_len) = if filtered {
+            (BTREE_HUGE_INDIRECT_FILTERED, os + ls + 4 + ls + ls)
+        } else {
+            (BTREE_HUGE_INDIRECT, os + ls + ls)
+        };
+        if hdr.tree_type != expected_type || usize::from(hdr.record_size) < rec_len {
+            return Err(heap_error("unexpected huge-object B-tree record type"));
+        }
+        let records =
+            collect_btree_v2_records(file_data, &hdr, self.offset_size, self.length_size)?;
+        for rec in &records {
+            let d = &rec.data;
+            if d.len() < rec_len {
+                continue;
+            }
+            let addr = le_uint(&d[..os]);
+            let len = le_uint(&d[os..os + ls]);
+            if filtered {
+                let mask = u32::from_le_bytes([
+                    d[os + ls],
+                    d[os + ls + 1],
+                    d[os + ls + 2],
+                    d[os + ls + 3],
+                ]);
+                let mem = le_uint(&d[os + ls + 4..os + 2 * ls + 4]);
+                let id = le_uint(&d[os + 2 * ls + 4..os + 3 * ls + 4]);
+                if id == key {
+                    return Ok((addr, len, mask, mem));
+                }
+            } else {
+                let id = le_uint(&d[os + ls..os + 2 * ls]);
+                if id == key {
+                    return Ok((addr, len, 0, len));
+                }
+            }
+        }
+        Err(heap_error("huge object not found in its B-tree"))
+    }
+
+    /// Read a tiny object (heap ID type 2), stored in the ID itself.
+    fn read_tiny_object(&self, id: &[u8]) -> Result<Vec<u8>, FormatError> {
+        // libhdf5 uses a one-byte length (low 4 bits of byte 0) unless the ID
+        // is long enough to need 12 bits, which then borrow byte 1.
+        let extended = usize::from(self.heap_id_length).saturating_sub(1) > 17;
+        let (len, start) = if extended {
+            ensure_len(id, 0, 2)?;
+            (
+                ((usize::from(id[0] & 0x0F)) << 8 | usize::from(id[1])) + 1,
+                2,
+            )
+        } else {
+            (usize::from(id[0] & 0x0F) + 1, 1)
+        };
+        ensure_len(id, start, len)?;
+        Ok(id[start..start + len].to_vec())
+    }
+
+    /// Read a managed object (heap ID type 0).
+    fn read_heap_managed(
+        &self,
+        file_data: &[u8],
+        id_bytes: &[u8],
+        offset_size: u8,
     ) -> Result<Vec<u8>, FormatError> {
         let (heap_offset, obj_len) = self.decode_managed_id(id_bytes)?;
 
@@ -289,12 +532,15 @@ impl FractalHeapHeader {
             // Root is a direct block
             self.read_from_direct_block(
                 file_data,
-                self.root_block_address as usize,
-                self.starting_block_size,
-                0, // block offset in heap = 0 for root
+                DirectBlock {
+                    addr: self.root_block_address as usize,
+                    size: self.starting_block_size,
+                    heap_offset: 0,
+                    filtered_size: self.root_direct_block_filtered_size,
+                    filter_mask: self.root_direct_block_filter_mask,
+                },
                 heap_offset,
                 obj_len as usize,
-                offset_size,
             )
         } else {
             // Root is an indirect block — limit recursion to 64 levels
@@ -313,27 +559,41 @@ impl FractalHeapHeader {
 
     /// Read an object from a direct block.
     ///
-    /// The heap offset is relative to the start of the block (including its header),
-    /// so we just add it to the block address minus the block's heap offset.
-    #[allow(clippy::too_many_arguments)]
+    /// The heap offset is relative to the start of the block (including its
+    /// header), so we just add it to the block address minus the block's heap
+    /// offset. A filtered heap stores each direct block (header included)
+    /// through its filter pipeline, so the block is decoded first.
     fn read_from_direct_block(
         &self,
         file_data: &[u8],
-        block_addr: usize,
-        _block_size: u64,
-        block_heap_offset: u64,
+        block: DirectBlock,
         target_offset: u64,
         length: usize,
-        _offset_size: u8,
     ) -> Result<Vec<u8>, FormatError> {
-        if target_offset < block_heap_offset {
+        if target_offset < block.heap_offset {
             return Err(FormatError::UnexpectedEof {
-                expected: block_heap_offset as usize,
+                expected: block.heap_offset as usize,
                 available: target_offset as usize,
             });
         }
-        let local_offset = (target_offset - block_heap_offset) as usize;
-        let pos = block_addr
+        let local_offset = (target_offset - block.heap_offset) as usize;
+        if let Some(pipeline) = &self.filter_pipeline {
+            let stored_len = usize::try_from(block.filtered_size)
+                .map_err(|_| heap_error("direct block size"))?;
+            let size = usize::try_from(block.size).map_err(|_| heap_error("direct block size"))?;
+            ensure_len(file_data, block.addr, stored_len)?;
+            let decoded = crate::filters::decompress_chunk_masked(
+                &file_data[block.addr..block.addr + stored_len],
+                pipeline,
+                size,
+                1,
+                block.filter_mask,
+            )?;
+            ensure_len(&decoded, local_offset, length)?;
+            return Ok(decoded[local_offset..local_offset + length].to_vec());
+        }
+        let pos = block
+            .addr
             .checked_add(local_offset)
             .ok_or(FormatError::UnexpectedEof {
                 expected: usize::MAX,
@@ -371,19 +631,13 @@ impl FractalHeapHeader {
         let iblock_header = 5 + offset_size as usize + block_offset_bytes;
         let mut pos = iblock_addr + iblock_header;
 
-        // Compute block sizes for each row using the doubling table
         let tw = self.table_width as u64;
-
         let nrows_usize = nrows as usize;
-
-        // Build table of (block_size, heap_offset) for each child entry
         let mut current_heap_offset = iblock_heap_offset;
 
         // Rows below max_direct_rows hold direct blocks; rows at/above hold
         // child indirect blocks. (NOT the FRHP "starting rows" field.)
         let start_indirect = self.max_direct_rows();
-
-        // Read child addresses for direct block rows
         let max_direct_rows = nrows_usize.min(start_indirect);
 
         for row in 0..max_direct_rows {
@@ -393,60 +647,74 @@ impl FractalHeapHeader {
                 let child_addr = read_offset(file_data, pos, offset_size)?;
                 pos += offset_size as usize;
 
-                if self.io_filter_encoded_length > 0 {
-                    // filtered_size(length_size) + filter_mask(4)
-                    // Skip for now - we don't handle filtered direct blocks in fractal heaps
-                    pos += 4; // filter_mask - simplified
-                }
+                // A filtered heap stores each direct block's filtered size
+                // (length_size) and filter mask (4) after its address.
+                let (filtered_size, filter_mask) = if self.filter_pipeline.is_some() {
+                    let size = read_offset(file_data, pos, self.length_size)?;
+                    pos += usize::from(self.length_size);
+                    ensure_len(file_data, pos, 4)?;
+                    let mask = u32::from_le_bytes([
+                        file_data[pos],
+                        file_data[pos + 1],
+                        file_data[pos + 2],
+                        file_data[pos + 3],
+                    ]);
+                    pos += 4;
+                    (size, mask)
+                } else {
+                    (0, 0)
+                };
 
-                if !is_undefined(child_addr, offset_size) {
-                    let block_end = current_heap_offset + block_size;
-                    if target_offset >= current_heap_offset && target_offset < block_end {
-                        return self.read_from_direct_block(
-                            file_data,
-                            child_addr as usize,
-                            block_size,
-                            current_heap_offset,
-                            target_offset,
-                            length,
-                            offset_size,
-                        );
-                    }
+                let block_end = current_heap_offset.saturating_add(block_size);
+                if !is_undefined(child_addr, offset_size)
+                    && target_offset >= current_heap_offset
+                    && target_offset < block_end
+                {
+                    return self.read_from_direct_block(
+                        file_data,
+                        DirectBlock {
+                            addr: child_addr as usize,
+                            size: block_size,
+                            heap_offset: current_heap_offset,
+                            filtered_size,
+                            filter_mask,
+                        },
+                        target_offset,
+                        length,
+                    );
                 }
-                current_heap_offset += block_size;
+                current_heap_offset = block_end;
             }
         }
 
-        // If we have indirect block rows
+        // Rows at and above `start_indirect` hold child indirect blocks. A
+        // child in row r spans exactly that row's block size of heap space,
+        // so it has as many rows as a table of that total size needs.
         for row in start_indirect..nrows_usize {
-            let _block_size = self.block_size_for_row(row);
-            let child_nrows = row - start_indirect + 1;
+            let child_space = self.block_size_for_row(row);
+            let child_nrows = self.rows_for_size(child_space);
 
             for _col in 0..tw {
                 let child_addr = read_offset(file_data, pos, offset_size)?;
                 pos += offset_size as usize;
 
-                if !is_undefined(child_addr, offset_size) {
-                    // Calculate total heap space covered by this indirect block child
-                    let total_child_space = self.indirect_block_heap_size(child_nrows);
-                    let block_end = current_heap_offset + total_child_space;
-                    if target_offset >= current_heap_offset && target_offset < block_end {
-                        return self.read_from_indirect_block(
-                            file_data,
-                            child_addr as usize,
-                            child_nrows as u16,
-                            current_heap_offset,
-                            target_offset,
-                            length,
-                            offset_size,
-                            depth_remaining - 1,
-                        );
-                    }
-                    current_heap_offset += total_child_space;
-                } else {
-                    let total_child_space = self.indirect_block_heap_size(child_nrows);
-                    current_heap_offset += total_child_space;
+                let block_end = current_heap_offset.saturating_add(child_space);
+                if !is_undefined(child_addr, offset_size)
+                    && target_offset >= current_heap_offset
+                    && target_offset < block_end
+                {
+                    return self.read_from_indirect_block(
+                        file_data,
+                        child_addr as usize,
+                        child_nrows,
+                        current_heap_offset,
+                        target_offset,
+                        length,
+                        offset_size,
+                        depth_remaining - 1,
+                    );
                 }
+                current_heap_offset = block_end;
             }
         }
 
@@ -475,25 +743,34 @@ impl FractalHeapHeader {
         log2 + 2
     }
 
+    /// Rows an indirect block needs to span `size` bytes of heap space:
+    /// `log2(size) - log2(starting_block_size * table_width) + 1`, as
+    /// libhdf5's `H5HF__dtable_size_to_rows`.
+    fn rows_for_size(&self, size: u64) -> u16 {
+        let log2 = |v: u64| 63u32.saturating_sub(v.max(1).leading_zeros());
+        let first_row_bits = log2(self.starting_block_size) + log2(u64::from(self.table_width));
+        (log2(size).saturating_sub(first_row_bits) + 1) as u16
+    }
+
     /// Get block size for a given row in the doubling table.
     fn block_size_for_row(&self, row: usize) -> u64 {
         let sbs = self.starting_block_size;
         if row <= 1 {
             sbs
         } else {
-            sbs * (1u64 << (row - 1))
+            sbs.saturating_mul(1u64.checked_shl((row - 1) as u32).unwrap_or(u64::MAX))
         }
     }
+}
 
-    /// Total heap space covered by an indirect block with the given number of rows.
-    fn indirect_block_heap_size(&self, nrows: usize) -> u64 {
-        let tw = self.table_width as u64;
-        let mut total = 0u64;
-        for row in 0..nrows {
-            total += self.block_size_for_row(row) * tw;
-        }
-        total
-    }
+/// A managed direct block's location, extent and (for a filtered heap) its
+/// stored size and filter mask.
+struct DirectBlock {
+    addr: usize,
+    size: u64,
+    heap_offset: u64,
+    filtered_size: u64,
+    filter_mask: u32,
 }
 
 #[cfg(test)]
@@ -641,7 +918,7 @@ mod tests {
         let hdr = FractalHeapHeader::parse(&file_data, 0, 8, 8).unwrap();
 
         // Build a managed heap ID:
-        // byte 0: type=0 (bits 6-7 = 00), version=0 (bits 4-5), reserved (bits 0-3)
+        // byte 0: version=0 (bits 6-7), type=0 (bits 4-5), reserved (bits 0-3)
         // bytes 1-6: offset (max_heap_size=16 bits) then length (remaining bits)
         // For offset=0, length=13:
         // payload = offset | (length << 16) = 0 | (13 << 16) = 0x000D0000
@@ -705,9 +982,46 @@ mod tests {
     fn invalid_heap_id_type() {
         let (file_data, _) = build_simple_heap(8, 8);
         let hdr = FractalHeapHeader::parse(&file_data, 0, 8, 8).unwrap();
-        // Type = 1 (tiny) in bits 6-7
-        let id = vec![0x40u8, 0, 0, 0, 0, 0, 0]; // bit 6 set = type 1
+        // Type = 1 (huge) in bits 4-5 is not a managed ID
+        let id = vec![0x10u8, 0, 0, 0, 0, 0, 0];
         let err = hdr.decode_managed_id(&id).unwrap_err();
         assert_eq!(err, FormatError::InvalidHeapIdType(1));
     }
+
+    #[test]
+    fn tiny_object_is_read_from_the_id() {
+        let (file_data, _) = build_simple_heap(8, 8);
+        let hdr = FractalHeapHeader::parse(&file_data, 0, 8, 8).unwrap();
+        // Type 2 (0x20), length - 1 in the low 4 bits, data after.
+        let id = [0x20 | 2, b'a', b'b', b'c', 0, 0, 0];
+        assert_eq!(hdr.read_managed_object(&file_data, &id, 8).unwrap(), b"abc");
+        // A length running past the ID is an error, not a short read.
+        let id = [0x20 | 9, b'a', b'b', b'c', 0, 0, 0];
+        assert!(hdr.read_managed_object(&file_data, &id, 8).is_err());
+    }
+
+    #[test]
+    fn huge_object_with_a_direct_id() {
+        // With IDs long enough for an address and a length, libhdf5 stores
+        // huge objects' location in the ID instead of the huge-object B-tree.
+        let (mut file_data, _) = build_simple_heap(8, 8);
+        let mut hdr = FractalHeapHeader::parse(&file_data, 0, 8, 8).unwrap();
+        hdr.heap_id_length = 17;
+        file_data[900..905].copy_from_slice(b"huge!");
+        let mut id = vec![0x10u8];
+        id.extend_from_slice(&900u64.to_le_bytes());
+        id.extend_from_slice(&5u64.to_le_bytes());
+        assert_eq!(
+            hdr.read_managed_object(&file_data, &id, 8).unwrap(),
+            b"huge!"
+        );
+    }
+
+    #[test]
+    fn unknown_heap_id_version_is_refused() {
+        let (file_data, _) = build_simple_heap(8, 8);
+        let hdr = FractalHeapHeader::parse(&file_data, 0, 8, 8).unwrap();
+        let id = [0x40u8, 0, 0, 0, 0, 0, 0];
+        assert!(hdr.read_managed_object(&file_data, &id, 8).is_err());
+    }
 }

crates/clawhdf5-format/src/group_v1.rs

@@ -45,9 +45,16 @@ pub fn resolve_v1_group_entries(
     )?;
 
     let mut entries = Vec::new();
+    let mut heap_checked = false;
     for snod_addr in snod_addrs {
         let snod = SymbolTableNode::parse(file_data, snod_addr as usize, offset_size)?;
         for entry in &snod.entries {
+            // Like libhdf5, look at the heap's free list only once a name is
+            // needed: an empty group with a damaged heap still lists.
+            if !heap_checked {
+                heap.validate_free_list(file_data, length_size)?;
+                heap_checked = true;
+            }
             let name = heap.read_string(file_data, entry.link_name_offset)?;
             entries.push(GroupEntry {
                 name,
@@ -73,6 +80,53 @@ pub fn find_v1_soft_link(
     offset_size: u8,
     length_size: u8,
 ) -> Result<Option<String>, FormatError> {
+    let mut found = None;
+    for_each_v1_soft_link(
+        file_data,
+        sym_table_msg,
+        offset_size,
+        length_size,
+        |link_name| link_name == name,
+        |_, target| {
+            found = Some(target);
+            false
+        },
+    )?;
+    Ok(found)
+}
+
+/// Every soft link in a v1 group, as `(name, target path)`.
+pub fn v1_soft_links(
+    file_data: &[u8],
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<(String, String)>, FormatError> {
+    let mut links = Vec::new();
+    for_each_v1_soft_link(
+        file_data,
+        sym_table_msg,
+        offset_size,
+        length_size,
+        |_| true,
+        |name, target| {
+            links.push((String::from(name), target));
+            true
+        },
+    )?;
+    Ok(links)
+}
+
+/// Visit the soft links of a v1 group whose name passes `wanted`, with their
+/// target paths, until `visit` returns false.
+fn for_each_v1_soft_link(
+    file_data: &[u8],
+    sym_table_msg: &SymbolTableMessage,
+    offset_size: u8,
+    length_size: u8,
+    wanted: impl Fn(&str) -> bool,
+    mut visit: impl FnMut(&str, String) -> bool,
+) -> Result<(), FormatError> {
     let heap = LocalHeap::parse(
         file_data,
         sym_table_msg.local_heap_address as usize,
@@ -85,13 +139,19 @@ pub fn find_v1_soft_link(
         offset_size,
         length_size,
     )?;
+    let mut heap_checked = false;
     for snod_addr in snod_addrs {
         let snod = SymbolTableNode::parse(file_data, snod_addr as usize, offset_size)?;
         for entry in &snod.entries {
             if entry.cache_type != CACHE_TYPE_SOFT_LINK {
                 continue;
             }
-            if heap.read_string(file_data, entry.link_name_offset)? != name {
+            if !heap_checked {
+                heap.validate_free_list(file_data, length_size)?;
+                heap_checked = true;
+            }
+            let name = heap.read_string(file_data, entry.link_name_offset)?;
+            if !wanted(&name) {
                 continue;
             }
             let value_offset = u32::from_le_bytes([
@@ -100,12 +160,19 @@ pub fn find_v1_soft_link(
                 entry.scratch_pad[2],
                 entry.scratch_pad[3],
             ]);
-            return heap
-                .read_string(file_data, u64::from(value_offset))
-                .map(Some);
+            let target = heap.read_string(file_data, u64::from(value_offset))?;
+            if !visit(&name, target) {
+                return Ok(());
+            }
         }
     }
-    Ok(None)
+    Ok(())
+}
+
+/// Whether a v1 symbol-table entry is a soft link (no object header of its
+/// own; its target path is in the local heap).
+pub fn is_v1_soft_link(entry: &GroupEntry) -> bool {
+    entry.cache_type == CACHE_TYPE_SOFT_LINK
 }
 
 /// Extract the SymbolTableMessage from an object header's messages.

crates/clawhdf5-format/src/group_v2.rs

@@ -38,6 +38,24 @@ pub fn resolve_v2_group_entries(
     }
 }
 
+/// First user-defined link type (HDF5 reserves 2-63; 64 is external).
+const FIRST_USER_DEFINED_LINK_TYPE: u8 = 65;
+
+/// Parse a Link message, or `None` for a user-defined link (type 65-255).
+///
+/// A user-defined link's target is only meaningful to the application that
+/// registered its class, so, like libhdf5 without that class, we cannot
+/// follow it. Leaving it out lets the rest of the group be listed and
+/// resolved instead of one such link failing the whole group; reserved
+/// types (2-63) are still an error.
+fn parse_link(data: &[u8], offset_size: u8) -> Result<Option<LinkMessage>, FormatError> {
+    match LinkMessage::parse(data, offset_size) {
+        Ok(link) => Ok(Some(link)),
+        Err(FormatError::InvalidLinkType(t)) if t >= FIRST_USER_DEFINED_LINK_TYPE => Ok(None),
+        Err(e) => Err(e),
+    }
+}
+
 /// Extract link entries from Link messages directly in the object header (compact storage).
 fn resolve_compact_entries(
     object_header: &ObjectHeader,
@@ -46,7 +64,9 @@ fn resolve_compact_entries(
     let mut entries = Vec::new();
     for msg in &object_header.messages {
         if msg.msg_type == MessageType::Link {
-            let link = LinkMessage::parse(&msg.data, offset_size)?;
+            let Some(link) = parse_link(&msg.data, offset_size)? else {
+                continue;
+            };
             if let LinkTarget::Hard {
                 object_header_address,
             } = link.link_target
@@ -98,7 +118,9 @@ fn for_each_dense_link(
 
         // Read managed object from fractal heap
         let link_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
-        visit(LinkMessage::parse(&link_data, offset_size)?);
+        if let Some(link) = parse_link(&link_data, offset_size)? {
+            visit(link);
+        }
     }
     Ok(())
 }
@@ -178,7 +200,9 @@ fn find_symbolic_link(
     } else {
         for msg in &object_header.messages {
             if msg.msg_type == MessageType::Link {
-                let link = LinkMessage::parse(&msg.data, offset_size)?;
+                let Some(link) = parse_link(&msg.data, offset_size)? else {
+                    continue;
+                };
                 if link.name == name && is_symbolic(&link.link_target) {
                     found = Some(link.link_target);
                 }
@@ -231,32 +255,135 @@ pub fn resolve_path_any(
     superblock: &Superblock,
     path: &str,
 ) -> Result<u64, FormatError> {
-    resolve_path_following_links(file_data, superblock, path, 0)
+    resolve_path_following_links(
+        file_data,
+        superblock,
+        superblock.root_group_address,
+        path,
+        0,
+    )
+}
+
+/// Resolve `path` relative to the group at `group_address` (an absolute path
+/// starts at the root group instead), following soft links. This is how a
+/// relative soft link's target is resolved: from the group holding the link.
+pub fn resolve_path_from(
+    file_data: &[u8],
+    superblock: &Superblock,
+    group_address: u64,
+    path: &str,
+) -> Result<u64, FormatError> {
+    let start = if path.starts_with('/') {
+        superblock.root_group_address
+    } else {
+        group_address
+    };
+    resolve_path_following_links(file_data, superblock, start, path, 0)
+}
+
+/// The children of the group at `group_address` that can be opened, as h5py
+/// lists them: hard links, and soft links resolved to the object they point
+/// at (under the soft link's own name). Links that cannot be followed are
+/// left out rather than failing the listing — a dangling or cyclic soft link
+/// (h5py lists its name but cannot open it), an external link (another
+/// file), and a user-defined link. An object header that is not a group has
+/// no children.
+///
+/// Any other error, such as a corrupt structure met while resolving a soft
+/// link, is returned.
+pub fn resolve_group_children(
+    file_data: &[u8],
+    superblock: &Superblock,
+    group_address: u64,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    let os = superblock.offset_size;
+    let ls = superblock.length_size;
+    let header = ObjectHeader::parse(file_data, group_address as usize, os, ls)?;
+
+    let mut entries = Vec::new();
+    let mut soft = Vec::new();
+    if is_v1_group(&header) {
+        let sym_msg = header
+            .messages
+            .iter()
+            .find(|m| m.msg_type == MessageType::SymbolTable)
+            .ok_or_else(|| FormatError::PathNotFound(String::from("no symbol table message")))?;
+        let stm = SymbolTableMessage::parse(&sym_msg.data, os)?;
+        let all = group_v1::resolve_v1_group_entries(file_data, &stm, os, ls)?;
+        if all.iter().any(group_v1::is_v1_soft_link) {
+            soft = group_v1::v1_soft_links(file_data, &stm, os, ls)?;
+        }
+        entries.extend(all.into_iter().filter(|e| !group_v1::is_v1_soft_link(e)));
+    } else if is_v2_group(&header) {
+        let mut visit = |link: LinkMessage| match link.link_target {
+            LinkTarget::Hard {
+                object_header_address,
+            } => entries.push(GroupEntry {
+                name: link.name,
+                object_header_address,
+                cache_type: 0,
+            }),
+            LinkTarget::Soft { target_path } => soft.push((link.name, target_path)),
+            LinkTarget::External { .. } => {}
+        };
+        let link_info = find_link_info(&header, os)?;
+        if let Some(fh_addr) = link_info.fractal_heap_address {
+            for_each_dense_link(file_data, &link_info, fh_addr, os, ls, visit)?;
+        } else {
+            for msg in &header.messages {
+                if msg.msg_type == MessageType::Link
+                    && let Some(link) = parse_link(&msg.data, os)?
+                {
+                    visit(link);
+                }
+            }
+        }
+    }
+
+    for (name, target) in soft {
+        match resolve_path_from(file_data, superblock, group_address, &target) {
+            Ok(object_header_address) => entries.push(GroupEntry {
+                name,
+                object_header_address,
+                cache_type: 0,
+            }),
+            // Dangling, cyclic, or ending in another file: not openable here.
+            Err(
+                FormatError::PathNotFound(_)
+                | FormatError::NestingDepthExceeded
+                | FormatError::ExternalLinkUnsupported { .. },
+            ) => {}
+            Err(e) => return Err(e),
+        }
+    }
+    Ok(entries)
 }
 
 /// Soft links followed while resolving one path. Guards against link cycles
 /// (`a -> b -> a`), which are legal to create.
 const MAX_SOFT_LINK_DEPTH: u8 = 16;
 
+/// Walk `path` from the group at `start`, following soft links.
 fn resolve_path_following_links(
     file_data: &[u8],
     superblock: &Superblock,
+    start: u64,
     path: &str,
     depth: u8,
 ) -> Result<u64, FormatError> {
-    let components: Vec<&str> = path.split('/').filter(|s| !s.is_empty()).collect();
+    let components: Vec<&str> = path
+        .split('/')
+        .filter(|s| !s.is_empty() && *s != ".")
+        .collect();
     if components.is_empty() {
-        return Ok(superblock.root_group_address);
+        return Ok(start);
     }
 
     let os = superblock.offset_size;
     let ls = superblock.length_size;
 
-    let root_header =
-        ObjectHeader::parse(file_data, superblock.root_group_address as usize, os, ls)?;
-
-    let mut current_addr = superblock.root_group_address;
-    let mut current_header = root_header;
+    let mut current_addr = start;
+    let mut current_header = ObjectHeader::parse(file_data, start as usize, os, ls)?;
 
     for (i, component) in components.iter().enumerate() {
         let entries = resolve_group_entries(file_data, &current_header, os, ls)?;
@@ -280,20 +407,17 @@ fn resolve_path_following_links(
                         }
                         // A relative target is relative to the group holding
                         // the link; then the rest of the original path.
-                        let mut full = String::new();
-                        if !target_path.starts_with('/') {
-                            for parent in &components[..i] {
-                                full.push('/');
-                                full.push_str(parent);
-                            }
-                        }
-                        full.push('/');
-                        full.push_str(&target_path);
+                        let from = if target_path.starts_with('/') {
+                            superblock.root_group_address
+                        } else {
+                            current_addr
+                        };
+                        let mut full = target_path;
                         for rest in &components[i + 1..] {
                             full.push('/');
                             full.push_str(rest);
                         }
-                        resolve_path_following_links(file_data, superblock, &full, depth + 1)
+                        resolve_path_following_links(file_data, superblock, from, &full, depth + 1)
                     }
                     Some(LinkTarget::External {
                         filename,

crates/clawhdf5-format/src/lib.rs

@@ -26,12 +26,13 @@
 //! use clawhdf5_format::{signature, superblock, object_header, group_v2,
 //!     datatype, dataspace, data_layout, data_read, message_type::MessageType};
 //!
-//! let file_data = std::fs::read("output.h5").unwrap();
-//! let sig = signature::find_signature(&file_data).unwrap();
-//! let sb  = superblock::Superblock::parse(&file_data, sig).unwrap();
-//! let addr = group_v2::resolve_path_any(&file_data, &sb, "data").unwrap();
+//! let bytes = std::fs::read("output.h5").unwrap();
+//! // Addresses are relative to the superblock: skip any user block.
+//! let (_user_block, file_data) = signature::split_user_block(&bytes).unwrap();
+//! let sb  = superblock::Superblock::parse(file_data, 0).unwrap();
+//! let addr = group_v2::resolve_path_any(file_data, &sb, "data").unwrap();
 //! let hdr = object_header::ObjectHeader::parse(
-//!     &file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
+//!     file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
 //! ```
 //!
 //! # Features
@@ -100,6 +101,7 @@ pub mod signature;
 pub mod superblock;
 pub mod symbol_table;
 pub mod type_builders;
+pub mod vds;
 pub mod vl_data;
 
 #[cfg(feature = "provenance")]

crates/clawhdf5-format/src/local_heap.rs

@@ -87,6 +87,57 @@ impl LocalHeap {
         })
     }
 
+    /// Walk the free list the way libhdf5 does when it loads a heap's data
+    /// (`H5HL__fl_deserialize`), rejecting a heap whose free list points
+    /// outside the data segment. libhdf5 refuses such a heap ("bad heap free
+    /// list"), and names read from it would be garbage.
+    ///
+    /// libhdf5 only loads a heap when it needs a name from it (an empty
+    /// group's broken heap goes unnoticed), so call this before the first
+    /// [`Self::read_string`], not on parse.
+    ///
+    /// The end of the list is `H5HL_FREE_NULL` (1); an all-ones value (the
+    /// undefined address) is accepted as "no free list" too.
+    pub fn validate_free_list(&self, file_data: &[u8], length_size: u8) -> Result<(), FormatError> {
+        const FREE_NULL: u64 = 1;
+        let ls = length_size as usize;
+        let undefined = if ls >= 8 {
+            u64::MAX
+        } else {
+            (1u64 << (8 * ls)) - 1
+        };
+        let size = self.data_segment_size;
+        let seg = self.data_segment_address;
+        let mut next = self.free_list_head_offset;
+        // Each free block holds two lengths, so a list longer than this
+        // revisits a block: a cycle.
+        let max_blocks = size / (2 * ls as u64) + 1;
+        let mut walked = 0u64;
+        while next != FREE_NULL && next != undefined {
+            if next >= size || walked >= max_blocks {
+                return Err(FormatError::InvalidLocalHeapFreeList);
+            }
+            walked += 1;
+            let at = seg
+                .checked_add(next)
+                .and_then(|a| usize::try_from(a).ok())
+                .ok_or(FormatError::InvalidLocalHeapFreeList)?;
+            let block_offset = next;
+            next = read_offset(file_data, at, length_size)?;
+            if next == 0 {
+                return Err(FormatError::InvalidLocalHeapFreeList);
+            }
+            let block_size = read_offset(file_data, at + ls, length_size)?;
+            if block_offset
+                .checked_add(block_size)
+                .is_none_or(|end| end > size)
+            {
+                return Err(FormatError::InvalidLocalHeapFreeList);
+            }
+        }
+        Ok(())
+    }
+
     /// Read a null-terminated string from the heap's data segment at the given byte offset.
     pub fn read_string(&self, file_data: &[u8], string_offset: u64) -> Result<String, FormatError> {
         let seg_addr = self.data_segment_address as usize;
@@ -162,8 +213,8 @@ mod tests {
         // data_segment_size
         write_val(&mut file, pos, data_seg_size as u64, length_size);
         pos += length_size as usize;
-        // free_list_head_offset
-        write_val(&mut file, pos, 0xFFFFFFFF, length_size);
+        // free_list_head_offset: H5HL_FREE_NULL (no free space)
+        write_val(&mut file, pos, 1, length_size);
         pos += length_size as usize;
         // data_segment_address
         write_val(&mut file, pos, data_seg_offset as u64, offset_size);
@@ -243,6 +294,50 @@ mod tests {
         assert_eq!(s, "test");
     }
 
+    /// Heap with data segment `[a, b, c, 0-padding]` whose free list starts
+    /// at `head` and has one block `(next, size)` at offset 8.
+    fn heap_with_free_block(head: u64, next: u64, size: u64) -> Vec<u8> {
+        let mut file = build_heap_file(0, 100, &["abcdefg"], 8, 8);
+        file.resize(200, 0);
+        write_val(&mut file, 8, 32, 8); // data segment size
+        write_val(&mut file, 16, head, 8);
+        write_val(&mut file, 108, next, 8);
+        write_val(&mut file, 116, size, 8);
+        file
+    }
+
+    #[test]
+    fn free_list_inside_the_segment_is_accepted() {
+        let file = heap_with_free_block(8, 1, 24);
+        let heap = LocalHeap::parse(&file, 0, 8, 8).unwrap();
+        heap.validate_free_list(&file, 8).unwrap();
+        assert_eq!(heap.read_string(&file, 0).unwrap(), "abcdefg");
+        // An all-ones head is "no free list" too.
+        let file = heap_with_free_block(u64::MAX, 0, 0);
+        let heap = LocalHeap::parse(&file, 0, 8, 8).unwrap();
+        assert!(heap.validate_free_list(&file, 8).is_ok());
+    }
+
+    #[test]
+    fn bad_free_list_is_rejected_like_libhdf5() {
+        for (head, next, size, why) in [
+            (40, 1, 8, "head past the segment"),
+            (8, 1, 25, "block runs past the segment"),
+            (8, 0, 8, "next offset of zero"),
+            (8, 8, 8, "cycle"),
+            (8, 999, 8, "next past the segment"),
+        ] {
+            let file = heap_with_free_block(head, next, size);
+            // The header itself parses; the free list is checked on use.
+            let heap = LocalHeap::parse(&file, 0, 8, 8).unwrap();
+            assert_eq!(
+                heap.validate_free_list(&file, 8).unwrap_err(),
+                FormatError::InvalidLocalHeapFreeList,
+                "{why}"
+            );
+        }
+    }
+
     #[test]
     fn invalid_version() {
         let mut file = build_heap_file(0, 100, &["x"], 8, 8);

crates/clawhdf5-format/src/selection.rs

@@ -229,44 +229,47 @@ impl Selection {
     /// self-describing in length, so the count lets a caller walk a packed list
     /// of selections — as the Virtual Dataset global-heap block does).
     ///
-    /// Only the forms needed for VDS assembly are decoded: `ALL`, `NONE`, and
-    /// **regular** hyperslabs serialized at **version 3** (the encoding HDF5
-    /// 1.10+/2.0 emit). Point selections, irregular hyperslabs, and older
-    /// hyperslab versions return an error rather than mis-decoding.
+    /// Decodes `ALL`, `NONE`, and hyperslabs at every version libhdf5 writes
+    /// (1: irregular, 4-byte coordinates — the default-format encoding; 2:
+    /// regular, 8-byte; 3: either, variable width). A regular hyperslab maps
+    /// to [`Selection::Hyperslab`]; an *irregular* one (a union of blocks)
+    /// maps to a single-block hyperslab when it has one block, and otherwise to
+    /// [`Selection::Points`] listing the union in row-major order (the order
+    /// libhdf5 iterates it in). Unlimited counts/blocks decode as `u64::MAX`
+    /// (see [`SerializedSelection::decode`] for the raw form). Point
+    /// selections are refused: libhdf5 does not allow them in virtual datasets
+    /// either.
     pub fn decode_serialized(data: &[u8]) -> Result<(Selection, usize), FormatError> {
-        if data.len() < 8 {
-            return Err(FormatError::UnexpectedEof {
-                expected: 8,
-                available: data.len(),
-            });
-        }
-        let sel_type = u32::from_le_bytes([data[0], data[1], data[2], data[3]]);
-        let version = u32::from_le_bytes([data[4], data[5], data[6], data[7]]);
-
-        match sel_type {
-            // ALL / NONE: type(4) + version(4) + reserved(4) + length(4) = 16 bytes.
-            3 | 0 => {
-                if data.len() < 16 {
-                    return Err(FormatError::UnexpectedEof {
-                        expected: 16,
-                        available: data.len(),
-                    });
-                }
-                let sel = if sel_type == 3 {
-                    Selection::All
+        let (raw, len) = SerializedSelection::decode(data)?;
+        let sel = match raw {
+            SerializedSelection::All => Selection::All,
+            SerializedSelection::None => Selection::None,
+            SerializedSelection::Regular {
+                start,
+                stride,
+                count,
+                block,
+            } => Selection::Hyperslab {
+                start,
+                stride,
+                count,
+                block,
+            },
+            SerializedSelection::Blocks { rank, starts, ends } => {
+                if starts.len() == rank {
+                    let block = starts.iter().zip(&ends).map(|(&s, &e)| e - s + 1).collect();
+                    Selection::Hyperslab {
+                        start: starts,
+                        stride: vec![1; rank],
+                        count: vec![1; rank],
+                        block,
+                    }
                 } else {
-                    Selection::None
-                };
-                Ok((sel, 16))
+                    Selection::Points(blocks_union_coords(rank, &starts, &ends)?)
+                }
             }
-            2 => decode_hyperslab_serialized(data, version),
-            1 => Err(FormatError::ChunkedReadError(
-                "VDS point selections are not supported".into(),
-            )),
-            _ => Err(FormatError::ChunkedReadError(
-                "unknown dataspace selection type".into(),
-            )),
-        }
+        };
+        Ok((sel, len))
     }
 
     /// Enumerate the selected element indices of a **1-D** dataspace of the
@@ -314,6 +317,11 @@ impl Selection {
                         "VDS selection rank does not match dataspace rank".into(),
                     ));
                 }
+                if count.iter().chain(block.iter()).any(|&v| v == UNLIMITED) {
+                    return Err(FormatError::ChunkedReadError(
+                        "unlimited selection must be clipped before it is enumerated".into(),
+                    ));
+                }
                 // Selected coordinates along each dimension, in order.
                 let mut per_dim: Vec<Vec<u64>> = Vec::with_capacity(rank);
                 for d in 0..rank {
@@ -400,84 +408,279 @@ impl Selection {
     }
 }
 
-/// Decode an `H5S_SEL_HYPER` selection in its serialized form. Only version-3
-/// **regular** hyperslabs are supported.
-fn decode_hyperslab_serialized(
-    data: &[u8],
-    version: u32,
-) -> Result<(Selection, usize), FormatError> {
-    if version != 3 {
-        return Err(FormatError::ChunkedReadError(
-            "only version-3 hyperslab selections are supported".into(),
-        ));
+/// Hyperslab count/block value meaning "unlimited" (`H5S_UNLIMITED`).
+pub const UNLIMITED: u64 = u64::MAX;
+
+/// Largest number of elements an irregular selection is expanded to when it
+/// is converted to a point list by [`Selection::decode_serialized`].
+const MAX_EXPANDED_POINTS: u64 = 1 << 26;
+
+/// A selection exactly as `H5S_select_serialize` stores it, before it is
+/// applied to any dataspace.
+///
+/// Unlike [`Selection`] this keeps an irregular hyperslab as its list of
+/// blocks, and a regular hyperslab's count/block may be [`UNLIMITED`] (the
+/// unlimited selections used by unlimited and "printf" virtual dataset
+/// mappings).
+#[derive(Debug, Clone, PartialEq)]
+pub enum SerializedSelection {
+    /// `H5S_SEL_ALL`.
+    All,
+    /// `H5S_SEL_NONE`.
+    None,
+    /// A regular hyperslab. `count[d]` or `block[d]` may be [`UNLIMITED`].
+    Regular {
+        start: Vec<u64>,
+        stride: Vec<u64>,
+        count: Vec<u64>,
+        block: Vec<u64>,
+    },
+    /// An irregular hyperslab: the union of `starts.len() / rank` blocks, each
+    /// given by its first (`starts`) and last (`ends`, inclusive) coordinate,
+    /// flattened block-major.
+    Blocks {
+        rank: usize,
+        starts: Vec<u64>,
+        ends: Vec<u64>,
+    },
+}
+
+fn sel_err(msg: &str) -> FormatError {
+    FormatError::ChunkedReadError(msg.into())
+}
+
+/// Bounds-checked little-endian reader over a serialized selection.
+struct SelReader<'a> {
+    data: &'a [u8],
+    pos: usize,
+}
+
+impl SelReader<'_> {
+    fn take(&mut self, n: usize) -> Result<&[u8], FormatError> {
+        let end = self.pos.checked_add(n).filter(|&e| e <= self.data.len());
+        let end = end.ok_or(FormatError::UnexpectedEof {
+            expected: self.pos.saturating_add(n),
+            available: self.data.len(),
+        })?;
+        let s = &self.data[self.pos..end];
+        self.pos = end;
+        Ok(s)
     }
-    // type(4) ver(4) flags(1) enc_size(1) rank(4) [start,stride,count,block]*rank
-    if data.len() < 14 {
-        return Err(FormatError::UnexpectedEof {
-            expected: 14,
-            available: data.len(),
-        });
+
+    fn uint(&mut self, size: usize) -> Result<u64, FormatError> {
+        let bytes = self.take(size)?;
+        Ok(bytes
+            .iter()
+            .enumerate()
+            .fold(0u64, |v, (i, &b)| v | (b as u64) << (i * 8)))
     }
-    let flags = data[8];
-    let enc_size = data[9] as usize;
-    // Bit 0 set => regular hyperslab. Irregular hyperslabs list explicit blocks.
-    if flags & 0x01 == 0 {
-        return Err(FormatError::ChunkedReadError(
-            "irregular VDS hyperslab selections are not supported".into(),
-        ));
+
+    fn remaining(&self) -> usize {
+        self.data.len() - self.pos
     }
-    if enc_size != 2 && enc_size != 4 && enc_size != 8 {
-        return Err(FormatError::ChunkedReadError(
-            "unsupported hyperslab coordinate encoding size".into(),
-        ));
-    }
-    let rank = u32::from_le_bytes([data[10], data[11], data[12], data[13]]) as usize;
-    // HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything larger so a
-    // corrupt rank can't drive a huge allocation or read loop.
-    if rank > 32 {
-        return Err(FormatError::ChunkedReadError(
-            "hyperslab selection rank exceeds maximum (32)".into(),
-        ));
-    }
-    let mut pos = 14;
-    let read_coord = |data: &[u8], pos: usize| -> Result<u64, FormatError> {
-        if pos + enc_size > data.len() {
-            return Err(FormatError::UnexpectedEof {
-                expected: pos + enc_size,
-                available: data.len(),
-            });
+}
+
+impl SerializedSelection {
+    /// Decode a serialized selection, returning it and the number of bytes it
+    /// occupies. Mirrors libhdf5's `H5S_select_deserialize`: `ALL`/`NONE` and
+    /// hyperslab versions 1-3 are decoded; point selections (which libhdf5
+    /// refuses in virtual datasets) and malformed input are errors.
+    pub fn decode(data: &[u8]) -> Result<(SerializedSelection, usize), FormatError> {
+        let mut r = SelReader { data, pos: 0 };
+        let sel_type = r.uint(4)?;
+        let version = r.uint(4)?;
+        match sel_type {
+            // ALL / NONE: type(4) + version(4) + reserved(4) + length(4).
+            0 | 3 => {
+                r.take(8)?;
+                let sel = if sel_type == 3 {
+                    SerializedSelection::All
+                } else {
+                    SerializedSelection::None
+                };
+                Ok((sel, r.pos))
+            }
+            2 => {
+                let sel = decode_hyperslab(&mut r, version)?;
+                Ok((sel, r.pos))
+            }
+            1 => Err(sel_err(
+                "VDS point selections are not supported (libhdf5 rejects them too)",
+            )),
+            _ => Err(sel_err("unknown dataspace selection type")),
         }
-        let mut v = 0u64;
-        for (i, &b) in data[pos..pos + enc_size].iter().enumerate() {
-            v |= (b as u64) << (i * 8);
+    }
+
+    /// The single dimension in which this selection is unlimited, if any.
+    pub fn unlimited_dim(&self) -> Option<usize> {
+        match self {
+            SerializedSelection::Regular { count, block, .. } => count
+                .iter()
+                .zip(block)
+                .position(|(&c, &b)| c == UNLIMITED || b == UNLIMITED),
+            _ => None,
         }
-        Ok(v)
+    }
+
+    /// The rank the selection was serialized with (`None` for ALL/NONE, which
+    /// carry no rank).
+    pub fn rank(&self) -> Option<usize> {
+        match self {
+            SerializedSelection::Regular { start, .. } => Some(start.len()),
+            SerializedSelection::Blocks { rank, .. } => Some(*rank),
+            _ => None,
+        }
+    }
+}
+
+/// `H5S__hyper_deserialize`: after the type and version words.
+fn decode_hyperslab(r: &mut SelReader, version: u64) -> Result<SerializedSelection, FormatError> {
+    const REGULAR: u8 = 0x01;
+    let (flags, enc_size) = match version {
+        // v1: reserved(4) + length(4), always irregular, 4-byte coordinates.
+        1 => {
+            r.take(8)?;
+            (0u8, 4usize)
+        }
+        // v2: flags(1) + length(4), 8-byte coordinates.
+        2 => {
+            let flags = r.take(1)?[0];
+            r.take(4)?;
+            (flags, 8)
+        }
+        // v3: flags(1) + encoding size(1).
+        3 => {
+            let flags = r.take(1)?[0];
+            let enc = r.take(1)?[0] as usize;
+            (flags, enc)
+        }
+        _ => return Err(sel_err("unsupported hyperslab selection version")),
     };
-    let (mut start, mut stride, mut count, mut block) = (
-        Vec::with_capacity(rank),
-        Vec::with_capacity(rank),
-        Vec::with_capacity(rank),
-        Vec::with_capacity(rank),
-    );
-    for _ in 0..rank {
-        start.push(read_coord(data, pos)?);
-        pos += enc_size;
-        stride.push(read_coord(data, pos)?);
-        pos += enc_size;
-        count.push(read_coord(data, pos)?);
-        pos += enc_size;
-        block.push(read_coord(data, pos)?);
-        pos += enc_size;
+    if flags & !REGULAR != 0 {
+        return Err(sel_err("unknown hyperslab selection flags"));
     }
-    Ok((
-        Selection::Hyperslab {
+    if !matches!(enc_size, 2 | 4 | 8) {
+        return Err(sel_err("unsupported hyperslab coordinate encoding size"));
+    }
+    let rank = r.uint(4)? as usize;
+    // HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything else so a
+    // corrupt rank can't drive a huge allocation or read loop.
+    if rank == 0 || rank > 32 {
+        return Err(sel_err("hyperslab selection rank must be 1..=32"));
+    }
+    // The all-ones value of the encoding width means "unlimited".
+    let unlim_raw = if enc_size == 8 {
+        u64::MAX
+    } else {
+        (1u64 << (enc_size * 8)) - 1
+    };
+
+    if flags & REGULAR != 0 {
+        let (mut start, mut stride, mut count, mut block) = (
+            Vec::with_capacity(rank),
+            Vec::with_capacity(rank),
+            Vec::with_capacity(rank),
+            Vec::with_capacity(rank),
+        );
+        for _ in 0..rank {
+            start.push(r.uint(enc_size)?);
+            stride.push(r.uint(enc_size)?);
+            let c = r.uint(enc_size)?;
+            count.push(if c == unlim_raw { UNLIMITED } else { c });
+            let b = r.uint(enc_size)?;
+            block.push(if b == unlim_raw { UNLIMITED } else { b });
+        }
+        let unlimited = count
+            .iter()
+            .zip(&block)
+            .filter(|&(&c, &b)| c == UNLIMITED || b == UNLIMITED)
+            .count();
+        if unlimited > 1 {
+            return Err(sel_err(
+                "hyperslab selection is unlimited in more than one dimension",
+            ));
+        }
+        for d in 0..rank {
+            // Overlapping blocks are not a valid regular hyperslab.
+            if count[d] > 1 && block[d] != UNLIMITED && block[d] > stride[d] {
+                return Err(sel_err("regular hyperslab blocks overlap"));
+            }
+        }
+        return Ok(SerializedSelection::Regular {
             start,
             stride,
             count,
             block,
-        },
-        pos,
-    ))
+        });
+    }
+
+    // Irregular: number of blocks, then each block's start and end corners.
+    let nblocks = r.uint(enc_size)?;
+    let per_block = (rank * 2 * enc_size) as u64;
+    // Untrusted count: it must fit in what is left of the buffer.
+    if nblocks
+        .checked_mul(per_block)
+        .is_none_or(|need| need > r.remaining() as u64)
+    {
+        return Err(FormatError::UnexpectedEof {
+            expected: r
+                .pos
+                .saturating_add(nblocks.saturating_mul(per_block) as usize),
+            available: r.data.len(),
+        });
+    }
+    let n = nblocks as usize * rank;
+    let (mut starts, mut ends) = (Vec::with_capacity(n), Vec::with_capacity(n));
+    for _ in 0..nblocks {
+        for _ in 0..rank {
+            starts.push(r.uint(enc_size)?);
+        }
+        for _ in 0..rank {
+            ends.push(r.uint(enc_size)?);
+        }
+    }
+    if starts.iter().zip(&ends).any(|(s, e)| e < s) {
+        return Err(sel_err("hyperslab block ends before it starts"));
+    }
+    Ok(SerializedSelection::Blocks { rank, starts, ends })
+}
+
+/// The coordinates of the union of the given blocks, in row-major order.
+fn blocks_union_coords(
+    rank: usize,
+    starts: &[u64],
+    ends: &[u64],
+) -> Result<Vec<Vec<u64>>, FormatError> {
+    let mut total = 0u64;
+    for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
+        let vol = s
+            .iter()
+            .zip(e)
+            .try_fold(1u64, |acc, (&s, &e)| acc.checked_mul(e - s + 1));
+        total = vol
+            .and_then(|v| total.checked_add(v))
+            .filter(|&t| t <= MAX_EXPANDED_POINTS)
+            .ok_or_else(|| sel_err("irregular hyperslab selection is too large to expand"))?;
+    }
+    let mut out = Vec::with_capacity(total as usize);
+    for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
+        let mut cur = s.to_vec();
+        'block: loop {
+            out.push(cur.clone());
+            for d in (0..rank).rev() {
+                if cur[d] < e[d] {
+                    cur[d] += 1;
+                    continue 'block;
+                }
+                cur[d] = s[d];
+            }
+            break;
+        }
+    }
+    // Lexicographic order of coordinates is row-major order.
+    out.sort_unstable();
+    out.dedup();
+    Ok(out)
 }
 
 // ---------------------------------------------------------------------------
@@ -642,11 +845,100 @@ mod tests {
     }
 
     #[test]
-    fn decode_irregular_hyperslab_rejected() {
+    fn decode_truncated_irregular_hyperslab_is_error() {
+        // Irregular, rank 1, but the block count is missing.
         let bytes = [0x02u8, 0, 0, 0, 0x03, 0, 0, 0, 0x00, 0x02, 0x01, 0, 0, 0];
         assert!(Selection::decode_serialized(&bytes).is_err());
     }
 
+    /// Version 1 as libhdf5 writes it for the default (earliest) format bounds:
+    /// type, version, reserved(4), length(4), rank(4), nblocks(4), then each
+    /// block's start and inclusive end corner as 4-byte values.
+    fn v1_blocks(rank: u32, blocks: &[(&[u32], &[u32])]) -> Vec<u8> {
+        let mut b = Vec::new();
+        for w in [2u32, 1, 0, 0, rank, blocks.len() as u32] {
+            b.extend_from_slice(&w.to_le_bytes());
+        }
+        for (s, e) in blocks {
+            for v in s.iter().chain(e.iter()) {
+                b.extend_from_slice(&v.to_le_bytes());
+            }
+        }
+        b
+    }
+
+    #[test]
+    fn decode_v1_irregular_single_block() {
+        // Exactly what h5py/HDF5 2.0 writes for `[0:4]` with default libver.
+        let bytes = v1_blocks(1, &[(&[0], &[3])]);
+        let (sel, used) = Selection::decode_serialized(&bytes).unwrap();
+        assert_eq!(used, bytes.len());
+        assert_eq!(sel.iter_linear_1d(8).unwrap(), vec![0, 1, 2, 3]);
+    }
+
+    #[test]
+    fn decode_v1_irregular_union_is_row_major() {
+        // Blocks given out of order and overlapping still enumerate once each,
+        // in row-major order (libhdf5 iterates the union, not the list).
+        let bytes = v1_blocks(2, &[(&[1, 0], &[1, 1]), (&[0, 2], &[1, 2])]);
+        let (sel, used) = Selection::decode_serialized(&bytes).unwrap();
+        assert_eq!(used, bytes.len());
+        // (0,2) (1,0) (1,1) (1,2) in a 2x3 space.
+        assert_eq!(sel.iter_linear(&[2, 3]).unwrap(), vec![2, 3, 4, 5]);
+    }
+
+    #[test]
+    fn decode_v2_regular_with_unlimited_count() {
+        // v2: flags(1) + length(4), then 8-byte start/stride/count/block.
+        let mut b = Vec::new();
+        b.extend_from_slice(&2u32.to_le_bytes());
+        b.extend_from_slice(&2u32.to_le_bytes());
+        b.push(0x01);
+        b.extend_from_slice(&36u32.to_le_bytes());
+        b.extend_from_slice(&1u32.to_le_bytes());
+        for v in [0u64, 10, u64::MAX, 10] {
+            b.extend_from_slice(&v.to_le_bytes());
+        }
+        let (raw, used) = SerializedSelection::decode(&b).unwrap();
+        assert_eq!(used, b.len());
+        assert_eq!(raw.unlimited_dim(), Some(0));
+        assert_eq!(
+            raw,
+            SerializedSelection::Regular {
+                start: vec![0],
+                stride: vec![10],
+                count: vec![UNLIMITED],
+                block: vec![10],
+            }
+        );
+        // An unclipped unlimited selection cannot be enumerated.
+        let (sel, _) = Selection::decode_serialized(&b).unwrap();
+        assert!(sel.iter_linear_1d(100).is_err());
+    }
+
+    #[test]
+    fn decode_v3_two_byte_all_ones_is_unlimited() {
+        let bytes = [
+            0x02, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0x01, 0, 0, 0, //
+            0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0xFF, 0xFF,
+        ];
+        let (raw, _) = SerializedSelection::decode(&bytes).unwrap();
+        assert_eq!(raw.unlimited_dim(), Some(0));
+    }
+
+    #[test]
+    fn decode_irregular_block_count_beyond_buffer_is_error() {
+        let mut b = v1_blocks(1, &[(&[0], &[3])]);
+        b[20..24].copy_from_slice(&u32::MAX.to_le_bytes());
+        assert!(Selection::decode_serialized(&b).is_err());
+    }
+
+    #[test]
+    fn decode_point_selection_is_refused() {
+        let bytes = [1u8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
+        assert!(Selection::decode_serialized(&bytes).is_err());
+    }
+
     #[test]
     fn iter_linear_2d_block_row_major() {
         // A 2x2 block at the top-left of a 4x4 space => linear 0,1,4,5.

crates/clawhdf5-format/src/shared_message.rs

@@ -154,13 +154,29 @@ pub fn is_shared(msg_flags: u8) -> bool {
 ///
 /// When the shared flag is set on a message, the data contains a reference
 /// instead of the actual message content.
+///
+/// Assumes the file's length size equals its offset size, which only matters
+/// for version-1 references; use [`parse_shared_ref_sized`] when the
+/// superblock's length size is known.
 pub fn parse_shared_ref(data: &[u8], offset_size: u8) -> Result<SharedMessageRef, FormatError> {
+    parse_shared_ref_sized(data, offset_size, offset_size)
+}
+
+/// [`parse_shared_ref`] with the superblock's length size, which locates the
+/// object header address in a version-1 reference.
+pub fn parse_shared_ref_sized(
+    data: &[u8],
+    offset_size: u8,
+    length_size: u8,
+) -> Result<SharedMessageRef, FormatError> {
     ensure_len(data, 0, 2)?;
     let version = data[0];
     let ref_type = data[1];
 
     // Layouts (HDF5 spec IV.A.2 "Shared Message", and libhdf5's decoder):
-    //   v1: version, type, reserved(6), address     — always "committed"
+    //   v1: version, type, reserved(6), then an old-style symbol table
+    //       entry: link-name offset(length_size), object header address,
+    //       cache type(4), reserved(4), scratch(16)  — always "committed"
     //   v2: version, type, address                  — always "committed"
     //   v3: version, type, then a fractal-heap ID if type == SOHM, otherwise
     //       an address
@@ -177,7 +193,7 @@ pub fn parse_shared_ref(data: &[u8], offset_size: u8) -> Result<SharedMessageRef
         })
     };
     match version {
-        1 => address_at(2 + 6),
+        1 => address_at(2 + 6 + length_size as usize),
         2 => address_at(2),
         3 if ref_type == SHARE_TYPE_SOHM => {
             ensure_len(data, 2, FHEAP_ID_LEN)?;
@@ -434,7 +450,7 @@ pub fn message_data_with_sohm<'a>(
     if !is_shared(msg.flags) {
         return Ok(Cow::Borrowed(&msg.data));
     }
-    let shared_ref = parse_shared_ref(&msg.data, offset_size)?;
+    let shared_ref = parse_shared_ref_sized(&msg.data, offset_size, length_size)?;
     let table = if shared_ref.heap_id.is_some() {
         load_sohm_table(file_data, offset_size, length_size)?
     } else {
@@ -514,7 +530,7 @@ pub fn message_data<'a>(
     if !is_shared(msg.flags) {
         return Ok(Cow::Borrowed(&msg.data));
     }
-    let shared_ref = parse_shared_ref(&msg.data, offset_size)?;
+    let shared_ref = parse_shared_ref_sized(&msg.data, offset_size, length_size)?;
     resolve_shared_message(
         file_data,
         &shared_ref,
@@ -529,7 +545,8 @@ pub fn message_data<'a>(
 ///
 /// For type 1/3 (shared in another object header), reads the target object header
 /// and finds the message of the specified type.
-/// For type 2 (SOHM), uses the fractal heap from the SOHM table.
+/// For type 2 (SOHM), uses the fractal heap from the file's SOHM table,
+/// loaded from the superblock extension on demand.
 pub fn resolve_shared_message(
     file_data: &[u8],
     shared_ref: &SharedMessageRef,
@@ -537,13 +554,18 @@ pub fn resolve_shared_message(
     offset_size: u8,
     length_size: u8,
 ) -> Result<Vec<u8>, FormatError> {
+    let table = if shared_ref.heap_id.is_some() {
+        load_sohm_table(file_data, offset_size, length_size)?
+    } else {
+        None
+    };
     resolve_shared_message_with_sohm(
         file_data,
         shared_ref,
         target_msg_type,
         offset_size,
         length_size,
-        None,
+        table.as_ref(),
     )
 }
 
@@ -649,15 +671,26 @@ mod tests {
 
     #[test]
     fn parse_v1_ref() {
-        let mut data = Vec::new();
-        data.push(1); // version
-        data.push(0); // type
-        data.extend_from_slice(&[0u8; 6]); // reserved
-        data.extend_from_slice(&0x5678u64.to_le_bytes());
+        // Datatype message of `/group1/dset2` in HDF5's `tcompound.h5`
+        // (written in 2000): version 1, six reserved bytes, then an old-style
+        // symbol table entry — link-name offset 0x10, object header address
+        // 0x590 (the committed datatype `/type1`), cache type, reserved and
+        // scratch.
+        let mut data = vec![1, 0, 0, 0, 0, 0, 0, 0];
+        data.extend_from_slice(&0x10u64.to_le_bytes());
+        data.extend_from_slice(&0x590u64.to_le_bytes());
+        data.extend_from_slice(&[0; 24]);
 
-        let shared = parse_shared_ref(&data, 8).unwrap();
+        let shared = parse_shared_ref_sized(&data, 8, 8).unwrap();
         assert_eq!(shared.version, 1);
-        assert_eq!(shared.object_header_address, Some(0x5678));
+        assert_eq!(shared.object_header_address, Some(0x590));
+
+        // The name offset is a length: 4 bytes here, then an 8-byte address.
+        let mut data = vec![1, 0, 0, 0, 0, 0, 0, 0];
+        data.extend_from_slice(&0x10u32.to_le_bytes());
+        data.extend_from_slice(&0x590u64.to_le_bytes());
+        let shared = parse_shared_ref_sized(&data, 8, 4).unwrap();
+        assert_eq!(shared.object_header_address, Some(0x590));
     }
 
     #[test]

crates/clawhdf5-format/src/signature.rs

@@ -11,6 +11,16 @@ pub const HDF5_SIGNATURE: [u8; 8] = [0x89, b'H', b'D', b'F', b'\r', b'\n', 0x1A,
 /// (powers of two starting at 512, plus offset 0).
 ///
 /// Returns the byte offset where the signature was found.
+///
+/// A non-zero offset means the file starts with a *user block*, and every
+/// address inside the file is relative to the superblock's position, not to
+/// byte 0 (libhdf5 uses the signature's position as the base address even
+/// when the stored base-address field disagrees). The parsers in this crate
+/// take addresses as indices into `file_data`, so they must be handed the
+/// bytes from the signature on — use [`split_user_block`]. [`Superblock::parse`]
+/// refuses a non-zero offset for this reason.
+///
+/// [`Superblock::parse`]: crate::superblock::Superblock::parse
 pub fn find_signature(data: &[u8]) -> Result<usize, FormatError> {
     // Check offset 0
     if data.len() >= 8 && data[..8] == HDF5_SIGNATURE {
@@ -29,6 +39,17 @@ pub fn find_signature(data: &[u8]) -> Result<usize, FormatError> {
     Err(FormatError::SignatureNotFound)
 }
 
+/// Split a file into its user block and its HDF5 bytes.
+///
+/// Returns `(user_block, hdf5)`: `user_block` is everything before the
+/// superblock signature (empty for most files) and `hdf5` is the rest, in
+/// which every HDF5 address is a plain index. Pass `hdf5` as `file_data` to
+/// every parser in this crate, and parse the superblock at offset 0 of it.
+pub fn split_user_block(data: &[u8]) -> Result<(&[u8], &[u8]), FormatError> {
+    let offset = find_signature(data)?;
+    Ok(data.split_at(offset))
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -88,6 +109,21 @@ mod tests {
         assert_eq!(find_signature(&data), Err(FormatError::SignatureNotFound));
     }
 
+    #[test]
+    fn split_user_block_rebases_at_the_signature() {
+        let mut data = vec![7u8; 1024];
+        data[512..520].copy_from_slice(&HDF5_SIGNATURE);
+        let (ub, hdf5) = split_user_block(&data).unwrap();
+        assert_eq!(ub.len(), 512);
+        assert_eq!(hdf5.len(), 512);
+        assert_eq!(&hdf5[..8], &HDF5_SIGNATURE);
+
+        data[..8].copy_from_slice(&HDF5_SIGNATURE);
+        let (ub, hdf5) = split_user_block(&data).unwrap();
+        assert!(ub.is_empty());
+        assert_eq!(hdf5.len(), 1024);
+    }
+
     #[test]
     fn signature_prefers_earliest() {
         // Signature at both 0 and 512, should return 0

crates/clawhdf5-format/src/superblock.rs

@@ -174,8 +174,18 @@ impl Superblock {
 
     /// Parse a superblock from `data` starting at `signature_offset`.
     ///
-    /// The signature must be present at the given offset.
+    /// The signature must be present at the given offset, and that offset
+    /// must be 0: every address in an HDF5 file is relative to the
+    /// superblock, so when a file has a user block (signature at 512, 1024,
+    /// …) the caller must pass the bytes from the signature on — see
+    /// [`crate::signature::split_user_block`] — and use that slice as
+    /// `file_data` everywhere. A non-zero offset is refused with
+    /// [`FormatError::UserBlockNotStripped`] because the addresses in the
+    /// returned superblock would otherwise be applied to the wrong bytes.
     pub fn parse(data: &[u8], signature_offset: usize) -> Result<Superblock, FormatError> {
+        if signature_offset != 0 {
+            return Err(FormatError::UserBlockNotStripped(signature_offset as u64));
+        }
         let d = data
             .get(signature_offset..)
             .ok_or(FormatError::UnexpectedEof {
@@ -676,7 +686,16 @@ mod tests {
         let mut data = vec![0u8; 1024];
         let v0 = build_v0_bytes(8);
         data[512..512 + v0.len()].copy_from_slice(&v0);
-        let sb = Superblock::parse(&data, 512).unwrap();
+        // Addresses are relative to the superblock, so parsing in place
+        // (where they would be applied to the whole buffer) is refused...
+        assert_eq!(
+            Superblock::parse(&data, 512),
+            Err(FormatError::UserBlockNotStripped(512))
+        );
+        // ...and the caller parses the bytes from the signature on.
+        let (ub, hdf5) = crate::signature::split_user_block(&data).unwrap();
+        assert_eq!(ub.len(), 512);
+        let sb = Superblock::parse(hdf5, 0).unwrap();
         assert_eq!(sb.version, 0);
         assert_eq!(sb.root_group_address, 96);
     }

crates/clawhdf5-format/tests/fixtures/legacy/README.md

@@ -0,0 +1,13 @@
+# Legacy (HDF5 1.4/1.6-era) fixtures
+
+Unmodified copies of the HDF Group's own test files from
+https://github.com/HDFGroup/hdf5 at a3cf1ea82cc7a66e50029a688121e1b105a7ce88
+(BSD-style license, see that repository's `LICENSE`). Current libraries cannot
+write these structures, so they are kept as files.
+
+| File | Upstream path | Exercises |
+|---|---|---|
+| `deflate.h5` | `test/testfiles/deflate.h5` | Data Layout message v1, chunked + deflate (v1 B-tree index) |
+| `h5ex_g_iterate.h5` | `HDF5Examples/C/H5G/h5ex_g_iterate.h5` | Data Layout message v2, contiguous; an unallocated dataset |
+| `tarrold.h5` | `test/testfiles/tarrold.h5` | Compound datatype v1 members with legacy array dimensions |
+| `tcompound.h5` | `tools/test/testfiles/tcompound.h5` | Version-1 shared messages (committed datatypes); compound v1 array members with data |

crates/clawhdf5-format/tests/integration_test.rs

@@ -83,6 +83,45 @@ fn read_chunked_dataset(file_data: &[u8], dataset_path: &str) -> (Vec<u8>, Datat
     (raw, datatype, dataspace)
 }
 
+/// Helper: read a virtual dataset with `vds::read_virtual_dataset`, giving it
+/// the dataset's own fill value (same-file sources only).
+fn read_virtual_fixture(file_data: &[u8], path: &str) -> (Vec<u8>, Datatype) {
+    let sig = find_signature(file_data).unwrap();
+    let sb = Superblock::parse(file_data, sig).unwrap();
+    let addr = resolve_path_any(file_data, &sb, path).unwrap();
+    let hdr =
+        ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
+    let msg = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t).unwrap();
+    let ds = Dataspace::parse(&msg(MessageType::Dataspace).data, sb.length_size).unwrap();
+    let (dt, _) = Datatype::parse(&msg(MessageType::Datatype).data).unwrap();
+    let layout = DataLayout::parse(
+        &msg(MessageType::DataLayout).data,
+        sb.offset_size,
+        sb.length_size,
+    )
+    .unwrap();
+    let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
+        file_data,
+        &hdr.messages,
+        sb.offset_size,
+        sb.length_size,
+    )
+    .unwrap();
+    let v = clawhdf5_format::vds::read_virtual_dataset(
+        file_data,
+        &layout,
+        &ds,
+        &dt,
+        fill.as_deref(),
+        sb.offset_size,
+        sb.length_size,
+        None,
+    )
+    .unwrap();
+    assert_eq!(v.dims, ds.dimensions);
+    (v.data, dt)
+}
+
 /// Helper: read any dataset (contiguous or chunked) as f64.
 fn read_dataset_f64_any(bytes: &[u8], path: &str) -> Vec<f64> {
     let sig = find_signature(bytes).unwrap();
@@ -672,7 +711,7 @@ fn v4_virtual_dataset_same_file_read() {
     //   virt[4:8]  <- (unmapped)                              => fill 0
     //   virt[8:12] <- src_b[0:4]  (ALL)                       => 20,21,22,23
     let file_data = include_bytes!("fixtures/vds_same_file.h5");
-    let (raw, datatype, _) = read_chunked_dataset(file_data, "virt");
+    let (raw, datatype) = read_virtual_fixture(file_data, "virt");
     let values = read_as_i32(&raw, &datatype).unwrap();
     assert_eq!(
         values,
@@ -681,6 +720,38 @@ fn v4_virtual_dataset_same_file_read() {
     );
 }
 
+#[test]
+fn v4_virtual_dataset_raw_api_refuses_to_guess_the_fill_value() {
+    // The raw read API has no fill value message, so a virtual dataset with an
+    // unmapped region is an error there instead of zeros that may be wrong.
+    let file_data = include_bytes!("fixtures/vds_same_file.h5");
+    let sig = find_signature(file_data).unwrap();
+    let sb = Superblock::parse(file_data, sig).unwrap();
+    let addr = resolve_path_any(file_data, &sb, "virt").unwrap();
+    let hdr =
+        ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
+    let msg = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t).unwrap();
+    let ds = Dataspace::parse(&msg(MessageType::Dataspace).data, sb.length_size).unwrap();
+    let (dt, _) = Datatype::parse(&msg(MessageType::Datatype).data).unwrap();
+    let layout = DataLayout::parse(
+        &msg(MessageType::DataLayout).data,
+        sb.offset_size,
+        sb.length_size,
+    )
+    .unwrap();
+    let err = read_raw_data_full(
+        file_data,
+        &layout,
+        &ds,
+        &dt,
+        None,
+        sb.offset_size,
+        sb.length_size,
+    )
+    .unwrap_err();
+    assert!(err.to_string().contains("fill value"), "{err}");
+}
+
 #[test]
 fn v4_virtual_dataset_2d_same_file_read() {
     // A 4x4 virtual dataset assembled from two 2x2 same-file sources placed as
@@ -689,7 +760,7 @@ fn v4_virtual_dataset_2d_same_file_read() {
     //   virt[2:4,2:4] <- src_b = [[5,6],[7,8]]
     //   everything else -> fill 0
     let file_data = include_bytes!("fixtures/vds_2d_same_file.h5");
-    let (raw, datatype, _) = read_chunked_dataset(file_data, "virt");
+    let (raw, datatype) = read_virtual_fixture(file_data, "virt");
     let values = read_as_i32(&raw, &datatype).unwrap();
     assert_eq!(
         values,

crates/clawhdf5-io/src/async_read.rs

@@ -268,28 +268,26 @@ impl AsyncHDF5File {
     ///
     /// Reads the entire file into memory, then parses the superblock.
     pub async fn open<R: AsyncHDF5Read>(reader: &R) -> Result<Self, AsyncHDF5Error> {
-        let data = reader.read_all().await?;
-        let sig_offset = find_signature(&data)?;
-        let superblock = Superblock::parse(&data, sig_offset)?;
-        Ok(Self { data, superblock })
+        Self::from_bytes(reader.read_all().await?)
     }
 
     /// Open an HDF5 file asynchronously from a file path.
     pub async fn open_path<P: AsRef<Path>>(path: P) -> Result<Self, AsyncHDF5Error> {
-        let data = tokio::fs::read(path).await?;
-        let sig_offset = find_signature(&data)?;
-        let superblock = Superblock::parse(&data, sig_offset)?;
-        Ok(Self { data, superblock })
+        Self::from_bytes(tokio::fs::read(path).await?)
     }
 
     /// Open an HDF5 file from bytes already in memory.
-    pub fn from_bytes(data: Vec<u8>) -> Result<Self, AsyncHDF5Error> {
-        let sig_offset = find_signature(&data)?;
-        let superblock = Superblock::parse(&data, sig_offset)?;
+    pub fn from_bytes(mut data: Vec<u8>) -> Result<Self, AsyncHDF5Error> {
+        // HDF5 addresses are relative to the superblock: drop any user block
+        // so they index `data` directly.
+        let user_block = find_signature(&data)?;
+        data.drain(..user_block);
+        let superblock = Superblock::parse(&data, 0)?;
         Ok(Self { data, superblock })
     }
 
-    /// Access the raw file bytes.
+    /// Access the file bytes from the superblock on (any user block is
+    /// dropped on open).
     pub fn as_bytes(&self) -> &[u8] {
         &self.data
     }

crates/clawhdf5-io/src/mpi_vol.rs

@@ -180,7 +180,7 @@ fn mpi_collective_read(vol: &MpiVol, location: &str, path: &str) -> Result<Vec<u
     use clawhdf5_format::{
         data_layout::DataLayout, data_read::read_raw_data_full, dataspace::Dataspace,
         datatype::Datatype, filter_pipeline::FilterPipeline, group_v2::resolve_path_any,
-        message_type::MessageType, object_header::ObjectHeader, signature::find_signature,
+        message_type::MessageType, object_header::ObjectHeader, signature::split_user_block,
         superblock::Superblock,
     };
     use mpi::traits::*;
@@ -192,9 +192,10 @@ fn mpi_collective_read(vol: &MpiVol, location: &str, path: &str) -> Result<Vec<u
     let mut len_buf = [0usize; 1];
 
     if rank == 0 {
-        let bytes = std::fs::read(location).map_err(VolError::Io)?;
-        let sig = find_signature(&bytes).map_err(|e| VolError::DataError(e.to_string()))?;
-        let sb = Superblock::parse(&bytes, sig).map_err(|e| VolError::DataError(e.to_string()))?;
+        let file = std::fs::read(location).map_err(VolError::Io)?;
+        // Addresses are relative to the superblock: skip any user block.
+        let (_, bytes) = split_user_block(&file).map_err(|e| VolError::DataError(e.to_string()))?;
+        let sb = Superblock::parse(bytes, 0).map_err(|e| VolError::DataError(e.to_string()))?;
         let addr = resolve_path_any(&bytes, &sb, path)
             .map_err(|e| VolError::NotFound(format!("{path}: {e}")))?;
         let oh = ObjectHeader::parse(&bytes, addr as usize, sb.offset_size, sb.length_size)

crates/clawhdf5-io/src/vol.rs

@@ -283,12 +283,13 @@ impl VirtualObjectLayer for NativeVol {
         use clawhdf5_format::{
             data_layout::DataLayout, data_read::read_raw_data_full, dataspace::Dataspace,
             datatype::Datatype, filter_pipeline::FilterPipeline, group_v2::resolve_path_any,
-            message_type::MessageType, object_header::ObjectHeader, signature::find_signature,
+            message_type::MessageType, object_header::ObjectHeader, signature::split_user_block,
             superblock::Superblock,
         };
 
-        let sig = find_signature(data).map_err(|e| VolError::DataError(e.to_string()))?;
-        let sb = Superblock::parse(data, sig).map_err(|e| VolError::DataError(e.to_string()))?;
+        // Addresses are relative to the superblock: skip any user block.
+        let (_, data) = split_user_block(data).map_err(|e| VolError::DataError(e.to_string()))?;
+        let sb = Superblock::parse(data, 0).map_err(|e| VolError::DataError(e.to_string()))?;
         let addr = resolve_path_any(data, &sb, path)
             .map_err(|e| VolError::NotFound(format!("{path}: {e}")))?;
 

crates/clawhdf5/src/lazy.rs

@@ -12,25 +12,23 @@
 use std::cell::RefCell;
 use std::collections::HashMap;
 
-use clawhdf5_format::attribute::extract_attributes_full;
 use clawhdf5_format::data_layout::DataLayout;
 use clawhdf5_format::data_read;
 use clawhdf5_format::dataspace::Dataspace;
 use clawhdf5_format::datatype::Datatype;
 use clawhdf5_format::error::FormatError;
 use clawhdf5_format::filter_pipeline::FilterPipeline;
-use clawhdf5_format::group_v1::{self, GroupEntry};
+use clawhdf5_format::group_v1::GroupEntry;
 use clawhdf5_format::group_v2;
 use clawhdf5_format::message_type::MessageType;
 use clawhdf5_format::object_header::ObjectHeader;
 use clawhdf5_format::signature;
 use clawhdf5_format::superblock::Superblock;
-use clawhdf5_format::symbol_table::SymbolTableMessage;
 
 use clawhdf5_io::HDF5Read;
 
 use crate::error::Error;
-use crate::types::{AttrValue, DType, attrs_to_map, classify_datatype};
+use crate::types::{AttrValue, DType, classify_datatype, read_attrs};
 
 /// A lazy HDF5 file handle that parses metadata on demand.
 ///
@@ -42,6 +40,9 @@ use crate::types::{AttrValue, DType, attrs_to_map, classify_datatype};
 /// `MemoryReader`, etc.
 pub struct LazyFile<R: HDF5Read> {
     reader: R,
+    /// Offset of the superblock in the file (the user-block size); every
+    /// HDF5 address is relative to it.
+    base: usize,
     superblock: Superblock,
     root_header: ObjectHeader,
     /// Cache of parsed object headers, keyed by address.
@@ -73,9 +74,9 @@ impl<R: HDF5Read> LazyFile<R> {
     ///
     /// Parses only the superblock and root group object header.
     pub fn open(reader: R) -> Result<Self, Error> {
-        let data = reader.as_bytes();
-        let sig_offset = signature::find_signature(data)?;
-        let superblock = Superblock::parse(data, sig_offset)?;
+        let (user_block, data) = signature::split_user_block(reader.as_bytes())?;
+        let base = user_block.len();
+        let superblock = Superblock::parse(data, 0)?;
         let root_header = ObjectHeader::parse(
             data,
             superblock.root_group_address as usize,
@@ -84,15 +85,26 @@ impl<R: HDF5Read> LazyFile<R> {
         )?;
         Ok(Self {
             reader,
+            base,
             superblock,
             root_header,
             header_cache: RefCell::new(HashMap::new()),
         })
     }
 
-    /// Returns the raw file bytes.
+    /// Returns the file's bytes from the superblock on (after any user
+    /// block), which is the space every HDF5 address in the file indexes.
     pub fn as_bytes(&self) -> &[u8] {
-        self.reader.as_bytes()
+        self.hdf5_bytes()
+    }
+
+    /// Size of the user block before the superblock (0 for most files).
+    pub fn user_block_size(&self) -> u64 {
+        self.base as u64
+    }
+
+    fn hdf5_bytes(&self) -> &[u8] {
+        &self.reader.as_bytes()[self.base..]
     }
 
     /// Returns a reference to the parsed superblock.
@@ -110,7 +122,7 @@ impl<R: HDF5Read> LazyFile<R> {
 
     /// Resolve a path and return a `LazyDataset` handle.
     pub fn dataset(&self, path: &str) -> Result<LazyDataset<'_, R>, Error> {
-        let data = self.reader.as_bytes();
+        let data = self.hdf5_bytes();
         let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
         let hdr = self.get_or_parse_header(addr)?;
         if !has_message(&hdr, MessageType::DataLayout) {
@@ -124,7 +136,7 @@ impl<R: HDF5Read> LazyFile<R> {
 
     /// Resolve a path and return a `LazyGroup` handle.
     pub fn group(&self, path: &str) -> Result<LazyGroup<'_, R>, Error> {
-        let data = self.reader.as_bytes();
+        let data = self.hdf5_bytes();
         let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
         Ok(LazyGroup {
             file: self,
@@ -163,7 +175,7 @@ impl<R: HDF5Read> LazyFile<R> {
         }
 
         // Parse and cache
-        let data = self.reader.as_bytes();
+        let data = self.hdf5_bytes();
         let hdr = ObjectHeader::parse(
             data,
             address as usize,
@@ -187,7 +199,7 @@ impl<R: HDF5Read> LazyFile<R> {
 impl<R: HDF5Read> std::fmt::Debug for LazyFile<R> {
     fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         f.debug_struct("LazyFile")
-            .field("size", &self.reader.as_bytes().len())
+            .field("size", &self.hdf5_bytes().len())
             .field("superblock_version", &self.superblock.version)
             .field("cached_headers", &self.header_cache.borrow().len())
             .finish()
@@ -232,17 +244,26 @@ impl<'f, R: HDF5Read> LazyGroup<'f, R> {
     }
 
     /// Read all attributes of this group.
+    ///
+    /// An attribute that cannot be read — a corrupt or unsupported attribute
+    /// message, or a dense-storage heap object that cannot be located — is
+    /// left out of the map instead of failing every attribute on the object;
+    /// [`attrs_with_errors`](Self::attrs_with_errors) reports which failed.
+    /// Values that are returned are complete (never partially decoded). An
+    /// error in the index of the attributes itself (the attribute info
+    /// message, the dense heap header or B-tree) still fails the call.
     pub fn attrs(&self) -> Result<HashMap<String, AttrValue>, Error> {
+        self.attrs_with_errors().map(|(attrs, _)| attrs)
+    }
+
+    /// Like [`attrs`](Self::attrs), also returning one error for each
+    /// attribute that could not be read and was left out.
+    pub fn attrs_with_errors(
+        &self,
+    ) -> Result<(HashMap<String, AttrValue>, Vec<FormatError>), Error> {
         let hdr = self.file.get_or_parse_header(self.address)?;
-        let data = self.file.reader.as_bytes();
-        let attr_msgs =
-            extract_attributes_full(data, &hdr, self.file.offset_size(), self.file.length_size())?;
-        Ok(attrs_to_map(
-            &attr_msgs,
-            data,
-            self.file.offset_size(),
-            self.file.length_size(),
-        ))
+        let data = self.file.hdf5_bytes();
+        read_attrs(data, &hdr, self.file.offset_size(), self.file.length_size())
     }
 
     /// Get a dataset within this group by name.
@@ -275,12 +296,14 @@ impl<'f, R: HDF5Read> LazyGroup<'f, R> {
         })
     }
 
+    /// This group's links that can be opened: hard links, and soft links
+    /// resolved to their targets (see
+    /// [`group_v2::resolve_group_children`]); dangling, external and
+    /// user-defined links are left out.
     fn children(&self) -> Result<Vec<GroupEntry>, Error> {
-        let hdr = self.file.get_or_parse_header(self.address)?;
-        let data = self.file.reader.as_bytes();
-        let os = self.file.offset_size();
-        let ls = self.file.length_size();
-        resolve_group_entries(data, &hdr, os, ls).map_err(Error::Format)
+        let data = self.file.hdf5_bytes();
+        group_v2::resolve_group_children(data, &self.file.superblock, self.address)
+            .map_err(Error::Format)
     }
 }
 
@@ -360,7 +383,7 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
         let dl = self.data_layout()?;
         let ds = self.dataspace()?;
         let dt = self.datatype()?;
-        let slice = data_read::read_raw_data_zerocopy(self.file.reader.as_bytes(), &dl, &ds, &dt)?;
+        let slice = data_read::read_raw_data_zerocopy(self.file.hdf5_bytes(), &dl, &ds, &dt)?;
         Ok(slice)
     }
 
@@ -400,20 +423,30 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
     }
 
     /// Read all attributes of this dataset.
+    ///
+    /// An attribute that cannot be read — a corrupt or unsupported attribute
+    /// message, or a dense-storage heap object that cannot be located — is
+    /// left out of the map instead of failing every attribute on the object;
+    /// [`attrs_with_errors`](Self::attrs_with_errors) reports which failed.
+    /// Values that are returned are complete (never partially decoded). An
+    /// error in the index of the attributes itself (the attribute info
+    /// message, the dense heap header or B-tree) still fails the call.
     pub fn attrs(&self) -> Result<HashMap<String, AttrValue>, Error> {
-        let data = self.file.reader.as_bytes();
-        let attr_msgs = extract_attributes_full(
+        self.attrs_with_errors().map(|(attrs, _)| attrs)
+    }
+
+    /// Like [`attrs`](Self::attrs), also returning one error for each
+    /// attribute that could not be read and was left out.
+    pub fn attrs_with_errors(
+        &self,
+    ) -> Result<(HashMap<String, AttrValue>, Vec<FormatError>), Error> {
+        let data = self.file.hdf5_bytes();
+        read_attrs(
             data,
             &self.header,
             self.file.offset_size(),
             self.file.length_size(),
-        )?;
-        Ok(attrs_to_map(
-            &attr_msgs,
-            data,
-            self.file.offset_size(),
-            self.file.length_size(),
-        ))
+        )
     }
 
     /// A header message's payload, resolved through the shared-message
@@ -479,7 +512,7 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
         let ds = self.dataspace()?;
         let dl = self.data_layout()?;
         let pipeline = self.filter_pipeline()?;
-        let data = self.file.reader.as_bytes();
+        let data = self.file.hdf5_bytes();
         // Unallocated storage reads as the dataset's fill value.
         clawhdf5_format::fill_value::read_full_with_fill(
             &self.header.messages,
@@ -530,33 +563,3 @@ fn is_group(header: &ObjectHeader) -> bool {
             || m.msg_type == MessageType::SymbolTable
     })
 }
-
-fn resolve_group_entries(
-    file_data: &[u8],
-    object_header: &ObjectHeader,
-    offset_size: u8,
-    length_size: u8,
-) -> Result<Vec<GroupEntry>, FormatError> {
-    let is_v1 = object_header
-        .messages
-        .iter()
-        .any(|m| m.msg_type == MessageType::SymbolTable);
-    let is_v2 = object_header
-        .messages
-        .iter()
-        .any(|m| m.msg_type == MessageType::LinkInfo || m.msg_type == MessageType::Link);
-
-    if is_v1 {
-        let sym_msg = object_header
-            .messages
-            .iter()
-            .find(|m| m.msg_type == MessageType::SymbolTable)
-            .ok_or_else(|| FormatError::PathNotFound("no symbol table message".into()))?;
-        let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
-        group_v1::resolve_v1_group_entries(file_data, &stm, offset_size, length_size)
-    } else if is_v2 {
-        group_v2::resolve_v2_group_entries(file_data, object_header, offset_size, length_size)
-    } else {
-        Ok(Vec::new())
-    }
-}

crates/clawhdf5/src/mmap_file.rs

@@ -7,25 +7,23 @@
 
 use std::collections::HashMap;
 
-use clawhdf5_format::attribute::extract_attributes_full;
 use clawhdf5_format::data_layout::DataLayout;
 use clawhdf5_format::data_read;
 use clawhdf5_format::dataspace::Dataspace;
 use clawhdf5_format::datatype::Datatype;
 use clawhdf5_format::error::FormatError;
 use clawhdf5_format::filter_pipeline::FilterPipeline;
-use clawhdf5_format::group_v1::{self, GroupEntry};
+use clawhdf5_format::group_v1::GroupEntry;
 use clawhdf5_format::group_v2;
 use clawhdf5_format::message_type::MessageType;
 use clawhdf5_format::object_header::ObjectHeader;
 use clawhdf5_format::signature;
 use clawhdf5_format::superblock::Superblock;
-use clawhdf5_format::symbol_table::SymbolTableMessage;
 
 use clawhdf5_io::MmapReader;
 
 use crate::error::Error;
-use crate::types::{AttrValue, DType, attrs_to_map, classify_datatype};
+use crate::types::{AttrValue, DType, classify_datatype, read_attrs};
 
 /// An HDF5 file opened via memory mapping.
 ///
@@ -34,6 +32,9 @@ use crate::types::{AttrValue, DType, attrs_to_map, classify_datatype};
 /// `&[u8]` slice via [`MmapDataset::read_raw_slice`].
 pub struct MmapFile {
     reader: MmapReader,
+    /// Offset of the superblock in the mapped file (the user-block size);
+    /// every HDF5 address is relative to it.
+    base: usize,
     superblock: Superblock,
 }
 
@@ -41,10 +42,25 @@ impl MmapFile {
     /// Open an HDF5 file using memory-mapped I/O.
     pub fn open<P: AsRef<std::path::Path>>(path: P) -> Result<Self, Error> {
         let reader = MmapReader::open(path).map_err(Error::Io)?;
-        let data = reader.as_bytes();
-        let sig_offset = signature::find_signature(data)?;
-        let superblock = Superblock::parse(data, sig_offset)?;
-        Ok(Self { reader, superblock })
+        let (user_block, data) = signature::split_user_block(reader.as_bytes())?;
+        let base = user_block.len();
+        let superblock = Superblock::parse(data, 0)?;
+        Ok(Self {
+            reader,
+            base,
+            superblock,
+        })
+    }
+
+    /// The file's bytes from the superblock on — the space HDF5 addresses
+    /// index into.
+    fn hdf5_bytes(&self) -> &[u8] {
+        &self.reader.as_bytes()[self.base..]
+    }
+
+    /// Size of the user block before the superblock (0 for most files).
+    pub fn user_block_size(&self) -> u64 {
+        self.base as u64
     }
 
     /// Returns a handle to the root group.
@@ -57,7 +73,7 @@ impl MmapFile {
 
     /// Resolve a path and return a `MmapDataset` handle.
     pub fn dataset(&self, path: &str) -> Result<MmapDataset<'_>, Error> {
-        let data = self.reader.as_bytes();
+        let data = self.hdf5_bytes();
         let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
         let hdr = self.parse_header(addr)?;
         if !has_message(&hdr, MessageType::DataLayout) {
@@ -71,7 +87,7 @@ impl MmapFile {
 
     /// Resolve a path and return a `MmapGroup` handle.
     pub fn group(&self, path: &str) -> Result<MmapGroup<'_>, Error> {
-        let data = self.reader.as_bytes();
+        let data = self.hdf5_bytes();
         let addr = group_v2::resolve_path_any(data, &self.superblock, path)?;
         Ok(MmapGroup {
             file: self,
@@ -79,9 +95,11 @@ impl MmapFile {
         })
     }
 
-    /// Returns the raw file bytes (zero-copy from mmap).
+    /// Returns the file's bytes from the superblock on (after any user
+    /// block), zero-copy from the mmap. Every HDF5 address in the file
+    /// indexes this slice.
     pub fn as_bytes(&self) -> &[u8] {
-        self.reader.as_bytes()
+        self.hdf5_bytes()
     }
 
     /// Returns a reference to the parsed superblock.
@@ -91,7 +109,7 @@ impl MmapFile {
 
     fn parse_header(&self, address: u64) -> Result<ObjectHeader, FormatError> {
         ObjectHeader::parse(
-            self.reader.as_bytes(),
+            self.hdf5_bytes(),
             address as usize,
             self.superblock.offset_size,
             self.superblock.length_size,
@@ -154,17 +172,26 @@ impl<'f> MmapGroup<'f> {
     }
 
     /// Read all attributes of this group.
+    ///
+    /// An attribute that cannot be read — a corrupt or unsupported attribute
+    /// message, or a dense-storage heap object that cannot be located — is
+    /// left out of the map instead of failing every attribute on the object;
+    /// [`attrs_with_errors`](Self::attrs_with_errors) reports which failed.
+    /// Values that are returned are complete (never partially decoded). An
+    /// error in the index of the attributes itself (the attribute info
+    /// message, the dense heap header or B-tree) still fails the call.
     pub fn attrs(&self) -> Result<HashMap<String, AttrValue>, Error> {
-        let data = self.file.reader.as_bytes();
+        self.attrs_with_errors().map(|(attrs, _)| attrs)
+    }
+
+    /// Like [`attrs`](Self::attrs), also returning one error for each
+    /// attribute that could not be read and was left out.
+    pub fn attrs_with_errors(
+        &self,
+    ) -> Result<(HashMap<String, AttrValue>, Vec<FormatError>), Error> {
         let hdr = self.file.parse_header(self.address)?;
-        let attr_msgs =
-            extract_attributes_full(data, &hdr, self.file.offset_size(), self.file.length_size())?;
-        Ok(attrs_to_map(
-            &attr_msgs,
-            data,
-            self.file.offset_size(),
-            self.file.length_size(),
-        ))
+        let data = self.file.hdf5_bytes();
+        read_attrs(data, &hdr, self.file.offset_size(), self.file.length_size())
     }
 
     /// Get a dataset within this group by name.
@@ -197,12 +224,14 @@ impl<'f> MmapGroup<'f> {
         })
     }
 
+    /// This group's links that can be opened: hard links, and soft links
+    /// resolved to their targets (see
+    /// [`group_v2::resolve_group_children`]); dangling, external and
+    /// user-defined links are left out.
     fn children(&self) -> Result<Vec<GroupEntry>, Error> {
-        let data = self.file.reader.as_bytes();
-        let hdr = self.file.parse_header(self.address)?;
-        let os = self.file.offset_size();
-        let ls = self.file.length_size();
-        resolve_group_entries(data, &hdr, os, ls).map_err(Error::Format)
+        let data = self.file.hdf5_bytes();
+        group_v2::resolve_group_children(data, &self.file.superblock, self.address)
+            .map_err(Error::Format)
     }
 }
 
@@ -326,7 +355,7 @@ impl<'f> MmapDataset<'f> {
                         actual: sz,
                     }));
                 }
-                let data = self.file.reader.as_bytes();
+                let data = self.file.hdf5_bytes();
                 let a = addr as usize;
                 if a + sz > data.len() {
                     return Err(Error::Format(FormatError::UnexpectedEof {
@@ -341,20 +370,30 @@ impl<'f> MmapDataset<'f> {
     }
 
     /// Read all attributes of this dataset.
+    ///
+    /// An attribute that cannot be read — a corrupt or unsupported attribute
+    /// message, or a dense-storage heap object that cannot be located — is
+    /// left out of the map instead of failing every attribute on the object;
+    /// [`attrs_with_errors`](Self::attrs_with_errors) reports which failed.
+    /// Values that are returned are complete (never partially decoded). An
+    /// error in the index of the attributes itself (the attribute info
+    /// message, the dense heap header or B-tree) still fails the call.
     pub fn attrs(&self) -> Result<HashMap<String, AttrValue>, Error> {
-        let data = self.file.reader.as_bytes();
-        let attr_msgs = extract_attributes_full(
+        self.attrs_with_errors().map(|(attrs, _)| attrs)
+    }
+
+    /// Like [`attrs`](Self::attrs), also returning one error for each
+    /// attribute that could not be read and was left out.
+    pub fn attrs_with_errors(
+        &self,
+    ) -> Result<(HashMap<String, AttrValue>, Vec<FormatError>), Error> {
+        let data = self.file.hdf5_bytes();
+        read_attrs(
             data,
             &self.header,
             self.file.offset_size(),
             self.file.length_size(),
-        )?;
-        Ok(attrs_to_map(
-            &attr_msgs,
-            data,
-            self.file.offset_size(),
-            self.file.length_size(),
-        ))
+        )
     }
 
     /// A header message's payload, resolved through the shared-message
@@ -423,7 +462,7 @@ impl<'f> MmapDataset<'f> {
         // Unallocated storage reads as the dataset's fill value.
         clawhdf5_format::fill_value::read_full_with_fill(
             &self.header.messages,
-            self.file.reader.as_bytes(),
+            self.file.hdf5_bytes(),
             &dl,
             &ds,
             dt.type_size() as usize,
@@ -431,7 +470,7 @@ impl<'f> MmapDataset<'f> {
             self.file.length_size(),
             || {
                 Ok(data_read::read_raw_data_full(
-                    self.file.reader.as_bytes(),
+                    self.file.hdf5_bytes(),
                     &dl,
                     &ds,
                     &dt,
@@ -470,33 +509,3 @@ fn is_group(header: &ObjectHeader) -> bool {
             || m.msg_type == MessageType::SymbolTable
     })
 }
-
-fn resolve_group_entries(
-    file_data: &[u8],
-    object_header: &ObjectHeader,
-    offset_size: u8,
-    length_size: u8,
-) -> Result<Vec<GroupEntry>, FormatError> {
-    let is_v1 = object_header
-        .messages
-        .iter()
-        .any(|m| m.msg_type == MessageType::SymbolTable);
-    let is_v2 = object_header
-        .messages
-        .iter()
-        .any(|m| m.msg_type == MessageType::LinkInfo || m.msg_type == MessageType::Link);
-
-    if is_v1 {
-        let sym_msg = object_header
-            .messages
-            .iter()
-            .find(|m| m.msg_type == MessageType::SymbolTable)
-            .ok_or_else(|| FormatError::PathNotFound("no symbol table message".into()))?;
-        let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
-        group_v1::resolve_v1_group_entries(file_data, &stm, offset_size, length_size)
-    } else if is_v2 {
-        group_v2::resolve_v2_group_entries(file_data, object_header, offset_size, length_size)
-    } else {
-        Ok(Vec::new())
-    }
-}

crates/clawhdf5/src/reader.rs

@@ -7,7 +7,6 @@
 
 use std::collections::HashMap;
 
-use clawhdf5_format::attribute::extract_attributes_full;
 use clawhdf5_format::chunk_cache::ChunkCache;
 use clawhdf5_format::data_layout::DataLayout;
 use clawhdf5_format::data_read;
@@ -15,36 +14,58 @@ use clawhdf5_format::dataspace::Dataspace;
 use clawhdf5_format::datatype::{Datatype, DatatypeByteOrder};
 use clawhdf5_format::error::FormatError;
 use clawhdf5_format::filter_pipeline::FilterPipeline;
-use clawhdf5_format::group_v1::{self, GroupEntry};
+use clawhdf5_format::group_v1::GroupEntry;
 use clawhdf5_format::group_v2;
 use clawhdf5_format::message_type::MessageType;
 use clawhdf5_format::object_header::ObjectHeader;
 use clawhdf5_format::signature;
 use clawhdf5_format::superblock::Superblock;
-use clawhdf5_format::symbol_table::SymbolTableMessage;
 
 use crate::error::Error;
-use crate::types::{AttrValue, DType, attrs_to_map, classify_datatype};
+use crate::types::{AttrValue, DType, classify_datatype, read_attrs};
 
 // ---------------------------------------------------------------------------
 // FileData — internal storage for either owned bytes or an mmap
 // ---------------------------------------------------------------------------
 
 /// Internal storage: either an owned `Vec<u8>` or a memory-mapped region.
-enum FileData {
+enum Backing {
     Owned(Vec<u8>),
     #[cfg(feature = "mmap")]
     Mmap(clawhdf5_io::MmapReader),
 }
 
-impl FileData {
-    fn as_bytes(&self) -> &[u8] {
+impl Backing {
+    fn whole_file(&self) -> &[u8] {
         match self {
-            FileData::Owned(v) => v,
+            Backing::Owned(v) => v,
             #[cfg(feature = "mmap")]
-            FileData::Mmap(r) => r.as_bytes(),
+            Backing::Mmap(r) => r.as_bytes(),
         }
     }
+}
+
+/// The file's bytes, viewed from the superblock on. A file may start with a
+/// user block (the superblock at 512, 1024, …); every HDF5 address is
+/// relative to the superblock, so all parsing goes through [`Self::as_bytes`].
+struct FileData {
+    backing: Backing,
+    /// Offset of the superblock in the file (the user-block size).
+    base: usize,
+}
+
+impl FileData {
+    /// Locate the superblock and parse it.
+    fn new(backing: Backing) -> Result<(Self, Superblock), Error> {
+        let (user_block, hdf5) = signature::split_user_block(backing.whole_file())?;
+        let base = user_block.len();
+        let superblock = Superblock::parse(hdf5, 0)?;
+        Ok((Self { backing, base }, superblock))
+    }
+
+    fn as_bytes(&self) -> &[u8] {
+        &self.backing.whole_file()[self.base..]
+    }
 
     fn len(&self) -> usize {
         self.as_bytes().len()
@@ -81,11 +102,9 @@ impl File {
         #[cfg(feature = "mmap")]
         {
             let reader = clawhdf5_io::MmapReader::open(path).map_err(Error::Io)?;
-            let data_ref = reader.as_bytes();
-            let sig_offset = signature::find_signature(data_ref)?;
-            let superblock = Superblock::parse(data_ref, sig_offset)?;
+            let (data, superblock) = FileData::new(Backing::Mmap(reader))?;
             Ok(Self {
-                data: FileData::Mmap(reader),
+                data,
                 superblock,
                 chunk_cache: ChunkCache::new(),
                 base_dir,
@@ -116,10 +135,9 @@ impl File {
     /// In-memory files have no directory, so external Virtual Dataset sources
     /// cannot be resolved automatically (same-file VDS still works).
     pub fn from_bytes(data: Vec<u8>) -> Result<Self, Error> {
-        let sig_offset = signature::find_signature(&data)?;
-        let superblock = Superblock::parse(&data, sig_offset)?;
+        let (data, superblock) = FileData::new(Backing::Owned(data))?;
         Ok(Self {
-            data: FileData::Owned(data),
+            data,
             superblock,
             chunk_cache: ChunkCache::new(),
             base_dir: None,
@@ -209,11 +227,19 @@ impl File {
         Ok(results.into_iter().map(|(_, data)| data).collect())
     }
 
-    /// Returns the raw file bytes.
+    /// Returns the file's bytes from the superblock on (after any user
+    /// block). Every HDF5 address in the file indexes this slice, so it is
+    /// what the `clawhdf5_format` parsers expect as `file_data`.
     pub fn as_bytes(&self) -> &[u8] {
         self.data.as_bytes()
     }
 
+    /// Size of the user block before the superblock (0 for most files).
+    /// Matches h5py's `File.userblock_size`.
+    pub fn user_block_size(&self) -> u64 {
+        self.data.base as u64
+    }
+
     /// Returns a reference to the parsed superblock.
     pub fn superblock(&self) -> &Superblock {
         &self.superblock
@@ -221,10 +247,10 @@ impl File {
 
     /// Returns `true` when the file is backed by memory-mapped I/O.
     pub fn is_mmap(&self) -> bool {
-        match &self.data {
-            FileData::Owned(_) => false,
+        match &self.data.backing {
+            Backing::Owned(_) => false,
             #[cfg(feature = "mmap")]
-            FileData::Mmap(_) => true,
+            Backing::Mmap(_) => true,
         }
     }
 
@@ -294,17 +320,26 @@ impl<'f> Group<'f> {
     }
 
     /// Read all attributes of this group.
+    ///
+    /// An attribute that cannot be read — a corrupt or unsupported attribute
+    /// message, or a dense-storage heap object that cannot be located — is
+    /// left out of the map instead of failing every attribute on the object;
+    /// [`attrs_with_errors`](Self::attrs_with_errors) reports which failed.
+    /// Values that are returned are complete (never partially decoded). An
+    /// error in the index of the attributes itself (the attribute info
+    /// message, the dense heap header or B-tree) still fails the call.
     pub fn attrs(&self) -> Result<HashMap<String, AttrValue>, Error> {
-        let data = self.file.data.as_bytes();
+        self.attrs_with_errors().map(|(attrs, _)| attrs)
+    }
+
+    /// Like [`attrs`](Self::attrs), also returning one error for each
+    /// attribute that could not be read and was left out.
+    pub fn attrs_with_errors(
+        &self,
+    ) -> Result<(HashMap<String, AttrValue>, Vec<FormatError>), Error> {
         let hdr = self.file.parse_header(self.address)?;
-        let attr_msgs =
-            extract_attributes_full(data, &hdr, self.file.offset_size(), self.file.length_size())?;
-        Ok(attrs_to_map(
-            &attr_msgs,
-            data,
-            self.file.offset_size(),
-            self.file.length_size(),
-        ))
+        let data = self.file.data.as_bytes();
+        read_attrs(data, &hdr, self.file.offset_size(), self.file.length_size())
     }
 
     /// Get a dataset within this group by name.
@@ -337,12 +372,14 @@ impl<'f> Group<'f> {
         })
     }
 
+    /// This group's links that can be opened: hard links, and soft links
+    /// resolved to their targets (see
+    /// [`group_v2::resolve_group_children`]); dangling, external and
+    /// user-defined links are left out.
     fn children(&self) -> Result<Vec<GroupEntry>, Error> {
         let data = self.file.data.as_bytes();
-        let hdr = self.file.parse_header(self.address)?;
-        let os = self.file.offset_size();
-        let ls = self.file.length_size();
-        resolve_group_entries(data, &hdr, os, ls).map_err(Error::Format)
+        group_v2::resolve_group_children(data, &self.file.superblock, self.address)
+            .map_err(Error::Format)
     }
 }
 
@@ -485,6 +522,7 @@ impl<'f> Dataset<'f> {
             self.file.length_size(),
         )?;
         let fill_matters = !clawhdf5_format::fill_value::has_storage(&dl)
+            || matches!(dl, DataLayout::Virtual { .. })
             || (matches!(dl, DataLayout::Chunked { .. })
                 && !clawhdf5_format::fill_value::is_default(fill.as_deref()));
         if fill_matters {
@@ -731,20 +769,30 @@ impl<'f> Dataset<'f> {
     }
 
     /// Read all attributes of this dataset.
+    ///
+    /// An attribute that cannot be read — a corrupt or unsupported attribute
+    /// message, or a dense-storage heap object that cannot be located — is
+    /// left out of the map instead of failing every attribute on the object;
+    /// [`attrs_with_errors`](Self::attrs_with_errors) reports which failed.
+    /// Values that are returned are complete (never partially decoded). An
+    /// error in the index of the attributes itself (the attribute info
+    /// message, the dense heap header or B-tree) still fails the call.
     pub fn attrs(&self) -> Result<HashMap<String, AttrValue>, Error> {
+        self.attrs_with_errors().map(|(attrs, _)| attrs)
+    }
+
+    /// Like [`attrs`](Self::attrs), also returning one error for each
+    /// attribute that could not be read and was left out.
+    pub fn attrs_with_errors(
+        &self,
+    ) -> Result<(HashMap<String, AttrValue>, Vec<FormatError>), Error> {
         let data = self.file.data.as_bytes();
-        let attr_msgs = extract_attributes_full(
+        read_attrs(
             data,
             &self.header,
             self.file.offset_size(),
             self.file.length_size(),
-        )?;
-        Ok(attrs_to_map(
-            &attr_msgs,
-            data,
-            self.file.offset_size(),
-            self.file.length_size(),
-        ))
+        )
     }
 
     /// Verify this dataset's content against its stored provenance hash
@@ -808,7 +856,22 @@ impl<'f> Dataset<'f> {
 
     fn dataspace(&self) -> Result<Dataspace, Error> {
         let data = self.required_payload(MessageType::Dataspace)?;
-        Ok(Dataspace::parse(&data, self.file.length_size())?)
+        let mut ds = Dataspace::parse(&data, self.file.length_size())?;
+        // libhdf5 reports a virtual dataset with unlimited or printf-style
+        // mappings at the extent its sources currently fill, not the stored
+        // one (`H5Dget_space`).
+        if let Ok(dl @ DataLayout::Virtual { .. }) = self.data_layout() {
+            let resolver = self.vds_resolver();
+            ds.dimensions = clawhdf5_format::vds::virtual_dataset_extent(
+                self.file.data.as_bytes(),
+                &dl,
+                &ds,
+                self.file.offset_size(),
+                self.file.length_size(),
+                Some(&resolver),
+            )?;
+        }
+        Ok(ds)
     }
 
     fn data_layout(&self) -> Result<DataLayout, Error> {
@@ -837,24 +900,9 @@ impl<'f> Dataset<'f> {
         let pipeline = self.filter_pipeline()?;
 
         // Virtual datasets are assembled from source datasets; the per-file
-        // chunk cache does not apply. Route them through the resolver path so
-        // external sibling files resolve relative to this file's directory.
+        // chunk cache does not apply.
         if matches!(dl, DataLayout::Virtual { .. }) {
-            let base_dir = self.file.base_dir.clone();
-            let resolver = move |name: &str| -> Option<Vec<u8>> {
-                let dir = base_dir.as_ref()?;
-                std::fs::read(dir.join(sibling_file_name(name)?)).ok()
-            };
-            return Ok(data_read::read_raw_data_full_with_resolver(
-                self.file.data.as_bytes(),
-                &dl,
-                &ds,
-                &dt,
-                pipeline.as_ref(),
-                self.file.offset_size(),
-                self.file.length_size(),
-                Some(&resolver),
-            )?);
+            return self.read_virtual(&dl, &ds, &dt);
         }
 
         // Unallocated storage reads as the dataset's fill value.
@@ -880,6 +928,62 @@ impl<'f> Dataset<'f> {
             },
         )
     }
+
+    /// Resolver for external Virtual Dataset source files: names are
+    /// resolved against the directory of the file that holds the virtual
+    /// dataset, as libhdf5 does. A missing file is `Ok(None)` (its mappings
+    /// read as the fill value); a name that would leave that directory is
+    /// refused with an error rather than read as fill.
+    fn vds_resolver(&self) -> impl Fn(&str) -> Result<Option<Vec<u8>>, FormatError> + use<> {
+        let base_dir = self.file.base_dir.clone();
+        move |name: &str| {
+            let Some(dir) = base_dir.as_ref() else {
+                return Err(FormatError::ChunkedReadError(format!(
+                    "virtual dataset source file {name:?} cannot be resolved for an in-memory file"
+                )));
+            };
+            let rel = sibling_file_name(name).ok_or_else(|| {
+                FormatError::ChunkedReadError(format!(
+                    "virtual dataset source file {name:?} is outside the virtual file's \
+                     directory and is not followed"
+                ))
+            })?;
+            match std::fs::read(dir.join(rel)) {
+                Ok(bytes) => Ok(Some(bytes)),
+                Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
+                Err(e) => Err(FormatError::ChunkedReadError(format!(
+                    "cannot read virtual dataset source file {name:?}: {e}"
+                ))),
+            }
+        }
+    }
+
+    /// Read a whole virtual dataset; unmapped elements hold its fill value.
+    fn read_virtual(
+        &self,
+        dl: &DataLayout,
+        ds: &Dataspace,
+        dt: &Datatype,
+    ) -> Result<Vec<u8>, Error> {
+        let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
+            self.file.data.as_bytes(),
+            &self.header.messages,
+            self.file.offset_size(),
+            self.file.length_size(),
+        )?;
+        let resolver = self.vds_resolver();
+        let v = clawhdf5_format::vds::read_virtual_dataset(
+            self.file.data.as_bytes(),
+            dl,
+            ds,
+            dt,
+            fill.as_deref(),
+            self.file.offset_size(),
+            self.file.length_size(),
+            Some(&resolver),
+        )?;
+        Ok(v.data)
+    }
 }
 
 // ---------------------------------------------------------------------------
@@ -959,37 +1063,6 @@ fn is_group(header: &ObjectHeader) -> bool {
     })
 }
 
-fn resolve_group_entries(
-    file_data: &[u8],
-    object_header: &ObjectHeader,
-    offset_size: u8,
-    length_size: u8,
-) -> Result<Vec<GroupEntry>, FormatError> {
-    let is_v1 = object_header
-        .messages
-        .iter()
-        .any(|m| m.msg_type == MessageType::SymbolTable);
-    let is_v2 = object_header
-        .messages
-        .iter()
-        .any(|m| m.msg_type == MessageType::LinkInfo || m.msg_type == MessageType::Link);
-
-    if is_v1 {
-        let sym_msg = object_header
-            .messages
-            .iter()
-            .find(|m| m.msg_type == MessageType::SymbolTable)
-            .ok_or_else(|| FormatError::PathNotFound("no symbol table message".into()))?;
-        let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
-        group_v1::resolve_v1_group_entries(file_data, &stm, offset_size, length_size)
-    } else if is_v2 {
-        group_v2::resolve_v2_group_entries(file_data, object_header, offset_size, length_size)
-    } else {
-        // Empty group or unrecognized — return empty
-        Ok(Vec::new())
-    }
-}
-
 #[cfg(test)]
 mod sibling_file_name_tests {
     use super::sibling_file_name;

crates/clawhdf5/src/types.rs

@@ -155,6 +155,33 @@ pub(crate) fn classify_datatype(dt: &clawhdf5_format::datatype::Datatype) -> DTy
 /// Read attribute messages into a `HashMap<String, AttrValue>`.
 ///
 /// Best-effort: attributes that can't be decoded are silently skipped.
+/// The attributes of the object with header `header` that could be read,
+/// and one error for each that could not (see
+/// [`extract_attributes_tolerant`](clawhdf5_format::attribute::extract_attributes_tolerant)).
+pub(crate) fn read_attrs(
+    file_data: &[u8],
+    header: &clawhdf5_format::object_header::ObjectHeader,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<
+    (
+        HashMap<String, AttrValue>,
+        Vec<clawhdf5_format::error::FormatError>,
+    ),
+    crate::Error,
+> {
+    let (msgs, errors) = clawhdf5_format::attribute::extract_attributes_tolerant(
+        file_data,
+        header,
+        offset_size,
+        length_size,
+    )?;
+    Ok((
+        attrs_to_map(&msgs, file_data, offset_size, length_size),
+        errors,
+    ))
+}
+
 pub(crate) fn attrs_to_map(
     attrs: &[clawhdf5_format::attribute::AttributeMessage],
     file_data: &[u8],

crates/clawhdf5/tests/dense_storage_interop.rs

@@ -0,0 +1,418 @@
+//! Dense ("new-style") link and attribute storage written by libhdf5 (via
+//! h5py): groups whose links live in a fractal heap indexed by a v2 B-tree,
+//! and objects whose attributes do. Every listing and value is compared with
+//! what h5py itself reports for the same file.
+//!
+//! Skipped when python3 with h5py is unavailable, unless
+//! `CLAWHDF5_REQUIRE_INTEROP=1`.
+
+use std::process::Command;
+
+use clawhdf5::{AttrValue, File};
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+fn python_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+macro_rules! skip_if_no_python {
+    () => {
+        if !python_available() {
+            assert!(
+                !interop_required(),
+                "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+            );
+            eprintln!("SKIP: python3 with h5py not available");
+            return;
+        }
+    };
+}
+
+fn run_python(script: &str) -> String {
+    let output = Command::new(python())
+        .args(["-c", script])
+        .output()
+        .expect("failed to run python");
+    if !output.status.success() {
+        panic!(
+            "Python script failed:\nSTDOUT: {}\nSTDERR: {}",
+            String::from_utf8_lossy(&output.stdout),
+            String::from_utf8_lossy(&output.stderr)
+        );
+    }
+    String::from_utf8_lossy(&output.stdout).trim().to_string()
+}
+
+/// Have h5py write a file with `body` (which sees `f`, `h5py` and `np`),
+/// returning the temp dir holding it and its path.
+fn h5py_file(body: &str) -> (tempfile::TempDir, String) {
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("dense.h5").display().to_string();
+    let script = format!(
+        "import h5py, numpy as np\n\
+         with h5py.File(r'{path}', 'w', libver='latest') as f:\n{}",
+        indent(body)
+    );
+    run_python(&script);
+    (dir, path)
+}
+
+fn indent(body: &str) -> String {
+    body.lines()
+        .map(|l| format!("    {l}\n"))
+        .collect::<String>()
+}
+
+/// The datasets and groups h5py lists in `group`, sorted: links h5py can
+/// resolve (hard and soft), without dangling soft links or external links.
+fn h5py_listing(path: &str, group: &str) -> (Vec<String>, Vec<String>) {
+    let out = run_python(&format!(
+        "import h5py\n\
+         ds, gs = [], []\n\
+         with h5py.File(r'{path}', 'r') as f:\n\
+         \x20   g = f[{group:?}]\n\
+         \x20   for k in g.keys():\n\
+         \x20       if isinstance(g.get(k, getlink=True), h5py.ExternalLink):\n\
+         \x20           continue\n\
+         \x20       try:\n\
+         \x20           o = g[k]\n\
+         \x20       except Exception:\n\
+         \x20           continue\n\
+         \x20       (ds if isinstance(o, h5py.Dataset) else gs).append(k)\n\
+         print('\\x1f'.join(sorted(ds)))\n\
+         print('\\x1f'.join(sorted(gs)))\n"
+    ));
+    let mut lines = out.lines();
+    let split = |l: Option<&str>| -> Vec<String> {
+        l.unwrap_or("")
+            .split('\x1f')
+            .filter(|s| !s.is_empty())
+            .map(str::to_string)
+            .collect()
+    };
+    let ds = split(lines.next());
+    let gs = split(lines.next());
+    (ds, gs)
+}
+
+fn our_listing(path: &str, group: &str) -> (Vec<String>, Vec<String>) {
+    let f = File::open(path).unwrap();
+    let g = f.group(group).unwrap();
+    let mut ds = g.datasets().unwrap();
+    let mut gs = g.groups().unwrap();
+    ds.sort();
+    gs.sort();
+    (ds, gs)
+}
+
+fn assert_same_listing(path: &str, group: &str) {
+    let ours = our_listing(path, group);
+    let theirs = h5py_listing(path, group);
+    assert_eq!(ours.0.len(), theirs.0.len(), "dataset count in {group}");
+    assert_eq!(ours, theirs, "listing of {group}");
+}
+
+#[test]
+fn dense_group_whose_heap_outgrows_the_root_direct_rows() {
+    skip_if_no_python!();
+    // Long link names make the link heap larger than the root indirect
+    // block's direct rows can hold (512 KiB with h5py's defaults), so links
+    // live in child indirect blocks. Those were sized from the wrong row
+    // count, and every link past the direct rows was unreachable.
+    let (_dir, path) = h5py_file(
+        "t = f.create_dataset('t', data=[1.0])\n\
+         g = f.create_group('g')\n\
+         for i in range(2500):\n\
+         \x20   g['n%05d_' % i + 'x' * 240] = t\n",
+    );
+    assert_same_listing(&path, "g");
+    let f = File::open(&path).unwrap();
+    let last = format!("g/n02499_{}", "x".repeat(240));
+    assert_eq!(f.dataset(&last).unwrap().read_f64().unwrap(), vec![1.0]);
+}
+
+#[test]
+fn dense_group_with_a_three_level_name_index() {
+    skip_if_no_python!();
+    // 24 000 links give the link-name v2 B-tree a depth of 3. Internal-node
+    // child pointers carry the subtree's total record count in a width that
+    // depends on the most records a subtree can hold; the reader estimated
+    // that as leaf_max^depth, read the root's pointers 3 bytes wide instead
+    // of 2, and decoded garbage heap IDs.
+    let (_dir, path) = h5py_file(
+        "t = f.create_dataset('t', data=[1.0])\n\
+         g = f.create_group('g')\n\
+         for i in range(24000):\n\
+         \x20   g['l%06d' % i] = t\n",
+    );
+    assert_same_listing(&path, "g");
+    let f = File::open(&path).unwrap();
+    assert_eq!(
+        f.dataset("g/l023999").unwrap().read_f64().unwrap(),
+        vec![1.0]
+    );
+}
+
+/// The attribute names h5py reports for `obj`, sorted.
+fn h5py_attr_names(path: &str, obj: &str) -> Vec<String> {
+    let out = run_python(&format!(
+        "import h5py\n\
+         with h5py.File(r'{path}', 'r') as f:\n\
+         \x20   print('\\x1f'.join(sorted(f[{obj:?}].attrs.keys())))\n"
+    ));
+    out.split('\x1f')
+        .filter(|s| !s.is_empty())
+        .map(str::to_string)
+        .collect()
+}
+
+#[test]
+fn dense_attribute_stored_as_a_huge_heap_object() {
+    skip_if_no_python!();
+    // More than 8 attributes puts them in dense storage; one larger than the
+    // heap's 4 KiB managed-object limit is stored as a "huge" object, outside
+    // the heap blocks and found through the huge-object v2 B-tree. Its heap ID
+    // (type bits 4-5 = 1) was misread as a managed ID, and the error made
+    // every attribute on the object unreadable. NetCDF-4 files hit this
+    // (netcdf-c's issue671.nc / issue672.nc).
+    let (_dir, path) = h5py_file(
+        "d = f.create_dataset('d', data=[1.0])\n\
+         for i in range(10):\n\
+         \x20   d.attrs['a%d' % i] = i\n\
+         d.attrs['big'] = np.arange(1024, dtype='f8')\n\
+         d.attrs['bigger'] = np.arange(20000, dtype='i8') * 3\n",
+    );
+    let f = File::open(&path).unwrap();
+    let attrs = f.dataset("d").unwrap().attrs().unwrap();
+    let mut names: Vec<String> = attrs.keys().cloned().collect();
+    names.sort();
+    assert_eq!(names, h5py_attr_names(&path, "d"));
+    for i in 0..10 {
+        assert!(
+            matches!(attrs[&format!("a{i}")], AttrValue::I64(v) if v == i),
+            "a{i}: {:?}",
+            attrs[&format!("a{i}")]
+        );
+    }
+    let big: Vec<f64> = (0..1024).map(f64::from).collect();
+    assert!(matches!(&attrs["big"], AttrValue::F64Array(v) if *v == big));
+    let bigger: Vec<i64> = (0..20000).map(|v| v * 3).collect();
+    assert!(matches!(&attrs["bigger"], AttrValue::I64Array(v) if *v == bigger));
+}
+
+/// A group whose link heap has a deflate I/O filter (set on the group
+/// creation property list), with 3 000 links and one link whose message is
+/// larger than the heap's managed-object limit, so it is a huge object.
+fn huge_link_group(filtered: bool) -> (tempfile::TempDir, String) {
+    let filter = if filtered {
+        "import ctypes, glob, os\n\
+         lib = ctypes.CDLL(glob.glob(os.path.join(os.path.dirname(h5py.__file__), '..', 'h5py.libs', 'libhdf5-*.so*'))[0])\n\
+         lib.H5Pset_deflate.argtypes = [ctypes.c_int64, ctypes.c_uint]\n\
+         assert lib.H5Pset_deflate(gcpl.id, 6) >= 0\n"
+    } else {
+        ""
+    };
+    h5py_file(&format!(
+        "t = f.create_dataset('t', data=[1.0])\n\
+         gcpl = h5py.h5p.create(h5py.h5p.GROUP_CREATE)\n\
+         {filter}\
+         h5py.h5g.create(f.id, b'g', gcpl=gcpl)\n\
+         g = f['g']\n\
+         for i in range(3000):\n\
+         \x20   g['l%05d' % i] = t\n\
+         g['L' * 5000] = t\n"
+    ))
+}
+
+fn check_huge_link_group(filtered: bool) {
+    let (_dir, path) = huge_link_group(filtered);
+    assert_same_listing(&path, "g");
+    let f = File::open(&path).unwrap();
+    let huge = format!("g/{}", "L".repeat(5000));
+    assert_eq!(f.dataset(&huge).unwrap().read_f64().unwrap(), vec![1.0]);
+    assert_eq!(
+        f.dataset("g/l02999").unwrap().read_f64().unwrap(),
+        vec![1.0]
+    );
+}
+
+#[test]
+fn dense_group_with_a_huge_link() {
+    skip_if_no_python!();
+    check_huge_link_group(false);
+}
+
+#[test]
+fn dense_group_with_a_filtered_link_heap() {
+    skip_if_no_python!();
+    // libhdf5 applies a group's filter pipeline to its link heap: direct
+    // blocks and huge objects are stored deflated, and the heap header
+    // carries the pipeline. The header's checksum was looked for in the
+    // wrong place, and filtered blocks were read raw.
+    if run_python(
+        "import h5py, glob, os\nprint(len(glob.glob(os.path.join(os.path.dirname(h5py.__file__), '..', 'h5py.libs', 'libhdf5-*.so*'))))",
+    ) == "0"
+    {
+        assert!(
+            !interop_required(),
+            "CLAWHDF5_REQUIRE_INTEROP=1 but h5py's bundled libhdf5 was not found"
+        );
+        eprintln!("SKIP: h5py's bundled libhdf5 not found (needed to set the filter)");
+        return;
+    }
+    check_huge_link_group(true);
+}
+
+fn fixture(name: &str) -> String {
+    format!("{}/tests/fixtures/{name}", env!("CARGO_MANIFEST_DIR"))
+}
+
+/// `tall.h5` and `tudlink.h5` are libhdf5's own tool test files
+/// (`tools/test/testfiles`, BSD-style HDF5 licence). Each has user-defined
+/// links of class 187, which h5py lists by name but cannot open, and h5dump
+/// prints as `USERDEFINED_LINK`. One such link made the whole group
+/// unlistable (`InvalidLinkType(187)`); it is now left out of the listing
+/// like any other link that cannot be followed.
+#[test]
+fn user_defined_links_do_not_break_the_listing() {
+    let f = File::open(fixture("tall.h5")).unwrap();
+    let g2 = f.group("g2").unwrap();
+    let mut ds = g2.datasets().unwrap();
+    ds.sort();
+    assert_eq!(ds, ["dset2.1", "dset2.2"]);
+    assert!(g2.groups().unwrap().is_empty());
+    assert!(f.dataset("g2/udlink").is_err());
+
+    let f = File::open(fixture("tudlink.h5")).unwrap();
+    assert!(f.root().datasets().unwrap().is_empty());
+    assert!(f.root().groups().unwrap().is_empty());
+}
+
+/// Soft links (absolute, relative, to a dataset and to a group), a dangling
+/// one, a cycle and an external link, in an old-style (symbol table) or
+/// new-style (link message) group.
+fn soft_link_file(libver: &str) -> (tempfile::TempDir, String) {
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("soft.h5").display().to_string();
+    run_python(&format!(
+        "import h5py, numpy as np\n\
+         with h5py.File(r'{path}', 'w', libver='{libver}') as f:\n\
+         \x20   f.create_dataset('a/b/deep', data=np.arange(3.0))\n\
+         \x20   f.create_dataset('plain', data=[1.0])\n\
+         \x20   f['soft_ds'] = h5py.SoftLink('/a/b/deep')\n\
+         \x20   f['soft_grp'] = h5py.SoftLink('/a')\n\
+         \x20   f['a/rel'] = h5py.SoftLink('b/deep')\n\
+         \x20   f['a/rel_grp'] = h5py.SoftLink('b')\n\
+         \x20   f['dangling'] = h5py.SoftLink('/nope')\n\
+         \x20   f['loop1'] = h5py.SoftLink('/loop2')\n\
+         \x20   f['loop2'] = h5py.SoftLink('/loop1')\n\
+         \x20   f['ext'] = h5py.ExternalLink('elsewhere.h5', '/x')\n"
+    ));
+    (dir, path)
+}
+
+fn check_soft_links(libver: &str) {
+    let (_dir, path) = soft_link_file(libver);
+    assert_same_listing(&path, "/");
+    assert_same_listing(&path, "a");
+
+    let f = File::open(&path).unwrap();
+    let root = f.root();
+    let deep = vec![0.0, 1.0, 2.0];
+    assert_eq!(root.dataset("soft_ds").unwrap().read_f64().unwrap(), deep);
+    let a = root.group("soft_grp").unwrap();
+    assert_eq!(a.dataset("rel").unwrap().read_f64().unwrap(), deep);
+    assert_eq!(a.group("rel_grp").unwrap().datasets().unwrap(), ["deep"]);
+    // A dangling link is not listed and cannot be opened.
+    assert!(root.dataset("dangling").is_err());
+    assert!(root.dataset("loop1").is_err());
+
+    // The memory-mapped and lazy handles list the same way.
+    let (ds, gs) = h5py_listing(&path, "/");
+    let m = clawhdf5::MmapFile::open(&path).unwrap();
+    let mut mds = m.root().datasets().unwrap();
+    let mut mgs = m.root().groups().unwrap();
+    mds.sort();
+    mgs.sort();
+    assert_eq!((mds, mgs), (ds.clone(), gs.clone()));
+    let l = clawhdf5::LazyFile::from_bytes(std::fs::read(&path).unwrap()).unwrap();
+    let mut lds = l.root().datasets().unwrap();
+    let mut lgs = l.root().groups().unwrap();
+    lds.sort();
+    lgs.sort();
+    assert_eq!((lds, lgs), (ds, gs));
+}
+
+/// Soft links were left out of `datasets()`/`groups()` (and could not be
+/// opened by name from a group handle); in old-style groups, where a soft
+/// link has no object header address, they made the listing fail. h5py
+/// lists a soft link under its own name as whatever it points at.
+#[test]
+fn soft_links_are_listed_as_their_targets() {
+    skip_if_no_python!();
+    check_soft_links("latest");
+    check_soft_links("earliest");
+}
+
+/// One unreadable attribute used to fail `attrs()` for every attribute on
+/// the object. Now it is left out (and reported by `attrs_with_errors`),
+/// and the others are returned with their full values.
+#[test]
+fn one_unreadable_attribute_does_not_hide_the_others() {
+    skip_if_no_python!();
+    let (dir, path) = h5py_file(
+        "d = f.create_dataset('d', data=[1.0])\n\
+         for i in range(10):\n\
+         \x20   d.attrs['a%d' % i] = float(i)\n\
+         d.attrs['zz_broken_attribute'] = 42.0\n",
+    );
+    // Dense attributes live in a fractal heap whose blocks carry no checksum
+    // by default: give the one named `zz_broken_attribute` a nonexistent
+    // attribute message version (the byte 9 before its name in a v3 message).
+    let mut bytes = std::fs::read(&path).unwrap();
+    let needle = b"zz_broken_attribute";
+    let at = bytes
+        .windows(needle.len())
+        .position(|w| w == needle)
+        .expect("attribute name in the file");
+    assert_eq!(bytes[at - 9], 3, "expected a version-3 attribute message");
+    bytes[at - 9] = 0x7f;
+    let broken = dir.path().join("broken.h5");
+    std::fs::write(&broken, &bytes).unwrap();
+
+    for file in [
+        File::open(&broken).unwrap(),
+        File::from_bytes(bytes.clone()).unwrap(),
+    ] {
+        let ds = file.dataset("d").unwrap();
+        let (attrs, errors) = ds.attrs_with_errors().unwrap();
+        assert_eq!(errors.len(), 1, "{errors:?}");
+        assert_eq!(attrs.len(), 10);
+        for i in 0..10 {
+            assert!(
+                matches!(attrs[&format!("a{i}")], AttrValue::F64(v) if v == f64::from(i)),
+                "a{i}"
+            );
+        }
+        assert!(!attrs.contains_key("zz_broken_attribute"));
+        assert_eq!(ds.attrs().unwrap().len(), 10);
+    }
+    let m = clawhdf5::MmapFile::open(&broken).unwrap();
+    assert_eq!(
+        m.dataset("d").unwrap().attrs_with_errors().unwrap().1.len(),
+        1
+    );
+    let l = clawhdf5::LazyFile::from_bytes(bytes).unwrap();
+    assert_eq!(l.dataset("d").unwrap().attrs().unwrap().len(), 10);
+}

crates/clawhdf5/tests/fixtures/vds/README.md

@@ -0,0 +1,14 @@
+# VDS test files from libhdf5
+
+Copied unchanged from the HDF Group's HDF5 repository,
+`tools/test/testfiles/vds/` (the h5dump/h5ls VDS test data). HDF5 is
+distributed under a BSD-style license (see `COPYING` in the HDF5 source).
+
+| File | What it exercises |
+|---|---|
+| `vds-eiger.h5` + `f-0.h5`, `f-3.h5` | printf-style source name `f-%b.h5`; `f-3.h5` lies past the first missing source and must be ignored (extent 5, not 20) |
+| `4_vds.h5` + `4_0.h5`..`4_2.h5` | printf-style `4_%b.h5` with version-2 (1.10 format) hyperslab selections |
+| `vds-percival-unlim-maxmin.h5` + `a.h5`..`d.h5` | four interleaved unlimited mappings whose sources have different lengths |
+
+Used by `crates/clawhdf5/tests/vds_interop.rs::vds_libhdf5_test_files`,
+which compares our reads with h5py's.

crates/clawhdf5/tests/legacy_format_interop.rs

@@ -0,0 +1,220 @@
+//! Files written by HDF5 1.4/1.6-era libraries: Data Layout message versions
+//! 1 and 2, compound datatype version 1 array members, and version-1 shared
+//! message references. The fixtures are HDF5's own test files (see
+//! `clawhdf5-format/tests/fixtures/legacy/README.md`).
+//!
+//! The expected values were read with h5py 3.16 / HDF5 2.0; the interop test
+//! re-checks every dataset byte for byte against h5py, and is skipped when
+//! python3 with h5py is unavailable unless `CLAWHDF5_REQUIRE_INTEROP=1`.
+
+use std::process::Command;
+
+use clawhdf5::{DType, File};
+use clawhdf5_format::selection::Selection;
+
+const FIXTURES: &str = concat!(
+    env!("CARGO_MANIFEST_DIR"),
+    "/../clawhdf5-format/tests/fixtures/legacy"
+);
+
+fn open(name: &str) -> File {
+    File::open(format!("{FIXTURES}/{name}")).unwrap()
+}
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+fn python_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+/// Layout v1, chunked (50x50 chunks of a 100x200 dataset), deflate: every
+/// read path goes through the version-1 B-tree chunk index.
+#[test]
+fn layout_v1_chunked_deflate() {
+    let file = open("deflate.h5");
+    let ds = file.dataset("Dataset1").unwrap();
+    assert_eq!(ds.shape().unwrap(), [100, 200]);
+    let expected: Vec<i32> = (0..100).flat_map(|_| (0..200).map(|j| j % 5)).collect();
+    assert_eq!(ds.read_i32().unwrap(), expected);
+
+    // A hyperslab that straddles four chunks.
+    let slab = Selection::Hyperslab {
+        start: vec![48, 48],
+        stride: vec![1, 1],
+        count: vec![4, 4],
+        block: vec![1, 1],
+    };
+    let raw = ds.read_selection(&slab).unwrap();
+    let got: Vec<i32> = raw
+        .as_chunks::<4>()
+        .0
+        .iter()
+        .map(|b| i32::from_le_bytes(*b))
+        .collect();
+    assert_eq!(got, [3, 4, 0, 1, 3, 4, 0, 1, 3, 4, 0, 1, 3, 4, 0, 1]);
+}
+
+/// Layout v2, contiguous: one dataset with storage, one never written (reads
+/// as its fill value, 0).
+#[test]
+fn layout_v2_contiguous() {
+    let file = open("h5ex_g_iterate.h5");
+    assert_eq!(file.dataset("G1/DS2").unwrap().read_i32().unwrap(), [1]);
+    assert_eq!(file.dataset("DS1").unwrap().read_i32().unwrap(), [0]);
+}
+
+/// Compound datatype version 1 members carrying legacy array dimensions
+/// (HDF5 before 1.4 had no array class). h5py: `[('i', '<i2'), ('f', '<f4',
+/// (4,)), ('l', '<i4', (4,)), ('d', '<f8')]`, itemsize 44.
+#[test]
+fn compound_v1_legacy_array_members() {
+    let file = open("tarrold.h5");
+    let ds = file.dataset("Dataset2").unwrap();
+    assert_eq!(
+        ds.dtype().unwrap(),
+        DType::Compound(vec![
+            ("i".into(), DType::I16),
+            ("f".into(), DType::Array(Box::new(DType::F32), vec![4])),
+            ("l".into(), DType::Array(Box::new(DType::I32), vec![4])),
+            ("d".into(), DType::F64),
+        ])
+    );
+    assert_eq!(ds.shape().unwrap(), [8, 9]);
+    assert_eq!(
+        ds.read_selection(&Selection::All).unwrap().len(),
+        8 * 9 * 44
+    );
+}
+
+/// Datasets whose committed datatype is referenced by a version-1 shared
+/// message, whose object header address follows a link-name offset. Values
+/// from h5py; the file is big-endian.
+#[test]
+fn shared_message_v1_committed_datatypes() {
+    let file = open("tcompound.h5");
+    let be_pairs = |name: &str| -> Vec<(i32, f32)> {
+        file.dataset(name)
+            .unwrap()
+            .read_selection(&Selection::All)
+            .unwrap()
+            .as_chunks::<8>()
+            .0
+            .iter()
+            .map(|b| {
+                (
+                    i32::from_be_bytes(b[..4].try_into().unwrap()),
+                    f32::from_be_bytes(b[4..].try_into().unwrap()),
+                )
+            })
+            .collect()
+    };
+    assert_eq!(
+        be_pairs("group1/dset2"),
+        [(0, 0.0), (1, 1.1), (2, 2.2), (3, 3.3), (4, 4.4)]
+    );
+    assert_eq!(
+        be_pairs("group2/dset5"),
+        [(0, 0.0), (1, 0.1), (2, 0.2), (3, 0.3), (4, 0.4)]
+    );
+
+    // `/type2`: { int_array: i32[4], float_array: f32[5][6] }, whose array
+    // members are compound v1 legacy dimensions.
+    let dset3 = file.dataset("group1/dset3").unwrap();
+    assert_eq!(
+        dset3.dtype().unwrap(),
+        DType::Compound(vec![
+            (
+                "int_array".into(),
+                DType::Array(Box::new(DType::I32), vec![4])
+            ),
+            (
+                "float_array".into(),
+                DType::Array(Box::new(DType::F32), vec![5, 6])
+            ),
+        ])
+    );
+    let raw = dset3.read_selection(&Selection::All).unwrap();
+    assert_eq!(raw.len(), 3 * 6 * (16 + 120));
+    assert_eq!(
+        &raw[..16],
+        &[0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3]
+    );
+    let first: Vec<f32> = raw[16..16 + 120]
+        .as_chunks::<4>()
+        .0
+        .iter()
+        .map(|b| f32::from_be_bytes(*b))
+        .collect();
+    let expected: Vec<f32> = (0..5)
+        .flat_map(|i| (0..6).map(move |j| (1 + i + j) as f32))
+        .collect();
+    assert_eq!(first, expected);
+}
+
+/// Every dataset in every fixture, byte for byte against h5py.
+#[test]
+fn legacy_fixtures_match_h5py() {
+    if !python_available() {
+        assert!(
+            !interop_required(),
+            "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+        );
+        eprintln!("SKIP: python3 with h5py not available");
+        return;
+    }
+    for (name, datasets) in [
+        ("deflate.h5", &["Dataset1"][..]),
+        ("h5ex_g_iterate.h5", &["DS1", "G1/DS2"][..]),
+        ("tarrold.h5", &["Dataset1", "Dataset2"][..]),
+        (
+            "tcompound.h5",
+            &[
+                "dset1",
+                "group1/dset2",
+                "group1/dset3",
+                "group1/dset4",
+                "group2/dset5",
+            ][..],
+        ),
+    ] {
+        let path = format!("{FIXTURES}/{name}");
+        let script = format!(
+            r#"
+import h5py, numpy as np
+f = h5py.File({path:?}, "r")
+for n in {datasets:?}:
+    print(n, np.ascontiguousarray(f[n][()]).tobytes().hex())
+"#
+        );
+        let out = Command::new(python())
+            .args(["-c", &script])
+            .output()
+            .unwrap();
+        assert!(
+            out.status.success(),
+            "h5py: {}",
+            String::from_utf8_lossy(&out.stderr)
+        );
+        let file = File::open(&path).unwrap();
+        for line in String::from_utf8(out.stdout).unwrap().lines() {
+            let (ds, hex) = line.split_once(' ').unwrap();
+            let ours = file
+                .dataset(ds)
+                .unwrap()
+                .read_selection(&Selection::All)
+                .unwrap();
+            let ours: String = ours.iter().map(|b| format!("{b:02x}")).collect();
+            assert_eq!(ours, hex, "{name}:{ds}");
+        }
+    }
+}

crates/clawhdf5/tests/local_heap_interop.rs

@@ -0,0 +1,131 @@
+//! Old-style (symbol-table) groups keep link names in a local heap. libhdf5
+//! validates the heap's free list when it loads the heap and refuses the
+//! group ("bad heap free list") when the list points outside the heap; we
+//! must refuse too instead of listing names read from a broken heap. Like
+//! libhdf5, the check happens when a name is needed, so an empty group with
+//! a broken heap still lists.
+//!
+//! h5py writes the files; skipped when python3 with h5py is unavailable,
+//! unless `CLAWHDF5_REQUIRE_INTEROP=1`.
+
+use std::process::Command;
+
+use clawhdf5::File;
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+fn python_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+macro_rules! skip_if_no_python {
+    () => {
+        if !python_available() {
+            assert!(
+                !interop_required(),
+                "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+            );
+            eprintln!("SKIP: python3 with h5py not available");
+            return;
+        }
+    };
+}
+
+fn run_python(script: &str) -> String {
+    let output = Command::new(python())
+        .args(["-c", script])
+        .output()
+        .expect("failed to run python");
+    assert!(
+        output.status.success(),
+        "python failed:\n{}",
+        String::from_utf8_lossy(&output.stderr)
+    );
+    String::from_utf8_lossy(&output.stdout).into_owned()
+}
+
+#[test]
+fn local_heap_free_list_checked_like_libhdf5() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let good = dir.path().join("good.h5");
+    // Writes `good.h5` (a deleted link leaves a real free block in the root
+    // group's heap) and two copies whose root heap free list is broken; for
+    // each prints what h5py lists, or `ERROR`.
+    let script = format!(
+        r#"
+import h5py, struct
+good = "{good}"
+with h5py.File(good, "w", libver="earliest") as f:
+    for name in ("alpha", "beta", "gamma"):
+        f.create_group(name)
+    del f["beta"]
+data = bytearray(open(good, "rb").read())
+heap = data.find(b"HEAP")  # the root group's heap is written first
+size, head, seg = struct.unpack_from("<QQQ", data, heap + 8)
+assert head != 1, "expected a free block"
+bad_head = bytearray(data)
+struct.pack_into("<Q", bad_head, heap + 16, size + 8)
+bad_block = bytearray(data)
+struct.pack_into("<Q", bad_block, seg + head + 8, size)  # block runs past the end
+# libhdf5 only loads a heap when it needs a name: an empty group with the
+# same damage still lists (as empty).
+empty = good.replace("good.h5", "empty_src.h5")
+with h5py.File(empty, "w", libver="earliest") as f:
+    pass
+bad_empty = bytearray(open(empty, "rb").read())
+eheap = bad_empty.find(b"HEAP")
+esize = struct.unpack_from("<Q", bad_empty, eheap + 8)[0]
+struct.pack_into("<Q", bad_empty, eheap + 16, esize + 8)
+for name, content in (("good", data), ("bad_head", bad_head), ("bad_block", bad_block),
+                      ("bad_empty", bad_empty)):
+    path = good.replace("good.h5", name + ".h5")
+    open(path, "wb").write(content)
+    try:
+        with h5py.File(path, "r") as f:
+            print(name, *sorted(f.keys()))
+    except Exception as e:
+        print(name, "ERROR")
+"#,
+        good = good.display()
+    );
+    let out = run_python(&script);
+    let lines: Vec<&str> = out.lines().collect();
+    assert_eq!(
+        lines,
+        [
+            "good alpha gamma",
+            "bad_head ERROR",
+            "bad_block ERROR",
+            "bad_empty"
+        ],
+        "h5py's view changed"
+    );
+
+    let file = File::open(&good).unwrap();
+    let mut groups = file.root().groups().unwrap();
+    groups.sort();
+    assert_eq!(groups, ["alpha", "gamma"]);
+
+    for name in ["bad_head", "bad_block"] {
+        let file = File::open(dir.path().join(format!("{name}.h5"))).unwrap();
+        let listed = file.root().groups();
+        assert!(
+            listed.is_err(),
+            "{name}: listed {listed:?} from a heap libhdf5 rejects"
+        );
+    }
+
+    let file = File::open(dir.path().join("bad_empty.h5")).unwrap();
+    assert_eq!(file.root().groups().unwrap(), Vec::<String>::new());
+}

crates/clawhdf5/tests/shared_message_v1.rs

@@ -0,0 +1,125 @@
+//! Version-1 shared messages (HDF5 1.6 era). A dataset that uses a committed
+//! datatype stores a *shared* datatype message pointing at the type's object
+//! header. In version 1 that pointer is a 1.6 "symbol table entry": after six
+//! reserved bytes comes a length-sized heap offset, *then* the address.
+//!
+//! Fixture: `tcompound.h5` from libhdf5's own tool tests
+//! (`tools/test/testfiles/tcompound.h5`, HDF5 source tree, BSD-style
+//! licence), 8 KiB. Its datasets use committed compound types through v1
+//! shared messages. The expected types are what h5dump 1.14.6 and h5py 3.16
+//! (HDF5 2.0) report; the h5py cross-check runs when python3 with h5py is
+//! available (required with `CLAWHDF5_REQUIRE_INTEROP=1`).
+
+use std::process::Command;
+
+use clawhdf5::{DType, File};
+
+const FIXTURE: &[u8] = include_bytes!("../../clawhdf5-format/tests/fixtures/tcompound.h5");
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+fn python_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+fn compound(fields: &[(&str, DType)]) -> DType {
+    DType::Compound(
+        fields
+            .iter()
+            .map(|(n, t)| (n.to_string(), t.clone()))
+            .collect(),
+    )
+}
+
+fn expected() -> Vec<(&'static str, DType)> {
+    let int_float = |i: &str, f: &str| compound(&[(i, DType::I32), (f, DType::F32)]);
+    vec![
+        ("group1/dset2", int_float("int_name", "float_name")),
+        (
+            "group1/dset3",
+            compound(&[
+                ("int_array", DType::Array(Box::new(DType::I32), vec![4])),
+                (
+                    "float_array",
+                    DType::Array(Box::new(DType::F32), vec![5, 6]),
+                ),
+            ]),
+        ),
+        ("group1/dset4", int_float("int", "float")),
+        ("group2/dset5", int_float("int", "float")),
+    ]
+}
+
+#[test]
+fn v1_shared_datatype_resolves_to_the_committed_type() {
+    // Reading the heap offset as the address used to land on the superblock
+    // and fail with InvalidObjectHeaderVersion.
+    let file = File::from_bytes(FIXTURE.to_vec()).unwrap();
+    for (path, dtype) in expected() {
+        assert_eq!(
+            file.dataset(path).unwrap().dtype().unwrap(),
+            dtype,
+            "{path}"
+        );
+    }
+}
+
+#[test]
+fn v1_shared_datatype_field_names_match_h5py() {
+    if !python_available() {
+        assert!(
+            !interop_required(),
+            "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+        );
+        eprintln!("SKIP: python3 with h5py not available");
+        return;
+    }
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("tcompound.h5");
+    std::fs::write(&path, FIXTURE).unwrap();
+    let script = format!(
+        r#"
+import h5py
+with h5py.File("{path}", "r") as f:
+    for p in ("group1/dset2", "group1/dset3", "group1/dset4", "group2/dset5"):
+        print(p, *f[p].dtype.names)
+"#,
+        path = path.display()
+    );
+    let out = Command::new(python())
+        .args(["-c", &script])
+        .output()
+        .unwrap();
+    assert!(
+        out.status.success(),
+        "{}",
+        String::from_utf8_lossy(&out.stderr)
+    );
+    let stdout = String::from_utf8_lossy(&out.stdout);
+    let theirs: Vec<&str> = stdout.lines().collect();
+    // Read the types through clawhdf5, not from `expected()`, so this checks
+    // our reader against libhdf5 rather than the table against h5py.
+    let file = File::from_bytes(FIXTURE.to_vec()).unwrap();
+    let ours: Vec<String> = expected()
+        .into_iter()
+        .map(|(p, _)| (p, file.dataset(p).unwrap().dtype().unwrap()))
+        .map(|(p, t)| match t {
+            DType::Compound(fields) => {
+                let names: Vec<String> = fields.into_iter().map(|(n, _)| n).collect();
+                format!("{p} {}", names.join(" "))
+            }
+            other => panic!("{other:?}"),
+        })
+        .collect();
+    assert_eq!(theirs, ours);
+}

crates/clawhdf5/tests/sohm_interop.rs

@@ -0,0 +1,141 @@
+//! Files with shared object header messages (SOHM: datatypes, dataspaces,
+//! filter pipelines and attributes stored once in a file-wide heap and
+//! referenced by heap ID), written by libhdf5 through h5py.
+//!
+//! Skipped when python3 with h5py is unavailable, unless
+//! `CLAWHDF5_REQUIRE_INTEROP=1`.
+
+use std::process::Command;
+
+use clawhdf5::{AttrValue, File};
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+fn python_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+fn run_python(script: &str) -> String {
+    let output = Command::new(python())
+        .args(["-c", script])
+        .output()
+        .expect("failed to run python");
+    if !output.status.success() {
+        panic!(
+            "Python script failed:\nSTDOUT: {}\nSTDERR: {}",
+            String::from_utf8_lossy(&output.stdout),
+            String::from_utf8_lossy(&output.stderr)
+        );
+    }
+    String::from_utf8_lossy(&output.stdout).trim().to_string()
+}
+
+/// h5py has no binding for the SOHM property-list calls, so they go through
+/// the libhdf5 that h5py bundles. `None` when that library is not found.
+fn sohm_file(path: &str, libver: &str, mesg_types: u32) -> Option<()> {
+    let out = run_python(&format!(
+        "import ctypes, glob, os, h5py, numpy as np\n\
+         libs = glob.glob(os.path.join(os.path.dirname(h5py.__file__), '..', 'h5py.libs', 'libhdf5-*.so*'))\n\
+         if not libs:\n\
+         \x20   print('nolib'); raise SystemExit\n\
+         lib = ctypes.CDLL(libs[0])\n\
+         lib.H5Pset_shared_mesg_nindexes.argtypes = [ctypes.c_int64, ctypes.c_uint]\n\
+         lib.H5Pset_shared_mesg_index.argtypes = [ctypes.c_int64, ctypes.c_uint, ctypes.c_uint, ctypes.c_uint]\n\
+         fcpl = h5py.h5p.create(h5py.h5p.FILE_CREATE)\n\
+         assert lib.H5Pset_shared_mesg_nindexes(fcpl.id, 1) >= 0\n\
+         assert lib.H5Pset_shared_mesg_index(fcpl.id, 0, {mesg_types}, 1) >= 0\n\
+         fapl = h5py.h5p.create(h5py.h5p.FILE_ACCESS)\n\
+         low = h5py.h5f.LIBVER_EARLIEST if '{libver}' == 'earliest' else h5py.h5f.LIBVER_LATEST\n\
+         fapl.set_libver_bounds(low, h5py.h5f.LIBVER_LATEST)\n\
+         fid = h5py.h5f.create(r'{path}'.encode(), h5py.h5f.ACC_TRUNC, fcpl=fcpl, fapl=fapl)\n\
+         with h5py.File(fid) as f:\n\
+         \x20   for i in range(4):\n\
+         \x20       ds = f.create_dataset('d%d' % i, shape=(50,), dtype='<f8', chunks=(10,), compression='gzip', fillvalue=-9.0)\n\
+         \x20       ds[0:20] = np.arange(20.0) + i\n\
+         \x20       ds.attrs['shared_attr'] = np.arange(10.0)\n\
+         \x20       ds.attrs['units'] = 'm/s'\n\
+         \x20   f.create_dataset('contig', data=np.arange(7, dtype='<i4') * 2)\n\
+         print('ok')\n"
+    ));
+    (out == "ok").then_some(())
+}
+
+/// Every message type libhdf5 can share (`H5O_SHMESG_ALL_FLAG`), and each on
+/// its own.
+const MESG_TYPES: [(u32, &str); 6] = [
+    (0x182A, "all"),
+    (0x02, "dataspace"),
+    (0x08, "datatype"),
+    (0x20, "fill value"),
+    (0x800, "filter pipeline"),
+    (0x1000, "attribute"),
+];
+
+/// A message shared through the SOHM heap was only resolved on the one path
+/// that loaded the SOHM table itself (shared fill values); datatypes,
+/// dataspaces, filter pipelines and attributes stored there failed with
+/// "invalid shared message version: 2", so such files' datasets could not be
+/// read at all.
+#[test]
+fn sohm_shared_messages_resolve() {
+    if !python_available() {
+        assert!(
+            !interop_required(),
+            "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+        );
+        eprintln!("SKIP: python3 with h5py not available");
+        return;
+    }
+    let dir = tempfile::tempdir().unwrap();
+    for libver in ["earliest", "latest"] {
+        for (flags, what) in MESG_TYPES {
+            let path = dir.path().join("sohm.h5").display().to_string();
+            if sohm_file(&path, libver, flags).is_none() {
+                assert!(
+                    !interop_required(),
+                    "CLAWHDF5_REQUIRE_INTEROP=1 but h5py's bundled libhdf5 was not found"
+                );
+                eprintln!("SKIP: h5py's bundled libhdf5 not found");
+                return;
+            }
+            let case = format!("{what}, libver {libver}");
+            let f = File::open(&path).unwrap();
+            let d2 = f.dataset("d2").unwrap_or_else(|e| panic!("{case}: {e}"));
+            let mut expected: Vec<f64> = (0..20).map(|v| f64::from(v) + 2.0).collect();
+            expected.resize(50, -9.0);
+            assert_eq!(
+                d2.read_f64().unwrap_or_else(|e| panic!("{case}: {e}")),
+                expected,
+                "{case}"
+            );
+            let (attrs, errors) = d2.attrs_with_errors().unwrap();
+            assert!(errors.is_empty(), "{case}: {errors:?}");
+            let shared: Vec<f64> = (0..10).map(f64::from).collect();
+            assert!(
+                matches!(&attrs["shared_attr"], AttrValue::F64Array(v) if *v == shared),
+                "{case}: {:?}",
+                attrs.get("shared_attr")
+            );
+            assert!(
+                matches!(&attrs["units"], AttrValue::String(s) if s == "m/s"),
+                "{case}: {:?}",
+                attrs.get("units")
+            );
+            assert_eq!(
+                f.dataset("contig").unwrap().read_i32().unwrap(),
+                [0, 2, 4, 6, 8, 10, 12],
+                "{case}"
+            );
+        }
+    }
+}

crates/clawhdf5/tests/userblock_interop.rs

@@ -0,0 +1,314 @@
+//! Files that start with a user block (`h5py.File(..., userblock_size=N)`,
+//! `h5jam`): the superblock sits at 512, 1024, ... and every address in the
+//! file is relative to it. Each reader (buffered, mmap, `MmapFile`,
+//! `LazyFile`) must apply that base, and read the same values h5py does.
+//!
+//! h5py writes the files; skipped when python3 with h5py is unavailable,
+//! unless `CLAWHDF5_REQUIRE_INTEROP=1`.
+
+use std::collections::HashMap;
+use std::path::Path;
+use std::process::Command;
+
+use clawhdf5::{AttrValue, File, LazyFile, MmapFile};
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+fn python_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+macro_rules! skip_if_no_python {
+    () => {
+        if !python_available() {
+            assert!(
+                !interop_required(),
+                "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+            );
+            eprintln!("SKIP: python3 with h5py not available");
+            return;
+        }
+    };
+}
+
+/// Run `script` and return its stdout as `key -> values` (one
+/// `key v1 v2 ...` line per key).
+fn run_python(script: &str) -> HashMap<String, Vec<String>> {
+    let output = Command::new(python())
+        .args(["-c", script])
+        .output()
+        .expect("failed to run python");
+    assert!(
+        output.status.success(),
+        "python failed:\n{}",
+        String::from_utf8_lossy(&output.stderr)
+    );
+    String::from_utf8_lossy(&output.stdout)
+        .lines()
+        .filter_map(|line| {
+            let mut words = line.split_whitespace().map(str::to_string);
+            Some((words.next()?, words.collect()))
+        })
+        .collect()
+}
+
+fn parse<T: std::str::FromStr>(values: &[String]) -> Vec<T>
+where
+    T::Err: std::fmt::Debug,
+{
+    values.iter().map(|v| v.parse().unwrap()).collect()
+}
+
+/// Write a file with a user block of `userblock` bytes holding contiguous,
+/// chunked (deflate), compact and committed-type datasets, nested groups,
+/// and attributes (compact and, under `latest`, dense). Prints what h5py
+/// reads back.
+fn write_file(path: &Path, userblock: u32, libver: &str) -> HashMap<String, Vec<String>> {
+    let script = format!(
+        r#"
+import h5py, numpy as np
+path = "{path}"
+with h5py.File(path, "w", userblock_size={userblock}, libver={libver}) as f:
+    f.attrs["title"] = "user block"
+    f.attrs["answer"] = np.int64(42)
+    f.create_dataset("contig", data=np.arange(12, dtype="<f8") * 0.5)
+    f.create_dataset("chunked", data=np.arange(1000, dtype="<i4") * 3 - 7,
+                     chunks=(128,), compression="gzip")
+    dcpl = h5py.h5p.create(h5py.h5p.DATASET_CREATE)
+    dcpl.set_layout(h5py.h5d.COMPACT)
+    space = h5py.h5s.create_simple((5,))
+    dsid = h5py.h5d.create(f.id, b"compact", h5py.h5t.STD_I64LE, space, dcpl=dcpl)
+    dsid.write(h5py.h5s.ALL, h5py.h5s.ALL, np.array([5, -4, 3, -2, 1], "<i8"))
+    f["named_type"] = np.dtype("<f4")
+    f.create_dataset("committed", data=np.array([1.25, -2.5], "<f4"),
+                     dtype=f["named_type"])
+    g = f.create_group("a/b")
+    g.create_dataset("deep", data=np.array([7, 8, 9], "<i8"))
+    g.attrs["scale"] = 2.5
+    d = f["contig"]
+    d.attrs["units"] = "m"
+    # Enough attributes that `latest` stores them densely (fractal heap).
+    for i in range(12):
+        f["a"].attrs["k%02d" % i] = np.int64(i * i)
+    # And enough links for a dense (fractal-heap) group under `latest`.
+    many = f.create_group("many")
+    for i in range(20):
+        many.create_dataset("d%02d" % i, data=np.array([i], "<i4"))
+with h5py.File(path, "r") as f:
+    print("userblock", f.userblock_size)
+    print("contig", *f["contig"][()])
+    print("chunked", *f["chunked"][()])
+    print("compact", *f["compact"][()])
+    print("committed", *f["committed"][()])
+    print("deep", *f["a/b/deep"][()])
+    print("many", *[int(f["many/d%02d" % i][0]) for i in range(20)])
+    print("k", *[int(f["a"].attrs["k%02d" % i]) for i in range(12)])
+"#,
+        path = path.display(),
+        libver = if libver == "default" {
+            "None".to_string()
+        } else {
+            format!("{libver:?}")
+        },
+    );
+    run_python(&script)
+}
+
+/// Attribute value rendered for comparison (`AttrValue` has no `PartialEq`).
+fn attr(map: &HashMap<String, AttrValue>, key: &str) -> String {
+    match map.get(key) {
+        Some(AttrValue::I64(v)) => format!("i64 {v}"),
+        Some(AttrValue::F64(v)) => format!("f64 {v}"),
+        Some(AttrValue::String(v)) => format!("str {v}"),
+        other => format!("{other:?}"),
+    }
+}
+
+fn i64s(v: &[String]) -> Vec<i64> {
+    parse(v)
+}
+
+/// Everything read through the `File` API must match h5py.
+fn check_file(file: &File, expected: &HashMap<String, Vec<String>>, label: &str) {
+    let ub: u64 = expected["userblock"][0].parse().unwrap();
+    assert_eq!(file.user_block_size(), ub, "{label}: user block size");
+    assert_eq!(
+        file.dataset("contig").unwrap().read_f64().unwrap(),
+        parse::<f64>(&expected["contig"]),
+        "{label}: contiguous"
+    );
+    assert_eq!(
+        file.dataset("chunked")
+            .unwrap()
+            .read_i32()
+            .unwrap()
+            .iter()
+            .map(|&v| v as i64)
+            .collect::<Vec<_>>(),
+        i64s(&expected["chunked"]),
+        "{label}: chunked"
+    );
+    assert_eq!(
+        file.dataset("compact").unwrap().read_i64().unwrap(),
+        i64s(&expected["compact"]),
+        "{label}: compact"
+    );
+    assert_eq!(
+        file.dataset("committed").unwrap().read_f32().unwrap(),
+        parse::<f32>(&expected["committed"]),
+        "{label}: committed datatype"
+    );
+    assert_eq!(
+        file.dataset("a/b/deep").unwrap().read_i64().unwrap(),
+        i64s(&expected["deep"]),
+        "{label}: nested group"
+    );
+    let many: Vec<i64> = (0..20)
+        .map(|i| {
+            file.dataset(&format!("many/d{i:02}"))
+                .unwrap()
+                .read_i32()
+                .unwrap()[0] as i64
+        })
+        .collect();
+    assert_eq!(many, i64s(&expected["many"]), "{label}: many links");
+
+    let root = file.root().attrs().unwrap();
+    assert_eq!(attr(&root, "title"), "str user block", "{label}");
+    assert_eq!(attr(&root, "answer"), "i64 42", "{label}");
+    let a = file.group("a").unwrap().attrs().unwrap();
+    let k: Vec<i64> = (0..12)
+        .map(|i| match &a[&format!("k{i:02}")] {
+            AttrValue::I64(v) => *v,
+            _ => panic!("{label}: k{i:02} is not an i64"),
+        })
+        .collect();
+    assert_eq!(k, i64s(&expected["k"]), "{label}: attributes");
+    assert_eq!(
+        attr(&file.group("a/b").unwrap().attrs().unwrap(), "scale"),
+        "f64 2.5",
+        "{label}"
+    );
+    assert_eq!(
+        attr(&file.dataset("contig").unwrap().attrs().unwrap(), "units"),
+        "str m",
+        "{label}"
+    );
+}
+
+fn check_all_readers(path: &Path, expected: &HashMap<String, Vec<String>>, label: &str) {
+    check_file(
+        &File::open(path).unwrap(),
+        expected,
+        &format!("{label} File::open"),
+    );
+    check_file(
+        &File::open_buffered(path).unwrap(),
+        expected,
+        &format!("{label} File::open_buffered"),
+    );
+    check_file(
+        &File::from_bytes(std::fs::read(path).unwrap()).unwrap(),
+        expected,
+        &format!("{label} File::from_bytes"),
+    );
+
+    let ub: u64 = expected["userblock"][0].parse().unwrap();
+
+    let mm = MmapFile::open(path).unwrap();
+    assert_eq!(mm.user_block_size(), ub, "{label} MmapFile");
+    assert_eq!(
+        mm.dataset("contig").unwrap().read_f64().unwrap(),
+        parse::<f64>(&expected["contig"]),
+        "{label} MmapFile contiguous"
+    );
+    assert_eq!(
+        mm.dataset("compact").unwrap().read_i64().unwrap(),
+        i64s(&expected["compact"]),
+        "{label} MmapFile compact"
+    );
+    assert_eq!(
+        mm.dataset("committed").unwrap().read_f32().unwrap(),
+        parse::<f32>(&expected["committed"]),
+        "{label} MmapFile committed"
+    );
+    assert_eq!(
+        mm.dataset("a/b/deep").unwrap().read_i64().unwrap(),
+        i64s(&expected["deep"]),
+        "{label} MmapFile nested"
+    );
+    assert_eq!(
+        attr(&mm.root().attrs().unwrap(), "answer"),
+        "i64 42",
+        "{label} MmapFile attrs"
+    );
+
+    let lazy = LazyFile::open_mmap(path).unwrap();
+    assert_eq!(lazy.user_block_size(), ub, "{label} LazyFile");
+    assert_eq!(
+        lazy.dataset("contig").unwrap().read_f64().unwrap(),
+        parse::<f64>(&expected["contig"]),
+        "{label} LazyFile contiguous"
+    );
+    assert_eq!(
+        lazy.dataset("chunked")
+            .unwrap()
+            .read_i32()
+            .unwrap()
+            .iter()
+            .map(|&v| v as i64)
+            .collect::<Vec<_>>(),
+        i64s(&expected["chunked"]),
+        "{label} LazyFile chunked"
+    );
+    assert_eq!(
+        lazy.dataset("committed").unwrap().read_f32().unwrap(),
+        parse::<f32>(&expected["committed"]),
+        "{label} LazyFile committed"
+    );
+    assert_eq!(
+        lazy.dataset("a/b/deep").unwrap().read_i64().unwrap(),
+        i64s(&expected["deep"]),
+        "{label} LazyFile nested"
+    );
+    assert_eq!(
+        attr(&lazy.root().attrs().unwrap(), "answer"),
+        "i64 42",
+        "{label} LazyFile attrs"
+    );
+}
+
+#[test]
+fn user_block_files_read_like_h5py() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    for userblock in [512u32, 4096] {
+        for libver in ["default", "latest"] {
+            let label = format!("userblock={userblock} libver={libver}");
+            let path = dir.path().join(format!("ub_{userblock}_{libver}.h5"));
+            let expected = write_file(&path, userblock, libver);
+            assert_eq!(expected["userblock"], [userblock.to_string()], "{label}");
+            check_all_readers(&path, &expected, &label);
+        }
+    }
+}
+
+#[test]
+fn file_without_user_block_reports_zero() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("no_ub.h5");
+    let expected = write_file(&path, 0, "default");
+    assert_eq!(expected["userblock"], ["0"]);
+    check_all_readers(&path, &expected, "userblock=0");
+}

crates/clawhdf5/tests/vds_interop.rs

@@ -0,0 +1,488 @@
+//! Virtual Dataset (VDS) reads checked against libhdf5 (through h5py).
+//!
+//! Each test has h5py build virtual datasets and their source files in a temp
+//! directory, record what libhdf5 reads back (shape and values) next to them,
+//! and then compares that with what clawhdf5 reads from the same files.
+//!
+//! Skipped when python3 or h5py is unavailable, unless
+//! `CLAWHDF5_REQUIRE_INTEROP=1`.
+
+use std::path::Path;
+use std::process::Command;
+
+use clawhdf5::File;
+
+/// The Python interpreter to drive interop checks with (`CLAWHDF5_PYTHON`
+/// lets these run against a virtualenv holding h5py).
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+/// When `CLAWHDF5_REQUIRE_INTEROP=1` (set in CI), a missing Python dependency
+/// is a test failure instead of a silent skip.
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
+fn python_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py, numpy"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+macro_rules! skip_if_no_python {
+    () => {
+        if !python_available() {
+            assert!(
+                !interop_required(),
+                "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+            );
+            eprintln!("SKIP: python3 with h5py not available");
+            return;
+        }
+    };
+}
+
+/// Prelude for every generator script: `expect(file, dset, tag)` records what
+/// libhdf5 reads for `file:dset` as `<tag>.expect` (shape line, values line).
+const PRELUDE: &str = r#"
+import h5py, numpy as np
+def expect(fn, dset, tag):
+    with h5py.File(fn, "r") as f:
+        d = f[dset]
+        a = d[...]
+        with open(tag + ".expect", "w") as out:
+            out.write(" ".join(str(n) for n in d.shape) + "\n")
+            out.write(" ".join(repr(float(v)) for v in a.ravel()) + "\n")
+"#;
+
+/// Run `body` (after [`PRELUDE`]) with `dir` as the working directory, so
+/// relative source file names land next to the virtual file.
+fn generate(dir: &Path, body: &str) {
+    let script = format!("{PRELUDE}\n{body}");
+    let out = Command::new(python())
+        .args(["-c", &script])
+        .current_dir(dir)
+        .output()
+        .expect("failed to run python");
+    assert!(
+        out.status.success(),
+        "generator failed:\nSTDOUT: {}\nSTDERR: {}",
+        String::from_utf8_lossy(&out.stdout),
+        String::from_utf8_lossy(&out.stderr)
+    );
+}
+
+/// What libhdf5 read for `tag`: (shape, values as f64).
+fn expected(dir: &Path, tag: &str) -> (Vec<u64>, Vec<f64>) {
+    let text = std::fs::read_to_string(dir.join(format!("{tag}.expect"))).unwrap();
+    let mut lines = text.lines();
+    let parse_line = |l: Option<&str>| -> Vec<String> {
+        l.unwrap_or("")
+            .split_whitespace()
+            .map(str::to_string)
+            .collect()
+    };
+    let shape = parse_line(lines.next())
+        .iter()
+        .map(|s| s.parse().unwrap())
+        .collect();
+    let values = parse_line(lines.next())
+        .iter()
+        .map(|s| s.parse().unwrap())
+        .collect();
+    (shape, values)
+}
+
+/// Assert clawhdf5 reads `file:dset` exactly as libhdf5 did for `tag`.
+fn assert_matches_libhdf5(dir: &Path, file: &str, dset: &str, tag: &str) {
+    let (shape, values) = expected(dir, tag);
+    let f = File::open(dir.join(file)).unwrap();
+    let ds = f.dataset(dset).unwrap();
+    assert_eq!(
+        ds.shape().unwrap(),
+        shape,
+        "{tag}: shape differs from libhdf5"
+    );
+    let got = ds
+        .read_f64()
+        .unwrap_or_else(|e| panic!("{tag}: read failed: {e}"));
+    assert_eq!(got.len(), values.len(), "{tag}: element count differs");
+    for (i, (g, e)) in got.iter().zip(&values).enumerate() {
+        assert!(
+            g == e || (g.is_nan() && e.is_nan()),
+            "{tag}: element {i} is {g}, libhdf5 reads {e}\n ours: {got:?}\n libhdf5: {values:?}"
+        );
+    }
+}
+
+// ---------------------------------------------------------------------------
+// Selection encodings
+// ---------------------------------------------------------------------------
+
+/// Files written with the default (earliest) format bounds serialize every
+/// VDS hyperslab as a version-1 *irregular* selection (4-byte block corners),
+/// and a strided selection as many blocks. These were refused outright.
+#[test]
+fn vds_version1_irregular_hyperslab_selections() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    generate(
+        dir.path(),
+        r#"
+with h5py.File("src.h5", "w") as s:
+    s.create_dataset("a", data=np.arange(12.0))
+    s.create_dataset("m", data=np.arange(20.0).reshape(4, 5))
+with h5py.File("v1.h5", "w") as f:   # default libver: hyperslab version 1
+    f.create_dataset("local", data=np.arange(10.0) * -1)
+    lay = h5py.VirtualLayout(shape=(12,), dtype="f8")
+    lay[0:4] = h5py.VirtualSource(".", "local", shape=(10,))[2:6]
+    lay[4:10] = h5py.VirtualSource("src.h5", "a", shape=(12,))[::2]
+    lay[10:12] = h5py.VirtualSource("src.h5", "a", shape=(12,))[10:12]
+    f.create_virtual_dataset("strided", lay)
+    lay = h5py.VirtualLayout(shape=(4, 6), dtype="f8")
+    lay[:, 0:2] = h5py.VirtualSource("src.h5", "m", shape=(4, 5))[:, 3:5]
+    lay[:, 2:6:2] = h5py.VirtualSource("src.h5", "m", shape=(4, 5))[:, 0:2]
+    lay[:, 3:6:2] = h5py.VirtualSource("src.h5", "m", shape=(4, 5))[:, 2:4]
+    f.create_virtual_dataset("grid", lay)
+expect("v1.h5", "strided", "strided")
+expect("v1.h5", "grid", "grid")
+"#,
+    );
+    assert_matches_libhdf5(dir.path(), "v1.h5", "strided", "strided");
+    assert_matches_libhdf5(dir.path(), "v1.h5", "grid", "grid");
+}
+
+// ---------------------------------------------------------------------------
+// Mapping list encoding
+// ---------------------------------------------------------------------------
+
+/// With a 2.0 low version bound libhdf5 writes the mapping list as block
+/// version 1: a flags byte per entry, and repeated names stored as the index
+/// of the entry that first spelled them out. The flags byte was mistaken for
+/// an empty (same-file) name.
+#[test]
+fn vds_mapping_block_version1_shared_names() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    generate(
+        dir.path(),
+        r#"
+name = "a_rather_long_dataset_name"
+with h5py.File("a_rather_long_source_file.h5", "w") as s:
+    s.create_dataset(name, data=np.arange(12.0) + 100)
+with h5py.File("shared.h5", "w", libver=("v200", "v200")) as f:
+    f.create_dataset(name, data=np.arange(12.0) * -1)
+    lay = h5py.VirtualLayout(shape=(4, 4), dtype="f8")
+    for i in range(3):
+        src = h5py.VirtualSource("a_rather_long_source_file.h5", name, shape=(12,))
+        lay[i] = src[4 * i:4 * i + 4]
+    lay[3] = h5py.VirtualSource(".", name, shape=(12,))[0:4]
+    f.create_virtual_dataset("v", lay)
+# the heap block must really be version 1 for this test to mean anything
+raw = open("shared.h5", "rb").read()
+gcol = raw.index(b"GCOL")
+assert raw[gcol + 32] == 1, "expected a version-1 VDS mapping block"
+expect("shared.h5", "v", "shared")
+"#,
+    );
+    assert_matches_libhdf5(dir.path(), "shared.h5", "v", "shared");
+}
+
+// ---------------------------------------------------------------------------
+// Fill value
+// ---------------------------------------------------------------------------
+
+/// Elements no mapping supplies read as the virtual dataset's fill value, not
+/// as 0: unmapped regions, a missing source file, a missing source dataset.
+/// A source's own unallocated chunks read as *its* fill value.
+#[test]
+fn vds_unmapped_regions_read_as_fill_value() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    generate(
+        dir.path(),
+        r#"
+for i in range(3):
+    with h5py.File(f"src_{i}.h5", "w") as s:
+        s.create_dataset("data", data=np.arange(10.0) + i * 100)
+with h5py.File("sparse_src.h5", "w") as s:
+    d = s.create_dataset("data", shape=(10,), chunks=(5,), dtype="f8", fillvalue=42.0)
+    d[0:5] = np.arange(5.0) + 1000   # the second chunk is never written
+for libver in ["earliest", "latest"]:
+    with h5py.File(f"fill_{libver}.h5", "w", libver=libver) as f:
+        f.create_dataset("local", data=np.arange(10.0) * -1)
+        lay = h5py.VirtualLayout(shape=(6, 10), dtype="f8")
+        for i in range(3):
+            lay[i] = h5py.VirtualSource(f"src_{i}.h5", "data", shape=(10,))
+        lay[3] = h5py.VirtualSource("no_such_file.h5", "data", shape=(10,))
+        lay[4] = h5py.VirtualSource("src_0.h5", "no_such_dataset", shape=(10,))
+        # row 5 is not mapped at all
+        f.create_virtual_dataset("files", lay, fillvalue=-1.0)
+        lay = h5py.VirtualLayout(shape=(20,), dtype="f8")
+        lay[0:10] = h5py.VirtualSource(".", "local", shape=(10,))
+        f.create_virtual_dataset("same_file", lay, fillvalue=7.0)
+        lay = h5py.VirtualLayout(shape=(12,), dtype="f8")
+        lay[1:11] = h5py.VirtualSource("sparse_src.h5", "data", shape=(10,))
+        f.create_virtual_dataset("sparse_source", lay, fillvalue=-3.5)
+        lay = h5py.VirtualLayout(shape=(3, 4), dtype="i4")
+        lay[1, :] = h5py.VirtualSource(".", "ints", shape=(4,))
+        f.create_dataset("ints", data=np.arange(4, dtype="i4") + 1)
+        f.create_virtual_dataset("int_fill", lay, fillvalue=-99)
+    for name in ["files", "same_file", "sparse_source", "int_fill"]:
+        expect(f"fill_{libver}.h5", name, f"{name}_{libver}")
+"#,
+    );
+    for libver in ["earliest", "latest"] {
+        let file = format!("fill_{libver}.h5");
+        for name in ["files", "same_file", "sparse_source", "int_fill"] {
+            assert_matches_libhdf5(dir.path(), &file, name, &format!("{name}_{libver}"));
+        }
+    }
+
+    // A selection read goes through the same fill-aware assembly.
+    let f = File::open(dir.path().join("fill_latest.h5")).unwrap();
+    let sel = clawhdf5::Selection::slice(std::slice::from_ref(&(8..14)));
+    let got = f
+        .dataset("same_file")
+        .unwrap()
+        .read_f64_selection(&sel)
+        .unwrap();
+    assert_eq!(got, vec![-8.0, -9.0, 7.0, 7.0, 7.0, 7.0]);
+}
+
+/// A source name that would leave the virtual file's directory is refused
+/// with an error; it used to be skipped and read silently as fill.
+#[test]
+fn vds_source_outside_directory_is_an_error_not_fill() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    std::fs::create_dir(dir.path().join("sub")).unwrap();
+    generate(
+        dir.path(),
+        r#"
+with h5py.File("src.h5", "w") as s:
+    s.create_dataset("data", data=np.arange(4.0))
+with h5py.File("sub/up.h5", "w", libver="latest") as f:
+    lay = h5py.VirtualLayout(shape=(4,), dtype="f8")
+    lay[:] = h5py.VirtualSource("../src.h5", "data", shape=(4,))
+    f.create_virtual_dataset("v", lay, fillvalue=-1.0)
+with h5py.File("nested.h5", "w", libver="latest") as f:
+    lay = h5py.VirtualLayout(shape=(4,), dtype="f8")
+    lay[:] = h5py.VirtualSource("sub/inner.h5", "data", shape=(4,))
+    f.create_virtual_dataset("v", lay, fillvalue=-1.0)
+with h5py.File("sub/inner.h5", "w") as s:
+    s.create_dataset("data", data=np.arange(4.0) + 10)
+expect("nested.h5", "v", "nested")
+"#,
+    );
+    // libhdf5 resolves "../src.h5" (and would read [0, 1, 2, 3]); we refuse
+    // to leave the directory, and say so.
+    let f = File::open(dir.path().join("sub/up.h5")).unwrap();
+    let err = f.dataset("v").unwrap().read_f64().unwrap_err();
+    assert!(
+        err.to_string().contains("not followed"),
+        "unexpected error: {err}"
+    );
+    // A relative name below the virtual file's directory resolves there.
+    assert_matches_libhdf5(dir.path(), "nested.h5", "v", "nested");
+}
+
+/// Variable-length and reference elements are addresses into their own file.
+/// Copied raw from an external source they would be decoded against the
+/// virtual dataset's file and name another object, so they are refused.
+#[test]
+fn vds_external_variable_length_source_is_an_error_not_foreign_addresses() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    generate(
+        dir.path(),
+        r#"
+st = h5py.string_dtype()
+with h5py.File("src.h5", "w") as s:
+    s.create_dataset("names", data=np.array(["alpha", "beta", "gamma"], dtype=object), dtype=st)
+    s.create_dataset("refs", data=[s.ref, s.ref], dtype=h5py.ref_dtype)
+with h5py.File("v.h5", "w", libver="latest") as f:
+    f.create_dataset("pad", data=np.arange(64.0))
+    lay = h5py.VirtualLayout(shape=(3,), dtype=st)
+    lay[:] = h5py.VirtualSource("src.h5", "names", shape=(3,))
+    f.create_virtual_dataset("names", lay)
+    lay = h5py.VirtualLayout(shape=(2,), dtype=h5py.ref_dtype)
+    lay[:] = h5py.VirtualSource("src.h5", "refs", shape=(2,))
+    f.create_virtual_dataset("refs", lay)
+"#,
+    );
+    let f = File::open(dir.path().join("v.h5")).unwrap();
+    for name in ["names", "refs"] {
+        let err = f
+            .dataset(name)
+            .unwrap()
+            .read_selection(&clawhdf5_format::selection::Selection::All)
+            .expect_err("raw addresses from another file must not be returned");
+        assert!(
+            err.to_string().contains("from another file"),
+            "{name}: unexpected error: {err}"
+        );
+    }
+}
+
+// ---------------------------------------------------------------------------
+// Unlimited and printf-style mappings
+// ---------------------------------------------------------------------------
+
+/// Helpers for building unlimited VDS mappings through h5py's low-level API.
+const UNLIMITED_HELPERS: &str = r#"
+U = h5py.h5s.UNLIMITED
+def space(dims, maxdims, start=None, count=None, stride=None, block=None):
+    s = h5py.h5s.create_simple(dims, maxdims)
+    if start is not None:
+        s.select_hyperslab(start, count, stride, block)
+    return s
+def make_vds(fn, name, dims, maxdims, maps, fill, libver="latest", mode="w"):
+    # maps: [(vsel_kwargs, source_file, source_dataset, source_space)]
+    with h5py.File(fn, mode, libver=libver) as f:
+        dcpl = h5py.h5p.create(h5py.h5p.DATASET_CREATE)
+        for vsel, sfile, sdset, sspace in maps:
+            dcpl.set_virtual(space(dims, maxdims, **vsel), sfile.encode(), sdset.encode(), sspace)
+        dcpl.set_fill_value(np.array(fill, dtype="f8"))
+        h5py.h5d.create(f.id, name.encode(), h5py.h5t.IEEE_F64LE,
+                        h5py.h5s.create_simple(dims, maxdims), dcpl=dcpl)
+"#;
+
+/// printf-style names: block `j` of the virtual selection comes from the
+/// source named with `j` in place of `%b` (`%%` is a literal `%`), probing
+/// j = 0, 1, ... until the first missing source. libhdf5 also recomputes the
+/// extent from what it finds, so the stored dataspace is not the shape.
+#[test]
+fn vds_printf_source_names() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let body = format!(
+        "{UNLIMITED_HELPERS}{}",
+        r#"
+for i in [0, 1, 2, 4]:   # 3 is missing: 4 is past the first gap and unused
+    with h5py.File(f"vds_src_{i}.h5", "w") as s:
+        s.create_dataset("data", data=np.arange(10.0) + i * 100)
+    with h5py.File(f"p%c_{i}.h5", "w") as s:
+        s.create_dataset("data", data=np.arange(10.0) - i * 100)
+for libver in ["earliest", "latest"]:
+    fn = f"printf_{libver}.h5"
+    make_vds(fn, "files", (10,), (U,),
+             [(dict(start=(0,), count=(U,), stride=(10,), block=(10,)), "vds_src_%b.h5", "data",
+               space((10,), (10,), (0,), (1,), (1,), (10,)))], -1.0, libver)
+    # interleaved blocks with gaps between them, and an escaped percent sign
+    make_vds(fn, "escaped", (4,), (U,),
+             [(dict(start=(1,), count=(U,), stride=(6,), block=(4,)), "p%%c_%b.h5", "data",
+               space((10,), (10,), (2,), (1,), (1,), (4,)))], -5.0, libver, "a")
+    # printf in the dataset name, same file, 2-D frames
+    with h5py.File(fn, "a") as f:
+        for i in range(3):
+            f.create_dataset(f"frame_{i}", data=np.arange(6.0).reshape(2, 3) + 10 * i)
+    with h5py.File(fn, "a", libver=libver) as f:
+        dcpl = h5py.h5p.create(h5py.h5p.DATASET_CREATE)
+        dcpl.set_virtual(space((1, 2, 3), (U, 2, 3), (0, 0, 0), (U, 1, 1), (1, 1, 1), (1, 2, 3)),
+                         b".", b"frame_%b", space((2, 3), (2, 3)))
+        h5py.h5d.create(f.id, b"frames", h5py.h5t.IEEE_F64LE,
+                        h5py.h5s.create_simple((1, 2, 3), (U, 2, 3)), dcpl=dcpl)
+    for name in ["files", "escaped", "frames"]:
+        expect(fn, name, f"{name}_{libver}")
+"#
+    );
+    generate(dir.path(), &body);
+    for libver in ["earliest", "latest"] {
+        for name in ["files", "escaped", "frames"] {
+            let file = format!("printf_{libver}.h5");
+            assert_matches_libhdf5(dir.path(), &file, name, &format!("{name}_{libver}"));
+        }
+    }
+    // The shape libhdf5 reports: three 10-element blocks.
+    let f = File::open(dir.path().join("printf_latest.h5")).unwrap();
+    assert_eq!(f.dataset("files").unwrap().shape().unwrap(), vec![30]);
+}
+
+/// Unlimited source and virtual selections: each mapping covers as much as
+/// its source's current extent fills (a partial last block included), the
+/// extent is the largest of them but never smaller than the limited
+/// mappings need, and a missing source contributes nothing.
+#[test]
+fn vds_unlimited_mappings_follow_source_extents() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let body = format!(
+        "{UNLIMITED_HELPERS}{}",
+        r#"
+with h5py.File("grow.h5", "w") as s:
+    s.create_dataset("a", data=np.arange(7.0) + 1, maxshape=(None,))
+    s.create_dataset("b", data=np.arange(5.0) + 100, maxshape=(None,))
+    s.create_dataset("rows", data=np.arange(12.0).reshape(4, 3) + 50, maxshape=(None, 3))
+unlim_src = lambda: space((1,), (U,), (0,), (U,), (1,), (1,))
+for libver in ["earliest", "latest"]:
+    fn = f"unlim_{libver}.h5"
+    make_vds(fn, "interleaved", (1,), (U,), [
+        # blocks of 3 every 4: 7 source elements end mid-block
+        (dict(start=(0,), count=(U,), stride=(4,), block=(3,)), "grow.h5", "a", unlim_src()),
+        (dict(start=(3,), count=(U,), stride=(4,), block=(1,)), "grow.h5", "b", unlim_src()),
+        (dict(start=(0,), count=(U,), stride=(1,), block=(1,)), "missing.h5", "a", unlim_src()),
+    ], -2.0, libver)
+    make_vds(fn, "rows", (6, 3), (U, 3), [
+        # an unlimited *block*, plus a limited mapping reaching row 5
+        (dict(start=(0, 0), count=(1, 1), stride=(1, 1), block=(U, 3)), "grow.h5", "rows",
+         space((1, 3), (U, 3), (0, 0), (1, 1), (1, 1), (U, 3))),
+        (dict(start=(5, 0), count=(1, 1), stride=(1, 1), block=(1, 3)), "grow.h5", "rows",
+         space((4, 3), (U, 3), (1, 0), (1, 1), (1, 1), (1, 3))),
+    ], -4.0, libver, "a")
+    for name in ["interleaved", "rows"]:
+        expect(fn, name, f"{name}_{libver}")
+"#
+    );
+    generate(dir.path(), &body);
+    for libver in ["earliest", "latest"] {
+        for name in ["interleaved", "rows"] {
+            let file = format!("unlim_{libver}.h5");
+            assert_matches_libhdf5(dir.path(), &file, name, &format!("{name}_{libver}"));
+        }
+    }
+    // "a" (7 elements, blocks of 3 every 4) ends at 9; "b" (5 elements from
+    // 3, every 4) at 20. "rows" fills 4 rows but a limited mapping needs 6.
+    let f = File::open(dir.path().join("unlim_latest.h5")).unwrap();
+    assert_eq!(f.dataset("interleaved").unwrap().shape().unwrap(), vec![20]);
+    assert_eq!(f.dataset("rows").unwrap().shape().unwrap(), vec![6, 3]);
+}
+
+/// libhdf5's own VDS test files (HDF5 `tools/test/testfiles/vds`):
+/// printf-style Eiger frames (with a source past the first gap that must be
+/// ignored), a printf mapping in the 1.10 format, and Percival's four
+/// interleaved unlimited sources of different lengths.
+#[test]
+fn vds_libhdf5_test_files() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let fixtures = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/vds");
+    for entry in std::fs::read_dir(&fixtures).unwrap() {
+        let path = entry.unwrap().path();
+        if path.extension().is_some_and(|e| e == "h5") {
+            std::fs::copy(&path, dir.path().join(path.file_name().unwrap())).unwrap();
+        }
+    }
+    let cases = [
+        ("vds-eiger.h5", "/VDS-Eiger"),
+        ("4_vds.h5", "/vds_dset"),
+        ("vds-percival-unlim-maxmin.h5", "/VDS-Percival-unlim-maxmin"),
+    ];
+    let mut body = String::new();
+    for (i, (file, dset)) in cases.iter().enumerate() {
+        body.push_str(&format!("expect({file:?}, {dset:?}, \"case{i}\")\n"));
+    }
+    generate(dir.path(), &body);
+    for (i, (file, dset)) in cases.iter().enumerate() {
+        assert_matches_libhdf5(dir.path(), file, dset, &format!("case{i}"));
+    }
+    // Stored as 20 frames; only f-0.h5 is found before the first gap.
+    let f = File::open(dir.path().join("vds-eiger.h5")).unwrap();
+    assert_eq!(
+        f.dataset("VDS-Eiger").unwrap().shape().unwrap(),
+        vec![5, 10, 10]
+    );
+}

docs/known-issues.md

@@ -57,30 +57,76 @@ After it, 448 read correctly and 23 differ. Of those 23:
   byte-swapped in h5py, and h5dump agrees with us.
 - The rest are object or attribute listing differences.
 
+**Update 2026-09-25:** the sweep is now in the repo (`conformance/run.sh`,
+corpora pinned by commit) and its current numbers are in `CONFORMANCE.md`,
+regenerated nightly by `.gitea/workflows/conformance.yml`. The probe now
+compares N-Bit floats as the values libhdf5 converts them to, so the N-Bit
+files above count as identical. Its file list is defined by
+`conformance/list_files.py` (697 files: netCDF classic files are left out,
+and 11 HDF5 files the ad-hoc sweep missed are in). On 42b81d9: 467 identical,
+123 our-error, 15 mismatch (2 are the h5py bug above), 92 that libhdf5 cannot
+read, and no panics, hangs or crashes.
+
 There were no panics, hangs or crashes before or after, including on all 147
 CVE and fuzzer files. On some of those files, h5dump 1.14.6 and h5py/HDF5 2.0
 segfault or abort.
 
 ## Gaps found by the 2026-09-25 HDF5 audit (open)
 
-**Status:** open. These fail with an error; none returns wrong data, except
-the VDS item, which is marked.
+**Status:** open. These fail with an error; none returns wrong data (the VDS
+fill-value item that did is fixed).
 
 - **Layout message versions 1 and 2** (HDF5 1.6-era files): 84 of the 686
   sweep files, `InvalidLayoutVersion`. This is the largest single gap.
+  **Fixed 2026-09-25:** versions 1 and 2 are parsed (compact, contiguous,
+  chunked via the v1 B-tree).
+- **Compound datatype version 1 array members** (found with the layout
+  fix; pre-1.4 files such as `tarrold.h5`): **wrong data** — the legacy
+  per-member dimensions were skipped, so an array member read as one scalar.
+  **Fixed 2026-09-25.**
 - **Virtual datasets:**
-  - **Wrong data:** unmapped regions read as 0 instead of the fill value.
-  - `%b` printf-style source names are not expanded.
-  - Hyperslab selection versions 1 and 2 are refused.
+  - ~~**Wrong data:** unmapped regions read as 0 instead of the fill value.~~
+    Fixed 2026-09-25: unmapped elements and missing sources read as the
+    virtual dataset's fill value.
+  - ~~`%b` printf-style source names are not expanded.~~ Fixed 2026-09-25:
+    printf-style and unlimited mappings are read, and the extent is
+    recomputed from the sources as libhdf5 does. Still open: the
+    "first missing" view and a printf gap other than 0 (libhdf5 access
+    properties we always read at their defaults), source-to-virtual type
+    conversion other than a byte swap, nested virtual sources, and source
+    files outside the virtual file's directory (refused with an error).
+  - ~~Hyperslab selection versions 1 and 2 are refused.~~ Fixed 2026-09-25:
+    versions 1-3 and irregular hyperslabs are decoded.
+  - ~~The version-1 mapping list written with a 2.0 low bound (flags byte,
+    shared names) was misparsed.~~ Found and fixed 2026-09-25.
 - **Files with a user block:** the base address is not applied.
+  **Fixed 2026-09-25:** every reader views the file from the superblock on
+  (`twithub.h5`, `twithub513.h5`, `h5clear_fsm_persist_user_*.h5`; the
+  `twithub` files still stop at the user-defined link type below).
 - **Old-style shared messages (version 1)** read the wrong address.
+  **Fixed 2026-09-25:** the address follows the length-sized link-name
+  offset of the embedded symbol table entry (`tcompound.h5`, `tcompound2.h5`).
 - **Groups and links:**
   - Groups with a user-defined link type (e.g. 187) cannot be listed.
+    **Fixed 2026-09-25:** user-defined links are skipped; the rest of the
+    group lists.
   - Dense groups with more than about 22 000 links cannot be listed.
-  - Soft links are left out of `datasets()`.
+    **Fixed 2026-09-25:** two bugs — fractal-heap child indirect blocks had
+    the wrong row count, and v2 B-tree internal nodes at depth 3+ were read
+    with the wrong pointer widths.
+  - Soft links are left out of `datasets()`. **Fixed 2026-09-25:** soft
+    links are listed as their targets; dangling ones are left out.
+  - **Wrong data (found while fixing user blocks):** an old-style group whose
+    local-heap free list points outside the heap listed garbage names where
+    libhdf5 refuses the heap. **Fixed 2026-09-25**
+    (`InvalidLocalHeapFreeList`, checked when a name is first read, as
+    libhdf5 does).
 - **Dense attributes:** a large attribute stored as a fractal-heap "huge"
   object makes every attribute on the object fail. This affects real NetCDF
-  files (`issue671.nc`).
+  files (`issue671.nc`). **Fixed 2026-09-25:** huge and tiny heap objects,
+  and filtered heaps, are read; and an attribute that still cannot be read
+  is left out of `attrs()` (reported by `attrs_with_errors()`) instead of
+  failing the others.
 - **Other readers:**
   - VL-string datasets are not readable through `File`.
   - Metadata cache images are not supported.