docs: record the 2026-09-25 HDF5 audit fixes and open gaps

CHANGELOG: upgrade notes (changed read results for max-shape files,
saturating conversions, new writer errors, format-crate API changes) and
the reader/writer correctness fixes. known-issues: the silent-wrong-data
table with before/after sweep numbers, the gaps still open, and a
correction to the Extensible Array entry, which said files we wrote were
unaffected. CLAUDE.md: clawhdf5-gpu is vector distance computation, not
I/O, and clawhdf5-filters holds only deflate backends (no Blosc).

Also a facade test that libhdf5's 20-bit N-Bit float test data reads as
libhdf5's values.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

CHANGELOG.md

@@ -3,6 +3,38 @@
 ## Unreleased
 
 ### Upgrade Notes
+- **HDF5 correctness audit (2026-09-25).** A sweep of 686 public files (the
+  libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c,
+  netcdf4-python, h5wasm, h5py and xarray corpora), a 567-case read matrix and
+  a 96-case write matrix against HDF5 1.10–2.0 found bugs that returned wrong
+  values with no error, and files we wrote that libhdf5 rejects. The fixes are
+  listed under Correctness and Interop. What changes for callers:
+  - **Chunked datasets whose max shape is larger than their current shape**,
+    or whose unlimited dimension is not the first, were indexed by the current
+    shape instead of the max shape, both when read and when written. Files from
+    libhdf5 now read correctly. Files clawhdf5 wrote with such a max shape were
+    laid out wrongly and now read the way libhdf5 always read them — rewrite
+    them. Agent stores and ClawBrainHub files have no max shape and are
+    unaffected.
+  - Integer reads (`read_i32`/`read_i64`/`read_u64`/...) of float data now
+    convert (truncate toward zero, saturate at the type's range, NaN reads as
+    0) instead of returning the IEEE bit pattern, and out-of-range integers
+    saturate instead of keeping the low bits.
+  - `FileWriter::finish()` now returns an error instead of writing a corrupt
+    file for: a header message over 64 KiB (e.g. an attribute larger than
+    ~64 KiB), a group/dataset/link name that is empty, `.` or contains `/`
+    (nested paths were written as one literal link), a max shape smaller than
+    the shape, a page size outside 512 B–1 GiB, and more than 65 535 chunks in
+    a dataset with several unlimited dimensions.
+  - **Breaking (format crate):** `ObjectHeaderWriter::serialize`,
+    `BatchObjectHeaderWriter::compute_sizes`/`serialize_all` and
+    `build_chunked_data_from_precompressed` return `Result`;
+    `read_fixed_array_chunks`/`read_extensible_array_chunks` take `max_dims`;
+    `build_fixed_array_at`/`ea_writer::build_extensible_array_at` take one
+    `Option<WrittenChunk>` per index slot; `fill_value::dataset_fill_value`
+    returns `UnresolvedSharedMessage` for a shared message it cannot resolve
+    instead of `None`. `FillTime::default()` is `IfSet` (libhdf5's default;
+    default files are byte-identical).
 - **ZeroClaw does not use clawhdf5.** The project described itself as
   ZeroClaw's memory backend ("imported as a `clawhdf5` Cargo feature"). Checked
   against ZeroClaw v0.8.5 (the latest release), the `osobh/zeroclaw` fork and
@@ -241,6 +273,53 @@
 - CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
 
 ### Correctness
+- `clawhdf5-format` reader — **values returned wrong with no error:**
+  - Fixed Array and Extensible Array chunk indexes were laid out by the
+    dataset's current shape instead of its max shape (23 libhdf5 test files,
+    and any h5py file with e.g. `maxshape=(10, None)` or `(20, 10)` under
+    `libver='latest'`).
+  - Files with 4-byte offsets: unfiltered chunked datasets read as zeros.
+    Chunk B-tree keys store offsets in 8 bytes whatever the file's offset
+    size.
+  - A chunk's filter mask skipped the whole pipeline when any bit was set;
+    only the flagged filters are skipped now.
+  - Float data read as an integer returned the bit pattern; narrowing integer
+    reads kept the low bits; bfloat16 was decoded as IEEE half. Floats are now
+    decoded from their datatype fields (bf16, FP8 E4M3/E5M2, IEEE half, single
+    and double).
+  - `vl_data::read_vl_bytes` truncated sequences of non-byte base types.
+  - A shared fill-value message read as zero fill; it is resolved now,
+    including from the file's shared-message (SOHM) table, which could never
+    resolve because its index version byte was skipped.
+  - Two threads reading two chunked datasets through one `File` could get each
+    other's chunks (the shared chunk cache was switched between datasets
+    across separate lock acquisitions). The cache is now keyed by dataset.
+- `clawhdf5-format` reader — errors on valid files: enum and bool datasets
+  through the numeric readers; the "don't filter partial edge chunks" layout
+  flag; Fletcher32 ahead of deflate (NetCDF-4's order). Unknown-message flags
+  follow libhdf5 (`tbogus.h5`): "fail if unknown" is refused, "fail if unknown
+  and writing" is ignored by a reader.
+- `clawhdf5-format` writer — **files libhdf5 rejects or reads wrong:**
+  - Extensible Array (one unlimited dimension): chunks from index 244 on were
+    written but never indexed and read as 0, by libhdf5 and by us.
+  - Fixed Array: more than 1 024 chunks gave checksum errors (data blocks
+    were never paged).
+  - A finite max shape larger than the shape gave libhdf5 "addr overflow"; an
+    unlimited dimension that is not the first scrambled the data; several
+    unlimited dimensions (`(None, None)`) broke the whole file. These now
+    write the index libhdf5 writes (swizzled Extensible Array, or a B-tree v2
+    index for several unlimited dimensions).
+  - Header messages over 64 KiB (the size field is 16 bits) and compact
+    datasets at 65 534–65 535 bytes produced corrupt files.
+  - Reference, Opaque, BitField and Time datatypes were written as empty
+    messages; they now encode as HDF5 2.0 does.
+  - `with_page_size` wrote a nonexistent superblock version 4; it now writes
+    the v3 superblock and File Space Info message libhdf5 writes.
+  - `FillTime` values were rotated on disk (NEVER was written as ALLOC, and so
+    on). New `DatasetBuilder::with_fill_value`.
+  - An empty-string attribute got a zero-size datatype, which made every
+    attribute on the object unreadable in libhdf5.
+  - `maxshape` equal to the shape no longer forces chunked layout.
 - `clawhdf5-format`: **a truncated deflate chunk read back short, with no
   error.** The deflate filter used flate2's streaming reader, which returns the
   bytes it has when the input runs out before the end-of-stream marker. It now

CLAUDE.md

@@ -1,7 +1,7 @@
 # clawhdf5
 
 ## Purpose
-Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
+Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
 
 ## Architecture
 
@@ -11,13 +11,13 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
 |-------|------|
 | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
 | `clawhdf5-io` | Read/write implementation |
-| `clawhdf5-filters` | Compression filters (gzip, LZ4, Zstd, Blosc) |
+| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec) live in `clawhdf5-format`. No Blosc. |
 | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
 | `clawhdf5` | Main facade crate |
 | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
 | `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
 | `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
-| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
+| `clawhdf5-gpu` | GPU vector distance computation via wgpu (hand-written WGSL compute shaders) — not dataset I/O |
 | `clawhdf5-accel` | CPU SIMD acceleration path |
 | `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
 | `clawhdf5-android` | Android JNI bindings |
@@ -148,7 +148,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
   Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
   `MemorySource` for this bookkeeping is inferred from the caller-supplied
   `source_channel` string (a heuristic, not an authenticated trust boundary).
-- GPU-accelerated batch I/O for large dataset processing
+- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
 - Python and Node.js bindings for cross-language use
 - NetCDF-4 compatibility for scientific data interop
 

crates/clawhdf5/tests/numeric_conversion_interop.rs

@@ -372,3 +372,29 @@ with h5py.File("{path}", "r") as f:
         }
     }
 }
+
+/// libhdf5's N-Bit float test data is stored as a 20-bit custom float
+/// (`le_data.h5` from the HDF5 test suite). The N-Bit filter restores the
+/// file type's bytes; the typed reader must then decode that layout the way
+/// libhdf5 converts it (h5py reads 0.3333435, 0.666687, 1, ...).
+#[test]
+fn nbit_custom_float_decodes_like_libhdf5() {
+    let path = concat!(
+        env!("CARGO_MANIFEST_DIR"),
+        "/../clawhdf5-format/tests/fixtures/filters/le_data.h5"
+    );
+    let f = File::open(path).unwrap();
+    // Exactly representable in the 20-bit type, so exact in f32 and f64.
+    let expected = [
+        0.333343505859375,
+        0.66668701171875,
+        1.0,
+        1.3333740234375,
+        1.6666259765625,
+        2.0,
+    ];
+    for name in ["Nbit_float_data_le", "Nbit_float_data_be"] {
+        let got = f.dataset(name).unwrap().read_f64().unwrap();
+        assert_eq!(&got[..6], &expected, "{name}");
+    }
+}

docs/known-issues.md

@@ -7,6 +7,99 @@ deleting it.
 
 ---
 
+## Silent wrong data found by the 2026-09-25 HDF5 audit
+
+**Status:** fixed after v2.7.0 (2026-09-25). **Every release up
+to and including v2.7.0 is affected.**
+
+An audit on tank checked clawhdf5 against libhdf5 in three ways:
+- a sweep of 686 public files: the libhdf5 test files, the HDF Group's
+  `cve_hdf5` reproducers, and the pyfive, netcdf-c, netcdf4-python, h5wasm,
+  h5py and xarray corpora;
+- 567 read cases generated with h5py 3.16 / HDF5 2.0;
+- 96 write cases checked with h5py builds linking HDF5 1.10, 1.12, 1.14 and
+  2.0, plus h5dump 1.14.6.
+
+It found these cases where a value came back wrong **without an error**:
+
+| Area | What happened | Who is affected |
+|---|---|---|
+| Chunk index (read) | Fixed/Extensible Array indexes laid out by the current shape, not the max shape: chunks returned from the wrong place | any file with a max shape larger than its shape and `libver='latest'` (h5py `maxshape=(10, None)`, `(20, 10)`) |
+| Chunk index (write) | Extensible Array chunks from index 244 on never indexed (read as 0); unlimited dimension not first: data scrambled | files we wrote with one unlimited dimension and > 244 chunks, or e.g. `maxshape=(20, None)` |
+| 4-byte offsets | unfiltered chunked datasets read as zeros | files created with `sizeof_addr = 4` |
+| Filter mask | any skipped filter skipped the whole pipeline | files with partially filtered chunks (optional filters, direct chunk writes) |
+| Numeric reads | float read as integer returned the bit pattern; narrowing integer reads kept the low bits; bfloat16 decoded as IEEE half | `read_i32`/`read_i64`/`read_u64` callers on float or wider data; HDF5 2.0 bf16 data |
+| SZIP | garbage or zeros | every libhdf5-written SZIP dataset |
+| Scale-offset | float values 1 ULP off | libhdf5 D-scale float data |
+| Shared fill value | read as zero fill | fill values stored as shared messages |
+| VL sequences | `read_vl_bytes` truncated non-byte base types | VL int/float sequences |
+| Chunk cache | two threads reading two chunked datasets through one `File` could get each other's chunks | multi-threaded readers, including Python with the GIL released |
+
+The audit also found files we wrote that libhdf5 **refuses**, now fixed:
+- Fixed Array datasets with more than 1 024 chunks.
+- Header messages over 64 KiB (large attributes).
+- Reference, Opaque, BitField and Time datatypes.
+- Files written with `with_page_size`.
+- Several unlimited dimensions.
+- A finite max shape larger than the shape.
+- An empty-string attribute, which broke every attribute on its object.
+- `FillTime` codes, which were rotated.
+
+Our LZ4 and Zstd output could not be read by libhdf5's registered plugins, and
+our pcodec filter used Granular BitRound's ID. The details are in
+`CHANGELOG.md` under Correctness and Interop.
+
+Before the fix, 419 of the 686 files read correctly and 43 differed from h5py.
+After it, 448 read correctly and 23 differ. Of those 23:
+- 17 are N-Bit float files. The probe compares raw file-type bytes; the typed
+  reader returns libhdf5's values (`nbit_custom_float_decodes_like_libhdf5`).
+- 2 are an h5py bug: VL data with a big-endian base type comes back
+  byte-swapped in h5py, and h5dump agrees with us.
+- The rest are object or attribute listing differences.
+
+There were no panics, hangs or crashes before or after, including on all 147
+CVE and fuzzer files. On some of those files, h5dump 1.14.6 and h5py/HDF5 2.0
+segfault or abort.
+
+## Gaps found by the 2026-09-25 HDF5 audit (open)
+
+**Status:** open. These fail with an error; none returns wrong data, except
+the VDS item, which is marked.
+
+- **Layout message versions 1 and 2** (HDF5 1.6-era files): 84 of the 686
+  sweep files, `InvalidLayoutVersion`. This is the largest single gap.
+- **Virtual datasets:**
+  - **Wrong data:** unmapped regions read as 0 instead of the fill value.
+  - `%b` printf-style source names are not expanded.
+  - Hyperslab selection versions 1 and 2 are refused.
+- **Files with a user block:** the base address is not applied.
+- **Old-style shared messages (version 1)** read the wrong address.
+- **Groups and links:**
+  - Groups with a user-defined link type (e.g. 187) cannot be listed.
+  - Dense groups with more than about 22 000 links cannot be listed.
+  - Soft links are left out of `datasets()`.
+- **Dense attributes:** a large attribute stored as a fractal-heap "huge"
+  object makes every attribute on the object fail. This affects real NetCDF
+  files (`issue671.nc`).
+- **Other readers:**
+  - VL-string datasets are not readable through `File`.
+  - Metadata cache images are not supported.
+  - x87 long double and binary128 are refused.
+  - N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
+- **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not
+  implemented.
+- **Header checks:** on 12 CVE datasets libhdf5 rejects a corrupt header and
+  we read data anyway. We need stricter header checks.
+- **Writer:**
+  - Nested groups beyond one level: path-like names are now refused, not
+    created.
+  - Dense attribute storage for attributes over 64 KiB.
+  - Output that HDF5 1.8 can read.
+  - A B-tree v2 chunk index larger than one leaf, so datasets with several
+    unlimited dimensions are limited to 65 535 chunks.
+
+---
+
 ## Compound datatype message version 5 is not parsed (HDF5 2.0)
 
 **Status:** fixed on `main` in `a13ff51` (2026-06-03); **not in the v2.1.0
@@ -226,7 +319,10 @@ block-offset field in the super block, and a page-init bitmap read from the
 wrong structure. All four are fixed and covered by interop tests against
 HDF5 2.0 at sizes that cross each boundary, including paged data blocks.
 
-Files written by this crate are unaffected — this was purely a read-path bug.
+Files written by this crate were not affected by *this* read bug, but the
+writer had its own: it indexed only the first 244 chunks, so later chunks
+read back as 0 in libhdf5 and in clawhdf5. See "Silent wrong data found by
+the 2026-09-25 HDF5 audit" below.
 
 ## Every `f32` dataset we wrote was unreadable by h5py / libhdf5