2026-09-26 02:42:54 +00:00
4 changed files with 206 additions and 5 deletions
@@ -3,6 +3,38 @@
 ## Unreleased
 ### Upgrade Notes
 - **HDF5 correctness audit (2026-09-25).** A sweep of 686 public files (the
  libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c,
  netcdf4-python, h5wasm, h5py and xarray corpora), a 567-case read matrix and
  a 96-case write matrix against HDF5 1.10–2.0 found bugs that returned wrong
  values with no error, and files we wrote that libhdf5 rejects. The fixes are
  listed under Correctness and Interop. What changes for callers:
  - **Chunked datasets whose max shape is larger than their current shape**,
    or whose unlimited dimension is not the first, were indexed by the current
    shape instead of the max shape, both when read and when written. Files from
    libhdf5 now read correctly. Files clawhdf5 wrote with such a max shape were
    laid out wrongly and now read the way libhdf5 always read them — rewrite
    them. Agent stores and ClawBrainHub files have no max shape and are
    unaffected.
  - Integer reads (`read_i32`/`read_i64`/`read_u64`/...) of float data now
    convert (truncate toward zero, saturate at the type's range, NaN reads as
    0) instead of returning the IEEE bit pattern, and out-of-range integers
    saturate instead of keeping the low bits.
  - `FileWriter::finish()` now returns an error instead of writing a corrupt
    file for: a header message over 64 KiB (e.g. an attribute larger than
    ~64 KiB), a group/dataset/link name that is empty, `.` or contains `/`
    (nested paths were written as one literal link), a max shape smaller than
    the shape, a page size outside 512 B–1 GiB, and more than 65 535 chunks in
    a dataset with several unlimited dimensions.
  - **Breaking (format crate):** `ObjectHeaderWriter::serialize`,
    `BatchObjectHeaderWriter::compute_sizes`/`serialize_all` and
    `build_chunked_data_from_precompressed` return `Result`;
    `read_fixed_array_chunks`/`read_extensible_array_chunks` take `max_dims`;
    `build_fixed_array_at`/`ea_writer::build_extensible_array_at` take one
    `Option<WrittenChunk>` per index slot; `fill_value::dataset_fill_value`
    returns `UnresolvedSharedMessage` for a shared message it cannot resolve
    instead of `None`. `FillTime::default()` is `IfSet` (libhdf5's default;
    default files are byte-identical).
 - **ZeroClaw does not use clawhdf5.** The project described itself as
  ZeroClaw's memory backend ("imported as a `clawhdf5` Cargo feature"). Checked
  against ZeroClaw v0.8.5 (the latest release), the `osobh/zeroclaw` fork and
@@ -241,6 +273,53 @@
 - CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
 ### Correctness
 - `clawhdf5-format` reader — **values returned wrong with no error:**
  - Fixed Array and Extensible Array chunk indexes were laid out by the
    dataset's current shape instead of its max shape (23 libhdf5 test files,
    and any h5py file with e.g. `maxshape=(10, None)` or `(20, 10)` under
    `libver='latest'`).
  - Files with 4-byte offsets: unfiltered chunked datasets read as zeros.
    Chunk B-tree keys store offsets in 8 bytes whatever the file's offset
    size.
  - A chunk's filter mask skipped the whole pipeline when any bit was set;
    only the flagged filters are skipped now.
  - Float data read as an integer returned the bit pattern; narrowing integer
    reads kept the low bits; bfloat16 was decoded as IEEE half. Floats are now
    decoded from their datatype fields (bf16, FP8 E4M3/E5M2, IEEE half, single
    and double).
  - `vl_data::read_vl_bytes` truncated sequences of non-byte base types.
  - A shared fill-value message read as zero fill; it is resolved now,
    including from the file's shared-message (SOHM) table, which could never
    resolve because its index version byte was skipped.
  - Two threads reading two chunked datasets through one `File` could get each
    other's chunks (the shared chunk cache was switched between datasets
    across separate lock acquisitions). The cache is now keyed by dataset.
 - `clawhdf5-format` reader — errors on valid files: enum and bool datasets
  through the numeric readers; the "don't filter partial edge chunks" layout
  flag; Fletcher32 ahead of deflate (NetCDF-4's order). Unknown-message flags
  follow libhdf5 (`tbogus.h5`): "fail if unknown" is refused, "fail if unknown
  and writing" is ignored by a reader.
 - `clawhdf5-format` writer — **files libhdf5 rejects or reads wrong:**
  - Extensible Array (one unlimited dimension): chunks from index 244 on were
    written but never indexed and read as 0, by libhdf5 and by us.
  - Fixed Array: more than 1 024 chunks gave checksum errors (data blocks
    were never paged).
  - A finite max shape larger than the shape gave libhdf5 "addr overflow"; an
    unlimited dimension that is not the first scrambled the data; several
    unlimited dimensions (`(None, None)`) broke the whole file. These now
    write the index libhdf5 writes (swizzled Extensible Array, or a B-tree v2
    index for several unlimited dimensions).
  - Header messages over 64 KiB (the size field is 16 bits) and compact
    datasets at 65 534–65 535 bytes produced corrupt files.
  - Reference, Opaque, BitField and Time datatypes were written as empty
    messages; they now encode as HDF5 2.0 does.
  - `with_page_size` wrote a nonexistent superblock version 4; it now writes
    the v3 superblock and File Space Info message libhdf5 writes.
  - `FillTime` values were rotated on disk (NEVER was written as ALLOC, and so
    on). New `DatasetBuilder::with_fill_value`.
  - An empty-string attribute got a zero-size datatype, which made every
    attribute on the object unreadable in libhdf5.
  - `maxshape` equal to the shape no longer forces chunked layout.
 - `clawhdf5-format`: **a truncated deflate chunk read back short, with no
  error.** The deflate filter used flate2's streaming reader, which returns the
  bytes it has when the input runs out before the end-of-stream marker. It now
@@ -1,7 +1,7 @@
 # clawhdf5
 ## Purpose
-Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
+Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
 ## Architecture
@@ -11,13 +11,13 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
 |-------|------|
 | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
 | `clawhdf5-io` | Read/write implementation |
-| `clawhdf5-filters` | Compression filters (gzip, LZ4, Zstd, Blosc) |
+| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec) live in `clawhdf5-format`. No Blosc. |
 | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
 | `clawhdf5` | Main facade crate |
 | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
 | `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
 | `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
-| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
+| `clawhdf5-gpu` | GPU vector distance computation via wgpu (hand-written WGSL compute shaders) — not dataset I/O |
 | `clawhdf5-accel` | CPU SIMD acceleration path |
 | `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
 | `clawhdf5-android` | Android JNI bindings |
@@ -148,7 +148,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
  Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
  `MemorySource` for this bookkeeping is inferred from the caller-supplied
  `source_channel` string (a heuristic, not an authenticated trust boundary).
- GPU-accelerated batch I/O for large dataset processing
+- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
 - Python and Node.js bindings for cross-language use
 - NetCDF-4 compatibility for scientific data interop
@@ -372,3 +372,29 @@ with h5py.File("{path}", "r") as f:
        }
    }
 }
 /// libhdf5's N-Bit float test data is stored as a 20-bit custom float
 /// (`le_data.h5` from the HDF5 test suite). The N-Bit filter restores the
 /// file type's bytes; the typed reader must then decode that layout the way
 /// libhdf5 converts it (h5py reads 0.3333435, 0.666687, 1, ...).
 #[test]
 fn nbit_custom_float_decodes_like_libhdf5() {
    let path = concat!(
        env!("CARGO_MANIFEST_DIR"),
        "/../clawhdf5-format/tests/fixtures/filters/le_data.h5"
    );
    let f = File::open(path).unwrap();
    // Exactly representable in the 20-bit type, so exact in f32 and f64.
    let expected = [
        0.333343505859375,
        0.66668701171875,
        1.0,
        1.3333740234375,
        1.6666259765625,
        2.0,
    ];
    for name in ["Nbit_float_data_le", "Nbit_float_data_be"] {
        let got = f.dataset(name).unwrap().read_f64().unwrap();
        assert_eq!(&got[..6], &expected, "{name}");
    }
 }
@@ -7,6 +7,99 @@ deleting it.
 ---
 ## Silent wrong data found by the 2026-09-25 HDF5 audit
 **Status:** fixed after v2.7.0 (2026-09-25). **Every release up
 to and including v2.7.0 is affected.**
 An audit on tank checked clawhdf5 against libhdf5 in three ways:
 - a sweep of 686 public files: the libhdf5 test files, the HDF Group's
  `cve_hdf5` reproducers, and the pyfive, netcdf-c, netcdf4-python, h5wasm,
  h5py and xarray corpora;
 - 567 read cases generated with h5py 3.16 / HDF5 2.0;
 - 96 write cases checked with h5py builds linking HDF5 1.10, 1.12, 1.14 and
  2.0, plus h5dump 1.14.6.
 It found these cases where a value came back wrong **without an error**:
 | Area | What happened | Who is affected |
 |---|---|---|
 | Chunk index (read) | Fixed/Extensible Array indexes laid out by the current shape, not the max shape: chunks returned from the wrong place | any file with a max shape larger than its shape and `libver='latest'` (h5py `maxshape=(10, None)`, `(20, 10)`) |
 | Chunk index (write) | Extensible Array chunks from index 244 on never indexed (read as 0); unlimited dimension not first: data scrambled | files we wrote with one unlimited dimension and > 244 chunks, or e.g. `maxshape=(20, None)` |
 | 4-byte offsets | unfiltered chunked datasets read as zeros | files created with `sizeof_addr = 4` |
 | Filter mask | any skipped filter skipped the whole pipeline | files with partially filtered chunks (optional filters, direct chunk writes) |
 | Numeric reads | float read as integer returned the bit pattern; narrowing integer reads kept the low bits; bfloat16 decoded as IEEE half | `read_i32`/`read_i64`/`read_u64` callers on float or wider data; HDF5 2.0 bf16 data |
 | SZIP | garbage or zeros | every libhdf5-written SZIP dataset |
 | Scale-offset | float values 1 ULP off | libhdf5 D-scale float data |
 | Shared fill value | read as zero fill | fill values stored as shared messages |
 | VL sequences | `read_vl_bytes` truncated non-byte base types | VL int/float sequences |
 | Chunk cache | two threads reading two chunked datasets through one `File` could get each other's chunks | multi-threaded readers, including Python with the GIL released |
 The audit also found files we wrote that libhdf5 **refuses**, now fixed:
 - Fixed Array datasets with more than 1 024 chunks.
 - Header messages over 64 KiB (large attributes).
 - Reference, Opaque, BitField and Time datatypes.
 - Files written with `with_page_size`.
 - Several unlimited dimensions.
 - A finite max shape larger than the shape.
 - An empty-string attribute, which broke every attribute on its object.
 - `FillTime` codes, which were rotated.
 Our LZ4 and Zstd output could not be read by libhdf5's registered plugins, and
 our pcodec filter used Granular BitRound's ID. The details are in
 `CHANGELOG.md` under Correctness and Interop.
 Before the fix, 419 of the 686 files read correctly and 43 differed from h5py.
 After it, 448 read correctly and 23 differ. Of those 23:
 - 17 are N-Bit float files. The probe compares raw file-type bytes; the typed
  reader returns libhdf5's values (`nbit_custom_float_decodes_like_libhdf5`).
 - 2 are an h5py bug: VL data with a big-endian base type comes back
  byte-swapped in h5py, and h5dump agrees with us.
 - The rest are object or attribute listing differences.
 There were no panics, hangs or crashes before or after, including on all 147
 CVE and fuzzer files. On some of those files, h5dump 1.14.6 and h5py/HDF5 2.0
 segfault or abort.
 ## Gaps found by the 2026-09-25 HDF5 audit (open)
 **Status:** open. These fail with an error; none returns wrong data, except
 the VDS item, which is marked.
 - **Layout message versions 1 and 2** (HDF5 1.6-era files): 84 of the 686
  sweep files, `InvalidLayoutVersion`. This is the largest single gap.
 - **Virtual datasets:**
  - **Wrong data:** unmapped regions read as 0 instead of the fill value.
  - `%b` printf-style source names are not expanded.
  - Hyperslab selection versions 1 and 2 are refused.
 - **Files with a user block:** the base address is not applied.
 - **Old-style shared messages (version 1)** read the wrong address.
 - **Groups and links:**
  - Groups with a user-defined link type (e.g. 187) cannot be listed.
  - Dense groups with more than about 22 000 links cannot be listed.
  - Soft links are left out of `datasets()`.
 - **Dense attributes:** a large attribute stored as a fractal-heap "huge"
  object makes every attribute on the object fail. This affects real NetCDF
  files (`issue671.nc`).
 - **Other readers:**
  - VL-string datasets are not readable through `File`.
  - Metadata cache images are not supported.
  - x87 long double and binary128 are refused.
  - N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
 - **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not
  implemented.
 - **Header checks:** on 12 CVE datasets libhdf5 rejects a corrupt header and
  we read data anyway. We need stricter header checks.
 - **Writer:**
  - Nested groups beyond one level: path-like names are now refused, not
    created.
  - Dense attribute storage for attributes over 64 KiB.
  - Output that HDF5 1.8 can read.
  - A B-tree v2 chunk index larger than one leaf, so datasets with several
    unlimited dimensions are limited to 65 535 chunks.
 ---
 ## Compound datatype message version 5 is not parsed (HDF5 2.0)
 **Status:** fixed on `main` in `a13ff51` (2026-06-03); **not in the v2.1.0
@@ -226,7 +319,10 @@ block-offset field in the super block, and a page-init bitmap read from the
 wrong structure. All four are fixed and covered by interop tests against
 HDF5 2.0 at sizes that cross each boundary, including paged data blocks.
-Files written by this crate are unaffected — this was purely a read-path bug.
+Files written by this crate were not affected by *this* read bug, but the
 writer had its own: it indexed only the first 244 chunks, so later chunks
 read back as 0 in libhdf5 and in clawhdf5. See "Silent wrong data found by
 the 2026-09-25 HDF5 audit" below.
 ## Every `f32` dataset we wrote was unreadable by h5py / libhdf5