diff --git a/CHANGELOG.md b/CHANGELOG.md index cbe22bb..8b87415 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,38 @@ ## Unreleased ### Upgrade Notes +- **HDF5 correctness audit (2026-09-25).** A sweep of 686 public files (the + libhdf5 test files, the HDF Group's CVE reproducers, pyfive, netcdf-c, + netcdf4-python, h5wasm, h5py and xarray corpora), a 567-case read matrix and + a 96-case write matrix against HDF5 1.10–2.0 found bugs that returned wrong + values with no error, and files we wrote that libhdf5 rejects. The fixes are + listed under Correctness and Interop. What changes for callers: + - **Chunked datasets whose max shape is larger than their current shape**, + or whose unlimited dimension is not the first, were indexed by the current + shape instead of the max shape, both when read and when written. Files from + libhdf5 now read correctly. Files clawhdf5 wrote with such a max shape were + laid out wrongly and now read the way libhdf5 always read them — rewrite + them. Agent stores and ClawBrainHub files have no max shape and are + unaffected. + - Integer reads (`read_i32`/`read_i64`/`read_u64`/...) of float data now + convert (truncate toward zero, saturate at the type's range, NaN reads as + 0) instead of returning the IEEE bit pattern, and out-of-range integers + saturate instead of keeping the low bits. + - `FileWriter::finish()` now returns an error instead of writing a corrupt + file for: a header message over 64 KiB (e.g. an attribute larger than + ~64 KiB), a group/dataset/link name that is empty, `.` or contains `/` + (nested paths were written as one literal link), a max shape smaller than + the shape, a page size outside 512 B–1 GiB, and more than 65 535 chunks in + a dataset with several unlimited dimensions. + - **Breaking (format crate):** `ObjectHeaderWriter::serialize`, + `BatchObjectHeaderWriter::compute_sizes`/`serialize_all` and + `build_chunked_data_from_precompressed` return `Result`; + `read_fixed_array_chunks`/`read_extensible_array_chunks` take `max_dims`; + `build_fixed_array_at`/`ea_writer::build_extensible_array_at` take one + `Option` per index slot; `fill_value::dataset_fill_value` + returns `UnresolvedSharedMessage` for a shared message it cannot resolve + instead of `None`. `FillTime::default()` is `IfSet` (libhdf5's default; + default files are byte-identical). - **ZeroClaw does not use clawhdf5.** The project described itself as ZeroClaw's memory backend ("imported as a `clawhdf5` Cargo feature"). Checked against ZeroClaw v0.8.5 (the latest release), the `osobh/zeroclaw` fork and @@ -241,6 +273,53 @@ - CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake. ### Correctness +- `clawhdf5-format` reader — **values returned wrong with no error:** + - Fixed Array and Extensible Array chunk indexes were laid out by the + dataset's current shape instead of its max shape (23 libhdf5 test files, + and any h5py file with e.g. `maxshape=(10, None)` or `(20, 10)` under + `libver='latest'`). + - Files with 4-byte offsets: unfiltered chunked datasets read as zeros. + Chunk B-tree keys store offsets in 8 bytes whatever the file's offset + size. + - A chunk's filter mask skipped the whole pipeline when any bit was set; + only the flagged filters are skipped now. + - Float data read as an integer returned the bit pattern; narrowing integer + reads kept the low bits; bfloat16 was decoded as IEEE half. Floats are now + decoded from their datatype fields (bf16, FP8 E4M3/E5M2, IEEE half, single + and double). + - `vl_data::read_vl_bytes` truncated sequences of non-byte base types. + - A shared fill-value message read as zero fill; it is resolved now, + including from the file's shared-message (SOHM) table, which could never + resolve because its index version byte was skipped. + - Two threads reading two chunked datasets through one `File` could get each + other's chunks (the shared chunk cache was switched between datasets + across separate lock acquisitions). The cache is now keyed by dataset. +- `clawhdf5-format` reader — errors on valid files: enum and bool datasets + through the numeric readers; the "don't filter partial edge chunks" layout + flag; Fletcher32 ahead of deflate (NetCDF-4's order). Unknown-message flags + follow libhdf5 (`tbogus.h5`): "fail if unknown" is refused, "fail if unknown + and writing" is ignored by a reader. +- `clawhdf5-format` writer — **files libhdf5 rejects or reads wrong:** + - Extensible Array (one unlimited dimension): chunks from index 244 on were + written but never indexed and read as 0, by libhdf5 and by us. + - Fixed Array: more than 1 024 chunks gave checksum errors (data blocks + were never paged). + - A finite max shape larger than the shape gave libhdf5 "addr overflow"; an + unlimited dimension that is not the first scrambled the data; several + unlimited dimensions (`(None, None)`) broke the whole file. These now + write the index libhdf5 writes (swizzled Extensible Array, or a B-tree v2 + index for several unlimited dimensions). + - Header messages over 64 KiB (the size field is 16 bits) and compact + datasets at 65 534–65 535 bytes produced corrupt files. + - Reference, Opaque, BitField and Time datatypes were written as empty + messages; they now encode as HDF5 2.0 does. + - `with_page_size` wrote a nonexistent superblock version 4; it now writes + the v3 superblock and File Space Info message libhdf5 writes. + - `FillTime` values were rotated on disk (NEVER was written as ALLOC, and so + on). New `DatasetBuilder::with_fill_value`. + - An empty-string attribute got a zero-size datatype, which made every + attribute on the object unreadable in libhdf5. + - `maxshape` equal to the shape no longer forces chunked layout. - `clawhdf5-format`: **a truncated deflate chunk read back short, with no error.** The deflate filter used flate2's streaming reader, which returns the bytes it has when the input runs out before the end-of-stream marker. It now diff --git a/CLAUDE.md b/CLAUDE.md index 1fa4581..3f8d9c1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,7 +1,7 @@ # clawhdf5 ## Purpose -Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true). +Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true). ## Architecture @@ -11,13 +11,13 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F |-------|------| | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants | | `clawhdf5-io` | Read/write implementation | -| `clawhdf5-filters` | Compression filters (gzip, LZ4, Zstd, Blosc) | +| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec) live in `clawhdf5-format`. No Blosc. | | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs | | `clawhdf5` | Main facade crate | | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer | | `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index | | `clawhdf5-agent` | Agent memory, session history, knowledge graph storage | -| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) | +| `clawhdf5-gpu` | GPU vector distance computation via wgpu (hand-written WGSL compute shaders) — not dataset I/O | | `clawhdf5-accel` | CPU SIMD acceleration path | | `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration | | `clawhdf5-android` | Android JNI bindings | @@ -148,7 +148,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`. `MemorySource` for this bookkeeping is inferred from the caller-supplied `source_channel` string (a heuristic, not an authenticated trust boundary). -- GPU-accelerated batch I/O for large dataset processing +- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only - Python and Node.js bindings for cross-language use - NetCDF-4 compatibility for scientific data interop diff --git a/crates/clawhdf5/tests/numeric_conversion_interop.rs b/crates/clawhdf5/tests/numeric_conversion_interop.rs index 357a37d..22310ed 100644 --- a/crates/clawhdf5/tests/numeric_conversion_interop.rs +++ b/crates/clawhdf5/tests/numeric_conversion_interop.rs @@ -372,3 +372,29 @@ with h5py.File("{path}", "r") as f: } } } + +/// libhdf5's N-Bit float test data is stored as a 20-bit custom float +/// (`le_data.h5` from the HDF5 test suite). The N-Bit filter restores the +/// file type's bytes; the typed reader must then decode that layout the way +/// libhdf5 converts it (h5py reads 0.3333435, 0.666687, 1, ...). +#[test] +fn nbit_custom_float_decodes_like_libhdf5() { + let path = concat!( + env!("CARGO_MANIFEST_DIR"), + "/../clawhdf5-format/tests/fixtures/filters/le_data.h5" + ); + let f = File::open(path).unwrap(); + // Exactly representable in the 20-bit type, so exact in f32 and f64. + let expected = [ + 0.333343505859375, + 0.66668701171875, + 1.0, + 1.3333740234375, + 1.6666259765625, + 2.0, + ]; + for name in ["Nbit_float_data_le", "Nbit_float_data_be"] { + let got = f.dataset(name).unwrap().read_f64().unwrap(); + assert_eq!(&got[..6], &expected, "{name}"); + } +} diff --git a/docs/known-issues.md b/docs/known-issues.md index f99dbe6..6dfa369 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -7,6 +7,99 @@ deleting it. --- +## Silent wrong data found by the 2026-09-25 HDF5 audit + +**Status:** fixed after v2.7.0 (2026-09-25). **Every release up +to and including v2.7.0 is affected.** + +An audit on tank checked clawhdf5 against libhdf5 in three ways: +- a sweep of 686 public files: the libhdf5 test files, the HDF Group's + `cve_hdf5` reproducers, and the pyfive, netcdf-c, netcdf4-python, h5wasm, + h5py and xarray corpora; +- 567 read cases generated with h5py 3.16 / HDF5 2.0; +- 96 write cases checked with h5py builds linking HDF5 1.10, 1.12, 1.14 and + 2.0, plus h5dump 1.14.6. + +It found these cases where a value came back wrong **without an error**: + +| Area | What happened | Who is affected | +|---|---|---| +| Chunk index (read) | Fixed/Extensible Array indexes laid out by the current shape, not the max shape: chunks returned from the wrong place | any file with a max shape larger than its shape and `libver='latest'` (h5py `maxshape=(10, None)`, `(20, 10)`) | +| Chunk index (write) | Extensible Array chunks from index 244 on never indexed (read as 0); unlimited dimension not first: data scrambled | files we wrote with one unlimited dimension and > 244 chunks, or e.g. `maxshape=(20, None)` | +| 4-byte offsets | unfiltered chunked datasets read as zeros | files created with `sizeof_addr = 4` | +| Filter mask | any skipped filter skipped the whole pipeline | files with partially filtered chunks (optional filters, direct chunk writes) | +| Numeric reads | float read as integer returned the bit pattern; narrowing integer reads kept the low bits; bfloat16 decoded as IEEE half | `read_i32`/`read_i64`/`read_u64` callers on float or wider data; HDF5 2.0 bf16 data | +| SZIP | garbage or zeros | every libhdf5-written SZIP dataset | +| Scale-offset | float values 1 ULP off | libhdf5 D-scale float data | +| Shared fill value | read as zero fill | fill values stored as shared messages | +| VL sequences | `read_vl_bytes` truncated non-byte base types | VL int/float sequences | +| Chunk cache | two threads reading two chunked datasets through one `File` could get each other's chunks | multi-threaded readers, including Python with the GIL released | + +The audit also found files we wrote that libhdf5 **refuses**, now fixed: +- Fixed Array datasets with more than 1 024 chunks. +- Header messages over 64 KiB (large attributes). +- Reference, Opaque, BitField and Time datatypes. +- Files written with `with_page_size`. +- Several unlimited dimensions. +- A finite max shape larger than the shape. +- An empty-string attribute, which broke every attribute on its object. +- `FillTime` codes, which were rotated. + +Our LZ4 and Zstd output could not be read by libhdf5's registered plugins, and +our pcodec filter used Granular BitRound's ID. The details are in +`CHANGELOG.md` under Correctness and Interop. + +Before the fix, 419 of the 686 files read correctly and 43 differed from h5py. +After it, 448 read correctly and 23 differ. Of those 23: +- 17 are N-Bit float files. The probe compares raw file-type bytes; the typed + reader returns libhdf5's values (`nbit_custom_float_decodes_like_libhdf5`). +- 2 are an h5py bug: VL data with a big-endian base type comes back + byte-swapped in h5py, and h5dump agrees with us. +- The rest are object or attribute listing differences. + +There were no panics, hangs or crashes before or after, including on all 147 +CVE and fuzzer files. On some of those files, h5dump 1.14.6 and h5py/HDF5 2.0 +segfault or abort. + +## Gaps found by the 2026-09-25 HDF5 audit (open) + +**Status:** open. These fail with an error; none returns wrong data, except +the VDS item, which is marked. + +- **Layout message versions 1 and 2** (HDF5 1.6-era files): 84 of the 686 + sweep files, `InvalidLayoutVersion`. This is the largest single gap. +- **Virtual datasets:** + - **Wrong data:** unmapped regions read as 0 instead of the fill value. + - `%b` printf-style source names are not expanded. + - Hyperslab selection versions 1 and 2 are refused. +- **Files with a user block:** the base address is not applied. +- **Old-style shared messages (version 1)** read the wrong address. +- **Groups and links:** + - Groups with a user-defined link type (e.g. 187) cannot be listed. + - Dense groups with more than about 22 000 links cannot be listed. + - Soft links are left out of `datasets()`. +- **Dense attributes:** a large attribute stored as a fractal-heap "huge" + object makes every attribute on the object fail. This affects real NetCDF + files (`issue671.nc`). +- **Other readers:** + - VL-string datasets are not readable through `File`. + - Metadata cache images are not supported. + - x87 long double and binary128 are refused. + - N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail. +- **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not + implemented. +- **Header checks:** on 12 CVE datasets libhdf5 rejects a corrupt header and + we read data anyway. We need stricter header checks. +- **Writer:** + - Nested groups beyond one level: path-like names are now refused, not + created. + - Dense attribute storage for attributes over 64 KiB. + - Output that HDF5 1.8 can read. + - A B-tree v2 chunk index larger than one leaf, so datasets with several + unlimited dimensions are limited to 65 535 chunks. + +--- + ## Compound datatype message version 5 is not parsed (HDF5 2.0) **Status:** fixed on `main` in `a13ff51` (2026-06-03); **not in the v2.1.0 @@ -226,7 +319,10 @@ block-offset field in the super block, and a page-init bitmap read from the wrong structure. All four are fixed and covered by interop tests against HDF5 2.0 at sizes that cross each boundary, including paged data blocks. -Files written by this crate are unaffected — this was purely a read-path bug. +Files written by this crate were not affected by *this* read bug, but the +writer had its own: it indexed only the first 244 chunks, so later chunks +read back as 0 in libhdf5 and in clawhdf5. See "Silent wrong data found by +the 2026-09-25 HDF5 audit" below. ## Every `f32` dataset we wrote was unreadable by h5py / libhdf5