clawhdf5/CHANGELOG.md

# Changelog

## Unreleased

### Bug Fixes
- `clawhdf5-format`: compound datatypes written with **default libver bounds**
  (datatype message version 1 — what plain `h5py.File(path, 'w')` produces)
  were mis-parsed. The v1 member layout carries 28 bytes of legacy array
  fields after the byte offset (the parser skipped 24), and v2 pads member
  names to 8 bytes and has no array fields at all (the parser did neither), so
  every member after the first byte offset was read from the wrong position —
  typically surfacing as `Overflow("compound member ...")` on read. Found by
  adding a default-libver axis to the h5py interop tests; byte-level regression
  tests for v1 and v2 added.
- `clawhdf5-gpu`: `gpu_tests` could hang forever under the default parallel
  test runner — every test created its own wgpu instance and device at once.
  Tests now serialise GPU access, and GPU→CPU readback waits are bounded
  (30 s) so a wedged driver returns `GpuError::BufferMap` instead of blocking.
- `clawhdf5-agent`: `benches/bench.rs` and `benches/memory_bench.rs` no longer
  compiled against the current `strategy`/`consolidation` APIs.

### Durability & Integrity
- `clawhdf5-agent`: a crash between writing a checkpoint and truncating the WAL
  no longer **duplicates every pending entry** on the next open. Each
  checkpoint records a `WalMark` (byte length + chained CRC of the WAL prefix it
  folded in) in `/meta`; `open()` skips exactly that prefix when it is still
  present. No WAL format change for this; older files behave as before.
- `clawhdf5-agent`: checkpoints and snapshots are durable as a unit — the temp
  file is synced before the rename and the directory after it. Individual WAL
  appends remain unsynced by design (documented in `CLAUDE.md`).
- `clawhdf5-agent`: `save_or_update` hits are logged as a new `Update` WAL
  record, so replay updates in place instead of appending a duplicate. WAL
  header version 3 → 4 (so older builds refuse the file rather than truncating
  a record they can't parse); v3 files are read and upgraded in place.
- `clawhdf5-agent`: loading validates every per-record dataset length (a
  truncated store is now `MemoryError::Schema`, not a later panic), fixes the
  `n.len() == n.len()` tautology that trusted a norms dataset of any length,
  and rejects `embedding_dim == 0` with records present.
- `clawhdf5-agent`: eight behavioural `MemoryConfig` fields are now persisted in
  `/meta`. Previously they reset to defaults on every open — a compressed store
  was rewritten uncompressed, `wal_enabled = false` flipped back to `true`.
- `clawhdf5-agent`: `compression = true` never worked in a default build (it
  requested Zstd without enabling the feature, so every checkpoint failed with
  `unsupported filter: 32015`). Default builds now use deflate; Zstd is the new
  opt-in `zstd` feature.
- `clawhdf5-agent`: **single-writer lock** (`<store>.h5.lock`,
  `MemoryError::Locked`) — two handles on one store used to silently destroy
  each other's data. New `HDF5Memory::open_read_only` gives a lock-free,
  never-writing view; the CLI's read-only subcommands use it.
- `clawhdf5-agent`: an unreadable WAL (torn header / bad magic) is quarantined
  (`HDF5Memory::quarantined_wal()`) instead of blocking `open()` of a healthy
  store. A WAL from an unknown newer version still fails and is left intact.
- `clawhdf5-agent`: provenance records are renumbered on compaction (they
  weren't, so every later `save_or_update` raised a false High integrity
  alert); pending anomaly alerts and tracked sessions are bounded;
  `snapshot()` includes entries still in the WAL.
- `clawhdf5-agent`: hybrid ranking is deterministic (index tie-breaks instead
  of `HashMap` order); a set of identical positive scores — including a single
  candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer
  reinforces zero-score filler results.
- `clawhdf5-format`: chunked/VDS/hyperslab reads size their buffers with
  overflow-checked arithmetic and fallible allocation, so crafted dimensions
  are `FormatError::Overflow` instead of a wrapped size or a process abort;
  `parallel_read` bounds checks use `checked_add`.
- `clawhdf5`: a malformed filter-pipeline message is an error instead of being
  treated as "no filters" (which returned compressed bytes as data);
  `FileBuilder::write` is atomic and synced instead of truncating the
  destination first.

### CI / Testing
- CI now lints every target (`cargo clippy --all-targets`) plus
  `clawhdf5-format`'s optional features, compiles all benches, and tests the
  format feature matrix. Previously test/bench code and feature-gated modules
  were never linted; the accumulated clippy backlog is fixed.
- CI installs python3 + h5py/numpy/netCDF4/xarray and sets
  `CLAWHDF5_REQUIRE_INTEROP=1`, which turns a missing interop dependency into a
  test **failure**. Until now every h5py/netCDF4 interop test silently skipped
  in CI, which is how the HDF5 2.0 compound bug fixed in v2.2.0 reached a user.
  The `#[ignore]`d `writer_h5py_tests` suite is run explicitly.
- h5py-generated-file tests now cover default libver bounds as well as
  `libver='latest'` (HDF5 2.0 raised the default low bound to 1.8).
- Optional fuzz smoke run (`CLAWHDF5_FUZZ_SECONDS=N scripts/ci-test.sh`); new
  datatype corpus seeds for v1 compound and native complex messages.

## v2.2.0 (2026-09-18)

### Security
- `clawhdf5-format`: bounded decompression output (`MAX_DECOMPRESS_SIZE`) for
  deflate/lz4/zstd/pcodec so a crafted compressed chunk can't drive an
  unbounded allocation (memory-exhaustion DoS).
- `clawhdf5-format`: `chunked_read.rs`/`data_read.rs`/`local_heap.rs` bounds
  audit — added `ensure_len` overflow guards at every plain-arithmetic
  offset+size check, a recursion-depth guard against a crafted
  self-referencing/cyclic B-tree chunk index, a fix for an unguarded
  compound-datatype `byte_offset` overrun in `read_compound_fields`, and an
  `ndims - 1` underflow guard for degenerate zero-dimension chunked layouts.
  Added a new `fuzz_dataset_read` cargo-fuzz target (walks every dataset in a
  parsed file and exercises the contiguous/chunked/compact raw-data read
  paths) which found and fixed 3 real crash bugs — an integer-multiply
  overflow in `copy_chunk_to_output`'s N-D assembly path, the `ndims - 1`
  underflow above, and an overflow in `local_heap.rs` — within the first few
  fuzzing runs.
- `clawhdf5-format`: `btree_v1.rs` overflow-safe bounds checks via a local
  `ensure_len` helper, closing a `usize`-overflow panic reachable from a
  crafted near-`usize::MAX` B-tree offset.
- `clawhdf5-agent`: WAL length-prefix caps (`MAX_WAL_FIELD_LEN`, 64 MiB) reject
  a corrupted/truncated length claim before allocating. Followed by a full
  per-entry CRC32 trailer (`WAL_VERSION` bumped to 2) — a bit-flip inside an
  entry now stops replay cleanly instead of silently accepting corrupted
  data. Old-format WAL files are still read correctly and migrated to the new
  format on next open.
- `clawhdf5-android`: validate `embedding_len`/`query_embedding_len` against
  the handle's configured `embedding_dim` (and reject null pointers) before
  constructing a slice from a raw pointer in `edgehdf5_save` /
  `edgehdf5_hybrid_search`.
- `clawhdf5-py`: bump pyo3/numpy `0.28` → `0.29`, clearing two RUSTSEC
  advisories (OOB read in `PyList`/`PyTuple` iterator; missing `Sync` bound on
  `PyCFunction::new_closure`).
- Clarified that the integrity hashes in `clawhdf5-agent::provenance`
  (FNV-1a) and `clawhdf5-format::provenance` (SHA-256) are unkeyed and detect
  only accidental corruption, not tampering — doc-only change, no behavior
  change.

### Performance
- `clawhdf5-format`: chunk cache lookup is now O(1) (`slot_index: HashMap`)
  instead of a linear scan, and cache hits return a shared `Arc` instead of
  cloning the decompressed buffer — the hottest path in chunked reads.
- `clawhdf5-ann`: optional `parallel` feature (rayon) parallelizes HNSW's
  `prune_connections` neighbor-distance computation. The outer build/insert
  loop is deliberately left sequential — it has genuine cross-iteration data
  dependencies and needs its own correctness-focused design pass.
- `clawhdf5-format/chunked_read.rs`: removed 12 unnecessary
  `chunk_dimensions[..rank].to_vec()` allocations where callees already
  accept `&[u32]`.

### Architecture
- Added `.gitea/workflows/ci.yml`, actually wiring the long-existing
  `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) into CI on every
  push/PR to `main`. Fixed stale package names in `ci-test.sh`/
  `check-nostd.sh` that had been silently no-op'ing the `clawhdf5-py`
  exclusion and the no_std check.
- Fixed a genuine no_std build break in `clawhdf5-format` (uncovered once the
  no_std CI check actually started running): `core::sync::atomic::AtomicU64`
  doesn't exist on `thumbv7em-none-eabihf` (switched to `portable-atomic`),
  missing `alloc` imports for `Box`/`Vec`/`format!` on a few no_std paths, and
  `f64::powi` (std/libm-only) replaced with a local exponentiation-by-squaring
  helper in the scale-offset filter.
- Added `[workspace.dependencies]` for `tempfile`/`criterion`/`half`/`serde`,
  fixing a real version skew on `half` (`2` vs `2.7` across crates).
- Fixed version skew: `clawhdf5-py` (`pyproject.toml`) and
  `packages/clawhdf5-node` (`package.json`) were both behind the actual crate
  version (2.1.0).
- Documented that the `mpi-io` feature's read/write paths are root-read
  +broadcast / gather-to-rank-0, not true collective I/O.

### Documentation
- BENCHMARKS.md: re-ran the previously-undated "LongMemEval Results", "SIMD &
  Parallelism", and "Vector Search Latency"/"Comparison to MemX" sections on
  a second machine (tank, Ryzen 7 7800X3D) with explicit dates and reproduce
  commands. Found and corrected a methodology issue in the SIMD/Parallelism
  benchmark selection (several originally-compared benchmarks didn't actually
  isolate the scalar/SIMD/parallel axis).
- README.md / ROADMAP.md / CLAUDE.md: corrected several stale facts —
  the `clawhdf5-types` crate (removed earlier) was still listed in the
  README crate map; the LongMemEval numbers in the README badge and table
  didn't match the actual (much better) benchmark results in BENCHMARKS.md;
  total line-of-code and test-count figures were stale; `clawhdf5-gpu`'s
  CubeCL→wgpu correction; documented the new `clawhdf5-ann` `parallel`
  feature flag, which had no entry in the Feature Flags table.

### New Features
- `clawhdf5-migrate`: substantial engine improvements:
  - **Real content validation** — the post-migration check now reads the written
    HDF5 back and compares actual content (chunk text, embeddings, and every
    session/entity/relation field) against the source, not just row counts. A
    representative sample of chunk rows is verified by default; `--validate-full`
    checks every row. A corrupt migration that preserves counts no longer passes.
  - **Configurable schema** — table names are no longer hardcoded; queries are
    built from a `SchemaConfig` (table + ordered column names, defaulting to the
    ZeroClaw layout) with `--chunks-table` / `--sessions-table` /
    `--entities-table` / `--relations-table` overrides.
  - **Streaming count pass** — `--dry-run` now does a `COUNT(*)`-only pass per
    table instead of loading every row into memory.
  - **Incremental migration** — `--incremental` reads the existing output, reads
    only source chunks newer than the last migrated id, and appends them
    (refreshing the metadata groups), instead of re-migrating everything.
- `clawhdf5-format`: read **IEEE-754 half-precision (f16)** floats. `read_as_f32`
  / `read_as_f64` previously only handled 4- and 8-byte floats; 2-byte floats
  (e.g. float16-stored embeddings) now decode via a no_std-safe bit conversion.
- `clawhdf5-format`: **write multi-block fractal heaps** (root indirect block).
  Dense attribute and dense link storage previously capped at a single direct
  block (~64 KiB of heap data — a few thousand attributes/links). When the
  objects exceed one direct block, the heap now lays out a root indirect block
  (FHIB) over multiple direct blocks sized by the doubling table, distributing
  objects across blocks with correct per-block heap offsets. Validated
  end-to-end: a 2,500-attribute object and a 2,500-link group round-trip
  through our reader and are read correctly by h5py. (Objects still may not
  span a block — no huge-object path.)
- `clawhdf5-format`: **write dense group link storage** (fractal heap + v2
  B-tree). A group with more than 8 links (libhdf5's compact `max_compact`
  default) is now written densely — its links live in a fractal heap indexed by
  a v2 B-tree of type 5 (link-name index) referenced from the group's LinkInfo
  message — instead of as inline Link messages. This matches libhdf5's
  compact→dense switchover and keeps large groups out of the object header.
  Reverse-engineered against libhdf5: link heaps use `heap_id_length` 7 /
  `max_heap_size` 32 (vs 8 / 40 for attributes). The shared single-direct-block
  fractal-heap builder is now parameterized and used by both dense attributes
  and dense links. Validated end-to-end: our reader round-trips, and h5py reads
  the dense groups we write. (Single direct block — up to ~a couple thousand
  links per group; beyond that needs indirect blocks, still unsupported.)

### Robustness
- `clawhdf5-format`: harden the readers added this cycle against malformed /
  hostile input — they parse untrusted bytes and must return errors, never
  panic, OOM, or recurse without bound. Fixed concrete vectors found by audit
  and locked in with adversarial tests:
  - **Paged Fixed Array**: `1 << max_nelmts_bits` shift overflow (a `u8` ≥ 64);
    element/page offset multiplications now checked; element count bounded by
    file size.
  - **H5S selection decoder**: `ALL`/`NONE` no longer claim 16 bytes they don't
    have; hyperslab `rank` capped at 32 (`H5S_MAX_RANK`) to stop a giant
    allocation; `iter_linear` coordinate/stride/product arithmetic is checked.
  - **VDS mapping parser**: no pre-allocation from the untrusted `nused`; all
    selection slicing is bounds-checked.
  - **scale-offset / N-Bit filters**: `1 << minbits` overflow at `minbits == 64`;
    N-Bit `bit_offset + precision` overflow; N-Bit type-tree recursion depth
    capped (no stack overflow from a crafted nested tree); element counts
    bounded by the chunk's expected decompressed size so a bogus count can't
    drive a huge allocation.
  - **Virtual Dataset assembly**: a virtual dataset whose source is itself
    virtual (a cycle) now errors instead of recursing into a stack overflow.

### New Features
- `clawhdf5-agent`: **compress fixed-length string datasets** (memory text
  chunks, session summaries, ids, tags, entity/relation names, …). These were
  always stored uncompressed with a "chunked compound not yet supported" note
  that was simply stale — chunked writes work for fixed-size string/compound
  datatypes like any other. `write_string_dataset` now chunks + deflates a
  string dataset once its payload reaches 4 KiB, so large, highly-redundant
  NullPad content shrinks substantially while tiny metadata stays contiguous
  (no chunk-overhead bloat).
- `clawhdf5-format`: decode the **scale-offset filter** (id 6) — both the
  integer variant (`H5Z_SO_INT`) and the floating-point **D-scale** variant
  (`H5Z_SO_FLOAT_DSCALE`). Handles signed/unsigned int sizes, f32/f64, negative
  minima, decimal scale factors and fill values; reverse-engineered against
  HDF5 2.0 and validated end-to-end. The float E-scale variant remains
  unsupported.
- `clawhdf5-format`: decode the **N-Bit filter** (id 5) — atomic, **compound**
  and **array** layouts (the full recursive type tree, nestable to any depth),
  previously unsupported. Signed and unsigned reduced-precision integers and
  float members all read end-to-end, validated against HDF5 2.0.

### New Features
- `clawhdf5` / `clawhdf5-format`: read **external-file Virtual Datasets (VDS)**.
  The format layer gains `read_raw_data_full_with_resolver` and a
  `VdsSourceResolver` callback (`Fn(&str) -> Option<Vec<u8>>`) that maps a
  stored source file name to its bytes, so the pure-byte reader can pull in
  external sources without a filesystem of its own. The `clawhdf5` `File` API
  wires a default resolver that reads sibling source files relative to the
  opened file's directory, so `File::open(...).dataset(...).read_*()` now
  transparently assembles cross-file VDS. A source file the resolver cannot
  supply leaves its region at the fill value (matching HDF5); an external
  source with no resolver at all is a clean error. In-memory files
  (`File::from_bytes`) have no directory, so only same-file VDS resolves there.
- `clawhdf5-format`: assemble **same-file Virtual Datasets (VDS)** of any rank.
  Previously a virtual layout returned `UnsupportedVersion`. The reader now
  decodes the global-heap mapping block (reverse-engineered against HDF5 2.0:
  `version · nused · [source-file · source-dataset · source-selection ·
  virtual-selection]* · checksum`, including the block-version-1 same-file
  marker), decodes the `H5S` source/virtual dataspace **selections** (ALL,
  NONE, and version-3 regular hyperslabs), reads each same-file source dataset,
  and scatters its selected elements into the virtual buffer in row-major order
  (so multi-dimensional block mappings land correctly); unmapped regions are
  left at the zero fill value. External-file sources return a clean unsupported
  error. The previous `parse_vds_mappings` used a guessed layout that did not
  match real files and is replaced.

### Tests
- `clawhdf5-format`: regression test for **scale-offset float E-scale**
  datasets. The HDF5 library does not implement E-scale encoding — when asked
  for it (`cd_values[0] = 1`) it stores the chunk raw and sets the chunk filter
  mask to skip the filter — so these files read back verbatim purely by
  honoring the per-chunk filter mask. The test locks in that behavior against a
  fixture produced via the HDF5 low-level API; no E-scale decoder is needed.

### Bug Fixes
- `clawhdf5-format`: **read multi-direct-block fractal heaps**. The reader split
  direct vs indirect block rows using the FRHP "Starting # of Rows in Root
  Indirect Block" field (a constant, typically 1), so any heap whose data spans
  more than one direct block — common in libhdf5 files with a large group or
  many dense attributes — was misread as having indirect blocks and failed with
  `InvalidFractalHeapSignature`. The split is now derived from the heap geometry
  (`max_direct_rows = log2(max_direct / start) + 2`). Validated against an
  h5py-written 400-dense-attribute group (root indirect block, 4 rows, 13 direct
  blocks).
- `clawhdf5-format`: scope the per-file **chunk cache by dataset**. The shared
  `ChunkCache` built its chunk index once and reused it for every chunked
  dataset in the file, keyed only by chunk coordinate with no dataset
  discrimination. With a single chunked dataset per file this was latent; once a
  file holds two chunked datasets of different rank (e.g. a 1-D compressed
  string array and the 2-D embeddings matrix), the first dataset's index was
  reused for the second, panicking with an out-of-bounds chunk coordinate. The
  cache now rebinds (dropping its index, chunk-index map, layout, and
  decompressed slots) whenever the dataset being read changes, while still
  caching repeated/sequential access to the same dataset.
- `clawhdf5-format`: read **paged Fixed Array** chunk indexes. A filtered,
  fixed-dimension dataset with more than one data-block page (>1024 chunks by
  default) previously failed with "paged Fixed Array data blocks not yet
  supported". The reader now walks the page-init bitmap (MSB-first), skips
  uninitialized pages, and resolves each page's fixed full-size slot (including
  the short final page). Reverse-engineered and validated end-to-end against an
  HDF5 2.0 file.
- `clawhdf5-format`: read **array-typed datatypes** (e.g. an array-typed
  compound member) via `read_as_i32/i64/u64/f32/f64` — previously a
  `TypeMismatch`. The array is read as a flat sequence of its base elements
  (recursing for nested arrays), applying base-type precision rules.
- `clawhdf5-format`: **sign-extend reduced-precision fixed-point integers** on
  read. A signed integer whose datatype precision is smaller than its storage
  size is stored zero-filled, so e.g. a 16-bit-precision `-1` previously read as
  `65535`. The integer read paths now extract the precision field and
  sign-extend (full-width types are unchanged). Completes signed N-Bit reads and
  also fixes un-filtered reduced-precision integer datasets.
- `clawhdf5-format`: read datasets written by modern HDF5 (1.14+/2.0, i.e.
  `libver=latest`). Compound (class 6) and array (class 10) datatype **version 5**
  messages and data layout **version 5** messages were rejected as invalid; they
  reuse the v3/v4 binary structure, so they are now accepted. This unblocks
  reading compound types and — critically — every chunked/compressed dataset
  written by HDF5 2.0. Found by running the h5py interop tests against
  h5py 3.16 / HDF5 2.0.
  Independently reported (with a patch) against the v2.1.0 tag by
  M. Scot Breitenfeld (The HDF Group) — v2.1.0 predates this fix.
- `clawhdf5-format`: parse HDF5 2.0 native complex datatypes (class 11,
  datatype version 5, e.g. `H5T_COMPLEX_IEEE_F64LE`). The properties are a
  single base floating-point datatype, not a compound-style member list; the
  old parser read the base type's bytes as member names, producing a garbage
  datatype, and failed with `UnexpectedEof` when a complex type was nested in
  a compound. It is now surfaced as the equivalent `{r, i}` compound (the
  shape h5py writes for numpy complex dtypes), with a size check against the
  base type. Validated end-to-end against an HDF5 2.0-written file.

### Performance
- `clawhdf5-format`: chunked writes now compress all chunks up front via
  `compress_all_chunks`, running across rayon threads under the `parallel`
  feature when there are more than 4 filtered chunks. On-disk layout is
  unchanged. Speeds up compressed embedding writes in `clawhdf5-agent` (which
  enables `parallel`).

### Documentation
- Fix stale package names across all 13 per-crate READMEs (`rustyhdf5-*` /
  `edgehdf5-*` → `clawhdf5-*`, usage versions → 2.1.0).
- Correct README workspace/test/crate stats and the CLAUDE.md CLI subcommand
  list; document the `hnsw` and format compression/checksum feature flags and
  the `entity_extract` / `async_memory` modules.

## v2.1.0 (2026-06-03)

### New Features
- `clawhdf5-agent`: HNSW now backs the vector stage of `hybrid_search`. The
  `hnsw` feature is **on by default**, so semantic search uses the approximate
  `clawhdf5-ann` index instead of a linear cosine scan. The index mirrors the
  memory cache (node id == cache index) and self-heals — it rebuilds whenever it
  drifts from the cache length, so no mutation path can desync it. Non-indexable
  stores (no/zero-dim/mixed embeddings) and dimension-mismatched queries fall
  back to the exact linear scan. Disable with
  `--no-default-features --features float16` for exact search.
- `clawhdf5-ann`: HNSW is now a live, mutable index — added `insert`,
  `mark_deleted` (soft-delete bitset; deleted nodes are traversed for
  connectivity but never returned), `compact` (drops deleted vectors and
  renumbers survivors), and `new` (empty index). Serialization gains a format
  version tag (`HNSW_FORMAT_VERSION` = 2) and persists the deleted bitset;
  pre-existing v1 files still load.
- `clawhdf5-agent`: `hybrid::merge_vector_keyword` exposes the shared
  normalize-and-fuse step used by both the linear and HNSW vector paths.
- Expose `max_dimensions()` API on Dataset, MmapDataset, and LazyDataset
- NetCDF-4 unlimited dimension detection now works correctly
- Python bindings (`clawhdf5-py`) build and link on macOS with system Python

### Bug Fixes
- `clawhdf5-py`: upgrade PyO3 and numpy `0.23` → `0.28` so the bindings build on
  Python 3.14 (PyO3 0.23 capped at 3.13 and hard-failed `cargo build
  --workspace`). Updated for the removed `PyObject` alias (`Py<PyAny>`) and the
  `Python::allow_threads` → `Python::detach` rename.
- Fix GPU L2 distance test (squared vs actual L2 mismatch in test helper)
- Mark Android JNI functions as `unsafe` for Rust 2024 edition compliance
- Add `# Safety` documentation to all public unsafe extern functions
- Fix all clippy warnings: needless_range_loop, manual_strip, ptr_arg, etc.
- Rename `RelationType::from_str` to `from_label` to avoid trait confusion
- Isolate h5py interop tests with `#[ignore]` when h5py unavailable

### Code Quality
- Full rustfmt pass across workspace (61 files)
- Refine inner unsafe blocks for Rust 2024 edition style
- Zero clippy warnings, zero clippy errors across entire workspace
- 1,546 tests passing, 0 failures

## v2.0.0 (2026-03-19)

- Unified rustyhdf5 (11 crates) and edgehdf5 (4 crates) into a single workspace
- All crates renamed to clawhdf5-* prefix
- Version bumped to 2.0.0 across all crates
- Git dependencies replaced with in-workspace path dependencies
- Added `agent` feature flag to clawhdf5-agent