Files

T

ClawHDF5 Coding Agent 3a30327f35 security(agent): chain WAL entry CRCs and restrict the legacy no-CRC reader

Two related gaps in the WAL format, both closed:

1. Each entry's CRC32 covered only its own bytes, with no sequence number
   or chaining — entries could be reordered, duplicated, or spliced (e.g.
   a Tombstone moved before/after its target Save) while every individual
   entry still passed its own CRC check, silently changing replayed cache
   state. Bump to WAL_VERSION 3: each entry's CRC32 trailer is now computed
   over its own bytes chained with the previous entry's stored CRC
   (crc32(entry_bytes ++ prev_crc)), seeded at 0 after a truncation. Moving,
   duplicating, or reordering an entry breaks the chain at that point, and
   replay stops there — same handling as a bit-flip or truncation. The
   previous per-entry-CRC-only format becomes WAL_VERSION_CRC_UNCHAINED (2)
   and remains fully readable (not restricted, since it still verifies each
   entry); WalFile::open migrates it to v3 by recreating the file fresh,
   same as the existing v1 migration.

   WalFile::open() on an existing v3 file scans it once to resume the CRC
   chain correctly for further appends — required because a process
   restart without an intervening flush reopens the same (non-truncated)
   WAL and keeps appending to it, so new entries must chain against the
   real last entry already on disk, not restart from 0.

2. WAL_VERSION_LEGACY_NO_CRC (v1, no integrity verification at all) was
   reachable through the public WalFile::read_entries — a version byte
   flipped from 2/3 down to 1 silently downgraded every entry to the
   fully-unverified pre-hardening parser for any caller, not just the
   one-time migration path. Split into WalFile::read_entries (rejects v1
   with a typed error; still reads v2/v3) and the pub(crate)
   read_entries_for_migration (accepts v1 too), used exclusively by
   HDF5Memory::open's migration flow.

INT-09

2026-08-17 01:01:29 +00:00

4.1 KiB

Raw Permalink Blame History

clawhdf5

Purpose

Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. Used by ZeroClaw as its persistent memory and knowledge graph backend.

Architecture

Cargo workspace with 16 crates under crates/ (plus libaec-sys, an internal FFI bindings crate for the optional szip feature):

Crate	Role
`clawhdf5-format`	HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants
`clawhdf5-io`	Read/write implementation
`clawhdf5-filters`	Compression filters (gzip, LZ4, Zstd, Blosc)
`clawhdf5-derive`	Proc-macro derive for HDF5-serializable structs
`clawhdf5`	Main facade crate
`clawhdf5-netcdf4`	NetCDF-4 compatibility layer
`clawhdf5-ann`	HNSW approximate nearest-neighbor vector index
`clawhdf5-agent`	Agent memory, session history, knowledge graph storage
`clawhdf5-gpu`	GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders)
`clawhdf5-accel`	CPU SIMD acceleration path
`clawhdf5-migrate`	Schema migration engine
`clawhdf5-android`	Android JNI bindings
`clawhdf5-cli`	Command-line interface
`clawhdf5-napi`	Node.js native addon bindings
`clawhdf5-py`	PyO3 Python bindings
`clawhdf5-bench`	Benchmark suite

Key Features

Zero-dependency HDF5 read/write (no libhdf5 C library required)
HNSW vector index for semantic similarity search over agent memories — the clawhdf5-agent hnsw feature is on by default, so hybrid_search uses the approximate clawhdf5-ann index for the vector stage (the index mirrors the cache and self-heals on drift). Build the agent with --no-default-features --features float16 to force the exact linear cosine scan.
WAL (write-ahead log) for crash-safe persistence, with a chained CRC32 trailer per entry (each entry's CRC folds in the previous entry's CRC) so a corrupted, reordered, duplicated, or spliced entry stops replay cleanly instead of loading bad or tampered data. The pre-chaining per-entry-CRC format (v2) is still fully readable; the oldest no-CRC format (v1) is only reachable through the one-time migration path in HDF5Memory::open, not through the public WalFile::read_entries.
Dataset::verify_provenance() (clawhdf5 facade, provenance feature, on by default) recomputes a dataset's SHA-256 and compares it against the _provenance_sha256 attribute written automatically on save when DatasetBuilder::with_provenance is used. It's opt-in per call, not run automatically on open — it decodes and hashes the whole dataset. The hash is unkeyed (tamper-evident, not tamper-proof): it detects accidental corruption, not a deliberate actor able to modify both the data and the stored hash.
clawhdf5-agent's HDF5Memory::save/save_batch/save_or_update run every write through an in-memory (session-scoped, not persisted to disk) provenance ledger and write-anomaly detector: a content hash per record (provenance.rs) for detecting accidental mid-session corruption, plus rate-limit/injection-pattern/source-distribution checks (anomaly.rs). Alerts never block a save — drain them with HDF5Memory::take_anomaly_alerts. MemorySource for this bookkeeping is inferred from the caller-supplied source_channel string (a heuristic, not an authenticated trust boundary).
GPU-accelerated batch I/O for large dataset processing
Python and Node.js bindings for cross-language use
NetCDF-4 compatibility for scientific data interop

Workflows

Build

cargo build --release

Test

cargo test --workspace

CLI

cargo run -p clawhdf5-cli -- --help
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands

Python bindings

cd crates/clawhdf5-py
maturin develop
python -c "import clawhdf5; print(clawhdf5.__version__)"

Integration

ZeroClaw imports this as a Cargo feature (clawhdf5 feature flag) to persist agent memory with HNSW vector search for context retrieval.

4.1 KiB Raw Permalink Blame History