ClawHDF5 Coding Agent
3a30327f35
Two related gaps in the WAL format, both closed: 1. Each entry's CRC32 covered only its own bytes, with no sequence number or chaining — entries could be reordered, duplicated, or spliced (e.g. a Tombstone moved before/after its target Save) while every individual entry still passed its own CRC check, silently changing replayed cache state. Bump to WAL_VERSION 3: each entry's CRC32 trailer is now computed over its own bytes chained with the previous entry's stored CRC (crc32(entry_bytes ++ prev_crc)), seeded at 0 after a truncation. Moving, duplicating, or reordering an entry breaks the chain at that point, and replay stops there — same handling as a bit-flip or truncation. The previous per-entry-CRC-only format becomes WAL_VERSION_CRC_UNCHAINED (2) and remains fully readable (not restricted, since it still verifies each entry); WalFile::open migrates it to v3 by recreating the file fresh, same as the existing v1 migration. WalFile::open() on an existing v3 file scans it once to resume the CRC chain correctly for further appends — required because a process restart without an intervening flush reopens the same (non-truncated) WAL and keeps appending to it, so new entries must chain against the real last entry already on disk, not restart from 0. 2. WAL_VERSION_LEGACY_NO_CRC (v1, no integrity verification at all) was reachable through the public WalFile::read_entries — a version byte flipped from 2/3 down to 1 silently downgraded every entry to the fully-unverified pre-hardening parser for any caller, not just the one-time migration path. Split into WalFile::read_entries (rejects v1 with a typed error; still reads v2/v3) and the pub(crate) read_entries_for_migration (accepts v1 too), used exclusively by HDF5Memory::open's migration flow. INT-09
90 lines
4.1 KiB
Markdown
90 lines
4.1 KiB
Markdown
# clawhdf5
|
|
|
|
## Purpose
|
|
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. Used by ZeroClaw as its persistent memory and knowledge graph backend.
|
|
|
|
## Architecture
|
|
|
|
Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
|
|
|
|
| Crate | Role |
|
|
|-------|------|
|
|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
|
| `clawhdf5-io` | Read/write implementation |
|
|
| `clawhdf5-filters` | Compression filters (gzip, LZ4, Zstd, Blosc) |
|
|
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
|
| `clawhdf5` | Main facade crate |
|
|
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
|
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
|
|
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
|
|
| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
|
|
| `clawhdf5-accel` | CPU SIMD acceleration path |
|
|
| `clawhdf5-migrate` | Schema migration engine |
|
|
| `clawhdf5-android` | Android JNI bindings |
|
|
| `clawhdf5-cli` | Command-line interface |
|
|
| `clawhdf5-napi` | Node.js native addon bindings |
|
|
| `clawhdf5-py` | PyO3 Python bindings |
|
|
| `clawhdf5-bench` | Benchmark suite |
|
|
|
|
## Key Features
|
|
- Zero-dependency HDF5 read/write (no libhdf5 C library required)
|
|
- HNSW vector index for semantic similarity search over agent memories — the
|
|
`clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses
|
|
the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
|
|
the cache and self-heals on drift). Build the agent with
|
|
`--no-default-features --features float16` to force the exact linear cosine scan.
|
|
- WAL (write-ahead log) for crash-safe persistence, with a chained CRC32
|
|
trailer per entry (each entry's CRC folds in the previous entry's CRC) so a
|
|
corrupted, reordered, duplicated, or spliced entry stops replay cleanly
|
|
instead of loading bad or tampered data. The pre-chaining per-entry-CRC
|
|
format (v2) is still fully readable; the oldest no-CRC format (v1) is only
|
|
reachable through the one-time migration path in `HDF5Memory::open`, not
|
|
through the public `WalFile::read_entries`.
|
|
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
|
|
default) recomputes a dataset's SHA-256 and compares it against the
|
|
`_provenance_sha256` attribute written automatically on save when
|
|
`DatasetBuilder::with_provenance` is used. It's opt-in per call, not run
|
|
automatically on open — it decodes and hashes the whole dataset. The hash
|
|
is unkeyed (tamper-*evident*, not tamper-*proof*): it detects accidental
|
|
corruption, not a deliberate actor able to modify both the data and the
|
|
stored hash.
|
|
- `clawhdf5-agent`'s `HDF5Memory::save`/`save_batch`/`save_or_update` run every
|
|
write through an in-memory (session-scoped, not persisted to disk)
|
|
provenance ledger and write-anomaly detector: a content hash per record
|
|
(`provenance.rs`) for detecting accidental mid-session corruption, plus
|
|
rate-limit/injection-pattern/source-distribution checks (`anomaly.rs`).
|
|
Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
|
|
`MemorySource` for this bookkeeping is inferred from the caller-supplied
|
|
`source_channel` string (a heuristic, not an authenticated trust boundary).
|
|
- GPU-accelerated batch I/O for large dataset processing
|
|
- Python and Node.js bindings for cross-language use
|
|
- NetCDF-4 compatibility for scientific data interop
|
|
|
|
## Workflows
|
|
|
|
### Build
|
|
```bash
|
|
cargo build --release
|
|
```
|
|
|
|
### Test
|
|
```bash
|
|
cargo test --workspace
|
|
```
|
|
|
|
### CLI
|
|
```bash
|
|
cargo run -p clawhdf5-cli -- --help
|
|
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands
|
|
```
|
|
|
|
### Python bindings
|
|
```bash
|
|
cd crates/clawhdf5-py
|
|
maturin develop
|
|
python -c "import clawhdf5; print(clawhdf5.__version__)"
|
|
```
|
|
|
|
## Integration
|
|
ZeroClaw imports this as a Cargo feature (`clawhdf5` feature flag) to persist agent memory with HNSW vector search for context retrieval.
|