osobhandClaude Opus 5.5
56abaec75e
Record the `blosc2` feature in the changelog, the README's feature table and the crate table, and mark the Blosc2 half of the known "Filters" issue fixed (dated, with the conformance run that shows h5ex_d_blosc2 reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features hdf5plugin never writes (dictionaries, lazy chunks, variable-length blocks, user-defined codecs and registered filters), which are errors. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
228 lines
14 KiB
Markdown
228 lines
14 KiB
Markdown
# clawhdf5
|
||
|
||
## Purpose
|
||
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
|
||
|
||
## Architecture
|
||
|
||
Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
|
||
|
||
| Crate | Role |
|
||
|-------|------|
|
||
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
||
| `clawhdf5-io` | Read/write implementation |
|
||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
|
||
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
||
| `clawhdf5` | Main facade crate |
|
||
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
||
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
|
||
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
|
||
| `clawhdf5-gpu` | GPU vector distance computation via wgpu (hand-written WGSL compute shaders) — not dataset I/O |
|
||
| `clawhdf5-accel` | CPU SIMD acceleration path |
|
||
| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
|
||
| `clawhdf5-android` | Android JNI bindings |
|
||
| `clawhdf5-cli` | Command-line interface (agent memory) |
|
||
| `clawhdf5-tools` | `h5rs`: pure-Rust HDF5 tools — `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` (structural + checksum validator) |
|
||
| `clawhdf5-napi` | Node.js native addon bindings |
|
||
| `clawhdf5-py` | PyO3 Python bindings |
|
||
| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
|
||
| `clawhdf5-bench` | Benchmark suite |
|
||
|
||
## Key Features
|
||
- Zero-C-dependency HDF5 read/write: no libhdf5, and deflate defaults to
|
||
pure-Rust zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake).
|
||
`ci-test.sh` fails if a C-building crate enters the core crates' default
|
||
tree. flate2 must keep `runtime_detection` with zlib-rs — without it zlib-rs
|
||
loses SIMD and inflates 3.5x slower. MSRV is 1.92 (`rust-version`, checked
|
||
in CI).
|
||
- HNSW vector index for semantic similarity search over agent memories — the
|
||
`clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses
|
||
the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
|
||
the cache and self-heals on drift). Build the agent with
|
||
`--no-default-features --features float16` to force the exact linear cosine scan.
|
||
The agent's `parallel` feature (also default) builds the index on a thread
|
||
pool; the graph is identical with or without it.
|
||
The index uses the HNSW paper's diversity heuristic for neighbour selection
|
||
(plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its
|
||
graph is saved to `<store>.h5.ann` at each checkpoint and reloaded by `open()`
|
||
(tied to the checkpoint by a generation id; stale/damaged sidecars are
|
||
ignored and the index rebuilt). `MemoryConfig::quantized_index` (**on by
|
||
default** for new stores, persisted; stores predating the setting load as
|
||
`false` and keep their f32 index — guarded by
|
||
`tests/fixtures/store_v2_5_0.h5`; CLI opt-out is `create --f32-index`)
|
||
stores the index's own copy of the embeddings as `i8`,
|
||
which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors
|
||
at 100K); because quantised distances are approximate and `ef` cannot
|
||
compensate, the query path then re-scores the candidate pool against the
|
||
exact embeddings, which holds recall at the f32 index's level. It is also
|
||
faster at equal recall: 1.63x the QPS on x86-64 (AVX2) and 1.18x on a
|
||
Raspberry Pi 5 (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm since
|
||
the intrinsic is unstable; plain NEON on pre-dotprod cores). The aarch64
|
||
code is `cfg`'d out on x86, so x86 CI never compiles or lints it — test it
|
||
on real ARM (`rpivision02`, 10.0.2.3, is a Pi 5). `hybrid_search` keeps one incremental BM25
|
||
index for the life of the store and never writes the store: Hebbian
|
||
activation boosts are persisted by the next checkpoint (or on drop), not per
|
||
query. Measure any search-path change with
|
||
`cargo run --release -p clawhdf5-bench --bin search_harness` (baselines in
|
||
`BENCHMARKS.md`).
|
||
- WAL (write-ahead log) for crash-safe persistence, with a chained CRC32
|
||
trailer per entry (each entry's CRC folds in the previous entry's CRC) so a
|
||
corrupted, reordered, duplicated, or spliced entry stops replay cleanly
|
||
instead of loading bad or tampered data. The pre-chaining per-entry-CRC
|
||
format (v2) is still fully readable; the oldest no-CRC format (v1) is only
|
||
reachable through the one-time migration path in `HDF5Memory::open`, not
|
||
through the public `WalFile::read_entries`.
|
||
**What the WAL guarantees:** integrity, ordering, and recovery from a
|
||
*process* crash at any point — including between a checkpoint and the WAL
|
||
truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips
|
||
the WAL prefix the `.h5` already contains, so entries are never applied
|
||
twice). Checkpoints and snapshots are made durable as a unit (temp file
|
||
synced, renamed, directory synced). **What it does not guarantee:**
|
||
individual WAL appends are *not* fsynced (a deliberate latency trade-off), so
|
||
saves made since the last checkpoint can be lost on power failure or kernel
|
||
panic. Current header version is 4 (adds the `Update` record used by
|
||
`save_or_update`); v3 files are read and upgraded in place.
|
||
- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive
|
||
advisory lock on `<store>.h5.lock` and a second opener gets
|
||
`MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free,
|
||
never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/
|
||
`export` do). An unreadable WAL (torn header, bad magic) is quarantined to
|
||
`<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
|
||
unknown *newer* version still fails and is left untouched.
|
||
- `MemoryConfig::float16` (**on by default** for new stores, persisted;
|
||
existing stores keep their recorded `false` — guarded by the v2.5.0
|
||
fixture in `tests/float16_store.rs`; CLI opt-out is `create --f32`) writes
|
||
`/memory/embeddings` as IEEE half precision (48% smaller file at 100K;
|
||
LongMemEval with real MiniLM embeddings identical to f32).
|
||
`MemoryCache::half_precision` rounds each embedding as it enters the cache (push, update, WAL replay, and on load of a store still
|
||
`f32` on disk), so memory and file agree bit for bit; the conversions live
|
||
in `clawhdf5_format::float16` and must stay the single implementation.
|
||
Values beyond ±65504 are `MemoryError::InvalidEntry`. Interop: every file
|
||
must open in h5py — `f32` datasets and empty datasets did not until
|
||
2026-09-23 (see `docs/known-issues.md`); the agent's `h5py_interop` test
|
||
guards a whole store.
|
||
- `HDF5Memory::search(query_emb, text, &SearchOptions)` is the full search
|
||
path: optional source-channel filter (applied before ranking; exact scan of
|
||
the allowed records whenever cheaper than `pool × M` index distance
|
||
evaluations, and as the fallback when the pool comes back short), fusion,
|
||
activation scaling, optional re-ranking and confidence rejection.
|
||
`hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend`
|
||
(the `openclaw` module) is `search` with re-rank + confidence on.
|
||
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an
|
||
OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"`
|
||
config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md`
|
||
records what a real plugin would need.
|
||
- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream
|
||
v0.8.5 and the `osobh/zeroclaw` fork, and their full history): no
|
||
`clawhdf5` feature or backend exists; ZeroClaw's memory backends are
|
||
sqlite/lucid/postgres/qdrant/markdown/none behind its own `Memory` trait.
|
||
`clawhdf5-migrate`'s default SQLite layout (`memory_chunks`, `sessions`,
|
||
`entities`, `relations`) is not ZeroClaw's schema either (ZeroClaw's is a
|
||
`memories` table). Don't reintroduce integration claims without an
|
||
integration and a test against the real consumer. Measure changes with
|
||
`search_harness --options-study`.
|
||
- `MemoryConfig::compression` is off by default; when on, embeddings are
|
||
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
|
||
- Signed checkpoints (`clawhdf5-agent` `signing` module): with
|
||
`HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed
|
||
manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph
|
||
hashes; per-record hashes in `/integrity/record_hashes`);
|
||
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly
|
||
what the file persists in the form the loader returns it (strings lose
|
||
trailing NULs; an empty WAL mark is not written) or untouched stores stop
|
||
verifying — `tests/signed_store.rs` round-trips awkward strings. The key is
|
||
never persisted; a signed store refuses to checkpoint without it
|
||
(`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`).
|
||
WAL entries after the checkpoint are not covered.
|
||
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
|
||
default) recomputes a dataset's SHA-256 and compares it against the
|
||
`_provenance_sha256` attribute written automatically on save when
|
||
`DatasetBuilder::with_provenance` is used. It's opt-in per call, not run
|
||
automatically on open — it decodes and hashes the whole dataset. The hash
|
||
is unkeyed (tamper-*evident*, not tamper-*proof*): it detects accidental
|
||
corruption, not a deliberate actor able to modify both the data and the
|
||
stored hash.
|
||
- `clawhdf5-agent`'s `HDF5Memory::save`/`save_batch`/`save_or_update` run every
|
||
write through an in-memory (session-scoped, not persisted to disk)
|
||
provenance ledger and write-anomaly detector: a content hash per record
|
||
(`provenance.rs`) for detecting accidental mid-session corruption, plus
|
||
rate-limit/injection-pattern/source-distribution checks (`anomaly.rs`).
|
||
Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
|
||
`MemorySource` for this bookkeeping is inferred from the caller-supplied
|
||
`source_channel` string (a heuristic, not an authenticated trust boundary).
|
||
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
|
||
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
|
||
no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.
|
||
`examples/wasm-viewer/test/run.sh` builds the package (needs the
|
||
`wasm-bindgen` CLI at the crate's exact version) and tests it under Node
|
||
and headless Chromium (a Playwright download in `~/.cache/ms-playwright`
|
||
on tank); the CI container has neither, so CI runs the native
|
||
`clawhdf5-wasm` `h5py_interop` test on the same fixture. Size numbers are
|
||
in the example's README.
|
||
- Python and Node.js bindings for cross-language use
|
||
- NetCDF-4 compatibility for scientific data interop
|
||
|
||
## Workflows
|
||
|
||
### Build
|
||
```bash
|
||
cargo build --release
|
||
```
|
||
|
||
### Test
|
||
```bash
|
||
cargo test --workspace
|
||
```
|
||
|
||
### CI
|
||
`.gitea/workflows/ci.yml` has two jobs, both green as of 2026-09-22:
|
||
- **`test`** (`ubuntu-latest`, in `rust:latest`) runs `scripts/ci-test.sh` with
|
||
the h5py/netCDF4 interop suites required (`CLAWHDF5_REQUIRE_INTEROP=1`).
|
||
Served by the `tank` and `architect` runners.
|
||
- **`test-arm64`** (`linux_arm64`) lints and tests the aarch64 code — the NEON
|
||
kernels are `cfg`'d out on x86, so this is the only place they are built.
|
||
Served by `vision-01` (host mode) and `vision-02` (Docker), so steps must
|
||
work in both.
|
||
|
||
Keep workflows free of JavaScript actions (`actions/checkout`, `actions/cache`,
|
||
…): `rust:latest` has no `node`, and not every runner reaches GitHub, where
|
||
they are fetched from. Check out with plain `git` instead. The `test` job
|
||
installs `cmake` for the opt-in `fast-deflate` (zlib-ng) steps; the default
|
||
build needs no C toolchain, so `test-arm64` does not.
|
||
All runners are on `gitea-runner` 3.5.0, from `docker.gitea.com/act_runner`
|
||
— `gitea/act_runner:latest` on Docker Hub is frozen at 0.6.1.
|
||
|
||
### CLI
|
||
```bash
|
||
cargo run -p clawhdf5-cli -- --help
|
||
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands
|
||
```
|
||
|
||
### HDF5 tools (`h5rs`, crate `clawhdf5-tools`)
|
||
```bash
|
||
cargo run -p clawhdf5-tools -- ls -r file.h5 # also dump [--json], stat, diff, check
|
||
bash scripts/h5rs-fuzz.sh # every subcommand over the CVE corpus: no panic/crash/hang
|
||
bash scripts/h5rs-check-ok-files.sh --data # check passes every fully-read conformance file
|
||
```
|
||
Its interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian
|
||
`hdf5-tools`, installed in CI); `dump` must stay byte-identical to h5dump on
|
||
the test files.
|
||
|
||
### Python bindings
|
||
```bash
|
||
cd crates/clawhdf5-py
|
||
maturin develop
|
||
python -c "import clawhdf5; print(clawhdf5.__version__)"
|
||
```
|
||
|
||
## Integration
|
||
- **ClawBrainHub** (`clawverse/clawbrainhub` on git.redclaw.dev) is the one
|
||
verified consumer: `cbh-core` reads and writes `.brain` files through the
|
||
facade (`File`, `FileBuilder`, `AttrValue`, `Selection`), `cbh-scanner`
|
||
uses the facade, and `cbh-cli` uses `clawhdf5_agent::bm25::BM25Index`. It
|
||
depends on this repo by path (`../clawhdf5`), so it builds against whatever
|
||
is checked out — changes to those APIs reach it directly. Verified
|
||
2026-09-25 against main: builds, and its 204 tests pass.
|
||
- OpenClaw and ZeroClaw were both described as consumers; neither integrates
|
||
clawhdf5 (see Key Features and `docs/openclaw.md`).
|