Omar Sobh
429c29b76b
CI / test (push) Failing after 3s
README.md: - Fix badly stale LongMemEval numbers (badge said Hit@5 46%, table showed fabricated ~46%/~0.34/~72% figures that never matched BENCHMARKS.md's actual results of Hit@5 100% session / 84.4% turn-level, MRR 1.0/0.6597) - Remove clawhdf5-types from the Crate Map — that crate was removed in an earlier cleanup pass but the README diagram was never updated; fix the crate count (16, not 17) and stale line-of-code figures (72,087/84K -> ~92K) - Fix a dead #benchmarks badge anchor (no such heading exists) -> #performance - Document the new clawhdf5-ann `parallel` feature (had no Feature Flags entry) - Note WAL's CRC32 per-entry check, link the new tank LongMemEval/SIMD/ vector-search reproduction section, update stale test-count comment (417+ -> 1,650+) and Phase 2 roadmap blurb (LongMemEval is now done) ROADMAP.md: - Check off "Academic benchmark cross-validation" (done via the tank LongMemEval re-run) and add a new "Recently closed out" section summarizing the Tier 3-4 hardening pass (Android JNI validation, pyo3 bump, WAL CRC32, bounds-check audit + fuzz harness that found 3 real bugs, HNSW optional parallel feature, workspace.dependencies) - Update stale test count (1,546 -> 1,650+) and last-updated date CLAUDE.md: mention WAL's per-entry CRC32 check CHANGELOG.md: add Security/Performance/Architecture/Documentation entries under Unreleased summarizing all of Tiers 1-4 (this had not been touched since 2026-06-04, predating the entire hardening pass)
68 lines
2.6 KiB
Markdown
68 lines
2.6 KiB
Markdown
# clawhdf5
|
|
|
|
## Purpose
|
|
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. Used by ZeroClaw as its persistent memory and knowledge graph backend.
|
|
|
|
## Architecture
|
|
|
|
Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
|
|
|
|
| Crate | Role |
|
|
|-------|------|
|
|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
|
| `clawhdf5-io` | Read/write implementation |
|
|
| `clawhdf5-filters` | Compression filters (gzip, LZ4, Zstd, Blosc) |
|
|
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
|
| `clawhdf5` | Main facade crate |
|
|
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
|
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
|
|
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
|
|
| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
|
|
| `clawhdf5-accel` | CPU SIMD acceleration path |
|
|
| `clawhdf5-migrate` | Schema migration engine |
|
|
| `clawhdf5-android` | Android JNI bindings |
|
|
| `clawhdf5-cli` | Command-line interface |
|
|
| `clawhdf5-napi` | Node.js native addon bindings |
|
|
| `clawhdf5-py` | PyO3 Python bindings |
|
|
| `clawhdf5-bench` | Benchmark suite |
|
|
|
|
## Key Features
|
|
- Zero-dependency HDF5 read/write (no libhdf5 C library required)
|
|
- HNSW vector index for semantic similarity search over agent memories — the
|
|
`clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses
|
|
the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
|
|
the cache and self-heals on drift). Build the agent with
|
|
`--no-default-features --features float16` to force the exact linear cosine scan.
|
|
- WAL (write-ahead log) for crash-safe persistence, with a CRC32 trailer per entry so a corrupted entry stops replay cleanly instead of loading bad data
|
|
- GPU-accelerated batch I/O for large dataset processing
|
|
- Python and Node.js bindings for cross-language use
|
|
- NetCDF-4 compatibility for scientific data interop
|
|
|
|
## Workflows
|
|
|
|
### Build
|
|
```bash
|
|
cargo build --release
|
|
```
|
|
|
|
### Test
|
|
```bash
|
|
cargo test --workspace
|
|
```
|
|
|
|
### CLI
|
|
```bash
|
|
cargo run -p clawhdf5-cli -- --help
|
|
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands
|
|
```
|
|
|
|
### Python bindings
|
|
```bash
|
|
cd crates/clawhdf5-py
|
|
maturin develop
|
|
python -c "import clawhdf5; print(clawhdf5.__version__)"
|
|
```
|
|
|
|
## Integration
|
|
ZeroClaw imports this as a Cargo feature (`clawhdf5` feature flag) to persist agent memory with HNSW vector search for context retrieval.
|