2.1 KiB
2.1 KiB
clawhdf5
Purpose
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. Used by ZeroClaw as its persistent memory and knowledge graph backend.
Architecture
Cargo workspace with 17 crates under crates/:
| Crate | Role |
|---|---|
clawhdf5-types |
Shared type definitions and physical constants |
clawhdf5-format |
HDF5 binary spec parser (superblock, B-tree, heap) |
clawhdf5-io |
Read/write implementation |
clawhdf5-filters |
Compression filters (gzip, LZ4, Zstd, Blosc) |
clawhdf5-derive |
Proc-macro derive for HDF5-serializable structs |
clawhdf5 |
Main facade crate |
clawhdf5-netcdf4 |
NetCDF-4 compatibility layer |
clawhdf5-ann |
HNSW approximate nearest-neighbor vector index |
clawhdf5-agent |
Agent memory, session history, knowledge graph storage |
clawhdf5-gpu |
GPU-accelerated I/O via CubeCL |
clawhdf5-accel |
CPU SIMD acceleration path |
clawhdf5-migrate |
Schema migration engine |
clawhdf5-android |
Android JNI bindings |
clawhdf5-cli |
Command-line interface |
clawhdf5-napi |
Node.js native addon bindings |
clawhdf5-py |
PyO3 Python bindings |
clawhdf5-bench |
Benchmark suite |
Key Features
- Zero-dependency HDF5 read/write (no libhdf5 C library required)
- HNSW vector index for semantic similarity search over agent memories
- WAL (write-ahead log) for crash-safe persistence
- GPU-accelerated batch I/O for large dataset processing
- Python and Node.js bindings for cross-language use
- NetCDF-4 compatibility for scientific data interop
Workflows
Build
cargo build --release
Test
cargo test --workspace
CLI
cargo run -p clawhdf5-cli -- --help
# inspect, dump, index, search subcommands
Python bindings
cd crates/clawhdf5-py
maturin develop
python -c "import clawhdf5; print(clawhdf5.__version__)"
Integration
ZeroClaw imports this as a Cargo feature (clawhdf5 feature flag) to persist agent memory with HNSW vector search for context retrieval.