# Synthesis & Actionable Next Steps *Research brief — generated 2026-08-12* --- ## 1. Executive Summary ClawHDF5 is a mature, well-tested pure-Rust project with: - **Complete HDF5 format coverage** for the most common real-world files (superblock v0–v4, all common filter codecs, fractal heaps, VDS, N-Bit, scale-offset) - **A research-grade agent memory engine** with hybrid retrieval, knowledge graph, temporal reasoning, and anomaly detection — all proven on LongMemEval - **Strong security baseline** for Environment A (untrusted file parsing): allocation guards, recursion depth caps, fuzz targets, CRC32 WAL integrity - **Known gaps** in distribution (no published packages), encryption at rest, and some format edge cases (extensible arrays, huge objects, true collective MPI-IO) The project is ready for **production use in its core use cases** (AI agent memory, HDF5 file I/O). The remaining work is primarily in hardening, publishing, and expanding the attack surface coverage. --- ## 2. Findings by Domain ### 2.1 Architecture - 16-crate workspace with clear separation between format, I/O, agent, and bindings layers - The `no_std` path works and is CI-checked; the embedded use case is viable - HNSW is the right default vector backend; the self-healing rebuild mechanism is a good robustness choice - The RRF hybrid pipeline design is well-founded in research; the 0.4/0.6 weight finding is a concrete, immediately actionable improvement ### 2.2 Performance - The biggest single improvement available is **changing the hybrid search default weights from 0.7/0.3 to 0.4/0.6** — a 30-minute change that yields +~6pp on retrieval recall - **Parallel chunk decompression** is the highest-effort-to-reward performance win (~2× read throughput for large chunked datasets, ~1–2 days effort) - **Persistent BM25 index** eliminates startup rebuild time that will become significant at 100K+ records - HNSW build parallelism is the highest-effort item but also the highest absolute-scale win ### 2.3 Robustness - The bounds-check audit is ~70% complete; the remaining `unwrap()` audit and additional fuzz targets should close this - WAL robustness is good but lacks an atomic commit marker for the flush path - Knowledge graph BFS has no cycle guard (easy to add) - Android JNI has no thread-safety guarantee (medium risk) ### 2.4 Security - Encryption at rest is entirely absent — the most significant security gap for `.brain` file and personal-data use cases - File signing (Ed25519) is absent — limits trust for distributed `.brain` files - Embedding-space poisoning detection is absent — text-level anomaly detection is not sufficient against sophisticated adversaries - Supply-chain hygiene (`cargo-audit`, `cargo-deny`) is not automated --- ## 3. Actionable Next Steps ### Immediate (< 1 week, zero risk) **STEP-1: Fix hybrid search default weights** - File: `crates/clawhdf5-agent/src/hybrid.rs` - Change: Default weight from `(0.7, 0.3)` to `(0.4, 0.6)` (vector, keyword) - Validation: Run LongMemEval benchmark and confirm improvement - Impact: +~6pp turn-level Hit@5 for all users who don't override the default **STEP-2: Add `overflow-checks = true` to release profile for format crate** - File: `crates/clawhdf5-format/Cargo.toml` (or root `Cargo.toml` `[profile.release]`) - Change: `overflow-checks = true` scoped to `clawhdf5-format` - Validation: `cargo test -p clawhdf5-format --release` passes - Impact: Defense-in-depth for untrusted file parsing **STEP-3: Add `cargo-audit` to CI** - File: `.gitea/workflows/ci.yml` - Change: Add step `cargo audit --deny warnings` - Impact: Continuous dependency advisory monitoring; catches RUSTSEC advisories before they reach users **STEP-4: Publish workspace to crates.io / npm / PyPI** - Add `publish = true` + `categories` + `keywords` to all public crate `Cargo.toml` files - Commit `packages/clawhdf5-node/package-lock.json` - Add `maturin` wheel build step to CI for Python - Add `npm ci && npx jest` step to CI for Node.js - Impact: Discoverability; external contribution; ecosystem adoption ### Short-Term (1–4 weeks) **STEP-5: Knowledge graph cycle guard** - File: `crates/clawhdf5-agent/src/knowledge.rs` - Change: Add `visited: HashSet` to `bfs_neighbors` and `spreading_activation` - Validation: Add test with a cyclic graph - Impact: Prevents infinite loops on corrupted or adversarially constructed graphs **STEP-6: WAL fuzz target** - File: `crates/clawhdf5-agent/fuzz/fuzz_targets/fuzz_wal_replay.rs` - Change: Feed arbitrary byte sequences into WAL replay path - Validation: Run for 1 hour; no crashes or panics - Impact: Verify CRC32 guard correctly short-circuits before any allocation on all malformed inputs **STEP-7: Parallel chunk decompression** - File: `crates/clawhdf5-format/src/chunked_read.rs` - Change: Add rayon `par_iter` over independent chunks when `parallel` feature is enabled - Validation: Criterion benchmark shows ~2× improvement for multi-chunk datasets - Impact: ~2× read throughput for large embeddings matrix reads **STEP-8: JNI `Mutex` wrapping** - File: `crates/clawhdf5-android/src/lib.rs` - Change: Store `Box>` instead of `Box`; wrap all JNI fn bodies with `lock().unwrap()` - Validation: Multi-threaded Android test (or a synthetic concurrent test in CI) - Impact: Prevent data races on multi-threaded Android apps **STEP-9: Persistent BM25 index** - Files: `crates/clawhdf5-agent/src/bm25.rs`, HDF5 schema under `/index/bm25/` - Change: Serialize posting lists to HDF5 on flush; deserialize on open - Validation: Verify BM25 search results are identical with/without persistence; measure startup time at 100K records - Impact: Eliminates O(N) rebuild on restart for large corpora **STEP-10: Media reference sandboxing** - File: `crates/clawhdf5-agent/src/multimodal.rs` - Change: Add `media_sandbox_dir: Option` to `MemoryConfig`; validate and canonicalize `MediaRef::Path` before resolution; add URL scheme allowlist for `MediaRef::Url` - Impact: Prevents path traversal attacks via adversarial memory content ### Medium-Term (1–2 months) **STEP-11: AES-256-GCM encryption at rest** - Add `encryption` feature using `aes-gcm` + `argon2` crates - Encrypt each chunk's data + WAL entries with AES-256-GCM - API: `MemoryConfig::with_passphrase(passphrase: &str)` - Impact: Confidentiality for `.brain` files and personal agent memories **STEP-12: Ed25519 file signing** - Add `signing` feature using `ed25519-dalek` - Sign the full provenance tree (all dataset SHA-256 hashes) with an Ed25519 key - CLI: `clawhdf5-cli sign --key signing.key memory.h5`; `clawhdf5-cli verify memory.h5` - Impact: Tamper detection for distributed `.brain` files on ClawBrainHub **STEP-13: HNSW batch insert parallelism** - File: `crates/clawhdf5-ann/src/lib.rs` - Change: Group inserts into batches; process batches with a coarse lock; explore lock-free per-node locking - Validation: Correctness tests under concurrent insert + search; Criterion shows improvement - Impact: 2–4× faster index build time at 100K+ vectors **STEP-14: Benchmark CI regression gate** - Add `cargo bench --save-baseline main` to CI on merge to main - Add a comparison step on PRs: `cargo bench --load-baseline main -- --verbose 2>&1 | grep "Performance has regressed"` - Impact: Catch performance regressions before they reach users **STEP-15: Embedding-space anomaly detection** - File: `crates/clawhdf5-agent/src/anomaly.rs` - Add Mahalanobis distance outlier detection on new embeddings - Quarantine embeddings from `MemorySource::Tool` pending explicit promotion - Impact: Defense against embedding-space poisoning attacks (MemoryGraft class of attacks) ### Long-Term (2+ months) **STEP-16: True collective MPI-IO** - File: `crates/clawhdf5-io/src/mpi_io.rs` - Replace root-read + broadcast with `MPI_File_read_at_all` / `MPI_File_write_at_all` - Impact: HPC scalability — I/O bandwidth now scales with rank count **STEP-17: Blosc2 filter support** - Filter id 32001, via `blosc2-sys` FFI or a pure-Rust implementation - Impact: Read compatibility with the most widely-used third-party HDF5 filter in scientific Python **STEP-18: Matryoshka / variable-dimension embedding support** - Allow `embedding_dim` to be a maximum dimension with a stored per-vector actual dimension - Support truncated cosine search at reduced dimensions - Impact: Alignment with OpenAI `text-embedding-3-small` and other MRL-trained models --- ## 4. Task Markers TASK: INT-01 — Fix hybrid search default weights to 0.4/0.6 TASK: INT-02 — Add overflow-checks=true to format crate release profile TASK: INT-03 — Add cargo-audit step to Gitea CI TASK: INT-04 — Publish clawhdf5-* to crates.io; npm; PyPI TASK: INT-05 — Add cycle guard to knowledge graph BFS and spreading activation TASK: INT-06 — Add WAL replay fuzz target TASK: INT-07 — Implement parallel chunk decompression (rayon, parallel feature) TASK: INT-08 — Wrap Android JNI handles in Mutex for thread safety TASK: INT-09 — Implement persistent BM25 index (serialize/deserialize to HDF5) TASK: INT-10 — Add media reference sandboxing (path canonicalization + URL allowlist) TASK: INT-11 — Implement AES-256-GCM encryption at rest (encryption feature) TASK: INT-12 — Implement Ed25519 file signing (signing feature + CLI commands) TASK: INT-13 — HNSW batch insert parallelism (design pass + implementation) TASK: INT-14 — Add Criterion benchmark regression gate to CI TASK: INT-15 — Embedding-space anomaly detection (Mahalanobis + source quarantine)