clawhdf5/research/07-synthesis-next-steps.md

Synthesis & Actionable Next Steps

Research brief — generated 2026-08-12

---

1. Executive Summary

ClawHDF5 is a mature, well-tested pure-Rust project with:

• Complete HDF5 format coverage for the most common real-world files (superblock v0–v4, all common filter codecs, fractal heaps, VDS, N-Bit, scale-offset)
• A research-grade agent memory engine with hybrid retrieval, knowledge graph, temporal reasoning, and anomaly detection — all proven on LongMemEval
• Strong security baseline for Environment A (untrusted file parsing): allocation guards, recursion depth caps, fuzz targets, CRC32 WAL integrity
• Known gaps in distribution (no published packages), encryption at rest, and some format edge cases (extensible arrays, huge objects, true collective MPI-IO)

The project is ready for production use in its core use cases (AI agent memory, HDF5 file I/O). The remaining work is primarily in hardening, publishing, and expanding the attack surface coverage.

---

2. Findings by Domain

2.1 Architecture

• 16-crate workspace with clear separation between format, I/O, agent, and bindings layers
• The no_std path works and is CI-checked; the embedded use case is viable
• HNSW is the right default vector backend; the self-healing rebuild mechanism is a good robustness choice
• The RRF hybrid pipeline design is well-founded in research; the 0.4/0.6 weight finding is a concrete, immediately actionable improvement

2.2 Performance

• The biggest single improvement available is changing the hybrid search default weights from 0.7/0.3 to 0.4/0.6 — a 30-minute change that yields +~6pp on retrieval recall
• Parallel chunk decompression is the highest-effort-to-reward performance win (~2× read throughput for large chunked datasets, ~1–2 days effort)
• Persistent BM25 index eliminates startup rebuild time that will become significant at 100K+ records
• HNSW build parallelism is the highest-effort item but also the highest absolute-scale win

2.3 Robustness

• The bounds-check audit is ~70% complete; the remaining unwrap() audit and additional fuzz targets should close this
• WAL robustness is good but lacks an atomic commit marker for the flush path
• Knowledge graph BFS has no cycle guard (easy to add)
• Android JNI has no thread-safety guarantee (medium risk)

2.4 Security

• Encryption at rest is entirely absent — the most significant security gap for .brain file and personal-data use cases
• File signing (Ed25519) is absent — limits trust for distributed .brain files
• Embedding-space poisoning detection is absent — text-level anomaly detection is not sufficient against sophisticated adversaries
• Supply-chain hygiene (cargo-audit, cargo-deny) is not automated

---

3. Actionable Next Steps

Immediate (< 1 week, zero risk)

STEP-1: Fix hybrid search default weights

• File: crates/clawhdf5-agent/src/hybrid.rs
• Change: Default weight from (0.7, 0.3) to (0.4, 0.6) (vector, keyword)
• Validation: Run LongMemEval benchmark and confirm improvement
• Impact: +~6pp turn-level Hit@5 for all users who don't override the default

STEP-2: Add overflow-checks = true to release profile for format crate

• File: crates/clawhdf5-format/Cargo.toml (or root Cargo.toml [profile.release])
• Change: overflow-checks = true scoped to clawhdf5-format
• Validation: cargo test -p clawhdf5-format --release passes
• Impact: Defense-in-depth for untrusted file parsing

STEP-3: Add cargo-audit to CI

• File: .gitea/workflows/ci.yml
• Change: Add step cargo audit --deny warnings
• Impact: Continuous dependency advisory monitoring; catches RUSTSEC advisories before they reach users

STEP-4: Publish workspace to crates.io / npm / PyPI

• Add publish = true + categories + keywords to all public crate Cargo.toml files
• Commit packages/clawhdf5-node/package-lock.json
• Add maturin wheel build step to CI for Python
• Add npm ci && npx jest step to CI for Node.js
• Impact: Discoverability; external contribution; ecosystem adoption

Short-Term (1–4 weeks)

STEP-5: Knowledge graph cycle guard

• File: crates/clawhdf5-agent/src/knowledge.rs
• Change: Add visited: HashSet<EntityId> to bfs_neighbors and spreading_activation
• Validation: Add test with a cyclic graph
• Impact: Prevents infinite loops on corrupted or adversarially constructed graphs

STEP-6: WAL fuzz target

• File: crates/clawhdf5-agent/fuzz/fuzz_targets/fuzz_wal_replay.rs
• Change: Feed arbitrary byte sequences into WAL replay path
• Validation: Run for 1 hour; no crashes or panics
• Impact: Verify CRC32 guard correctly short-circuits before any allocation on all malformed inputs

STEP-7: Parallel chunk decompression

• File: crates/clawhdf5-format/src/chunked_read.rs
• Change: Add rayon par_iter over independent chunks when parallel feature is enabled
• Validation: Criterion benchmark shows ~2× improvement for multi-chunk datasets
• Impact: ~2× read throughput for large embeddings matrix reads

STEP-8: JNI Mutex wrapping

• File: crates/clawhdf5-android/src/lib.rs
• Change: Store Box<Mutex<HDF5Memory>> instead of Box<HDF5Memory>; wrap all JNI fn bodies with lock().unwrap()
• Validation: Multi-threaded Android test (or a synthetic concurrent test in CI)
• Impact: Prevent data races on multi-threaded Android apps

STEP-9: Persistent BM25 index

• Files: crates/clawhdf5-agent/src/bm25.rs, HDF5 schema under /index/bm25/
• Change: Serialize posting lists to HDF5 on flush; deserialize on open
• Validation: Verify BM25 search results are identical with/without persistence; measure startup time at 100K records
• Impact: Eliminates O(N) rebuild on restart for large corpora

STEP-10: Media reference sandboxing

• File: crates/clawhdf5-agent/src/multimodal.rs
• Change: Add media_sandbox_dir: Option<PathBuf> to MemoryConfig; validate and canonicalize MediaRef::Path before resolution; add URL scheme allowlist for MediaRef::Url
• Impact: Prevents path traversal attacks via adversarial memory content

Medium-Term (1–2 months)

STEP-11: AES-256-GCM encryption at rest

• Add encryption feature using aes-gcm + argon2 crates
• Encrypt each chunk's data + WAL entries with AES-256-GCM
• API: MemoryConfig::with_passphrase(passphrase: &str)
• Impact: Confidentiality for .brain files and personal agent memories

STEP-12: Ed25519 file signing

• Add signing feature using ed25519-dalek
• Sign the full provenance tree (all dataset SHA-256 hashes) with an Ed25519 key
• CLI: clawhdf5-cli sign --key signing.key memory.h5; clawhdf5-cli verify memory.h5
• Impact: Tamper detection for distributed .brain files on ClawBrainHub

STEP-13: HNSW batch insert parallelism

• File: crates/clawhdf5-ann/src/lib.rs
• Change: Group inserts into batches; process batches with a coarse lock; explore lock-free per-node locking
• Validation: Correctness tests under concurrent insert + search; Criterion shows improvement
• Impact: 2–4× faster index build time at 100K+ vectors

STEP-14: Benchmark CI regression gate

• Add cargo bench --save-baseline main to CI on merge to main
• Add a comparison step on PRs: cargo bench --load-baseline main -- --verbose 2>&1 | grep "Performance has regressed"
• Impact: Catch performance regressions before they reach users

STEP-15: Embedding-space anomaly detection

• File: crates/clawhdf5-agent/src/anomaly.rs
• Add Mahalanobis distance outlier detection on new embeddings
• Quarantine embeddings from MemorySource::Tool pending explicit promotion
• Impact: Defense against embedding-space poisoning attacks (MemoryGraft class of attacks)

Long-Term (2+ months)

STEP-16: True collective MPI-IO

• File: crates/clawhdf5-io/src/mpi_io.rs
• Replace root-read + broadcast with MPI_File_read_at_all / MPI_File_write_at_all
• Impact: HPC scalability — I/O bandwidth now scales with rank count

STEP-17: Blosc2 filter support

• Filter id 32001, via blosc2-sys FFI or a pure-Rust implementation
• Impact: Read compatibility with the most widely-used third-party HDF5 filter in scientific Python

STEP-18: Matryoshka / variable-dimension embedding support

• Allow embedding_dim to be a maximum dimension with a stored per-vector actual dimension
• Support truncated cosine search at reduced dimensions
• Impact: Alignment with OpenAI text-embedding-3-small and other MRL-trained models

---

4. Task Markers

TASK: INT-01 — Fix hybrid search default weights to 0.4/0.6 TASK: INT-02 — Add overflow-checks=true to format crate release profile TASK: INT-03 — Add cargo-audit step to Gitea CI TASK: INT-04 — Publish clawhdf5-* to crates.io; npm; PyPI TASK: INT-05 — Add cycle guard to knowledge graph BFS and spreading activation TASK: INT-06 — Add WAL replay fuzz target TASK: INT-07 — Implement parallel chunk decompression (rayon, parallel feature) TASK: INT-08 — Wrap Android JNI handles in Mutex for thread safety TASK: INT-09 — Implement persistent BM25 index (serialize/deserialize to HDF5) TASK: INT-10 — Add media reference sandboxing (path canonicalization + URL allowlist) TASK: INT-11 — Implement AES-256-GCM encryption at rest (encryption feature) TASK: INT-12 — Implement Ed25519 file signing (signing feature + CLI commands) TASK: INT-13 — HNSW batch insert parallelism (design pass + implementation) TASK: INT-14 — Add Criterion benchmark regression gate to CI TASK: INT-15 — Embedding-space anomaly detection (Mahalanobis + source quarantine)