clawhdf5/research/07-synthesis-next-steps.md

# Synthesis & Actionable Next Steps

*Research brief — generated 2026-08-12*

---

## 1. Executive Summary

ClawHDF5 is a mature, well-tested pure-Rust project with:
- **Complete HDF5 format coverage** for the most common real-world files (superblock v0–v4, all common filter codecs, fractal heaps, VDS, N-Bit, scale-offset)
- **A research-grade agent memory engine** with hybrid retrieval, knowledge graph, temporal reasoning, and anomaly detection — all proven on LongMemEval
- **Strong security baseline** for Environment A (untrusted file parsing): allocation guards, recursion depth caps, fuzz targets, CRC32 WAL integrity
- **Known gaps** in distribution (no published packages), encryption at rest, and some format edge cases (extensible arrays, huge objects, true collective MPI-IO)

The project is ready for **production use in its core use cases** (AI agent memory, HDF5 file I/O). The remaining work is primarily in hardening, publishing, and expanding the attack surface coverage.

---

## 2. Findings by Domain

### 2.1 Architecture
- 16-crate workspace with clear separation between format, I/O, agent, and bindings layers
- The `no_std` path works and is CI-checked; the embedded use case is viable
- HNSW is the right default vector backend; the self-healing rebuild mechanism is a good robustness choice
- The RRF hybrid pipeline design is well-founded in research; the 0.4/0.6 weight finding is a concrete, immediately actionable improvement

### 2.2 Performance
- The biggest single improvement available is **changing the hybrid search default weights from 0.7/0.3 to 0.4/0.6** — a 30-minute change that yields +~6pp on retrieval recall
- **Parallel chunk decompression** is the highest-effort-to-reward performance win (~2× read throughput for large chunked datasets, ~1–2 days effort)
- **Persistent BM25 index** eliminates startup rebuild time that will become significant at 100K+ records
- HNSW build parallelism is the highest-effort item but also the highest absolute-scale win

### 2.3 Robustness
- The bounds-check audit is ~70% complete; the remaining `unwrap()` audit and additional fuzz targets should close this
- WAL robustness is good but lacks an atomic commit marker for the flush path
- Knowledge graph BFS has no cycle guard (easy to add)
- Android JNI has no thread-safety guarantee (medium risk)

### 2.4 Security
- Encryption at rest is entirely absent — the most significant security gap for `.brain` file and personal-data use cases
- File signing (Ed25519) is absent — limits trust for distributed `.brain` files
- Embedding-space poisoning detection is absent — text-level anomaly detection is not sufficient against sophisticated adversaries
- Supply-chain hygiene (`cargo-audit`, `cargo-deny`) is not automated

---

## 3. Actionable Next Steps

### Immediate (< 1 week, zero risk)

**STEP-1: Fix hybrid search default weights**
- File: `crates/clawhdf5-agent/src/hybrid.rs`
- Change: Default weight from `(0.7, 0.3)` to `(0.4, 0.6)` (vector, keyword)
- Validation: Run LongMemEval benchmark and confirm improvement
- Impact: +~6pp turn-level Hit@5 for all users who don't override the default

**STEP-2: Add `overflow-checks = true` to release profile for format crate**
- File: `crates/clawhdf5-format/Cargo.toml` (or root `Cargo.toml` `[profile.release]`)
- Change: `overflow-checks = true` scoped to `clawhdf5-format`
- Validation: `cargo test -p clawhdf5-format --release` passes
- Impact: Defense-in-depth for untrusted file parsing

**STEP-3: Add `cargo-audit` to CI**
- File: `.gitea/workflows/ci.yml`
- Change: Add step `cargo audit --deny warnings`
- Impact: Continuous dependency advisory monitoring; catches RUSTSEC advisories before they reach users

**STEP-4: Publish workspace to crates.io / npm / PyPI**
- Add `publish = true` + `categories` + `keywords` to all public crate `Cargo.toml` files
- Commit `packages/clawhdf5-node/package-lock.json`
- Add `maturin` wheel build step to CI for Python
- Add `npm ci && npx jest` step to CI for Node.js
- Impact: Discoverability; external contribution; ecosystem adoption

### Short-Term (1–4 weeks)

**STEP-5: Knowledge graph cycle guard**
- File: `crates/clawhdf5-agent/src/knowledge.rs`
- Change: Add `visited: HashSet<EntityId>` to `bfs_neighbors` and `spreading_activation`
- Validation: Add test with a cyclic graph
- Impact: Prevents infinite loops on corrupted or adversarially constructed graphs

**STEP-6: WAL fuzz target**
- File: `crates/clawhdf5-agent/fuzz/fuzz_targets/fuzz_wal_replay.rs`
- Change: Feed arbitrary byte sequences into WAL replay path
- Validation: Run for 1 hour; no crashes or panics
- Impact: Verify CRC32 guard correctly short-circuits before any allocation on all malformed inputs

**STEP-7: Parallel chunk decompression**
- File: `crates/clawhdf5-format/src/chunked_read.rs`
- Change: Add rayon `par_iter` over independent chunks when `parallel` feature is enabled
- Validation: Criterion benchmark shows ~2× improvement for multi-chunk datasets
- Impact: ~2× read throughput for large embeddings matrix reads

**STEP-8: JNI `Mutex` wrapping**
- File: `crates/clawhdf5-android/src/lib.rs`
- Change: Store `Box<Mutex<HDF5Memory>>` instead of `Box<HDF5Memory>`; wrap all JNI fn bodies with `lock().unwrap()`
- Validation: Multi-threaded Android test (or a synthetic concurrent test in CI)
- Impact: Prevent data races on multi-threaded Android apps

**STEP-9: Persistent BM25 index**
- Files: `crates/clawhdf5-agent/src/bm25.rs`, HDF5 schema under `/index/bm25/`
- Change: Serialize posting lists to HDF5 on flush; deserialize on open
- Validation: Verify BM25 search results are identical with/without persistence; measure startup time at 100K records
- Impact: Eliminates O(N) rebuild on restart for large corpora

**STEP-10: Media reference sandboxing**
- File: `crates/clawhdf5-agent/src/multimodal.rs`
- Change: Add `media_sandbox_dir: Option<PathBuf>` to `MemoryConfig`; validate and canonicalize `MediaRef::Path` before resolution; add URL scheme allowlist for `MediaRef::Url`
- Impact: Prevents path traversal attacks via adversarial memory content

### Medium-Term (1–2 months)

**STEP-11: AES-256-GCM encryption at rest**
- Add `encryption` feature using `aes-gcm` + `argon2` crates
- Encrypt each chunk's data + WAL entries with AES-256-GCM
- API: `MemoryConfig::with_passphrase(passphrase: &str)`
- Impact: Confidentiality for `.brain` files and personal agent memories

**STEP-12: Ed25519 file signing**
- Add `signing` feature using `ed25519-dalek`
- Sign the full provenance tree (all dataset SHA-256 hashes) with an Ed25519 key
- CLI: `clawhdf5-cli sign --key signing.key memory.h5`; `clawhdf5-cli verify memory.h5`
- Impact: Tamper detection for distributed `.brain` files on ClawBrainHub

**STEP-13: HNSW batch insert parallelism**
- File: `crates/clawhdf5-ann/src/lib.rs`
- Change: Group inserts into batches; process batches with a coarse lock; explore lock-free per-node locking
- Validation: Correctness tests under concurrent insert + search; Criterion shows improvement
- Impact: 2–4× faster index build time at 100K+ vectors

**STEP-14: Benchmark CI regression gate**
- Add `cargo bench --save-baseline main` to CI on merge to main
- Add a comparison step on PRs: `cargo bench --load-baseline main -- --verbose 2>&1 | grep "Performance has regressed"`
- Impact: Catch performance regressions before they reach users

**STEP-15: Embedding-space anomaly detection**
- File: `crates/clawhdf5-agent/src/anomaly.rs`
- Add Mahalanobis distance outlier detection on new embeddings
- Quarantine embeddings from `MemorySource::Tool` pending explicit promotion
- Impact: Defense against embedding-space poisoning attacks (MemoryGraft class of attacks)

### Long-Term (2+ months)

**STEP-16: True collective MPI-IO**
- File: `crates/clawhdf5-io/src/mpi_io.rs`
- Replace root-read + broadcast with `MPI_File_read_at_all` / `MPI_File_write_at_all`
- Impact: HPC scalability — I/O bandwidth now scales with rank count

**STEP-17: Blosc2 filter support**
- Filter id 32001, via `blosc2-sys` FFI or a pure-Rust implementation
- Impact: Read compatibility with the most widely-used third-party HDF5 filter in scientific Python

**STEP-18: Matryoshka / variable-dimension embedding support**
- Allow `embedding_dim` to be a maximum dimension with a stored per-vector actual dimension
- Support truncated cosine search at reduced dimensions
- Impact: Alignment with OpenAI `text-embedding-3-small` and other MRL-trained models

---

## 4. Task Markers

TASK: INT-01 — Fix hybrid search default weights to 0.4/0.6
TASK: INT-02 — Add overflow-checks=true to format crate release profile
TASK: INT-03 — Add cargo-audit step to Gitea CI
TASK: INT-04 — Publish clawhdf5-* to crates.io; npm; PyPI
TASK: INT-05 — Add cycle guard to knowledge graph BFS and spreading activation
TASK: INT-06 — Add WAL replay fuzz target
TASK: INT-07 — Implement parallel chunk decompression (rayon, parallel feature)
TASK: INT-08 — Wrap Android JNI handles in Mutex for thread safety
TASK: INT-09 — Implement persistent BM25 index (serialize/deserialize to HDF5)
TASK: INT-10 — Add media reference sandboxing (path canonicalization + URL allowlist)
TASK: INT-11 — Implement AES-256-GCM encryption at rest (encryption feature)
TASK: INT-12 — Implement Ed25519 file signing (signing feature + CLI commands)
TASK: INT-13 — HNSW batch insert parallelism (design pass + implementation)
TASK: INT-14 — Add Criterion benchmark regression gate to CI
TASK: INT-15 — Embedding-space anomaly detection (Mahalanobis + source quarantine)