clawhdf5/ROADMAP.md

# ClawhDF5 Roadmap — Agent Memory Evolution

> Making clawhdf5 the defacto agentic memory solution.
> Single file. Pure Rust. Zero dependencies. Trusted everywhere.

---

## Track 1: Knowledge Graph in HDF5
**Status:** 🟢 Phase 1 Complete
**Priority:** Critical
**Crate:** `clawhdf5-agent`

- [x] **1.1** Entity storage — entities with properties, embeddings, timestamps (created_at/updated_at)
- [x] **1.2** Relation storage — typed edges with RelationType enum (Temporal/Causal/Associative/Hierarchical/Custom), metadata, timestamps
- [x] **1.3** Entity extraction helpers — rule-based extraction (Person, Org, Location, Date, Technology, Project) with extract_and_store_entities() integration
- [x] **1.4** Entity resolution — fuzzy name matching (Levenshtein distance) via resolve_or_create()
- [x] **1.5** Graph traversal queries — BFS neighbors with depth, subgraph extraction from seeds
- [x] **1.6** Spreading activation — weighted activation propagation with configurable decay
- [x] **1.7** Graph-aware retrieval — get_entity_context() for formatted context injection
- [x] **1.8** Tests — comprehensive tests for all new features

**Research:** Graph-Native Cognitive Memory (2026), Graph-based Agent Memory survey (2026), SYNAPSE (2025)

---

## Track 2: Memory Consolidation Engine
**Status:** 🟢 Phase 1 Complete
**Priority:** Critical
**Crate:** `clawhdf5-agent`

- [x] **2.1** Importance scoring — surprise (novelty), correction boost, length scoring with configurable weights
- [x] **2.2** Three-tier memory model — Working → Episodic → Semantic with bounded capacities
- [x] **2.3** Time-decay with reactivation — exponential decay with configurable half-life, access resets timestamp
- [x] **2.4** Bounded memory with graceful degradation — evict lowest-decay entries when over capacity
- [x] **2.5** Consolidation cycles — promote/evict across tiers based on importance and access thresholds
- [x] **2.6** Memory statistics — ConsolidationStats with per-tier counts, eviction/promotion tracking
- [x] **2.7** Tests — comprehensive tests for all features

**Research:** CraniMem (2026), D-MEM (2026), AI Hippocampus survey (2026)

---

## Track 3: Hybrid Retrieval Pipeline
**Status:** 🟢 Phase 1 Complete
**Priority:** High
**Crate:** `clawhdf5-agent`

- [x] **3.1** Reciprocal Rank Fusion (RRF) — rrf_hybrid_search() with k=60 constant
- [x] **3.2** Multi-factor re-ranking — temporal decay, source authority hierarchy, activation scores (reranker.rs)
- [x] **3.3** Low-confidence rejection — min_score threshold, gap filtering, max_results (confidence.rs)
- [x] **3.4** Query expansion — synonyms, acronyms, temporal rewrites, morphological variants, knowledge graph aliases + expanded_search() with RRF merge
- [x] **3.5** Result explanation — ReRankResult with full score breakdown per factor
- [x] **3.6** Configurable pipeline — ReRankConfig + ConfidenceConfig with tunable weights/thresholds
- [x] **3.7** Tests + MemX-comparable benchmarks — 5 integration tests (Hit@1≥90%, search<500ms@100K, BM25<200ms@100K, hybrid<50ms@10K, compact<200ms@10K)

**Research:** MemX (2026), SwiftMem (2026)

---

## Track 4: Temporal Reasoning
**Status:** 🟢 Phase 1 Complete
**Priority:** High
**Crate:** `clawhdf5-agent`

- [x] **4.1** Temporal index — sorted timestamp index with binary search, insert/remove
- [x] **4.2** Time-range queries — range_query, before, after, latest, earliest
- [x] **4.3** Session DAG — parent/child linking, chain walking, time-range overlap queries
- [x] **4.4** Temporal re-ranking — query hint enum (Latest/Earliest/Around/Between/None) with boost scoring
- [x] **4.5** Temporal entity tracking — EntityTimeline with state change history + point-in-time reconstruction
- [x] **4.6** Tests — comprehensive tests for all features

**Research:** MemX temporal gaps (≤43.6% Hit@5), MemoryArena multi-session tasks (2026)

---

## Track 5: Memory Security & Provenance
**Status:** 🟢 Phase 1 Complete
**Priority:** Medium-High
**Crate:** `clawhdf5-agent`

- [x] **5.1** Source attribution — MemoryProvenance with source, creator, session, FNV-1a content hash
- [x] **5.2** Write anomaly detection — rate limiting, 15 injection patterns, source distribution analysis
- [x] **5.3** Source isolation — per-MemorySource sub-stores preventing cross-contamination
- [x] **5.4** Memory integrity verification — content hash comparison via verify_integrity()
- [x] **5.5** Poisoning resistance — pattern detection for prompt injection attempts
- [x] **5.6** Tests — comprehensive tests including adversarial patterns

**Research:** MemoryGraft (2025), SSGM Framework (2026)

---

## Track 6: Multi-Modal Memory
**Status:** 🟢 Phase 1 Complete
**Priority:** Medium
**Crate:** `clawhdf5-agent`

- [x] **6.1** Image embedding storage — ModalEmbedding with model provenance (CLIP, SigLIP, etc.)
- [x] **6.2** Audio fingerprints — Audio modality with embedding storage
- [x] **6.3** Multi-modal search — search_by_modality (filtered) + search_cross_modal (all embeddings)
- [x] **6.4** Observation records — raw perception vs interpretation with confidence scoring
- [x] **6.5** Media reference storage — MediaRef with Path/Url/Inline, MIME types, FNV-1a checksums
- [x] **6.6** Tests — 35 comprehensive tests

**Research:** Neuro-Symbolic Memory (2026), RAGdb multi-modal RAG (2025)

---

## Track 7: OpenClaw Integration — withdrawn (2026-09-25)
**Status:** ⚪ Withdrawn (the items below were library work; no OpenClaw integration shipped)
**Priority:** Critical (for adoption)
**Crates:** `clawhdf5-agent`, `clawhdf5-napi`

- [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
- [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
- [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
- [x] **7.4** `search()` — backed by the full hybrid retrieval pipeline (a Rust method; no OpenClaw tool was ever registered)
- [x] **7.5** `get()` — read back by path, with a line slice (not an OpenClaw tool either)
- [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
- [ ] **7.7** ~~Config surface — `memory.backend = "clawhdf5"`~~ — never valid OpenClaw config; docs removed
- [ ] **7.8** ~~Documentation + migration guide~~ — removed: they described an integration that never worked

**Node.js bridge:** `clawhdf5-napi` (napi-rs) and a TypeScript wrapper in `packages/clawhdf5-node` exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).

---

> **Withdrawn.** None of this track produced a working OpenClaw integration: no
> plugin was built, the documented `memory.backend = "clawhdf5"` config was never
> valid in any OpenClaw release, and the Node package was never published. The
> Rust `ClawhdfBackend` remains as a library API. Not pursued for now; see
> [docs/openclaw.md](docs/openclaw.md) for what a plugin would need today.

## Track 8: Benchmarking & Validation
**Status:** 🟢 Complete
**Priority:** High
**Crates:** `clawhdf5-agent`, `clawhdf5-bench`

- [x] **8.1** MemoryArena benchmark — 35 queries, 50 sessions, Hit@10=91.4%, MRR=0.547
- [x] **8.2** LongMemEval benchmark — 500 queries, session Hit@1=100%, turn Hit@5=84.4% (beats MemX 51.6%), MRR=0.660
- [x] **8.3** Latency benchmarks — vector search at 1K/10K/100K, hybrid/RRF, graph traversal, consolidation, temporal
- [x] **8.4** Memory footprint — 1.7 KB/record uncompressed, 282 B compressed (6.2x ratio), 100K+ rec/s ingestion
- [x] **8.5** Consolidation efficiency — 8.8x search speedup, 90% noise eviction, zero quality loss
- [x] **8.6** Cross-platform benchmarks — x86 measured, ARM estimated, cross_platform.sh script
- [x] **8.7** Published results in BENCHMARKS.md with ephemeral tier Redis comparison (70-140x faster)

---

## Implementation Order

**Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
**Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
**Phase 3:** ~~Track 6 (multi-modal)~~ 🟢 Complete; Track 7 (OpenClaw integration) withdrawn
**Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete

All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.

---

## What's Next

Verified against current repo state on 2026-08-05 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped):

- [ ] TypeScript bridge not wired into CI — `packages/clawhdf5-node/` already has a complete, working napi-rs package (package.json, tsconfig, hand-written TS wrapper matching all 21 `#[napi]` items, Jest test suite, README); it isn't published to npm and has no committed lockfile
- [ ] Publish crates to crates.io — no `publish` config anywhere in the workspace yet
- [ ] Python wheel distribution via maturin — `crates/clawhdf5-py/pyproject.toml` exists (maturin-buildable locally) but wheels aren't published anywhere
- [ ] `chunked_read.rs`/`data_read.rs` full bounds-check audit + scheduled fuzz campaigns (the new `fuzz_dataset_read` target covers the two files' main entry points; a full manual audit of every indexing site is still open) — see Tier 4 below
- [ ] WAL per-entry checksum landed as CRC32 (see below); a stronger per-entry format (explicit length prefix, avoiding the read-then-verify restructuring) could still be revisited if profiling shows it matters
- [ ] HNSW build parallelism is still narrow (only `prune_connections`); the correctness-sensitive outer insert loop needs its own dedicated design pass before parallelizing

### Recently closed out (2026-08-05, Tier 3–4 hardening pass)

- [x] Academic benchmark cross-validation — LongMemEval reproduced against MemX on tank (Ryzen 7 7800X3D): turn-level Hit@5 84.4% vs MemX's 51.6%; recall numbers are deterministic and reproduce exactly across machines. SIMD/Parallelism and Vector Search sections also re-run and dated. See [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05)
- [x] Android JNI (`clawhdf5-android`): validate `embedding_len`/`query_embedding_len` against the handle's configured `embedding_dim` before constructing a slice from a raw pointer
- [x] `clawhdf5-py`: bumped pyo3/numpy 0.28 → 0.29, clearing two RUSTSEC advisories
- [x] WAL (`clawhdf5-agent`): length-prefix caps (`MAX_WAL_FIELD_LEN`) to reject a corrupted length claim before allocating, then a full per-entry CRC32 trailer (`WAL_VERSION` 2) so a bit-flip stops replay cleanly instead of loading corrupted data; old-format WAL files still read correctly and are migrated on next open
- [x] `chunked_read.rs`/`data_read.rs`/`local_heap.rs` bounds-check audit: added `ensure_len` overflow guards, a recursion-depth guard against cyclic B-trees, and a fix for an unguarded compound-datatype byte-offset overrun. Added a new `fuzz_dataset_read` cargo-fuzz target exercising the contiguous/chunked/compact read paths — it found and we fixed 3 real crash bugs (integer-overflow panics) within the first few runs
- [x] `clawhdf5-ann`: optional `parallel` feature (rayon) for HNSW's `prune_connections` neighbor-distance computation
- [x] `[workspace.dependencies]` added for `tempfile`/`criterion`/`half`/`serde`, fixing a real version skew on `half` (2 vs 2.7)

### Recently closed out (2026-08-05 hardening pass)

- [x] CI/CD pipeline — `.gitea/workflows/ci.yml` now runs `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) on push/PR to `main`
- [x] Fixed no_std build breakage in `clawhdf5-format` (missing alloc imports, `AtomicU64` unsupported on thumbv7em, `f64::powi` requiring std/libm)
- [x] Fixed version skew: `clawhdf5-py` (pyproject.toml) and `packages/clawhdf5-node` (package.json) were both behind the actual crate version

### Recently closed out (2026-08-03 cleanup pass)

- [x] Removed `clawhdf5-types` — it was an empty 1-line stub crate; shared type definitions already live in `clawhdf5-format`, so CLAUDE.md and the workspace manifest were corrected instead of filling it in
- [x] Superblock v4 (page-buffer mode) read/write — the only unimplemented task from `docs/superpowers/plans/2026-06-29-format-write-extensions.md`; now done (`Superblock::parse_v4`/`serialize`, `FileWriter::with_page_size`)
- [x] Reconciled the three `docs/superpowers/plans/*.md` docs against actual shipped code — they were pre-work plans for `d6c4d4f` (2026-06-30), committed to git late; checkboxes now reflect reality

---

_Last updated: 2026-08-05_