clawhdf5/ROADMAP.md

ClawhDF5 Roadmap — Agent Memory Evolution

Making clawhdf5 the defacto agentic memory solution. Single file. Pure Rust. Zero dependencies. Trusted everywhere.

---

Track 1: Knowledge Graph in HDF5

Status: 🟢 Phase 1 Complete Priority: Critical Crate: clawhdf5-agent

• 1.1 Entity storage — entities with properties, embeddings, timestamps (created_at/updated_at)
• 1.2 Relation storage — typed edges with RelationType enum (Temporal/Causal/Associative/Hierarchical/Custom), metadata, timestamps
• 1.3 Entity extraction helpers — rule-based extraction (Person, Org, Location, Date, Technology, Project) with extract_and_store_entities() integration
• 1.4 Entity resolution — fuzzy name matching (Levenshtein distance) via resolve_or_create()
• 1.5 Graph traversal queries — BFS neighbors with depth, subgraph extraction from seeds
• 1.6 Spreading activation — weighted activation propagation with configurable decay
• 1.7 Graph-aware retrieval — get_entity_context() for formatted context injection
• 1.8 Tests — comprehensive tests for all new features

Research: Graph-Native Cognitive Memory (2026), Graph-based Agent Memory survey (2026), SYNAPSE (2025)

---

Track 2: Memory Consolidation Engine

Status: 🟢 Phase 1 Complete Priority: Critical Crate: clawhdf5-agent

• 2.1 Importance scoring — surprise (novelty), correction boost, length scoring with configurable weights
• 2.2 Three-tier memory model — Working → Episodic → Semantic with bounded capacities
• 2.3 Time-decay with reactivation — exponential decay with configurable half-life, access resets timestamp
• 2.4 Bounded memory with graceful degradation — evict lowest-decay entries when over capacity
• 2.5 Consolidation cycles — promote/evict across tiers based on importance and access thresholds
• 2.6 Memory statistics — ConsolidationStats with per-tier counts, eviction/promotion tracking
• 2.7 Tests — comprehensive tests for all features

Research: CraniMem (2026), D-MEM (2026), AI Hippocampus survey (2026)

---

Track 3: Hybrid Retrieval Pipeline

Status: 🟢 Phase 1 Complete Priority: High Crate: clawhdf5-agent

• 3.1 Reciprocal Rank Fusion (RRF) — rrf_hybrid_search() with k=60 constant
• 3.2 Multi-factor re-ranking — temporal decay, source authority hierarchy, activation scores (reranker.rs)
• 3.3 Low-confidence rejection — min_score threshold, gap filtering, max_results (confidence.rs)
• 3.4 Query expansion — synonyms, acronyms, temporal rewrites, morphological variants, knowledge graph aliases + expanded_search() with RRF merge
• 3.5 Result explanation — ReRankResult with full score breakdown per factor
• 3.6 Configurable pipeline — ReRankConfig + ConfidenceConfig with tunable weights/thresholds
• 3.7 Tests + MemX-comparable benchmarks — 5 integration tests (Hit@1≥90%, search<500ms@100K, BM25<200ms@100K, hybrid<50ms@10K, compact<200ms@10K)

Research: MemX (2026), SwiftMem (2026)

---

Track 4: Temporal Reasoning

Status: 🟢 Phase 1 Complete Priority: High Crate: clawhdf5-agent

• 4.1 Temporal index — sorted timestamp index with binary search, insert/remove
• 4.2 Time-range queries — range_query, before, after, latest, earliest
• 4.3 Session DAG — parent/child linking, chain walking, time-range overlap queries
• 4.4 Temporal re-ranking — query hint enum (Latest/Earliest/Around/Between/None) with boost scoring
• 4.5 Temporal entity tracking — EntityTimeline with state change history + point-in-time reconstruction
• 4.6 Tests — comprehensive tests for all features

Research: MemX temporal gaps (≤43.6% Hit@5), MemoryArena multi-session tasks (2026)

---

Track 5: Memory Security & Provenance

Status: 🟢 Phase 1 Complete Priority: Medium-High Crate: clawhdf5-agent

• 5.1 Source attribution — MemoryProvenance with source, creator, session, FNV-1a content hash
• 5.2 Write anomaly detection — rate limiting, 15 injection patterns, source distribution analysis
• 5.3 Source isolation — per-MemorySource sub-stores preventing cross-contamination
• 5.4 Memory integrity verification — content hash comparison via verify_integrity()
• 5.5 Poisoning resistance — pattern detection for prompt injection attempts
• 5.6 Tests — comprehensive tests including adversarial patterns

Research: MemoryGraft (2025), SSGM Framework (2026)

---

Track 6: Multi-Modal Memory

Status: 🟢 Phase 1 Complete Priority: Medium Crate: clawhdf5-agent

• 6.1 Image embedding storage — ModalEmbedding with model provenance (CLIP, SigLIP, etc.)
• 6.2 Audio fingerprints — Audio modality with embedding storage
• 6.3 Multi-modal search — search_by_modality (filtered) + search_cross_modal (all embeddings)
• 6.4 Observation records — raw perception vs interpretation with confidence scoring
• 6.5 Media reference storage — MediaRef with Path/Url/Inline, MIME types, FNV-1a checksums
• 6.6 Tests — 35 comprehensive tests

Research: Neuro-Symbolic Memory (2026), RAGdb multi-modal RAG (2025)

---

Track 7: OpenClaw Integration — withdrawn (2026-09-25)

Status: ⚪ Withdrawn (the items below were library work; no OpenClaw integration shipped) Priority: Critical (for adoption) Crates: clawhdf5-agent, clawhdf5-napi

• 7.1 Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
• 7.2 Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
• 7.3 Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
• 7.4 search() — backed by the full hybrid retrieval pipeline (a Rust method; no OpenClaw tool was ever registered)
• 7.5 get() — read back by path, with a line slice (not an OpenClaw tool either)
• 7.6 Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
• 7.7 Config surface — memory.backend = "clawhdf5" — never valid OpenClaw config; docs removed
• 7.8 Documentation + migration guide — removed: they described an integration that never worked

Node.js bridge: clawhdf5-napi (napi-rs) and a TypeScript wrapper in packages/clawhdf5-node exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).

---

Withdrawn. None of this track produced a working OpenClaw integration: no plugin was built, the documented memory.backend = "clawhdf5" config was never valid in any OpenClaw release, and the Node package was never published. The Rust ClawhdfBackend remains as a library API. Not pursued for now; see docs/openclaw.md for what a plugin would need today.

Track 8: Benchmarking & Validation

Status: 🟢 Complete Priority: High Crates: clawhdf5-agent, clawhdf5-bench

• 8.1 MemoryArena benchmark — 35 queries, 50 sessions, Hit@10=91.4%, MRR=0.547
• 8.2 LongMemEval benchmark — 500 queries, session Hit@1=100%, turn Hit@5=84.4% (beats MemX 51.6%), MRR=0.660
• 8.3 Latency benchmarks — vector search at 1K/10K/100K, hybrid/RRF, graph traversal, consolidation, temporal
• 8.4 Memory footprint — 1.7 KB/record uncompressed, 282 B compressed (6.2x ratio), 100K+ rec/s ingestion
• 8.5 Consolidation efficiency — 8.8x search speedup, 90% noise eviction, zero quality loss
• 8.6 Cross-platform benchmarks — x86 measured, ARM estimated, cross_platform.sh script
• 8.7 Published results in BENCHMARKS.md with ephemeral tier Redis comparison (70-140x faster)

---

Implementation Order

Phase 1: Tracks 1, 2, 3 — core memory intelligence 🟢 Complete Phase 2: Track 4 (temporal) + Track 5 (security) 🟢 Complete Phase 3: Track 6 (multi-modal) 🟢 Complete; Track 7 (OpenClaw integration) withdrawn Phase 4: Track 8 (benchmarking + validation) 🟢 Complete

All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.

---

What's Next

Verified against current repo state on 2026-08-05 (see also docs/superpowers/plans/ for the filter-codec/format-write/MPI-IO work, now shipped):

• TypeScript bridge not wired into CI — packages/clawhdf5-node/ already has a complete, working napi-rs package (package.json, tsconfig, hand-written TS wrapper matching all 21 #[napi] items, Jest test suite, README); it isn't published to npm and has no committed lockfile
• Publish crates to crates.io — no publish config anywhere in the workspace yet
• Python wheel distribution via maturin — crates/clawhdf5-py/pyproject.toml exists (maturin-buildable locally) but wheels aren't published anywhere
• chunked_read.rs/data_read.rs full bounds-check audit + scheduled fuzz campaigns (the new fuzz_dataset_read target covers the two files' main entry points; a full manual audit of every indexing site is still open) — see Tier 4 below
• WAL per-entry checksum landed as CRC32 (see below); a stronger per-entry format (explicit length prefix, avoiding the read-then-verify restructuring) could still be revisited if profiling shows it matters
• HNSW build parallelism is still narrow (only prune_connections); the correctness-sensitive outer insert loop needs its own dedicated design pass before parallelizing

Recently closed out (2026-08-05, Tier 3–4 hardening pass)

• Academic benchmark cross-validation — LongMemEval reproduced against MemX on tank (Ryzen 7 7800X3D): turn-level Hit@5 84.4% vs MemX's 51.6%; recall numbers are deterministic and reproduce exactly across machines. SIMD/Parallelism and Vector Search sections also re-run and dated. See BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector Search
• Android JNI (clawhdf5-android): validate embedding_len/query_embedding_len against the handle's configured embedding_dim before constructing a slice from a raw pointer
• clawhdf5-py: bumped pyo3/numpy 0.28 → 0.29, clearing two RUSTSEC advisories
• WAL (clawhdf5-agent): length-prefix caps (MAX_WAL_FIELD_LEN) to reject a corrupted length claim before allocating, then a full per-entry CRC32 trailer (WAL_VERSION 2) so a bit-flip stops replay cleanly instead of loading corrupted data; old-format WAL files still read correctly and are migrated on next open
• chunked_read.rs/data_read.rs/local_heap.rs bounds-check audit: added ensure_len overflow guards, a recursion-depth guard against cyclic B-trees, and a fix for an unguarded compound-datatype byte-offset overrun. Added a new fuzz_dataset_read cargo-fuzz target exercising the contiguous/chunked/compact read paths — it found and we fixed 3 real crash bugs (integer-overflow panics) within the first few runs
• clawhdf5-ann: optional parallel feature (rayon) for HNSW's prune_connections neighbor-distance computation
• [workspace.dependencies] added for tempfile/criterion/half/serde, fixing a real version skew on half (2 vs 2.7)

Recently closed out (2026-08-05 hardening pass)

• CI/CD pipeline — .gitea/workflows/ci.yml now runs scripts/ci-test.sh (fmt, clippy, tests, no_std check) on push/PR to main
• Fixed no_std build breakage in clawhdf5-format (missing alloc imports, AtomicU64 unsupported on thumbv7em, f64::powi requiring std/libm)
• Fixed version skew: clawhdf5-py (pyproject.toml) and packages/clawhdf5-node (package.json) were both behind the actual crate version

Recently closed out (2026-08-03 cleanup pass)

• Removed clawhdf5-types — it was an empty 1-line stub crate; shared type definitions already live in clawhdf5-format, so CLAUDE.md and the workspace manifest were corrected instead of filling it in
• Superblock v4 (page-buffer mode) read/write — the only unimplemented task from docs/superpowers/plans/2026-06-29-format-write-extensions.md; now done (Superblock::parse_v4/serialize, FileWriter::with_page_size)
• Reconciled the three docs/superpowers/plans/*.md docs against actual shipped code — they were pre-work plans for d6c4d4f (2026-06-30), committed to git late; checkboxes now reflect reality

---

Last updated: 2026-08-05