Omar Sobh
429c29b76b
CI / test (push) Failing after 3s
README.md: - Fix badly stale LongMemEval numbers (badge said Hit@5 46%, table showed fabricated ~46%/~0.34/~72% figures that never matched BENCHMARKS.md's actual results of Hit@5 100% session / 84.4% turn-level, MRR 1.0/0.6597) - Remove clawhdf5-types from the Crate Map — that crate was removed in an earlier cleanup pass but the README diagram was never updated; fix the crate count (16, not 17) and stale line-of-code figures (72,087/84K -> ~92K) - Fix a dead #benchmarks badge anchor (no such heading exists) -> #performance - Document the new clawhdf5-ann `parallel` feature (had no Feature Flags entry) - Note WAL's CRC32 per-entry check, link the new tank LongMemEval/SIMD/ vector-search reproduction section, update stale test-count comment (417+ -> 1,650+) and Phase 2 roadmap blurb (LongMemEval is now done) ROADMAP.md: - Check off "Academic benchmark cross-validation" (done via the tank LongMemEval re-run) and add a new "Recently closed out" section summarizing the Tier 3-4 hardening pass (Android JNI validation, pyo3 bump, WAL CRC32, bounds-check audit + fuzz harness that found 3 real bugs, HNSW optional parallel feature, workspace.dependencies) - Update stale test count (1,546 -> 1,650+) and last-updated date CLAUDE.md: mention WAL's per-entry CRC32 check CHANGELOG.md: add Security/Performance/Architecture/Documentation entries under Unreleased summarizing all of Tiers 1-4 (this had not been touched since 2026-06-04, predating the entire hardening pass)
188 lines
12 KiB
Markdown
188 lines
12 KiB
Markdown
# ClawhDF5 Roadmap — Agent Memory Evolution
|
||
|
||
> Making clawhdf5 the defacto agentic memory solution.
|
||
> Single file. Pure Rust. Zero dependencies. Trusted everywhere.
|
||
|
||
---
|
||
|
||
## Track 1: Knowledge Graph in HDF5
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** Critical
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **1.1** Entity storage — entities with properties, embeddings, timestamps (created_at/updated_at)
|
||
- [x] **1.2** Relation storage — typed edges with RelationType enum (Temporal/Causal/Associative/Hierarchical/Custom), metadata, timestamps
|
||
- [x] **1.3** Entity extraction helpers — rule-based extraction (Person, Org, Location, Date, Technology, Project) with extract_and_store_entities() integration
|
||
- [x] **1.4** Entity resolution — fuzzy name matching (Levenshtein distance) via resolve_or_create()
|
||
- [x] **1.5** Graph traversal queries — BFS neighbors with depth, subgraph extraction from seeds
|
||
- [x] **1.6** Spreading activation — weighted activation propagation with configurable decay
|
||
- [x] **1.7** Graph-aware retrieval — get_entity_context() for formatted context injection
|
||
- [x] **1.8** Tests — comprehensive tests for all new features
|
||
|
||
**Research:** Graph-Native Cognitive Memory (2026), Graph-based Agent Memory survey (2026), SYNAPSE (2025)
|
||
|
||
---
|
||
|
||
## Track 2: Memory Consolidation Engine
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** Critical
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **2.1** Importance scoring — surprise (novelty), correction boost, length scoring with configurable weights
|
||
- [x] **2.2** Three-tier memory model — Working → Episodic → Semantic with bounded capacities
|
||
- [x] **2.3** Time-decay with reactivation — exponential decay with configurable half-life, access resets timestamp
|
||
- [x] **2.4** Bounded memory with graceful degradation — evict lowest-decay entries when over capacity
|
||
- [x] **2.5** Consolidation cycles — promote/evict across tiers based on importance and access thresholds
|
||
- [x] **2.6** Memory statistics — ConsolidationStats with per-tier counts, eviction/promotion tracking
|
||
- [x] **2.7** Tests — comprehensive tests for all features
|
||
|
||
**Research:** CraniMem (2026), D-MEM (2026), AI Hippocampus survey (2026)
|
||
|
||
---
|
||
|
||
## Track 3: Hybrid Retrieval Pipeline
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** High
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **3.1** Reciprocal Rank Fusion (RRF) — rrf_hybrid_search() with k=60 constant
|
||
- [x] **3.2** Multi-factor re-ranking — temporal decay, source authority hierarchy, activation scores (reranker.rs)
|
||
- [x] **3.3** Low-confidence rejection — min_score threshold, gap filtering, max_results (confidence.rs)
|
||
- [x] **3.4** Query expansion — synonyms, acronyms, temporal rewrites, morphological variants, knowledge graph aliases + expanded_search() with RRF merge
|
||
- [x] **3.5** Result explanation — ReRankResult with full score breakdown per factor
|
||
- [x] **3.6** Configurable pipeline — ReRankConfig + ConfidenceConfig with tunable weights/thresholds
|
||
- [x] **3.7** Tests + MemX-comparable benchmarks — 5 integration tests (Hit@1≥90%, search<500ms@100K, BM25<200ms@100K, hybrid<50ms@10K, compact<200ms@10K)
|
||
|
||
**Research:** MemX (2026), SwiftMem (2026)
|
||
|
||
---
|
||
|
||
## Track 4: Temporal Reasoning
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** High
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **4.1** Temporal index — sorted timestamp index with binary search, insert/remove
|
||
- [x] **4.2** Time-range queries — range_query, before, after, latest, earliest
|
||
- [x] **4.3** Session DAG — parent/child linking, chain walking, time-range overlap queries
|
||
- [x] **4.4** Temporal re-ranking — query hint enum (Latest/Earliest/Around/Between/None) with boost scoring
|
||
- [x] **4.5** Temporal entity tracking — EntityTimeline with state change history + point-in-time reconstruction
|
||
- [x] **4.6** Tests — comprehensive tests for all features
|
||
|
||
**Research:** MemX temporal gaps (≤43.6% Hit@5), MemoryArena multi-session tasks (2026)
|
||
|
||
---
|
||
|
||
## Track 5: Memory Security & Provenance
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** Medium-High
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **5.1** Source attribution — MemoryProvenance with source, creator, session, FNV-1a content hash
|
||
- [x] **5.2** Write anomaly detection — rate limiting, 15 injection patterns, source distribution analysis
|
||
- [x] **5.3** Source isolation — per-MemorySource sub-stores preventing cross-contamination
|
||
- [x] **5.4** Memory integrity verification — content hash comparison via verify_integrity()
|
||
- [x] **5.5** Poisoning resistance — pattern detection for prompt injection attempts
|
||
- [x] **5.6** Tests — comprehensive tests including adversarial patterns
|
||
|
||
**Research:** MemoryGraft (2025), SSGM Framework (2026)
|
||
|
||
---
|
||
|
||
## Track 6: Multi-Modal Memory
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** Medium
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **6.1** Image embedding storage — ModalEmbedding with model provenance (CLIP, SigLIP, etc.)
|
||
- [x] **6.2** Audio fingerprints — Audio modality with embedding storage
|
||
- [x] **6.3** Multi-modal search — search_by_modality (filtered) + search_cross_modal (all embeddings)
|
||
- [x] **6.4** Observation records — raw perception vs interpretation with confidence scoring
|
||
- [x] **6.5** Media reference storage — MediaRef with Path/Url/Inline, MIME types, FNV-1a checksums
|
||
- [x] **6.6** Tests — 35 comprehensive tests
|
||
|
||
**Research:** Neuro-Symbolic Memory (2026), RAGdb multi-modal RAG (2025)
|
||
|
||
---
|
||
|
||
## Track 7: OpenClaw Integration
|
||
**Status:** 🟢 Complete
|
||
**Priority:** Critical (for adoption)
|
||
**Crates:** `clawhdf5-agent`, `clawhdf5-napi`
|
||
|
||
- [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
|
||
- [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
|
||
- [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
|
||
- [x] **7.4** memory_search tool — backed by full hybrid retrieval pipeline
|
||
- [x] **7.5** memory_get tool — get() with path + line range support
|
||
- [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
|
||
- [x] **7.7** Config surface — `memory.backend = "clawhdf5"` schema documented in docs/openclaw-config.md
|
||
- [x] **7.8** Documentation + migration guide — docs/migration-guide.md, docs/openclaw-integration.md (architecture, full API reference, code patterns)
|
||
|
||
**Node.js bridge:** `clawhdf5-napi` (napi-rs) → `@redclaw/clawhdf5` npm package with full TypeScript types.
|
||
|
||
---
|
||
|
||
## Track 8: Benchmarking & Validation
|
||
**Status:** 🟢 Complete
|
||
**Priority:** High
|
||
**Crates:** `clawhdf5-agent`, `clawhdf5-bench`
|
||
|
||
- [x] **8.1** MemoryArena benchmark — 35 queries, 50 sessions, Hit@10=91.4%, MRR=0.547
|
||
- [x] **8.2** LongMemEval benchmark — 500 queries, session Hit@1=100%, turn Hit@5=84.4% (beats MemX 51.6%), MRR=0.660
|
||
- [x] **8.3** Latency benchmarks — vector search at 1K/10K/100K, hybrid/RRF, graph traversal, consolidation, temporal
|
||
- [x] **8.4** Memory footprint — 1.7 KB/record uncompressed, 282 B compressed (6.2x ratio), 100K+ rec/s ingestion
|
||
- [x] **8.5** Consolidation efficiency — 8.8x search speedup, 90% noise eviction, zero quality loss
|
||
- [x] **8.6** Cross-platform benchmarks — x86 measured, ARM estimated, cross_platform.sh script
|
||
- [x] **8.7** Published results in BENCHMARKS.md with ephemeral tier Redis comparison (70-140x faster)
|
||
|
||
---
|
||
|
||
## Implementation Order
|
||
|
||
**Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
|
||
**Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
|
||
**Phase 3:** ~~Track 6 (multi-modal) + Track 7 (OpenClaw integration)~~ 🟢 Complete
|
||
**Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
|
||
|
||
All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.
|
||
|
||
---
|
||
|
||
## What's Next
|
||
|
||
Verified against current repo state on 2026-08-05 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped):
|
||
|
||
- [ ] TypeScript bridge not wired into CI — `packages/clawhdf5-node/` already has a complete, working napi-rs package (package.json, tsconfig, hand-written TS wrapper matching all 21 `#[napi]` items, Jest test suite, README); it isn't published to npm and has no committed lockfile
|
||
- [ ] Publish crates to crates.io — no `publish` config anywhere in the workspace yet
|
||
- [ ] Python wheel distribution via maturin — `crates/clawhdf5-py/pyproject.toml` exists (maturin-buildable locally) but wheels aren't published anywhere
|
||
- [ ] `chunked_read.rs`/`data_read.rs` full bounds-check audit + scheduled fuzz campaigns (the new `fuzz_dataset_read` target covers the two files' main entry points; a full manual audit of every indexing site is still open) — see Tier 4 below
|
||
- [ ] WAL per-entry checksum landed as CRC32 (see below); a stronger per-entry format (explicit length prefix, avoiding the read-then-verify restructuring) could still be revisited if profiling shows it matters
|
||
- [ ] HNSW build parallelism is still narrow (only `prune_connections`); the correctness-sensitive outer insert loop needs its own dedicated design pass before parallelizing
|
||
|
||
### Recently closed out (2026-08-05, Tier 3–4 hardening pass)
|
||
|
||
- [x] Academic benchmark cross-validation — LongMemEval reproduced against MemX on tank (Ryzen 7 7800X3D): turn-level Hit@5 84.4% vs MemX's 51.6%; recall numbers are deterministic and reproduce exactly across machines. SIMD/Parallelism and Vector Search sections also re-run and dated. See [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05)
|
||
- [x] Android JNI (`clawhdf5-android`): validate `embedding_len`/`query_embedding_len` against the handle's configured `embedding_dim` before constructing a slice from a raw pointer
|
||
- [x] `clawhdf5-py`: bumped pyo3/numpy 0.28 → 0.29, clearing two RUSTSEC advisories
|
||
- [x] WAL (`clawhdf5-agent`): length-prefix caps (`MAX_WAL_FIELD_LEN`) to reject a corrupted length claim before allocating, then a full per-entry CRC32 trailer (`WAL_VERSION` 2) so a bit-flip stops replay cleanly instead of loading corrupted data; old-format WAL files still read correctly and are migrated on next open
|
||
- [x] `chunked_read.rs`/`data_read.rs`/`local_heap.rs` bounds-check audit: added `ensure_len` overflow guards, a recursion-depth guard against cyclic B-trees, and a fix for an unguarded compound-datatype byte-offset overrun. Added a new `fuzz_dataset_read` cargo-fuzz target exercising the contiguous/chunked/compact read paths — it found and we fixed 3 real crash bugs (integer-overflow panics) within the first few runs
|
||
- [x] `clawhdf5-ann`: optional `parallel` feature (rayon) for HNSW's `prune_connections` neighbor-distance computation
|
||
- [x] `[workspace.dependencies]` added for `tempfile`/`criterion`/`half`/`serde`, fixing a real version skew on `half` (2 vs 2.7)
|
||
|
||
### Recently closed out (2026-08-05 hardening pass)
|
||
|
||
- [x] CI/CD pipeline — `.gitea/workflows/ci.yml` now runs `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) on push/PR to `main`
|
||
- [x] Fixed no_std build breakage in `clawhdf5-format` (missing alloc imports, `AtomicU64` unsupported on thumbv7em, `f64::powi` requiring std/libm)
|
||
- [x] Fixed version skew: `clawhdf5-py` (pyproject.toml) and `packages/clawhdf5-node` (package.json) were both behind the actual crate version
|
||
|
||
### Recently closed out (2026-08-03 cleanup pass)
|
||
|
||
- [x] Removed `clawhdf5-types` — it was an empty 1-line stub crate; shared type definitions already live in `clawhdf5-format`, so CLAUDE.md and the workspace manifest were corrected instead of filling it in
|
||
- [x] Superblock v4 (page-buffer mode) read/write — the only unimplemented task from `docs/superpowers/plans/2026-06-29-format-write-extensions.md`; now done (`Superblock::parse_v4`/`serialize`, `FileWriter::with_page_size`)
|
||
- [x] Reconciled the three `docs/superpowers/plans/*.md` docs against actual shipped code — they were pre-work plans for `d6c4d4f` (2026-06-30), committed to git late; checkboxes now reflect reality
|
||
|
||
---
|
||
|
||
_Last updated: 2026-08-05_
|