osobhandClaude Opus 5.5
b39a705e77
Oracle and full longmemeval_s haystack, f32 and --float16, plus --sweep (both corpora) and --rerank-sweep, on tank at7a8fae0(MiniLM on the RTX 5060 Ti). Not idle: two orphaned h5py test processes kept the 1-minute load at 2.1-2.7 (up to 8.4 during runs), so no latency figure was updated. The headline (hybrid 0.4/0.6 turn Hit@5 81.4%) and the float16 claim reproduce. Oracle hybrid is 86.8%, not 85.2%: the old figure was at the 0.7/0.3 default of the time (today's 0.7/0.3 gives 85.2%). The ablation's 0.7/0.3 row, vector-only MRR and seven sweep rows move in the last digit or by one or two questions, most likely from the index tie-break of3ed0489. RRF turn MRR 0.5967 -> 0.5969 is not explained. Recency counts vary by one question between identical runs, so the float16 "flips" note is corrected. ROADMAP 8.2 no longer repeats the retracted session-level and MemX claims. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
194 lines
13 KiB
Markdown
194 lines
13 KiB
Markdown
# ClawhDF5 Roadmap — Agent Memory Evolution
|
||
|
||
> Making clawhdf5 the defacto agentic memory solution.
|
||
> Single file. Pure Rust. Zero dependencies. Trusted everywhere.
|
||
|
||
---
|
||
|
||
## Track 1: Knowledge Graph in HDF5
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** Critical
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **1.1** Entity storage — entities with properties, embeddings, timestamps (created_at/updated_at)
|
||
- [x] **1.2** Relation storage — typed edges with RelationType enum (Temporal/Causal/Associative/Hierarchical/Custom), metadata, timestamps
|
||
- [x] **1.3** Entity extraction helpers — rule-based extraction (Person, Org, Location, Date, Technology, Project) with extract_and_store_entities() integration
|
||
- [x] **1.4** Entity resolution — fuzzy name matching (Levenshtein distance) via resolve_or_create()
|
||
- [x] **1.5** Graph traversal queries — BFS neighbors with depth, subgraph extraction from seeds
|
||
- [x] **1.6** Spreading activation — weighted activation propagation with configurable decay
|
||
- [x] **1.7** Graph-aware retrieval — get_entity_context() for formatted context injection
|
||
- [x] **1.8** Tests — comprehensive tests for all new features
|
||
|
||
**Research:** Graph-Native Cognitive Memory (2026), Graph-based Agent Memory survey (2026), SYNAPSE (2025)
|
||
|
||
---
|
||
|
||
## Track 2: Memory Consolidation Engine
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** Critical
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **2.1** Importance scoring — surprise (novelty), correction boost, length scoring with configurable weights
|
||
- [x] **2.2** Three-tier memory model — Working → Episodic → Semantic with bounded capacities
|
||
- [x] **2.3** Time-decay with reactivation — exponential decay with configurable half-life, access resets timestamp
|
||
- [x] **2.4** Bounded memory with graceful degradation — evict lowest-decay entries when over capacity
|
||
- [x] **2.5** Consolidation cycles — promote/evict across tiers based on importance and access thresholds
|
||
- [x] **2.6** Memory statistics — ConsolidationStats with per-tier counts, eviction/promotion tracking
|
||
- [x] **2.7** Tests — comprehensive tests for all features
|
||
|
||
**Research:** CraniMem (2026), D-MEM (2026), AI Hippocampus survey (2026)
|
||
|
||
---
|
||
|
||
## Track 3: Hybrid Retrieval Pipeline
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** High
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **3.1** Reciprocal Rank Fusion (RRF) — rrf_hybrid_search() with k=60 constant
|
||
- [x] **3.2** Multi-factor re-ranking — temporal decay, source authority hierarchy, activation scores (reranker.rs)
|
||
- [x] **3.3** Low-confidence rejection — min_score threshold, gap filtering, max_results (confidence.rs)
|
||
- [x] **3.4** Query expansion — synonyms, acronyms, temporal rewrites, morphological variants, knowledge graph aliases + expanded_search() with RRF merge
|
||
- [x] **3.5** Result explanation — ReRankResult with full score breakdown per factor
|
||
- [x] **3.6** Configurable pipeline — ReRankConfig + ConfidenceConfig with tunable weights/thresholds
|
||
- [x] **3.7** Tests + MemX-comparable benchmarks — 5 integration tests (Hit@1≥90%, search<500ms@100K, BM25<200ms@100K, hybrid<50ms@10K, compact<200ms@10K)
|
||
|
||
**Research:** MemX (2026), SwiftMem (2026)
|
||
|
||
---
|
||
|
||
## Track 4: Temporal Reasoning
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** High
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **4.1** Temporal index — sorted timestamp index with binary search, insert/remove
|
||
- [x] **4.2** Time-range queries — range_query, before, after, latest, earliest
|
||
- [x] **4.3** Session DAG — parent/child linking, chain walking, time-range overlap queries
|
||
- [x] **4.4** Temporal re-ranking — query hint enum (Latest/Earliest/Around/Between/None) with boost scoring
|
||
- [x] **4.5** Temporal entity tracking — EntityTimeline with state change history + point-in-time reconstruction
|
||
- [x] **4.6** Tests — comprehensive tests for all features
|
||
|
||
**Research:** MemX temporal gaps (≤43.6% Hit@5), MemoryArena multi-session tasks (2026)
|
||
|
||
---
|
||
|
||
## Track 5: Memory Security & Provenance
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** Medium-High
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **5.1** Source attribution — MemoryProvenance with source, creator, session, FNV-1a content hash
|
||
- [x] **5.2** Write anomaly detection — rate limiting, 15 injection patterns, source distribution analysis
|
||
- [x] **5.3** Source isolation — per-MemorySource sub-stores preventing cross-contamination
|
||
- [x] **5.4** Memory integrity verification — content hash comparison via verify_integrity()
|
||
- [x] **5.5** Poisoning resistance — pattern detection for prompt injection attempts
|
||
- [x] **5.6** Tests — comprehensive tests including adversarial patterns
|
||
|
||
**Research:** MemoryGraft (2025), SSGM Framework (2026)
|
||
|
||
---
|
||
|
||
## Track 6: Multi-Modal Memory
|
||
**Status:** 🟢 Phase 1 Complete
|
||
**Priority:** Medium
|
||
**Crate:** `clawhdf5-agent`
|
||
|
||
- [x] **6.1** Image embedding storage — ModalEmbedding with model provenance (CLIP, SigLIP, etc.)
|
||
- [x] **6.2** Audio fingerprints — Audio modality with embedding storage
|
||
- [x] **6.3** Multi-modal search — search_by_modality (filtered) + search_cross_modal (all embeddings)
|
||
- [x] **6.4** Observation records — raw perception vs interpretation with confidence scoring
|
||
- [x] **6.5** Media reference storage — MediaRef with Path/Url/Inline, MIME types, FNV-1a checksums
|
||
- [x] **6.6** Tests — 35 comprehensive tests
|
||
|
||
**Research:** Neuro-Symbolic Memory (2026), RAGdb multi-modal RAG (2025)
|
||
|
||
---
|
||
|
||
## Track 7: OpenClaw Integration — withdrawn (2026-09-25)
|
||
**Status:** ⚪ Withdrawn (the items below were library work; no OpenClaw integration shipped)
|
||
**Priority:** Critical (for adoption)
|
||
**Crates:** `clawhdf5-agent`, `clawhdf5-napi`
|
||
|
||
- [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
|
||
- [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
|
||
- [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
|
||
- [x] **7.4** `search()` — backed by the full hybrid retrieval pipeline (a Rust method; no OpenClaw tool was ever registered)
|
||
- [x] **7.5** `get()` — read back by path, with a line slice (not an OpenClaw tool either)
|
||
- [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
|
||
- [ ] **7.7** ~~Config surface — `memory.backend = "clawhdf5"`~~ — never valid OpenClaw config; docs removed
|
||
- [ ] **7.8** ~~Documentation + migration guide~~ — removed: they described an integration that never worked
|
||
|
||
**Node.js bridge:** `clawhdf5-napi` (napi-rs) and a TypeScript wrapper in `packages/clawhdf5-node` exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).
|
||
|
||
---
|
||
|
||
> **Withdrawn.** None of this track produced a working OpenClaw integration: no
|
||
> plugin was built, the documented `memory.backend = "clawhdf5"` config was never
|
||
> valid in any OpenClaw release, and the Node package was never published. The
|
||
> Rust `ClawhdfBackend` remains as a library API. Not pursued for now; see
|
||
> [docs/openclaw.md](docs/openclaw.md) for what a plugin would need today.
|
||
|
||
## Track 8: Benchmarking & Validation
|
||
**Status:** 🟢 Complete
|
||
**Priority:** High
|
||
**Crates:** `clawhdf5-agent`, `clawhdf5-bench`
|
||
|
||
- [x] **8.1** MemoryArena benchmark — 35 queries, 50 sessions, Hit@10=91.4%, MRR=0.547
|
||
- [x] **8.2** LongMemEval benchmark — 500 questions, retrieval recall (not QA accuracy). Full `longmemeval_s` haystack, hybrid 0.4/0.6 with MiniLM embeddings: turn Hit@5 81.4%, MRR 0.643; session Hit@5 96.8% (re-run 2026-09-27 on tank). Oracle variant: BM25-only turn Hit@5 84.4%, MRR 0.660; hybrid 86.8%. The session Hit@1 of 100% first recorded here was degenerate on the oracle variant, and the "beats MemX 51.6%" claim compared a different granularity. Both are retracted; see [BENCHMARKS.md § LongMemEval Results](BENCHMARKS.md#longmemeval-results)
|
||
- [x] **8.3** Latency benchmarks — vector search at 1K/10K/100K, hybrid/RRF, graph traversal, consolidation, temporal
|
||
- [x] **8.4** Memory footprint — 1.7 KB/record uncompressed, 282 B compressed (6.2x ratio), 100K+ rec/s ingestion
|
||
- [x] **8.5** Consolidation efficiency — 8.8x search speedup, 90% noise eviction, zero quality loss
|
||
- [x] **8.6** Cross-platform benchmarks — x86 measured, ARM estimated, cross_platform.sh script
|
||
- [x] **8.7** Published results in BENCHMARKS.md with ephemeral tier Redis comparison (70-140x faster)
|
||
|
||
---
|
||
|
||
## Implementation Order
|
||
|
||
**Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
|
||
**Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
|
||
**Phase 3:** ~~Track 6 (multi-modal)~~ 🟢 Complete; Track 7 (OpenClaw integration) withdrawn
|
||
**Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
|
||
|
||
All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.
|
||
|
||
---
|
||
|
||
## What's Next
|
||
|
||
Verified against current repo state on 2026-08-05 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped):
|
||
|
||
- [ ] TypeScript bridge not wired into CI — `packages/clawhdf5-node/` already has a complete, working napi-rs package (package.json, tsconfig, hand-written TS wrapper matching all 21 `#[napi]` items, Jest test suite, README); it isn't published to npm and has no committed lockfile
|
||
- [ ] Publish crates to crates.io — no `publish` config anywhere in the workspace yet
|
||
- [ ] Python wheel distribution via maturin — `crates/clawhdf5-py/pyproject.toml` exists (maturin-buildable locally) but wheels aren't published anywhere
|
||
- [ ] `chunked_read.rs`/`data_read.rs` full bounds-check audit + scheduled fuzz campaigns (the new `fuzz_dataset_read` target covers the two files' main entry points; a full manual audit of every indexing site is still open) — see Tier 4 below
|
||
- [ ] WAL per-entry checksum landed as CRC32 (see below); a stronger per-entry format (explicit length prefix, avoiding the read-then-verify restructuring) could still be revisited if profiling shows it matters
|
||
- [ ] HNSW build parallelism is still narrow (only `prune_connections`); the correctness-sensitive outer insert loop needs its own dedicated design pass before parallelizing
|
||
|
||
### Recently closed out (2026-08-05, Tier 3–4 hardening pass)
|
||
|
||
- [x] Academic benchmark cross-validation — LongMemEval reproduced on tank (Ryzen 7 7800X3D): turn-level Hit@5 84.4% on the oracle variant (the comparison with MemX's 51.6% made here was later retracted, since MemX measures fact-level granularity over a far larger corpus); recall numbers are deterministic and reproduce exactly across machines. SIMD/Parallelism and Vector Search sections also re-run and dated. See [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05)
|
||
- [x] Android JNI (`clawhdf5-android`): validate `embedding_len`/`query_embedding_len` against the handle's configured `embedding_dim` before constructing a slice from a raw pointer
|
||
- [x] `clawhdf5-py`: bumped pyo3/numpy 0.28 → 0.29, clearing two RUSTSEC advisories
|
||
- [x] WAL (`clawhdf5-agent`): length-prefix caps (`MAX_WAL_FIELD_LEN`) to reject a corrupted length claim before allocating, then a full per-entry CRC32 trailer (`WAL_VERSION` 2) so a bit-flip stops replay cleanly instead of loading corrupted data; old-format WAL files still read correctly and are migrated on next open
|
||
- [x] `chunked_read.rs`/`data_read.rs`/`local_heap.rs` bounds-check audit: added `ensure_len` overflow guards, a recursion-depth guard against cyclic B-trees, and a fix for an unguarded compound-datatype byte-offset overrun. Added a new `fuzz_dataset_read` cargo-fuzz target exercising the contiguous/chunked/compact read paths — it found and we fixed 3 real crash bugs (integer-overflow panics) within the first few runs
|
||
- [x] `clawhdf5-ann`: optional `parallel` feature (rayon) for HNSW's `prune_connections` neighbor-distance computation
|
||
- [x] `[workspace.dependencies]` added for `tempfile`/`criterion`/`half`/`serde`, fixing a real version skew on `half` (2 vs 2.7)
|
||
|
||
### Recently closed out (2026-08-05 hardening pass)
|
||
|
||
- [x] CI/CD pipeline — `.gitea/workflows/ci.yml` now runs `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) on push/PR to `main`
|
||
- [x] Fixed no_std build breakage in `clawhdf5-format` (missing alloc imports, `AtomicU64` unsupported on thumbv7em, `f64::powi` requiring std/libm)
|
||
- [x] Fixed version skew: `clawhdf5-py` (pyproject.toml) and `packages/clawhdf5-node` (package.json) were both behind the actual crate version
|
||
|
||
### Recently closed out (2026-08-03 cleanup pass)
|
||
|
||
- [x] Removed `clawhdf5-types` — it was an empty 1-line stub crate; shared type definitions already live in `clawhdf5-format`, so CLAUDE.md and the workspace manifest were corrected instead of filling it in
|
||
- [x] Superblock v4 (page-buffer mode) read/write — the only unimplemented task from `docs/superpowers/plans/2026-06-29-format-write-extensions.md`; now done (`Superblock::parse_v4`/`serialize`, `FileWriter::with_page_size`)
|
||
- [x] Reconciled the three `docs/superpowers/plans/*.md` docs against actual shipped code — they were pre-work plans for `d6c4d4f` (2026-06-30), committed to git late; checkboxes now reflect reality
|
||
|
||
---
|
||
|
||
_Last updated: 2026-08-05_
|