docs: ROADMAP rewritten from CHANGELOG, git log and known-issues
The old file was a tracker for the mid-2026 agent-memory tracks, last updated 2026-08-05, with an OpenClaw track and "what's next" items that have since shipped (CI, fuzz target, WAL checksums, HNSW parallel build). Now: releases v2.0.0-v2.7.0 and every PR merged since (#3-#21, merge dates from git log), range-read milestones M0-M5, a one-paragraph summary of the agent-memory work, and what is genuinely next: crates.io and PyPI publishing (plus the broken Node package), SWMR writing, MPI collective I/O, paged-metadata single-request reads, Blosc2/ZFP encoders, and the open items of docs/known-issues.md. OpenClaw and ZeroClaw are listed only as withdrawn. No dates are given for future work. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
+146
-177
@@ -1,193 +1,162 @@
|
||||
# ClawhDF5 Roadmap — Agent Memory Evolution
|
||||
# clawhdf5 roadmap
|
||||
|
||||
> Making clawhdf5 the defacto agentic memory solution.
|
||||
> Single file. Pure Rust. Zero dependencies. Trusted everywhere.
|
||||
What has shipped, and what is genuinely next. Everything here is checked
|
||||
against `CHANGELOG.md`, `git log` and [`docs/known-issues.md`](docs/known-issues.md);
|
||||
dates are merge dates on `main`. Nothing after v2.7.0 has been released:
|
||||
the work since then is on `main` under `CHANGELOG.md` "Unreleased".
|
||||
|
||||
_Last updated: 2026-09-28 (at `9b5803f`, PR #21)._
|
||||
|
||||
---
|
||||
|
||||
## Track 1: Knowledge Graph in HDF5
|
||||
**Status:** 🟢 Phase 1 Complete
|
||||
**Priority:** Critical
|
||||
**Crate:** `clawhdf5-agent`
|
||||
## Done
|
||||
|
||||
- [x] **1.1** Entity storage — entities with properties, embeddings, timestamps (created_at/updated_at)
|
||||
- [x] **1.2** Relation storage — typed edges with RelationType enum (Temporal/Causal/Associative/Hierarchical/Custom), metadata, timestamps
|
||||
- [x] **1.3** Entity extraction helpers — rule-based extraction (Person, Org, Location, Date, Technology, Project) with extract_and_store_entities() integration
|
||||
- [x] **1.4** Entity resolution — fuzzy name matching (Levenshtein distance) via resolve_or_create()
|
||||
- [x] **1.5** Graph traversal queries — BFS neighbors with depth, subgraph extraction from seeds
|
||||
- [x] **1.6** Spreading activation — weighted activation propagation with configurable decay
|
||||
- [x] **1.7** Graph-aware retrieval — get_entity_context() for formatted context injection
|
||||
- [x] **1.8** Tests — comprehensive tests for all new features
|
||||
### Releases
|
||||
|
||||
**Research:** Graph-Native Cognitive Memory (2026), Graph-based Agent Memory survey (2026), SYNAPSE (2025)
|
||||
| Version | Date | Headline |
|
||||
|---|---|---|
|
||||
| v2.0.0 | 2026-03-19 | rustyhdf5 (11 crates) and edgehdf5 (4 crates) unified into one workspace as `clawhdf5-*` |
|
||||
| v2.1.0 | 2026-06-03 | HNSW backs the agent's vector search by default; live, mutable HNSW index |
|
||||
| v2.2.0 – v2.7.0 | 2026-09-18 – 2026-09-20 | bounded decompression and read-path bounds checks, single-writer store locking, WAL v4, HNSW recall fix (0.31 -> 0.98 recall@10 at 100K), fusion weights tuned on LongMemEval, int8 index, Extensible Array read fix and chunk-index checksums |
|
||||
|
||||
Details per release: [`CHANGELOG.md`](CHANGELOG.md).
|
||||
|
||||
### Since v2.7.0 (unreleased, on `main`)
|
||||
|
||||
| PR | Merged | What |
|
||||
|---|---|---|
|
||||
| #3 | 2026-09-23 | pure-Rust deflate (zlib-rs) by default, no C in the core crates' default build (checked in CI), MSRV 1.92 |
|
||||
| #4 | 2026-09-25 | files open in h5py again (every `f32` and every empty dataset clawhdf5 wrote was unreadable by libhdf5); float16 embedding storage |
|
||||
| #5 | 2026-09-25 | `HDF5Memory::search` with `SearchOptions` (source filters, re-ranking, confidence); float16 on by default |
|
||||
| #6 | 2026-09-25 | `clawhdf5-migrate` writes real agent stores; knowledge-graph fix; dated benchmark re-run |
|
||||
| #7 | 2026-09-25 | consolidation benchmark completed (cheaper novelty scoring) |
|
||||
| #8 | 2026-09-25 | Ed25519-signed checkpoints (`HDF5Memory::verify`) |
|
||||
| #9, #10 | 2026-09-25 | OpenClaw and ZeroClaw integration claims withdrawn — neither ever integrated clawhdf5 |
|
||||
| #11 | 2026-09-26 | silent wrong data and libhdf5 interop bugs found by the HDF5 audit fixed |
|
||||
| #12 | 2026-09-26 | reproducible conformance sweep over eight public corpora, nightly CI job ([`CONFORMANCE.md`](CONFORMANCE.md)) |
|
||||
| #13 | 2026-09-26 | reads HDF5 1.6-era layouts, user blocks, virtual datasets, dense attributes, very large groups |
|
||||
| #14 | 2026-09-26 | `h5rs` tools (`ls`, `dump`, `stat`, `diff`, `check`), the browser reader (`clawhdf5-wasm`), libhdf5's header checks, plugin filters (LZF, bitshuffle, bzip2, Blosc), concurrency benchmark |
|
||||
| #15 | 2026-09-26 | fast contiguous and concurrent reads, variable-length data, nested groups and links in the writer, Python bindings |
|
||||
| #16 | 2026-09-26 | chunked full reads faster than an h5py process pool, writer B-trees of any size, Blosc2 (read), 599/697 conformance |
|
||||
| #17 | 2026-09-26 | range reads M0/M1 (indexed name lookups, the `Storage` trait), ZFP (read), in-place editing (`FileEditor`) |
|
||||
| #18 | 2026-09-27 | range reads M2/M3 (`File::open_storage`; `clawhdf5-remote`: HTTP(S), S3, GCS, Azure), in-place editing of every chunk index, shrinking, dense attributes |
|
||||
| #19 | 2026-09-27 | remote files in the browser (`openUrl`, M4), SWMR reader (`File::open_swmr`, M5), Python remote reads and `'r+'` editing |
|
||||
| #20 | 2026-09-28 | benchmarks re-measured: LongMemEval with real MiniLM embeddings, local reads on an idle machine |
|
||||
| #21 | 2026-09-28 | remote files open in a few requests (group lookups down the B-tree, `Storage::hint`), `ObjectHeader::parse` back to its earlier speed, the last conformance mismatches resolved: 602/697 ok, 0 our-error, 0 mismatch |
|
||||
|
||||
### Range reads (design: [`docs/design/range-reads.md`](docs/design/range-reads.md))
|
||||
|
||||
- [x] M0 — indexed name lookups (#17)
|
||||
- [x] M1 — metadata parsed through the `Storage` trait (#17)
|
||||
- [x] M2 — raw data through `Storage`, `File::open_storage` (#18)
|
||||
- [x] M3 — `clawhdf5-remote`: HTTP(S) range requests and object stores through a block cache; `h5rs` URLs (#18); Python URLs (#19)
|
||||
- [x] M4 — `openUrl` in the browser, restartable "NeedBytes" cache (#19; fewer round trips in #21)
|
||||
- [x] M5 — reading files a SWMR writer is appending to ([`docs/design/swmr.md`](docs/design/swmr.md), #19)
|
||||
|
||||
### Agent memory (`clawhdf5-agent`)
|
||||
|
||||
Shipped before and during the v2 releases, and kept current since:
|
||||
knowledge graph with entity extraction and resolution; three-tier
|
||||
consolidation with decay; hybrid retrieval (HNSW + BM25, weighted or RRF
|
||||
fusion, re-ranking, confidence rejection, query expansion); temporal index
|
||||
and session DAG; per-save provenance ledger and write-anomaly detection;
|
||||
multi-modal embeddings; WAL with chained CRC32; single-writer locking;
|
||||
signed checkpoints. Retrieval is measured, not claimed: see
|
||||
[`BENCHMARKS.md`](BENCHMARKS.md) ("LongMemEval Results" reports retrieval
|
||||
recall, not QA accuracy; earlier headline numbers that compared different
|
||||
granularities were retracted there).
|
||||
|
||||
---
|
||||
|
||||
## Track 2: Memory Consolidation Engine
|
||||
**Status:** 🟢 Phase 1 Complete
|
||||
**Priority:** Critical
|
||||
**Crate:** `clawhdf5-agent`
|
||||
## Next
|
||||
|
||||
- [x] **2.1** Importance scoring — surprise (novelty), correction boost, length scoring with configurable weights
|
||||
- [x] **2.2** Three-tier memory model — Working → Episodic → Semantic with bounded capacities
|
||||
- [x] **2.3** Time-decay with reactivation — exponential decay with configurable half-life, access resets timestamp
|
||||
- [x] **2.4** Bounded memory with graceful degradation — evict lowest-decay entries when over capacity
|
||||
- [x] **2.5** Consolidation cycles — promote/evict across tiers based on importance and access thresholds
|
||||
- [x] **2.6** Memory statistics — ConsolidationStats with per-tier counts, eviction/promotion tracking
|
||||
- [x] **2.7** Tests — comprehensive tests for all features
|
||||
Not scheduled; listed roughly by how much they unblock. None has a date.
|
||||
|
||||
**Research:** CraniMem (2026), D-MEM (2026), AI Hippocampus survey (2026)
|
||||
### Distribution
|
||||
|
||||
- [ ] **Publish the crates to crates.io.** Nothing is published; the READMEs
|
||||
say to depend on git. Before publishing: no `publish` settings exist
|
||||
(only `clawhdf5-wasm` has `publish = false`), and several `Cargo.toml`
|
||||
descriptions still name the old `rustyhdf5`/`edgehdf5` (`clawhdf5-accel`,
|
||||
`-derive`, `-gpu`, `-io`, `-netcdf4`, `-android`).
|
||||
- [ ] **Publish Python wheels to PyPI.** `crates/clawhdf5-py` builds with
|
||||
maturin and is tested in CI, but no wheel is published. The default wheel
|
||||
reads plain `http://` only; `https`/`s3`/`gcs`/`azure` wheels compile C
|
||||
(ring, aws-lc-rs).
|
||||
- [ ] **The Node.js package** (`packages/clawhdf5-node` over
|
||||
`clawhdf5-napi`) has never worked and is not in CI: fix it and add CI, or
|
||||
remove it ([known issue](docs/known-issues.md)).
|
||||
|
||||
### HDF5 features
|
||||
|
||||
- [ ] **SWMR writing.** The reader is done (M5); writing a file while
|
||||
libhdf5 readers follow it is not. Also not covered: remote SWMR (a remote
|
||||
file is pinned at open), `MmapFile`/`LazyFile` SWMR reads, refreshing
|
||||
groups or attributes.
|
||||
- [ ] **MPI collective I/O.** `clawhdf5-io`'s `MpiVol` (`mpi-io`) is
|
||||
root-read + broadcast and gather-to-root writes, not collective MPI-IO
|
||||
(`MPI_File_read_at_all`/`write_at_all`).
|
||||
- [ ] **Paged-metadata single-request reads.** Files written with paged
|
||||
aggregation (`H5Pset_file_space_strategy(PAGE)`, `h5repack -S PAGE`)
|
||||
keep their metadata in a few pages; range reads could fetch those in one
|
||||
request and use the file's page size as the block size. Today the block
|
||||
size is fixed (1 MiB) and only the first block is read ahead
|
||||
(range-reads design, option (c) as a policy).
|
||||
- [ ] **Blosc2 and ZFP encoders.** Both filters are read-only; the other
|
||||
plugin filters (LZF, bitshuffle, bzip2, Blosc 1) read and write.
|
||||
- [ ] **External links and external raw data** are explicit errors, not
|
||||
followed.
|
||||
- [ ] **Virtual datasets:** the "first missing" view and printf gaps other
|
||||
than 0, source-to-virtual type conversion other than a byte swap, nested
|
||||
virtual sources, source files outside the virtual file's directory.
|
||||
- [ ] **Datatypes:** x87 long double and binary128 are refused.
|
||||
- [ ] **Writer:** one attribute or link message over 65 515 bytes in dense
|
||||
storage is an error (huge fractal-heap objects); no option to write
|
||||
files HDF5 1.8 can read.
|
||||
- [ ] **`FileEditor`:** new chunks in implicit indexes, variable-length and
|
||||
reference data, filters it cannot encode (scale-offset, N-Bit, SZIP),
|
||||
some dense-attribute heap layouts, creating or deleting objects and
|
||||
attributes (also from Python `'r+'`), and no journal (a crash mid-edit
|
||||
can leave the file inconsistent). Freed space is reused only within one
|
||||
editor.
|
||||
- [ ] **Selection reads** decode the whole dataset when the selection's
|
||||
bounding box covers more than half of it (a strided `ds[::100]`), and
|
||||
for compact/virtual datasets or a non-default fill value: correct, but
|
||||
more work than needed.
|
||||
- [ ] **Readers:** `LazyFile` and `MmapFile` still need the whole file;
|
||||
the zero-copy methods need the file in memory.
|
||||
|
||||
### Remote and browser
|
||||
|
||||
- [ ] Run the `s3`/`gcs`/`azure` backends against real buckets (only built
|
||||
and URL-parsing-tested so far).
|
||||
- [ ] `h5rs` options for request headers and cache settings.
|
||||
- [ ] Browser limits in [`docs/known-issues.md`](docs/known-issues.md)
|
||||
("`clawhdf5-wasm` (browser) limits"): files of 4 GiB or more (wasm32),
|
||||
compound/reference/opaque datasets, round trips per index level. The
|
||||
package doubled in size with `openUrl`
|
||||
([size table](examples/wasm-viewer/README.md#size)); dropping the
|
||||
function-name section would take a third off the raw size (13% gzipped).
|
||||
|
||||
### Quality
|
||||
|
||||
- [ ] Scheduled fuzz campaigns: the cargo-fuzz targets
|
||||
([`crates/clawhdf5-format/fuzz`](crates/clawhdf5-format/fuzz/README.md),
|
||||
and the agent's WAL target) run only by hand or with
|
||||
`CLAWHDF5_FUZZ_SECONDS`.
|
||||
|
||||
---
|
||||
|
||||
## Track 3: Hybrid Retrieval Pipeline
|
||||
**Status:** 🟢 Phase 1 Complete
|
||||
**Priority:** High
|
||||
**Crate:** `clawhdf5-agent`
|
||||
## Withdrawn
|
||||
|
||||
- [x] **3.1** Reciprocal Rank Fusion (RRF) — rrf_hybrid_search() with k=60 constant
|
||||
- [x] **3.2** Multi-factor re-ranking — temporal decay, source authority hierarchy, activation scores (reranker.rs)
|
||||
- [x] **3.3** Low-confidence rejection — min_score threshold, gap filtering, max_results (confidence.rs)
|
||||
- [x] **3.4** Query expansion — synonyms, acronyms, temporal rewrites, morphological variants, knowledge graph aliases + expanded_search() with RRF merge
|
||||
- [x] **3.5** Result explanation — ReRankResult with full score breakdown per factor
|
||||
- [x] **3.6** Configurable pipeline — ReRankConfig + ConfidenceConfig with tunable weights/thresholds
|
||||
- [x] **3.7** Tests + MemX-comparable benchmarks — 5 integration tests (Hit@1≥90%, search<500ms@100K, BM25<200ms@100K, hybrid<50ms@10K, compact<200ms@10K)
|
||||
- **OpenClaw integration** (withdrawn 2026-09-25, PR #9). clawhdf5 was
|
||||
never an OpenClaw memory plugin; the documented
|
||||
`memory.backend = "clawhdf5"` was never valid. The Rust `ClawhdfBackend`
|
||||
remains as a library API. [`docs/openclaw.md`](docs/openclaw.md) records
|
||||
what a real plugin would need.
|
||||
- **ZeroClaw integration** (withdrawn 2026-09-25, PR #10). ZeroClaw has no
|
||||
clawhdf5 backend, and `clawhdf5-migrate`'s SQLite layout is not
|
||||
ZeroClaw's schema.
|
||||
|
||||
**Research:** MemX (2026), SwiftMem (2026)
|
||||
|
||||
---
|
||||
|
||||
## Track 4: Temporal Reasoning
|
||||
**Status:** 🟢 Phase 1 Complete
|
||||
**Priority:** High
|
||||
**Crate:** `clawhdf5-agent`
|
||||
|
||||
- [x] **4.1** Temporal index — sorted timestamp index with binary search, insert/remove
|
||||
- [x] **4.2** Time-range queries — range_query, before, after, latest, earliest
|
||||
- [x] **4.3** Session DAG — parent/child linking, chain walking, time-range overlap queries
|
||||
- [x] **4.4** Temporal re-ranking — query hint enum (Latest/Earliest/Around/Between/None) with boost scoring
|
||||
- [x] **4.5** Temporal entity tracking — EntityTimeline with state change history + point-in-time reconstruction
|
||||
- [x] **4.6** Tests — comprehensive tests for all features
|
||||
|
||||
**Research:** MemX temporal gaps (≤43.6% Hit@5), MemoryArena multi-session tasks (2026)
|
||||
|
||||
---
|
||||
|
||||
## Track 5: Memory Security & Provenance
|
||||
**Status:** 🟢 Phase 1 Complete
|
||||
**Priority:** Medium-High
|
||||
**Crate:** `clawhdf5-agent`
|
||||
|
||||
- [x] **5.1** Source attribution — MemoryProvenance with source, creator, session, FNV-1a content hash
|
||||
- [x] **5.2** Write anomaly detection — rate limiting, 15 injection patterns, source distribution analysis
|
||||
- [x] **5.3** Source isolation — per-MemorySource sub-stores preventing cross-contamination
|
||||
- [x] **5.4** Memory integrity verification — content hash comparison via verify_integrity()
|
||||
- [x] **5.5** Poisoning resistance — pattern detection for prompt injection attempts
|
||||
- [x] **5.6** Tests — comprehensive tests including adversarial patterns
|
||||
|
||||
**Research:** MemoryGraft (2025), SSGM Framework (2026)
|
||||
|
||||
---
|
||||
|
||||
## Track 6: Multi-Modal Memory
|
||||
**Status:** 🟢 Phase 1 Complete
|
||||
**Priority:** Medium
|
||||
**Crate:** `clawhdf5-agent`
|
||||
|
||||
- [x] **6.1** Image embedding storage — ModalEmbedding with model provenance (CLIP, SigLIP, etc.)
|
||||
- [x] **6.2** Audio fingerprints — Audio modality with embedding storage
|
||||
- [x] **6.3** Multi-modal search — search_by_modality (filtered) + search_cross_modal (all embeddings)
|
||||
- [x] **6.4** Observation records — raw perception vs interpretation with confidence scoring
|
||||
- [x] **6.5** Media reference storage — MediaRef with Path/Url/Inline, MIME types, FNV-1a checksums
|
||||
- [x] **6.6** Tests — 35 comprehensive tests
|
||||
|
||||
**Research:** Neuro-Symbolic Memory (2026), RAGdb multi-modal RAG (2025)
|
||||
|
||||
---
|
||||
|
||||
## Track 7: OpenClaw Integration — withdrawn (2026-09-25)
|
||||
**Status:** ⚪ Withdrawn (the items below were library work; no OpenClaw integration shipped)
|
||||
**Priority:** Critical (for adoption)
|
||||
**Crates:** `clawhdf5-agent`, `clawhdf5-napi`
|
||||
|
||||
- [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
|
||||
- [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
|
||||
- [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
|
||||
- [x] **7.4** `search()` — backed by the full hybrid retrieval pipeline (a Rust method; no OpenClaw tool was ever registered)
|
||||
- [x] **7.5** `get()` — read back by path, with a line slice (not an OpenClaw tool either)
|
||||
- [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
|
||||
- [ ] **7.7** ~~Config surface — `memory.backend = "clawhdf5"`~~ — never valid OpenClaw config; docs removed
|
||||
- [ ] **7.8** ~~Documentation + migration guide~~ — removed: they described an integration that never worked
|
||||
|
||||
**Node.js bridge:** `clawhdf5-napi` (napi-rs) and a TypeScript wrapper in `packages/clawhdf5-node` exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).
|
||||
|
||||
---
|
||||
|
||||
> **Withdrawn.** None of this track produced a working OpenClaw integration: no
|
||||
> plugin was built, the documented `memory.backend = "clawhdf5"` config was never
|
||||
> valid in any OpenClaw release, and the Node package was never published. The
|
||||
> Rust `ClawhdfBackend` remains as a library API. Not pursued for now; see
|
||||
> [docs/openclaw.md](docs/openclaw.md) for what a plugin would need today.
|
||||
|
||||
## Track 8: Benchmarking & Validation
|
||||
**Status:** 🟢 Complete
|
||||
**Priority:** High
|
||||
**Crates:** `clawhdf5-agent`, `clawhdf5-bench`
|
||||
|
||||
- [x] **8.1** MemoryArena benchmark — 35 queries, 50 sessions, Hit@10=91.4%, MRR=0.547
|
||||
- [x] **8.2** LongMemEval benchmark — 500 questions, retrieval recall (not QA accuracy). Full `longmemeval_s` haystack, hybrid 0.4/0.6 with MiniLM embeddings: turn Hit@5 81.4%, MRR 0.643; session Hit@5 96.8% (re-run 2026-09-27 on tank). Oracle variant: BM25-only turn Hit@5 84.4%, MRR 0.660; hybrid 86.8%. The session Hit@1 of 100% first recorded here was degenerate on the oracle variant, and the "beats MemX 51.6%" claim compared a different granularity. Both are retracted; see [BENCHMARKS.md § LongMemEval Results](BENCHMARKS.md#longmemeval-results)
|
||||
- [x] **8.3** Latency benchmarks — vector search at 1K/10K/100K, hybrid/RRF, graph traversal, consolidation, temporal
|
||||
- [x] **8.4** Memory footprint — 1.7 KB/record uncompressed, 282 B compressed (6.2x ratio), 100K+ rec/s ingestion
|
||||
- [x] **8.5** Consolidation efficiency — 8.8x search speedup, 90% noise eviction, zero quality loss
|
||||
- [x] **8.6** Cross-platform benchmarks — x86 measured, ARM estimated, cross_platform.sh script
|
||||
- [x] **8.7** Published results in BENCHMARKS.md with ephemeral tier Redis comparison (70-140x faster)
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
**Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
|
||||
**Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
|
||||
**Phase 3:** ~~Track 6 (multi-modal)~~ 🟢 Complete; Track 7 (OpenClaw integration) withdrawn
|
||||
**Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
|
||||
|
||||
All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.
|
||||
|
||||
---
|
||||
|
||||
## What's Next
|
||||
|
||||
Verified against current repo state on 2026-08-05 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped):
|
||||
|
||||
- [ ] TypeScript bridge not wired into CI — `packages/clawhdf5-node/` already has a complete, working napi-rs package (package.json, tsconfig, hand-written TS wrapper matching all 21 `#[napi]` items, Jest test suite, README); it isn't published to npm and has no committed lockfile
|
||||
- [ ] Publish crates to crates.io — no `publish` config anywhere in the workspace yet
|
||||
- [ ] Python wheel distribution via maturin — `crates/clawhdf5-py/pyproject.toml` exists (maturin-buildable locally) but wheels aren't published anywhere
|
||||
- [ ] `chunked_read.rs`/`data_read.rs` full bounds-check audit + scheduled fuzz campaigns (the new `fuzz_dataset_read` target covers the two files' main entry points; a full manual audit of every indexing site is still open) — see Tier 4 below
|
||||
- [ ] WAL per-entry checksum landed as CRC32 (see below); a stronger per-entry format (explicit length prefix, avoiding the read-then-verify restructuring) could still be revisited if profiling shows it matters
|
||||
- [ ] HNSW build parallelism is still narrow (only `prune_connections`); the correctness-sensitive outer insert loop needs its own dedicated design pass before parallelizing
|
||||
|
||||
### Recently closed out (2026-08-05, Tier 3–4 hardening pass)
|
||||
|
||||
- [x] Academic benchmark cross-validation — LongMemEval reproduced on tank (Ryzen 7 7800X3D): turn-level Hit@5 84.4% on the oracle variant (the comparison with MemX's 51.6% made here was later retracted, since MemX measures fact-level granularity over a far larger corpus); recall numbers are deterministic and reproduce exactly across machines. SIMD/Parallelism and Vector Search sections also re-run and dated. See [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05)
|
||||
- [x] Android JNI (`clawhdf5-android`): validate `embedding_len`/`query_embedding_len` against the handle's configured `embedding_dim` before constructing a slice from a raw pointer
|
||||
- [x] `clawhdf5-py`: bumped pyo3/numpy 0.28 → 0.29, clearing two RUSTSEC advisories
|
||||
- [x] WAL (`clawhdf5-agent`): length-prefix caps (`MAX_WAL_FIELD_LEN`) to reject a corrupted length claim before allocating, then a full per-entry CRC32 trailer (`WAL_VERSION` 2) so a bit-flip stops replay cleanly instead of loading corrupted data; old-format WAL files still read correctly and are migrated on next open
|
||||
- [x] `chunked_read.rs`/`data_read.rs`/`local_heap.rs` bounds-check audit: added `ensure_len` overflow guards, a recursion-depth guard against cyclic B-trees, and a fix for an unguarded compound-datatype byte-offset overrun. Added a new `fuzz_dataset_read` cargo-fuzz target exercising the contiguous/chunked/compact read paths — it found and we fixed 3 real crash bugs (integer-overflow panics) within the first few runs
|
||||
- [x] `clawhdf5-ann`: optional `parallel` feature (rayon) for HNSW's `prune_connections` neighbor-distance computation
|
||||
- [x] `[workspace.dependencies]` added for `tempfile`/`criterion`/`half`/`serde`, fixing a real version skew on `half` (2 vs 2.7)
|
||||
|
||||
### Recently closed out (2026-08-05 hardening pass)
|
||||
|
||||
- [x] CI/CD pipeline — `.gitea/workflows/ci.yml` now runs `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) on push/PR to `main`
|
||||
- [x] Fixed no_std build breakage in `clawhdf5-format` (missing alloc imports, `AtomicU64` unsupported on thumbv7em, `f64::powi` requiring std/libm)
|
||||
- [x] Fixed version skew: `clawhdf5-py` (pyproject.toml) and `packages/clawhdf5-node` (package.json) were both behind the actual crate version
|
||||
|
||||
### Recently closed out (2026-08-03 cleanup pass)
|
||||
|
||||
- [x] Removed `clawhdf5-types` — it was an empty 1-line stub crate; shared type definitions already live in `clawhdf5-format`, so CLAUDE.md and the workspace manifest were corrected instead of filling it in
|
||||
- [x] Superblock v4 (page-buffer mode) read/write — the only unimplemented task from `docs/superpowers/plans/2026-06-29-format-write-extensions.md`; now done (`Superblock::parse_v4`/`serialize`, `FileWriter::with_page_size`)
|
||||
- [x] Reconciled the three `docs/superpowers/plans/*.md` docs against actual shipped code — they were pre-work plans for `d6c4d4f` (2026-06-30), committed to git late; checkboxes now reflect reality
|
||||
|
||||
---
|
||||
|
||||
_Last updated: 2026-08-05_
|
||||
The old track-by-track tracker this file used to be (agent-memory
|
||||
Tracks 1–8, mid-2026) is in git history (`git log -- ROADMAP.md`).
|
||||
|
||||
Reference in New Issue
Block a user