Merge pull request 'Withdraw the OpenClaw integration claims' (#9) from docs/withdraw-openclaw into main
Reviewed-on: #9
This commit was merged in pull request #9.
This commit is contained in:
@@ -3,6 +3,15 @@
|
|||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
### Upgrade Notes
|
### Upgrade Notes
|
||||||
|
- **OpenClaw is not supported, and never was.** The docs described a
|
||||||
|
"drop-in" OpenClaw memory backend enabled with `memory.backend = "clawhdf5"`.
|
||||||
|
That config was never valid in any OpenClaw release (v2026.2–v2026.7
|
||||||
|
accepted only `builtin`/`qmd` and rejected unknown keys, so a Gateway given
|
||||||
|
it refuses to start; OpenClaw 2.0 removed the key), no plugin was ever built,
|
||||||
|
and `@redclaw/clawhdf5` was never published. The integration docs
|
||||||
|
(`openclaw-integration.md`, `openclaw-config.md`, `migration-guide.md`) are
|
||||||
|
removed; `docs/openclaw.md` explains the status and what a real plugin would
|
||||||
|
need against OpenClaw v2026.9.6. `ClawhdfBackend` stays as a library API.
|
||||||
- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained
|
- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained
|
||||||
`SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants
|
`SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants
|
||||||
will no longer be breaking.
|
will no longer be breaking.
|
||||||
@@ -126,6 +135,13 @@
|
|||||||
activation of the `k` results it returns, not of the whole `3k` candidate
|
activation of the `k` results it returns, not of the whole `3k` candidate
|
||||||
pool it re-ranks.
|
pool it re-ranks.
|
||||||
|
|
||||||
|
### Documentation
|
||||||
|
- OpenClaw claims withdrawn across the README, QUICKSTART, USE_CASES, ROADMAP
|
||||||
|
(Track 7 marked withdrawn) and the `openclaw` module docs; the dead
|
||||||
|
`github.com/redclawsystems/openclaw` link is gone. The Node package is
|
||||||
|
marked unpublished and broken (now `"private": true` so it cannot be
|
||||||
|
published by accident), with its bugs recorded in `docs/known-issues.md`.
|
||||||
|
|
||||||
### Benchmarks
|
### Benchmarks
|
||||||
- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
|
- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
|
||||||
machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
|
machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
|
||||||
|
|||||||
@@ -104,8 +104,12 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
|
|||||||
the allowed records whenever cheaper than `pool × M` index distance
|
the allowed records whenever cheaper than `pool × M` index distance
|
||||||
evaluations, and as the fallback when the pool comes back short), fusion,
|
evaluations, and as the fallback when the pool comes back short), fusion,
|
||||||
activation scaling, optional re-ranking and confidence rejection.
|
activation scaling, optional re-ranking and confidence rejection.
|
||||||
`hybrid_search`/`hybrid_search_with` are thin wrappers; the OpenClaw
|
`hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend`
|
||||||
backend is `search` with re-rank + confidence on. Measure changes with
|
(the `openclaw` module) is `search` with re-rank + confidence on.
|
||||||
|
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an
|
||||||
|
OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"`
|
||||||
|
config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md`
|
||||||
|
records what a real plugin would need. ZeroClaw is the integration target. Measure changes with
|
||||||
`search_harness --options-study`.
|
`search_harness --options-study`.
|
||||||
- `MemoryConfig::compression` is off by default; when on, embeddings are
|
- `MemoryConfig::compression` is off by default; when on, embeddings are
|
||||||
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
|
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
|
||||||
|
|||||||
@@ -71,7 +71,7 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
|||||||
100K). It no longer rebuilds BM25 or rewrites the store per query, and the
|
100K). It no longer rebuilds BM25 or rewrites the store per query, and the
|
||||||
HNSW graph is persisted (v2.4.0).
|
HNSW graph is persisted (v2.4.0).
|
||||||
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
|
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
|
||||||
been discarding the retrieval score, costing the OpenClaw backend 40.6pp of
|
been discarding the retrieval score, costing the Markdown backend 40.6pp of
|
||||||
Hit@1; fixed in v2.6.0.
|
Hit@1; fixed in v2.6.0.
|
||||||
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
|
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
|
||||||
0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
|
0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
|
||||||
@@ -93,7 +93,7 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
|
|||||||
identical LongMemEval retrieval on real embeddings.
|
identical LongMemEval retrieval on real embeddings.
|
||||||
- `HDF5Memory::search` with `SearchOptions`: filter by source channel (exact
|
- `HDF5Memory::search` with `SearchOptions`: filter by source channel (exact
|
||||||
filtered top-k, never slower than unfiltered), and opt-in re-ranking and
|
filtered top-k, never slower than unfiltered), and opt-in re-ranking and
|
||||||
confidence rejection, which used to be OpenClaw-only.
|
confidence rejection, which used to be reachable only through `ClawhdfBackend`.
|
||||||
|
|
||||||
**Tooling**
|
**Tooling**
|
||||||
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
|
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
|
||||||
@@ -228,7 +228,7 @@ is for. The weights matter more than the stages: a sweep of `vector_weight` from
|
|||||||
0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by
|
0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by
|
||||||
`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since
|
`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since
|
||||||
v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by
|
v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by
|
||||||
`unified_search`, `hybrid_search_with` and the OpenClaw backend); callers that
|
`unified_search`, `hybrid_search_with` and `ClawhdfBackend`); callers that
|
||||||
pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if
|
pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if
|
||||||
rank-1 precision matters most. Reciprocal rank fusion is selectable
|
rank-1 precision matters most. Reciprocal rank fusion is selectable
|
||||||
(`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See
|
(`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See
|
||||||
@@ -329,7 +329,7 @@ ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory syst
|
|||||||
│ × √(Hebbian activation) │
|
│ × √(Hebbian activation) │
|
||||||
└─────────────────┬──────────────────┘
|
└─────────────────┬──────────────────┘
|
||||||
│ opt-in (SearchOptions);
|
│ opt-in (SearchOptions);
|
||||||
│ the OpenClaw backend turns both on
|
│ ClawhdfBackend turns both on
|
||||||
┌─────────────────▼──────────────────┐
|
┌─────────────────▼──────────────────┐
|
||||||
│ Multi-factor re-ranking │
|
│ Multi-factor re-ranking │
|
||||||
│ relevance · recency · authority · │
|
│ relevance · recency · authority · │
|
||||||
@@ -365,14 +365,14 @@ directly; the store persists the records, sessions and graph they work over.
|
|||||||
| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
|
| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
|
||||||
| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
|
| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
|
||||||
| **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
|
| **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
|
||||||
| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Opt-in via `SearchOptions::with_rerank`; on in the OpenClaw backend |
|
| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Opt-in via `SearchOptions::with_rerank`; on in `ClawhdfBackend` |
|
||||||
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend |
|
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in `ClawhdfBackend` |
|
||||||
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
|
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
|
||||||
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
|
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
|
||||||
| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record |
|
| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record |
|
||||||
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
|
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
|
||||||
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
|
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
|
||||||
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
|
| **`openclaw`** | `ClawhdfBackend`: a Markdown-oriented backend (ingest by section, search, read back by path, export). Named for OpenClaw, but **not an OpenClaw plugin** — see [docs/openclaw.md](docs/openclaw.md) |
|
||||||
| **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
|
| **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
|
||||||
| **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW |
|
| **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW |
|
||||||
| **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming |
|
| **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming |
|
||||||
@@ -448,7 +448,7 @@ let work = memory.search(
|
|||||||
);
|
);
|
||||||
|
|
||||||
// Re-rank by relevance, recency, source authority and activation, then drop
|
// Re-rank by relevance, recency, source authority and activation, then drop
|
||||||
// low-confidence results — the pipeline the OpenClaw backend runs.
|
// low-confidence results — the pipeline ClawhdfBackend runs.
|
||||||
let careful = memory.search(
|
let careful = memory.search(
|
||||||
&query_embedding,
|
&query_embedding,
|
||||||
"user preferences",
|
"user preferences",
|
||||||
@@ -557,7 +557,13 @@ let ids = index.range_query(1700000000.0, 1700010800.0);
|
|||||||
let recent = index.latest(10);
|
let recent = index.latest(10);
|
||||||
```
|
```
|
||||||
|
|
||||||
### OpenClaw Integration
|
### Markdown Backend
|
||||||
|
|
||||||
|
`ClawhdfBackend` ingests Markdown by section and searches it with the full
|
||||||
|
pipeline. It is a library API — clawhdf5 is **not** an OpenClaw memory plugin
|
||||||
|
([docs/openclaw.md](docs/openclaw.md)). Sections stored this way carry no
|
||||||
|
embedding, so their search is keyword-only unless you save records with
|
||||||
|
vectors through `save_entry`.
|
||||||
|
|
||||||
```rust
|
```rust
|
||||||
use clawhdf5_agent::openclaw::*;
|
use clawhdf5_agent::openclaw::*;
|
||||||
@@ -851,10 +857,10 @@ See [ROADMAP.md](ROADMAP.md) for the full implementation tracker.
|
|||||||
- ✅ Temporal reasoning with sub-µs queries
|
- ✅ Temporal reasoning with sub-µs queries
|
||||||
- ✅ Memory security + anomaly detection
|
- ✅ Memory security + anomaly detection
|
||||||
- ✅ Multi-modal memory (text/image/audio/video)
|
- ✅ Multi-modal memory (text/image/audio/video)
|
||||||
- ✅ OpenClaw integration layer
|
- ✅ Markdown ingest/export backend (`ClawhdfBackend`); an OpenClaw plugin was never built — see [docs/openclaw.md](docs/openclaw.md)
|
||||||
- ✅ Comprehensive Criterion benchmarks
|
- ✅ Comprehensive Criterion benchmarks
|
||||||
|
|
||||||
**Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: publish the OpenClaw TypeScript bridge to npm, crates.io/PyPI publishing.
|
**Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: crates.io/PyPI publishing. The Node bindings are unpublished and known to be broken ([known issues](docs/known-issues.md)).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+14
-8
@@ -105,24 +105,30 @@
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Track 7: OpenClaw Integration
|
## Track 7: OpenClaw Integration — withdrawn (2026-09-25)
|
||||||
**Status:** 🟢 Complete
|
**Status:** ⚪ Withdrawn (the items below were library work; no OpenClaw integration shipped)
|
||||||
**Priority:** Critical (for adoption)
|
**Priority:** Critical (for adoption)
|
||||||
**Crates:** `clawhdf5-agent`, `clawhdf5-napi`
|
**Crates:** `clawhdf5-agent`, `clawhdf5-napi`
|
||||||
|
|
||||||
- [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
|
- [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
|
||||||
- [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
|
- [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
|
||||||
- [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
|
- [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
|
||||||
- [x] **7.4** memory_search tool — backed by full hybrid retrieval pipeline
|
- [x] **7.4** `search()` — backed by the full hybrid retrieval pipeline (a Rust method; no OpenClaw tool was ever registered)
|
||||||
- [x] **7.5** memory_get tool — get() with path + line range support
|
- [x] **7.5** `get()` — read back by path, with a line slice (not an OpenClaw tool either)
|
||||||
- [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
|
- [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
|
||||||
- [x] **7.7** Config surface — `memory.backend = "clawhdf5"` schema documented in docs/openclaw-config.md
|
- [ ] **7.7** ~~Config surface — `memory.backend = "clawhdf5"`~~ — never valid OpenClaw config; docs removed
|
||||||
- [x] **7.8** Documentation + migration guide — docs/migration-guide.md, docs/openclaw-integration.md (architecture, full API reference, code patterns)
|
- [ ] **7.8** ~~Documentation + migration guide~~ — removed: they described an integration that never worked
|
||||||
|
|
||||||
**Node.js bridge:** `clawhdf5-napi` (napi-rs) → `@redclaw/clawhdf5` npm package with full TypeScript types.
|
**Node.js bridge:** `clawhdf5-napi` (napi-rs) and a TypeScript wrapper in `packages/clawhdf5-node` exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
> **Withdrawn.** None of this track produced a working OpenClaw integration: no
|
||||||
|
> plugin was built, the documented `memory.backend = "clawhdf5"` config was never
|
||||||
|
> valid in any OpenClaw release, and the Node package was never published. The
|
||||||
|
> Rust `ClawhdfBackend` remains as a library API. Not pursued for now; see
|
||||||
|
> [docs/openclaw.md](docs/openclaw.md) for what a plugin would need today.
|
||||||
|
|
||||||
## Track 8: Benchmarking & Validation
|
## Track 8: Benchmarking & Validation
|
||||||
**Status:** 🟢 Complete
|
**Status:** 🟢 Complete
|
||||||
**Priority:** High
|
**Priority:** High
|
||||||
@@ -142,7 +148,7 @@
|
|||||||
|
|
||||||
**Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
|
**Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
|
||||||
**Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
|
**Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
|
||||||
**Phase 3:** ~~Track 6 (multi-modal) + Track 7 (OpenClaw integration)~~ 🟢 Complete
|
**Phase 3:** ~~Track 6 (multi-modal)~~ 🟢 Complete; Track 7 (OpenClaw integration) withdrawn
|
||||||
**Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
|
**Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
|
||||||
|
|
||||||
All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.
|
All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.
|
||||||
|
|||||||
@@ -203,7 +203,7 @@ impl ImportanceScorer {
|
|||||||
/// Novelty score: 1.0 − max cosine similarity against all existing records.
|
/// Novelty score: 1.0 − max cosine similarity against all existing records.
|
||||||
/// Returns 1.0 when there are no existing memories.
|
/// Returns 1.0 when there are no existing memories.
|
||||||
///
|
///
|
||||||
/// Same result as [`Self::cosine_similarity`] against each record, but the
|
/// Same result as the reference cosine similarity against each record, but the
|
||||||
/// new embedding's norm is computed once rather than per record, each
|
/// new embedding's norm is computed once rather than per record, each
|
||||||
/// record costs one fused pass (dot product and its norm together) rather
|
/// record costs one fused pass (dot product and its norm together) rather
|
||||||
/// than three, and a large working set is scored in parallel. Every insert
|
/// than three, and a large working set is scored in parallel. Every insert
|
||||||
|
|||||||
@@ -1,10 +1,12 @@
|
|||||||
//! OpenClaw Integration Layer.
|
//! A Markdown-oriented memory backend over [`crate::HDF5Memory`].
|
||||||
//!
|
//!
|
||||||
//! Bridge between OpenClaw agent gateway (Markdown + sqlite-vec) and the
|
//! Named for OpenClaw, whose workspace memory is Markdown, but **not an
|
||||||
//! clawhdf5 HDF5-backed memory backend. Provides:
|
//! OpenClaw plugin**: nothing here registers with OpenClaw, and the
|
||||||
|
//! integration it was written for never worked (see `docs/openclaw.md`).
|
||||||
|
//! Provides:
|
||||||
//!
|
//!
|
||||||
//! - [`MemoryBackend`] — the trait OpenClaw implements against.
|
//! - [`MemoryBackend`] — search / read back / write / ingest / export.
|
||||||
//! - [`ClawhdfBackend`] — concrete HDF5-backed implementation.
|
//! - [`ClawhdfBackend`] — the HDF5-backed implementation.
|
||||||
//! - [`MarkdownParser`] — splits Markdown into [`MarkdownSection`] records.
|
//! - [`MarkdownParser`] — splits Markdown into [`MarkdownSection`] records.
|
||||||
//! - [`MarkdownExporter`] — renders sections back to Markdown text.
|
//! - [`MarkdownExporter`] — renders sections back to Markdown text.
|
||||||
|
|
||||||
@@ -61,7 +63,8 @@ pub struct BackendStats {
|
|||||||
// MemoryBackend trait
|
// MemoryBackend trait
|
||||||
// ─────────────────────────────────────────────────────────────────────────────
|
// ─────────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
/// Interface that OpenClaw uses to interact with a memory backend.
|
/// A Markdown-oriented memory backend: search, read back by path, write,
|
||||||
|
/// ingest and export.
|
||||||
///
|
///
|
||||||
/// Implementors provide persistent storage, full-text + vector search,
|
/// Implementors provide persistent storage, full-text + vector search,
|
||||||
/// Markdown ingestion / export, and statistics.
|
/// Markdown ingestion / export, and statistics.
|
||||||
@@ -318,7 +321,7 @@ impl MarkdownExporter {
|
|||||||
///
|
///
|
||||||
/// # Path mapping
|
/// # Path mapping
|
||||||
///
|
///
|
||||||
/// OpenClaw addresses memories by file path (e.g. `"memory/user.md"`).
|
/// Memories are addressed by file path (e.g. `"memory/user.md"`).
|
||||||
/// Internally every [`MemoryEntry`] stores the originating path as its
|
/// Internally every [`MemoryEntry`] stores the originating path as its
|
||||||
/// `source_channel`. Section sub-paths are stored as
|
/// `source_channel`. Section sub-paths are stored as
|
||||||
/// `"<path>::<heading>"`.
|
/// `"<path>::<heading>"`.
|
||||||
@@ -421,7 +424,7 @@ impl ClawhdfBackend {
|
|||||||
|
|
||||||
// ── Compaction & Consolidation hooks (7.6) ────────────────────────────
|
// ── Compaction & Consolidation hooks (7.6) ────────────────────────────
|
||||||
|
|
||||||
/// Run a compaction cycle — called by OpenClaw during session compaction.
|
/// Run a compaction cycle (decay, compaction, WAL flush).
|
||||||
///
|
///
|
||||||
/// Sequence:
|
/// Sequence:
|
||||||
/// 1. `tick_session()` — apply Hebbian decay to all activation weights.
|
/// 1. `tick_session()` — apply Hebbian decay to all activation weights.
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
//!
|
//!
|
||||||
//! Records the origin, authorship, and a content hash of every memory chunk
|
//! Records the origin, authorship, and a content hash of every memory chunk
|
||||||
//! so the system can detect *accidental* corruption and trace data lineage.
|
//! so the system can detect *accidental* corruption and trace data lineage.
|
||||||
//! The hash is unkeyed (see [`fnv1a_64`]) — this is not a tamper-evidence or
|
//! The hash is unkeyed (FNV-1a) — this is not a tamper-evidence or
|
||||||
//! authenticity guarantee.
|
//! authenticity guarantee.
|
||||||
|
|
||||||
use std::collections::HashMap;
|
use std::collections::HashMap;
|
||||||
|
|||||||
+19
-61
@@ -11,7 +11,7 @@ ClawhDF5 serves three audiences with different entry points:
|
|||||||
| You Are | You Want | Start Here |
|
| You Are | You Want | Start Here |
|
||||||
|---------|----------|------------|
|
|---------|----------|------------|
|
||||||
| **AI agent developer** | Persistent memory for your agent | [Agent Memory (Rust)](#1-agent-memory-rust-library) |
|
| **AI agent developer** | Persistent memory for your agent | [Agent Memory (Rust)](#1-agent-memory-rust-library) |
|
||||||
| **OpenClaw user** | Better memory for your OpenClaw agent | [OpenClaw Integration](#2-openclaw-integration) |
|
| **OpenClaw user** | clawhdf5 is not an OpenClaw memory plugin | [Status](openclaw.md) |
|
||||||
| **Data scientist** | Read/write HDF5 files in Rust | [HDF5 File I/O](#3-hdf5-file-io) |
|
| **Data scientist** | Read/write HDF5 files in Rust | [HDF5 File I/O](#3-hdf5-file-io) |
|
||||||
| **CLI user** | Inspect and manage agent memories | [CLI Tool](#4-cli-tool) |
|
| **CLI user** | Inspect and manage agent memories | [CLI Tool](#4-cli-tool) |
|
||||||
| **Python user** | Use clawhdf5 from Python | [Python Bindings](#5-python-bindings) |
|
| **Python user** | Use clawhdf5 from Python | [Python Bindings](#5-python-bindings) |
|
||||||
@@ -197,80 +197,38 @@ if let Some(alert) = detector.check_rate_anomaly() {
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. OpenClaw Integration
|
## 2. Markdown Memory (and OpenClaw)
|
||||||
|
|
||||||
ClawhDF5 can serve as the memory backend for [OpenClaw](https://docs.openclaw.ai) agents, replacing the default Markdown + sqlite-vec approach.
|
**clawhdf5 is not an OpenClaw memory backend.** Earlier versions of this guide
|
||||||
|
described one; it never worked — see [openclaw.md](openclaw.md) for what
|
||||||
|
happened and what a real plugin would need.
|
||||||
|
|
||||||
### How It Works
|
What does exist is `ClawhdfBackend`, a library API that ingests Markdown files
|
||||||
|
by section and searches them with the full pipeline (hybrid retrieval,
|
||||||
```
|
re-ranking, confidence rejection):
|
||||||
OpenClaw Agent
|
|
||||||
│
|
|
||||||
├── memory_search("user preferences")
|
|
||||||
│ │
|
|
||||||
│ └── ClawhdfBackend
|
|
||||||
│ ├── Vector search (cosine)
|
|
||||||
│ ├── BM25 keyword search
|
|
||||||
│ ├── Reciprocal Rank Fusion
|
|
||||||
│ ├── Multi-factor re-ranking
|
|
||||||
│ └── Low-confidence rejection
|
|
||||||
│
|
|
||||||
└── agent_memory.h5 (single file, portable)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Migration from Markdown
|
|
||||||
|
|
||||||
```rust
|
```rust
|
||||||
use clawhdf5_agent::openclaw::*;
|
use clawhdf5_agent::openclaw::*;
|
||||||
|
use std::path::Path;
|
||||||
|
|
||||||
// Create a new HDF5 backend
|
let mut backend = ClawhdfBackend::create(Path::new("memory.h5"), 384)?;
|
||||||
let mut backend = ClawhdfBackend::create("memory.h5", "my-agent", 384)?;
|
|
||||||
|
|
||||||
// Import your existing MEMORY.md
|
// Each heading becomes a record, stored under "MEMORY.md::<heading>".
|
||||||
let md = std::fs::read_to_string("~/.openclaw/workspace/MEMORY.md")?;
|
let md = std::fs::read_to_string("MEMORY.md")?;
|
||||||
let count = backend.ingest_markdown("MEMORY.md", &md)?;
|
let count = backend.ingest_markdown("MEMORY.md", &md)?;
|
||||||
println!("Imported {} sections", count);
|
println!("Imported {count} sections");
|
||||||
|
|
||||||
// Import daily logs
|
|
||||||
for entry in std::fs::read_dir("~/.openclaw/workspace/memory/")? {
|
|
||||||
let path = entry?.path();
|
|
||||||
if path.extension().map(|e| e == "md").unwrap_or(false) {
|
|
||||||
let content = std::fs::read_to_string(&path)?;
|
|
||||||
let name = path.file_name().unwrap().to_string_lossy();
|
|
||||||
backend.ingest_markdown(&name, &content)?;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Search using the full pipeline
|
|
||||||
let results = backend.search("what are user preferences", &query_embedding, 5);
|
let results = backend.search("what are user preferences", &query_embedding, 5);
|
||||||
for r in &results {
|
for r in &results {
|
||||||
println!("[{:.3}] {} (from {})", r.score, r.text, r.path);
|
println!("[{:.3}] {} (from {})", r.score, r.text, r.path);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Export back to Markdown (lossless roundtrip)
|
|
||||||
let exported = backend.export_markdown("MEMORY.md")?;
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### What You Get Over sqlite-vec
|
Limits to know: sections ingested this way carry no embedding (search over them
|
||||||
|
is keyword-only unless you save records with vectors via `save_entry`);
|
||||||
| Feature | sqlite-vec | ClawhDF5 |
|
ingesting the same file again adds the sections again rather than replacing
|
||||||
|---------|-----------|----------|
|
them; and `export_markdown` rewrites every heading as `##`, so it is not a
|
||||||
| Vector search | ✅ | ✅ (8× faster at 100K) |
|
lossless round trip.
|
||||||
| Keyword search | ❌ | ✅ BM25 |
|
|
||||||
| Hybrid fusion | ❌ | ✅ RRF |
|
|
||||||
| Re-ranking | ❌ | ✅ Multi-factor |
|
|
||||||
| Confidence rejection | ❌ | ✅ |
|
|
||||||
| Knowledge graph | ❌ | ✅ |
|
|
||||||
| Memory consolidation | ❌ | ✅ |
|
|
||||||
| Temporal queries | ❌ | ✅ (716ns) |
|
|
||||||
| Anomaly detection | ❌ | ✅ |
|
|
||||||
| Provenance tracking | ❌ | ✅ |
|
|
||||||
| Multi-modal | ❌ | ✅ |
|
|
||||||
| Single portable file | ❌ (SQLite + MD files) | ✅ |
|
|
||||||
|
|
||||||
### Future: Native OpenClaw Plugin
|
|
||||||
|
|
||||||
The Phase 2 roadmap includes a native OpenClaw plugin (`memory.backend = "clawhdf5"`) that transparently replaces sqlite-vec. Until then, the Rust library can be wrapped via NAPI or used from the CLI.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -547,7 +505,7 @@ let final_results = confidence::reject_low_confidence(
|
|||||||
|
|
||||||
**Why not a vector database?** Pinecone, Qdrant, Weaviate — they're cloud services or heavy servers. Agent memory should be local, portable, and zero-dependency. An agent's memories should travel with it.
|
**Why not a vector database?** Pinecone, Qdrant, Weaviate — they're cloud services or heavy servers. Agent memory should be local, portable, and zero-dependency. An agent's memories should travel with it.
|
||||||
|
|
||||||
**Why not Markdown?** OpenClaw uses Markdown today and it works for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't.
|
**Why not Markdown?** Plain Markdown files work for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't.
|
||||||
|
|
||||||
**Why HDF5 specifically?**
|
**Why HDF5 specifically?**
|
||||||
- Native N-dimensional array storage (perfect for embeddings)
|
- Native N-dimensional array storage (perfect for embeddings)
|
||||||
|
|||||||
+3
-30
@@ -42,37 +42,10 @@ conversation → embedding → save to agent.h5
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2. OpenClaw Memory Upgrade
|
## 2. OpenClaw
|
||||||
|
|
||||||
**Scenario:** You run OpenClaw and the default Markdown + sqlite-vec memory works OK for simple recall but falls short on complex queries like "what did we decide about the deployment architecture last Tuesday?" or "who's responsible for the billing system?"
|
Not supported: clawhdf5 is not an OpenClaw memory plugin, and the config this
|
||||||
|
section used to show was never valid. See [openclaw.md](openclaw.md).
|
||||||
**Problem:** Markdown files have no semantic structure. sqlite-vec does flat vector search — no keyword fusion, no re-ranking, no temporal reasoning, no knowledge graph.
|
|
||||||
|
|
||||||
**ClawhDF5 solution:**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Migrate existing memories
|
|
||||||
clawhdf5 --path memory.h5 create --agent-id openclaw --dim 384
|
|
||||||
|
|
||||||
# Import your MEMORY.md and daily logs
|
|
||||||
# (programmatically via ClawhdfBackend::ingest_markdown)
|
|
||||||
```
|
|
||||||
|
|
||||||
Then in your OpenClaw config (future):
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"memory": {
|
|
||||||
"backend": "clawhdf5",
|
|
||||||
"path": "~/.openclaw/agents/main/memory.h5"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**What changes:**
|
|
||||||
- "What did we discuss last Tuesday?" → temporal index finds the session, returns memories from that time range
|
|
||||||
- "Who owns the billing system?" → knowledge graph traversal: billing_system → owned_by → Alice
|
|
||||||
- "Preferences about deployment" → hybrid search (vector + BM25) finds relevant memories even with different wording
|
|
||||||
- Bad search results get filtered out by confidence rejection instead of confusing the agent
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -279,3 +279,26 @@ the same agent-store interop test.
|
|||||||
|
|
||||||
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
||||||
which is what libhdf5 itself writes.
|
which is what libhdf5 itself writes.
|
||||||
|
|
||||||
|
## The Node.js package (`packages/clawhdf5-node`) does not work
|
||||||
|
|
||||||
|
**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
|
||||||
|
|
||||||
|
The TypeScript wrapper over `crates/clawhdf5-napi` has never run successfully:
|
||||||
|
|
||||||
|
- napi-rs converts `#[napi(object)]` fields to camelCase, but the wrapper reads
|
||||||
|
snake_case (`r.line_range`, `s.total_records`, `s.working_count`, …), so
|
||||||
|
every stats and consolidation field comes back `undefined`
|
||||||
|
(`src/index.ts:76-120`).
|
||||||
|
- It loads `../clawhdf5.node`, but `napi build --platform` produces
|
||||||
|
`clawhdf5.<triple>.node`; `main` points at `index.js` while `tsc` writes to
|
||||||
|
`dist/`; `napi prepublish` expects per-platform packages that are not
|
||||||
|
defined.
|
||||||
|
- `save`/`saveBatch` exist in the napi layer but not in the wrapper, so a
|
||||||
|
TypeScript caller cannot store an embedding at all.
|
||||||
|
- The WAL for `agent.brain` is `agent.h5.wal` (the store uses
|
||||||
|
`with_extension("h5.wal")`), not `agent.brain.wal` as the old docs and the
|
||||||
|
test cleanup assume.
|
||||||
|
|
||||||
|
It was written for an OpenClaw integration that is not being pursued (see
|
||||||
|
`docs/openclaw.md`). Fix and add CI, or remove it, before anyone depends on it.
|
||||||
|
|||||||
@@ -1,213 +0,0 @@
|
|||||||
# Migration Guide: OpenClaw sqlite-vec → clawhdf5
|
|
||||||
|
|
||||||
This guide walks through migrating an OpenClaw agent from its default
|
|
||||||
sqlite-vec + Markdown file memory to the `clawhdf5` HDF5 backend.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Why migrate?
|
|
||||||
|
|
||||||
| Feature | sqlite-vec + Markdown | clawhdf5 |
|
|
||||||
|---------|----------------------|----------|
|
|
||||||
| Storage format | SQLite WAL + flat .md files | Single HDF5 binary file |
|
|
||||||
| Vector search | sqlite-vec (SQLite extension) | Pure-Rust SIMD (clawhdf5-accel) |
|
|
||||||
| Full-text search | External (FTS5 or plain string match) | Built-in BM25 |
|
|
||||||
| Hybrid search | Manual combination | Automatic RRF blend |
|
|
||||||
| Memory tiers | Flat | Working → Episodic → Semantic |
|
|
||||||
| Hebbian decay | Not built-in | Automatic activation weighting |
|
|
||||||
| Portability | SQLite binary required | Zero native deps (all Rust) |
|
|
||||||
| Crash recovery | SQLite WAL | clawhdf5 WAL |
|
|
||||||
| Compaction | Manual | Auto-threshold + session-end |
|
|
||||||
| Embedding dim change | New DB required | New file required (same) |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Step 1: Install `@redclaw/clawhdf5`
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npm install @redclaw/clawhdf5
|
|
||||||
```
|
|
||||||
|
|
||||||
Or, if building from the monorepo source:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npm install -g @napi-rs/cli
|
|
||||||
cd packages/clawhdf5-node
|
|
||||||
npm install
|
|
||||||
npm run build
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Step 2: Update your OpenClaw config
|
|
||||||
|
|
||||||
Change `backend` from `"sqlite-vec"` (or `"markdown"`) to `"clawhdf5"`:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"memory": {
|
|
||||||
"backend": "clawhdf5",
|
|
||||||
"clawhdf5": {
|
|
||||||
"path": "./agent.brain",
|
|
||||||
"embeddingDim": 768
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
See [openclaw-config.md](openclaw-config.md) for the full schema.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Step 3: Run the one-time migration
|
|
||||||
|
|
||||||
clawhdf5 ships a migration helper that reads your existing Markdown memory
|
|
||||||
files and ingests them via `ingestMarkdown()`.
|
|
||||||
|
|
||||||
### Automated migration script
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
|
||||||
import { readFileSync, readdirSync, statSync } from 'fs';
|
|
||||||
import { join, relative } from 'path';
|
|
||||||
|
|
||||||
async function migrate(
|
|
||||||
memoryDir: string,
|
|
||||||
brainPath: string,
|
|
||||||
embeddingDim: number = 768,
|
|
||||||
): Promise<void> {
|
|
||||||
const mem = ClawhdfMemory.create(brainPath, embeddingDim);
|
|
||||||
|
|
||||||
// Walk all .md files under memoryDir
|
|
||||||
function walk(dir: string): string[] {
|
|
||||||
return readdirSync(dir).flatMap((entry) => {
|
|
||||||
const full = join(dir, entry);
|
|
||||||
return statSync(full).isDirectory() ? walk(full) : [full];
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
const files = walk(memoryDir).filter((f) => f.endsWith('.md'));
|
|
||||||
let totalSections = 0;
|
|
||||||
|
|
||||||
for (const file of files) {
|
|
||||||
const content = readFileSync(file, 'utf8');
|
|
||||||
const relPath = relative(process.cwd(), file);
|
|
||||||
const count = mem.ingestMarkdown(relPath, content);
|
|
||||||
console.log(` ${relPath}: ${count} sections`);
|
|
||||||
totalSections += count;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Force WAL merge after bulk import
|
|
||||||
mem.flushWal();
|
|
||||||
|
|
||||||
console.log(`\nMigration complete: ${files.length} files, ${totalSections} sections`);
|
|
||||||
const s = mem.stats();
|
|
||||||
console.log(` Total records: ${s.totalRecords}`);
|
|
||||||
console.log(` File size: ${(s.fileSizeBytes / 1024).toFixed(1)} KB`);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Usage
|
|
||||||
migrate('./memory', './agent.brain', 768).catch(console.error);
|
|
||||||
```
|
|
||||||
|
|
||||||
### What the migration does
|
|
||||||
|
|
||||||
1. Walks all `.md` files under your memory directory.
|
|
||||||
2. Parses each file into sections using the same `MarkdownParser` used by
|
|
||||||
OpenClaw (splits on ATX headings `#`, `##`, `###`, …).
|
|
||||||
3. Stores each section as a separate record in the HDF5 file with the file
|
|
||||||
path as the `source_channel` (e.g. `memory/user.md::Goals`).
|
|
||||||
4. Flushes the WAL to merge everything into the `.brain` file.
|
|
||||||
|
|
||||||
After migration, **the original `.md` files are not modified or deleted**.
|
|
||||||
You can keep them as a backup or remove them once you have verified the
|
|
||||||
migrated data.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Step 4: Verify
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
|
||||||
|
|
||||||
const mem = ClawhdfMemory.open('./agent.brain');
|
|
||||||
const s = mem.stats();
|
|
||||||
console.log('Records after migration:', s.totalRecords);
|
|
||||||
|
|
||||||
// Spot-check: retrieve a known path
|
|
||||||
const userMd = mem.get('memory/MEMORY.md');
|
|
||||||
console.log(userMd?.slice(0, 200));
|
|
||||||
|
|
||||||
// Round-trip a file back to Markdown
|
|
||||||
const exported = mem.exportMarkdown('memory/MEMORY.md');
|
|
||||||
console.log(exported.slice(0, 500));
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Step 5: Update agent code
|
|
||||||
|
|
||||||
If your agent code reads memory files directly from disk, update it to use
|
|
||||||
the clawhdf5 API instead:
|
|
||||||
|
|
||||||
**Before (sqlite-vec + file reads):**
|
|
||||||
```typescript
|
|
||||||
const content = readFileSync('memory/user.md', 'utf8');
|
|
||||||
const sections = parseMarkdown(content);
|
|
||||||
const results = await vectorSearch(query, sections, k);
|
|
||||||
```
|
|
||||||
|
|
||||||
**After (clawhdf5):**
|
|
||||||
```typescript
|
|
||||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
|
||||||
|
|
||||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
|
||||||
const embedding = await embed(query); // your embedding function
|
|
||||||
const results = mem.search(query, new Float32Array(embedding), k);
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Step 6: Session lifecycle hooks
|
|
||||||
|
|
||||||
Add compaction at session end for best long-term memory health:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// At the start of your agent process
|
|
||||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
|
||||||
|
|
||||||
// ... agent runs ...
|
|
||||||
|
|
||||||
// At the end of each session
|
|
||||||
mem.tickSession(); // decay activation weights
|
|
||||||
const stats = mem.runConsolidation(Date.now() / 1000); // promote memories
|
|
||||||
console.log('[memory] consolidation:', stats);
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Rollback
|
|
||||||
|
|
||||||
If you need to roll back to sqlite-vec:
|
|
||||||
|
|
||||||
1. Change `memory.backend` back to `"sqlite-vec"` in your config.
|
|
||||||
2. The original `.md` files are unchanged (if you kept them).
|
|
||||||
3. Delete `agent.brain` (and `agent.brain.wal` if present).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### `Error: no records found for path: memory/user.md`
|
|
||||||
The path passed to `get()` or `exportMarkdown()` must exactly match the
|
|
||||||
relative path used during `ingestMarkdown()`. Check for leading `./`
|
|
||||||
differences.
|
|
||||||
|
|
||||||
### Memory is empty after reopening
|
|
||||||
Make sure `flushWal()` was called after bulk writes. Without it, entries
|
|
||||||
remain in the WAL and may be lost if the process exits abnormally.
|
|
||||||
|
|
||||||
### Embedding dimension mismatch
|
|
||||||
The `embeddingDim` passed to `create()` cannot be changed after the file is
|
|
||||||
created. If you switch embedding models, create a new `.brain` file and
|
|
||||||
re-run the migration script.
|
|
||||||
@@ -1,144 +0,0 @@
|
|||||||
# OpenClaw × clawhdf5 Configuration Reference
|
|
||||||
|
|
||||||
This document describes the full configuration schema for integrating
|
|
||||||
`clawhdf5` as the memory backend in an OpenClaw agent gateway.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Minimal example
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"memory": {
|
|
||||||
"backend": "clawhdf5",
|
|
||||||
"clawhdf5": {
|
|
||||||
"path": "./agent.brain",
|
|
||||||
"embeddingDim": 768
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Full schema
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"memory": {
|
|
||||||
"backend": "clawhdf5",
|
|
||||||
"clawhdf5": {
|
|
||||||
"path": "./agent.brain",
|
|
||||||
"embeddingDim": 768,
|
|
||||||
"walEnabled": true,
|
|
||||||
"walMaxEntries": 500,
|
|
||||||
"consolidation": {
|
|
||||||
"workingCapacity": 100,
|
|
||||||
"episodicCapacity": 10000,
|
|
||||||
"episodicHalfLifeDays": 7,
|
|
||||||
"semanticHalfLifeDays": 30,
|
|
||||||
"promotionThreshold": 0.6,
|
|
||||||
"semanticAccessThreshold": 10
|
|
||||||
},
|
|
||||||
"compaction": {
|
|
||||||
"autoCompactThreshold": 0.3,
|
|
||||||
"tickOnSessionEnd": true,
|
|
||||||
"consolidateOnCompaction": true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Field reference
|
|
||||||
|
|
||||||
### Top level
|
|
||||||
|
|
||||||
| Field | Type | Default | Description |
|
|
||||||
|-------|------|---------|-------------|
|
|
||||||
| `memory.backend` | `string` | `"clawhdf5"` | Must be `"clawhdf5"` to activate this backend |
|
|
||||||
|
|
||||||
### `clawhdf5`
|
|
||||||
|
|
||||||
| Field | Type | Default | Description |
|
|
||||||
|-------|------|---------|-------------|
|
|
||||||
| `path` | `string` | `"./agent.brain"` | Filesystem path for the `.brain` (HDF5) file. Relative to the OpenClaw working directory. |
|
|
||||||
| `embeddingDim` | `number` | `768` | Dimension of the embedding vectors. Must match the embedder model. Common values: `384` (MiniLM), `768` (nomic-embed-text, BGE-base), `1536` (OpenAI text-embedding-3-small). |
|
|
||||||
| `walEnabled` | `boolean` | `true` | Enable the Write-Ahead Log for crash recovery. Disable only on read-only stores or when crash safety is not required. |
|
|
||||||
| `walMaxEntries` | `number` | `500` | Number of WAL entries to accumulate before an automatic merge to the .h5 file. Lower values = more frequent flushes (safer, slightly slower). |
|
|
||||||
|
|
||||||
### `clawhdf5.consolidation`
|
|
||||||
|
|
||||||
Controls the hippocampal three-tier memory engine (Working → Episodic →
|
|
||||||
Semantic).
|
|
||||||
|
|
||||||
| Field | Type | Default | Description |
|
|
||||||
|-------|------|---------|-------------|
|
|
||||||
| `workingCapacity` | `number` | `100` | Maximum records in the Working tier before lowest-decay entries are evicted. |
|
|
||||||
| `episodicCapacity` | `number` | `10000` | Maximum records in the Episodic tier. |
|
|
||||||
| `episodicHalfLifeDays` | `number` | `7` | Half-life (in days) for exponential decay of Episodic records. Records not accessed within roughly one half-life drop in importance. |
|
|
||||||
| `semanticHalfLifeDays` | `number` | `30` | Half-life for Semantic records. Longer than Episodic — semantic knowledge decays slowly. |
|
|
||||||
| `promotionThreshold` | `number` | `0.6` | Importance score (0–1) above which a Working record is promoted to the Episodic tier. Higher = more selective. |
|
|
||||||
| `semanticAccessThreshold` | `number` | `10` | Minimum access count for an Episodic record to be promoted to Semantic. |
|
|
||||||
|
|
||||||
### `clawhdf5.compaction`
|
|
||||||
|
|
||||||
| Field | Type | Default | Description |
|
|
||||||
|-------|------|---------|-------------|
|
|
||||||
| `autoCompactThreshold` | `number` | `0.3` | Fraction of tombstoned records (0–1) that triggers automatic compaction. `0.3` = compact when 30% of records are deleted. Set to `0` to disable auto-compact. |
|
|
||||||
| `tickOnSessionEnd` | `boolean` | `true` | Run `tickSession()` (Hebbian decay) automatically when the agent session closes. |
|
|
||||||
| `consolidateOnCompaction` | `boolean` | `true` | Run the hippocampal consolidation engine after each compaction cycle. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Embedder compatibility
|
|
||||||
|
|
||||||
The `embeddingDim` must remain constant for the lifetime of a `.brain` file.
|
|
||||||
Mixing embedding models in the same file is not supported.
|
|
||||||
|
|
||||||
| Embedder | `embeddingDim` |
|
|
||||||
|----------|---------------|
|
|
||||||
| `all-MiniLM-L6-v2` | `384` |
|
|
||||||
| `nomic-embed-text` | `768` |
|
|
||||||
| `BGE-base-en-v1.5` | `768` |
|
|
||||||
| `OpenAI text-embedding-3-small` | `1536` |
|
|
||||||
| `OpenAI text-embedding-3-large` | `3072` |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## OpenClaw integration code
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
|
||||||
|
|
||||||
// Load config from your OpenClaw config file
|
|
||||||
const cfg = loadConfig(); // your config loading logic
|
|
||||||
|
|
||||||
const mem = ClawhdfMemory.openOrCreate(
|
|
||||||
cfg.memory.clawhdf5.path,
|
|
||||||
cfg.memory.clawhdf5.embeddingDim ?? 768,
|
|
||||||
);
|
|
||||||
|
|
||||||
// On session end
|
|
||||||
if (cfg.memory.clawhdf5.compaction?.tickOnSessionEnd) {
|
|
||||||
mem.tickSession();
|
|
||||||
}
|
|
||||||
if (cfg.memory.clawhdf5.compaction?.consolidateOnCompaction) {
|
|
||||||
const stats = mem.runConsolidation(Date.now() / 1000);
|
|
||||||
console.log('[clawhdf5] consolidation:', stats);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Environment variables
|
|
||||||
|
|
||||||
The following environment variables override config file values when set:
|
|
||||||
|
|
||||||
| Variable | Overrides |
|
|
||||||
|----------|-----------|
|
|
||||||
| `CLAWHDF5_PATH` | `clawhdf5.path` |
|
|
||||||
| `CLAWHDF5_EMBEDDING_DIM` | `clawhdf5.embeddingDim` |
|
|
||||||
| `CLAWHDF5_WAL_ENABLED` | `clawhdf5.walEnabled` (`"true"` / `"false"`) |
|
|
||||||
@@ -1,337 +0,0 @@
|
|||||||
# OpenClaw × clawhdf5 Integration
|
|
||||||
|
|
||||||
clawhdf5 provides a drop-in HDF5-backed memory backend for the
|
|
||||||
[OpenClaw](https://github.com/redclawsystems/openclaw) agent gateway.
|
|
||||||
This document covers architecture, the full Node.js API reference, and code
|
|
||||||
examples for common operations.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
|
||||||
│ OpenClaw (Node.js/TypeScript) │
|
|
||||||
│ │
|
|
||||||
│ ┌─────────────────┐ ┌──────────────────────────────┐ │
|
|
||||||
│ │ Agent runtime │───▶│ @redclaw/clawhdf5 (Node.js) │ │
|
|
||||||
│ └─────────────────┘ │ TypeScript wrapper │ │
|
|
||||||
│ └──────────────┬─────────────┘ │
|
|
||||||
│ │ napi-rs FFI │
|
|
||||||
└────────────────────────────────────────┼────────────────────┘
|
|
||||||
│
|
|
||||||
┌────────────────────────────────────────▼────────────────────┐
|
|
||||||
│ clawhdf5-napi (Rust, cdylib) │
|
|
||||||
│ │
|
|
||||||
│ ClawhdfMemory ──▶ ClawhdfBackend ──▶ HDF5Memory │
|
|
||||||
│ MemoryBackend ├─ MemoryCache │
|
|
||||||
│ trait impl ├─ WalFile │
|
|
||||||
│ ├─ SessionCache │
|
|
||||||
│ └─ KnowledgeCache │
|
|
||||||
│ │
|
|
||||||
│ ConsolidationEngine (hippocampal tiers) │
|
|
||||||
│ Working (100) ──▶ Episodic (10k) ──▶ Semantic (∞) │
|
|
||||||
└─────────────────────────────────────────────────────────────┘
|
|
||||||
│
|
|
||||||
┌────────────────▼────────────┐
|
|
||||||
│ agent.brain (HDF5 file) │
|
|
||||||
│ agent.brain.wal (WAL log) │
|
|
||||||
└─────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
### Key design decisions
|
|
||||||
|
|
||||||
- **Single file**: everything lives in one `.brain` HDF5 file (+ WAL sidecar).
|
|
||||||
- **In-memory cache**: the full embedding matrix and chunk list are loaded into RAM for fast search.
|
|
||||||
- **Hybrid search**: vector similarity (70%) and BM25 full-text (30%) are blended with Reciprocal Rank Fusion (RRF), then re-ranked by Hebbian activation weight and temporal recency.
|
|
||||||
- **Hippocampal tiers**: records are classified as Working, Episodic, or Semantic based on importance and access frequency. Tier promotion and eviction happen during `runConsolidation()`.
|
|
||||||
- **WAL**: writes are journaled before hitting the .h5 file. On crash, the WAL is replayed at next open.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Installation
|
|
||||||
|
|
||||||
```bash
|
|
||||||
npm install @redclaw/clawhdf5
|
|
||||||
```
|
|
||||||
|
|
||||||
See [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md)
|
|
||||||
for build-from-source instructions.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Node.js API reference
|
|
||||||
|
|
||||||
### `ClawhdfMemory` (class)
|
|
||||||
|
|
||||||
All instance methods are synchronous. The native Rust code is single-threaded
|
|
||||||
on the Node.js side; do **not** share a `ClawhdfMemory` instance across Worker
|
|
||||||
threads without external locking.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### Static factory methods
|
|
||||||
|
|
||||||
##### `ClawhdfMemory.create(path: string, embeddingDim: number): ClawhdfMemory`
|
|
||||||
|
|
||||||
Create a new `.brain` file. Throws if the file already exists.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const mem = ClawhdfMemory.create('./agent.brain', 768);
|
|
||||||
```
|
|
||||||
|
|
||||||
##### `ClawhdfMemory.open(path: string): ClawhdfMemory`
|
|
||||||
|
|
||||||
Open an existing file. Replays the WAL automatically.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const mem = ClawhdfMemory.open('./agent.brain');
|
|
||||||
```
|
|
||||||
|
|
||||||
##### `ClawhdfMemory.openOrCreate(path: string, embeddingDim: number): ClawhdfMemory`
|
|
||||||
|
|
||||||
**Recommended entry point.** Opens if the file exists, otherwise creates it.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `search(queryText, queryEmbedding, k): MemorySearchResult[]`
|
|
||||||
|
|
||||||
Hybrid BM25 + vector search.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const embedding = new Float32Array(await embed(query));
|
|
||||||
const results = mem.search(query, embedding, 10);
|
|
||||||
for (const r of results) {
|
|
||||||
console.log(r.score.toFixed(3), r.path, r.text.slice(0, 80));
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Pass an empty `Float32Array` to use BM25 only (no vector similarity).
|
|
||||||
|
|
||||||
**Parameters:**
|
|
||||||
- `queryText: string` — used for BM25 term matching
|
|
||||||
- `queryEmbedding: Float32Array` — dense vector of length `embeddingDim`
|
|
||||||
- `k: number` — maximum results to return
|
|
||||||
|
|
||||||
**Returns:** `MemorySearchResult[]`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `get(path, fromLine?, numLines?): string | null`
|
|
||||||
|
|
||||||
Retrieve stored content by path.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const md = mem.get('memory/user.md'); // all content
|
|
||||||
const lines = mem.get('memory/user.md', 5, 10); // lines 5–14
|
|
||||||
const section = mem.get('memory/user.md::Goals'); // specific section
|
|
||||||
```
|
|
||||||
|
|
||||||
Section sub-paths use the `::heading` suffix produced by `ingestMarkdown`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `write(path, content): void`
|
|
||||||
|
|
||||||
Store raw content at `path`.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
mem.write('memory/session.md', '# Session\n\nWorking on task X.');
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `ingestMarkdown(path, content): number`
|
|
||||||
|
|
||||||
Parse `content` as Markdown, split on ATX headings, and store each section
|
|
||||||
separately. Returns the number of sections ingested.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { readFileSync } from 'fs';
|
|
||||||
const md = readFileSync('./memory/MEMORY.md', 'utf8');
|
|
||||||
const count = mem.ingestMarkdown('memory/MEMORY.md', md);
|
|
||||||
console.log(`Ingested ${count} sections`);
|
|
||||||
```
|
|
||||||
|
|
||||||
Sections are addressable as `memory/MEMORY.md::HeadingName`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `exportMarkdown(path): string`
|
|
||||||
|
|
||||||
Reconstruct stored sections for `path` back into a Markdown string.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const md = mem.exportMarkdown('memory/MEMORY.md');
|
|
||||||
writeFileSync('./memory/MEMORY.md', md);
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `stats(): BackendStats`
|
|
||||||
|
|
||||||
Return aggregate statistics.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const s = mem.stats();
|
|
||||||
console.log(`Records: ${s.totalRecords}, Size: ${s.fileSizeBytes} bytes`);
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `compact(): number`
|
|
||||||
|
|
||||||
Remove tombstoned records from the store. Returns count removed.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `tickSession(): void`
|
|
||||||
|
|
||||||
Apply Hebbian decay to all activation weights. Call at session end.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `flushWal(): void`
|
|
||||||
|
|
||||||
Force a WAL merge: flush `.h5` and truncate the WAL log.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `runConsolidation(nowSecs: number): ConsolidationStats`
|
|
||||||
|
|
||||||
Run one full hippocampal consolidation cycle.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const stats = mem.runConsolidation(Date.now() / 1000);
|
|
||||||
console.log(stats);
|
|
||||||
// { workingCount: 42, episodicCount: 310, semanticCount: 5,
|
|
||||||
// totalEvictions: 0, totalPromotions: 7 }
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
#### `walPendingCount(): number`
|
|
||||||
|
|
||||||
Number of pending WAL entries (0 if WAL is disabled).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Type reference
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
interface MemorySearchResult {
|
|
||||||
text: string;
|
|
||||||
score: number; // 0–1, higher = more relevant
|
|
||||||
path: string; // source file path
|
|
||||||
lineRange?: [number, number];
|
|
||||||
timestamp?: number; // Unix epoch seconds
|
|
||||||
source: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
interface BackendStats {
|
|
||||||
totalRecords: number;
|
|
||||||
totalEmbeddings: number;
|
|
||||||
fileSizeBytes: number;
|
|
||||||
modalities: string[]; // e.g. ["text"]
|
|
||||||
lastUpdated?: number; // Unix epoch seconds
|
|
||||||
}
|
|
||||||
|
|
||||||
interface ConsolidationStats {
|
|
||||||
workingCount: number;
|
|
||||||
episodicCount: number;
|
|
||||||
semanticCount: number;
|
|
||||||
totalEvictions: number;
|
|
||||||
totalPromotions: number;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Common patterns
|
|
||||||
|
|
||||||
### Session lifecycle
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
|
||||||
|
|
||||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
|
||||||
|
|
||||||
// --- agent session runs ---
|
|
||||||
|
|
||||||
// On session end: decay + consolidate
|
|
||||||
mem.tickSession();
|
|
||||||
const consolidationStats = mem.runConsolidation(Date.now() / 1000);
|
|
||||||
console.log('[memory] consolidation:', consolidationStats);
|
|
||||||
```
|
|
||||||
|
|
||||||
### Ingest all memory files at startup
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { readdirSync, readFileSync, statSync } from 'fs';
|
|
||||||
import { join, relative } from 'path';
|
|
||||||
|
|
||||||
function ingestDirectory(mem: ClawhdfMemory, dir: string): void {
|
|
||||||
for (const entry of readdirSync(dir)) {
|
|
||||||
const full = join(dir, entry);
|
|
||||||
if (statSync(full).isDirectory()) {
|
|
||||||
ingestDirectory(mem, full);
|
|
||||||
} else if (entry.endsWith('.md')) {
|
|
||||||
const content = readFileSync(full, 'utf8');
|
|
||||||
const path = relative(process.cwd(), full);
|
|
||||||
mem.ingestMarkdown(path, content);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
mem.flushWal();
|
|
||||||
}
|
|
||||||
|
|
||||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
|
||||||
ingestDirectory(mem, './memory');
|
|
||||||
```
|
|
||||||
|
|
||||||
### Search with real embeddings
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import OpenAI from 'openai';
|
|
||||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
|
||||||
|
|
||||||
const ai = new OpenAI();
|
|
||||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 1536);
|
|
||||||
|
|
||||||
async function searchMemory(query: string, k = 5) {
|
|
||||||
const resp = await ai.embeddings.create({
|
|
||||||
model: 'text-embedding-3-small',
|
|
||||||
input: query,
|
|
||||||
});
|
|
||||||
const embedding = new Float32Array(resp.data[0].embedding);
|
|
||||||
return mem.search(query, embedding, k);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Error handling
|
|
||||||
|
|
||||||
All methods that can fail throw a `NapiError` (a standard JS `Error` subclass)
|
|
||||||
with the Rust error message as `message`.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
try {
|
|
||||||
const md = mem.exportMarkdown('nonexistent.md');
|
|
||||||
} catch (e) {
|
|
||||||
console.error('Export failed:', (e as Error).message);
|
|
||||||
// "no records found for path: nonexistent.md"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## See also
|
|
||||||
|
|
||||||
- [openclaw-config.md](openclaw-config.md) — Full configuration schema
|
|
||||||
- [migration-guide.md](migration-guide.md) — Migrating from sqlite-vec
|
|
||||||
- [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md) — Build instructions
|
|
||||||
- [BENCHMARKS.md](../BENCHMARKS.md) — Performance results
|
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
# OpenClaw: not supported
|
||||||
|
|
||||||
|
**clawhdf5 does not currently work as an [OpenClaw](https://docs.openclaw.ai)
|
||||||
|
memory backend, and never has.** Earlier versions of these docs described a
|
||||||
|
"drop-in" backend enabled with `memory.backend = "clawhdf5"`. That
|
||||||
|
configuration was never valid: from v2026.2 through v2026.7 OpenClaw's
|
||||||
|
`memory.backend` accepted only `"builtin"` or `"qmd"` and rejected unknown
|
||||||
|
keys, and since v2026.8.1 ("OpenClaw 2.0") the key no longer exists. A Gateway
|
||||||
|
given that config refuses to start. No plugin was ever built or tested against
|
||||||
|
OpenClaw, and the `@redclaw/clawhdf5` npm package was never published.
|
||||||
|
|
||||||
|
As of 2026-09-25 we are not pursuing an OpenClaw plugin; the maintained
|
||||||
|
integration target is ZeroClaw. This page records what a plugin would need,
|
||||||
|
for when that changes.
|
||||||
|
|
||||||
|
## What OpenClaw expects today (v2026.9.6)
|
||||||
|
|
||||||
|
Checked against the OpenClaw source at tag `v2026.9.6` and its docs on
|
||||||
|
2026-09-25. OpenClaw marks every plugin API as experimental, so re-check before
|
||||||
|
building anything.
|
||||||
|
|
||||||
|
- **Memory lives in Markdown files**, which are the source of truth: `MEMORY.md`,
|
||||||
|
`USER.md`, daily notes in `memory/YYYY-MM-DD.md` in the agent workspace. The
|
||||||
|
memory engine is an index over them
|
||||||
|
([concepts/memory](https://docs.openclaw.ai/concepts/memory)).
|
||||||
|
- **A memory plugin is selected with `plugins.slots.memory: "<plugin-id>"`**
|
||||||
|
(default `memory-core`), its settings under
|
||||||
|
`plugins.entries.<plugin-id>.config`, validated against the plugin's own
|
||||||
|
schema ([gateway/config-extensions](https://docs.openclaw.ai/gateway/config-extensions)).
|
||||||
|
Memory search settings are under `memory.search`
|
||||||
|
([reference/memory-config](https://docs.openclaw.ai/reference/memory-config)).
|
||||||
|
- **A plugin needs** an `openclaw.plugin.json` manifest with `id`,
|
||||||
|
`configSchema`, `"kind": "memory"` and every tool listed in `contracts.tools`
|
||||||
|
([plugins/manifest](https://docs.openclaw.ai/plugins/manifest)); a
|
||||||
|
`package.json` with `openclaw.extensions`, `openclaw.compat.pluginApi` and an
|
||||||
|
`openclaw` peer dependency; and an entry built with `definePluginEntry`.
|
||||||
|
- **Two ways to integrate** (both exist upstream): tools only, as
|
||||||
|
`memory-lancedb` does (`api.registerTool`), or a full memory engine, as
|
||||||
|
`memory-core` does, through `api.registerMemoryCapability({ runtime, ... })`,
|
||||||
|
whose runtime returns a `MemorySearchManager` implementing `search`,
|
||||||
|
`readFile` (returning `status: "ok" | "not_found"`), `status`,
|
||||||
|
`probeEmbeddingAvailability` and `probeVectorAvailability`. Active Memory
|
||||||
|
expects `memory_search` and `memory_get` tools
|
||||||
|
([plugins/sdk-overview/memory-and-context](https://docs.openclaw.ai/plugins/sdk-overview/memory-and-context)).
|
||||||
|
- **Embeddings come from OpenClaw's providers** (`memory.search.provider`), or a
|
||||||
|
plugin registers one with `api.registerEmbeddingProvider`.
|
||||||
|
- **Native code**: plugin installs run with `--ignore-scripts`, so a napi addon
|
||||||
|
has to ship as prebuilt per-platform packages (the pattern `memory-lancedb`
|
||||||
|
uses for LanceDB), loaded lazily
|
||||||
|
([plugins/dependency-resolution](https://docs.openclaw.ai/plugins/dependency-resolution)).
|
||||||
|
- **Distribution**: `openclaw plugins install` from npm or ClawHub; a first
|
||||||
|
install from an arbitrary source needs explicit review, and community ClawHub
|
||||||
|
packages go through a security audit.
|
||||||
|
- **Churn to plan for**: the memory SDK was reshaped in 2026 (separate
|
||||||
|
registration functions merged into `registerMemoryCapability`;
|
||||||
|
`registerMemoryEmbeddingProvider` removed on 2026-08-21), and further SDK
|
||||||
|
surfaces become eligible for removal on 2026-10-01
|
||||||
|
([plugins/sdk-migration/removal-timeline](https://docs.openclaw.ai/plugins/sdk-migration/removal-timeline)).
|
||||||
|
|
||||||
|
## What this repository has
|
||||||
|
|
||||||
|
Building blocks, usable as a library today, but not an OpenClaw plugin:
|
||||||
|
|
||||||
|
- `clawhdf5_agent::openclaw::ClawhdfBackend` — a Markdown-oriented backend over
|
||||||
|
`HDF5Memory`: ingest Markdown by section, hybrid search with re-ranking and
|
||||||
|
confidence rejection, read back by path, export. Gaps a plugin would have to
|
||||||
|
close: `write`/`ingest_markdown` store no embeddings (search is keyword-only
|
||||||
|
for that content unless records are saved with `save_entry`), re-ingesting
|
||||||
|
appends rather than replaces, there is no delete, `line_range` is never set,
|
||||||
|
and export rewrites every heading as `##`.
|
||||||
|
- `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a
|
||||||
|
TypeScript wrapper. **Not published, not built or tested in CI, and known to
|
||||||
|
be broken**; see `docs/known-issues.md`.
|
||||||
@@ -5,20 +5,20 @@ HDF5-backed agent memory system with hippocampal consolidation.
|
|||||||
|
|
||||||
Built with [napi-rs](https://napi.rs).
|
Built with [napi-rs](https://napi.rs).
|
||||||
|
|
||||||
|
> **Status: unpublished and known to be broken.** This package is not on npm,
|
||||||
|
> no binaries are built, nothing in CI builds or tests it, and the wrapper
|
||||||
|
> reads field names the native layer does not produce. It is not an OpenClaw
|
||||||
|
> plugin. See [known issues](../../docs/known-issues.md) and
|
||||||
|
> [docs/openclaw.md](../../docs/openclaw.md) before using it.
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
|
Not published. Building from source needs `@napi-rs/cli`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm install @redclaw/clawhdf5
|
npm install && npm run build
|
||||||
```
|
```
|
||||||
|
|
||||||
Pre-built binaries are published for:
|
|
||||||
|
|
||||||
| Platform | Architecture |
|
|
||||||
|----------|-------------|
|
|
||||||
| Linux (glibc) | x64, aarch64 |
|
|
||||||
| macOS | x64, aarch64 (Apple Silicon) |
|
|
||||||
| Windows | x64 |
|
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
```ts
|
```ts
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "@redclaw/clawhdf5",
|
"name": "@redclaw/clawhdf5",
|
||||||
"version": "2.7.0",
|
"version": "2.7.0",
|
||||||
"description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
|
"description": "Node.js bindings for clawhdf5 \u2014 HDF5-backed agent memory with hippocampal consolidation",
|
||||||
"main": "index.js",
|
"main": "index.js",
|
||||||
"types": "index.d.ts",
|
"types": "index.d.ts",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
@@ -14,8 +14,7 @@
|
|||||||
"memory",
|
"memory",
|
||||||
"hdf5",
|
"hdf5",
|
||||||
"vector-search",
|
"vector-search",
|
||||||
"embedding",
|
"embedding"
|
||||||
"openclaw"
|
|
||||||
],
|
],
|
||||||
"napi": {
|
"napi": {
|
||||||
"name": "clawhdf5",
|
"name": "clawhdf5",
|
||||||
@@ -51,5 +50,6 @@
|
|||||||
},
|
},
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">= 16"
|
"node": ">= 16"
|
||||||
}
|
},
|
||||||
|
"private": true
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user