docs: withdraw the OpenClaw integration claims
The docs described a "drop-in" OpenClaw memory backend enabled with `memory.backend = "clawhdf5"`. Checked against OpenClaw's source and docs (v2026.2.26 through v2026.9.6): that config was never valid — v2026.2-v2026.7 accepted only "builtin"/"qmd" and rejected unknown keys, so a Gateway given it refuses to start, and v2026.8.1 (OpenClaw 2.0) removed the key. No plugin was ever built (no manifest, no registration, no tools), nothing was tested against OpenClaw, the linked github.com/redclawsystems/openclaw is a 404, and @redclaw/clawhdf5 was never published. Decision (2026-09-25): not pursuing an OpenClaw plugin for now; ZeroClaw is the integration target. - Remove openclaw-integration.md, openclaw-config.md and migration-guide.md; add docs/openclaw.md: the status, what a memory plugin needs against v2026.9.6 (plugins.slots.memory, manifest with kind "memory", registerMemoryCapability / MemorySearchManager, prebuilt native packages), and what this repo has as building blocks. - README, QUICKSTART, USE_CASES, ROADMAP (Track 7 withdrawn), CLAUDE.md and the `openclaw` module docs describe ClawhdfBackend as what it is: a Markdown-oriented library backend, not an OpenClaw plugin. The QUICKSTART example is corrected (the old one called a three-argument create that does not exist) and states its limits. - packages/clawhdf5-node: marked unpublished and broken, "private": true so it cannot be published by accident; its bugs (snake_case vs camelCase fields, wrong addon path, no way to store an embedding, wrong WAL name) are recorded in docs/known-issues.md. - Two broken rustdoc links fixed along the way. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
+19
-61
@@ -11,7 +11,7 @@ ClawhDF5 serves three audiences with different entry points:
|
||||
| You Are | You Want | Start Here |
|
||||
|---------|----------|------------|
|
||||
| **AI agent developer** | Persistent memory for your agent | [Agent Memory (Rust)](#1-agent-memory-rust-library) |
|
||||
| **OpenClaw user** | Better memory for your OpenClaw agent | [OpenClaw Integration](#2-openclaw-integration) |
|
||||
| **OpenClaw user** | clawhdf5 is not an OpenClaw memory plugin | [Status](openclaw.md) |
|
||||
| **Data scientist** | Read/write HDF5 files in Rust | [HDF5 File I/O](#3-hdf5-file-io) |
|
||||
| **CLI user** | Inspect and manage agent memories | [CLI Tool](#4-cli-tool) |
|
||||
| **Python user** | Use clawhdf5 from Python | [Python Bindings](#5-python-bindings) |
|
||||
@@ -197,80 +197,38 @@ if let Some(alert) = detector.check_rate_anomaly() {
|
||||
|
||||
---
|
||||
|
||||
## 2. OpenClaw Integration
|
||||
## 2. Markdown Memory (and OpenClaw)
|
||||
|
||||
ClawhDF5 can serve as the memory backend for [OpenClaw](https://docs.openclaw.ai) agents, replacing the default Markdown + sqlite-vec approach.
|
||||
**clawhdf5 is not an OpenClaw memory backend.** Earlier versions of this guide
|
||||
described one; it never worked — see [openclaw.md](openclaw.md) for what
|
||||
happened and what a real plugin would need.
|
||||
|
||||
### How It Works
|
||||
|
||||
```
|
||||
OpenClaw Agent
|
||||
│
|
||||
├── memory_search("user preferences")
|
||||
│ │
|
||||
│ └── ClawhdfBackend
|
||||
│ ├── Vector search (cosine)
|
||||
│ ├── BM25 keyword search
|
||||
│ ├── Reciprocal Rank Fusion
|
||||
│ ├── Multi-factor re-ranking
|
||||
│ └── Low-confidence rejection
|
||||
│
|
||||
└── agent_memory.h5 (single file, portable)
|
||||
```
|
||||
|
||||
### Migration from Markdown
|
||||
What does exist is `ClawhdfBackend`, a library API that ingests Markdown files
|
||||
by section and searches them with the full pipeline (hybrid retrieval,
|
||||
re-ranking, confidence rejection):
|
||||
|
||||
```rust
|
||||
use clawhdf5_agent::openclaw::*;
|
||||
use std::path::Path;
|
||||
|
||||
// Create a new HDF5 backend
|
||||
let mut backend = ClawhdfBackend::create("memory.h5", "my-agent", 384)?;
|
||||
let mut backend = ClawhdfBackend::create(Path::new("memory.h5"), 384)?;
|
||||
|
||||
// Import your existing MEMORY.md
|
||||
let md = std::fs::read_to_string("~/.openclaw/workspace/MEMORY.md")?;
|
||||
// Each heading becomes a record, stored under "MEMORY.md::<heading>".
|
||||
let md = std::fs::read_to_string("MEMORY.md")?;
|
||||
let count = backend.ingest_markdown("MEMORY.md", &md)?;
|
||||
println!("Imported {} sections", count);
|
||||
println!("Imported {count} sections");
|
||||
|
||||
// Import daily logs
|
||||
for entry in std::fs::read_dir("~/.openclaw/workspace/memory/")? {
|
||||
let path = entry?.path();
|
||||
if path.extension().map(|e| e == "md").unwrap_or(false) {
|
||||
let content = std::fs::read_to_string(&path)?;
|
||||
let name = path.file_name().unwrap().to_string_lossy();
|
||||
backend.ingest_markdown(&name, &content)?;
|
||||
}
|
||||
}
|
||||
|
||||
// Search using the full pipeline
|
||||
let results = backend.search("what are user preferences", &query_embedding, 5);
|
||||
for r in &results {
|
||||
println!("[{:.3}] {} (from {})", r.score, r.text, r.path);
|
||||
}
|
||||
|
||||
// Export back to Markdown (lossless roundtrip)
|
||||
let exported = backend.export_markdown("MEMORY.md")?;
|
||||
```
|
||||
|
||||
### What You Get Over sqlite-vec
|
||||
|
||||
| Feature | sqlite-vec | ClawhDF5 |
|
||||
|---------|-----------|----------|
|
||||
| Vector search | ✅ | ✅ (8× faster at 100K) |
|
||||
| Keyword search | ❌ | ✅ BM25 |
|
||||
| Hybrid fusion | ❌ | ✅ RRF |
|
||||
| Re-ranking | ❌ | ✅ Multi-factor |
|
||||
| Confidence rejection | ❌ | ✅ |
|
||||
| Knowledge graph | ❌ | ✅ |
|
||||
| Memory consolidation | ❌ | ✅ |
|
||||
| Temporal queries | ❌ | ✅ (716ns) |
|
||||
| Anomaly detection | ❌ | ✅ |
|
||||
| Provenance tracking | ❌ | ✅ |
|
||||
| Multi-modal | ❌ | ✅ |
|
||||
| Single portable file | ❌ (SQLite + MD files) | ✅ |
|
||||
|
||||
### Future: Native OpenClaw Plugin
|
||||
|
||||
The Phase 2 roadmap includes a native OpenClaw plugin (`memory.backend = "clawhdf5"`) that transparently replaces sqlite-vec. Until then, the Rust library can be wrapped via NAPI or used from the CLI.
|
||||
Limits to know: sections ingested this way carry no embedding (search over them
|
||||
is keyword-only unless you save records with vectors via `save_entry`);
|
||||
ingesting the same file again adds the sections again rather than replacing
|
||||
them; and `export_markdown` rewrites every heading as `##`, so it is not a
|
||||
lossless round trip.
|
||||
|
||||
---
|
||||
|
||||
@@ -547,7 +505,7 @@ let final_results = confidence::reject_low_confidence(
|
||||
|
||||
**Why not a vector database?** Pinecone, Qdrant, Weaviate — they're cloud services or heavy servers. Agent memory should be local, portable, and zero-dependency. An agent's memories should travel with it.
|
||||
|
||||
**Why not Markdown?** OpenClaw uses Markdown today and it works for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't.
|
||||
**Why not Markdown?** Plain Markdown files work for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't.
|
||||
|
||||
**Why HDF5 specifically?**
|
||||
- Native N-dimensional array storage (perfect for embeddings)
|
||||
|
||||
+3
-30
@@ -42,37 +42,10 @@ conversation → embedding → save to agent.h5
|
||||
|
||||
---
|
||||
|
||||
## 2. OpenClaw Memory Upgrade
|
||||
## 2. OpenClaw
|
||||
|
||||
**Scenario:** You run OpenClaw and the default Markdown + sqlite-vec memory works OK for simple recall but falls short on complex queries like "what did we decide about the deployment architecture last Tuesday?" or "who's responsible for the billing system?"
|
||||
|
||||
**Problem:** Markdown files have no semantic structure. sqlite-vec does flat vector search — no keyword fusion, no re-ranking, no temporal reasoning, no knowledge graph.
|
||||
|
||||
**ClawhDF5 solution:**
|
||||
|
||||
```bash
|
||||
# Migrate existing memories
|
||||
clawhdf5 --path memory.h5 create --agent-id openclaw --dim 384
|
||||
|
||||
# Import your MEMORY.md and daily logs
|
||||
# (programmatically via ClawhdfBackend::ingest_markdown)
|
||||
```
|
||||
|
||||
Then in your OpenClaw config (future):
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"backend": "clawhdf5",
|
||||
"path": "~/.openclaw/agents/main/memory.h5"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**What changes:**
|
||||
- "What did we discuss last Tuesday?" → temporal index finds the session, returns memories from that time range
|
||||
- "Who owns the billing system?" → knowledge graph traversal: billing_system → owned_by → Alice
|
||||
- "Preferences about deployment" → hybrid search (vector + BM25) finds relevant memories even with different wording
|
||||
- Bad search results get filtered out by confidence rejection instead of confusing the agent
|
||||
Not supported: clawhdf5 is not an OpenClaw memory plugin, and the config this
|
||||
section used to show was never valid. See [openclaw.md](openclaw.md).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -279,3 +279,26 @@ the same agent-store interop test.
|
||||
|
||||
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
||||
which is what libhdf5 itself writes.
|
||||
|
||||
## The Node.js package (`packages/clawhdf5-node`) does not work
|
||||
|
||||
**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
|
||||
|
||||
The TypeScript wrapper over `crates/clawhdf5-napi` has never run successfully:
|
||||
|
||||
- napi-rs converts `#[napi(object)]` fields to camelCase, but the wrapper reads
|
||||
snake_case (`r.line_range`, `s.total_records`, `s.working_count`, …), so
|
||||
every stats and consolidation field comes back `undefined`
|
||||
(`src/index.ts:76-120`).
|
||||
- It loads `../clawhdf5.node`, but `napi build --platform` produces
|
||||
`clawhdf5.<triple>.node`; `main` points at `index.js` while `tsc` writes to
|
||||
`dist/`; `napi prepublish` expects per-platform packages that are not
|
||||
defined.
|
||||
- `save`/`saveBatch` exist in the napi layer but not in the wrapper, so a
|
||||
TypeScript caller cannot store an embedding at all.
|
||||
- The WAL for `agent.brain` is `agent.h5.wal` (the store uses
|
||||
`with_extension("h5.wal")`), not `agent.brain.wal` as the old docs and the
|
||||
test cleanup assume.
|
||||
|
||||
It was written for an OpenClaw integration that is not being pursued (see
|
||||
`docs/openclaw.md`). Fix and add CI, or remove it, before anyone depends on it.
|
||||
|
||||
@@ -1,213 +0,0 @@
|
||||
# Migration Guide: OpenClaw sqlite-vec → clawhdf5
|
||||
|
||||
This guide walks through migrating an OpenClaw agent from its default
|
||||
sqlite-vec + Markdown file memory to the `clawhdf5` HDF5 backend.
|
||||
|
||||
---
|
||||
|
||||
## Why migrate?
|
||||
|
||||
| Feature | sqlite-vec + Markdown | clawhdf5 |
|
||||
|---------|----------------------|----------|
|
||||
| Storage format | SQLite WAL + flat .md files | Single HDF5 binary file |
|
||||
| Vector search | sqlite-vec (SQLite extension) | Pure-Rust SIMD (clawhdf5-accel) |
|
||||
| Full-text search | External (FTS5 or plain string match) | Built-in BM25 |
|
||||
| Hybrid search | Manual combination | Automatic RRF blend |
|
||||
| Memory tiers | Flat | Working → Episodic → Semantic |
|
||||
| Hebbian decay | Not built-in | Automatic activation weighting |
|
||||
| Portability | SQLite binary required | Zero native deps (all Rust) |
|
||||
| Crash recovery | SQLite WAL | clawhdf5 WAL |
|
||||
| Compaction | Manual | Auto-threshold + session-end |
|
||||
| Embedding dim change | New DB required | New file required (same) |
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Install `@redclaw/clawhdf5`
|
||||
|
||||
```bash
|
||||
npm install @redclaw/clawhdf5
|
||||
```
|
||||
|
||||
Or, if building from the monorepo source:
|
||||
|
||||
```bash
|
||||
npm install -g @napi-rs/cli
|
||||
cd packages/clawhdf5-node
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Update your OpenClaw config
|
||||
|
||||
Change `backend` from `"sqlite-vec"` (or `"markdown"`) to `"clawhdf5"`:
|
||||
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"backend": "clawhdf5",
|
||||
"clawhdf5": {
|
||||
"path": "./agent.brain",
|
||||
"embeddingDim": 768
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See [openclaw-config.md](openclaw-config.md) for the full schema.
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Run the one-time migration
|
||||
|
||||
clawhdf5 ships a migration helper that reads your existing Markdown memory
|
||||
files and ingests them via `ingestMarkdown()`.
|
||||
|
||||
### Automated migration script
|
||||
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
import { readFileSync, readdirSync, statSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
|
||||
async function migrate(
|
||||
memoryDir: string,
|
||||
brainPath: string,
|
||||
embeddingDim: number = 768,
|
||||
): Promise<void> {
|
||||
const mem = ClawhdfMemory.create(brainPath, embeddingDim);
|
||||
|
||||
// Walk all .md files under memoryDir
|
||||
function walk(dir: string): string[] {
|
||||
return readdirSync(dir).flatMap((entry) => {
|
||||
const full = join(dir, entry);
|
||||
return statSync(full).isDirectory() ? walk(full) : [full];
|
||||
});
|
||||
}
|
||||
|
||||
const files = walk(memoryDir).filter((f) => f.endsWith('.md'));
|
||||
let totalSections = 0;
|
||||
|
||||
for (const file of files) {
|
||||
const content = readFileSync(file, 'utf8');
|
||||
const relPath = relative(process.cwd(), file);
|
||||
const count = mem.ingestMarkdown(relPath, content);
|
||||
console.log(` ${relPath}: ${count} sections`);
|
||||
totalSections += count;
|
||||
}
|
||||
|
||||
// Force WAL merge after bulk import
|
||||
mem.flushWal();
|
||||
|
||||
console.log(`\nMigration complete: ${files.length} files, ${totalSections} sections`);
|
||||
const s = mem.stats();
|
||||
console.log(` Total records: ${s.totalRecords}`);
|
||||
console.log(` File size: ${(s.fileSizeBytes / 1024).toFixed(1)} KB`);
|
||||
}
|
||||
|
||||
// Usage
|
||||
migrate('./memory', './agent.brain', 768).catch(console.error);
|
||||
```
|
||||
|
||||
### What the migration does
|
||||
|
||||
1. Walks all `.md` files under your memory directory.
|
||||
2. Parses each file into sections using the same `MarkdownParser` used by
|
||||
OpenClaw (splits on ATX headings `#`, `##`, `###`, …).
|
||||
3. Stores each section as a separate record in the HDF5 file with the file
|
||||
path as the `source_channel` (e.g. `memory/user.md::Goals`).
|
||||
4. Flushes the WAL to merge everything into the `.brain` file.
|
||||
|
||||
After migration, **the original `.md` files are not modified or deleted**.
|
||||
You can keep them as a backup or remove them once you have verified the
|
||||
migrated data.
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Verify
|
||||
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
const mem = ClawhdfMemory.open('./agent.brain');
|
||||
const s = mem.stats();
|
||||
console.log('Records after migration:', s.totalRecords);
|
||||
|
||||
// Spot-check: retrieve a known path
|
||||
const userMd = mem.get('memory/MEMORY.md');
|
||||
console.log(userMd?.slice(0, 200));
|
||||
|
||||
// Round-trip a file back to Markdown
|
||||
const exported = mem.exportMarkdown('memory/MEMORY.md');
|
||||
console.log(exported.slice(0, 500));
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Update agent code
|
||||
|
||||
If your agent code reads memory files directly from disk, update it to use
|
||||
the clawhdf5 API instead:
|
||||
|
||||
**Before (sqlite-vec + file reads):**
|
||||
```typescript
|
||||
const content = readFileSync('memory/user.md', 'utf8');
|
||||
const sections = parseMarkdown(content);
|
||||
const results = await vectorSearch(query, sections, k);
|
||||
```
|
||||
|
||||
**After (clawhdf5):**
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
const embedding = await embed(query); // your embedding function
|
||||
const results = mem.search(query, new Float32Array(embedding), k);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Session lifecycle hooks
|
||||
|
||||
Add compaction at session end for best long-term memory health:
|
||||
|
||||
```typescript
|
||||
// At the start of your agent process
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
|
||||
// ... agent runs ...
|
||||
|
||||
// At the end of each session
|
||||
mem.tickSession(); // decay activation weights
|
||||
const stats = mem.runConsolidation(Date.now() / 1000); // promote memories
|
||||
console.log('[memory] consolidation:', stats);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
If you need to roll back to sqlite-vec:
|
||||
|
||||
1. Change `memory.backend` back to `"sqlite-vec"` in your config.
|
||||
2. The original `.md` files are unchanged (if you kept them).
|
||||
3. Delete `agent.brain` (and `agent.brain.wal` if present).
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### `Error: no records found for path: memory/user.md`
|
||||
The path passed to `get()` or `exportMarkdown()` must exactly match the
|
||||
relative path used during `ingestMarkdown()`. Check for leading `./`
|
||||
differences.
|
||||
|
||||
### Memory is empty after reopening
|
||||
Make sure `flushWal()` was called after bulk writes. Without it, entries
|
||||
remain in the WAL and may be lost if the process exits abnormally.
|
||||
|
||||
### Embedding dimension mismatch
|
||||
The `embeddingDim` passed to `create()` cannot be changed after the file is
|
||||
created. If you switch embedding models, create a new `.brain` file and
|
||||
re-run the migration script.
|
||||
@@ -1,144 +0,0 @@
|
||||
# OpenClaw × clawhdf5 Configuration Reference
|
||||
|
||||
This document describes the full configuration schema for integrating
|
||||
`clawhdf5` as the memory backend in an OpenClaw agent gateway.
|
||||
|
||||
---
|
||||
|
||||
## Minimal example
|
||||
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"backend": "clawhdf5",
|
||||
"clawhdf5": {
|
||||
"path": "./agent.brain",
|
||||
"embeddingDim": 768
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full schema
|
||||
|
||||
```json
|
||||
{
|
||||
"memory": {
|
||||
"backend": "clawhdf5",
|
||||
"clawhdf5": {
|
||||
"path": "./agent.brain",
|
||||
"embeddingDim": 768,
|
||||
"walEnabled": true,
|
||||
"walMaxEntries": 500,
|
||||
"consolidation": {
|
||||
"workingCapacity": 100,
|
||||
"episodicCapacity": 10000,
|
||||
"episodicHalfLifeDays": 7,
|
||||
"semanticHalfLifeDays": 30,
|
||||
"promotionThreshold": 0.6,
|
||||
"semanticAccessThreshold": 10
|
||||
},
|
||||
"compaction": {
|
||||
"autoCompactThreshold": 0.3,
|
||||
"tickOnSessionEnd": true,
|
||||
"consolidateOnCompaction": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Field reference
|
||||
|
||||
### Top level
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `memory.backend` | `string` | `"clawhdf5"` | Must be `"clawhdf5"` to activate this backend |
|
||||
|
||||
### `clawhdf5`
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `path` | `string` | `"./agent.brain"` | Filesystem path for the `.brain` (HDF5) file. Relative to the OpenClaw working directory. |
|
||||
| `embeddingDim` | `number` | `768` | Dimension of the embedding vectors. Must match the embedder model. Common values: `384` (MiniLM), `768` (nomic-embed-text, BGE-base), `1536` (OpenAI text-embedding-3-small). |
|
||||
| `walEnabled` | `boolean` | `true` | Enable the Write-Ahead Log for crash recovery. Disable only on read-only stores or when crash safety is not required. |
|
||||
| `walMaxEntries` | `number` | `500` | Number of WAL entries to accumulate before an automatic merge to the .h5 file. Lower values = more frequent flushes (safer, slightly slower). |
|
||||
|
||||
### `clawhdf5.consolidation`
|
||||
|
||||
Controls the hippocampal three-tier memory engine (Working → Episodic →
|
||||
Semantic).
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `workingCapacity` | `number` | `100` | Maximum records in the Working tier before lowest-decay entries are evicted. |
|
||||
| `episodicCapacity` | `number` | `10000` | Maximum records in the Episodic tier. |
|
||||
| `episodicHalfLifeDays` | `number` | `7` | Half-life (in days) for exponential decay of Episodic records. Records not accessed within roughly one half-life drop in importance. |
|
||||
| `semanticHalfLifeDays` | `number` | `30` | Half-life for Semantic records. Longer than Episodic — semantic knowledge decays slowly. |
|
||||
| `promotionThreshold` | `number` | `0.6` | Importance score (0–1) above which a Working record is promoted to the Episodic tier. Higher = more selective. |
|
||||
| `semanticAccessThreshold` | `number` | `10` | Minimum access count for an Episodic record to be promoted to Semantic. |
|
||||
|
||||
### `clawhdf5.compaction`
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `autoCompactThreshold` | `number` | `0.3` | Fraction of tombstoned records (0–1) that triggers automatic compaction. `0.3` = compact when 30% of records are deleted. Set to `0` to disable auto-compact. |
|
||||
| `tickOnSessionEnd` | `boolean` | `true` | Run `tickSession()` (Hebbian decay) automatically when the agent session closes. |
|
||||
| `consolidateOnCompaction` | `boolean` | `true` | Run the hippocampal consolidation engine after each compaction cycle. |
|
||||
|
||||
---
|
||||
|
||||
## Embedder compatibility
|
||||
|
||||
The `embeddingDim` must remain constant for the lifetime of a `.brain` file.
|
||||
Mixing embedding models in the same file is not supported.
|
||||
|
||||
| Embedder | `embeddingDim` |
|
||||
|----------|---------------|
|
||||
| `all-MiniLM-L6-v2` | `384` |
|
||||
| `nomic-embed-text` | `768` |
|
||||
| `BGE-base-en-v1.5` | `768` |
|
||||
| `OpenAI text-embedding-3-small` | `1536` |
|
||||
| `OpenAI text-embedding-3-large` | `3072` |
|
||||
|
||||
---
|
||||
|
||||
## OpenClaw integration code
|
||||
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
// Load config from your OpenClaw config file
|
||||
const cfg = loadConfig(); // your config loading logic
|
||||
|
||||
const mem = ClawhdfMemory.openOrCreate(
|
||||
cfg.memory.clawhdf5.path,
|
||||
cfg.memory.clawhdf5.embeddingDim ?? 768,
|
||||
);
|
||||
|
||||
// On session end
|
||||
if (cfg.memory.clawhdf5.compaction?.tickOnSessionEnd) {
|
||||
mem.tickSession();
|
||||
}
|
||||
if (cfg.memory.clawhdf5.compaction?.consolidateOnCompaction) {
|
||||
const stats = mem.runConsolidation(Date.now() / 1000);
|
||||
console.log('[clawhdf5] consolidation:', stats);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Environment variables
|
||||
|
||||
The following environment variables override config file values when set:
|
||||
|
||||
| Variable | Overrides |
|
||||
|----------|-----------|
|
||||
| `CLAWHDF5_PATH` | `clawhdf5.path` |
|
||||
| `CLAWHDF5_EMBEDDING_DIM` | `clawhdf5.embeddingDim` |
|
||||
| `CLAWHDF5_WAL_ENABLED` | `clawhdf5.walEnabled` (`"true"` / `"false"`) |
|
||||
@@ -1,337 +0,0 @@
|
||||
# OpenClaw × clawhdf5 Integration
|
||||
|
||||
clawhdf5 provides a drop-in HDF5-backed memory backend for the
|
||||
[OpenClaw](https://github.com/redclawsystems/openclaw) agent gateway.
|
||||
This document covers architecture, the full Node.js API reference, and code
|
||||
examples for common operations.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ OpenClaw (Node.js/TypeScript) │
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌──────────────────────────────┐ │
|
||||
│ │ Agent runtime │───▶│ @redclaw/clawhdf5 (Node.js) │ │
|
||||
│ └─────────────────┘ │ TypeScript wrapper │ │
|
||||
│ └──────────────┬─────────────┘ │
|
||||
│ │ napi-rs FFI │
|
||||
└────────────────────────────────────────┼────────────────────┘
|
||||
│
|
||||
┌────────────────────────────────────────▼────────────────────┐
|
||||
│ clawhdf5-napi (Rust, cdylib) │
|
||||
│ │
|
||||
│ ClawhdfMemory ──▶ ClawhdfBackend ──▶ HDF5Memory │
|
||||
│ MemoryBackend ├─ MemoryCache │
|
||||
│ trait impl ├─ WalFile │
|
||||
│ ├─ SessionCache │
|
||||
│ └─ KnowledgeCache │
|
||||
│ │
|
||||
│ ConsolidationEngine (hippocampal tiers) │
|
||||
│ Working (100) ──▶ Episodic (10k) ──▶ Semantic (∞) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
┌────────────────▼────────────┐
|
||||
│ agent.brain (HDF5 file) │
|
||||
│ agent.brain.wal (WAL log) │
|
||||
└─────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **Single file**: everything lives in one `.brain` HDF5 file (+ WAL sidecar).
|
||||
- **In-memory cache**: the full embedding matrix and chunk list are loaded into RAM for fast search.
|
||||
- **Hybrid search**: vector similarity (70%) and BM25 full-text (30%) are blended with Reciprocal Rank Fusion (RRF), then re-ranked by Hebbian activation weight and temporal recency.
|
||||
- **Hippocampal tiers**: records are classified as Working, Episodic, or Semantic based on importance and access frequency. Tier promotion and eviction happen during `runConsolidation()`.
|
||||
- **WAL**: writes are journaled before hitting the .h5 file. On crash, the WAL is replayed at next open.
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install @redclaw/clawhdf5
|
||||
```
|
||||
|
||||
See [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md)
|
||||
for build-from-source instructions.
|
||||
|
||||
---
|
||||
|
||||
## Node.js API reference
|
||||
|
||||
### `ClawhdfMemory` (class)
|
||||
|
||||
All instance methods are synchronous. The native Rust code is single-threaded
|
||||
on the Node.js side; do **not** share a `ClawhdfMemory` instance across Worker
|
||||
threads without external locking.
|
||||
|
||||
---
|
||||
|
||||
#### Static factory methods
|
||||
|
||||
##### `ClawhdfMemory.create(path: string, embeddingDim: number): ClawhdfMemory`
|
||||
|
||||
Create a new `.brain` file. Throws if the file already exists.
|
||||
|
||||
```typescript
|
||||
const mem = ClawhdfMemory.create('./agent.brain', 768);
|
||||
```
|
||||
|
||||
##### `ClawhdfMemory.open(path: string): ClawhdfMemory`
|
||||
|
||||
Open an existing file. Replays the WAL automatically.
|
||||
|
||||
```typescript
|
||||
const mem = ClawhdfMemory.open('./agent.brain');
|
||||
```
|
||||
|
||||
##### `ClawhdfMemory.openOrCreate(path: string, embeddingDim: number): ClawhdfMemory`
|
||||
|
||||
**Recommended entry point.** Opens if the file exists, otherwise creates it.
|
||||
|
||||
```typescript
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `search(queryText, queryEmbedding, k): MemorySearchResult[]`
|
||||
|
||||
Hybrid BM25 + vector search.
|
||||
|
||||
```typescript
|
||||
const embedding = new Float32Array(await embed(query));
|
||||
const results = mem.search(query, embedding, 10);
|
||||
for (const r of results) {
|
||||
console.log(r.score.toFixed(3), r.path, r.text.slice(0, 80));
|
||||
}
|
||||
```
|
||||
|
||||
Pass an empty `Float32Array` to use BM25 only (no vector similarity).
|
||||
|
||||
**Parameters:**
|
||||
- `queryText: string` — used for BM25 term matching
|
||||
- `queryEmbedding: Float32Array` — dense vector of length `embeddingDim`
|
||||
- `k: number` — maximum results to return
|
||||
|
||||
**Returns:** `MemorySearchResult[]`
|
||||
|
||||
---
|
||||
|
||||
#### `get(path, fromLine?, numLines?): string | null`
|
||||
|
||||
Retrieve stored content by path.
|
||||
|
||||
```typescript
|
||||
const md = mem.get('memory/user.md'); // all content
|
||||
const lines = mem.get('memory/user.md', 5, 10); // lines 5–14
|
||||
const section = mem.get('memory/user.md::Goals'); // specific section
|
||||
```
|
||||
|
||||
Section sub-paths use the `::heading` suffix produced by `ingestMarkdown`.
|
||||
|
||||
---
|
||||
|
||||
#### `write(path, content): void`
|
||||
|
||||
Store raw content at `path`.
|
||||
|
||||
```typescript
|
||||
mem.write('memory/session.md', '# Session\n\nWorking on task X.');
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `ingestMarkdown(path, content): number`
|
||||
|
||||
Parse `content` as Markdown, split on ATX headings, and store each section
|
||||
separately. Returns the number of sections ingested.
|
||||
|
||||
```typescript
|
||||
import { readFileSync } from 'fs';
|
||||
const md = readFileSync('./memory/MEMORY.md', 'utf8');
|
||||
const count = mem.ingestMarkdown('memory/MEMORY.md', md);
|
||||
console.log(`Ingested ${count} sections`);
|
||||
```
|
||||
|
||||
Sections are addressable as `memory/MEMORY.md::HeadingName`.
|
||||
|
||||
---
|
||||
|
||||
#### `exportMarkdown(path): string`
|
||||
|
||||
Reconstruct stored sections for `path` back into a Markdown string.
|
||||
|
||||
```typescript
|
||||
const md = mem.exportMarkdown('memory/MEMORY.md');
|
||||
writeFileSync('./memory/MEMORY.md', md);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `stats(): BackendStats`
|
||||
|
||||
Return aggregate statistics.
|
||||
|
||||
```typescript
|
||||
const s = mem.stats();
|
||||
console.log(`Records: ${s.totalRecords}, Size: ${s.fileSizeBytes} bytes`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `compact(): number`
|
||||
|
||||
Remove tombstoned records from the store. Returns count removed.
|
||||
|
||||
---
|
||||
|
||||
#### `tickSession(): void`
|
||||
|
||||
Apply Hebbian decay to all activation weights. Call at session end.
|
||||
|
||||
---
|
||||
|
||||
#### `flushWal(): void`
|
||||
|
||||
Force a WAL merge: flush `.h5` and truncate the WAL log.
|
||||
|
||||
---
|
||||
|
||||
#### `runConsolidation(nowSecs: number): ConsolidationStats`
|
||||
|
||||
Run one full hippocampal consolidation cycle.
|
||||
|
||||
```typescript
|
||||
const stats = mem.runConsolidation(Date.now() / 1000);
|
||||
console.log(stats);
|
||||
// { workingCount: 42, episodicCount: 310, semanticCount: 5,
|
||||
// totalEvictions: 0, totalPromotions: 7 }
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `walPendingCount(): number`
|
||||
|
||||
Number of pending WAL entries (0 if WAL is disabled).
|
||||
|
||||
---
|
||||
|
||||
### Type reference
|
||||
|
||||
```typescript
|
||||
interface MemorySearchResult {
|
||||
text: string;
|
||||
score: number; // 0–1, higher = more relevant
|
||||
path: string; // source file path
|
||||
lineRange?: [number, number];
|
||||
timestamp?: number; // Unix epoch seconds
|
||||
source: string;
|
||||
}
|
||||
|
||||
interface BackendStats {
|
||||
totalRecords: number;
|
||||
totalEmbeddings: number;
|
||||
fileSizeBytes: number;
|
||||
modalities: string[]; // e.g. ["text"]
|
||||
lastUpdated?: number; // Unix epoch seconds
|
||||
}
|
||||
|
||||
interface ConsolidationStats {
|
||||
workingCount: number;
|
||||
episodicCount: number;
|
||||
semanticCount: number;
|
||||
totalEvictions: number;
|
||||
totalPromotions: number;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common patterns
|
||||
|
||||
### Session lifecycle
|
||||
|
||||
```typescript
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
|
||||
// --- agent session runs ---
|
||||
|
||||
// On session end: decay + consolidate
|
||||
mem.tickSession();
|
||||
const consolidationStats = mem.runConsolidation(Date.now() / 1000);
|
||||
console.log('[memory] consolidation:', consolidationStats);
|
||||
```
|
||||
|
||||
### Ingest all memory files at startup
|
||||
|
||||
```typescript
|
||||
import { readdirSync, readFileSync, statSync } from 'fs';
|
||||
import { join, relative } from 'path';
|
||||
|
||||
function ingestDirectory(mem: ClawhdfMemory, dir: string): void {
|
||||
for (const entry of readdirSync(dir)) {
|
||||
const full = join(dir, entry);
|
||||
if (statSync(full).isDirectory()) {
|
||||
ingestDirectory(mem, full);
|
||||
} else if (entry.endsWith('.md')) {
|
||||
const content = readFileSync(full, 'utf8');
|
||||
const path = relative(process.cwd(), full);
|
||||
mem.ingestMarkdown(path, content);
|
||||
}
|
||||
}
|
||||
mem.flushWal();
|
||||
}
|
||||
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
|
||||
ingestDirectory(mem, './memory');
|
||||
```
|
||||
|
||||
### Search with real embeddings
|
||||
|
||||
```typescript
|
||||
import OpenAI from 'openai';
|
||||
import { ClawhdfMemory } from '@redclaw/clawhdf5';
|
||||
|
||||
const ai = new OpenAI();
|
||||
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 1536);
|
||||
|
||||
async function searchMemory(query: string, k = 5) {
|
||||
const resp = await ai.embeddings.create({
|
||||
model: 'text-embedding-3-small',
|
||||
input: query,
|
||||
});
|
||||
const embedding = new Float32Array(resp.data[0].embedding);
|
||||
return mem.search(query, embedding, k);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Error handling
|
||||
|
||||
All methods that can fail throw a `NapiError` (a standard JS `Error` subclass)
|
||||
with the Rust error message as `message`.
|
||||
|
||||
```typescript
|
||||
try {
|
||||
const md = mem.exportMarkdown('nonexistent.md');
|
||||
} catch (e) {
|
||||
console.error('Export failed:', (e as Error).message);
|
||||
// "no records found for path: nonexistent.md"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- [openclaw-config.md](openclaw-config.md) — Full configuration schema
|
||||
- [migration-guide.md](migration-guide.md) — Migrating from sqlite-vec
|
||||
- [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md) — Build instructions
|
||||
- [BENCHMARKS.md](../BENCHMARKS.md) — Performance results
|
||||
@@ -0,0 +1,73 @@
|
||||
# OpenClaw: not supported
|
||||
|
||||
**clawhdf5 does not currently work as an [OpenClaw](https://docs.openclaw.ai)
|
||||
memory backend, and never has.** Earlier versions of these docs described a
|
||||
"drop-in" backend enabled with `memory.backend = "clawhdf5"`. That
|
||||
configuration was never valid: from v2026.2 through v2026.7 OpenClaw's
|
||||
`memory.backend` accepted only `"builtin"` or `"qmd"` and rejected unknown
|
||||
keys, and since v2026.8.1 ("OpenClaw 2.0") the key no longer exists. A Gateway
|
||||
given that config refuses to start. No plugin was ever built or tested against
|
||||
OpenClaw, and the `@redclaw/clawhdf5` npm package was never published.
|
||||
|
||||
As of 2026-09-25 we are not pursuing an OpenClaw plugin; the maintained
|
||||
integration target is ZeroClaw. This page records what a plugin would need,
|
||||
for when that changes.
|
||||
|
||||
## What OpenClaw expects today (v2026.9.6)
|
||||
|
||||
Checked against the OpenClaw source at tag `v2026.9.6` and its docs on
|
||||
2026-09-25. OpenClaw marks every plugin API as experimental, so re-check before
|
||||
building anything.
|
||||
|
||||
- **Memory lives in Markdown files**, which are the source of truth: `MEMORY.md`,
|
||||
`USER.md`, daily notes in `memory/YYYY-MM-DD.md` in the agent workspace. The
|
||||
memory engine is an index over them
|
||||
([concepts/memory](https://docs.openclaw.ai/concepts/memory)).
|
||||
- **A memory plugin is selected with `plugins.slots.memory: "<plugin-id>"`**
|
||||
(default `memory-core`), its settings under
|
||||
`plugins.entries.<plugin-id>.config`, validated against the plugin's own
|
||||
schema ([gateway/config-extensions](https://docs.openclaw.ai/gateway/config-extensions)).
|
||||
Memory search settings are under `memory.search`
|
||||
([reference/memory-config](https://docs.openclaw.ai/reference/memory-config)).
|
||||
- **A plugin needs** an `openclaw.plugin.json` manifest with `id`,
|
||||
`configSchema`, `"kind": "memory"` and every tool listed in `contracts.tools`
|
||||
([plugins/manifest](https://docs.openclaw.ai/plugins/manifest)); a
|
||||
`package.json` with `openclaw.extensions`, `openclaw.compat.pluginApi` and an
|
||||
`openclaw` peer dependency; and an entry built with `definePluginEntry`.
|
||||
- **Two ways to integrate** (both exist upstream): tools only, as
|
||||
`memory-lancedb` does (`api.registerTool`), or a full memory engine, as
|
||||
`memory-core` does, through `api.registerMemoryCapability({ runtime, ... })`,
|
||||
whose runtime returns a `MemorySearchManager` implementing `search`,
|
||||
`readFile` (returning `status: "ok" | "not_found"`), `status`,
|
||||
`probeEmbeddingAvailability` and `probeVectorAvailability`. Active Memory
|
||||
expects `memory_search` and `memory_get` tools
|
||||
([plugins/sdk-overview/memory-and-context](https://docs.openclaw.ai/plugins/sdk-overview/memory-and-context)).
|
||||
- **Embeddings come from OpenClaw's providers** (`memory.search.provider`), or a
|
||||
plugin registers one with `api.registerEmbeddingProvider`.
|
||||
- **Native code**: plugin installs run with `--ignore-scripts`, so a napi addon
|
||||
has to ship as prebuilt per-platform packages (the pattern `memory-lancedb`
|
||||
uses for LanceDB), loaded lazily
|
||||
([plugins/dependency-resolution](https://docs.openclaw.ai/plugins/dependency-resolution)).
|
||||
- **Distribution**: `openclaw plugins install` from npm or ClawHub; a first
|
||||
install from an arbitrary source needs explicit review, and community ClawHub
|
||||
packages go through a security audit.
|
||||
- **Churn to plan for**: the memory SDK was reshaped in 2026 (separate
|
||||
registration functions merged into `registerMemoryCapability`;
|
||||
`registerMemoryEmbeddingProvider` removed on 2026-08-21), and further SDK
|
||||
surfaces become eligible for removal on 2026-10-01
|
||||
([plugins/sdk-migration/removal-timeline](https://docs.openclaw.ai/plugins/sdk-migration/removal-timeline)).
|
||||
|
||||
## What this repository has
|
||||
|
||||
Building blocks, usable as a library today, but not an OpenClaw plugin:
|
||||
|
||||
- `clawhdf5_agent::openclaw::ClawhdfBackend` — a Markdown-oriented backend over
|
||||
`HDF5Memory`: ingest Markdown by section, hybrid search with re-ranking and
|
||||
confidence rejection, read back by path, export. Gaps a plugin would have to
|
||||
close: `write`/`ingest_markdown` store no embeddings (search is keyword-only
|
||||
for that content unless records are saved with `save_entry`), re-ingesting
|
||||
appends rather than replaces, there is no delete, `line_range` is never set,
|
||||
and export rewrites every heading as `##`.
|
||||
- `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a
|
||||
TypeScript wrapper. **Not published, not built or tested in CI, and known to
|
||||
be broken**; see `docs/known-issues.md`.
|
||||
Reference in New Issue
Block a user