clawhdf5/docs/migration-guide.md

Migration Guide: OpenClaw sqlite-vec → clawhdf5

This guide walks through migrating an OpenClaw agent from its default sqlite-vec + Markdown file memory to the clawhdf5 HDF5 backend.

---

Why migrate?

Feature	sqlite-vec + Markdown	clawhdf5
Storage format	SQLite WAL + flat .md files	Single HDF5 binary file
Vector search	sqlite-vec (SQLite extension)	Pure-Rust SIMD (clawhdf5-accel)
Full-text search	External (FTS5 or plain string match)	Built-in BM25
Hybrid search	Manual combination	Automatic RRF blend
Memory tiers	Flat	Working → Episodic → Semantic
Hebbian decay	Not built-in	Automatic activation weighting
Portability	SQLite binary required	Zero native deps (all Rust)
Crash recovery	SQLite WAL	clawhdf5 WAL
Compaction	Manual	Auto-threshold + session-end
Embedding dim change	New DB required	New file required (same)

---

Step 1: Install @redclaw/clawhdf5

npm install @redclaw/clawhdf5

Or, if building from the monorepo source:

npm install -g @napi-rs/cli
cd packages/clawhdf5-node
npm install
npm run build

---

Step 2: Update your OpenClaw config

Change backend from "sqlite-vec" (or "markdown") to "clawhdf5":

{
  "memory": {
    "backend": "clawhdf5",
    "clawhdf5": {
      "path": "./agent.brain",
      "embeddingDim": 768
    }
  }
}

See openclaw-config.md for the full schema.

---

Step 3: Run the one-time migration

clawhdf5 ships a migration helper that reads your existing Markdown memory files and ingests them via ingestMarkdown().

Automated migration script

import { ClawhdfMemory } from '@redclaw/clawhdf5';
import { readFileSync, readdirSync, statSync } from 'fs';
import { join, relative } from 'path';

async function migrate(
  memoryDir: string,
  brainPath: string,
  embeddingDim: number = 768,
): Promise<void> {
  const mem = ClawhdfMemory.create(brainPath, embeddingDim);

  // Walk all .md files under memoryDir
  function walk(dir: string): string[] {
    return readdirSync(dir).flatMap((entry) => {
      const full = join(dir, entry);
      return statSync(full).isDirectory() ? walk(full) : [full];
    });
  }

  const files = walk(memoryDir).filter((f) => f.endsWith('.md'));
  let totalSections = 0;

  for (const file of files) {
    const content = readFileSync(file, 'utf8');
    const relPath = relative(process.cwd(), file);
    const count = mem.ingestMarkdown(relPath, content);
    console.log(`  ${relPath}: ${count} sections`);
    totalSections += count;
  }

  // Force WAL merge after bulk import
  mem.flushWal();

  console.log(`\nMigration complete: ${files.length} files, ${totalSections} sections`);
  const s = mem.stats();
  console.log(`  Total records:    ${s.totalRecords}`);
  console.log(`  File size:        ${(s.fileSizeBytes / 1024).toFixed(1)} KB`);
}

// Usage
migrate('./memory', './agent.brain', 768).catch(console.error);

What the migration does

1. Walks all .md files under your memory directory.
2. Parses each file into sections using the same MarkdownParser used by OpenClaw (splits on ATX headings #, ##, ###, …).
3. Stores each section as a separate record in the HDF5 file with the file path as the source_channel (e.g. memory/user.md::Goals).
4. Flushes the WAL to merge everything into the .brain file.

After migration, the original .md files are not modified or deleted. You can keep them as a backup or remove them once you have verified the migrated data.

---

Step 4: Verify

import { ClawhdfMemory } from '@redclaw/clawhdf5';

const mem = ClawhdfMemory.open('./agent.brain');
const s = mem.stats();
console.log('Records after migration:', s.totalRecords);

// Spot-check: retrieve a known path
const userMd = mem.get('memory/MEMORY.md');
console.log(userMd?.slice(0, 200));

// Round-trip a file back to Markdown
const exported = mem.exportMarkdown('memory/MEMORY.md');
console.log(exported.slice(0, 500));

---

Step 5: Update agent code

If your agent code reads memory files directly from disk, update it to use the clawhdf5 API instead:

Before (sqlite-vec + file reads):

const content = readFileSync('memory/user.md', 'utf8');
const sections = parseMarkdown(content);
const results = await vectorSearch(query, sections, k);

After (clawhdf5):

import { ClawhdfMemory } from '@redclaw/clawhdf5';

const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
const embedding = await embed(query); // your embedding function
const results = mem.search(query, new Float32Array(embedding), k);

---

Step 6: Session lifecycle hooks

Add compaction at session end for best long-term memory health:

// At the start of your agent process
const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);

// ... agent runs ...

// At the end of each session
mem.tickSession();                                 // decay activation weights
const stats = mem.runConsolidation(Date.now() / 1000);  // promote memories
console.log('[memory] consolidation:', stats);

---

Rollback

If you need to roll back to sqlite-vec:

1. Change memory.backend back to "sqlite-vec" in your config.
2. The original .md files are unchanged (if you kept them).
3. Delete agent.brain (and agent.brain.wal if present).

---

Troubleshooting

Error: no records found for path: memory/user.md

The path passed to get() or exportMarkdown() must exactly match the relative path used during ingestMarkdown(). Check for leading ./ differences.

Memory is empty after reopening

Make sure flushWal() was called after bulk writes. Without it, entries remain in the WAL and may be lost if the process exits abnormally.

Embedding dimension mismatch

The embeddingDim passed to create() cannot be changed after the file is created. If you switch embedding models, create a new .brain file and re-run the migration script.