2026-09-28 17:24:26 +00:00
1 changed files with 496 additions and 0 deletions
@@ -0,0 +1,496 @@
 # Agent memory (`clawhdf5-agent`)
 `clawhdf5-agent` is a persistent, searchable memory store for AI agents,
 built on clawhdf5's HDF5 writer: records (text, embedding, source channel,
 timestamp, session, tags), sessions and a knowledge graph in one `.h5` file,
 with a write-ahead log beside it. This page is the long form of the agent
 part of the [README](../README.md); every number on it comes from
 [BENCHMARKS.md](../BENCHMARKS.md), where the commands and machines are.
 - [Quick start](#quick-start) · [Search](#search) · [Signed checkpoints](#signed-checkpoints)
 - [Architecture](#architecture) · [Modules](#modules) · [Library components](#library-components)
 - [Performance](#performance) · [LongMemEval](#longmemeval-retrieval-recall) · [Footprint](#memory-footprint)
 - [Feature flags and settings](#feature-flags-and-settings) · [File schema](#file-schema)
 - [CLI](#cli) · [Migrating from SQLite](#migrating-from-sqlite) · [Research foundation](#research-foundation)
 Integration status: ClawBrainHub's CLI uses this crate's `bm25::BM25Index`;
 no agent framework uses the store. clawhdf5 is **not** an OpenClaw memory
 plugin ([openclaw.md](openclaw.md)), and ZeroClaw does not use it.
 ## Quick start
 ```toml
 [dependencies]
 clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # not on crates.io yet
 ```
 ```rust
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions};
 // A new store: 384-dim embeddings (float16 on disk and an int8 HNSW index by default).
 let mut memory = HDF5Memory::create(MemoryConfig::new("agent.h5".into(), "my-agent", 384))?;
 memory.save(MemoryEntry {
    chunk: "User prefers dark mode and vim keybindings.".into(),
    embedding: embed("User prefers dark mode and vim keybindings."), // your embedder
    source_channel: "chat".into(),
    timestamp: now,
    session_id: "session-001".into(),
    tags: "preference".into(),
 })?;
 // Hybrid search: HNSW vector + BM25 keyword, fused 0.4 / 0.6 (the measured default).
 let query = embed("what editor does the user like?");
 for r in memory.search(&query, "editor preferences", &SearchOptions::new(5)) {
    println!("[{:.3}] {}", r.score, r.chunk);
 }
 memory.flush_wal()?; // checkpoint the WAL into agent.h5
 ```
 clawhdf5 stores embeddings; it does not compute them. Any dimension works,
 fixed when the store is created. `HDF5Memory::open(path)` reopens a store
 (holding its single-writer lock); `HDF5Memory::open_read_only(path)` gives a
 lock-free point-in-time view.
 ## Search
 `HDF5Memory::search(query_emb, text, &SearchOptions)` is the full search
 path; `hybrid_search(query_emb, text, vector_weight, keyword_weight, k)` and
 `hybrid_search_with` are thin wrappers over it.
 ```rust
 use clawhdf5_agent::confidence::ConfidenceConfig;
 use clawhdf5_agent::reranker::ReRankConfig;
 // Only memories from these source channels; still a full page of k results.
 let work = memory.search(&query, "deadline", &SearchOptions::new(5).with_sources(["slack", "email"]));
 // Re-rank (relevance, recency, source authority, activation), then drop
 // low-confidence results: the pipeline ClawhdfBackend runs.
 let careful = memory.search(
    &query,
    "user preferences",
    &SearchOptions::new(5)
        .with_rerank(ReRankConfig::default())
        .with_confidence(ConfidenceConfig::default()),
 );
 ```
 The source-channel filter is applied before ranking: an exact scan of the
 allowed records whenever that is cheaper than the index would be, and as
 the fallback when the index returns a short pool. Hebbian activation boosts
 are persisted by the next checkpoint (or on drop), not per query; search
 never writes the store.
 ## Signed checkpoints
 ```rust
 use clawhdf5_agent::signing;
 let key = signing::generate_key();          // keep the secret key; publish the public one
 let public = key.verifying_key();
 memory.set_signing_key(key);                 // never written to disk
 memory.flush_wal()?;                         // this checkpoint is signed
 let report = HDF5Memory::verify(std::path::Path::new("agent.h5"), &public)?;
 assert!(report.is_valid());                  // report.changed_records names edited records
 ```
 The Ed25519 signature covers every record (text, embedding as stored,
 channel, timestamp, session, tags, deleted flag, activation) through a
 SHA-256 Merkle tree, plus the store's settings, sessions and knowledge
 graph, so a change made with any tool is caught and located. It covers
 checkpoints, not saves still in the WAL (`report.wal_entries_unsigned`
 counts those). A signed store refuses to checkpoint without the key
 (`MemoryError::SigningKeyRequired`). CLI: `clawhdf5 keygen`,
 `--signing-key <file>` on writing commands, and `verify --public-key`.
 Signing adds about 20% to a checkpoint and 32 bytes per record to the file
 ([BENCHMARKS.md § Signed checkpoints](../BENCHMARKS.md#signed-checkpoints)).
 ## Architecture
 ```
                         ┌─────────────────┐
                         │   Agent Query   │
                         └────────┬────────┘
                                  │
                ┌─────────────────▼──────────────────┐
                │ HDF5Memory::search                 │
                │ optional source-channel filter     │
                │ HNSW vector + BM25 keyword         │
                │ weighted fusion (0.4 / 0.6)        │
                │ × √(Hebbian activation)            │
                └─────────────────┬──────────────────┘
                                  │  opt-in (SearchOptions);
                                  │  ClawhdfBackend turns both on
                ┌─────────────────▼──────────────────┐
                │ Multi-factor re-ranking            │
                │ relevance · recency · authority ·  │
                │ activation                         │
                ├────────────────────────────────────┤
                │ Confidence rejection               │
                │ (suppress bad matches)             │
                └─────────────────┬──────────────────┘
                                  │
     ┌────────────────────────────▼────────────────────────────┐
     │ In memory                                               │
     │  cache (embeddings) · BM25 index · HNSW index           │
     │  provenance ledger + anomaly alerts (session-scoped)    │
     └────────────────────────────┬────────────────────────────┘
                                  │ WAL append; checkpoint
     ┌────────────────────────────▼────────────────────────────┐
     │ agent_memory.h5   /meta · /memory · /sessions ·         │
     │                   /knowledge_graph                      │
     │ agent_memory.h5.wal   chained-CRC write-ahead log       │
     │ agent_memory.h5.ann   HNSW graph (derived, rebuildable) │
     │ agent_memory.h5.lock  single-writer lock                │
     └─────────────────────────────────────────────────────────┘
 ```
 **Durability.** Every WAL entry carries a CRC32 chained to the previous
 entry's, so a corrupted, reordered, duplicated or spliced entry stops replay
 instead of loading bad data. Each checkpoint records a WAL mark in `/meta`,
 so a crash between a checkpoint and the WAL truncate never applies an entry
 twice. Checkpoints and snapshots are made durable as a unit (temp file
 synced, renamed, directory synced). **Individual WAL appends are not
 fsynced** (a latency trade-off): saves since the last checkpoint can be lost
 on power failure or a kernel panic, not on a process crash. An unreadable
 WAL is quarantined to `<store>.h5.wal.corrupt-<ts>` rather than blocking
 `open()`.
 **Single writer.** `create`/`open` take an exclusive advisory lock on
 `<store>.h5.lock`; a second opener gets `MemoryError::Locked`.
 **Write bookkeeping.** `save`/`save_batch`/`save_or_update` run each write
 through an in-memory (session-scoped, not persisted) provenance ledger — an
 unkeyed content hash per record, for detecting accidental corruption, not
 tampering — and a write-anomaly detector (rate limits, injection patterns,
 source distribution). Alerts never block a save; drain them with
 `take_anomaly_alerts`. The source classification is inferred from the
 caller's `source_channel` string, a heuristic, not an authenticated trust
 boundary.
 ## Modules
 | Module | What it does |
 |--------|-------------|
 | `hybrid` | Vector + BM25 fusion: min-max-normalised weighted sum, vector 0.4 / keyword 0.6 by default (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF via `Fusion::Rrf` / `hybrid_search_with` (measured worse) |
 | `reranker` | Re-ranking by retrieval relevance (leads, weight 1.0), recency, source authority, activation. Opt-in via `SearchOptions::with_rerank`; on in `ClawhdfBackend` |
 | `confidence` | Low-confidence rejection. Opt-in via `SearchOptions::with_confidence`; on in `ClawhdfBackend` |
 | `bm25` | Incremental Okapi BM25 index kept for the life of the store; optional stemming |
 | `signing` | Ed25519-signed checkpoints (above) |
 | `wal` | Write-ahead log, format v4, chained CRC32 per entry; reads v2 and v3 (v1 only through the one-time migration in `open`) |
 | `knowledge` | Entity/relation graph: BFS, spreading activation, fuzzy (Levenshtein) entity resolution |
 | `consolidation` | Three tiers (Working → Episodic → Semantic): importance, novelty, time decay |
 | `temporal` | Sorted timestamp index, session DAG, entity timeline |
 | `multimodal` | Cross-modal search over text/image/audio/video embeddings (exact scan) |
 | `provenance`, `anomaly` | Session-scoped write bookkeeping (above) |
 | `openclaw` | `ClawhdfBackend`, a Markdown-oriented backend (below). Named for OpenClaw, but **not an OpenClaw plugin** ([openclaw.md](openclaw.md)) |
 | `vector_search` | Flat cosine search paths: pre-normed, SIMD, BLAS, GPU, parallel |
 | `ivf` / `pq` | Standalone IVF and IVF-PQ indexes; not used by `HDF5Memory`, whose index is HNSW |
 | `query_expand`, `entity_extract` | Synonym/acronym/temporal query expansion; rule-based entity extraction into the graph |
 | `memory_strategy`, `decision_gate` | When to save: save-every, semantic shift, user correction; trivial/substantive classification |
 | `ephemeral` | In-memory TTL/LFU working tier |
 | `async_memory` | Tokio wrapper over the store (`async` feature) |
 ## Library components
 The consolidation tiers, the graph algorithms and the temporal and
 multi-modal indexes are components you drive directly; the store persists
 the records, sessions and graph they work over.
 ```rust
 use clawhdf5_agent::knowledge::KnowledgeCache;
 let mut kg = KnowledgeCache::new();
 let alice = kg.add_entity("Alice", "person", -1);
 let bob = kg.add_entity("Bob", "person", -1);
 let acme = kg.add_entity("Acme Corp", "company", -1);
 kg.add_relation(alice, acme, "works_at", 1.0);
 kg.add_relation(alice, bob, "manages", 0.8);
 let neighbors = kg.bfs_neighbors(alice, 2);                     // 2-hop neighbourhood
 let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5); // related entities
 let (id, created) = kg.resolve_or_create("alice", "person", -1, 2); // fuzzy (Levenshtein <= 2)
 assert_eq!((id, created), (alice, false));
 ```
 ```rust
 use clawhdf5_agent::consolidation::{ConsolidationConfig, ConsolidationEngine, UntrustedSource};
 let mut engine = ConsolidationEngine::new(ConsolidationConfig {
    working_capacity: 100,
    ..Default::default()
 });
 let id = engine.add_memory("User prefers dark mode".into(), embed("dark mode"), UntrustedSource::User, now);
 engine.access_memory(id, now + 60.0);   // reactivates it
 engine.consolidate(now + 3600.0);        // promote (Working -> Episodic -> Semantic) and evict
 let stats = engine.get_stats();
 println!("working {} episodic {} semantic {}", stats.working_count, stats.episodic_count, stats.semantic_count);
 ```
 System and correction sources get elevated importance and go through a
 separate entry point, `add_trusted_memory(.., TrustedSource::System, ..)`,
 so untrusted content cannot claim them.
 ```rust
 use clawhdf5_agent::temporal::TemporalIndex;
 let mut index = TemporalIndex::new();
 index.insert(1, 1_700_000_000.0);
 index.insert(2, 1_700_003_600.0);                               // an hour later
 let in_range = index.range_query(1_700_000_000.0, 1_700_010_800.0);
 let recent = index.latest(10);
 ```
 ### Markdown backend
 `ClawhdfBackend` ingests Markdown by section and searches it with the full
 pipeline. It is a library API, not an OpenClaw plugin.
 ```rust
 use clawhdf5_agent::openclaw::{ClawhdfBackend, MemoryBackend};
 let mut backend = ClawhdfBackend::create(std::path::Path::new("memory.h5"), 384)?;
 let md = std::fs::read_to_string("MEMORY.md")?;
 let sections = backend.ingest_markdown("MEMORY.md", &md)?;   // one record per heading
 for r in backend.search("dark mode", &embed("dark mode"), 5) {
    println!("[{:.3}] {} ({})", r.score, r.text, r.path);
 }
 let exported = backend.export_markdown("MEMORY.md")?;
 ```
 Limits: ingested sections carry no embedding, so their search is
 keyword-only unless you save records with vectors through `save_entry`;
 ingesting a file again adds its sections again; `export_markdown` writes
 every heading as `##`, so it is not a lossless round trip.
 ## Performance
 Unless marked otherwise, measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D,
 8C/16T), commit 5c8323c, 384-dim embeddings; commands in
 [BENCHMARKS.md](../BENCHMARKS.md).
 **HNSW (the default vector stage)** — `search_harness`, clustered data,
 N = 100K, M = 16, ef_construction = 64, ef = 64, recall against an exact scan
 ([§ Quantising the index copy](../BENCHMARKS.md#quantising-the-index-copy-quantized_index)):
 | index | recall@10 | QPS | build |
 |---|---:|---:|---:|
 | `f32` | 0.9945 | 13 399 | 3.2 s |
 | `i8` + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
 A paired comparison (medians of alternating runs, same binary), not re-run
 on 2026-09-24: a single `f32` run that day measured recall 0.9945, 19 001
 QPS and a 2.7 s build, so the 1.63x ratio has not been re-checked. On a
 Raspberry Pi 5 (NEON `SDOT`) the int8 index is 1.18x the `f32` QPS at equal
 recall. Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was
 0.31.
 **Operations:**
 | Operation | Latency | Scale |
 |-----------|---------|-------|
 | `hybrid_search` p50 | 0.07 ms / 0.49 ms / 4.69 ms | 1K / 10K / 100K records |
 | BM25 keyword search | 20.4 µs | 1K records |
 | Knowledge graph BFS | 23.1 µs | 1K entities |
 | Spreading activation | 10.1 µs | 100 entities |
 | Temporal range query | 622 ns | 10K timestamps |
 | Consolidation cycle | 115.2 µs | 1K records |
 | Cross-modal search (exact scan, 2 embeddings per record) | 842.0 µs / 8.44 ms | 1K / 10K records |
 | Memory write (WAL append) | 26.1 µs | per record |
 `float16` stores (the default) add about 2 µs per write for rounding
 ([§ Write Path](../BENCHMARKS.md#write-path)).
 **Brute-force and IVF** (Criterion; not used by `HDF5Memory`):
 | Scale | Flat | IVF (nprobe=10) | IVF-PQ |
 |-------|------|-----------------|--------|
 | 1K | 47.4 µs | — | — |
 | 10K | 500.5 µs | 24.8 µs | — |
 | 100K | 6.58 ms | 592 µs | 869 µs |
 No comparison with MemX is made: its published figure is end-to-end and
 ours is one component ([BENCHMARKS.md](../BENCHMARKS.md#comparison-to-memx-arxiv260316171)).
 **Consolidation** — 1,000 records (10 signal + 990 noise),
 `working_capacity = 100`: the store goes from 1,000 to 100 records with
 Hit@1 on the signal records staying at 100%, and search from 2.22 ms to
 0.24 ms ([§ Consolidation Efficiency](../BENCHMARKS.md#consolidation-efficiency)).
 ## LongMemEval retrieval recall
 Full `longmemeval_s` haystack, all 500 questions (47.7 sessions and 493.5
 turns each; 4.0% of sessions are evidence), real `all-MiniLM-L6-v2`
 embeddings, k = 10. Re-run 2026-09-27 on tank; the headline reproduced
 exactly ([§ LongMemEval Results](../BENCHMARKS.md#longmemeval-results)):
 | Mode | Turn-level Hit@5 | Session-level Hit@5 |
 |------|------------------|---------------------|
 | BM25 only | 75.0% | 93.6% |
 | Vector only (MiniLM) | 71.8% | 94.2% |
 | Hybrid 0.4 / 0.6 (default) | **81.4%** | **96.8%** |
 This is **retrieval recall** (did a gold turn appear in the top k), not the
 official LongMemEval QA accuracy; the two are not comparable. A weight sweep
 found the old 0.7 / 0.3 default strictly dominated by 0.4 / 0.6, the default
 since v2.5.0; use 0.3 / 0.7 if rank-1 precision matters most. Earlier
 session-level figures of 100% and a claimed win over MemX were retracted
 ([BENCHMARKS.md](../BENCHMARKS.md#retracted-session-level-recall-and-the-memx-comparison)).
 The benchmark's vector stage needs `clawhdf5-bench`'s `embeddings` feature.
 ## Memory footprint
 **On disk** — `float16` embeddings (the default), 200-character synthetic
 text, `footprint_bench`: 810.4 KB at 1K records, 7.8 MB at 10K, 76.7 MB at
 100K (803–829 bytes per record). The synthetic text is far more repetitive
 than real text (40 distinct strings, deflated), so real records will be
 larger; the embeddings alone are 768 B per record. On the same data, 100K ×
 384 takes 80.8 MiB as `float16` and 154.0 MiB as `f32`
 ([§ Memory Footprint](../BENCHMARKS.md#memory-footprint-1)).
 **In memory** — a store reopened from disk, counting allocator
 ([§ Memory footprint](../BENCHMARKS.md#memory-footprint)):
 | Records | Raw vectors | `f32` index | `i8` index (default) |
 |---------|-------------|-------------|----------------------|
 | 1K | 1 MiB | 4 MiB (2.40x) | 2 MiB (1.64x) |
 | 10K | 15 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
 | 100K | 146 MiB | 399 MiB (2.72x) | 256 MiB (1.74x) |
 The `f32` column was re-measured on 2026-09-24; the `i8` column was not.
 ## Feature flags and settings
 | `clawhdf5-agent` flag | Default | Description |
 |------|---------|-------------|
 | `float16` | **yes** | Half-precision cosine kernel. Half-precision *storage* is the `MemoryConfig::float16` setting, not this feature |
 | `hnsw` | **yes** | HNSW index for the vector stage (`clawhdf5-ann`); without it, an exact linear scan |
 | `parallel` | **yes** | Parallel HNSW bulk build (identical graph) and Rayon search strategies |
 | `zstd` | no | Zstd instead of deflate for embeddings when `MemoryConfig::compression` is on (links libzstd) |
 | `fast-math` / `openblas` / `accelerate` | no | BLAS matrix-vector multiply (generic / OpenBLAS / Apple Accelerate) |
 | `gpu` | no | GPU distance computation via wgpu (`clawhdf5-gpu`) |
 | `async` | no | Tokio async wrapper with background flush |
 For an exact linear scan: `--no-default-features --features float16`.
 Settings stored in the file (`MemoryConfig`):
 - `float16` (**on** for new stores): embeddings on disk as IEEE half
  precision, rounded as they enter the cache so memory and file agree;
  values must lie within ±65504. On LongMemEval with real MiniLM embeddings
  every retrieval metric matches `f32`. Opt out with `float16 = false` or
  `clawhdf5 create --f32`. Existing stores keep their setting.
 - `quantized_index` (**on** for new stores): the HNSW index's copy of the
  embeddings as `i8`, re-scored against the exact embeddings; see the table
  above. Opt out with `quantized_index = false` or `create --f32-index`.
 - `hnsw_m`, `hnsw_ef_construction`, `hnsw_ef_search`: 16 / 64 / scaled with
  `k` by default.
 - `compression` (off): deflate (or Zstd) for embeddings; text of 4 KiB or
  more is always deflated.
 - `wal_enabled` (on), `wal_max_entries`, `hebbian_boost`, `decay_factor`.
 ## File schema
 ```
 agent_memory.h5
 ├── /meta                          (attributes)
 │   ├── schema_version, edgehdf5_version (writer tag, kept for compatibility)
 │   ├── agent_id, embedder, embedding_dim, chunk_size, overlap, created_at
 │   ├── float16, compression, compression_level, compact_threshold,
 │   │   hebbian_boost, decay_factor, wal_enabled, wal_max_entries
 │   ├── quantized_index, hnsw_m, hnsw_ef_construction, hnsw_ef_search
 │   ├── wal_applied_len, wal_applied_crc   (WAL mark of the last checkpoint)
 │   └── ann_generation                     (ties the .ann sidecar to this checkpoint)
 ├── /memory
 │   ├── chunks:              string[N]
 │   ├── embeddings:          f32[N × D], or f16 for a `float16` store (chunked)
 │   ├── source_channel, session_ids, tags:  string[N]
 │   ├── timestamps:          f64[N]
 │   ├── tombstones:          u8[N]
 │   ├── norms:               f32[N]      (pre-computed L2)
 │   └── activation_weights:  f32[N]      (Hebbian)
 ├── /sessions
 │   ├── ids, channels, summaries:  string[S]
 │   ├── start_idxs, end_idxs:      i64[S]
 │   └── timestamps:                f64[S]
 ├── /knowledge_graph
 │   ├── entity_ids, entity_emb_idxs: i64[E]; entity_names, entity_types: string[E]
 │   ├── relation_srcs, relation_tgts: i64[R]; relation_types: string[R]
 │   ├── relation_weights: f32[R]; relation_ts: f64[R]
 │   └── alias_strings: string[A]; alias_entity_ids: i64[A]   (when aliases exist)
 └── /integrity                     (signed stores: per-record hashes and the signed manifest)
 ```
 A store is an ordinary HDF5 file: h5py, h5dump and `h5rs` read it (the
 agent's `h5py_interop` test checks a whole store). Beside it:
 `<store>.h5.wal`, `<store>.h5.ann` (HNSW graph; derived, safe to delete)
 and `<store>.h5.lock`.
 ## CLI
 `clawhdf5-cli` installs a binary named `clawhdf5`:
 ```bash
 cargo install --path crates/clawhdf5-cli
 clawhdf5 --path agent.h5 create --agent-id my-agent --dim 384 --wal
 echo '{"chunk":"User prefers dark mode","embedding":[0.1, ...],"source_channel":"chat","timestamp":1700000000.0,"session_id":"s1","tags":"pref"}' \
  | clawhdf5 --path agent.h5 save
 clawhdf5 --path agent.h5 search --embedding '[0.1, ...]' --query 'dark mode preferences' \
  --top-k 5 --vector-weight 0.4 --keyword-weight 0.6
 clawhdf5 --path agent.h5 stats          # also: recall <index>, export, agents-md, flush-wal
 clawhdf5 --path agent.h5 snapshot backup.h5
 clawhdf5 keygen --out signing.key       # then --signing-key signing.key; verify --public-key <hex>
 ```
 Output is JSON. The CLI's `search` defaults to weights 0.7 / 0.3, not the
 library's 0.4 / 0.6, so pass them. `recall`, `stats`, `agents-md` and
 `export` open the store read-only.
 ## Migrating from SQLite
 ```bash
 cargo install --path crates/clawhdf5-migrate
 clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedder minilm
 ```
 The output is an ordinary agent store, written through the agent's API. The
 source must use the `memory_chunks` / `sessions` / `entities` / `relations`
 layout (names configurable with `--*-table`); this is not ZeroClaw's schema,
 and ZeroClaw does not use clawhdf5. What carries over:
 | SQLite | Agent store |
 |--------|-------------|
 | `memory_chunks` | records (text, embedding, source channel, timestamp, session id, tags); rows with `deleted = 1` become deleted records, or are left out with `--skip-deleted` |
 | `sessions` | sessions (id, start/end index, channel, summary, timestamp) |
 | `entities`, `relations` | knowledge-graph entities and relations; entities get new ids and relations are re-pointed |
 Records are written in `id` order and numbered from 0. Embeddings are
 stored as float16 like any new store; `--f32` keeps full precision (and is
 required for values beyond ±65504). The dimension is detected from the
 first row unless `--embedding-dim` is given, and a row of another length is
 an error, never truncated or padded; a source with no records needs
 `--embedding-dim`. Every row is checked before the output is created.
 `--incremental` adds only rows the store does not hold (records already in
 it take the source's deleted flag). The tool reads the result back
 read-only, compares it with the source (every row with `--validate-full`)
 and checks that a migrated record is found by search; `--dry-run` only
 counts rows. `clawhdf5-migrate` bundles SQLite, so it compiles C.
 Older crate names: `rustyhdf5*` is now `clawhdf5*`, `edgehdf5-memory` is
 `clawhdf5-agent`, and the `edgehdf5` CLI is `clawhdf5-cli`.
 ## Research foundation
 The design draws on recent papers on agent memory:
 | Paper | Idea | Module |
 |-------|------|--------|
 | MemX (2026) | Hybrid fusion + multi-factor re-ranking | `hybrid`, `reranker` |
 | Graph-Native Cognitive Memory (2026) | Weighted, timestamped relations; entity timelines | `knowledge`, `temporal` |
 | CraniMem (2026) | Bounded hippocampal memory | `consolidation` |
 | D-MEM (2026) | Surprise-gated storage (as a novelty score) | `consolidation` |
 | SYNAPSE (2025) | Spreading activation for recall | `knowledge` |
 | RAGdb (2025) | Zero-dependency edge RAG | architecture |
 | MemoryGraft (2025) | Memory poisoning attacks | `anomaly`, `provenance` |
 | MemoryArena (2026) | Multi-session benchmark | `temporal` |
 | AI Hippocampus (2026) | Memory taxonomy survey | overall design |