clawhdf5/README.md

# ClawhDF5

**The memory layer AI agents deserve. One file. Pure Rust. No libhdf5.**

[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Rust](https://img.shields.io/badge/rust-1.94%2B-orange.svg)](https://www.rust-lang.org)
[![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building)
[![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
[![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)

ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.

> **Two things live here:**
> - **A general-purpose, pure-Rust HDF5 library** — no libhdf5, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
> - **An agent memory layer built on top of it** — vector search, knowledge graph, hippocampal-style consolidation, in `clawhdf5-agent`.

The crates are not on crates.io yet, so depend on them from git:

```toml
[dependencies]
clawhdf5       = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # core HDF5 read/write
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # + agent memory layer
```

> **C dependencies, precisely:** the HDF5 format code is pure Rust and never
> links libhdf5. The default deflate backend is zlib-ng (`fast-deflate`), a C
> library built from source, so a default build needs `cmake` and a C
> compiler. Opt-in codecs (`zstd`, `szip`) and BLAS backends link C too.

> **New here?** Start with the **[Quickstart Guide](docs/QUICKSTART.md)** · See **[Use Cases](docs/USE_CASES.md)** · Read **[Benchmarks](BENCHMARKS.md)**

## What's new (v2.2 → v2.7, and unreleased)

Five releases in September 2026. Details, including upgrade notes and every
breaking change, are in [CHANGELOG.md](CHANGELOG.md).

**HDF5 correctness (read these if you read files with an earlier release)**
- **Extensible Array chunk indexes returned wrong data** past the 36th chunk —
  any dataset with one unlimited dimension. Silent: plausible numbers from the
  wrong chunks. Fixed in v2.7.0; re-read affected data.
- Fixed and Extensible Array checksums are now verified, so a corrupt chunk
  index is `ChecksumMismatch` instead of wrong data (v2.7.0).
- Compound datatypes written with default libver bounds (plain
  `h5py.File(path, 'w')`) were mis-parsed; HDF5 2.0 compound v5 and native
  complex (class 11) types now parse (v2.2.0–v2.3.0).
- Committed datatypes, fill values, soft links and `H5T_STD_REF` references now
  read correctly; external links and external raw data are explicit errors;
  `attrs()` no longer silently drops attributes (v2.3.0–v2.5.0).
- Datasets indexed by a version-2 B-tree now read (v2.5.0).

**Security and robustness**
- A crafted file could abort any reader via B-tree v2 recursion or explode it
  via shared children; both are now fast errors (v2.7.0).
- Virtual-dataset source paths are confined to the file's directory; chunked
  reads use overflow-checked sizes and fallible allocation, and the facade
  writes files atomically (v2.3.0).
- Agent store: single-writer lock plus `open_read_only`; a crash between
  checkpoint and WAL truncate no longer duplicates entries; unreadable WALs are
  quarantined instead of blocking `open()` (v2.3.0).

**Search quality and speed**
- HNSW neighbour selection now uses the paper's diversity heuristic: recall@10
  at 100K went from 0.31 to 0.98 (v2.4.0).
- `hybrid_search` is 79–190× faster than v2.3.0 (p50 0.07 ms at 1K, 4.65 ms at
  100K). It no longer rebuilds BM25 or rewrites the store per query, and the
  HNSW graph is persisted (v2.4.0).
- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
  been discarding the retrieval score, costing the OpenClaw backend 40.6pp of
  Hit@1; fixed in v2.6.0.
- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
  0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).

**Memory**
- A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
  int8 HNSW index, **on by default for new stores** (unreleased), brings a
  100K × 384 store to 1.74× the raw vectors. At equal recall it is also faster
  than `f32`: 1.63× QPS on AVX2, 1.18× on a Raspberry Pi 5 (NEON `SDOT`).

**Tooling**
- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
  silently) and runs an aarch64 job for the NEON kernels.

---

## Why ClawhDF5?

Every AI agent needs memory. Today that means scattered Markdown files, SQLite databases, cloud-hosted vector stores, and glue code. ClawhDF5 replaces all of it:

| Problem | Status Quo | ClawhDF5 |
|---------|-----------|----------|
| Vector search | External DB (Pinecone, Qdrant) | Built-in, sub-millisecond |
| Keyword search | Separate FTS engine | Integrated BM25 |
| Knowledge graph | Neo4j or none | In-file graph with spreading activation |
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
| Temporal queries | Custom code | Native temporal index (716ns) |
| Multi-modal | Multiple stores | Unified cross-modal search |
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |

---

## Performance

Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note).

### HDF5 Core I/O (vs libhdf5 1.14.6)

*Benchmark numbers are being validated in collaboration with engineers from the HDF5 Group to confirm methodology and reproducibility.*

Figures below are from an independent reproduction run on a second machine (AMD Ryzen 7 7800X3D, 2026-08-03). Full methodology, the original i7-12650H run, and two additional benchmarks added to close prior coverage gaps (an I/O-inclusive metadata-open comparison and an honest zero-copy-mmap measurement) are in [BENCHMARKS.md § Independent Validation](BENCHMARKS.md#independent-validation-tank-ryzen-7-7800x3d-2026-08-03).

| Operation | ClawhDF5 | libhdf5 | Speedup |
|-----------|----------|---------|---------|
| Attribute write (128 attrs) | 85.2 µs | 877 µs | **10.3×** |
| Group create (64 groups) | 130 µs | 1.37 ms | **10.6×** |
| Chunked write, deflate-6 (512×512 f32) | 1.44 ms | 65.0 ms | **45.3×** |
| Sequential read (100K f32) | 23.3 µs | 63.6 µs | **2.7×** |
| Sequential write (100K f32) | 210 µs | 189 µs | **≈ tie** |

### Vector Search

**HNSW (the default backend for `hybrid_search`)** — `search_harness`, clustered
384-dim data, M = 16, ef_construction = 64, recall measured against an exact scan.
See [BENCHMARKS.md § Search harness](BENCHMARKS.md#search-harness-baseline-v230)
and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index):

| N = 100K, ef = 64 | recall@10 | QPS | build |
|---|---:|---:|---:|
| `f32` index | 0.9945 | 13 399 | 3.2 s |
| `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |

Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.

**Brute-force and IVF paths** (Criterion, i7-12650H):

| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
|-------|------|-----------------|--------|----------|
| 1K | **54 µs** | — | — | — |
| 10K | 753 µs | **27 µs** | — | — |
| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) |

> Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected,
> apples-to-apples SIMD/scalar/parallel comparison methodology — see
> [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector
> Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05).

### Agent Memory Operations

| Operation | Latency | Scale |
|-----------|---------|-------|
| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records |
| BM25 keyword search | **67 µs** | 1K records |
| Knowledge graph BFS | **24 µs** | 1K entities |
| Spreading activation | **17 µs** | 100 entities |
| Temporal range query | **716 ns** | 10K timestamps |
| Consolidation cycle | **164 µs** | 1K records |
| Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) |
| Importance gate | **61 ns** | per record |

### Chunked Write Throughput (codec comparison)

Measured with Criterion on f32 matrices. Auto-shuffle is applied before all compression codecs
by default (AoS→SoA byte transpose, +157–204% throughput for float data):

| Codec | 128×128 f32 | 512×512 f32 | Notes |
|-------|-------------|-------------|-------|
| Zstd level 3 | **148 µs / 422 MiB/s** | **1.34 ms / 748 MiB/s** | With auto-shuffle |
| Deflate level 6 | 153 µs / 407 MiB/s | 1.39 ms / 719 MiB/s | With auto-shuffle |
| Pcodec | 528 µs / 118 MiB/s | 1.69 ms / 591 MiB/s | Best compression ratio |

Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition.

> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).

### LongMemEval Retrieval Recall

Evaluated against the full **`longmemeval_s`** haystack — all 500 questions, 47.7
sessions and 493.5 turns each, with only 4.0% of haystack sessions being evidence
sessions. See [BENCHMARKS.md § LongMemEval
Results](BENCHMARKS.md#longmemeval-results) for the full scoring-target
declaration:

| Mode | Turn-Level Hit@5 | Session-Level Hit@5 |
|------|------------------|---------------------|
| BM25 only | 75.0% | 93.6% |
| Vector only (MiniLM) | 71.8% | 94.2% |
| Hybrid (0.4/0.6, tuned) | **81.4%** | **96.8%** |

Hybrid is the strongest configuration, which is what running two retrieval stages
is for. The weights matter more than the stages: a sweep of `vector_weight` from
0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by
`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since
v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by
`unified_search`, `hybrid_search_with` and the OpenClaw backend); callers that
pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if
rank-1 precision matters most. Reciprocal rank fusion is selectable
(`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See
[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#weight-sweep--full-haystack-n500).

The benchmark's vector stage requires `clawhdf5-bench`'s `embeddings` feature
(real MiniLM embeddings); without it the vector stage is inert and only the BM25 row is produced, which is what every previously published
number here measured.

On the easier `longmemeval_oracle` variant (evidence sessions only) the same
harness scores 84.4% turn-level Hit@5 / MRR 0.6597, reproduced identically on a
second machine. The 9.4-point gap is the cost of the real haystack, and is why the
full-haystack number is the one quoted here.

This is **retrieval recall** (did the gold memory appear in the top-k), not the
official LongMemEval QA-accuracy metric — the two are not comparable, and
retrieval recall reported as QA accuracy typically overstates by 20–30 points.

> **Previously reported here and now retracted:** session-level Hit@5 of 100.0% /
> MRR 1.0000, and a claim of beating MemX's 51.6%. Those session-level figures were
> degenerate on the oracle variant (any returned document is a hit by
> construction); the 93.6% above is a different, real measurement on a corpus where
> evidence sessions are 4.0% of the haystack. The MemX comparison stays withdrawn —
> MemX measures fact-level granularity over 220,349 records, which running the full
> haystack does not fix. Details in
> [BENCHMARKS.md](BENCHMARKS.md#retracted-session-level-recall-and-the-memx-comparison).

> Enable embeddings via `hybrid_search(query_emb, text, 0.4, 0.6, k)` for substantially higher recall. The vector stage is served by the HNSW index by default (the `hnsw` feature is on by default); build with `--no-default-features --features float16` to fall back to an exact linear cosine scan.

### Memory Footprint

**On disk** — 384-dim embeddings, 200-char text
([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):

| Records | File Size | Bytes/Record | Gzip-6 compressed |
|---------|-----------|--------------|-------------------|
| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) |
| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |

**In memory** — a store reopened from disk, 384-dim `f32`, measured with a
counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):

| Records | Raw vectors | Reopened, `f32` index | Reopened, `i8` index (default) |
|---------|-------------|-----------------------|--------------------------------|
| 1K | 1 MiB | 4 MiB (2.40x) | 2 MiB (1.64x) |
| 10K | 15 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
| 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |

Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
embedding twice.

### Consolidation Efficiency

1,000 records (10 signal + 990 noise), `working_capacity = 100`
([BENCHMARKS.md § Consolidation Efficiency](BENCHMARKS.md#consolidation-efficiency)):

| Metric | Before | After | Delta |
|--------|--------|-------|-------|
| Records in store | 1,000 | 100 | −90% |
| Hit@1 recall (signal records) | 100% | 100% | no loss |
| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** |

**Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**

---

## Agent Memory Architecture

ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory systems (see [Research Foundation](#research-foundation)).

```
                         ┌─────────────────┐
                         │   Agent Query   │
                         └────────┬────────┘
                                  │
                ┌─────────────────▼──────────────────┐
                │ HDF5Memory::hybrid_search          │
                │ HNSW vector + BM25 keyword         │
                │ weighted fusion (0.4 / 0.6)        │
                │ × √(Hebbian activation)            │
                └─────────────────┬──────────────────┘
                                  │  OpenClaw backend adds:
                ┌─────────────────▼──────────────────┐
                │ Multi-factor re-ranking            │
                │ relevance · recency · authority ·  │
                │ activation                         │
                ├────────────────────────────────────┤
                │ Confidence rejection               │
                │ (suppress bad matches)             │
                └─────────────────┬──────────────────┘
                                  │
     ┌────────────────────────────▼────────────────────────────┐
     │ In memory                                               │
     │  cache (flat f32 embeddings) · BM25 index · HNSW index  │
     │  provenance ledger + anomaly alerts (session-scoped)    │
     └────────────────────────────┬────────────────────────────┘
                                  │ WAL append; checkpoint
     ┌────────────────────────────▼────────────────────────────┐
     │ agent_memory.h5   /meta · /memory · /sessions ·         │
     │                   /knowledge_graph                      │
     │ agent_memory.h5.wal   chained-CRC write-ahead log       │
     │ agent_memory.h5.ann   HNSW graph (derived, rebuildable) │
     │ agent_memory.h5.lock  single-writer lock                │
     └─────────────────────────────────────────────────────────┘
```

Consolidation tiers (Working → Episodic → Semantic), the knowledge-graph
algorithms, temporal and multi-modal indexes are library components you drive
directly; the store persists the records, sessions and graph they work over.

### Module Overview

| Module | What It Does |
|--------|-------------|
| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
| **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Used by the OpenClaw backend |
| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches (OpenClaw backend) |
| **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
| **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
| **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
| **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW |
| **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming |
| **`query_expand`** | Synonym / acronym / temporal query expansion |
| **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph |
| **`wal`** | Write-ahead log (v4) with a chained CRC32 per entry, so a corrupted, reordered, duplicated or spliced entry stops replay; checkpoints record a WAL mark so nothing is applied twice. Appends are not fsynced |
| **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection |
| **`decision_gate`** | Sub-microsecond trivial/substantive classification |
| **`ephemeral`** | In-memory TTL/LFU working tier |
| **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) |

---

## Quick Start

### HDF5 File I/O

```rust
use clawhdf5::{File, FileBuilder, AttrValue};

// Write
let mut builder = FileBuilder::new();
builder.create_dataset("temperatures")
    .with_f64_data(&[22.5, 23.1, 21.8])
    .with_shape(&[3]);
builder.write("output.h5")?;

// Read
let file = File::open("output.h5")?;
let ds = file.dataset("temperatures")?;
let values = ds.read_f64()?;
assert_eq!(values, vec![22.5, 23.1, 21.8]);
```

### Agent Memory

```rust
use clawhdf5_agent::{HDF5Memory, MemoryConfig, MemoryEntry, AgentMemory};

// Create memory store
let config = MemoryConfig::new("agent.h5".into(), "my-agent", 384);
let mut memory = HDF5Memory::create(config)?;

// Save a memory
memory.save(MemoryEntry {
    chunk: "User prefers dark mode and vim keybindings.".into(),
    embedding: embed("User prefers dark mode..."),  // your embedder
    source_channel: "chat".into(),
    timestamp: now(),
    session_id: "session-001".into(),
    tags: "preference".into(),
})?;

// Hybrid search: vector + BM25, weighted 0.4 / 0.6 (the measured default)
let results = memory.hybrid_search(&query_embedding, "user preferences", 0.4, 0.6, 5);
for result in results {
    println!("[{:.3}] {}", result.score, result.chunk);
}
```

### Knowledge Graph

```rust
use clawhdf5_agent::knowledge::KnowledgeCache;

let mut kg = KnowledgeCache::new();

// Add entities
let alice = kg.add_entity("Alice", "person", -1);
let bob = kg.add_entity("Bob", "person", -1);
let acme = kg.add_entity("Acme Corp", "company", -1);

// Add relations
kg.add_relation(alice, acme, "works_at", 1.0);
kg.add_relation(bob, acme, "works_at", 1.0);
kg.add_relation(alice, bob, "manages", 0.8);

// Traverse
let neighbors = kg.bfs_neighbors(alice, 2);  // 2-hop neighborhood

// Spreading activation — find related entities
let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5);

// Entity resolution — fuzzy matching
let (id, created) = kg.resolve_or_create("alice", "person", -1, 2);
// id == alice, created == false: matched the existing entity (Levenshtein distance ≤ 2)
```

### Memory Consolidation

```rust
use clawhdf5_agent::consolidation::*;

let config = ConsolidationConfig::default();
let mut engine = ConsolidationEngine::new(config);

let now = 1_700_000_000.0; // seconds since the epoch

// Add memories — automatically scored for importance.
// Elevated sources (System, …) go through a separate, explicit API.
let id = engine.add_memory("User prefers dark mode".into(), vec![0.1, 0.2, ...], UntrustedSource::User, now);
engine.add_trusted_memory("ok".into(), vec![0.0, 0.0, ...], TrustedSource::System, now);

// Access a memory (reactivates it)
engine.access_memory(id, now);

// Run consolidation cycle
engine.consolidate(now);
let stats = engine.get_stats();
// Working memories promote to Episodic (if important enough)
// Episodic memories promote to Semantic (if accessed enough)
// Low-decay memories get evicted when tiers are full
```

### Temporal Queries

```rust
use clawhdf5_agent::temporal::*;

let mut index = TemporalIndex::new();
index.insert(1, 1700000000.0);  // record 1 at timestamp
index.insert(2, 1700003600.0);  // record 2, 1 hour later

// Range query — "what happened between 2pm and 5pm?"
let ids = index.range_query(1700000000.0, 1700010800.0);

// Latest 10 memories
let recent = index.latest(10);
```

### OpenClaw Integration

```rust
use clawhdf5_agent::openclaw::*;

// Create backend
let mut backend = ClawhdfBackend::create(std::path::Path::new("memory.h5"), 384)?;

// Ingest existing Markdown memory files
let md = std::fs::read_to_string("MEMORY.md")?;
let count = backend.ingest_markdown("MEMORY.md", &md)?;

// Search (full pipeline: weighted vector + BM25 fusion → re-rank → confidence filter)
let results = backend.search("user preferences", &query_embedding, 5);

// Export back to Markdown
let exported = backend.export_markdown("MEMORY.md")?;
```

---

## Crate Map

```
clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
                     and benches; plus libaec-sys, an internal FFI bindings
                     crate for the optional szip feature)
│
├── Core HDF5
│   ├── clawhdf5-format      — Binary parser/writer (no_std-capable), shared type definitions
│   ├── clawhdf5-io          — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
│   ├── clawhdf5-filters     — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format
│   ├── clawhdf5-derive      — Proc macros
│   ├── clawhdf5             — High-level API
│   ├── clawhdf5-netcdf4     — NetCDF-4 support
│   ├── clawhdf5-accel       — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
│   └── clawhdf5-gpu         — GPU compute (wgpu, hand-written WGSL compute shaders)
│
├── Agent Memory
│   ├── clawhdf5-agent       — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor (default backend; f32 or int8 storage; `parallel` build)
│   ├── clawhdf5-migrate     — SQLite → HDF5 migration
│   ├── clawhdf5-android     — Android JNI bridge
│   └── clawhdf5-cli         — CLI tool
│
├── Bindings
│   ├── clawhdf5-py          — Python (PyO3)
│   └── clawhdf5-napi        — Node.js (napi-rs)
│
└── Tooling
    └── clawhdf5-bench       — Benchmark suite
```

---

## Research Foundation

ClawhDF5's agent memory design draws from 15+ recent papers:

| Paper | Key Insight | ClawhDF5 Module |
|-------|-------------|-----------------|
| **MemX** (2026) | Hybrid fusion + multi-factor re-ranking | `hybrid`, `reranker` |
| **Graph-Native Cognitive Memory** (2026) | Graph-structured memory (weighted, timestamped relations; entity timelines) | `knowledge`, `temporal` |
| **CraniMem** (2026) | Bounded hippocampal memory | `consolidation` |
| **D-MEM** (2026) | Surprise-gated storage (implemented as a novelty score) | `consolidation` |
| **SYNAPSE** (2025) | Spreading activation for recall | `knowledge` |
| **RAGdb** (2025) | Zero-dependency edge RAG | Architecture |
| **MemoryGraft** (2025) | Memory poisoning attacks | `anomaly`, `provenance` |
| **MemoryArena** (2026) | Multi-session benchmark | `temporal` |
| **AI Hippocampus** (2026) | Memory taxonomy survey | Overall design |

---

## Feature Flags

### `clawhdf5-agent`

| Flag | Default | Description |
|------|---------|-------------|
| `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). The store itself always writes `f32` embeddings; `MemoryConfig::float16` is recorded in `/meta` but not yet applied |
| `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
| `parallel` | **yes** | Parallel HNSW bulk build (same graph, ~3× faster on 16 cores) and Rayon brute-force search strategies |
| `zstd` | no | Compress embeddings with Zstd instead of deflate when `MemoryConfig::compression` is on (links libzstd) |
| `fast-math` | no | BLAS matrix-vector multiply |
| `accelerate` | no | Apple Accelerate / AMX (macOS) |
| `openblas` | no | OpenBLAS (Linux) |
| `gpu` | no | GPU search via wgpu |
| `async` | no | Tokio async with background flush |
| `agent` | no | Reserved; currently enables nothing (the agent layer is always built) |

To opt out of the parallel build: `--no-default-features --features float16,hnsw`.
For an exact linear cosine scan instead of HNSW: `--no-default-features --features float16`.

`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search` tune the
vector index (16 / 64 / scale-with-`k` by default) and are stored with the
file.

`MemoryConfig::quantized_index` (**on by default** for new stores) holds the
HNSW index's own copy of the embeddings as `i8`, roughly halving a loaded
store's memory (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised
distances are approximate, so the query path re-scores the candidate pool
against the exact embeddings the store already holds, which keeps recall at the
`f32` index's level. It is also **faster**: 1.63x the queries per second at
equal recall on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with
index builds 1.8x and 2.3x faster respectively. Stores created before the
setting existed keep their `f32` index; opt out for new stores with
`quantized_index = false` or `clawhdf5-cli create --f32-index`. See
[BENCHMARKS.md § Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index).

### `clawhdf5-format`

| Flag | Default | Description |
|------|---------|-------------|
| `std` | yes | Standard library (disable for `no_std`) |
| `deflate` | yes | Deflate compression |
| `checksum` | yes | Jenkins lookup3 verification |
| `provenance` | yes | SHA-256 provenance attributes |
| `fast-deflate` | **yes** | zlib-ng backend for faster deflate (C; needs `cmake`) |
| `system-zlib-decompress` | **yes** | Use Apple's system libz for decompression (macOS only; no effect elsewhere) |
| `parallel` | no | Parallel chunk encoding + compression (rayon) |
| `fast-checksum` | no | crc32fast-accelerated checksums |
| `lz4` | no | LZ4 block compression filter (id 32004) |
| `zstd` | no | Zstandard compression filter (id 32015) |
| `pcodec` | no | Pcodec lossless numerical codec (id 32023, via `pco` crate) |
| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
| `blake3_hash` | no | BLAKE3 content hashing for provenance |
| `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) |

### `clawhdf5-ann`

| Flag | Default | Description |
|------|---------|-------------|
| `parallel` | no | Batched bulk build runs neighbour planning and back-link pruning on a Rayon pool; the graph is identical with or without it (enabled by `clawhdf5-agent`'s default `parallel`) |

### `clawhdf5-io`

| Flag | Default | Description |
|------|---------|-------------|
| `mmap` | no | Memory-mapped reads (`memmap2`) |
| `async` | no | Tokio-based async I/O |
| `hsds` | no | HSDS (HDF REST service) client |
| `mpi-io` | no | MPI-backed I/O via the `mpi` crate |

> **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read
> followed by a broadcast, and its write path gathers all ranks' shards to
> rank 0 before writing — not true collective I/O
> (`MPI_File_read_at_all`/`write_at_all`). It does not provide I/O bandwidth
> that scales with rank count; true collective I/O is tracked as future work.

---

## Building

```bash
# Default (needs cmake + a C compiler for zlib-ng)
cargo build --workspace

# Agent memory with all accelerations (Linux)
cargo build -p clawhdf5-agent --features fast-math

# Agent memory with Apple Accelerate (macOS)
cargo build -p clawhdf5-agent --features "accelerate,gpu"

# Tests
cargo test --workspace            # all 1,850+ tests
cargo test -p clawhdf5-agent      # agent memory tests
scripts/ci-test.sh                # what CI runs: fmt, clippy matrix, tests,
                                  # h5py/netCDF4 interop, no_std

# The interop suites need a Python with h5py; on a PEP 668 system that has to
# be a virtualenv. `ci-test.sh` finds `.venv` on its own, or set
# CLAWHDF5_PYTHON. Without one they skip — set CLAWHDF5_REQUIRE_INTEROP=1 to
# make that a failure instead.
python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4 xarray

# Benchmarks
cargo bench -p clawhdf5-agent               # agent memory suite
cargo bench -p clawhdf5-bench               # h5bench-equivalent I/O suite
```

---

## HDF5 File Schema

```
agent_memory.h5
├── /meta                          (attributes)
│   ├── schema_version: "1.0", edgehdf5_version
│   ├── agent_id, embedder, embedding_dim, chunk_size, overlap, created_at
│   ├── float16, compression, compression_level, compact_threshold,
│   │   hebbian_boost, decay_factor, wal_enabled, wal_max_entries
│   ├── quantized_index, hnsw_m, hnsw_ef_construction, hnsw_ef_search
│   ├── wal_applied_len, wal_applied_crc   (WAL mark of the last checkpoint)
│   └── ann_generation                     (ties the .ann sidecar to this checkpoint)
├── /memory
│   ├── chunks:              string[N]
│   ├── embeddings:          f32[N × D]  (chunked; deflate, or Zstd with the
│   │                                     `zstd` feature, when compression is on)
│   ├── source_channel:      string[N]
│   ├── timestamps:          f64[N]
│   ├── session_ids:         string[N]
│   ├── tags:                string[N]
│   ├── tombstones:          u8[N]
│   ├── norms:               f32[N]      (pre-computed L2)
│   └── activation_weights:  f32[N]      (Hebbian)
├── /sessions
│   ├── ids, channels, summaries:  string[S]
│   ├── start_idxs, end_idxs:      i64[S]
│   └── timestamps:                f64[S]
└── /knowledge_graph
    ├── entity_ids, entity_emb_idxs: i64[E]; entity_names, entity_types: string[E]
    ├── relation_srcs, relation_tgts: i64[R]; relation_types: string[R]
    ├── relation_weights: f32[R]; relation_ts: f64[R]
    └── alias_strings: string[A]; alias_entity_ids: i64[A]   (when aliases exist)
```

Alongside the store: `<store>.h5.wal` (write-ahead log), `<store>.h5.ann`
(HNSW graph; derived, safe to delete) and `<store>.h5.lock` (single-writer
lock). A second writer gets `MemoryError::Locked`; use
`HDF5Memory::open_read_only` for a lock-free point-in-time view.

---

## Migration

### From rustyhdf5 / edgehdf5

Replace in `Cargo.toml` and source:

| Old | New |
|-----|-----|
| `rustyhdf5*` | `clawhdf5*` |
| `edgehdf5-memory` | `clawhdf5-agent` |
| `edgehdf5` (CLI) | `clawhdf5-cli` |

### From SQLite

```bash
cargo install --path crates/clawhdf5-migrate
clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384
```

---

## Roadmap

See [ROADMAP.md](ROADMAP.md) for the full implementation tracker.

**Phase 1 complete** — all 8 tracks delivered:
- ✅ Knowledge Graph with spreading activation
- ✅ Hippocampal memory consolidation
- ✅ RRF hybrid retrieval + re-ranking + confidence rejection
- ✅ Temporal reasoning with sub-µs queries
- ✅ Memory security + anomaly detection
- ✅ Multi-modal memory (text/image/audio/video)
- ✅ OpenClaw integration layer
- ✅ Comprehensive Criterion benchmarks

**Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: publish the OpenClaw TypeScript bridge to npm, crates.io/PyPI publishing.

---

## Part of the RedClaw Ecosystem

ClawhDF5 powers the `.brain` format for [ClawBrainHub](https://clawbrainhub.com) — the brain registry for AI agents. One file that packages identity, skills, memory, knowledge, and cryptographic provenance.

---

## License

MIT

---

<p align="center">
  <em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em><br>
  <em>~86,000 lines of Rust. No libhdf5. One file to remember everything.</em>
</p>