clawhdf5/research/01-architecture-overview.md

# ClawHDF5 Architecture Overview

*Research brief — generated 2026-08-12*

---

## 1. Project Identity

ClawHDF5 (package prefix `clawhdf5-*`) is a **pure-Rust HDF5 implementation** combined with a **research-grade agent memory engine**. It ships zero C dependencies, targets `no_std` environments (embedded / WASM), and stores all agent state in a single portable `.h5` file.

Current version: **2.1.0** (released 2026-06-03; unreleased work-in-progress is the effective HEAD).

Repository: Cargo workspace with **16 crates** (plus `libaec-sys`, an internal FFI-bindings crate for the optional SZIP feature). Total size ~92K lines of Rust.

---

## 2. Crate Map

```
clawhdf5 workspace
│
├── Core HDF5
│   ├── clawhdf5-format     — Binary parser/writer (no_std), shared type defs
│   ├── clawhdf5-io         — I/O abstraction: buffered, mmap, async, MPI-IO stub
│   ├── clawhdf5-filters    — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip live in format
│   ├── clawhdf5-derive     — Proc-macro #[derive(HDF5)]
│   ├── clawhdf5            — High-level facade (File, Dataset, FileBuilder)
│   ├── clawhdf5-netcdf4    — NetCDF-4 compatibility shim
│   ├── clawhdf5-accel      — CPU SIMD (AVX2, AVX-512, NEON) acceleration
│   └── clawhdf5-gpu        — GPU compute via wgpu + hand-written WGSL shaders
│
├── Agent Memory
│   ├── clawhdf5-agent      — Memory engine (20.9K lines, 32 modules)
│   ├── clawhdf5-ann        — HNSW ANN index (default vector backend)
│   ├── clawhdf5-migrate    — SQLite → HDF5 migration tool
│   ├── clawhdf5-android    — Android JNI bridge
│   └── clawhdf5-cli        — CLI (create / save / search / recall / stats / …)
│
├── Bindings
│   ├── clawhdf5-py         — Python via PyO3 (pyo3/numpy 0.29)
│   └── clawhdf5-napi       — Node.js via napi-rs (@redclaw/clawhdf5 npm package)
│
└── Tooling
    └── clawhdf5-bench      — Criterion benchmark suite
```

---

## 3. HDF5 Format Layer (`clawhdf5-format`)

### 3.1 Parser Coverage

The format crate implements a ground-up HDF5 binary parser. Notable capabilities shipped as of HEAD:

| Feature | Status |
|---------|--------|
| Superblock v0–v4 (incl. page-buffer mode) | ✅ Full |
| B-tree v1 (symbol, chunk) | ✅ Full |
| B-tree v2 (link-name index type 5) | ✅ Full |
| Fractal heap (single-direct-block) | ✅ Full |
| Fractal heap (multi-direct-block / root indirect) | ✅ Full |
| Fractal heap (multi-level indirect) | ❌ Not yet |
| Dense group link storage (fractal heap + v2 B-tree) | ✅ Full |
| Dense attribute storage | ✅ Full |
| Compact / contiguous / chunked data layouts | ✅ Full |
| Fixed Array chunk index | ✅ Full (incl. paged) |
| Extensible Array chunk index | ⚠️ Partial (fixed rows only) |
| Virtual Datasets (same-file) | ✅ Full |
| Virtual Datasets (external-file) | ✅ Via `VdsSourceResolver` callback |
| Filter: deflate (zlib-ng fast path) | ✅ |
| Filter: shuffle | ✅ |
| Filter: fletcher32 | ✅ |
| Filter: LZ4 (id 32004) | ✅ (feature-gated) |
| Filter: Zstandard (id 32015) | ✅ (feature-gated) |
| Filter: Pcodec (id 32023) | ✅ (feature-gated) |
| Filter: N-Bit (id 5) | ✅ Full (atomic, compound, array) |
| Filter: Scale-offset D-scale / integer (id 6) | ✅ Full |
| Filter: Scale-offset E-scale (id 6, type 1) | ✅ Full |
| Filter: SZIP (id 4) | ✅ Feature-gated (`szip` via `libaec-sys` FFI) |
| Datatype: fixed-point (int) | ✅ Full incl. reduced-precision + sign extension |
| Datatype: floating-point (f32/f64/f16) | ✅ Full |
| Datatype: string (fixed/variable) | ✅ Full |
| Datatype: compound (class 6, v1–v5) | ✅ Full |
| Datatype: array (class 10, v1–v5) | ✅ Full |
| Datatype: reference | ⚠️ Partial |

### 3.2 Write Path

- `FileBuilder` API for high-level file construction.
- Dense attribute/link writes via single-direct-block fractal heap + v2 B-tree (validated against h5py 3.16 / HDF5 2.0).
- Multi-direct-block write path shipped (root indirect block).
- Objects spanning blocks (huge-object path) not yet supported.
- Chunked write with parallel compression (rayon, `parallel` feature).
- Auto-shuffle (AoS→SoA byte transpose): +157–204% throughput on float data.

### 3.3 Chunk Cache

O(1) lookup via `slot_index: HashMap`. Cache hits return a shared `Arc` (no clone). Cache is scoped per-dataset to prevent cross-dataset index collisions.

---

## 4. Agent Memory Layer (`clawhdf5-agent`)

### 4.1 Module Map (32 modules)

| Module | Responsibility |
|--------|----------------|
| `knowledge` | Entity/relation graph; BFS; spreading activation; fuzzy entity resolution (Levenshtein) |
| `consolidation` | Three-tier memory (Working → Episodic → Semantic) with importance scoring and time-decay |
| `hybrid` | RRF (k=60) fusion of vector + BM25; exposes `merge_vector_keyword` |
| `reranker` | Multi-factor re-ranking: temporal recency, source authority, activation weight |
| `confidence` | Low-confidence rejection — suppresses spurious recalls |
| `temporal` | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
| `multimodal` | Cross-modal search (text / image / audio / video) |
| `provenance` | FNV-1a content hash, SHA-256 attributes, source attribution |
| `anomaly` | 15 injection-pattern detectors, write rate limiter, source distribution analysis |
| `openclaw` | `MemoryBackend` trait; Markdown ↔ HDF5 import/export |
| `vector_search` | Flat cosine, pre-normed, SIMD, BLAS, GPU paths |
| `ivf` / `pq` | IVF-PQ ANN for billion-scale search |
| `bm25` | BM25 keyword index with TF-IDF |
| `entity_extract` | Rule-based entity extraction from text chunks |
| `wal` | CRC32-per-entry WAL; `WAL_VERSION` 2; length-prefix caps (`MAX_WAL_FIELD_LEN` = 64 MiB) |
| `memory_strategy` | Pluggable strategies: save-every, semantic-shift, user-correction detection |
| `decision_gate` | Sub-microsecond trivial/substantive classification |
| `async_memory` | Tokio async wrapper (`async` feature) |

### 4.2 HDF5 Schema

```
agent_memory.h5
├── /meta               — schema_version, agent_id, embedder, embedding_dim, created_at
├── /memory
│   ├── chunks:         string[N]
│   ├── embeddings:     f32[N × D]  (f16 with float16 flag — 2× space savings)
│   ├── tombstones:     u8[N]
│   └── norms:          f32[N]  (pre-computed L2)
├── /sessions
│   ├── ids:            string[S]
│   └── summaries:      string[S]
└── /knowledge_graph
    ├── entity_names:   string[E]
    ├── relation_srcs:  i64[R]
    ├── relation_tgts:  i64[R]
    └── relation_types: string[R]
```

### 4.3 HNSW Vector Index (`clawhdf5-ann`)

- Default vector backend for `hybrid_search` (on by default via `hnsw` feature).
- Mutable live index: `insert`, `mark_deleted` (soft-delete bitset), `compact`, serialization (format version 2).
- Self-healing: rebuilds on drift from memory cache length.
- Optional `parallel` feature (rayon) for `prune_connections`.
- Outer build/insert loop is deliberately sequential (cross-iteration data dependencies).
- Fallback: exact linear cosine scan via `--no-default-features --features float16`.

### 4.4 Retrieval Pipeline

```
Agent query
    │
    ▼
Hybrid search (HNSW vector + BM25)
    │
    ▼
RRF fusion (k=60)
    │
    ▼
Multi-factor re-ranking
  · temporal recency
  · source authority
  · spreading activation weight
    │
    ▼
Confidence rejection (min_score threshold + gap filter)
    │
    ▼
Results
```

LongMemEval results (full `longmemeval_s` haystack, 500 questions):
- BM25 only: 75.0% turn-level Hit@5
- Vector only (MiniLM): 71.8%
- Hybrid (weights 0.4/0.6 — tuned): **81.4%**

---

## 5. Cross-Language Bindings

| Binding | Crate | Status |
|---------|-------|--------|
| Python | `clawhdf5-py` (PyO3 0.29 / numpy 0.29) | Build works locally; wheels not published |
| Node.js | `clawhdf5-napi` + `packages/clawhdf5-node` | Complete package; not published to npm |
| Android | `clawhdf5-android` (JNI) | Shipped; bounds/null checks added for JNI unsafe |

---

## 6. CI/CD

`.gitea/workflows/ci.yml` runs `scripts/ci-test.sh` on every push/PR to `main`:
- `rustfmt` check
- `clippy` (zero warnings)
- Full test suite (`cargo test --workspace`, 1,650+ tests)
- `no_std` check (`scripts/check-nostd.sh`)

---

## 7. Key Design Decisions

1. **Zero C dependencies** — enables `no_std`, static linking, cross-compilation, and eliminates the HDF5 C library as an attack surface. Tradeoff: manual implementation of every HDF5 format detail.
2. **Single-file storage** — all agent state (vectors, BM25 index, knowledge graph, WAL) lives in one `.h5` file. Portability > convenience for multi-component setups.
3. **CRC32 per WAL entry** — crash safety without journaling overhead; corrupted entry stops replay cleanly.
4. **`float16` storage** — 2× space savings on embeddings; defaults on.
5. **HNSW on by default** — sub-millisecond ANN at 10K–100K vectors; exact scan always available as fallback.
6. **`parallel` feature off by default** — correctness-safe default; enables Rayon where safe (chunk compression, HNSW `prune_connections`).