clawhdf5/research/01-architecture-overview.md

ClawHDF5 Architecture Overview

Research brief — generated 2026-08-12

---

1. Project Identity

ClawHDF5 (package prefix clawhdf5-*) is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It ships zero C dependencies, targets no_std environments (embedded / WASM), and stores all agent state in a single portable .h5 file.

Current version: 2.1.0 (released 2026-06-03; unreleased work-in-progress is the effective HEAD).

Repository: Cargo workspace with 16 crates (plus libaec-sys, an internal FFI-bindings crate for the optional SZIP feature). Total size ~92K lines of Rust.

---

2. Crate Map

clawhdf5 workspace
│
├── Core HDF5
│   ├── clawhdf5-format     — Binary parser/writer (no_std), shared type defs
│   ├── clawhdf5-io         — I/O abstraction: buffered, mmap, async, MPI-IO stub
│   ├── clawhdf5-filters    — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip live in format
│   ├── clawhdf5-derive     — Proc-macro #[derive(HDF5)]
│   ├── clawhdf5            — High-level facade (File, Dataset, FileBuilder)
│   ├── clawhdf5-netcdf4    — NetCDF-4 compatibility shim
│   ├── clawhdf5-accel      — CPU SIMD (AVX2, AVX-512, NEON) acceleration
│   └── clawhdf5-gpu        — GPU compute via wgpu + hand-written WGSL shaders
│
├── Agent Memory
│   ├── clawhdf5-agent      — Memory engine (20.9K lines, 32 modules)
│   ├── clawhdf5-ann        — HNSW ANN index (default vector backend)
│   ├── clawhdf5-migrate    — SQLite → HDF5 migration tool
│   ├── clawhdf5-android    — Android JNI bridge
│   └── clawhdf5-cli        — CLI (create / save / search / recall / stats / …)
│
├── Bindings
│   ├── clawhdf5-py         — Python via PyO3 (pyo3/numpy 0.29)
│   └── clawhdf5-napi       — Node.js via napi-rs (@redclaw/clawhdf5 npm package)
│
└── Tooling
    └── clawhdf5-bench      — Criterion benchmark suite

---

3. HDF5 Format Layer (clawhdf5-format)

3.1 Parser Coverage

The format crate implements a ground-up HDF5 binary parser. Notable capabilities shipped as of HEAD:

Feature	Status
Superblock v0–v4 (incl. page-buffer mode)	✅ Full
B-tree v1 (symbol, chunk)	✅ Full
B-tree v2 (link-name index type 5)	✅ Full
Fractal heap (single-direct-block)	✅ Full
Fractal heap (multi-direct-block / root indirect)	✅ Full
Fractal heap (multi-level indirect)	❌ Not yet
Dense group link storage (fractal heap + v2 B-tree)	✅ Full
Dense attribute storage	✅ Full
Compact / contiguous / chunked data layouts	✅ Full
Fixed Array chunk index	✅ Full (incl. paged)
Extensible Array chunk index	⚠️ Partial (fixed rows only)
Virtual Datasets (same-file)	✅ Full
Virtual Datasets (external-file)	✅ Via VdsSourceResolver callback
Filter: deflate (zlib-ng fast path)	✅
Filter: shuffle	✅
Filter: fletcher32	✅
Filter: LZ4 (id 32004)	✅ (feature-gated)
Filter: Zstandard (id 32015)	✅ (feature-gated)
Filter: Pcodec (id 32023)	✅ (feature-gated)
Filter: N-Bit (id 5)	✅ Full (atomic, compound, array)
Filter: Scale-offset D-scale / integer (id 6)	✅ Full
Filter: Scale-offset E-scale (id 6, type 1)	✅ Full
Filter: SZIP (id 4)	✅ Feature-gated (szip via libaec-sys FFI)
Datatype: fixed-point (int)	✅ Full incl. reduced-precision + sign extension
Datatype: floating-point (f32/f64/f16)	✅ Full
Datatype: string (fixed/variable)	✅ Full
Datatype: compound (class 6, v1–v5)	✅ Full
Datatype: array (class 10, v1–v5)	✅ Full
Datatype: reference	⚠️ Partial

3.2 Write Path

• FileBuilder API for high-level file construction.
• Dense attribute/link writes via single-direct-block fractal heap + v2 B-tree (validated against h5py 3.16 / HDF5 2.0).
• Multi-direct-block write path shipped (root indirect block).
• Objects spanning blocks (huge-object path) not yet supported.
• Chunked write with parallel compression (rayon, parallel feature).
• Auto-shuffle (AoS→SoA byte transpose): +157–204% throughput on float data.

3.3 Chunk Cache

O(1) lookup via slot_index: HashMap. Cache hits return a shared Arc (no clone). Cache is scoped per-dataset to prevent cross-dataset index collisions.

---

4. Agent Memory Layer (clawhdf5-agent)

4.1 Module Map (32 modules)

Module	Responsibility
knowledge	Entity/relation graph; BFS; spreading activation; fuzzy entity resolution (Levenshtein)
consolidation	Three-tier memory (Working → Episodic → Semantic) with importance scoring and time-decay
hybrid	RRF (k=60) fusion of vector + BM25; exposes merge_vector_keyword
reranker	Multi-factor re-ranking: temporal recency, source authority, activation weight
confidence	Low-confidence rejection — suppresses spurious recalls
temporal	Sorted timestamp index, session DAG, entity timeline, temporal query hints
multimodal	Cross-modal search (text / image / audio / video)
provenance	FNV-1a content hash, SHA-256 attributes, source attribution
anomaly	15 injection-pattern detectors, write rate limiter, source distribution analysis
openclaw	MemoryBackend trait; Markdown ↔ HDF5 import/export
vector_search	Flat cosine, pre-normed, SIMD, BLAS, GPU paths
ivf / pq	IVF-PQ ANN for billion-scale search
bm25	BM25 keyword index with TF-IDF
entity_extract	Rule-based entity extraction from text chunks
wal	CRC32-per-entry WAL; WAL_VERSION 2; length-prefix caps (MAX_WAL_FIELD_LEN = 64 MiB)
memory_strategy	Pluggable strategies: save-every, semantic-shift, user-correction detection
decision_gate	Sub-microsecond trivial/substantive classification
async_memory	Tokio async wrapper (async feature)

4.2 HDF5 Schema

agent_memory.h5
├── /meta               — schema_version, agent_id, embedder, embedding_dim, created_at
├── /memory
│   ├── chunks:         string[N]
│   ├── embeddings:     f32[N × D]  (f16 with float16 flag — 2× space savings)
│   ├── tombstones:     u8[N]
│   └── norms:          f32[N]  (pre-computed L2)
├── /sessions
│   ├── ids:            string[S]
│   └── summaries:      string[S]
└── /knowledge_graph
    ├── entity_names:   string[E]
    ├── relation_srcs:  i64[R]
    ├── relation_tgts:  i64[R]
    └── relation_types: string[R]

4.3 HNSW Vector Index (clawhdf5-ann)

• Default vector backend for hybrid_search (on by default via hnsw feature).
• Mutable live index: insert, mark_deleted (soft-delete bitset), compact, serialization (format version 2).
• Self-healing: rebuilds on drift from memory cache length.
• Optional parallel feature (rayon) for prune_connections.
• Outer build/insert loop is deliberately sequential (cross-iteration data dependencies).
• Fallback: exact linear cosine scan via --no-default-features --features float16.

4.4 Retrieval Pipeline

Agent query
    │
    ▼
Hybrid search (HNSW vector + BM25)
    │
    ▼
RRF fusion (k=60)
    │
    ▼
Multi-factor re-ranking
  · temporal recency
  · source authority
  · spreading activation weight
    │
    ▼
Confidence rejection (min_score threshold + gap filter)
    │
    ▼
Results

LongMemEval results (full longmemeval_s haystack, 500 questions):

• BM25 only: 75.0% turn-level Hit@5
• Vector only (MiniLM): 71.8%
• Hybrid (weights 0.4/0.6 — tuned): 81.4%

---

5. Cross-Language Bindings

Binding	Crate	Status
Python	clawhdf5-py (PyO3 0.29 / numpy 0.29)	Build works locally; wheels not published
Node.js	clawhdf5-napi + packages/clawhdf5-node	Complete package; not published to npm
Android	clawhdf5-android (JNI)	Shipped; bounds/null checks added for JNI unsafe

---

6. CI/CD

.gitea/workflows/ci.yml runs scripts/ci-test.sh on every push/PR to main:

• rustfmt check
• clippy (zero warnings)
• Full test suite (cargo test --workspace, 1,650+ tests)
• no_std check (scripts/check-nostd.sh)

---

7. Key Design Decisions

1. Zero C dependencies — enables no_std, static linking, cross-compilation, and eliminates the HDF5 C library as an attack surface. Tradeoff: manual implementation of every HDF5 format detail.
2. Single-file storage — all agent state (vectors, BM25 index, knowledge graph, WAL) lives in one .h5 file. Portability > convenience for multi-component setups.
3. CRC32 per WAL entry — crash safety without journaling overhead; corrupted entry stops replay cleanly.
4. float16 storage — 2× space savings on embeddings; defaults on.
5. HNSW on by default — sub-millisecond ANN at 10K–100K vectors; exact scan always available as fallback.
6. parallel feature off by default — correctness-safe default; enables Rayon where safe (chunk compression, HNSW prune_connections).