bench: re-run every stale BENCHMARKS.md section, dated and traced
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).
Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
same as today, so the old 18 us / 6.17 ms figures (undated, other
hardware) are not reproducible; float16 adds ~2 us per save and the
int8 index nothing (both isolated by switching the bench's config).
Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
migration section (from the previous migrate commit);
- CHANGELOG for this branch.
Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -6,7 +6,7 @@
|
||||
[](https://www.rust-lang.org)
|
||||
[](#building)
|
||||
[](BENCHMARKS.md#longmemeval-results)
|
||||
[](BENCHMARKS.md#memory-footprint-1)
|
||||
[](BENCHMARKS.md#memory-footprint-1)
|
||||
|
||||
ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
|
||||
|
||||
@@ -111,8 +111,8 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
|
||||
| Keyword search | Separate FTS engine | Integrated BM25 |
|
||||
| Knowledge graph | Neo4j or none | In-file graph with spreading activation |
|
||||
| Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
|
||||
| Temporal queries | Custom code | Native temporal index (716ns) |
|
||||
| Multi-modal | Multiple stores | Unified cross-modal search |
|
||||
| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
|
||||
| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
|
||||
| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
|
||||
| Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
|
||||
|
||||
@@ -120,7 +120,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
|
||||
|
||||
## Performance
|
||||
|
||||
Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note).
|
||||
The brute-force/IVF vector search, agent-memory, on-disk footprint and consolidation figures below were measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D, 8C/16T), commit 5c8323c, 384-dim embeddings; the commands are in [BENCHMARKS.md](BENCHMARKS.md). Exceptions are marked where they appear: the HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note), and the HNSW `f32`/`i8` table and the in-memory `i8` column were not re-measured on 2026-09-24.
|
||||
|
||||
### HDF5 Core I/O (vs libhdf5 1.14.6)
|
||||
|
||||
@@ -154,33 +154,45 @@ and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quant
|
||||
| `f32` index | 0.9945 | 13 399 | 3.2 s |
|
||||
| `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
|
||||
|
||||
Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.
|
||||
Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. These
|
||||
two rows are a paired comparison (medians of alternating runs, same binary).
|
||||
A single `f32` run on 2026-09-24 measured recall 0.9945, 19 001 QPS and a
|
||||
2.7 s build; the int8 row was not re-run, so the pair has not been re-checked
|
||||
([§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index)).
|
||||
|
||||
**Brute-force and IVF paths** (Criterion, i7-12650H):
|
||||
**Brute-force and IVF paths** (Criterion, tank, 2026-09-24):
|
||||
|
||||
| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
|
||||
| Scale | Flat | IVF (nprobe=10) | IVF-PQ | MemX¹ (claimed, end-to-end) |
|
||||
|-------|------|-----------------|--------|----------|
|
||||
| 1K | **54 µs** | — | — | — |
|
||||
| 10K | 753 µs | **27 µs** | — | — |
|
||||
| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) |
|
||||
| 1K | **47.4 µs** | — | — | — |
|
||||
| 10K | 500.5 µs | **24.8 µs** | — | — |
|
||||
| 100K | 6.58 ms | 592 µs | **869 µs** | <90 ms |
|
||||
|
||||
> Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected,
|
||||
> apples-to-apples SIMD/scalar/parallel comparison methodology — see
|
||||
> [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector
|
||||
> Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05).
|
||||
> These replace figures from the original i7-12650H run (flat 54 µs / 753 µs /
|
||||
> 11.4 ms); a 2026-08-05 run on tank had already matched the new ones — see
|
||||
> [BENCHMARKS.md § Vector Search Latency](BENCHMARKS.md#vector-search-latency).
|
||||
|
||||
### Agent Memory Operations
|
||||
|
||||
| Operation | Latency | Scale |
|
||||
|-----------|---------|-------|
|
||||
| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records |
|
||||
| BM25 keyword search | **67 µs** | 1K records |
|
||||
| Knowledge graph BFS | **24 µs** | 1K entities |
|
||||
| Spreading activation | **17 µs** | 100 entities |
|
||||
| Temporal range query | **716 ns** | 10K timestamps |
|
||||
| Consolidation cycle | **164 µs** | 1K records |
|
||||
| Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) |
|
||||
| Importance gate | **61 ns** | per record |
|
||||
| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **0.07 ms** / 0.49 ms / 4.69 ms | 1K / 10K / 100K records |
|
||||
| BM25 keyword search | **20.4 µs** | 1K records |
|
||||
| Knowledge graph BFS | **23.1 µs** | 1K entities |
|
||||
| Spreading activation | **10.1 µs** | 100 entities |
|
||||
| Temporal range query | **622 ns** | 10K timestamps |
|
||||
| Consolidation cycle | **115.2 µs** | 1K records |
|
||||
| Cross-modal search (exact scan, 2 embeddings per record) | **842.0 µs** / 8.44 ms | 1K / 10K records |
|
||||
| Memory write (WAL) | **26.1 µs** | per record (group-commit append; HDF5 batched at flush) |
|
||||
| Importance gate | **57.6 ns** | per record (trivial skip) |
|
||||
|
||||
The old 18 µs WAL write was undated, from another machine: v2.3.0 measures
|
||||
24.3 µs on the same hardware as this table, the same as an `f32` store today.
|
||||
`float16` stores (the new default) add ~2 µs for rounding; the int8 index adds
|
||||
nothing. See [BENCHMARKS.md § Write Path](BENCHMARKS.md#write-path).
|
||||
Knowledge-graph traversal was briefly 6.5x slower (155 µs) until this re-run
|
||||
found and fixed an adjacency index rebuilt on every traversal; see
|
||||
[§ Knowledge Graph](BENCHMARKS.md#knowledge-graph).
|
||||
|
||||
### Chunked Write Throughput (codec comparison)
|
||||
|
||||
@@ -195,7 +207,7 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data):
|
||||
|
||||
Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition.
|
||||
|
||||
> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
|
||||
> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search), so the two columns are not comparable and no ratio is given. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
|
||||
|
||||
### LongMemEval Retrieval Recall
|
||||
|
||||
@@ -248,18 +260,26 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
|
||||
|
||||
### Memory Footprint
|
||||
|
||||
**On disk** — 384-dim embeddings, 200-char text
|
||||
**On disk** — 384-dim `float16` embeddings (the default for new stores),
|
||||
200-char text, `footprint_bench`
|
||||
([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):
|
||||
|
||||
| Records | File Size | Bytes/Record | Gzip-6 compressed |
|
||||
|---------|-----------|--------------|-------------------|
|
||||
| 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) |
|
||||
| 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
|
||||
| 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |
|
||||
| 1K | 810.4 KB | 829 B | 56.4 KB |
|
||||
| 10K | 7.8 MB | 820 B | 471.3 KB |
|
||||
| 100K | 76.7 MB | 803 B | 4.5 MB |
|
||||
|
||||
These figures are `f32` embeddings. New agent stores default to
|
||||
`MemoryConfig::float16`, which halves them: 100K × 384 records take 80.8 MiB
|
||||
instead of 154.0.
|
||||
The benchmark's synthetic embeddings and text are far more repetitive than
|
||||
real data (only 40 distinct texts), so no column here is an expectation for
|
||||
real data. The compressed column is an upper bound, and the Bytes/Record
|
||||
column is optimistic too: it is not an uncompressed figure, because the store
|
||||
always deflates its text (any string dataset of 4 KiB or more) whatever
|
||||
`MemoryConfig::compression` says. The `float16` embeddings alone are 768 B per
|
||||
record, so 200 characters of real text would take a record above 820 B.
|
||||
This table used to show `f32` stores (1.7 KB per record, 169.8 MB at 100K);
|
||||
those were not re-measured. The float16 study compares the two on the same
|
||||
data: 100K × 384 records take 80.8 MiB as `float16` and 154.0 MiB as `f32`.
|
||||
|
||||
**In memory** — a store reopened from disk, 384-dim `f32`, measured with a
|
||||
counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
|
||||
@@ -271,7 +291,8 @@ counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-foo
|
||||
| 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |
|
||||
|
||||
Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
|
||||
embedding twice.
|
||||
embedding twice. The `f32` column was re-measured on 2026-09-24 and reproduced
|
||||
exactly; the `i8` column was not re-run.
|
||||
|
||||
### Consolidation Efficiency
|
||||
|
||||
@@ -282,7 +303,10 @@ embedding twice.
|
||||
|--------|--------|-------|-------|
|
||||
| Records in store | 1,000 | 100 | −90% |
|
||||
| Hit@1 recall (signal records) | 100% | 100% | no loss |
|
||||
| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** |
|
||||
| Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** |
|
||||
|
||||
The consolidation cycle that does this took 0.13 ms; at 10K records a cycle
|
||||
takes 2.16 ms.
|
||||
|
||||
**Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**
|
||||
|
||||
@@ -754,9 +778,37 @@ Replace in `Cargo.toml` and source:
|
||||
|
||||
```bash
|
||||
cargo install --path crates/clawhdf5-migrate
|
||||
clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384
|
||||
clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedder minilm
|
||||
```
|
||||
|
||||
The output is an ordinary `clawhdf5-agent` store, written through the agent's
|
||||
own API: open it with `HDF5Memory::open` (or `clawhdf5-cli --path memory.h5 …`)
|
||||
and search it straight away. What carries over from the ZeroClaw tables:
|
||||
|
||||
| SQLite | Agent store |
|
||||
|--------|-------------|
|
||||
| `memory_chunks` | memory records (text, embedding, source channel, timestamp, session id, tags); rows with `deleted = 1` become deleted records, or are left out with `--skip-deleted` |
|
||||
| `sessions` | sessions (id, start/end index, channel, summary, timestamp) |
|
||||
| `entities`, `relations` | knowledge graph entities and relations; entities get new ids and relations are re-pointed at them |
|
||||
|
||||
The chunk `id` column has no counterpart in the agent store, so records are
|
||||
written in `id` order and numbered from 0. Embeddings are stored as float16
|
||||
like any new store; `--f32` keeps full precision (and is required for values
|
||||
beyond ±65504). The embedding dimension is detected from the first row unless
|
||||
`--embedding-dim` is given, and every row must have it: a row of another length
|
||||
is an error, never truncated or padded. A source with no memory records (only
|
||||
sessions or the graph) needs `--embedding-dim`, since a store's dimension is
|
||||
fixed when it is created. Every row is checked before the output is created,
|
||||
so a source that cannot be migrated leaves an existing store at `--hdf5` as it
|
||||
was. `--incremental` adds to an existing store only the rows it does not
|
||||
already hold; the source must have the store's dimension, and records already
|
||||
in the store take the source's deleted flag (a row deleted in SQLite since the
|
||||
last run is deleted in the store; one un-deleted there is written again, as
|
||||
the agent has no un-delete). The tool reads the result back with
|
||||
`HDF5Memory::open_read_only`, compares it with the source (every row with
|
||||
`--validate-full`) and checks that a migrated record is found by search;
|
||||
`--dry-run` only counts the rows.
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
|
||||
Reference in New Issue
Block a user