bench: re-run every stale BENCHMARKS.md section, dated and traced

Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).

Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
  same as today, so the old 18 us / 6.17 ms figures (undated, other
  hardware) are not reproducible; float16 adds ~2 us per save and the
  int8 index nothing (both isolated by switching the bench's config).

Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
  README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
  takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
  migration section (from the previous migrate commit);
- CHANGELOG for this branch.

Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-24 23:45:15 -05:00
co-authored by Claude Opus 5.5
parent 1b3bbb054a
commit dce5559ff2
6 changed files with 880 additions and 175 deletions
+51
View File
@@ -3,6 +3,17 @@
## Unreleased
### Upgrade Notes
- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be
a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no
`/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be
used as agent memory. Files it wrote before this release are not agent
stores; re-run the migration. Also: embeddings default to `float16` like
any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with
the wrong embedding length is an error instead of being truncated or
padded; `--incremental` now matches rows by content against an existing
store and follows the source's deleted flags; a source with no memory rows
needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of
the old layout are gone (the agent schema has no place for them).
- **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
dataset we wrote — including every agent store's embeddings — was refused
with "sign bit position out of bounds", and every empty dataset with
@@ -46,6 +57,24 @@
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
### Migration
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
/ `open`, `save_batch`, the session cache and knowledge graph), so there is
no second copy of the schema. Sessions and entities/relations carry over;
deleted rows become deleted records (or are left out with
`--skip-deleted`). Every source row is checked before the output is created,
so a source that cannot be migrated leaves an existing store untouched.
Validation reads the result back with `HDF5Memory::open_read_only`, compares
every field (embeddings bit for bit — `round_to_f16` of the source for a
`float16` store) and checks that a migrated record is found by search. The
`half`-based conversion is gone; `clawhdf5_format::float16` is the only one.
42 tests, including h5py opening a migrated store; an adversarial review's
two blocker and four major findings are fixed with regression tests.
- `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`,
`HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never
auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry`
re-exported from the crate root.
### Search
- `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
filtering, re-ranking and confidence rejection in the store's own search
@@ -68,6 +97,20 @@
activation of the `k` results it returns, not of the whole `3k` candidate
pool it re-ranks.
### Benchmarks
- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
each and every number traced back to the raw output by a separate check.
Where a figure moved, the section says so. Two apparent regressions were
isolated rather than published: knowledge-graph traversal (a real bug,
fixed above) and the write path, which measures the same at v2.3.0 on this
machine — the old 18 µs / 6.17 ms figures came from an undated run on other
hardware; `float16` adds ~2 µs per save and the int8 index nothing.
- New `multimodal_bench`: cross-modal search at 1K and 10K records, which the
README claimed but nothing measured.
- `footprint_bench` reports whether it built `float16` or `f32` stores and
takes `--f32`; it had kept printing "f32" after the default changed.
### Interop
- `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
libhdf5.** The float datatype encoder hard-coded the sign bit's position to
@@ -146,6 +189,14 @@
knew to ask; it now only ever switches the default off.
### Performance
- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K
entities took 155 µs. The index is now cached on `KnowledgeCache` and
checked against a fingerprint of the graph on each use — one pass over
entity ids and relation endpoints, no allocation — so any change, including
direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K
entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
- `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
whole chunk in one call, into a buffer allocated once, instead of streaming
it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's