bench: re-run every stale BENCHMARKS.md section, dated and traced
Every undated or pre-September section re-run on one machine on one day
(tank, AMD Ryzen 7 7800X3D, 2026-09-24, commit 5c8323c), 24 commands run
serially with the load average checked before each, with the command
recorded for each section. A separate check traced every changed number
back to the raw output; its corrections are applied (e.g. the on-disk
~820 B/record is float16 plus always-deflated text on a synthetic corpus
of 40 distinct texts, not float16 alone).
Two apparent regressions were isolated rather than published:
- knowledge-graph traversal: a real bug, fixed in the previous commit;
- the write path: v2.3.0 built and run on the same machine measures the
same as today, so the old 18 us / 6.17 ms figures (undated, other
hardware) are not reproducible; float16 adds ~2 us per save and the
int8 index nothing (both isolated by switching the bench's config).
Also:
- new multimodal_bench: cross-modal search at 1K/10K records, which the
README claimed but nothing measured;
- footprint_bench reports whether it built float16 or f32 stores and
takes --f32 (it kept printing "f32" after the default changed);
- README: performance tables, the "Why" table figures and the SQLite
migration section (from the previous migrate commit);
- CHANGELOG for this branch.
Not re-run: consolidation_efficiency's 100K row and its memory-reduction
part (stopped for time), and cross_platform.sh.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -3,6 +3,17 @@
|
||||
## Unreleased
|
||||
|
||||
### Upgrade Notes
|
||||
- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be
|
||||
a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no
|
||||
`/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be
|
||||
used as agent memory. Files it wrote before this release are not agent
|
||||
stores; re-run the migration. Also: embeddings default to `float16` like
|
||||
any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with
|
||||
the wrong embedding length is an error instead of being truncated or
|
||||
padded; `--incremental` now matches rows by content against an existing
|
||||
store and follows the source's deleted flags; a source with no memory rows
|
||||
needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of
|
||||
the old layout are gone (the agent schema has no place for them).
|
||||
- **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
|
||||
dataset we wrote — including every agent store's embeddings — was refused
|
||||
with "sign bit position out of bounds", and every empty dataset with
|
||||
@@ -46,6 +57,24 @@
|
||||
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
|
||||
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
|
||||
|
||||
### Migration
|
||||
- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
|
||||
/ `open`, `save_batch`, the session cache and knowledge graph), so there is
|
||||
no second copy of the schema. Sessions and entities/relations carry over;
|
||||
deleted rows become deleted records (or are left out with
|
||||
`--skip-deleted`). Every source row is checked before the output is created,
|
||||
so a source that cannot be migrated leaves an existing store untouched.
|
||||
Validation reads the result back with `HDF5Memory::open_read_only`, compares
|
||||
every field (embeddings bit for bit — `round_to_f16` of the source for a
|
||||
`float16` store) and checks that a migrated record is found by search. The
|
||||
`half`-based conversion is gone; `clawhdf5_format::float16` is the only one.
|
||||
42 tests, including h5py opening a migrated store; an adversarial review's
|
||||
two blocker and four major findings are fixed with regression tests.
|
||||
- `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`,
|
||||
`HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never
|
||||
auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry`
|
||||
re-exported from the crate root.
|
||||
|
||||
### Search
|
||||
- `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
|
||||
filtering, re-ranking and confidence rejection in the store's own search
|
||||
@@ -68,6 +97,20 @@
|
||||
activation of the `k` results it returns, not of the whole `3k` candidate
|
||||
pool it re-ranks.
|
||||
|
||||
### Benchmarks
|
||||
- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
|
||||
machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
|
||||
each and every number traced back to the raw output by a separate check.
|
||||
Where a figure moved, the section says so. Two apparent regressions were
|
||||
isolated rather than published: knowledge-graph traversal (a real bug,
|
||||
fixed above) and the write path, which measures the same at v2.3.0 on this
|
||||
machine — the old 18 µs / 6.17 ms figures came from an undated run on other
|
||||
hardware; `float16` adds ~2 µs per save and the int8 index nothing.
|
||||
- New `multimodal_bench`: cross-modal search at 1K and 10K records, which the
|
||||
README claimed but nothing measured.
|
||||
- `footprint_bench` reports whether it built `float16` or `f32` stores and
|
||||
takes `--f32`; it had kept printing "f32" after the default changed.
|
||||
|
||||
### Interop
|
||||
- `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
|
||||
libhdf5.** The float datatype encoder hard-coded the sign bit's position to
|
||||
@@ -146,6 +189,14 @@
|
||||
knew to ask; it now only ever switches the default off.
|
||||
|
||||
### Performance
|
||||
- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
|
||||
should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
|
||||
index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K
|
||||
entities took 155 µs. The index is now cached on `KnowledgeCache` and
|
||||
checked against a fingerprint of the graph on each use — one pass over
|
||||
entity ids and relation endpoints, no allocation — so any change, including
|
||||
direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K
|
||||
entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
|
||||
- `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
|
||||
whole chunk in one call, into a buffer allocated once, instead of streaming
|
||||
it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
|
||||
|
||||
Reference in New Issue
Block a user