docs: the int8 index is faster than f32 on AVX2, not slower

Measured with `clawhdf5_accel::dot_i8` in place, medians of three
alternating runs at N = 100 000 x 384, same binary:

  build      f32 3197 ms   int8 1778 ms   int8+re-score 1826 ms
  ef = 64    f32 13 399 QPS @ 0.9945   int8+re-score 21 848 QPS @ 0.9940

So at equal recall the quantised index is 1.63x the queries per second
and 1.8x the build speed, holding a quarter of the vectors. The earlier
"~13% of QPS" figure compared a scalar int8 loop against hand-written
AVX2 f32 kernels and was measuring the missing kernel; it is kept in
BENCHMARKS.md with that explanation rather than quietly replaced.

Still off by default, now for portability rather than performance: the
kernel is AVX2-only and aarch64 falls back to scalar, where the original
trade applies. A NEON kernel would settle it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-20 17:30:56 -07:00
co-authored by Claude Opus 5
parent 97e65f2adf
commit dea02f5214
4 changed files with 48 additions and 9 deletions
+5 -2
View File
@@ -437,8 +437,11 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
copy of the embeddings as `i8`, roughly halving a loaded store's memory
(2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are
approximate, so the query path re-scores the candidate pool against the exact
embeddings the store already holds recall matches the `f32` index, at about
13% fewer queries per second. See `BENCHMARKS.md`, "Quantising the index copy".
embeddings the store already holds, which keeps recall at the `f32` index's
level. On AVX2 it is also **faster** — 1.63x the queries per second and 1.8x
the build speed at equal recall — because the int8 kernel is SIMD too. It
stays off by default only because that kernel is AVX2-only and aarch64 falls
back to a scalar loop. See `BENCHMARKS.md`, "Quantising the index copy".
| `parallel` | no | Rayon parallel search |
| `fast-math` | no | BLAS matrix-vector multiply |
| `accelerate` | no | Apple Accelerate / AMX (macOS) |