docs: the int8 index is faster than f32 on AVX2, not slower

Measured with `clawhdf5_accel::dot_i8` in place, medians of three
alternating runs at N = 100 000 x 384, same binary:

  build      f32 3197 ms   int8 1778 ms   int8+re-score 1826 ms
  ef = 64    f32 13 399 QPS @ 0.9945   int8+re-score 21 848 QPS @ 0.9940

So at equal recall the quantised index is 1.63x the queries per second
and 1.8x the build speed, holding a quarter of the vectors. The earlier
"~13% of QPS" figure compared a scalar int8 loop against hand-written
AVX2 f32 kernels and was measuring the missing kernel; it is kept in
BENCHMARKS.md with that explanation rather than quietly replaced.

Still off by default, now for portability rather than performance: the
kernel is AVX2-only and aarch64 falls back to scalar, where the original
trade applies. A NEON kernel would settle it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-20 17:30:56 -07:00
co-authored by Claude Opus 5
parent 97e65f2adf
commit dea02f5214
4 changed files with 48 additions and 9 deletions
+25 -5
View File
@@ -88,11 +88,31 @@ back**, because the loss is in the distances rather than in the graph
Re-scoring closes the gap: the store already holds the exact embeddings, so
the query path re-scores the candidate pool against them before fusion. That
is done automatically whenever the index is quantised. What it costs is
throughput — about 13% of QPS and 16% of build time at 100 000 x 384. So the
setting trades ~13% of query speed for ~36% of the process's memory at equal
recall. It is **off by default**: the right side of that trade depends on
whether the deployment is short of memory or short of CPU.
is done automatically whenever the index is quantised.
**On AVX2 this costs nothing — it pays.** The first measurement of this put
the cost at ~13% of QPS and ~16% of build time, but that compared a scalar
int8 loop against `clawhdf5-accel`'s hand-written AVX2 kernels for `f32`:
the gap was a missing kernel, not a property of int8. With
`clawhdf5_accel::dot_i8` (AVX2: sign-extend to `i16`, then `madd_epi16`),
medians of three alternating runs at N = 100 000, same binary:
| | f32 | int8 | int8 + re-score |
|---|---:|---:|---:|
| build | 3197 ms | **1778 ms** | 1826 ms |
| QPS at ef = 64 | 13 399 | 29 195 | **21 848** |
| recall@10 at ef = 64 | 0.9945 | 0.9625 | **0.9940** |
So at equal recall the quantised index answers **1.63x as many queries per
second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare
only at equal `ef`: with re-scoring the harness raises `ef` to at least the
candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.)
It is still **off by default**, for portability rather than performance: the
int8 kernel is AVX2-only, and on aarch64 — including `clawhdf5-android` — it
falls back to the scalar loop, where the original trade still applies. A NEON
kernel would remove that caveat. On an x86-64 deployment, turning it on is a
win on every axis measured.
A measurement trap worth recording: the synthetic `clustered` generator in the
`clawhdf5-ann` tests draws clusters far tighter than any real embedding, so