docs: measured ARM numbers, and a correction
On a Raspberry Pi 5 at N = 100 000 and equal recall (0.9940 vs 0.9945), medians of three runs: f32 33 413 ms build 6 164 QPS int8 scalar 18 950 ms ~6 190 QPS (what v2.7.0 shipped) int8 NEON ~17 000 ms 6 640 QPS int8 SDOT 14 464 ms 7 267 QPS 1.18x f32, 2.3x build The docs said quantised search stayed off by default because aarch64 "falls back to the scalar loop, where the original trade still applies" — that it was ~13% slower than f32 there, as on x86. That was extrapolated rather than measured, and it was wrong: x86's portable baseline is SSE2 against hand-written AVX2 f32 kernels, but on aarch64 NEON is the baseline and the scalar loop vectorises well, so it already matched f32. Corrected in BENCHMARKS.md, README.md and CLAUDE.md; the released v2.7.0 changelog entry is left as it was and the correction is recorded in a new one. Labelled as Pi 5 figures throughout — a Pi's memory bandwidth and cache are far below an M-series or flagship phone, so the ratios will move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
+31
-5
@@ -108,11 +108,37 @@ second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare
|
||||
only at equal `ef`: with re-scoring the harness raises `ef` to at least the
|
||||
candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.)
|
||||
|
||||
It is still **off by default**, for portability rather than performance: the
|
||||
int8 kernel is AVX2-only, and on aarch64 — including `clawhdf5-android` — it
|
||||
falls back to the scalar loop, where the original trade still applies. A NEON
|
||||
kernel would remove that caveat. On an x86-64 deployment, turning it on is a
|
||||
win on every axis measured.
|
||||
#### On ARM (Raspberry Pi 5, Cortex-A76)
|
||||
|
||||
`dot_i8` has two aarch64 kernels: `SDOT` for CPUs with the ARMv8.2
|
||||
dot-product extension (Cortex-A76 and later, Neoverse-N1, all Apple Silicon)
|
||||
and plain NEON (`vmull_s8` + `vpadalq_s16`) otherwise. Medians of three runs
|
||||
at N = 100 000, ef = 64, recall@10 0.9940 in every int8 row against f32's
|
||||
0.9945:
|
||||
|
||||
| int8 kernel | build | QPS | vs f32 |
|
||||
|---|---:|---:|---:|
|
||||
| *(f32 baseline)* | 33 413 ms | 6 164 | 1.00x |
|
||||
| scalar (what v2.7.0 shipped) | 18 950 ms | ~6 190 | 1.00x |
|
||||
| plain NEON | ~17 000 ms | 6 640 | 1.08x |
|
||||
| **SDOT** | **14 464 ms** | **7 267** | **1.18x** |
|
||||
|
||||
These are Pi 5 numbers, not "ARM" numbers: a Pi has far less memory bandwidth
|
||||
and cache than an Apple M-series or a flagship phone, so the ratios will move
|
||||
on other hardware. The plain-NEON row is that code on an A76 with `SDOT`
|
||||
disabled, not a measurement of a pre-A76 core.
|
||||
|
||||
**A correction.** Until this was measured, this section said aarch64 "falls
|
||||
back to the scalar loop, where the original trade still applies" — that is,
|
||||
that quantised search was ~13% slower than f32 on ARM. That was extrapolated
|
||||
from x86 and it was wrong. On x86-64 the portable baseline is SSE2 while the
|
||||
f32 kernels are hand-written AVX2, so scalar int8 lost; on aarch64 NEON *is*
|
||||
the baseline, the compiler vectorises the scalar loop well, and scalar int8
|
||||
already matched f32 for search while building 1.76x faster.
|
||||
|
||||
So on every configuration measured — x86-64 AVX2, and Pi 5 with each of the
|
||||
three int8 kernels — the quantised index is at least as fast as f32 at equal
|
||||
recall, builds faster, and holds a quarter of the vectors.
|
||||
|
||||
A measurement trap worth recording: the synthetic `clustered` generator in the
|
||||
`clawhdf5-ann` tests draws clusters far tighter than any real embedding, so
|
||||
|
||||
Reference in New Issue
Block a user