diff --git a/BENCHMARKS.md b/BENCHMARKS.md index bfea271..7668fb8 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -88,11 +88,31 @@ back**, because the loss is in the distances rather than in the graph Re-scoring closes the gap: the store already holds the exact embeddings, so the query path re-scores the candidate pool against them before fusion. That -is done automatically whenever the index is quantised. What it costs is -throughput — about 13% of QPS and 16% of build time at 100 000 x 384. So the -setting trades ~13% of query speed for ~36% of the process's memory at equal -recall. It is **off by default**: the right side of that trade depends on -whether the deployment is short of memory or short of CPU. +is done automatically whenever the index is quantised. + +**On AVX2 this costs nothing — it pays.** The first measurement of this put +the cost at ~13% of QPS and ~16% of build time, but that compared a scalar +int8 loop against `clawhdf5-accel`'s hand-written AVX2 kernels for `f32`: +the gap was a missing kernel, not a property of int8. With +`clawhdf5_accel::dot_i8` (AVX2: sign-extend to `i16`, then `madd_epi16`), +medians of three alternating runs at N = 100 000, same binary: + +| | f32 | int8 | int8 + re-score | +|---|---:|---:|---:| +| build | 3197 ms | **1778 ms** | 1826 ms | +| QPS at ef = 64 | 13 399 | 29 195 | **21 848** | +| recall@10 at ef = 64 | 0.9945 | 0.9625 | **0.9940** | + +So at equal recall the quantised index answers **1.63x as many queries per +second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare +only at equal `ef`: with re-scoring the harness raises `ef` to at least the +candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.) + +It is still **off by default**, for portability rather than performance: the +int8 kernel is AVX2-only, and on aarch64 — including `clawhdf5-android` — it +falls back to the scalar loop, where the original trade still applies. A NEON +kernel would remove that caveat. On an x86-64 deployment, turning it on is a +win on every axis measured. A measurement trap worth recording: the synthetic `clustered` generator in the `clawhdf5-ann` tests draws clusters far tighter than any real embedding, so diff --git a/CHANGELOG.md b/CHANGELOG.md index e4ea2a1..da4972d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,20 @@ ## Unreleased +### Performance +- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2: + sign-extend each half to `i16`, then `madd_epi16`; scalar fallback + elsewhere). The quantised HNSW index used a scalar loop while the `f32` path + it was measured against ran AVX2, so the ~13% throughput cost recorded for + `MemoryConfig::quantized_index` was a missing kernel rather than a property + of int8. With the kernel, at N = 100 000 x 384 and equal recall, the + quantised index answers **1.63x as many queries per second** (21 848 vs + 13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778 + vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three + alternating runs. It remains off by default only because the kernel is + AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so + the SIMD path is tested to agree with scalar bit for bit. + ### Correctness - `clawhdf5-format`: **datasets indexed by an Extensible Array returned wrong data beyond their first few dozen chunks.** One unlimited dimension gives a diff --git a/CLAUDE.md b/CLAUDE.md index 60bd9a0..b295dd3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -44,8 +44,10 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors at 100K); because quantised distances are approximate and `ef` cannot compensate, the query path then re-scores the candidate pool against the - exact embeddings, which holds recall at the f32 index's level and costs - ~13% of QPS. `hybrid_search` keeps one incremental BM25 + exact embeddings, which holds recall at the f32 index's level. On AVX2 it is + also 1.63x the QPS and 1.8x the build speed (`clawhdf5_accel::dot_i8`); it + stays off by default only because that kernel is AVX2-only and aarch64 falls + back to scalar. `hybrid_search` keeps one incremental BM25 index for the life of the store and never writes the store: Hebbian activation boosts are persisted by the next checkpoint (or on drop), not per query. Measure any search-path change with diff --git a/README.md b/README.md index 53e8b85..97a294d 100644 --- a/README.md +++ b/README.md @@ -437,8 +437,11 @@ ClawhDF5's agent memory design draws from 15+ recent papers: copy of the embeddings as `i8`, roughly halving a loaded store's memory (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are approximate, so the query path re-scores the candidate pool against the exact -embeddings the store already holds — recall matches the `f32` index, at about -13% fewer queries per second. See `BENCHMARKS.md`, "Quantising the index copy". +embeddings the store already holds, which keeps recall at the `f32` index's +level. On AVX2 it is also **faster** — 1.63x the queries per second and 1.8x +the build speed at equal recall — because the int8 kernel is SIMD too. It +stays off by default only because that kernel is AVX2-only and aarch64 falls +back to a scalar loop. See `BENCHMARKS.md`, "Quantising the index copy". | `parallel` | no | Rayon parallel search | | `fast-math` | no | BLAS matrix-vector multiply | | `accelerate` | no | Apple Accelerate / AMX (macOS) |