feat(agent): optional int8 vector index, re-scored against exact embeddings
`MemoryConfig::quantized_index` stores the HNSW index's own copy of the embeddings as i8 rather than f32. At 100k x 384 that takes the index from 266 to 123 MiB and the whole reopened store from 399 to 256 MiB — 2.72x to 1.74x the raw vectors, the largest remaining item in the footprint. Quantised distances are approximate and `ef` cannot compensate, because the loss is in the distances rather than in the graph: recall@10 tops out at 0.967 against f32's 0.9995 and does not move between ef=128 and ef=256. The store already holds the exact embeddings, though, so when the index is quantised the query path re-scores the candidate pool against them before fusion. That restores recall (0.9940 vs 0.9945 at ef=64) and costs about 13% of QPS. Off by default: it trades query speed for memory and which side is worth more depends on the deployment. The flag is persisted in `/meta`, so a reopened store does not silently revert to four times the index memory, and the sidecar graph is rehydrated into the configured storage. Also on the CLI as `create --quantized-index`. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
@@ -432,6 +432,13 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
|
||||
| `agent` | no | Full agent memory layer |
|
||||
| `float16` | **yes** | Half-precision embedding storage (2× compression) |
|
||||
| `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
|
||||
|
||||
`MemoryConfig::quantized_index` (off by default) stores the HNSW index's own
|
||||
copy of the embeddings as `i8`, roughly halving a loaded store's memory
|
||||
(2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are
|
||||
approximate, so the query path re-scores the candidate pool against the exact
|
||||
embeddings the store already holds — recall matches the `f32` index, at about
|
||||
13% fewer queries per second. See `BENCHMARKS.md`, "Quantising the index copy".
|
||||
| `parallel` | no | Rayon parallel search |
|
||||
| `fast-math` | no | BLAS matrix-vector multiply |
|
||||
| `accelerate` | no | Apple Accelerate / AMX (macOS) |
|
||||
|
||||
Reference in New Issue
Block a user