feat(agent): optional int8 vector index, re-scored against exact embeddings
`MemoryConfig::quantized_index` stores the HNSW index's own copy of the embeddings as i8 rather than f32. At 100k x 384 that takes the index from 266 to 123 MiB and the whole reopened store from 399 to 256 MiB — 2.72x to 1.74x the raw vectors, the largest remaining item in the footprint. Quantised distances are approximate and `ef` cannot compensate, because the loss is in the distances rather than in the graph: recall@10 tops out at 0.967 against f32's 0.9995 and does not move between ef=128 and ef=256. The store already holds the exact embeddings, though, so when the index is quantised the query path re-scores the candidate pool against them before fusion. That restores recall (0.9940 vs 0.9945 at ef=64) and costs about 13% of QPS. Off by default: it trades query speed for memory and which side is worth more depends on the deployment. The flag is persisted in `/meta`, so a reopened store does not silently revert to four times the index memory, and the sidecar graph is rehydrated into the configured storage. Also on the CLI as `create --quantized-index`. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,26 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Memory
|
||||
- `clawhdf5-agent`: **`MemoryConfig::quantized_index`** stores the vector
|
||||
index's own copy of the embeddings as `i8` rather than `f32`, which at 100k
|
||||
384-dim entries takes the index from 266 to 123 MiB and the whole reopened
|
||||
store from 399 to 256 MiB (2.72x -> **1.74x** the raw vectors). Quantised
|
||||
distances are approximate and `ef` cannot compensate — recall@10 tops out at
|
||||
0.967 against f32's 0.9995 — so the query path re-scores the candidate pool
|
||||
against the exact embeddings the store already holds, which restores recall
|
||||
(0.9940 vs 0.9945 at ef=64) for about 13% of QPS. **Off by default**: it
|
||||
trades query speed for memory, and which side is worth more depends on the
|
||||
deployment. The setting is persisted, so a reopened store does not silently
|
||||
revert to four times the index memory.
|
||||
- `clawhdf5-ann`: `Storage::Int8` and the `build_with` / `new_with` /
|
||||
`from_graph_bytes_with` constructors that select it. The scale is per row,
|
||||
not global — a fixed `[-1, 1]` scale spends fewer than 12 of the 255 levels
|
||||
on a unit-length 128-dim vector and is unusable (0.35 top-10 overlap against
|
||||
an exact ranking, versus 0.99 per row). `compact()` keeps the storage it was
|
||||
given; serialized indexes still carry f32 vectors, so a quantised index is
|
||||
rebuilt rather than loaded.
|
||||
|
||||
### Memory
|
||||
- `clawhdf5-agent`: **a loaded store holds ~30% less memory** (100k 384-dim
|
||||
entries: 505 -> 357 MiB, 3.44x -> 2.43x the raw vectors). The cache kept
|
||||
|
||||
Reference in New Issue
Block a user