feat(agent): new stores use the int8 vector index by default
`MemoryConfig::quantized_index` now defaults to `true`. It holds a quarter of the index memory and, with the exact re-score, is faster at equal recall on every configuration measured: 1.63x the queries per second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with builds 1.8x and 2.3x faster. The one argument for keeping it off — that int8 search was slower on ARM — did not survive being measured. Existing stores do not change. A store written with v2.6.0 or later keeps its persisted setting. One written before the setting existed has no stored value, and it loads as `false` rather than as the new default, so reopening it never changes how its index is held. That case is guarded by a real store written with the v2.5.0 CLI, committed as `tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens with an f32 index and still searches, and it fails if the load default is changed to `true`. The CLI needed more than a new default. `create --quantized-index` assigned its value straight into the config, so under the new default every CLI-created store would have been forced back to f32 unless the caller knew to ask for int8. It is replaced by `--f32-index`, which only ever switches the default off; `--quantized-index` is still accepted, hidden, as a no-op, and the two conflict. The whole agent suite passes under the new default, including the brute-force recall oracle, now running on int8 plus re-score without being asked to. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
@@ -134,13 +134,19 @@ pub struct MemoryConfig {
|
||||
pub wal_enabled: bool,
|
||||
pub wal_max_entries: usize,
|
||||
/// Store the vector index's own copy of the embeddings as int8 rather than
|
||||
/// f32, a quarter of the memory.
|
||||
/// f32, a quarter of the memory. **On by default** for new stores.
|
||||
///
|
||||
/// The index's copy is the single largest part of a loaded store's
|
||||
/// footprint. Quantised distances are approximate, so the candidate pool
|
||||
/// is re-scored against the cache's exact embeddings before fusion, which
|
||||
/// restores recall; what it costs is throughput — roughly 13% of queries
|
||||
/// per second and 16% of build time at 100K x 384. See `BENCHMARKS.md`.
|
||||
/// holds recall at the f32 index's level. It is also faster, not slower:
|
||||
/// at equal recall, 1.63x the queries per second on x86-64 (AVX2) and
|
||||
/// 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with builds 1.8x and 2.3x
|
||||
/// faster. See `BENCHMARKS.md`.
|
||||
///
|
||||
/// Persisted with the store. Stores written before this setting existed
|
||||
/// have no stored value and open as `false`, so reopening an old store
|
||||
/// never changes how its index is held.
|
||||
///
|
||||
/// Has no effect without the `hnsw` feature.
|
||||
pub quantized_index: bool,
|
||||
@@ -182,7 +188,7 @@ impl MemoryConfig {
|
||||
created_at,
|
||||
wal_enabled: true,
|
||||
wal_max_entries: 500,
|
||||
quantized_index: false,
|
||||
quantized_index: true,
|
||||
hnsw_m: 16,
|
||||
hnsw_ef_construction: 64,
|
||||
hnsw_ef_search: 0,
|
||||
|
||||
@@ -497,6 +497,9 @@ pub fn validate_and_load(
|
||||
wal_max_entries: optional_i64_attr(&attrs, "wal_max_entries")
|
||||
.and_then(|v| usize::try_from(v).ok())
|
||||
.unwrap_or(500),
|
||||
// `false`, not the new-store default: a store written before this
|
||||
// setting existed was built with an f32 index, and reopening it must
|
||||
// not silently change that.
|
||||
quantized_index: optional_bool_attr(&attrs, "quantized_index", false),
|
||||
hnsw_m: optional_i64_attr(&attrs, "hnsw_m")
|
||||
.and_then(|v| usize::try_from(v).ok())
|
||||
|
||||
Reference in New Issue
Block a user