feat(agent): new stores use the int8 vector index by default
`MemoryConfig::quantized_index` now defaults to `true`. It holds a quarter of the index memory and, with the exact re-score, is faster at equal recall on every configuration measured: 1.63x the queries per second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with builds 1.8x and 2.3x faster. The one argument for keeping it off — that int8 search was slower on ARM — did not survive being measured. Existing stores do not change. A store written with v2.6.0 or later keeps its persisted setting. One written before the setting existed has no stored value, and it loads as `false` rather than as the new default, so reopening it never changes how its index is held. That case is guarded by a real store written with the v2.5.0 CLI, committed as `tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens with an f32 index and still searches, and it fails if the load default is changed to `true`. The CLI needed more than a new default. `create --quantized-index` assigned its value straight into the config, so under the new default every CLI-created store would have been forced back to f32 unless the caller knew to ask for int8. It is replaced by `--f32-index`, which only ever switches the default off; `--quantized-index` is still accepted, hidden, as a no-op, and the two conflict. The whole agent suite passes under the new default, including the brute-force recall oracle, now running on int8 plus re-score without being asked to. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
@@ -28,9 +28,13 @@ enum Commands {
|
||||
/// Enable write-ahead log
|
||||
#[arg(long)]
|
||||
wal: bool,
|
||||
/// Store the vector index's copy of the embeddings as int8, roughly
|
||||
/// halving a loaded store's memory at about 13% fewer queries/second
|
||||
/// Hold the vector index's copy of the embeddings as f32 instead of
|
||||
/// the default int8 (which uses a quarter of the memory and is faster
|
||||
/// at equal recall)
|
||||
#[arg(long)]
|
||||
f32_index: bool,
|
||||
/// Accepted for compatibility; int8 is now the default
|
||||
#[arg(long, hide = true, conflicts_with = "f32_index")]
|
||||
quantized_index: bool,
|
||||
},
|
||||
/// Save a memory entry (reads JSON from stdin or --json)
|
||||
@@ -96,11 +100,18 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
agent_id,
|
||||
dim,
|
||||
wal,
|
||||
quantized_index,
|
||||
f32_index,
|
||||
quantized_index: _,
|
||||
} => {
|
||||
let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim);
|
||||
config.wal_enabled = wal;
|
||||
config.quantized_index = quantized_index;
|
||||
// Only ever switch *off* the library default: assigning the flag
|
||||
// outright would force every CLI-created store back to f32 unless
|
||||
// the caller knew to ask for int8.
|
||||
if f32_index {
|
||||
config.quantized_index = false;
|
||||
}
|
||||
let config_quantized = config.quantized_index;
|
||||
let mem = HDF5Memory::create(config)?;
|
||||
let j = serde_json::json!({
|
||||
"status": "created",
|
||||
@@ -108,7 +119,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
|
||||
"agent_id": agent_id,
|
||||
"embedding_dim": dim,
|
||||
"wal_enabled": wal,
|
||||
"quantized_index": quantized_index,
|
||||
"quantized_index": config_quantized,
|
||||
"count": mem.count(),
|
||||
});
|
||||
println!("{}", serde_json::to_string_pretty(&j)?);
|
||||
|
||||
Reference in New Issue
Block a user