From 8b85d9364b2b5a14746339bc9711573d52fe7449 Mon Sep 17 00:00:00 2001 From: osobh Date: Mon, 21 Sep 2026 18:27:59 -0700 Subject: [PATCH] feat(agent): new stores use the int8 vector index by default MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `MemoryConfig::quantized_index` now defaults to `true`. It holds a quarter of the index memory and, with the exact re-score, is faster at equal recall on every configuration measured: 1.63x the queries per second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with builds 1.8x and 2.3x faster. The one argument for keeping it off — that int8 search was slower on ARM — did not survive being measured. Existing stores do not change. A store written with v2.6.0 or later keeps its persisted setting. One written before the setting existed has no stored value, and it loads as `false` rather than as the new default, so reopening it never changes how its index is held. That case is guarded by a real store written with the v2.5.0 CLI, committed as `tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens with an f32 index and still searches, and it fails if the load default is changed to `true`. The CLI needed more than a new default. `create --quantized-index` assigned its value straight into the config, so under the new default every CLI-created store would have been forced back to f32 unless the caller knew to ask for int8. It is replaced by `--f32-index`, which only ever switches the default off; `--quantized-index` is still accepted, hidden, as a no-op, and the two conflict. The whole agent suite passes under the new default, including the brute-force recall oracle, now running on int8 plus re-score without being asked to. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 23 +++++++ CLAUDE.md | 7 +- README.md | 3 +- crates/clawhdf5-agent/src/lib.rs | 14 ++-- crates/clawhdf5-agent/src/schema.rs | 3 + .../tests/fixtures/store_v2_5_0.h5 | Bin 0 -> 6784 bytes .../clawhdf5-agent/tests/hnsw_integration.rs | 60 ++++++++++++++++++ crates/clawhdf5-cli/src/main.rs | 21 ++++-- docs/QUICKSTART.md | 9 +-- 9 files changed, 124 insertions(+), 16 deletions(-) create mode 100644 crates/clawhdf5-agent/tests/fixtures/store_v2_5_0.h5 diff --git a/CHANGELOG.md b/CHANGELOG.md index c2a8e6d..cdccb42 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,29 @@ ## Unreleased +### Upgrade Notes +- **New stores use the int8 vector index by default.** + `MemoryConfig::quantized_index` now defaults to `true`: a quarter of the + index memory, builds 1.8x (x86-64) and 2.3x (Raspberry Pi 5) faster, and + searches 1.63x and 1.18x faster at equal recall, measured on every + configuration tested. **Existing stores are unaffected** — a store written + with v2.6.0 or later keeps its persisted setting, and one written before the + setting existed opens as `false` and keeps its f32 index. Set + `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt + out. The CLI's `--quantized-index` is still accepted but is now a no-op. + +### Defaults +- `clawhdf5-agent`: `MemoryConfig::quantized_index` defaults to `true` for new + stores. The reason it had been off — that int8 search was slower on ARM — + did not survive measurement (see Corrections). Stores that predate the + setting still load it as `false`, so reopening one never changes how its + index is held; a store written by the v2.5.0 CLI is now a test fixture that + guards exactly that, and the test fails if the load default is changed. +- `clawhdf5-cli`: `create --f32-index` opts out. `create` used to assign + `--quantized-index` straight into the config, which under the new default + would have forced every CLI-created store back to f32 unless the caller + knew to ask; it now only ever switches the default off. + ### Performance - `clawhdf5-accel`: **`dot_i8` has aarch64 kernels** — `SDOT` for CPUs with the ARMv8.2 dot-product extension (Cortex-A76 and later, Neoverse-N1, every diff --git a/CLAUDE.md b/CLAUDE.md index 021d3a3..a032bca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,8 +39,11 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F (plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its graph is saved to `.h5.ann` at each checkpoint and reloaded by `open()` (tied to the checkpoint by a generation id; stale/damaged sidecars are - ignored and the index rebuilt). `MemoryConfig::quantized_index` (off by - default, persisted) stores the index's own copy of the embeddings as `i8`, + ignored and the index rebuilt). `MemoryConfig::quantized_index` (**on by + default** for new stores, persisted; stores predating the setting load as + `false` and keep their f32 index — guarded by + `tests/fixtures/store_v2_5_0.h5`; CLI opt-out is `create --f32-index`) + stores the index's own copy of the embeddings as `i8`, which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors at 100K); because quantised distances are approximate and `ef` cannot compensate, the query path then re-scores the candidate pool against the diff --git a/README.md b/README.md index 838dd67..e235648 100644 --- a/README.md +++ b/README.md @@ -437,7 +437,8 @@ ClawhDF5's agent memory design draws from 15+ recent papers: vector index (16 / 64 / scale-with-`k` by default) and are stored with the file. -`MemoryConfig::quantized_index` (off by default) stores the HNSW index's own +`MemoryConfig::quantized_index` (**on by default** for new stores) holds the +HNSW index's own copy of the embeddings as `i8`, roughly halving a loaded store's memory (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are approximate, so the query path re-scores the candidate pool against the exact diff --git a/crates/clawhdf5-agent/src/lib.rs b/crates/clawhdf5-agent/src/lib.rs index d973743..5d660ea 100644 --- a/crates/clawhdf5-agent/src/lib.rs +++ b/crates/clawhdf5-agent/src/lib.rs @@ -134,13 +134,19 @@ pub struct MemoryConfig { pub wal_enabled: bool, pub wal_max_entries: usize, /// Store the vector index's own copy of the embeddings as int8 rather than - /// f32, a quarter of the memory. + /// f32, a quarter of the memory. **On by default** for new stores. /// /// The index's copy is the single largest part of a loaded store's /// footprint. Quantised distances are approximate, so the candidate pool /// is re-scored against the cache's exact embeddings before fusion, which - /// restores recall; what it costs is throughput — roughly 13% of queries - /// per second and 16% of build time at 100K x 384. See `BENCHMARKS.md`. + /// holds recall at the f32 index's level. It is also faster, not slower: + /// at equal recall, 1.63x the queries per second on x86-64 (AVX2) and + /// 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with builds 1.8x and 2.3x + /// faster. See `BENCHMARKS.md`. + /// + /// Persisted with the store. Stores written before this setting existed + /// have no stored value and open as `false`, so reopening an old store + /// never changes how its index is held. /// /// Has no effect without the `hnsw` feature. pub quantized_index: bool, @@ -182,7 +188,7 @@ impl MemoryConfig { created_at, wal_enabled: true, wal_max_entries: 500, - quantized_index: false, + quantized_index: true, hnsw_m: 16, hnsw_ef_construction: 64, hnsw_ef_search: 0, diff --git a/crates/clawhdf5-agent/src/schema.rs b/crates/clawhdf5-agent/src/schema.rs index c1600d3..a2fd15a 100644 --- a/crates/clawhdf5-agent/src/schema.rs +++ b/crates/clawhdf5-agent/src/schema.rs @@ -497,6 +497,9 @@ pub fn validate_and_load( wal_max_entries: optional_i64_attr(&attrs, "wal_max_entries") .and_then(|v| usize::try_from(v).ok()) .unwrap_or(500), + // `false`, not the new-store default: a store written before this + // setting existed was built with an f32 index, and reopening it must + // not silently change that. quantized_index: optional_bool_attr(&attrs, "quantized_index", false), hnsw_m: optional_i64_attr(&attrs, "hnsw_m") .and_then(|v| usize::try_from(v).ok()) diff --git a/crates/clawhdf5-agent/tests/fixtures/store_v2_5_0.h5 b/crates/clawhdf5-agent/tests/fixtures/store_v2_5_0.h5 new file mode 100644 index 0000000000000000000000000000000000000000..de6ef1c04a47f16a7e1c5dd20444b6598fe936b4 GIT binary patch literal 6784 zcmeHLeQZ-z6hE)+wyqnjgAK-jFqI&vgR+gU0aCXy21r0y6cMbi?OR)E`^I~3gGrbR z5eWno6caE}3I4&Cs2N5K2I5CRO^n|P1~K>*U=lTIqLCOp=YBk(+scfKA%T;v_rCLW z&pr3t-@WgxtE;K)<0n+G&&N5=&r#zxNN zSHY6c;1Uekibt<#uhgKKZK*Kjkr?ep-o@ub`@Cmr>uRb%hRSq)TYbuKLLP>R#%3t0 zCDcH(W>DIoKc?1+S`PdlJKE=j9Ih5JG}Y3=0o7tE1wy6FS2n)XH>tF&tlsCV95)gF z7C9lC+o%yOX$4|oW|1i@;#x!vwb(jZBB+JKn!$923LU*sORBL-OIu-;a)nqjQd(YW zCe(P`34I}#BDW=ha4Z2{OggVNKMP(~9?gKMhoVi%r2#Yckk(0;8Q|DX^uTm=Oh;U8 z?4($RN<_9|LtIy_vEw@_ay3%S6{G8kMuV4T8s&bSlml_CS&OGNDWGJ8(7zh80#+19 ziR$q%X;qQ@<>jAFqdYRxs1^*y)MOy2>!x)Dv$xT#4>t>IA+;sY zfP?BrIx85p_-C{dd04K-1Gu2=@<>h%G<%GHd4J;$3nVk%N7lrPGd|q+1!K2}Z zahH`d?M|AwZ}h~`5_KpG*9cYB*VSNlFax-?Av|)wKr1gulnK65JEN-p&RW#kLm4dx z+`#}Hkrfc(>-u2F-8H_=p%|I3R}fDX+j>@-ZH;ln{Wk4j+c^h;b_*@&OcS!#{y~ zF&8>PiVu{10c%JDN1@YUYxOPW5MzE{o;HsU*e z?BZbaQ6`b>gN+hNBwPN(>PmFeHi&4pC3jtd<n>W3Q*8&Fg z<}7hkE8wvY3Ui~)>}%>xMo0^UqG~d!#lN!!9xIm6OiN8Pnj`Q=L8L&AxK-d`W;WSe zm!(F`*X&1+r6+=>r6)D>yv@u=>PEt>r1O$_>W6q4YgR2hYXi%*SR`th>x7W2T>tH9 z1{knqP@ND;ghy@=)tYTr*}>d=2LrZ@Bi=0~P>5>z@OjhTneZ5P3nBab>vzEj)_uG+ zRu2zjHxD+2k39(cu{#t_JiN<*=dtSr_4K5D5X0^OI5ceA*92cM2xeBl+X401Erd57 zO_1hwy1FvJfo*#?w|hKr6ma6MJ}x$4CKd@o1_FRVSJxHhV5%A<@p_>p!;BrgoGoC) z+%4JM%|$FZCj~f;0@-F$0^ca`PBWPup8(s}oG!i$gBOZ7lny$rklq~gbRazj7yK_6 z+#%uuh^(yqieaz}mBJi$^!RFm+@uw=S_m#%#RL_|70jfH0DGhp+iGdlO!q)?LMh4< z+6dk(;j?W?A47|C05y$}`9v3dD`J^n+Wfw@J{zK>NDkq(1pk#BMRne)*>@8s(=>E& z#T1qwTyeSRa;2ndIyYPA+z{+lH}>|rv}Sq|$;Am>v5I@*?uRXrBUf9Bdm%Ty;Nt44 zmtKmd7xH44U3Wmyld>DrK5{QFJKLSymZ@0W6L(a_LXW;%wv_mIGnO}Hyea2R1#iZ2 zgSKgMfX*$+x|1`$zk4~L%%4t> = (0..40).map(|_| make_vector(&mut seed, 8)).collect(); + for (i, v) in vectors.iter().enumerate() { + mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap(); + } + assert_eq!( + mem.hybrid_search(&vectors[11], "", 1.0, 0.0, 1)[0].index, + 11 + ); + mem.flush_wal().unwrap(); + drop(mem); + assert!(HDF5Memory::open(&path).unwrap().config().quantized_index); +} + +#[test] +fn a_store_written_before_the_setting_existed_stays_f32() { + // `store_v2_5_0.h5` was written by the v2.5.0 CLI, before + // `quantized_index` or the HNSW parameters were persisted, so it carries + // none of them. Flipping the default for new stores must not reach back + // and change how an existing store's index is held. + let dir = TempDir::new().unwrap(); + let path = dir.path().join("legacy.h5"); + std::fs::copy( + concat!( + env!("CARGO_MANIFEST_DIR"), + "/tests/fixtures/store_v2_5_0.h5" + ), + &path, + ) + .unwrap(); + + let bytes = std::fs::read(&path).unwrap(); + assert!( + !bytes.windows(15).any(|w| w == b"quantized_index"), + "the fixture must predate the setting, or it tests nothing" + ); + + let mut mem = HDF5Memory::open(&path).unwrap(); + assert!( + !mem.config().quantized_index, + "an old store must reopen with an f32 index" + ); + assert_eq!(mem.config().hnsw_m, 16); + assert_eq!(mem.config().hnsw_ef_construction, 64); + assert_eq!(mem.count(), 6); + // And it still searches: entry 3's own embedding finds it first. + let hit = mem.hybrid_search(&[3.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "", 1.0, 0.0, 1); + assert_eq!(hit[0].index, 3); +} diff --git a/crates/clawhdf5-cli/src/main.rs b/crates/clawhdf5-cli/src/main.rs index 53831a0..ed02190 100644 --- a/crates/clawhdf5-cli/src/main.rs +++ b/crates/clawhdf5-cli/src/main.rs @@ -28,9 +28,13 @@ enum Commands { /// Enable write-ahead log #[arg(long)] wal: bool, - /// Store the vector index's copy of the embeddings as int8, roughly - /// halving a loaded store's memory at about 13% fewer queries/second + /// Hold the vector index's copy of the embeddings as f32 instead of + /// the default int8 (which uses a quarter of the memory and is faster + /// at equal recall) #[arg(long)] + f32_index: bool, + /// Accepted for compatibility; int8 is now the default + #[arg(long, hide = true, conflicts_with = "f32_index")] quantized_index: bool, }, /// Save a memory entry (reads JSON from stdin or --json) @@ -96,11 +100,18 @@ fn run(cli: Cli) -> Result<(), Box> { agent_id, dim, wal, - quantized_index, + f32_index, + quantized_index: _, } => { let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim); config.wal_enabled = wal; - config.quantized_index = quantized_index; + // Only ever switch *off* the library default: assigning the flag + // outright would force every CLI-created store back to f32 unless + // the caller knew to ask for int8. + if f32_index { + config.quantized_index = false; + } + let config_quantized = config.quantized_index; let mem = HDF5Memory::create(config)?; let j = serde_json::json!({ "status": "created", @@ -108,7 +119,7 @@ fn run(cli: Cli) -> Result<(), Box> { "agent_id": agent_id, "embedding_dim": dim, "wal_enabled": wal, - "quantized_index": quantized_index, + "quantized_index": config_quantized, "count": mem.count(), }); println!("{}", serde_json::to_string_pretty(&j)?); diff --git a/docs/QUICKSTART.md b/docs/QUICKSTART.md index af9faf8..c751768 100644 --- a/docs/QUICKSTART.md +++ b/docs/QUICKSTART.md @@ -364,10 +364,11 @@ cargo install --path crates/clawhdf5-cli clawhdf5 --path agent.h5 create --agent-id my-agent --dim 384 --wal ``` -Add `--quantized-index` to store the vector index's copy of the embeddings as -int8. That roughly halves a loaded store's memory at about 13% fewer queries -per second, with recall unchanged — the query path re-scores candidates -against the exact embeddings. The setting is recorded in the file. +New stores hold the vector index's copy of the embeddings as int8, which +roughly halves a loaded store's memory and is faster at equal recall — the +query path re-scores candidates against the exact embeddings. Pass +`--f32-index` to keep an f32 index instead. The setting is recorded in the +file, and stores created before it existed keep their f32 index. Output: ```json