diff --git a/BENCHMARKS.md b/BENCHMARKS.md index a7a1148..1e4b98d 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -226,6 +226,30 @@ The in-memory cache holds the half-rounded values, so the store searches the same before and after a reopen; RAM use is unchanged (the cache is still `f32`). What `float16` saves is disk, and the I/O that goes with it. +**On real embeddings.** The table above is synthetic clustered data. The full +LongMemEval haystack (`longmemeval_s`, 500 questions, ~494 turns each) with +real all-MiniLM-L6-v2 embeddings, run once with `f32` stores and once with +`--float16`, measured 2026-09-24 on tank (embeddings on an RTX 5060 Ti): + +```bash +cargo run --release -p clawhdf5-bench --bin longmemeval_bench --features embeddings-cuda -- \ + benchmarks/longmemeval/longmemeval_s.json --embeddings weights/all-minilm-l6-v2 [--float16] +``` + +| mode | turn Hit@1 | turn Hit@5 | turn Hit@10 | turn MRR | session Hit@5 | session MRR | +|---|---:|---:|---:|---:|---:|---:| +| Hybrid 0.4 / 0.6, f32 | 51.6% | 81.4% | 87.8% | 0.6430 | 96.8% | 0.9347 | +| Hybrid 0.4 / 0.6, float16 | 51.6% | 81.4% | 87.8% | 0.6430 | 96.8% | 0.9347 | +| Vector only, f32 | 36.0% | 71.8% | 81.6% | 0.5031 | 94.2% | 0.8901 | +| Vector only, float16 | 36.0% | 71.8% | 81.6% | 0.5031 | 94.2% | 0.8901 | + +All eight modes the harness runs (BM25, vector, hybrid, RRF, stemmed, and +both re-rank variants) were identical at every Hit@k and MRR, turn and session +level, except RRF's session MRR (0.9253 vs 0.9254) and one or two flips in +which of two gold sessions ranks first, out of ~320. Those flips show the +half-precision path was in effect; they do not change a single hit. The f32 +run reproduces the published hybrid numbers exactly. + ### Opening a store (`read_from_disk`) `HDF5Memory::open` memory-mapped the file, copied the whole mapping into a diff --git a/CHANGELOG.md b/CHANGELOG.md index 53c6fa4..8525407 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -89,7 +89,9 @@ precision.** At 100K x 384 the file goes from 154.0 to 80.8 MiB (−48%), a checkpoint from 752 to 512 ms and open from 300 to 252 ms, with the same vector recall@10 against an exact scan (0.999 vs 0.994) and the same - `hybrid_search` latency; at 10K open is 3 ms slower. The cache rounds each + `hybrid_search` latency; at 10K open is 3 ms slower. On the full + LongMemEval haystack with real MiniLM embeddings every retrieval metric is + identical to `f32` (`longmemeval_bench --float16`). The cache rounds each embedding as it is saved, so memory and file agree bit for bit and a store returns the same results before and after a reopen (tested). Out-of-range values are refused with `MemoryError::InvalidEntry` rather than stored as diff --git a/crates/clawhdf5-bench/src/bin/longmemeval_bench.rs b/crates/clawhdf5-bench/src/bin/longmemeval_bench.rs index 7d4858c..7943f53 100644 --- a/crates/clawhdf5-bench/src/bin/longmemeval_bench.rs +++ b/crates/clawhdf5-bench/src/bin/longmemeval_bench.rs @@ -64,6 +64,11 @@ use tempfile::TempDir; const EMBEDDING_DIM: usize = 384; +/// `--float16`: build every per-question store with `MemoryConfig::float16`, +/// so embeddings are rounded to half precision as they are saved — exactly +/// what such a store searches over. +static FLOAT16: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); + /// A mode's fusion, as one short string for the reports. fn describe(mode: Mode) -> String { let fusion = match mode.fusion { @@ -431,6 +436,7 @@ fn evaluate_question( let mut config = MemoryConfig::new(dir.path().join("lme.h5"), "lme-bench", EMBEDDING_DIM); config.wal_enabled = false; config.compact_threshold = 0.0; + config.float16 = FLOAT16.load(std::sync::atomic::Ordering::Relaxed); let mut memory = HDF5Memory::create(config).expect("failed to create HDF5Memory"); memory.set_token_filter(mode.tokens); @@ -940,6 +946,10 @@ fn main() { limit = Some(v.parse().expect("--limit must be a positive integer")); } "--sweep" => sweep = true, + "--float16" => { + FLOAT16.store(true, std::sync::atomic::Ordering::Relaxed); + eprintln!("Stores use MemoryConfig::float16 (half-precision embeddings)"); + } "--rerank-sweep" => { // Re-ranking needs the vector stage to have candidates worth // reordering, so this is an embeddings-only comparison. @@ -971,6 +981,9 @@ fn main() { --rerank-sweep\n\ compare re-ranking off, metadata-only (the old\n\ behaviour) and blended at several half-lives.\n\ + --float16\n\ + build each store with MemoryConfig::float16, to\n\ + compare retrieval on half-precision embeddings.\n\ --sweep instead of the three named modes, sweep vector_weight\n\ from 0.0 to 1.0 in 0.1 steps. The 0.7/0.3 default was\n\ never searched; this is what searches it."