quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

bench(longmemeval): --float16, and float16 measured on real embeddings

Browse Source 
`longmemeval_bench --float16` builds every per-question store with
MemoryConfig::float16, so the vector stage searches half-rounded
embeddings exactly as such a store holds them.

Full longmemeval_s (500 questions, ~494 turns each) with real
all-MiniLM-L6-v2 embeddings, f32 vs float16, on tank (CUDA): identical
at every Hit@k and MRR, turn and session level, in all eight modes —
bar RRF session MRR 0.9253 vs 0.9254 and one or two flips out of ~320
in which gold session ranks first. The f32 run reproduces the published
hybrid numbers exactly. The earlier float16 evidence was synthetic
clustered data only; this is the real-embedding check.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-24 19:21:14 -05:00
co-authored by Claude Opus 5.5
parent c470244a6f
commit dbaf3f505d
3 changed files with 40 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BENCHMARKS.md
+24
View File
@@ -226,6 +226,30 @@ The in-memory cache holds the half-rounded values, so the store searches the
same before and after a reopen; RAM use is unchanged (the cache is still
`f32`). What `float16` saves is disk, and the I/O that goes with it.
**On real embeddings.** The table above is synthetic clustered data. The full
LongMemEval haystack (`longmemeval_s`, 500 questions, ~494 turns each) with
real all-MiniLM-L6-v2 embeddings, run once with `f32` stores and once with
`--float16`, measured 2026-09-24 on tank (embeddings on an RTX 5060 Ti):
```bash
cargo run --release -p clawhdf5-bench --bin longmemeval_bench --features embeddings-cuda -- \
benchmarks/longmemeval/longmemeval_s.json --embeddings weights/all-minilm-l6-v2 [--float16]
```
| mode | turn Hit@1 | turn Hit@5 | turn Hit@10 | turn MRR | session Hit@5 | session MRR |
|---|---:|---:|---:|---:|---:|---:|
| Hybrid 0.4 / 0.6, f32 | 51.6% | 81.4% | 87.8% | 0.6430 | 96.8% | 0.9347 |
| Hybrid 0.4 / 0.6, float16 | 51.6% | 81.4% | 87.8% | 0.6430 | 96.8% | 0.9347 |
| Vector only, f32 | 36.0% | 71.8% | 81.6% | 0.5031 | 94.2% | 0.8901 |
| Vector only, float16 | 36.0% | 71.8% | 81.6% | 0.5031 | 94.2% | 0.8901 |
All eight modes the harness runs (BM25, vector, hybrid, RRF, stemmed, and
both re-rank variants) were identical at every Hit@k and MRR, turn and session
level, except RRF's session MRR (0.9253 vs 0.9254) and one or two flips in
which of two gold sessions ranks first, out of ~320. Those flips show the
half-precision path was in effect; they do not change a single hit. The f32
run reproduces the published hybrid numbers exactly.
### Opening a store (`read_from_disk`)
`HDF5Memory::open` memory-mapped the file, copied the whole mapping into a