1 Commits

Include renames

Author	SHA1	Message	Date
Omar Sobh	c913cd1cbf	bench: make the vector stage real, and measure BM25 vs vector vs hybrid (Tier 4b) ... Every LongMemEval number this project has published measured BM25 alone. The bench passed zero-vector embeddings with vector_weight=0.0, so the HNSW/vector stage — the thing the README credits for retrieval quality — contributed nothing and was never tested. An optional `embeddings` feature loads all-MiniLM-L6-v2 via candle and encodes the corpus for real. It is off by default and nothing in the shipped crates depends on it, so a project that advertises no heavyweight dependencies keeps that property; without the feature the bench behaves exactly as before. Full haystack, n=500, turn-level: Hit@1 Hit@5 Hit@10 MRR BM25 only 53.8% 75.0% 81.6% 0.6320 Vector only 36.0% 71.8% 81.6% 0.5027 Hybrid 0.7/0.3 44.4% 79.2% 86.0% 0.5868 Session-level, hybrid leads outright: 88.2 / 95.8 / 97.8 / 0.9158. The hybrid claim holds for depth and not for precision@1. Hybrid is the best configuration at Hit@5 and Hit@10 at both granularities — turn-level Hit@5 gains 4.2 points over BM25 and 7.4 over vector-only, which is the result that justifies running two stages at all. But BM25 alone still leads turn-level Hit@1 and MRR, so fusing buys deeper recall and pays at rank 1. Callers assembling five memories of context want hybrid; callers taking a single top hit are better served by BM25 today. The 0.7/0.3 weights are the documented default, not a searched optimum. omni-cortex's four-signal ablation found the same direction independently — there, adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR. Two codebases, two fusion schemes, same trade. Vector-only trailing BM25 at every turn-level cutoff except Hit@10 is stated plainly rather than buried: LongMemEval questions share heavy vocabulary with their evidence turns, which is close to the best case for lexical matching, and MiniLM at 384-d is a small model. Implementation notes: - Texts are deduplicated before encoding. The haystack sessions are drawn from a shared pool, so 500 questions x 493.5 turns collapses to 190,015 unique strings — the difference between encoding the corpus once and per question. - `embeddings-cuda` adds the GPU path, and it is not a convenience: 190k texts take ~13 min on an RTX 5060 Ti, while the same work on 8 CPU cores was still unfinished after 30 minutes. The device is selected at runtime with a CPU fallback, so a machine without CUDA still works. - Mean-pooling is masked and the output L2-normalised, which is the published recipe for this checkpoint (not the [CLS] pooler). One measurement wrinkle, recorded rather than smoothed over: on the oracle variant BM25-only reads 84.2% Hit@5 with real embedding vectors present against 84.4% with zero vectors — one question of 500 changes rank, MRR identical at 0.6597. On the full haystack the two agree exactly. Weight 0.0 evidently does not make the vector stage bit-for-bit absent from candidate selection on a small corpus. Verified on the Linux dev host: 49 groups / 1659 passed / 0 failed, clippy clean under -D warnings, fmt clean, with and without the feature.	2026-08-07 07:22:20 -07:00