bench: sweep the hybrid weights, and correct the recommendation

Tier 4b reported hybrid retrieval at 0.7/0.3 and noted the weights were "the
documented default, not a searched optimum". `--sweep` searches them: 0.0 to 1.0
in 0.1 steps, reusing the one-time embedding table so eleven configurations cost
barely more than three.

The result is not a refinement. 0.7/0.3 is **strictly dominated**:

    vector/keyword   Hit@1   Hit@5  Hit@10     MRR   sHit@5
    0.0 / 1.0        53.8%   75.0%   81.6%  0.6320    93.6%
    0.3 / 0.7        53.2%   78.8%   87.2%  0.6463    96.0%
    0.4 / 0.6        51.6%   81.4%   87.8%  0.6429    96.8%
    0.5 / 0.5        48.2%   81.4%   88.2%  0.6234    97.4%
    0.7 / 0.3        44.4%   79.2%   86.0%  0.5868    95.8%
    1.0 / 0.0        36.0%   71.8%   81.6%  0.5027    94.2%

0.4/0.6 beats 0.7/0.3 on every metric at both granularities — Hit@1 +7.2pp,
Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. No trade is being made; the default simply
sat on the wrong side of the peak. It is now 0.4/0.6, and README's usage snippet
recommends the same.

This corrects a conclusion I published one commit ago. Measuring only 0.7/0.3, I
wrote that fusion "buys deeper recall and pays for it at rank 1" and advised
callers taking a single top hit to prefer BM25. That was an artifact of the bad
weight, not a property of fusion: at 0.3/0.7 hybrid *beats* BM25 on MRR (0.6463
vs 0.6320) and Hit@5 (78.8% vs 75.0%) while giving up 0.6pp of Hit@1. Both
BENCHMARKS.md and README carry the correction rather than a quiet edit, since
the old text told readers to configure their systems a particular way.

The three-mode ablation rows are kept at their original settings — they measure
the shape of each stage in isolation, and the operating point now comes from the
sweep instead.

BENCHMARKS.md

@@ -276,18 +276,49 @@ Session-level:
 | Vector only | 85.4% | 94.2% | 96.6% | 0.8901 |
 | Hybrid | **88.2%** | **95.8%** | **97.8%** | **0.9158** |
 
-**The hybrid claim holds for depth, not for precision@1.** Hybrid is the best
+### Weight sweep — full haystack, n=500
-configuration at Hit@5 and Hit@10 at both granularities — turn-level Hit@5 rises
-4.2 points over BM25 and 7.4 over vector-only, which is the result that justifies
-running two stages. But BM25 alone remains the best turn-level Hit@1 (53.8% vs
-44.4%) and MRR (0.6320 vs 0.5868). Fusing at `0.7/0.3` buys deeper recall and
-pays for it at rank 1.
 
-That trade matters for how the index is consumed: a caller assembling five
+`0.7/0.3` was a documented default, never a searched one. Sweeping
-memories of context should use hybrid, and a caller taking the single top hit is
+`vector_weight` from 0.0 to 1.0 (`--sweep`, reusing the one-time embedding
-better served by BM25 today. The weights are not tuned — `0.7/0.3` is the
+table) shows it is not merely suboptimal but **strictly dominated**:
-documented default, not a searched optimum, and a Hit@1-oriented deployment
+
-should sweep them.
+| vector / keyword | Hit@1 | Hit@5 | Hit@10 | MRR | session Hit@5 |
+|---|---|---|---|---|---|
+| 0.0 / 1.0 (BM25) | **53.8%** | 75.0% | 81.6% | 0.6320 | 93.6% |
+| 0.1 / 0.9 | 53.2% | 77.4% | 83.8% | 0.6374 | 95.0% |
+| 0.2 / 0.8 | 53.6% | 78.2% | 85.6% | 0.6440 | 95.4% |
+| 0.3 / 0.7 | 53.2% | 78.8% | 87.2% | **0.6463** | 96.0% |
+| **0.4 / 0.6** | 51.6% | **81.4%** | 87.8% | 0.6429 | 96.8% |
+| 0.5 / 0.5 | 48.2% | **81.4%** | **88.2%** | 0.6234 | **97.4%** |
+| 0.6 / 0.4 | 46.6% | 79.8% | 87.4% | 0.6069 | 96.6% |
+| 0.7 / 0.3 *(old default)* | 44.4% | 79.2% | 86.0% | 0.5868 | 95.8% |
+| 0.8 / 0.2 | 40.6% | 76.2% | 85.4% | 0.5571 | 95.2% |
+| 0.9 / 0.1 | 37.8% | 73.4% | 84.6% | 0.5289 | 94.2% |
+| 1.0 / 0.0 (vector) | 36.0% | 71.8% | 81.6% | 0.5027 | 94.2% |
+
+**`0.4/0.6` beats `0.7/0.3` on every metric at both granularities** — Hit@1
++7.2pp, Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. There is no trade being made; the
+old default was simply on the wrong side of the peak. **`0.4/0.6` is the
+recommended setting**, with `0.3/0.7` preferable if rank-1 precision matters
+most (it takes the best MRR in the sweep and gives up only 0.6pp of Hit@1
+against pure BM25).
+
+**Correction.** An earlier revision of this section, measuring only `0.7/0.3`,
+concluded that fusion "buys deeper recall and pays for it at rank 1" and advised
+callers taking a single top hit to prefer BM25. That was an artifact of the
+badly-chosen weight, not a property of fusion. At `0.3/0.7` hybrid *beats* BM25
+on MRR (0.6463 vs 0.6320) and on Hit@5 (78.8% vs 75.0%) while costing 0.6pp of
+Hit@1. The advice below is corrected accordingly.
+
+**Hybrid wins, once the weights are right.** At the old `0.7/0.3` the picture
+looked like a trade: best at Hit@5 and Hit@10, worse than BM25 at Hit@1 and MRR.
+The sweep above shows that was the weight, not fusion. At `0.4/0.6` hybrid leads
+Hit@5 and Hit@10 outright; at `0.3/0.7` it also leads MRR and is within 0.6pp of
+BM25 at Hit@1. Both dominate `0.7/0.3`.
+
+The rows below are kept at the three original settings because they are what the
+mode ablation measured — read them as "the shape of each stage in isolation",
+and take the operating point from the sweep.
 
 The same pattern shows up independently in omni-cortex's four-signal RRF ablation,
 where adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR.

README.md

@@ -111,13 +111,14 @@ declaration:
 |------|------------------|---------------------|
 | BM25 only | 75.0% | 93.6% |
 | Vector only (MiniLM) | 71.8% | 94.2% |
-| Hybrid (0.7/0.3) | **79.2%** | **95.8%** |
+| Hybrid (0.4/0.6, tuned) | **81.4%** | **96.8%** |
 
-Hybrid is the strongest configuration at Hit@5 and Hit@10, which is what running
+Hybrid is the strongest configuration, which is what running two retrieval stages
-two retrieval stages is for. It is *not* strongest at rank 1 — BM25 alone leads
+is for. The weights matter more than the stages: a sweep of `vector_weight` from
-turn-level Hit@1 (53.8% vs 44.4%) and MRR (0.6320 vs 0.5868), so a caller taking
+0.0 to 1.0 found the long-standing `0.7/0.3` default is **strictly dominated** by
-a single top hit is better served by BM25 today. The `0.7/0.3` weights are the
+`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Use
-documented default, not a searched optimum.
+`0.4/0.6`, or `0.3/0.7` if rank-1 precision matters most. See
+[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#longmemeval-results).
 
 Vector embeddings require `--features embeddings`; without it the vector stage is
 inert and only the BM25 row is produced, which is what every previously published
@@ -141,7 +142,7 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 > haystack does not fix. Details in
 > [BENCHMARKS.md](BENCHMARKS.md#retracted-session-level-recall-and-the-memx-comparison).
 
-> Enable embeddings via `hybrid_search(query_emb, text, 0.7, 0.3, k)` for substantially higher recall. The vector stage is served by the HNSW index by default (the `hnsw` feature is on by default); build with `--no-default-features --features float16` to fall back to an exact linear cosine scan.
+> Enable embeddings via `hybrid_search(query_emb, text, 0.4, 0.6, k)` for substantially higher recall. The vector stage is served by the HNSW index by default (the `hnsw` feature is on by default); build with `--no-default-features --features float16` to fall back to an exact linear cosine scan.
 
 ### Memory Footprint
 

crates/clawhdf5-bench/src/bin/longmemeval_bench.rs

@@ -82,13 +82,37 @@ const VECTOR_ONLY: Mode = Mode {
     vector_weight: 1.0,
     keyword_weight: 0.0,
 };
+/// Tuned by `--sweep` over the full haystack. The former 0.7/0.3 was a
+/// documented default that had never been searched, and the sweep found it
+/// strictly dominated: 0.4/0.6 is better on Hit@1, Hit@5, Hit@10 and MRR at
+/// both granularities.
 #[cfg(feature = "embeddings")]
 const HYBRID: Mode = Mode {
-    label: "Hybrid (0.7 vector / 0.3 BM25)",
+    label: "Hybrid (0.4 vector / 0.6 BM25, tuned)",
-    vector_weight: 0.7,
+    vector_weight: 0.4,
-    keyword_weight: 0.3,
+    keyword_weight: 0.6,
 };
 
+/// Every 0.1 step of vector weight, keyword weight taking the remainder.
+///
+/// Labels are leaked to `&'static str` because `Mode::label` is a `&'static
+/// str` for the eleven named modes and a sweep is a short-lived process; the
+/// alternative is threading a lifetime through the whole report path for a
+/// diagnostic mode.
+#[cfg(feature = "embeddings")]
+fn sweep_modes() -> Vec<Mode> {
+    (0..=10)
+        .map(|i| {
+            let v = i as f32 / 10.0;
+            Mode {
+                label: Box::leak(format!("sweep v={v:.1} / k={:.1}", 1.0 - v).into_boxed_str()),
+                vector_weight: v,
+                keyword_weight: 1.0 - v,
+            }
+        })
+        .collect()
+}
+
 /// Text -> embedding, built once for the whole corpus.
 type EmbeddingMap = HashMap<String, Vec<f32>>;
 
@@ -651,6 +675,7 @@ fn main() {
     let mut json_path: Option<String> = None;
     let mut limit: Option<usize> = None;
     let mut weights_dir: Option<String> = None;
+    let mut sweep = false;
     let mut args = std::env::args().skip(1);
     while let Some(arg) = args.next() {
         match arg.as_str() {
@@ -658,6 +683,7 @@ fn main() {
                 let v = args.next().expect("--limit needs a value");
                 limit = Some(v.parse().expect("--limit must be a positive integer"));
             }
+            "--sweep" => sweep = true,
             "--embeddings" => {
                 weights_dir = Some(args.next().expect("--embeddings needs a directory"));
             }
@@ -675,7 +701,10 @@ fn main() {
                               and tokenizer.json. Enables the vector stage and reports\n\
                               BM25-only, vector-only, and hybrid separately. Requires\n\
                               --features embeddings; without it the vector stage is\n\
-                              inert and only the BM25 row is produced."
+                              inert and only the BM25 row is produced.\n\
+                     --sweep  instead of the three named modes, sweep vector_weight\n\
+                              from 0.0 to 1.0 in 0.1 steps. The 0.7/0.3 default was\n\
+                              never searched; this is what searches it."
                 );
                 return;
             }
@@ -736,13 +765,20 @@ fn main() {
     let modes: Vec<Mode> = if embeddings.is_some() {
         #[cfg(feature = "embeddings")]
         {
+            if sweep {
+                sweep_modes()
+            } else {
                 vec![BM25_ONLY, VECTOR_ONLY, HYBRID]
             }
+        }
         #[cfg(not(feature = "embeddings"))]
         {
             vec![BM25_ONLY]
         }
     } else {
+        if sweep {
+            eprintln!("warning: --sweep needs --embeddings; running BM25 only");
+        }
         vec![BM25_ONLY]
     };