bench: sweep the hybrid weights, and correct the recommendation
CI / test (push) Failing after 3s

Tier 4b reported hybrid retrieval at 0.7/0.3 and noted the weights were "the
documented default, not a searched optimum". `--sweep` searches them: 0.0 to 1.0
in 0.1 steps, reusing the one-time embedding table so eleven configurations cost
barely more than three.

The result is not a refinement. 0.7/0.3 is **strictly dominated**:

    vector/keyword   Hit@1   Hit@5  Hit@10     MRR   sHit@5
    0.0 / 1.0        53.8%   75.0%   81.6%  0.6320    93.6%
    0.3 / 0.7        53.2%   78.8%   87.2%  0.6463    96.0%
    0.4 / 0.6        51.6%   81.4%   87.8%  0.6429    96.8%
    0.5 / 0.5        48.2%   81.4%   88.2%  0.6234    97.4%
    0.7 / 0.3        44.4%   79.2%   86.0%  0.5868    95.8%
    1.0 / 0.0        36.0%   71.8%   81.6%  0.5027    94.2%

0.4/0.6 beats 0.7/0.3 on every metric at both granularities — Hit@1 +7.2pp,
Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. No trade is being made; the default simply
sat on the wrong side of the peak. It is now 0.4/0.6, and README's usage snippet
recommends the same.

This corrects a conclusion I published one commit ago. Measuring only 0.7/0.3, I
wrote that fusion "buys deeper recall and pays for it at rank 1" and advised
callers taking a single top hit to prefer BM25. That was an artifact of the bad
weight, not a property of fusion: at 0.3/0.7 hybrid *beats* BM25 on MRR (0.6463
vs 0.6320) and Hit@5 (78.8% vs 75.0%) while giving up 0.6pp of Hit@1. Both
BENCHMARKS.md and README carry the correction rather than a quiet edit, since
the old text told readers to configure their systems a particular way.

The three-mode ablation rows are kept at their original settings — they measure
the shape of each stage in isolation, and the operating point now comes from the
sweep instead.
This commit is contained in:
Omar Sobh
2026-08-07 11:10:12 -07:00
parent 12d9d8462f
commit 1537a9464a
3 changed files with 91 additions and 23 deletions
@@ -82,13 +82,37 @@ const VECTOR_ONLY: Mode = Mode {
vector_weight: 1.0,
keyword_weight: 0.0,
};
/// Tuned by `--sweep` over the full haystack. The former 0.7/0.3 was a
/// documented default that had never been searched, and the sweep found it
/// strictly dominated: 0.4/0.6 is better on Hit@1, Hit@5, Hit@10 and MRR at
/// both granularities.
#[cfg(feature = "embeddings")]
const HYBRID: Mode = Mode {
label: "Hybrid (0.7 vector / 0.3 BM25)",
vector_weight: 0.7,
keyword_weight: 0.3,
label: "Hybrid (0.4 vector / 0.6 BM25, tuned)",
vector_weight: 0.4,
keyword_weight: 0.6,
};
/// Every 0.1 step of vector weight, keyword weight taking the remainder.
///
/// Labels are leaked to `&'static str` because `Mode::label` is a `&'static
/// str` for the eleven named modes and a sweep is a short-lived process; the
/// alternative is threading a lifetime through the whole report path for a
/// diagnostic mode.
#[cfg(feature = "embeddings")]
fn sweep_modes() -> Vec<Mode> {
(0..=10)
.map(|i| {
let v = i as f32 / 10.0;
Mode {
label: Box::leak(format!("sweep v={v:.1} / k={:.1}", 1.0 - v).into_boxed_str()),
vector_weight: v,
keyword_weight: 1.0 - v,
}
})
.collect()
}
/// Text -> embedding, built once for the whole corpus.
type EmbeddingMap = HashMap<String, Vec<f32>>;
@@ -651,6 +675,7 @@ fn main() {
let mut json_path: Option<String> = None;
let mut limit: Option<usize> = None;
let mut weights_dir: Option<String> = None;
let mut sweep = false;
let mut args = std::env::args().skip(1);
while let Some(arg) = args.next() {
match arg.as_str() {
@@ -658,6 +683,7 @@ fn main() {
let v = args.next().expect("--limit needs a value");
limit = Some(v.parse().expect("--limit must be a positive integer"));
}
"--sweep" => sweep = true,
"--embeddings" => {
weights_dir = Some(args.next().expect("--embeddings needs a directory"));
}
@@ -675,7 +701,10 @@ fn main() {
and tokenizer.json. Enables the vector stage and reports\n\
BM25-only, vector-only, and hybrid separately. Requires\n\
--features embeddings; without it the vector stage is\n\
inert and only the BM25 row is produced."
inert and only the BM25 row is produced.\n\
--sweep instead of the three named modes, sweep vector_weight\n\
from 0.0 to 1.0 in 0.1 steps. The 0.7/0.3 default was\n\
never searched; this is what searches it."
);
return;
}
@@ -736,13 +765,20 @@ fn main() {
let modes: Vec<Mode> = if embeddings.is_some() {
#[cfg(feature = "embeddings")]
{
vec![BM25_ONLY, VECTOR_ONLY, HYBRID]
if sweep {
sweep_modes()
} else {
vec![BM25_ONLY, VECTOR_ONLY, HYBRID]
}
}
#[cfg(not(feature = "embeddings"))]
{
vec![BM25_ONLY]
}
} else {
if sweep {
eprintln!("warning: --sweep needs --embeddings; running BM25 only");
}
vec![BM25_ONLY]
};