Tier 4b reported hybrid retrieval at 0.7/0.3 and noted the weights were "the
documented default, not a searched optimum". `--sweep` searches them: 0.0 to 1.0
in 0.1 steps, reusing the one-time embedding table so eleven configurations cost
barely more than three.
The result is not a refinement. 0.7/0.3 is **strictly dominated**:
vector/keyword Hit@1 Hit@5 Hit@10 MRR sHit@5
0.0 / 1.0 53.8% 75.0% 81.6% 0.6320 93.6%
0.3 / 0.7 53.2% 78.8% 87.2% 0.6463 96.0%
0.4 / 0.6 51.6% 81.4% 87.8% 0.6429 96.8%
0.5 / 0.5 48.2% 81.4% 88.2% 0.6234 97.4%
0.7 / 0.3 44.4% 79.2% 86.0% 0.5868 95.8%
1.0 / 0.0 36.0% 71.8% 81.6% 0.5027 94.2%
0.4/0.6 beats 0.7/0.3 on every metric at both granularities — Hit@1 +7.2pp,
Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. No trade is being made; the default simply
sat on the wrong side of the peak. It is now 0.4/0.6, and README's usage snippet
recommends the same.
This corrects a conclusion I published one commit ago. Measuring only 0.7/0.3, I
wrote that fusion "buys deeper recall and pays for it at rank 1" and advised
callers taking a single top hit to prefer BM25. That was an artifact of the bad
weight, not a property of fusion: at 0.3/0.7 hybrid *beats* BM25 on MRR (0.6463
vs 0.6320) and Hit@5 (78.8% vs 75.0%) while giving up 0.6pp of Hit@1. Both
BENCHMARKS.md and README carry the correction rather than a quiet edit, since
the old text told readers to configure their systems a particular way.
The three-mode ablation rows are kept at their original settings — they measure
the shape of each stage in isolation, and the operating point now comes from the
sweep instead.
This commit is contained in:
@@ -82,13 +82,37 @@ const VECTOR_ONLY: Mode = Mode {
|
||||
vector_weight: 1.0,
|
||||
keyword_weight: 0.0,
|
||||
};
|
||||
/// Tuned by `--sweep` over the full haystack. The former 0.7/0.3 was a
|
||||
/// documented default that had never been searched, and the sweep found it
|
||||
/// strictly dominated: 0.4/0.6 is better on Hit@1, Hit@5, Hit@10 and MRR at
|
||||
/// both granularities.
|
||||
#[cfg(feature = "embeddings")]
|
||||
const HYBRID: Mode = Mode {
|
||||
label: "Hybrid (0.7 vector / 0.3 BM25)",
|
||||
vector_weight: 0.7,
|
||||
keyword_weight: 0.3,
|
||||
label: "Hybrid (0.4 vector / 0.6 BM25, tuned)",
|
||||
vector_weight: 0.4,
|
||||
keyword_weight: 0.6,
|
||||
};
|
||||
|
||||
/// Every 0.1 step of vector weight, keyword weight taking the remainder.
|
||||
///
|
||||
/// Labels are leaked to `&'static str` because `Mode::label` is a `&'static
|
||||
/// str` for the eleven named modes and a sweep is a short-lived process; the
|
||||
/// alternative is threading a lifetime through the whole report path for a
|
||||
/// diagnostic mode.
|
||||
#[cfg(feature = "embeddings")]
|
||||
fn sweep_modes() -> Vec<Mode> {
|
||||
(0..=10)
|
||||
.map(|i| {
|
||||
let v = i as f32 / 10.0;
|
||||
Mode {
|
||||
label: Box::leak(format!("sweep v={v:.1} / k={:.1}", 1.0 - v).into_boxed_str()),
|
||||
vector_weight: v,
|
||||
keyword_weight: 1.0 - v,
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Text -> embedding, built once for the whole corpus.
|
||||
type EmbeddingMap = HashMap<String, Vec<f32>>;
|
||||
|
||||
@@ -651,6 +675,7 @@ fn main() {
|
||||
let mut json_path: Option<String> = None;
|
||||
let mut limit: Option<usize> = None;
|
||||
let mut weights_dir: Option<String> = None;
|
||||
let mut sweep = false;
|
||||
let mut args = std::env::args().skip(1);
|
||||
while let Some(arg) = args.next() {
|
||||
match arg.as_str() {
|
||||
@@ -658,6 +683,7 @@ fn main() {
|
||||
let v = args.next().expect("--limit needs a value");
|
||||
limit = Some(v.parse().expect("--limit must be a positive integer"));
|
||||
}
|
||||
"--sweep" => sweep = true,
|
||||
"--embeddings" => {
|
||||
weights_dir = Some(args.next().expect("--embeddings needs a directory"));
|
||||
}
|
||||
@@ -675,7 +701,10 @@ fn main() {
|
||||
and tokenizer.json. Enables the vector stage and reports\n\
|
||||
BM25-only, vector-only, and hybrid separately. Requires\n\
|
||||
--features embeddings; without it the vector stage is\n\
|
||||
inert and only the BM25 row is produced."
|
||||
inert and only the BM25 row is produced.\n\
|
||||
--sweep instead of the three named modes, sweep vector_weight\n\
|
||||
from 0.0 to 1.0 in 0.1 steps. The 0.7/0.3 default was\n\
|
||||
never searched; this is what searches it."
|
||||
);
|
||||
return;
|
||||
}
|
||||
@@ -736,13 +765,20 @@ fn main() {
|
||||
let modes: Vec<Mode> = if embeddings.is_some() {
|
||||
#[cfg(feature = "embeddings")]
|
||||
{
|
||||
vec![BM25_ONLY, VECTOR_ONLY, HYBRID]
|
||||
if sweep {
|
||||
sweep_modes()
|
||||
} else {
|
||||
vec![BM25_ONLY, VECTOR_ONLY, HYBRID]
|
||||
}
|
||||
}
|
||||
#[cfg(not(feature = "embeddings"))]
|
||||
{
|
||||
vec![BM25_ONLY]
|
||||
}
|
||||
} else {
|
||||
if sweep {
|
||||
eprintln!("warning: --sweep needs --embeddings; running BM25 only");
|
||||
}
|
||||
vec![BM25_ONLY]
|
||||
};
|
||||
|
||||
|
||||
Reference in New Issue
Block a user