quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(agent): selectable fusion; adopt the measured 0.4/0.6 default weights

Browse Source 
BENCHMARKS.md has recorded since the weight sweep that the 0.7/0.3 default is
strictly dominated by 0.4/0.6 over the full LongMemEval haystack, but the
shipping code never adopted it: unified_search and the OpenClaw backend both
passed 0.7/0.3. Re-running the sweep here (500 questions, real MiniLM
embeddings on a GPU) reproduces it — turn-level Hit@1 51.6% vs 44.2%, Hit@5
81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.6430 vs 0.5856 — so both now use
hybrid::DEFAULT_FUSION, which is that operating point and carries the
reasoning. A unit test pins it.

Fusion is also selectable now. hybrid::Fusion is either Weighted { vector,
keyword } or Rrf { k }; hybrid::fuse applies either to one candidate list per
stage, and merge_vector_keyword / hybrid_search delegate to it, so the public
API is unchanged. New HDF5Memory::hybrid_search_with and
hybrid::hybrid_search_fused take a Fusion. Reciprocal rank fusion was
implemented but reachable only as a free function over a linear scan, so it
had never been compared with the weighted sum on equal terms; it is now a mode
in the LongMemEval bench (measurement to follow).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-19 16:57:51 -07:00
co-authored by Claude Opus 5
parent b23946e62d
commit 29baabbed2
7 changed files with 256 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-agent/src/hybrid.rs
+155 -11
View File
@@ -29,12 +29,39 @@ pub fn hybrid_search(
query_embedding: &[f32],
query_text: &str,
vectors: &[Vec<f32>],
_chunks: &[String],
chunks: &[String],
tombstones: &[u8],
bm25_index: &BM25Index,
vector_weight: f32,
keyword_weight: f32,
k: usize,
) -> Vec<(usize, f32)> {
hybrid_search_fused(
query_embedding,
query_text,
vectors,
chunks,
tombstones,
bm25_index,
Fusion::Weighted {
vector: vector_weight,
keyword: keyword_weight,
},
k,
)
}
/// [`hybrid_search`] with the fusion method chosen explicitly.
#[allow(clippy::too_many_arguments)]
pub fn hybrid_search_fused(
query_embedding: &[f32],
query_text: &str,
vectors: &[Vec<f32>],
_chunks: &[String],
tombstones: &[u8],
bm25_index: &BM25Index,
fusion: Fusion,
k: usize,
) -> Vec<(usize, f32)> {
// Get raw scores from both systems. Request all results so normalization
// covers the full distribution.
@@ -60,7 +87,7 @@ pub fn hybrid_search(
};
let kw_scores = bm25_index.scores(query_text);
merge_vector_keyword(vec_scores, kw_scores, vector_weight, keyword_weight, k)
fuse(vec_scores, kw_scores, fusion, k)
}
/// Merge pre-computed vector-similarity and keyword scores into a single ranking.
@@ -76,18 +103,92 @@ pub fn merge_vector_keyword(
keyword_weight: f32,
k: usize,
) -> Vec<(usize, f32)> {
// Normalize each set to [0, 1].
let vec_normalized = normalize_scores(&vec_scores);
let kw_normalized = normalize_scores(&kw_scores);
fuse(
vec_scores,
kw_scores,
Fusion::Weighted {
vector: vector_weight,
keyword: keyword_weight,
},
k,
)
}
// Merge scores with weights.
let mut merged: HashMap<usize, f32> = HashMap::new();
/// How the vector and keyword stages are combined into one ranking.
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum Fusion {
/// Min-max normalise each stage over its own candidates, then take a
/// weighted sum. Uses the *scores*, so a stage that separates its
/// candidates sharply keeps that separation — and a stage whose candidates
/// are all near-identical contributes little.
Weighted {
/// Weight on the vector stage.
vector: f32,
/// Weight on the keyword stage.
keyword: f32,
},
/// Reciprocal rank fusion: each stage contributes `1 / (k + rank)`,
/// ignoring score magnitudes entirely. Robust when the two stages'
/// scores aren't comparable, at the cost of discarding confidence.
Rrf {
/// The rank-damping constant; 60 is the value from the original paper.
k: f32,
},
}
for (idx, score) in &vec_normalized {
*merged.entry(*idx).or_insert(0.0) += vector_weight * score;
impl Default for Fusion {
fn default() -> Self {
DEFAULT_FUSION
}
for (idx, score) in &kw_normalized {
*merged.entry(*idx).or_insert(0.0) += keyword_weight * score;
}
/// The fusion `hybrid_search` uses unless told otherwise.
///
/// The weights are not a guess: a sweep of every 0.1 step over the full
/// LongMemEval haystack (500 questions, real MiniLM embeddings) found the
/// long-standing 0.7/0.3 default *strictly dominated* — 0.4/0.6 is better at
/// Hit@1, Hit@5, Hit@10 and MRR, at both turn and session granularity. See
/// `BENCHMARKS.md`, "Weight sweep".
pub const DEFAULT_FUSION: Fusion = Fusion::Weighted {
vector: 0.4,
keyword: 0.6,
};
/// Combine one ranked candidate list from each stage into a single top-`k`.
///
/// Neither list need be sorted; both are consumed.
pub fn fuse(
vec_scores: Vec<(usize, f32)>,
kw_scores: Vec<(usize, f32)>,
fusion: Fusion,
k: usize,
) -> Vec<(usize, f32)> {
let mut merged: HashMap<usize, f32> = HashMap::new();
match fusion {
Fusion::Weighted { vector, keyword } => {
// Normalize each set to [0, 1].
for (idx, score) in &normalize_scores(&vec_scores) {
*merged.entry(*idx).or_insert(0.0) += vector * score;
}
for (idx, score) in &normalize_scores(&kw_scores) {
*merged.entry(*idx).or_insert(0.0) += keyword * score;
}
}
Fusion::Rrf { k: damping } => {
for mut stage in [vec_scores, kw_scores] {
// Rank 1 is the best score. Ties break by index so a stage's
// contribution doesn't depend on the candidate order it
// happened to be produced in.
stage.sort_by(|a, b| {
b.1.partial_cmp(&a.1)
.unwrap_or(std::cmp::Ordering::Equal)
.then(a.0.cmp(&b.0))
});
for (rank, (idx, _)) in stage.iter().enumerate() {
*merged.entry(*idx).or_insert(0.0) += 1.0 / (damping + (rank + 1) as f32);
}
}
}
}
let mut results: Vec<(usize, f32)> = merged.into_iter().collect();
@@ -353,6 +454,49 @@ mod tests {
assert_eq!(result[0].1, 1.0);
}
#[test]
fn default_fusion_is_the_tuned_operating_point() {
// A sweep over the full LongMemEval haystack found 0.7/0.3 strictly
// dominated by 0.4/0.6 (BENCHMARKS.md). This guards the finding
// against being quietly undone.
assert_eq!(
DEFAULT_FUSION,
Fusion::Weighted {
vector: 0.4,
keyword: 0.6
}
);
}
#[test]
fn rrf_rewards_agreement_between_the_stages_and_ignores_magnitudes() {
// Doc 1 is second-best in both stages; doc 0 is best in one and absent
// from the other. RRF prefers the doc both stages liked.
let vec_scores = vec![(0, 100.0), (1, 0.9)];
let kw_scores = vec![(2, 5.0), (1, 4.9)];
let ranked = fuse(vec_scores, kw_scores, Fusion::Rrf { k: 60.0 }, 3);
assert_eq!(ranked[0].0, 1, "{ranked:?}");
// Scaling one stage's scores cannot change an RRF ranking, only the
// order within that stage can.
let a = fuse(
vec![(0, 1.0), (1, 0.5)],
vec![(1, 2.0), (0, 1.0)],
Fusion::Rrf { k: 60.0 },
2,
);
let b = fuse(
vec![(0, 1e6), (1, -3.0)],
vec![(1, 0.002), (0, 0.001)],
Fusion::Rrf { k: 60.0 },
2,
);
assert_eq!(
a.iter().map(|r| r.0).collect::<Vec<_>>(),
b.iter().map(|r| r.0).collect::<Vec<_>>()
);
}
#[test]
fn merge_top_k_matches_a_full_sort() {
// Many ties (scores repeat) so the index tie-break is exercised.