diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 44cb0f7..39b6dc0 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -597,6 +597,34 @@ Session-level: | Vector only | 85.4% | 94.2% | 96.6% | 0.8901 | | Hybrid | **88.2%** | **95.8%** | **97.8%** | **0.9158** | +### Fusion method — weighted vs. RRF, full haystack, n=500 + +Reciprocal rank fusion has been in the codebase since early on but was only +reachable as a free function over a linear scan, so it had never been compared +with the weighted sum on equal terms. `HDF5Memory::hybrid_search_with` now +takes a `Fusion`, and both run over the same HNSW + BM25 candidates: + +| Mode | turn Hit@1 | Hit@5 | Hit@10 | MRR | session Hit@1 | session MRR | +|---|---|---|---|---|---|---| +| BM25 only | **53.8%** | 75.0% | 81.6% | 0.6320 | 86.2% | 0.8948 | +| Vector only | 36.0% | 71.8% | 81.6% | 0.5031 | 85.4% | 0.8901 | +| **Weighted 0.4 / 0.6** | 51.6% | **81.4%** | **87.8%** | **0.6430** | **91.0%** | **0.9347** | +| RRF (k=60) | 45.0% | 78.8% | 87.6% | 0.5967 | 89.6% | 0.9253 | + +**RRF loses to the tuned weighted sum** — 6.6pp of turn Hit@1 and 0.046 of MRR +— and lands almost exactly where the old `0.7/0.3` weighting did (44.2% / +0.5856). That is not a coincidence: RRF combines the two stages by rank with +*equal* influence, and on this corpus the stages are not equally good. BM25 +alone beats the vector stage by 17.8pp at Hit@1, so any scheme that treats them +as peers gives up rank-1 accuracy, and RRF discards the score magnitudes that +would say which stage to believe. + +This is a property of the corpus, not a defect in RRF: its selling point is +robustness when the two stages' scores are not comparable and there is no +labelled data to tune against. Here there is, so the weighted sum is kept as +the default. `Fusion::Rrf` remains available for callers whose stages are more +evenly matched. + ### Weight sweep — full haystack, n=500 `0.7/0.3` was a documented default, never a searched one. Sweeping @@ -639,7 +667,8 @@ BM25 at Hit@1. Both dominate `0.7/0.3`. The rows below are kept at the three original settings because they are what the mode ablation measured — read them as "the shape of each stage in isolation", -and take the operating point from the sweep. +and take the operating point from the sweep. `0.4/0.6` is now the shipped +default (`hybrid::DEFAULT_FUSION`). The same pattern shows up independently in omni-cortex's four-signal RRF ablation, where adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR.