fix(agent): re-ranking threw away the retrieval score

reranker::rerank built its combined score from temporal decay, source
authority and Hebbian activation. RerankInput carried no relevance score, so
it could not have used one: re-ranking a candidate pool reordered it purely by
age and discarded the retriever's ordering. The OpenClaw backend re-ranks
every search, so that was its shipping behaviour.

Measured over the full LongMemEval haystack (500 questions, real MiniLM
embeddings), ordering by metadata alone costs 40.6pp of Hit@1 (11.0% vs 51.6%)
and two thirds of MRR (0.1829 vs 0.6430) — the results are the newest memories
in the pool rather than the ones answering the question.

RerankInput::relevance and ReRankConfig::relevance_weight (1.0 by default)
make relevance lead, with the metadata signals breaking near-ties. Retrieval
is preserved (Hit@1 52.0%, +0.4pp against no re-ranking; MRR -0.003) and
recency discrimination improves 6-7pp, from chance to ~52%.

A half-life sweep (1, 7, 30, 90 days) moves recency 1.4pp and MRR 0.003 —
inside the noise — because the temporal term is capped by its weight while
relevance gaps are larger. The 24-hour default is kept: there is no measured
reason to change it. The two ends of the trade-off are recorded in
BENCHMARKS.md rather than just the good news.

Breaking: RerankInput and ReRankConfig gained fields.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-19 20:03:48 -07:00
co-authored by Claude Opus 5
parent 1e18ff5a86
commit 8ea455bbcb
2 changed files with 63 additions and 0 deletions
+23
View File
@@ -1,5 +1,28 @@
# Changelog
## Unreleased
### Retrieval quality
- `clawhdf5-agent`: **re-ranking discarded the retrieval score.**
`reranker::rerank` built its combined score from temporal decay, source
authority and Hebbian activation only — `RerankInput` had no relevance field
— so re-ranking a candidate pool reordered it by age and threw the
retriever's ordering away. The OpenClaw backend re-ranked every search, so
this was its shipping behaviour: measured over the full LongMemEval haystack
it cost **40.6pp of Hit@1** (11.0% vs 51.6%) and two thirds of MRR (0.183 vs
0.643). `RerankInput::relevance` and `ReRankConfig::relevance_weight` (1.0 by
default) fix it: relevance leads and the metadata signals break near-ties,
which restores retrieval (Hit@1 +0.4pp vs no re-ranking) and improves
recency discrimination by 67pp. **Breaking:** `RerankInput` and
`ReRankConfig` gained fields, so literal constructions need updating;
`..Default::default()` does not.
- `clawhdf5-bench`: the LongMemEval harness feeds the dataset's real session
dates to the store instead of a synthetic counter (decay needs true
intervals, not just the right order), and reports `newest_gold_first` — on a
`knowledge-update` question, did the newest gold session outrank the stale
one it supersedes? Plain recall cannot see this, because both are labelled
gold. New `--rerank-sweep`.
## v2.5.0 (2026-09-19)
### Upgrade Notes