feat(agent): optional keyword stemming, measured and left off by default
The keyword stage had no stemming, so "training" and "trains" were unrelated terms. bm25::TokenFilter::Stemmed strips common English inflections (plurals, -ing/-ed, with consonant un-doubling) from documents and queries alike; BM25Index::build_with and HDF5Memory::set_token_filter select it, and the index records which filter built it so a stale one is rebuilt rather than mixed. Measured over the full LongMemEval haystack (500 questions, real MiniLM embeddings) rather than adopted on principle — and it is a trade, not a win: BM25 only Hit@1 53.8% Hit@5 75.0% Hit@10 81.6% MRR 0.6320 BM25 stemmed Hit@1 52.0% Hit@5 77.8% Hit@10 84.0% MRR 0.6320 Hybrid 0.4/0.6 Hit@1 51.6% Hit@5 81.4% Hit@10 87.8% MRR 0.6430 Hybrid stemmed Hit@1 50.2% Hit@5 81.4% Hit@10 88.2% MRR 0.6394 Conflation buys depth and costs the top rank: on BM25 alone MRR is unchanged to four decimal places, the deeper gains exactly offsetting the rank-1 loss. On the shipping hybrid configuration the vector stage already supplies most of that recall, so the trade is narrower and slightly negative. Default stays Plain; Stemmed is there for callers who want Hit@5/@10 over rank-1 precision. The stemmer is deliberately conservative — it only strips inflections, and only when the stem stays long enough to be meaningful, since an aggressive one also conflates unrelated words. Tests pin both the pairs that must meet and the pairs that must not. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
@@ -3,6 +3,12 @@
|
||||
## Unreleased
|
||||
|
||||
### Retrieval quality
|
||||
- `clawhdf5-agent`: optional keyword stemming — `bm25::TokenFilter::Stemmed`
|
||||
and `HDF5Memory::set_token_filter`, so "training" and "trains" match. **Off
|
||||
by default**, on measurement rather than principle: over the full LongMemEval
|
||||
haystack it buys depth and costs the top rank (BM25 alone: Hit@5 +2.8pp,
|
||||
Hit@10 +2.4pp, Hit@1 −1.8pp, MRR unchanged), and on the shipping hybrid
|
||||
configuration the trade is narrower still. See `BENCHMARKS.md`.
|
||||
- `clawhdf5-agent`: **`QueryExpander::expand` panicked on ordinary non-ASCII
|
||||
input** — `"İ AI"` was enough. It searched a lowercased copy of the query and
|
||||
then sliced the *original* with those offsets, which only works while
|
||||
|
||||
Reference in New Issue
Block a user