feat(agent): optional keyword stemming, measured and left off by default

The keyword stage had no stemming, so "training" and "trains" were unrelated
terms. bm25::TokenFilter::Stemmed strips common English inflections (plurals,
-ing/-ed, with consonant un-doubling) from documents and queries alike;
BM25Index::build_with and HDF5Memory::set_token_filter select it, and the
index records which filter built it so a stale one is rebuilt rather than
mixed.

Measured over the full LongMemEval haystack (500 questions, real MiniLM
embeddings) rather than adopted on principle — and it is a trade, not a win:

  BM25 only         Hit@1 53.8%  Hit@5 75.0%  Hit@10 81.6%  MRR 0.6320
  BM25 stemmed      Hit@1 52.0%  Hit@5 77.8%  Hit@10 84.0%  MRR 0.6320
  Hybrid 0.4/0.6    Hit@1 51.6%  Hit@5 81.4%  Hit@10 87.8%  MRR 0.6430
  Hybrid stemmed    Hit@1 50.2%  Hit@5 81.4%  Hit@10 88.2%  MRR 0.6394

Conflation buys depth and costs the top rank: on BM25 alone MRR is unchanged
to four decimal places, the deeper gains exactly offsetting the rank-1 loss.
On the shipping hybrid configuration the vector stage already supplies most of
that recall, so the trade is narrower and slightly negative. Default stays
Plain; Stemmed is there for callers who want Hit@5/@10 over rank-1 precision.

The stemmer is deliberately conservative — it only strips inflections, and
only when the stem stays long enough to be meaningful, since an aggressive one
also conflates unrelated words. Tests pin both the pairs that must meet and
the pairs that must not.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-19 18:20:19 -07:00
co-authored by Claude Opus 5
parent 55ed87d2e8
commit 84a39ef3c5
5 changed files with 259 additions and 11 deletions
+6
View File
@@ -3,6 +3,12 @@
## Unreleased
### Retrieval quality
- `clawhdf5-agent`: optional keyword stemming — `bm25::TokenFilter::Stemmed`
and `HDF5Memory::set_token_filter`, so "training" and "trains" match. **Off
by default**, on measurement rather than principle: over the full LongMemEval
haystack it buys depth and costs the top rank (BM25 alone: Hit@5 +2.8pp,
Hit@10 +2.4pp, Hit@1 1.8pp, MRR unchanged), and on the shipping hybrid
configuration the trade is narrower still. See `BENCHMARKS.md`.
- `clawhdf5-agent`: **`QueryExpander::expand` panicked on ordinary non-ASCII
input** — `"İ AI"` was enough. It searched a lowercased copy of the query and
then sliced the *original* with those offsets, which only works while