fix(agent): query expansion panicked on non-ASCII and rewrote text inside words

Probing what QueryExpander::expand actually produces for LongMemEval questions
turned up two defects in the same helper.

`replace_word_case_insensitive` did a plain substring replace despite its
name, so acronym expansion fired inside ordinary words: "training" became
"trArtificial Intelligencening" ("ai") and "programming" became
"Pull Requestogramming" ("pr"). Nearly every acronym expansion of prose was
corrupt. Matching now requires word boundaries at both ends; real acronyms
(API, database) still expand in both directions.

The same helper searched `text.to_lowercase()` and then sliced `text` with the
offsets it found. That holds only while lowercasing preserves byte length, and
it does not — Turkish 'İ' is 2 bytes and lowercases to 3. Offsets after such a
character drifted, so output was silently corrupted ("İstanbul AI trip" lost a
character) or the slice landed inside a character or past the end and
panicked: `expand("İ AI")` was enough, from a plain query string. Matching now
walks the original string, comparing case-insensitively char by char, so
offsets are always valid.

Regression tests cover both, plus whole-word matching at string edges. The
morphological rules remain crude ("during" -> "dured"); that is a quality
limit, not a correctness bug, and is now documented as a reason to measure
before enabling expansion on a retrieval path.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-19 17:25:20 -07:00
co-authored by Claude Opus 5
parent aa92fef7bb
commit 6531158d9f
2 changed files with 175 additions and 8 deletions
+13
View File
@@ -3,6 +3,19 @@
## Unreleased
### Retrieval quality
- `clawhdf5-agent`: **`QueryExpander::expand` panicked on ordinary non-ASCII
input** — `"İ AI"` was enough. It searched a lowercased copy of the query and
then sliced the *original* with those offsets, which only works while
lowercasing preserves byte length (Turkish `İ` is 2 bytes and lowercases to
3). Depending on where the offsets drifted it either corrupted the output
("İstanbul AI trip" lost a character) or panicked. Matching now walks the
original string.
- `clawhdf5-agent`: query expansion no longer rewrites text inside words.
`replace_word_case_insensitive` did a plain substring replace despite its
name, so "training" became "trArtificial Intelligencening" and "programming"
became "Pull Requestogramming" — every acronym expansion of ordinary prose
was corrupt. Matches now require word boundaries; genuine acronyms
(`API`, `database`) still expand.
- `clawhdf5-agent`: **the default fusion weights are now the measured ones.**
A sweep of every 0.1 step over the full LongMemEval haystack (500 questions,
real MiniLM embeddings) shows the long-standing `0.7/0.3` default is