fix(agent): query expansion panicked on non-ASCII and rewrote text inside words
Probing what QueryExpander::expand actually produces for LongMemEval questions
turned up two defects in the same helper.
`replace_word_case_insensitive` did a plain substring replace despite its
name, so acronym expansion fired inside ordinary words: "training" became
"trArtificial Intelligencening" ("ai") and "programming" became
"Pull Requestogramming" ("pr"). Nearly every acronym expansion of prose was
corrupt. Matching now requires word boundaries at both ends; real acronyms
(API, database) still expand in both directions.
The same helper searched `text.to_lowercase()` and then sliced `text` with the
offsets it found. That holds only while lowercasing preserves byte length, and
it does not — Turkish 'İ' is 2 bytes and lowercases to 3. Offsets after such a
character drifted, so output was silently corrupted ("İstanbul AI trip" lost a
character) or the slice landed inside a character or past the end and
panicked: `expand("İ AI")` was enough, from a plain query string. Matching now
walks the original string, comparing case-insensitively char by char, so
offsets are always valid.
Regression tests cover both, plus whole-word matching at string edges. The
morphological rules remain crude ("during" -> "dured"); that is a quality
limit, not a correctness bug, and is now documented as a reason to measure
before enabling expansion on a retrieval path.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
@@ -3,6 +3,19 @@
|
||||
## Unreleased
|
||||
|
||||
### Retrieval quality
|
||||
- `clawhdf5-agent`: **`QueryExpander::expand` panicked on ordinary non-ASCII
|
||||
input** — `"İ AI"` was enough. It searched a lowercased copy of the query and
|
||||
then sliced the *original* with those offsets, which only works while
|
||||
lowercasing preserves byte length (Turkish `İ` is 2 bytes and lowercases to
|
||||
3). Depending on where the offsets drifted it either corrupted the output
|
||||
("İstanbul AI trip" lost a character) or panicked. Matching now walks the
|
||||
original string.
|
||||
- `clawhdf5-agent`: query expansion no longer rewrites text inside words.
|
||||
`replace_word_case_insensitive` did a plain substring replace despite its
|
||||
name, so "training" became "trArtificial Intelligencening" and "programming"
|
||||
became "Pull Requestogramming" — every acronym expansion of ordinary prose
|
||||
was corrupt. Matches now require word boundaries; genuine acronyms
|
||||
(`API`, `database`) still expand.
|
||||
- `clawhdf5-agent`: **the default fusion weights are now the measured ones.**
|
||||
A sweep of every 0.1 step over the full LongMemEval haystack (500 questions,
|
||||
real MiniLM embeddings) shows the long-standing `0.7/0.3` default is
|
||||
|
||||
Reference in New Issue
Block a user