osobhandClaude Opus 5
84a39ef3c5
The keyword stage had no stemming, so "training" and "trains" were unrelated terms. bm25::TokenFilter::Stemmed strips common English inflections (plurals, -ing/-ed, with consonant un-doubling) from documents and queries alike; BM25Index::build_with and HDF5Memory::set_token_filter select it, and the index records which filter built it so a stale one is rebuilt rather than mixed. Measured over the full LongMemEval haystack (500 questions, real MiniLM embeddings) rather than adopted on principle — and it is a trade, not a win: BM25 only Hit@1 53.8% Hit@5 75.0% Hit@10 81.6% MRR 0.6320 BM25 stemmed Hit@1 52.0% Hit@5 77.8% Hit@10 84.0% MRR 0.6320 Hybrid 0.4/0.6 Hit@1 51.6% Hit@5 81.4% Hit@10 87.8% MRR 0.6430 Hybrid stemmed Hit@1 50.2% Hit@5 81.4% Hit@10 88.2% MRR 0.6394 Conflation buys depth and costs the top rank: on BM25 alone MRR is unchanged to four decimal places, the deeper gains exactly offsetting the rank-1 loss. On the shipping hybrid configuration the vector stage already supplies most of that recall, so the trade is narrower and slightly negative. Default stays Plain; Stemmed is there for callers who want Hit@5/@10 over rank-1 precision. The stemmer is deliberately conservative — it only strips inflections, and only when the stem stays long enough to be meaningful, since an aggressive one also conflates unrelated words. Tests pin both the pairs that must meet and the pairs that must not. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
clawhdf5-agent
HDF5-backed persistent memory store for on-device AI agents.
Built on clawhdf5, clawhdf5-agent provides a vector-searchable memory backend optimized for edge AI workloads. Store embeddings, text chunks, and metadata in a single HDF5 file with SIMD-accelerated similarity search.
Features
- Persistent vector store in HDF5 format
- Cosine similarity and L2 distance search
- SIMD-accelerated via clawhdf5-accel (AVX2, NEON)
- Optional GPU acceleration via clawhdf5-gpu
- Memory-mapped access for large stores
- f16 storage support for compact embeddings
Usage
[dependencies]
clawhdf5-agent = "2.1.0"
License
MIT