`HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the
store's full search path. `SearchOptions::new(k)` is plain hybrid search
with the tuned default fusion; each further stage is opt-in:
- `with_sources([..])`: only records from these source channels. The
filter applies before ranking, so a filtered search still returns up
to k results, normalised over what it can return. The HNSW pool is
over-fetched in proportion to what the filter removes, and the allowed
records are scanned exactly whenever that costs fewer distance
evaluations than the index would (~pool x M) — and as the fallback if
the pool comes back short. Keyword matches are filtered too.
- `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by
relevance, recency, source authority and activation;
`with_confidence(ConfidenceConfig)` drops low-confidence results;
`at_time(now)` pins the recency clock.
These were reachable only through the OpenClaw backend, which is now
`search` with both on. Its Hebbian boost now goes to the k results it
returns rather than the whole 3k candidate pool. `hybrid_search` and
`hybrid_search_with` are wrappers and unchanged (tested bit for bit).
Measured on tank (search_harness --options-study --full, 3 runs): at
100K every filter — 50%, 10%, 1% of the store, and records far from the
query — returns the exact filtered top 10, and none is slower than an
unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs
about 3%. A first version decided between index and exact scan by pool
size vs store size; it measured 0.976 recall at 12.3 ms on the
far-from-query filter, which is why the rule compares costs instead.
Tests: tests/search_options.rs (filter correctness and full pages via
both paths, far-from-query fallback, edge cases, equality with
hybrid_search_with, re-rank recency, confidence, boost scope).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
MemoryCache held every embedding in two places: a `Vec<Vec<f32>>` and a
flattened copy for the batched kernels, kept in lock-step on every push,
update and compaction. A store loaded from disk therefore carried the corpus
twice, plus one heap allocation per entry.
A new `cache::Embeddings` owns just the flat `[N x dim]` buffer and indexes
into it, so `embeddings[i]` still reads as a `&[f32]` row. The batch kernels
take a `VectorSet` (implemented for both `Embeddings` and `Vec<Vec<f32>>`)
instead of `&[Vec<f32>]`, so their callers and tests are unchanged. Loading no
longer unflattens what it just read.
100k 384-dim entries, reopened from disk: 505 -> 357 MiB, 3.44x -> 2.43x the
raw vectors. Recall (1.0000 at ef=64) and query latency are unchanged.
Rows are now always exactly `dim` long, shorter ones zero-padded. The old
representation allowed ragged rows, which silently misaligned the flattened
copy — every row after a wrong-length embedding — and `update` carried a
comment about falling back to a rebuild to avoid exactly that. It is now
unrepresentable. A record saved without an embedding holds a zero row and is
told apart by its norm, which is what `total_embeddings` now counts.
Measured with a counting allocator rather than RSS: freeing a structure
returns its pages to the allocator's pool, not the OS, so an RSS reading from
inside the process showed the two representations as identical.
Breaking: MemoryCache::embeddings changes type, embeddings_flat is replaced by
flat_embeddings(), rebuild_flat() is a deprecated no-op.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
BENCHMARKS.md has recorded since the weight sweep that the 0.7/0.3 default is
strictly dominated by 0.4/0.6 over the full LongMemEval haystack, but the
shipping code never adopted it: unified_search and the OpenClaw backend both
passed 0.7/0.3. Re-running the sweep here (500 questions, real MiniLM
embeddings on a GPU) reproduces it — turn-level Hit@1 51.6% vs 44.2%, Hit@5
81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.6430 vs 0.5856 — so both now use
hybrid::DEFAULT_FUSION, which is that operating point and carries the
reasoning. A unit test pins it.
Fusion is also selectable now. hybrid::Fusion is either Weighted { vector,
keyword } or Rrf { k }; hybrid::fuse applies either to one candidate list per
stage, and merge_vector_keyword / hybrid_search delegate to it, so the public
API is unchanged. New HDF5Memory::hybrid_search_with and
hybrid::hybrid_search_fused take a Fusion. Reciprocal rank fusion was
implemented but reachable only as a free function over a linear scan, so it
had never been compared with the weighted sum on equal terms; it is now a mode
in the LongMemEval bench (measurement to follow).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Fusion min-max normalises over every keyword match, so hybrid_search asked
BM25 for a ranked list of the whole corpus: a hash insert per posting, then a
sort of every match, then the merge sorted every candidate again to keep k.
- BM25Index::scores returns every match unsorted, accumulated in a dense array
(contributions are strictly positive, so zero means untouched). search() is
built on it with the bounded heap.
- merge_vector_keyword partitions out its top k (select_nth) and orders only
those, with the same score-then-id order.
- Both hybrid paths use scores().
Rankings are identical (equivalence tests for both changes). p50 0.24 -> 0.07
ms (1K), 2.1 -> 0.49 ms (10K), 23 -> 4.65 ms (100K).
The harness gains --fusion-study, which measured the alternative — capping the
keyword pool — and found it changes the top-10 for most queries (overlap
0.83-0.92, different #1 for 10-35%) for only a 2x saving. Not adopted.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
test_hebbian_activation_boost failed intermittently. Root causes, all in the
query path:
- normalize_scores mapped a set of identical scores — including the
single-candidate case — to 0.0, so a lone perfect match contributed nothing
to the fused score. Identical positive scores now normalise to 1.0 (all
equally the best match); identical non-positive scores stay 0.0.
- merge_vector_keyword sorted a HashMap's entries by score alone and then
truncated, so which ties survived varied from run to run; hybrid_search had
the same problem in its final sort. Both now break ties by index.
- hybrid_search applied the Hebbian boost to every returned record, including
the zero-score filler that pads the list when fewer than k records match.
With random tie-breaking a filler record could collect as many boosts as the
real hit. Only records with a positive fused score are reinforced now.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Resolves two gaps found in a project-state review:
1. Python build was broken: PyO3/numpy 0.23 caps at Python 3.13 but the
environment has 3.14. Bumped to 0.28 and updated the two breaking APIs
(PyObject -> Py<PyAny>, allow_threads -> detach). The extension module now
imports and round-trips under Python 3.14, unblocking cargo build --workspace.
2. The "HNSW vector search over agent memories" headline was unwired:
clawhdf5-ann had zero dependents and the agent used a linear cosine+BM25 scan.
- clawhdf5-ann is now a live index: insert, mark_deleted (soft delete with a
deleted bitset, traversed but never returned), compact, and a format
version tag (v2) with backward-compatible load of v1 files.
- clawhdf5-agent wires HNSW behind the `hnsw` feature (ON by default). The
index mirrors the cache (node id == cache index) and self-heals: it rebuilds
whenever hnsw_synced_len drifts from cache.len(), so unhooked pushes can't
desync it. Non-indexable stores (no/zero-dim/mixed embeddings) and queries
whose dim doesn't match fall back to the exact linear scan.
- hybrid.rs gains merge_vector_keyword, shared by the linear and HNSW paths.
- tests/hnsw_integration.rs validates recall vs a brute-force oracle plus
insert/delete/batch behaviour.
Disable HNSW for exact search with `--no-default-features --features float16`.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
