feat(agent): HDF5Memory::search with source filters, re-ranking, confidence

`HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the
store's full search path. `SearchOptions::new(k)` is plain hybrid search
with the tuned default fusion; each further stage is opt-in:

- `with_sources([..])`: only records from these source channels. The
  filter applies before ranking, so a filtered search still returns up
  to k results, normalised over what it can return. The HNSW pool is
  over-fetched in proportion to what the filter removes, and the allowed
  records are scanned exactly whenever that costs fewer distance
  evaluations than the index would (~pool x M) — and as the fallback if
  the pool comes back short. Keyword matches are filtered too.
- `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by
  relevance, recency, source authority and activation;
  `with_confidence(ConfidenceConfig)` drops low-confidence results;
  `at_time(now)` pins the recency clock.

These were reachable only through the OpenClaw backend, which is now
`search` with both on. Its Hebbian boost now goes to the k results it
returns rather than the whole 3k candidate pool. `hybrid_search` and
`hybrid_search_with` are wrappers and unchanged (tested bit for bit).

Measured on tank (search_harness --options-study --full, 3 runs): at
100K every filter — 50%, 10%, 1% of the store, and records far from the
query — returns the exact filtered top 10, and none is slower than an
unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs
about 3%. A first version decided between index and exact scan by pool
size vs store size; it measured 0.976 recall at 12.3 ms on the
far-from-query filter, which is why the rule compares costs instead.

Tests: tests/search_options.rs (filter correctness and full pages via
both paths, far-from-query fallback, edge cases, equality with
hybrid_search_with, re-rank recency, confidence, boost scope).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-24 16:35:42 -05:00
co-authored by Claude Opus 5.5
parent d0db83812b
commit c470244a6f
10 changed files with 951 additions and 110 deletions
+22
View File
@@ -40,6 +40,28 @@
`quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
out. The CLI's `--quantized-index` is still accepted but is now a no-op.
### Search
- `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
filtering, re-ranking and confidence rejection in the store's own search
path. Re-ranking and confidence rejection used to be reachable only
through the OpenClaw backend, which now calls `search` with both on.
- `with_sources([..])` restricts a search to records from those source
channels. It applies before ranking, so a filtered search still returns up
to `k` results, normalised over what it can return. Measured at 100K: the
exact filtered top 10 for filters keeping 50%, 10% and 1% of the store and
for records far from the query, and never slower than an unfiltered search
(2.3 ms for a 1% filter vs 4.6 ms unfiltered). See `BENCHMARKS.md`,
"Search options".
- `with_rerank(ReRankConfig)` re-ranks a pool of `max(3k, 10)` candidates
(`rerank_pool` to change it) by relevance, recency, source authority and
activation; `with_confidence(ConfidenceConfig)` drops low-confidence
results; `at_time(now)` pins the clock for recency. About 3% on latency.
- `hybrid_search` and `hybrid_search_with` are unchanged (tested bit for
bit against `search` with default options).
- `clawhdf5-agent`: the OpenClaw backend's search now boosts the Hebbian
activation of the `k` results it returns, not of the whole `3k` candidate
pool it re-ranks.
### Interop
- `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
libhdf5.** The float datatype encoder hard-coded the sign bit's position to