quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

Search options (source filters, re-ranking, confidence); float16 default #5

Open
osobh wants to merge 3 commits from feat/search-options into main
pull from: feat/search-options
merge into: :main
Conversation 0 Commits 3 Files Changed 13
+1100 -136
osobh commented 2026-09-25 03:18:50 +00:00
Owner
Copy Link
Copy Source

Summary

Three commits: search options, the real-embedding check for float16, and float16 as the default for new stores.

HDF5Memory::search with SearchOptions (c470244)

The full search path, with each stage beyond plain hybrid search opt-in:

  • with_sources([..]): only records from those source channels. Applied before ranking, so a filtered search still returns up to k results. The HNSW pool is over-fetched in proportion to what the filter removes. The allowed records are scanned exactly whenever that's cheaper than the index (roughly pool × M distance evaluations), and also as the fallback if the pool comes back short.
  • with_rerank(..) / with_confidence(..) / at_time(..): re-ranking and confidence rejection, previously reachable only through the OpenClaw backend. The backend is now search with both on.
  • hybrid_search / hybrid_search_with are unchanged; they're tested bit for bit against search with default options.
100K × 384, tank, median of 3 filtered recall@10 p50
No filter 0.9995 4.60 ms
Filter keeping 50% / 10% / 1% 1.0000 / 1.0000 / 1.0000 4.77 / 3.24 / 2.29 ms
Filter to records far from the query 1.0000 2.30 ms
Re-rank + confidence — 4.75 ms

Behaviour change: the OpenClaw backend now boosts activation only for the k results it returns, not its whole 3k candidate pool.

float16 on real embeddings (dbaf3f5)

longmemeval_bench --float16. On the full longmemeval_s haystack (500 questions) with real all-MiniLM-L6-v2 embeddings, float16 and f32 are identical at every Hit@k and MRR, at turn and session level, in all eight modes. The only differences are one MRR differing by 0.0001, and one or two ties out of about 320 in which gold session ranks first.

New stores default to float16 (5c8323c)

  • Existing stores keep f32: each recorded float16 = false, and a v2.5.0 fixture test checks it survives a save and checkpoint bit for bit.
  • Opt-out: float16 = false or create --f32; --float16 is now a no-op.
  • Values beyond ±65504 are refused, so the upgrade note recommends f32 for unnormalised vectors.

Test plan

  • scripts/ci-test.sh locally on tank: 14/14 steps, including MSRV 1.92 and h5py interop
  • tests/search_options.rs: filter correctness and full pages (index path and exact path), the far-from-query fallback, edge cases, equality with hybrid_search_with, recency re-ranking, confidence, boost scope
  • tests/float16_store.rs: new stores default to float16; the v2.5.0 fixture stays f32
  • search_harness --options-study --full (3 runs), and LongMemEval f32 vs float16 (CUDA)
  • Gitea CI on this PR

🤖 Generated with Claude Code

## Summary Three commits: search options, the real-embedding check for `float16`, and `float16` as the default for new stores. ### `HDF5Memory::search` with `SearchOptions` (`c470244`) The full search path, with each stage beyond plain hybrid search opt-in: - **`with_sources([..])`:** only records from those source channels. Applied *before* ranking, so a filtered search still returns up to `k` results. The HNSW pool is over-fetched in proportion to what the filter removes. The allowed records are scanned exactly whenever that's cheaper than the index (roughly `pool × M` distance evaluations), and also as the fallback if the pool comes back short. - **`with_rerank(..)` / `with_confidence(..)` / `at_time(..)`:** re-ranking and confidence rejection, previously reachable only through the OpenClaw backend. The backend is now `search` with both on. - **`hybrid_search` / `hybrid_search_with`** are unchanged; they're tested bit for bit against `search` with default options. | 100K × 384, tank, median of 3 | filtered recall@10 | p50 | |---|---:|---:| | No filter | 0.9995 | 4.60 ms | | Filter keeping 50% / 10% / 1% | 1.0000 / 1.0000 / 1.0000 | 4.77 / 3.24 / 2.29 ms | | Filter to records far from the query | 1.0000 | 2.30 ms | | Re-rank + confidence | — | 4.75 ms | **Behaviour change:** the OpenClaw backend now boosts activation only for the `k` results it returns, not its whole 3k candidate pool. ### `float16` on real embeddings (`dbaf3f5`) `longmemeval_bench --float16`. On the full `longmemeval_s` haystack (500 questions) with real all-MiniLM-L6-v2 embeddings, `float16` and `f32` are identical at every Hit@k and MRR, at turn and session level, in all eight modes. The only differences are one MRR differing by 0.0001, and one or two ties out of about 320 in which gold session ranks first. ### New stores default to `float16` (`5c8323c`) - Existing stores keep `f32`: each recorded `float16 = false`, and a v2.5.0 fixture test checks it survives a save and checkpoint bit for bit. - Opt-out: `float16 = false` or `create --f32`; `--float16` is now a no-op. - Values beyond ±65504 are refused, so the upgrade note recommends `f32` for unnormalised vectors. ## Test plan - [x] `scripts/ci-test.sh` locally on tank: 14/14 steps, including MSRV 1.92 and h5py interop - [x] `tests/search_options.rs`: filter correctness and full pages (index path and exact path), the far-from-query fallback, edge cases, equality with `hybrid_search_with`, recency re-ranking, confidence, boost scope - [x] `tests/float16_store.rs`: new stores default to `float16`; the v2.5.0 fixture stays `f32` - [x] `search_harness --options-study --full` (3 runs), and LongMemEval `f32` vs `float16` (CUDA) - [ ] Gitea CI on this PR 🤖 Generated with [Claude Code](https://claude.com/claude-code)
osobh added 3 commits 2026-09-25 03:18:50 +00:00
feat(agent): HDF5Memory::search with source filters, re-ranking, confidence c470244a6f 
`HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the
store's full search path. `SearchOptions::new(k)` is plain hybrid search
with the tuned default fusion; each further stage is opt-in:

- `with_sources([..])`: only records from these source channels. The
  filter applies before ranking, so a filtered search still returns up
  to k results, normalised over what it can return. The HNSW pool is
  over-fetched in proportion to what the filter removes, and the allowed
  records are scanned exactly whenever that costs fewer distance
  evaluations than the index would (~pool x M) — and as the fallback if
  the pool comes back short. Keyword matches are filtered too.
- `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by
  relevance, recency, source authority and activation;
  `with_confidence(ConfidenceConfig)` drops low-confidence results;
  `at_time(now)` pins the recency clock.

These were reachable only through the OpenClaw backend, which is now
`search` with both on. Its Hebbian boost now goes to the k results it
returns rather than the whole 3k candidate pool. `hybrid_search` and
`hybrid_search_with` are wrappers and unchanged (tested bit for bit).

Measured on tank (search_harness --options-study --full, 3 runs): at
100K every filter — 50%, 10%, 1% of the store, and records far from the
query — returns the exact filtered top 10, and none is slower than an
unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs
about 3%. A first version decided between index and exact scan by pool
size vs store size; it measured 0.976 recall at 12.3 ms on the
far-from-query filter, which is why the rule compares costs instead.

Tests: tests/search_options.rs (filter correctness and full pages via
both paths, far-from-query fallback, edge cases, equality with
hybrid_search_with, re-rank recency, confidence, boost scope).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
bench(longmemeval): --float16, and float16 measured on real embeddings dbaf3f505d 
`longmemeval_bench --float16` builds every per-question store with
MemoryConfig::float16, so the vector stage searches half-rounded
embeddings exactly as such a store holds them.

Full longmemeval_s (500 questions, ~494 turns each) with real
all-MiniLM-L6-v2 embeddings, f32 vs float16, on tank (CUDA): identical
at every Hit@k and MRR, turn and session level, in all eight modes —
bar RRF session MRR 0.9253 vs 0.9254 and one or two flips out of ~320
in which gold session ranks first. The f32 run reproduces the published
hybrid numbers exactly. The earlier float16 evidence was synthetic
clustered data only; this is the real-embedding check.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
feat(agent): new stores default to float16 embeddings
CI / test (pull_request) Successful in 5m27s
Details
CI / test-arm64 (pull_request) Successful in 1m9s
Details
5c8323cb1e 
MemoryConfig::float16 now defaults to true for new stores, on
measurement: on the full LongMemEval haystack with real MiniLM
embeddings every retrieval metric matched f32 (previous commit), and at
100K the file is 48% smaller with faster checkpoints and opens.

Existing stores are unaffected: every agent store has recorded
`float16 = false` in /meta and keeps it. A test opens the v2.5.0
fixture, saves and checkpoints, and checks the embeddings are still f32
with the old rows bit-identical; another checks a new store is float16.

CLI: `create --f32` opts out; like `--f32-index` it only ever switches
the default off. `--float16` is still accepted and now a no-op.
Values beyond +-65504 are refused, so f32 remains the choice for
unnormalised vectors — the upgrade note says so.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
All checks were successful
CI / test-arm64 (pull_request) Successful in 1m9s
Details
CI / test (pull_request) Successful in 5m27s
Details
You are not authorized to merge this pull request.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/search-options:feat/search-options
git checkout feat/search-options
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
nsobh osobh scottb (Scott B.)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#5