feat(agent): new stores default to float16 embeddings

MemoryConfig::float16 now defaults to true for new stores, on measurement: on the full LongMemEval haystack with real MiniLM embeddings every retrieval metric matched f32 (previous commit), and at 100K the file is 48% smaller with faster checkpoints and opens. Existing stores are unaffected: every agent store has recorded `float16 = false` in /meta and keeps it. A test opens the v2.5.0 fixture, saves and checkpoints, and checks the embeddings are still f32 with the old rows bit-identical; another checks a new store is float16. CLI: `create --f32` opts out; like `--f32-index` it only ever switches the default off. `--float16` is still accepted and now a no-op. Values beyond +-65504 are refused, so f32 remains the choice for unnormalised vectors — the upgrade note says so. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
bench(longmemeval): --float16, and float16 measured on real embeddings
2026-09-24 19:39:33 -05:00 · 2026-09-24 19:21:14 -05:00 · 2026-09-24 16:35:42 -05:00 · 2026-09-24 12:00:38 -05:00 · 2026-09-24 12:00:26 -05:00
25 changed files with 2304 additions and 155 deletions
@@ -149,6 +149,107 @@ vectors, and recall is measured against brute-force ground truth rather than
 against the f32 index, whose own approximation errors a re-scored search is
 entitled to get right.
 ### Search options: source filters, re-ranking, confidence
 Measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D). `HDF5Memory::search`
 with `SearchOptions`, clustered 384-dim data, k = 10, Hebbian boosting off.
 Filters keep 50%, 10% or 1% of the store at random, or two whole clusters
 chosen *away* from each query's own — the case an ANN index handles worst,
 because nothing it finds near the query is allowed. Recall is vector-only
 against an exact scan of the allowed records; latency is full hybrid search
 (vector + BM25). 200 queries, medians of three runs (recall was identical in
 every run).
 ```bash
 cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
 ```
 | N | options | filtered recall@10 | p50 ms | p99 ms |
 |---:|---|---:|---:|---:|
 | 10 000 | no filter | 1.0000 | 0.488 | 0.518 |
 | 10 000 | random 50% | 1.0000 | 0.520 | 0.563 |
 | 10 000 | random 10% | 1.0000 | 0.342 | 0.367 |
 | 10 000 | random 1% | 1.0000 | 0.220 | 0.239 |
 | 10 000 | 2 clusters away from the query | 1.0000 | 0.252 | 0.278 |
 | 10 000 | re-rank | — | 0.562 | 0.674 |
 | 10 000 | re-rank + confidence | — | 0.554 | 0.575 |
 | 100 000 | no filter | 0.9995 | 4.600 | 5.245 |
 | 100 000 | random 50% | 1.0000 | 4.770 | 5.614 |
 | 100 000 | random 10% | 1.0000 | 3.244 | 4.008 |
 | 100 000 | random 1% | 1.0000 | 2.288 | 2.943 |
 | 100 000 | 2 clusters away from the query | 1.0000 | 2.298 | 3.048 |
 | 100 000 | re-rank | — | 4.747 | 5.411 |
 | 100 000 | re-rank + confidence | — | 4.748 | 5.526 |
 A filtered search finds the exact filtered top 10 and is never slower than an
 unfiltered one. The filter applies before ranking — over-fetching the index in
 proportion to what the filter removes, and scanning the allowed records
 exactly whenever that costs fewer distance evaluations than the index would
 (roughly `pool × M`). The first version compared the over-fetch to the store
 size instead, and that measured badly at 100K: the 1% filter took 5.9 ms at
 recall 0.9965 and the away-from-query filter 12.3 ms at 0.976, both through an
 index asked for ~16 000 candidates, where scanning the few hundred or thousand
 allowed records is exact and cheap. Re-ranking a 3k candidate pool and
 confidence rejection add about 3%.
 ### float16 embedding storage (`MemoryConfig::float16`)
 Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered
 384-dim data in an `f32` store and a `float16` store, both with the default
 int8 index and Hebbian boosting off (so every query sees the same store).
 Recall is vector-only `hybrid_search` against an exact scan of the original
 `f32` vectors, 200 queries. Six runs, three with each store going first;
 medians. Nothing depended on the order.
 ```bash
 cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
 cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full --f16-first
 ```
 | N | embeddings | file MiB | checkpoint ms | open ms | recall@10 | top-10 overlap | hybrid p50 ms |
 |---:|---|---:|---:|---:|---:|---:|---:|
 | 1 000 | f32 | 1.6 | 8 | 1.6 | 1.0000 | | 0.072 |
 | 1 000 | float16 | 0.8 | 6 | 1.9 | 0.9980 | 0.9980 | 0.072 |
 | 10 000 | f32 | 15.4 | 66 | 15.2 | 1.0000 | | 0.495 |
 | 10 000 | float16 | 8.2 | 45 | 18.1 | 1.0000 | 1.0000 | 0.494 |
 | 100 000 | f32 | 154.0 | 752 | 299.8 | 0.9940 | | 4.676 |
 | 100 000 | float16 | **80.8** | **512** | **252.1** | 0.9990 | 0.9940 | 4.654 |
 The file is 48% smaller, checkpoints write less and open reads less. At
 small N opening is slightly slower (widening halves costs more than the I/O it
 saves: +3 ms at 10K). Recall does not move: half precision keeps about three
 significant digits, far finer than the gaps between neighbours on unit-length
 embeddings. Recall was identical in every run; the 0.999 against 0.994 at
 100K is two slightly different HNSW graphs, not an improvement to claim.
 The in-memory cache holds the half-rounded values, so the store searches the
 same before and after a reopen; RAM use is unchanged (the cache is still
 `f32`). What `float16` saves is disk, and the I/O that goes with it.
 **On real embeddings.** The table above is synthetic clustered data. The full
 LongMemEval haystack (`longmemeval_s`, 500 questions, ~494 turns each) with
 real all-MiniLM-L6-v2 embeddings, run once with `f32` stores and once with
 `--float16`, measured 2026-09-24 on tank (embeddings on an RTX 5060 Ti):
 ```bash
 cargo run --release -p clawhdf5-bench --bin longmemeval_bench --features embeddings-cuda -- \
  benchmarks/longmemeval/longmemeval_s.json --embeddings weights/all-minilm-l6-v2 [--float16]
 ```
 | mode | turn Hit@1 | turn Hit@5 | turn Hit@10 | turn MRR | session Hit@5 | session MRR |
 |---|---:|---:|---:|---:|---:|---:|
 | Hybrid 0.4 / 0.6, f32 | 51.6% | 81.4% | 87.8% | 0.6430 | 96.8% | 0.9347 |
 | Hybrid 0.4 / 0.6, float16 | 51.6% | 81.4% | 87.8% | 0.6430 | 96.8% | 0.9347 |
 | Vector only, f32 | 36.0% | 71.8% | 81.6% | 0.5031 | 94.2% | 0.8901 |
 | Vector only, float16 | 36.0% | 71.8% | 81.6% | 0.5031 | 94.2% | 0.8901 |
 All eight modes the harness runs (BM25, vector, hybrid, RRF, stemmed, and
 both re-rank variants) were identical at every Hit@k and MRR, turn and session
 level, except RRF's session MRR (0.9253 vs 0.9254) and one or two flips in
 which of two gold sessions ranks first, out of ~320. Those flips show the
 half-precision path was in effect; they do not change a single hit. The f32
 run reproduces the published hybrid numbers exactly.
 ### Opening a store (`read_from_disk`)
 `HDF5Memory::open` memory-mapped the file, copied the whole mapping into a
@@ -3,6 +3,29 @@
 ## Unreleased
 ### Upgrade Notes
 - **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
  dataset we wrote — including every agent store's embeddings — was refused
  with "sign bit position out of bounds", and every empty dataset with
  "invalid dataset size". Both were write-side bugs present in every release;
  clawhdf5's own reader was unaffected. An agent store is rewritten in full at
  each checkpoint, so it becomes readable at its next checkpoint on this
  version; other files with `f32` or empty datasets need rewriting. Details in
  `docs/known-issues.md`.
 - **New stores store embeddings as half precision by default.**
  `MemoryConfig::float16` was persisted and otherwise ignored; it now writes
  `float16` embeddings (48% smaller files at 100K) and rounds each embedding
  to half precision as it is saved — and it defaults to `true` for new
  stores. On the full LongMemEval haystack with real MiniLM embeddings every
  retrieval metric matched `f32`. **Existing stores are unaffected**: every
  agent store has recorded `float16 = false`, and keeps it (a v2.5.0 fixture
  guards this). A store that already had `float16 = true` rounds its
  embeddings when next opened and writes them as `float16` at its next
  checkpoint. Opt out with `float16 = false` or `create --f32`; the CLI's
  `--float16` is still accepted and now a no-op. Values beyond ±65504 are
  refused, so keep `f32` for unnormalised vectors.
 - **Breaking:** `MemoryError` gained `InvalidEntry`, returned when a
  `float16` store is given an embedding value beyond ±65504. Exhaustive
  matches need the new arm.
 - **The default build no longer compiles any C.** Deflate now defaults to the
  pure-Rust zlib-rs instead of zlib-ng, so building the core crates needs
  neither cmake nor a C compiler. Speed on HDF5 reads and writes is within 6%
@@ -23,6 +46,64 @@
  `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
  out. The CLI's `--quantized-index` is still accepted but is now a no-op.
 ### Search
 - `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
  filtering, re-ranking and confidence rejection in the store's own search
  path. Re-ranking and confidence rejection used to be reachable only
  through the OpenClaw backend, which now calls `search` with both on.
  - `with_sources([..])` restricts a search to records from those source
    channels. It applies before ranking, so a filtered search still returns up
    to `k` results, normalised over what it can return. Measured at 100K: the
    exact filtered top 10 for filters keeping 50%, 10% and 1% of the store and
    for records far from the query, and never slower than an unfiltered search
    (2.3 ms for a 1% filter vs 4.6 ms unfiltered). See `BENCHMARKS.md`,
    "Search options".
  - `with_rerank(ReRankConfig)` re-ranks a pool of `max(3k, 10)` candidates
    (`rerank_pool` to change it) by relevance, recency, source authority and
    activation; `with_confidence(ConfidenceConfig)` drops low-confidence
    results; `at_time(now)` pins the clock for recency. About 3% on latency.
  - `hybrid_search` and `hybrid_search_with` are unchanged (tested bit for
    bit against `search` with default options).
 - `clawhdf5-agent`: the OpenClaw backend's search now boosts the Hebbian
  activation of the `k` results it returns, not of the whole `3k` candidate
  pool it re-ranks.
 ### Interop
 - `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
  libhdf5.** The float datatype encoder hard-coded the sign bit's position to
  63, correct only for `f64`; libhdf5 validates it and refused the dataset. It
  is now derived from the type (15 / 31 / 63). Our reader ignores the field,
  and the interop suites only wrote `f64`, which is how it went unnoticed.
 - `clawhdf5-format`: **every empty dataset was unreadable by h5py and
  libhdf5.** It was written with a real address and zero bytes, which trips
  libhdf5's `addr + size <= addr` overflow check. An empty contiguous dataset
  now gets the undefined address, as libhdf5 writes it. This affected every
  agent store without sessions or a knowledge graph.
 - New interop tests: `f32` and `float16` datasets in both directions (our
  `float16` rounding matches numpy's bit for bit on 4 020 probe values,
  including ties, subnormals and the overflow boundary), and an agent store —
  `f32` and `float16` — opened by h5py with every dataset decoded.
 ### Storage
 - `clawhdf5-format`: **half-precision datasets.**
  `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy `float16`),
  rounding to nearest-even; `make_f16_type`, and `clawhdf5_format::float16`
  with the conversions, which are checked against the `half` crate on 16.7M
  values and round-trip all 65 536 half values. Reading `float16` as `f32`
  gained a little-endian fast path.
 - `clawhdf5-agent`: **`MemoryConfig::float16` stores embeddings as half
  precision.** At 100K x 384 the file goes from 154.0 to 80.8 MiB (−48%), a
  checkpoint from 752 to 512 ms and open from 300 to 252 ms, with the same
  vector recall@10 against an exact scan (0.999 vs 0.994) and the same
  `hybrid_search` latency; at 10K open is 3 ms slower. On the full
  LongMemEval haystack with real MiniLM embeddings every retrieval metric is
  identical to `f32` (`longmemeval_bench --float16`). The cache rounds each
  embedding as it is saved, so memory and file agree bit for bit and a store
  returns the same results before and after a reopen (tested). Out-of-range
  values are refused with `MemoryError::InvalidEntry` rather than stored as
  infinity; batches are all or nothing. CLI: `create --float16`. See
  `BENCHMARKS.md`, "float16 embedding storage".
 ### Build
 - **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
  `clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
@@ -48,6 +129,11 @@
  now an error.
 ### Defaults
 - `clawhdf5-agent`: `MemoryConfig::float16` defaults to `true` for new stores,
  measured rather than assumed: identical LongMemEval retrieval on real
  embeddings, 48% smaller files and faster checkpoints and opens at 100K.
  `clawhdf5-cli create --f32` opts out; like `--f32-index`, it only ever
  switches the default off.
 - `clawhdf5-agent`: `MemoryConfig::quantized_index` defaults to `true` for new
  stores. The reason it had been off — that int8 search was slower on ARM —
  did not survive measurement (see Corrections). Stores that predate the
@@ -87,6 +87,26 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
  `export` do). An unreadable WAL (torn header, bad magic) is quarantined to
  `<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
  unknown *newer* version still fails and is left untouched.
 - `MemoryConfig::float16` (**on by default** for new stores, persisted;
  existing stores keep their recorded `false` — guarded by the v2.5.0
  fixture in `tests/float16_store.rs`; CLI opt-out is `create --f32`) writes
  `/memory/embeddings` as IEEE half precision (48% smaller file at 100K;
  LongMemEval with real MiniLM embeddings identical to f32).
  `MemoryCache::half_precision` rounds each embedding as it enters the cache (push, update, WAL replay, and on load of a store still
  `f32` on disk), so memory and file agree bit for bit; the conversions live
  in `clawhdf5_format::float16` and must stay the single implementation.
  Values beyond ±65504 are `MemoryError::InvalidEntry`. Interop: every file
  must open in h5py — `f32` datasets and empty datasets did not until
  2026-09-23 (see `docs/known-issues.md`); the agent's `h5py_interop` test
  guards a whole store.
 - `HDF5Memory::search(query_emb, text, &SearchOptions)` is the full search
  path: optional source-channel filter (applied before ranking; exact scan of
  the allowed records whenever cheaper than `pool × M` index distance
  evaluations, and as the fallback when the pool comes back short), fusion,
  activation scaling, optional re-ranking and confidence rejection.
  `hybrid_search`/`hybrid_search_with` are thin wrappers; the OpenClaw
  backend is `search` with re-rank + confidence on. Measure changes with
  `search_harness --options-study`.
 - `MemoryConfig::compression` is off by default; when on, embeddings are
  deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
 - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
@@ -82,6 +82,19 @@ breaking change, are in [CHANGELOG.md](CHANGELOG.md).
  100K × 384 store to 1.74× the raw vectors. At equal recall it is also faster
  than `f32`: 1.63× QPS on AVX2, 1.18× on a Raspberry Pi 5 (NEON `SDOT`).
 **Interop and search (unreleased)**
 - **Files we write now open in h5py and libhdf5.** Every `f32` dataset —
  including every agent store's embeddings — and every empty dataset was
  refused by libhdf5. Both were write-side bugs in every release; agent stores
  fix themselves at their next checkpoint. See
  [docs/known-issues.md](docs/known-issues.md).
 - `MemoryConfig::float16` now stores half-precision embeddings (it was
  ignored), and is on by default for new stores: 48% smaller files, and
  identical LongMemEval retrieval on real embeddings.
 - `HDF5Memory::search` with `SearchOptions`: filter by source channel (exact
  filtered top-k, never slower than unfiltered), and opt-in re-ranking and
  confidence rejection, which used to be OpenClaw-only.
 **Tooling**
 - CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
  silently) and runs an aarch64 job for the NEON kernels.
@@ -244,6 +257,10 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 | 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
 | 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |
 These figures are `f32` embeddings. New agent stores default to
 `MemoryConfig::float16`, which halves them: 100K × 384 records take 80.8 MiB
 instead of 154.0.
 **In memory** — a store reopened from disk, 384-dim `f32`, measured with a
 counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
@@ -281,12 +298,14 @@ ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory syst
                         └────────┬────────┘
                                  │
                ┌─────────────────▼──────────────────┐
-                │ HDF5Memory::hybrid_search          │
+                │ HDF5Memory::search                 │
                │ optional source-channel filter     │
                │ HNSW vector + BM25 keyword         │
                │ weighted fusion (0.4 / 0.6)        │
                │ × √(Hebbian activation)            │
                └─────────────────┬──────────────────┘
-                                  │  OpenClaw backend adds:
+                                  │  opt-in (SearchOptions);
                                  │  the OpenClaw backend turns both on
                ┌─────────────────▼──────────────────┐
                │ Multi-factor re-ranking            │
                │ relevance · recency · authority ·  │
@@ -322,8 +341,8 @@ directly; the store persists the records, sessions and graph they work over.
 | **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
 | **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
 | **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
-| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Used by the OpenClaw backend |
+| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Opt-in via `SearchOptions::with_rerank`; on in the OpenClaw backend |
-| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches (OpenClaw backend) |
+| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in the OpenClaw backend |
 | **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
 | **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
 | **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
@@ -389,6 +408,31 @@ for result in results {
 }
 ```
 ### Search Options
 ```rust
 use clawhdf5_agent::SearchOptions;
 use clawhdf5_agent::confidence::ConfidenceConfig;
 use clawhdf5_agent::reranker::ReRankConfig;
 // Only memories from these source channels; still a full page of k results.
 let work = memory.search(
    &query_embedding,
    "deadline",
    &SearchOptions::new(5).with_sources(["slack", "email"]),
 );
 // Re-rank by relevance, recency, source authority and activation, then drop
 // low-confidence results — the pipeline the OpenClaw backend runs.
 let careful = memory.search(
    &query_embedding,
    "user preferences",
    &SearchOptions::new(5)
        .with_rerank(ReRankConfig::default())
        .with_confidence(ConfidenceConfig::default()),
 );
 ```
 ### Knowledge Graph
 ```rust
@@ -538,7 +582,7 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
 | Flag | Default | Description |
 |------|---------|-------------|
-| `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). The store itself always writes `f32` embeddings; `MemoryConfig::float16` is recorded in `/meta` but not yet applied |
+| `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). Half-precision *storage* is the `MemoryConfig::float16` setting below, and needs no feature |
 | `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
 | `parallel` | **yes** | Parallel HNSW bulk build (same graph, ~3× faster on 16 cores) and Rayon brute-force search strategies |
 | `zstd` | no | Compress embeddings with Zstd instead of deflate when `MemoryConfig::compression` is on (links libzstd) |
@@ -568,6 +612,16 @@ setting existed keep their `f32` index; opt out for new stores with
 `quantized_index = false` or `clawhdf5-cli create --f32-index`. See
 [BENCHMARKS.md § Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index).
 `MemoryConfig::float16` (**on by default** for new stores) stores the
 embeddings on disk as IEEE half precision (numpy `float16`): at 100K × 384 the
 file drops from 154 to 81 MiB, checkpoints and opens get faster, and on the
 full LongMemEval haystack with real MiniLM embeddings every retrieval metric
 matches `f32`. Embeddings are rounded as they are saved, so the store searches
 the same before and after a reopen; values must lie within ±65504. Existing
 stores keep their setting. Opt out with `float16 = false` or
 `clawhdf5-cli create --f32` — e.g. for unnormalised vectors. See
 [BENCHMARKS.md § float16 embedding storage](BENCHMARKS.md#float16-embedding-storage-memoryconfigfloat16).
 ### `clawhdf5-format`
 | Flag | Default | Description |
@@ -656,8 +710,9 @@ agent_memory.h5
 │   └── ann_generation                     (ties the .ann sidecar to this checkpoint)
 ├── /memory
 │   ├── chunks:              string[N]
-│   ├── embeddings:          f32[N × D]  (chunked; deflate, or Zstd with the
+│   ├── embeddings:          f32[N × D], or f16 for a `float16` store
-│   │                                     `zstd` feature, when compression is on)
+│   │                        (chunked; deflate, or Zstd with the `zstd`
 │   │                         feature, when compression is on)
 │   ├── source_channel:      string[N]
 │   ├── timestamps:          f64[N]
 │   ├── session_ids:         string[N]
@@ -1,6 +1,7 @@
 //! In-memory cache for memory entries, sessions, and knowledge graph.
 use crate::vector_search;
 use clawhdf5_format::float16::round_to_f16;
 /// Every entry's embedding, in one contiguous `[N x dim]` buffer.
 ///
@@ -149,6 +150,11 @@ pub struct MemoryCache {
    pub norms: Vec<f32>,
    /// Hebbian activation weights (default 1.0 per entry).
    pub activation_weights: Vec<f32>,
    /// Round every embedding to IEEE half precision as it enters the cache,
    /// so the cache holds exactly what a `float16` store writes to disk. Set
    /// it with [`MemoryCache::set_half_precision`], which also rounds the
    /// rows already held.
    pub half_precision: bool,
 }
 impl MemoryCache {
@@ -164,9 +170,44 @@ impl MemoryCache {
            embedding_dim,
            norms: Vec::new(),
            activation_weights: Vec::new(),
            half_precision: false,
        }
    }
    /// Switch half-precision rounding on or off. Turning it on rounds every
    /// embedding already held (and recomputes norms where one changed) —
    /// e.g. a `float16` store whose last checkpoint predates half-precision
    /// storage and so is still `f32` on disk.
    pub fn set_half_precision(&mut self, on: bool) {
        self.half_precision = on;
        if !on {
            return;
        }
        for i in 0..self.embeddings.len() {
            let row = &self.embeddings[i];
            if row
                .iter()
                .all(|&v| round_to_f16(v).to_bits() == v.to_bits())
            {
                continue;
            }
            let rounded: Vec<f32> = row.iter().map(|&v| round_to_f16(v)).collect();
            self.norms[i] = vector_search::compute_norm(&rounded);
            self.embeddings.set(i, &rounded);
        }
    }
    /// The embedding as the cache will hold it: rounded to half precision
    /// when [`Self::half_precision`] is on, otherwise unchanged.
    fn stored_form(&self, mut embedding: Vec<f32>) -> Vec<f32> {
        if self.half_precision {
            for v in &mut embedding {
                *v = round_to_f16(*v);
            }
        }
        embedding
    }
    /// Kept for callers that used to have to re-flatten after a bulk load.
    /// The buffer is always flat now, so there is nothing to rebuild.
    #[deprecated(note = "embeddings are stored flat; this is a no-op")]
@@ -202,6 +243,7 @@ impl MemoryCache {
        tags: String,
    ) -> usize {
        let idx = self.chunks.len();
        let embedding = self.stored_form(embedding);
        let norm = vector_search::compute_norm(&embedding);
        self.chunks.push(chunk);
        self.embeddings.push(&embedding);
@@ -240,6 +282,7 @@ impl MemoryCache {
        session_id: String,
    ) {
        if idx < self.chunks.len() {
            let embedding = self.stored_form(embedding);
            let norm = vector_search::compute_norm(&embedding);
            self.chunks[idx] = chunk;
            self.embeddings.set(idx, &embedding);
@@ -439,4 +482,60 @@ mod tests {
            .reset_from(2, vec![vec![1.0, 2.0], vec![3.0, 4.0]]);
        assert_eq!(cache.embeddings.as_flat(), vec![1.0, 2.0, 3.0, 4.0]);
    }
    #[test]
    fn set_half_precision_rounds_existing_rows_and_their_norms() {
        // A store with float16 set whose checkpoint is still f32 on disk
        // loads full-precision rows; switching rounding on must bring them to
        // exactly what the next checkpoint will write.
        let mut cache = MemoryCache::new(3);
        cache.push(
            "a".into(),
            vec![0.1, 0.2, 0.3],
            "c".into(),
            0.0,
            "s".into(),
            "".into(),
        );
        cache.push(
            "b".into(),
            vec![0.5, 0.25, 1.0],
            "c".into(),
            0.0,
            "s".into(),
            "".into(),
        );
        let exact_norm = cache.norms[0];
        cache.set_half_precision(true);
        let row0: Vec<f32> = [0.1f32, 0.2, 0.3]
            .iter()
            .map(|&v| round_to_f16(v))
            .collect();
        assert_eq!(&cache.embeddings[0], row0.as_slice());
        assert_eq!(cache.norms[0], vector_search::compute_norm(&row0));
        assert_ne!(cache.norms[0], exact_norm);
        // Already representable: untouched.
        assert_eq!(&cache.embeddings[1], &[0.5, 0.25, 1.0]);
        // New rows are rounded as they arrive, and updates too.
        cache.push(
            "c".into(),
            vec![0.1, 0.0, 0.0],
            "c".into(),
            0.0,
            "s".into(),
            "".into(),
        );
        assert_eq!(cache.embeddings[2][0], round_to_f16(0.1));
        cache.update(
            2,
            "c".into(),
            vec![0.3, 0.0, 0.0],
            "c".into(),
            0.0,
            "s".into(),
        );
        assert_eq!(cache.embeddings[2][0], round_to_f16(0.3));
    }
 }
@@ -65,31 +65,34 @@ pub fn hybrid_search_fused(
 ) -> Vec<(usize, f32)> {
    // Get raw scores from both systems. Request all results so normalization
    // covers the full distribution.
-    // Use parallel search when rayon feature is enabled and vector count > 10K.
+    let vec_scores = exact_vector_scores(query_embedding, vectors, tombstones);
    let vec_scores = {
        #[cfg(feature = "parallel")]
        {
            if vectors.count() > 10_000 {
                vector_search::parallel_cosine_batch(
                    query_embedding,
                    vectors,
                    tombstones,
                    vectors.count(),
                )
            } else {
                vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
            }
        }
        #[cfg(not(feature = "parallel"))]
        {
            vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
        }
    };
    let kw_scores = bm25_index.scores(query_text);
    fuse(vec_scores, kw_scores, fusion, k)
 }
 /// Cosine similarity of `query_embedding` to every vector whose `skip` byte is
 /// 0 (a tombstone, or any other exclusion mask). Parallel above 10K vectors
 /// when the `parallel` feature is on.
 pub fn exact_vector_scores(
    query_embedding: &[f32],
    vectors: &(impl crate::vector_search::VectorSet + Sync + ?Sized),
    skip: &[u8],
 ) -> Vec<(usize, f32)> {
    #[cfg(feature = "parallel")]
    {
        if vectors.count() > 10_000 {
            return vector_search::parallel_cosine_batch(
                query_embedding,
                vectors,
                skip,
                vectors.count(),
            );
        }
    }
    vector_search::cosine_similarity_batch(query_embedding, vectors, skip)
 }
 /// Merge pre-computed vector-similarity and keyword scores into a single ranking.
 ///
 /// Both score sets are independently min-max normalized to [0, 1] and combined
@@ -63,6 +63,7 @@ use std::path::{Path, PathBuf};
 use cache::MemoryCache;
 #[cfg(feature = "hnsw")]
 use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};
 use clawhdf5_format::float16::round_to_f16;
 use ephemeral::{EphemeralConfig, EphemeralStore};
 // EphemeralEntry and EphemeralStats are part of the crate public API via
@@ -71,6 +72,7 @@ use ephemeral::{EphemeralConfig, EphemeralStore};
 pub use ephemeral::{EphemeralEntry, EphemeralStats};
 use knowledge::KnowledgeCache;
 use memory_strategy::{Exchange, MemoryStrategy, StrategyOutput};
 pub use search::SearchOptions;
 use session::SessionCache;
 // --- Error type ---
@@ -83,6 +85,9 @@ pub enum MemoryError {
    NotFound(String),
    /// Another `HDF5Memory` (in this or another process) has the store open.
    Locked(String),
    /// A record the store cannot hold as given, e.g. an embedding value
    /// outside the half-precision range of a `float16` store.
    InvalidEntry(String),
 }
 impl std::fmt::Display for MemoryError {
@@ -93,6 +98,7 @@ impl std::fmt::Display for MemoryError {
            MemoryError::Schema(e) => write!(f, "schema error: {e}"),
            MemoryError::NotFound(e) => write!(f, "not found: {e}"),
            MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
            MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
        }
    }
 }
@@ -124,6 +130,19 @@ pub struct MemoryConfig {
    pub embedding_dim: usize,
    pub chunk_size: usize,
    pub overlap: usize,
    /// Store embeddings as IEEE half precision (numpy `float16`): half the
    /// bytes of the embeddings dataset on disk. Every embedding is rounded to
    /// the nearest half as it enters the store, in memory as well as on disk,
    /// so search results are the same before and after a reopen. Values must
    /// lie within ±65504; a save outside that is `MemoryError::InvalidEntry`.
    /// Fixed when the store is created (persisted in `/meta`).
    ///
    /// **On by default for new stores**: on the full LongMemEval haystack with
    /// real MiniLM embeddings every retrieval metric matched `f32`, and at
    /// 100K records the file is 48% smaller (`BENCHMARKS.md`). Existing
    /// stores keep the setting they were created with. Set it to `false` for
    /// full-precision embeddings, e.g. for unnormalised vectors that may
    /// exceed the half-precision range.
    pub float16: bool,
    pub compression: bool,
    pub compression_level: u32,
@@ -179,7 +198,7 @@ impl MemoryConfig {
            embedding_dim,
            chunk_size: 512,
            overlap: 50,
-            float16: false,
+            float16: true,
            compression: false,
            compression_level: 0,
            compact_threshold: 0.3,
@@ -317,7 +336,8 @@ impl HDF5Memory {
    /// Create a new HDF5 memory file with the given configuration.
    pub fn create(config: MemoryConfig) -> Result<Self> {
        let lock = store_lock::StoreLock::acquire(&config.path)?;
-        let cache = MemoryCache::new(config.embedding_dim);
+        let mut cache = MemoryCache::new(config.embedding_dim);
        cache.set_half_precision(config.float16);
        let sessions = SessionCache::new();
        let knowledge = KnowledgeCache::new();
@@ -1070,7 +1090,29 @@ impl HDF5Memory {
    /// Upsert: if an active entry with the same tags (key) exists, update it in-place.
    /// Otherwise append a new entry. Use this for key-based memory stores where
    /// the same key should not create duplicates.
    /// A `float16` store holds embeddings as IEEE half precision, which has no
    /// finite value beyond ±65504. Refuse such an embedding rather than
    /// silently store infinity. (Values that are already infinite or NaN are
    /// stored as they are, as in an `f32` store.)
    fn check_embedding(&self, embedding: &[f32]) -> Result<()> {
        if !self.config.float16 {
            return Ok(());
        }
        let overflow = embedding
            .iter()
            .enumerate()
            .find(|&(_, &v)| v.is_finite() && round_to_f16(v).is_infinite());
        match overflow {
            None => Ok(()),
            Some((i, v)) => Err(MemoryError::InvalidEntry(format!(
                "embedding[{i}] = {v} is outside the half-precision range (±65504) \
                 of this float16 store"
            ))),
        }
    }
    pub fn save_or_update(&mut self, entry: MemoryEntry) -> Result<usize> {
        self.check_embedding(&entry.embedding)?;
        if let Some(existing_idx) = self.cache.find_by_tags(&entry.tags) {
            if let Some(ref mut w) = self.wal {
                let wal_entry = wal::WalEntry {
@@ -1126,6 +1168,7 @@ impl HDF5Memory {
 impl AgentMemory for HDF5Memory {
    fn save(&mut self, entry: MemoryEntry) -> Result<usize> {
        self.check_embedding(&entry.embedding)?;
        if let Some(ref mut w) = self.wal {
            let wal_entry = wal::WalEntry {
                entry_type: wal::WalEntryType::Save,
@@ -1167,6 +1210,10 @@ impl AgentMemory for HDF5Memory {
    }
    fn save_batch(&mut self, entries: Vec<MemoryEntry>) -> Result<Vec<usize>> {
        // All or nothing: check every entry before storing any.
        for entry in &entries {
            self.check_embedding(&entry.embedding)?;
        }
        let mut indices = Vec::with_capacity(entries.len());
        for entry in entries {
            let idx = self.cache.push(
@@ -1327,6 +1374,9 @@ impl HDF5Memory {
        })?;
        let view = memory_strategy::CacheStoreView::new(&self.cache, &self.knowledge);
        let output = strat.evaluate(&exchange, &view);
        for e in &output.entries {
            self.check_embedding(&e.embedding)?;
        }
        for e in &output.entries {
            self.cache.push(
                e.chunk.clone(),
@@ -1411,6 +1461,16 @@ impl HDF5Memory {
        let mut promoted = 0;
        for key in candidates {
            // Check before taking, so a rejected entry stays in the ephemeral
            // tier rather than being lost.
            if let Some(emb) = self
                .ephemeral
                .as_ref()
                .and_then(|s| s.get_entry(&key))
                .and_then(|e| e.embedding.as_deref())
            {
                self.check_embedding(emb)?;
            }
            let entry = match self
                .ephemeral
                .as_mut()
@@ -13,9 +13,8 @@ use std::path::{Path, PathBuf};
 use std::time::{SystemTime, UNIX_EPOCH};
 use crate::{
-    AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry,
+    AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions,
-    confidence::{ConfidenceConfig, ScoredResult, reject_low_confidence},
+    confidence::ConfidenceConfig, reranker::ReRankConfig,
    reranker::{ReRankConfig, RerankInput, rerank},
 };
 // ─────────────────────────────────────────────────────────────────────────────
@@ -524,71 +523,27 @@ impl ClawhdfBackend {
 impl MemoryBackend for ClawhdfBackend {
    /// Search using hybrid vector + BM25 retrieval, then re-rank and
-    /// confidence-filter.
+    /// confidence-filter — [`HDF5Memory::search`] with both stages on.
    fn search(
        &mut self,
        query_text: &str,
        query_embedding: &[f32],
        k: usize,
    ) -> Vec<MemorySearchResult> {
-        // 1. Hybrid retrieval (vector + BM25, fused by score).
+        let options = SearchOptions::new(k)
-        let candidates = k.saturating_mul(3).max(10);
+            .with_rerank(self.rerank_config)
-        let raw = self.memory.hybrid_search_with(
+            .with_confidence(self.confidence_config.clone())
-            query_embedding,
+            .at_time(Self::now_secs());
-            query_text,
+        self.memory
-            crate::hybrid::DEFAULT_FUSION,
+            .search(query_embedding, query_text, &options)
            candidates,
        );
        if raw.is_empty() {
            return Vec::new();
        }
        let now = Self::now_secs();
        // 2. Re-rank using temporal recency, source authority, Hebbian weight.
        let rerank_inputs: Vec<RerankInput> = raw
            .iter()
            .map(|r| RerankInput {
                index: r.index,
                timestamp: r.timestamp,
                source_channel: r.source_channel.clone(),
                raw_activation: r.activation,
                relevance: r.score,
            })
            .collect();
        let reranked = rerank(&rerank_inputs, &self.rerank_config, now);
        // 3. Confidence rejection.
        let scored: Vec<ScoredResult> = reranked
            .iter()
            .map(|r| ScoredResult {
                index: r.index,
                score: r.combined_score,
            })
            .collect();
        let confident = reject_low_confidence(&scored, &self.confidence_config);
        // 4. Map back to MemorySearchResult; preserve raw text via index lookup.
        let raw_by_idx: HashMap<usize, &crate::SearchResult> =
            raw.iter().map(|r| (r.index, r)).collect();
        confident
            .into_iter()
-            .take(k)
+            .map(|r| MemorySearchResult {
-            .filter_map(|sr| {
+                text: r.chunk,
-                let r = raw_by_idx.get(&sr.index)?;
+                score: r.score,
-                let path = r.source_channel.clone();
+                path: r.source_channel.clone(),
                Some(MemorySearchResult {
                    text: r.chunk.clone(),
                    score: sr.score,
                    path: path.clone(),
                line_range: None,
                timestamp: Some(r.timestamp),
-                    source: path,
+                source: r.source_channel,
                })
            })
            .collect()
    }
@@ -159,20 +159,27 @@ fn build_memory_group(
    // chunks: fixed-length string array
    write_string_dataset(&mut group, "chunks", &cache.chunks);
-    // embeddings: f32 [N x D]
+    // embeddings: [N x D], f32 — or IEEE half precision for a `float16`
    // store. The cache already holds half-rounded values then, so this
    // conversion is exact and a reopened store sees the same numbers.
    let n = cache.embeddings.len() as u64;
    let d = cache.embedding_dim as u64;
    let flat = cache.flat_embeddings();
    {
-        let ds = group
+        let ds = group.create_dataset("embeddings");
-            .create_dataset("embeddings")
+        let elem_bytes: u64 = if config.float16 {
-            .with_f32_data(flat)
+            ds.with_f16_data(flat);
-            .with_shape(&[n, d]);
+            2
        } else {
            ds.with_f32_data(flat);
            4
        };
        ds.with_shape(&[n, d]);
        // Chunk size tuning: target ~256KB per chunk for optimal I/O
        if n > 0 && d > 0 {
            let target_chunk_bytes: u64 = 256 * 1024;
-            let rows_per_chunk = (target_chunk_bytes / (d * 4)).max(1).min(n);
+            let rows_per_chunk = (target_chunk_bytes / (d * elem_bytes)).max(1).min(n);
            ds.with_chunks(&[rows_per_chunk, d]);
            // Compression. Shuffle is applied automatically (auto-shuffle
@@ -513,7 +520,16 @@ pub fn validate_and_load(
    };
    // Load /memory group
-    let memory_cache = load_memory_group(file, embedding_dim)?;
+    let mut memory_cache = load_memory_group(file, embedding_dim)?;
    // A float16 store's cache holds half-rounded embeddings. Embeddings read
    // from an f16 dataset already are; a float16 store whose last checkpoint
    // predates half-precision storage is still f32 on disk and is rounded
    // here.
    if config.float16 && embeddings_are_f16(file) {
        memory_cache.half_precision = true;
    } else {
        memory_cache.set_half_precision(config.float16);
    }
    // Load /sessions group
    let session_cache = load_sessions_group(file)?;
@@ -756,6 +772,13 @@ fn read_string_dataset_from_group(
        .map_err(|e| MemoryError::Hdf5(format!("cannot read strings from {name}: {e}")))
 }
 /// Whether `/memory/embeddings` is stored as IEEE half precision.
 fn embeddings_are_f16(file: &clawhdf5::File) -> bool {
    file.dataset("memory/embeddings")
        .and_then(|ds| ds.dtype())
        .is_ok_and(|dt| matches!(dt, clawhdf5::DType::Other(ref s) if s == "float16"))
 }
 fn read_f32_dataset(group: &clawhdf5::Group<'_>, name: &str) -> Result<Vec<f32>, MemoryError> {
    let ds = group
        .dataset(name)
@@ -2,18 +2,107 @@
 use std::path::Path;
 use std::collections::HashSet;
 use crate::bm25;
 use crate::confidence::{ConfidenceConfig, ScoredResult, reject_low_confidence};
 use crate::hybrid;
 use crate::reranker::{ReRankConfig, RerankInput, rerank};
 use crate::{HDF5Memory, MAX_ACTIVATION_WEIGHT, MemoryError, Result, SearchResult};
 /// Options for [`HDF5Memory::search`].
 ///
 /// [`SearchOptions::new`] is plain hybrid search with the tuned default
 /// fusion — the same as `hybrid_search_with(.., hybrid::DEFAULT_FUSION, k)`.
 /// Every stage beyond that is opt-in.
 #[derive(Debug, Clone)]
 pub struct SearchOptions {
    /// Number of results to return.
    pub k: usize,
    /// How the vector and keyword stages are combined.
    pub fusion: hybrid::Fusion,
    /// Only consider records whose `source_channel` is one of these. The
    /// filter applies *before* ranking, so a filtered search still returns up
    /// to `k` results and scores are normalised over the records it can
    /// return. `None` searches everything; an empty list matches nothing.
    pub source_channels: Option<Vec<String>>,
    /// Re-rank a candidate pool by retrieval relevance, recency, source
    /// authority and activation — the pipeline the OpenClaw backend runs.
    pub rerank: Option<ReRankConfig>,
    /// Candidates retrieved for re-ranking; 0 means `max(3k, 10)`.
    pub rerank_pool: usize,
    /// Drop low-confidence results (after re-ranking, when that is on).
    pub confidence: Option<ConfidenceConfig>,
    /// The time recency is measured from, in seconds since the epoch.
    /// `None` uses the system clock.
    pub now: Option<f64>,
 }
 impl SearchOptions {
    pub fn new(k: usize) -> Self {
        Self {
            k,
            fusion: hybrid::DEFAULT_FUSION,
            source_channels: None,
            rerank: None,
            rerank_pool: 0,
            confidence: None,
            now: None,
        }
    }
    pub fn with_fusion(mut self, fusion: hybrid::Fusion) -> Self {
        self.fusion = fusion;
        self
    }
    /// Search only records from these source channels.
    pub fn with_sources<S: Into<String>>(mut self, channels: impl IntoIterator<Item = S>) -> Self {
        self.source_channels = Some(channels.into_iter().map(Into::into).collect());
        self
    }
    pub fn with_rerank(mut self, config: ReRankConfig) -> Self {
        self.rerank = Some(config);
        self
    }
    pub fn with_confidence(mut self, config: ConfidenceConfig) -> Self {
        self.confidence = Some(config);
        self
    }
    /// Measure recency from `now` (seconds since the epoch) instead of the
    /// system clock — for reproducible results and tests.
    pub fn at_time(mut self, now: f64) -> Self {
        self.now = Some(now);
        self
    }
 }
 impl Default for SearchOptions {
    fn default() -> Self {
        Self::new(10)
    }
 }
 impl HDF5Memory {
-    /// Vector + keyword scoring stage of [`HDF5Memory::hybrid_search`].
+    /// Vector + keyword scoring stage of [`HDF5Memory::search`].
    ///
    /// Without the `hnsw` feature this is a full linear cosine scan (the exact
    /// previous behaviour, also used as the correctness oracle in tests). With
    /// `hnsw` enabled and an index available, the vector candidates come from an
    /// approximate-nearest-neighbour search over an over-fetched pool, then merge
    /// with BM25 via the shared [`hybrid::merge_vector_keyword`].
    ///
    /// `exclude`, when given, marks records that must not be returned (1 =
    /// excluded; it covers tombstones too). The index is over-fetched in
    /// proportion to how much the mask removes. Surfacing `pool` candidates
    /// costs the index roughly `pool × M` distance evaluations, while an exact
    /// scan of the allowed records costs one each — so whenever that scan is
    /// the cheaper of the two it is used instead, and it is also the fallback
    /// if the pool comes back with too few allowed hits (the allowed records
    /// sit away from the query). A filtered search never comes back short.
    #[cfg(feature = "hnsw")]
    fn vector_keyword_search(
        &mut self,
@@ -22,16 +111,30 @@ impl HDF5Memory {
        bm25: &bm25::BM25Index,
        fusion: hybrid::Fusion,
        k: usize,
        exclude: Option<&[u8]>,
    ) -> Vec<(usize, f32)> {
        self.ensure_hnsw_fresh();
        let n = self.cache.len();
        // Over-fetch so the merge sees a useful vector pool. `ef` is
        // configurable, but the pool the fusion stage sees is not tied to it:
        // a caller lowering `ef` for speed should not silently narrow what
        // fusion has to work with.
        let mut pool = (k * 8).max(64);
        let mut allowed = n;
        if let Some(ex) = exclude {
            allowed = ex.iter().filter(|&&e| e == 0).count();
            if allowed == 0 {
                return Vec::new();
            }
            // Expect `pool` allowed hits if the filter is independent of the
            // query's neighbourhood.
            pool = pool.saturating_mul(n).div_ceil(allowed);
            if allowed <= pool.saturating_mul(self.hnsw_m()) {
                return self.exact_masked_search(query_embedding, query_text, bm25, fusion, k, ex);
            }
        }
        match self.hnsw.as_ref() {
            Some(index) if !index.is_empty() && index.dimension() == query_embedding.len() => {
                // Over-fetch so the merge sees a useful vector pool; cosine
                // distance from the index converts back to similarity (1 - d).
                // `ef` is configurable, but the pool the fusion stage sees is
                // not tied to it: a caller lowering `ef` for speed should not
                // silently narrow what fusion has to work with.
                let pool = (k * 8).max(64);
                let ef = self.hnsw_ef_search(k).max(pool);
                let candidates = index.search(query_embedding, pool, ef);
                // A quantised index returns approximate distances, and no
@@ -42,6 +145,7 @@ impl HDF5Memory {
                let exact = index.storage() == clawhdf5_ann::Storage::Int8;
                let vec_scores: Vec<(usize, f32)> = candidates
                    .into_iter()
                    .filter(|(id, _)| exclude.is_none_or(|ex| ex[*id] == 0))
                    .map(|(id, dist)| {
                        let score = if exact {
                            crate::vector_search::cosine_similarity(
@@ -56,10 +160,54 @@ impl HDF5Memory {
                    .collect();
                // Fusion normalises over every keyword match, so it needs all
                // the scores — but not ranked.
-                let kw_scores = bm25.scores(query_text);
+                let mut kw_scores = bm25.scores(query_text);
                if let Some(ex) = exclude {
                    if vec_scores.len() < k.min(allowed) {
                        // The allowed records are not where the index looked.
                        return self.exact_masked_search(
                            query_embedding,
                            query_text,
                            bm25,
                            fusion,
                            k,
                            ex,
                        );
                    }
                    kw_scores.retain(|(id, _)| ex[*id] == 0);
                }
                hybrid::fuse(vec_scores, kw_scores, fusion, k)
            }
-            _ => hybrid::hybrid_search_fused(
+            _ => match exclude {
                Some(ex) => {
                    self.exact_masked_search(query_embedding, query_text, bm25, fusion, k, ex)
                }
                None => hybrid::hybrid_search_fused(
                    query_embedding,
                    query_text,
                    &self.cache.embeddings,
                    &self.cache.chunks,
                    &self.cache.tombstones,
                    bm25,
                    fusion,
                    k,
                ),
            },
        }
    }
    #[cfg(not(feature = "hnsw"))]
    fn vector_keyword_search(
        &mut self,
        query_embedding: &[f32],
        query_text: &str,
        bm25: &bm25::BM25Index,
        fusion: hybrid::Fusion,
        k: usize,
        exclude: Option<&[u8]>,
    ) -> Vec<(usize, f32)> {
        match exclude {
            Some(ex) => self.exact_masked_search(query_embedding, query_text, bm25, fusion, k, ex),
            None => hybrid::hybrid_search_fused(
                query_embedding,
                query_text,
                &self.cache.embeddings,
@@ -72,25 +220,33 @@ impl HDF5Memory {
        }
    }
-    #[cfg(not(feature = "hnsw"))]
+    /// Exact hybrid search over the records `exclude` leaves (0 = allowed).
-    fn vector_keyword_search(
+    fn exact_masked_search(
-        &mut self,
+        &self,
        query_embedding: &[f32],
        query_text: &str,
        bm25: &bm25::BM25Index,
        fusion: hybrid::Fusion,
        k: usize,
        exclude: &[u8],
    ) -> Vec<(usize, f32)> {
-        hybrid::hybrid_search_fused(
+        let vec_scores =
-            query_embedding,
+            hybrid::exact_vector_scores(query_embedding, &self.cache.embeddings, exclude);
-            query_text,
+        let mut kw_scores = bm25.scores(query_text);
-            &self.cache.embeddings,
+        kw_scores.retain(|(id, _)| exclude.get(*id) == Some(&0));
-            &self.cache.chunks,
+        hybrid::fuse(vec_scores, kw_scores, fusion, k)
-            &self.cache.tombstones,
+    }
-            bm25,
+
-            fusion,
+    /// The exclusion mask for a source-channel filter: 1 for a tombstoned
-            k,
+    /// record or one from a channel not in `channels`.
-        )
+    fn source_mask(&self, channels: &[String]) -> Vec<u8> {
        let allowed: HashSet<&str> = channels.iter().map(String::as_str).collect();
        self.cache
            .source_channels
            .iter()
            .zip(&self.cache.tombstones)
            .map(|(ch, &t)| u8::from(t != 0 || !allowed.contains(ch.as_str())))
            .collect()
    }
    /// Perform hybrid search combining cosine vector similarity and BM25 keyword search.
@@ -125,12 +281,53 @@ impl HDF5Memory {
        fusion: hybrid::Fusion,
        k: usize,
    ) -> Vec<SearchResult> {
        self.search(
            query_embedding,
            query_text,
            &SearchOptions::new(k).with_fusion(fusion),
        )
    }
    /// Hybrid search with optional source filtering, re-ranking and
    /// confidence rejection — see [`SearchOptions`].
    ///
    /// Stages, in order: vector + keyword retrieval over the records the
    /// source filter allows; fusion; scaling by Hebbian activation; re-ranking
    /// (if on) of a `rerank_pool` of candidates; confidence rejection (if on);
    /// the top `k`. The records returned with a positive score get their
    /// Hebbian boost.
    pub fn search(
        &mut self,
        query_embedding: &[f32],
        query_text: &str,
        options: &SearchOptions,
    ) -> Vec<SearchResult> {
        let k = options.k;
        let fetch = match options.rerank {
            Some(_) if options.rerank_pool > 0 => options.rerank_pool.max(k),
            Some(_) => k.saturating_mul(3).max(10),
            None => k,
        };
        let exclude = options
            .source_channels
            .as_deref()
            .map(|channels| self.source_mask(channels));
        // The keyword index lives for the life of the store and is updated
        // incrementally. Take it out for the duration of the call so the
        // vector stage can borrow `self` mutably, then put it back.
        self.ensure_bm25_fresh();
        let bm25 = self.bm25.take().expect("ensure_bm25_fresh leaves an index");
-        let scored = self.vector_keyword_search(query_embedding, query_text, &bm25, fusion, k);
+        let scored = self.vector_keyword_search(
            query_embedding,
            query_text,
            &bm25,
            options.fusion,
            fetch,
            exclude.as_deref(),
        );
        self.bm25 = Some(bm25);
        let mut results: Vec<SearchResult> = scored
            .into_iter()
            .map(|(idx, score)| {
@@ -154,6 +351,25 @@ impl HDF5Memory {
                .then(a.index.cmp(&b.index))
        });
        if let Some(config) = &options.rerank {
            results = Self::rerank_results(results, config, options.now);
        }
        if let Some(config) = &options.confidence {
            let scored: Vec<ScoredResult> = results
                .iter()
                .map(|r| ScoredResult {
                    index: r.index,
                    score: r.score,
                })
                .collect();
            let keep: HashSet<usize> = reject_low_confidence(&scored, config)
                .into_iter()
                .map(|r| r.index)
                .collect();
            results.retain(|r| keep.contains(&r.index));
        }
        results.truncate(k);
        // Only reinforce records that actually matched. When fewer than `k`
        // records are relevant, the rest of the list is zero-score filler;
        // boosting it would teach the store that arbitrary records are
@@ -164,11 +380,45 @@ impl HDF5Memory {
            .map(|r| r.index)
            .collect();
        self.apply_hebbian_boost(&hit_indices);
        self.bm25 = Some(bm25);
        results
    }
    /// Reorder by the re-ranker's combined score, which also becomes each
    /// result's `score`.
    fn rerank_results(
        results: Vec<SearchResult>,
        config: &ReRankConfig,
        now: Option<f64>,
    ) -> Vec<SearchResult> {
        let now = now.unwrap_or_else(|| {
            std::time::SystemTime::now()
                .duration_since(std::time::UNIX_EPOCH)
                .map(|d| d.as_secs_f64())
                .unwrap_or(0.0)
        });
        let inputs: Vec<RerankInput> = results
            .iter()
            .map(|r| RerankInput {
                index: r.index,
                timestamp: r.timestamp,
                source_channel: r.source_channel.clone(),
                raw_activation: r.activation,
                relevance: r.score,
            })
            .collect();
        let mut by_index: std::collections::HashMap<usize, SearchResult> =
            results.into_iter().map(|r| (r.index, r)).collect();
        rerank(&inputs, config, now)
            .into_iter()
            .filter_map(|rr| {
                let mut r = by_index.remove(&rr.index)?;
                r.score = rr.combined_score;
                Some(r)
            })
            .collect()
    }
    /// Reinforce the records a query returned. The new weights are persisted by
    /// the next checkpoint (any write that flushes, `flush_wal`, or drop) — not
    /// by rewriting the whole store inside the query, which is what made
@@ -0,0 +1,260 @@
 //! `MemoryConfig::float16`: embeddings stored as IEEE half precision.
 //!
 //! The setting used to be recorded in `/meta` and otherwise ignored — the
 //! embeddings dataset was always `f32`. These tests pin what it now does: the
 //! dataset is `float16`, the in-memory cache holds exactly the values the file
 //! holds (so search results survive a reopen bit for bit), and a value half
 //! precision cannot represent is refused rather than stored as infinity.
 use std::path::{Path, PathBuf};
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError};
 use clawhdf5_format::float16::round_to_f16;
 use tempfile::TempDir;
 const DIM: usize = 64;
 /// Deterministic, embedding-like unit vectors.
 fn embedding(seed: u64) -> Vec<f32> {
    let mut x = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
    let v: Vec<f32> = (0..DIM)
        .map(|_| {
            x ^= x << 13;
            x ^= x >> 7;
            x ^= x << 17;
            (x >> 40) as f32 / (1u64 << 24) as f32 - 0.5
        })
        .collect();
    let norm = v.iter().map(|a| a * a).sum::<f32>().sqrt();
    v.iter().map(|a| a / norm).collect()
 }
 fn entry(i: u64) -> MemoryEntry {
    MemoryEntry {
        chunk: format!("memory number {i} about topic {}", i % 7),
        embedding: embedding(i),
        source_channel: "test".into(),
        timestamp: i as f64,
        session_id: "s".into(),
        tags: format!("t{i}"),
    }
 }
 fn config(dir: &TempDir, name: &str, float16: bool) -> MemoryConfig {
    let mut c = MemoryConfig::new(dir.path().join(name), "agent", DIM);
    c.float16 = float16;
    c
 }
 fn embeddings_dtype_and_values(path: &Path) -> (String, Vec<f32>) {
    let file = clawhdf5::File::open(path).unwrap();
    let ds = file.dataset("memory/embeddings").unwrap();
    (format!("{:?}", ds.dtype().unwrap()), ds.read_f32().unwrap())
 }
 fn search_bits(m: &mut HDF5Memory, q: u64) -> Vec<(usize, u32)> {
    m.hybrid_search(&embedding(q), "memory topic 3", 0.4, 0.6, 10)
        .iter()
        .map(|r| (r.index, r.score.to_bits()))
        .collect()
 }
 #[test]
 fn float16_store_writes_half_precision_and_reopens_identically() {
    let dir = TempDir::new().unwrap();
    // Two identical stores. Search is not read-only (it boosts the Hebbian
    // activation of what it returns, and checkpoints persist that), so each
    // is queried exactly once: one live, one after a checkpoint and reopen.
    let live_cfg = config(&dir, "live.h5", true);
    let cfg = config(&dir, "f16.h5", true);
    let path: PathBuf = cfg.path.clone();
    let mut live = HDF5Memory::create(live_cfg).unwrap();
    live.save_batch((0..200).map(entry).collect()).unwrap();
    let mut m = HDF5Memory::create(cfg).unwrap();
    m.save_batch((0..200).map(entry).collect()).unwrap();
    drop(m);
    // On disk: a genuine float16 dataset holding the rounded inputs.
    let (dtype, values) = embeddings_dtype_and_values(&path);
    assert_eq!(dtype, "Other(\"float16\")");
    let expected: Vec<u32> = (0..200)
        .flat_map(|i| embedding(i).into_iter().map(|v| round_to_f16(v).to_bits()))
        .collect();
    let got: Vec<u32> = values.iter().map(|v| v.to_bits()).collect();
    assert_eq!(got, expected);
    // Reopened, the store answers exactly as the live one does: the cache
    // held the half-rounded values before the checkpoint.
    let mut reopened = HDF5Memory::open(&path).unwrap();
    for q in 0..5 {
        assert_eq!(
            search_bits(&mut live, 1000 + q),
            search_bits(&mut reopened, 1000 + q),
            "query {q}"
        );
    }
 }
 #[test]
 fn float16_halves_the_embeddings_on_disk() {
    let dir = TempDir::new().unwrap();
    let mut sizes = Vec::new();
    for float16 in [false, true] {
        let cfg = config(&dir, &format!("s{float16}.h5"), float16);
        let path = cfg.path.clone();
        let mut m = HDF5Memory::create(cfg).unwrap();
        m.save_batch((0..2000).map(entry).collect()).unwrap();
        drop(m);
        sizes.push(std::fs::metadata(&path).unwrap().len());
    }
    let embedding_bytes_f32 = (2000 * DIM * 4) as u64;
    let saved = sizes[0] - sizes[1];
    // Half of the f32 embeddings, give or take metadata and alignment.
    assert!(
        saved.abs_diff(embedding_bytes_f32 / 2) < 16 * 1024,
        "f32 {} B, f16 {} B, saved {saved} B, expected ~{} B",
        sizes[0],
        sizes[1],
        embedding_bytes_f32 / 2
    );
 }
 #[test]
 fn f32_store_is_unchanged() {
    let dir = TempDir::new().unwrap();
    let cfg = config(&dir, "f32.h5", false);
    let path = cfg.path.clone();
    let mut m = HDF5Memory::create(cfg).unwrap();
    m.save_batch((0..50).map(entry).collect()).unwrap();
    drop(m);
    let (dtype, values) = embeddings_dtype_and_values(&path);
    assert_eq!(dtype, "F32");
    let expected: Vec<f32> = (0..50).flat_map(embedding).collect();
    assert_eq!(values, expected);
 }
 #[test]
 fn out_of_range_values_are_refused_not_stored_as_infinity() {
    let dir = TempDir::new().unwrap();
    let mut cfg = config(&dir, "range.h5", true);
    cfg.wal_enabled = true;
    let path = cfg.path.clone();
    let mut m = HDF5Memory::create(cfg).unwrap();
    m.save(entry(1)).unwrap();
    let mut bad = entry(2);
    bad.embedding[5] = 70_000.0;
    match m.save(bad.clone()) {
        Err(MemoryError::InvalidEntry(msg)) => assert!(msg.contains("embedding[5]"), "{msg}"),
        other => panic!("expected InvalidEntry, got {other:?}"),
    }
    assert!(matches!(
        m.save_or_update(bad.clone()),
        Err(MemoryError::InvalidEntry(_))
    ));
    // A batch is all or nothing.
    assert!(matches!(
        m.save_batch(vec![entry(3), bad.clone(), entry(4)]),
        Err(MemoryError::InvalidEntry(_))
    ));
    assert_eq!(m.count(), 1);
    // The largest finite half, and values that round down to it, are fine.
    let mut edge = entry(5);
    edge.embedding[0] = 65504.0;
    edge.embedding[1] = -65519.0;
    m.save(edge).unwrap();
    assert_eq!(m.count(), 2);
    drop(m);
    // Nothing rejected reached the WAL or the file.
    let m = HDF5Memory::open(&path).unwrap();
    assert_eq!(m.count(), 2);
    // An f32 store takes the same value as it always did.
    let mut m32 = HDF5Memory::create(config(&dir, "range32.h5", false)).unwrap();
    m32.save(bad).unwrap();
 }
 #[test]
 fn wal_replay_rounds_like_a_live_save() {
    let dir = TempDir::new().unwrap();
    let mut cfg = config(&dir, "wal.h5", true);
    cfg.wal_enabled = true;
    cfg.wal_max_entries = 10_000; // keep everything in the WAL
    let path = cfg.path.clone();
    let mut m = HDF5Memory::create(cfg).unwrap();
    for i in 0..30 {
        m.save(entry(i)).unwrap();
    }
    let live = search_bits(&mut m, 77);
    // Crash image: the .h5 is still the empty checkpoint; everything is in
    // the WAL, which holds the caller's f32 values.
    let crash = TempDir::new().unwrap();
    let image = crash.path().join("image.h5");
    std::fs::copy(&path, &image).unwrap();
    std::fs::copy(
        path.with_extension("h5.wal"),
        image.with_extension("h5.wal"),
    )
    .unwrap();
    drop(m);
    let mut recovered = HDF5Memory::open(&image).unwrap();
    assert_eq!(recovered.count(), 30);
    assert_eq!(search_bits(&mut recovered, 77), live);
 }
 #[test]
 fn new_stores_default_to_float16() {
    let dir = TempDir::new().unwrap();
    let path = dir.path().join("default.h5");
    let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", DIM)).unwrap();
    assert!(m.config().float16);
    m.save_batch((0..10).map(entry).collect()).unwrap();
    drop(m);
    assert_eq!(embeddings_dtype_and_values(&path).0, "Other(\"float16\")");
    assert!(HDF5Memory::open(&path).unwrap().config().float16);
 }
 #[test]
 fn an_existing_f32_store_stays_f32() {
    // Written by the v2.5.0 CLI, with `float16 = 0` in /meta (every agent
    // store has recorded it). Flipping the default for new stores must not
    // reach back and round an existing store's embeddings.
    let dir = TempDir::new().unwrap();
    let path = dir.path().join("legacy.h5");
    std::fs::copy(
        concat!(
            env!("CARGO_MANIFEST_DIR"),
            "/tests/fixtures/store_v2_5_0.h5"
        ),
        &path,
    )
    .unwrap();
    let before = embeddings_dtype_and_values(&path);
    assert_eq!(before.0, "F32");
    let mut m = HDF5Memory::open(&path).unwrap();
    assert!(!m.config().float16, "an old store must reopen as f32");
    let dim = m.config().embedding_dim;
    let odd: Vec<f32> = (0..dim).map(|i| 0.1 + i as f32 * 1e-4).collect();
    m.save_batch(vec![MemoryEntry {
        chunk: "added after the upgrade".into(),
        embedding: odd.clone(),
        source_channel: "test".into(),
        timestamp: 1.0,
        session_id: "s".into(),
        tags: String::new(),
    }])
    .unwrap();
    drop(m);
    // Checkpointed: still f32, the old rows untouched and the new one exact.
    let (dtype, values) = embeddings_dtype_and_values(&path);
    assert_eq!(dtype, "F32");
    assert_eq!(&values[..before.1.len()], before.1.as_slice());
    assert_eq!(&values[before.1.len()..], odd.as_slice());
 }
@@ -0,0 +1,94 @@
 //! An agent store is a standard HDF5 file: h5py can open it and read every
 //! dataset.
 //!
 //! It could not: the float datatype's sign-bit position was hard-coded for
 //! f64, so every f32 dataset (embeddings, norms, activation weights) made
 //! libhdf5 refuse the file with "sign bit position out of bounds".
 use std::process::Command;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
 fn python() -> String {
    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
 }
 fn h5py_available() -> bool {
    Command::new(python())
        .args(["-c", "import h5py"])
        .output()
        .map(|o| o.status.success())
        .unwrap_or(false)
 }
 #[test]
 fn h5py_reads_every_dataset_of_an_agent_store() {
    if !h5py_available() {
        assert!(
            std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
            "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
        );
        eprintln!("SKIP: python3 with h5py not available");
        return;
    }
    let dir = tempfile::tempdir().unwrap();
    for float16 in [false, true] {
        let path = dir.path().join(format!("store_{float16}.h5"));
        let mut cfg = MemoryConfig::new(path.clone(), "agent", 8);
        cfg.float16 = float16;
        let mut m = HDF5Memory::create(cfg).unwrap();
        // save_batch checkpoints, so the records are in the .h5, not the WAL.
        m.save_batch(
            (0..20)
                .map(|i| MemoryEntry {
                    chunk: format!("memory {i}"),
                    embedding: (0..8).map(|j| ((i * 8 + j) as f32).sin()).collect(),
                    source_channel: "test".into(),
                    timestamp: i as f64,
                    session_id: "s".into(),
                    tags: String::new(),
                })
                .collect(),
        )
        .unwrap();
        drop(m);
        // Exact expected values, as bits: numpy's sin need not match Rust's
        // to the last place.
        let bits = (0..160)
            .map(|k| (k as f32).sin().to_bits().to_string())
            .collect::<Vec<_>>()
            .join(",");
        let script = format!(
            r#"
 import h5py, numpy as np
 want = np.float16 if {py_bool} else np.float32
 with h5py.File("{path}", "r") as f:
    names = []
    f.visititems(lambda n, o: names.append(n) if isinstance(o, h5py.Dataset) else None)
    for n in names:
        f[n][()]  # every dataset must decode
    e = f["memory/embeddings"]
    assert e.dtype == want, e.dtype
    assert e.shape == (20, 8), e.shape
    ref = np.array([{bits}], dtype=np.uint32).view(np.float32).astype(want).reshape(20, 8)
    assert (e[()] == ref).all()
    assert f["memory/norms"].dtype == np.float32
 print(len(names))
 "#,
            py_bool = if float16 { "True" } else { "False" },
            path = path.display()
        );
        let out = Command::new(python())
            .args(["-c", &script])
            .output()
            .unwrap();
        assert!(
            out.status.success(),
            "float16={float16}: {}",
            String::from_utf8_lossy(&out.stderr)
        );
        let n: usize = String::from_utf8_lossy(&out.stdout).trim().parse().unwrap();
        assert!(n >= 10, "only {n} datasets");
    }
 }
@@ -0,0 +1,344 @@
 //! `HDF5Memory::search` with `SearchOptions`: source filtering, re-ranking and
 //! confidence rejection in the store's own search path.
 use std::collections::HashSet;
 use clawhdf5_agent::confidence::ConfidenceConfig;
 use clawhdf5_agent::reranker::ReRankConfig;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions, hybrid};
 use tempfile::TempDir;
 const DIM: usize = 32;
 const N: usize = 3000;
 const CLUSTERS: usize = 20;
 struct Rng(u64);
 impl Rng {
    fn next(&mut self) -> u64 {
        self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
        let mut z = self.0;
        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
        z ^ (z >> 31)
    }
    fn unit(&mut self) -> f32 {
        (self.next() >> 40) as f32 / (1u64 << 24) as f32 - 0.5
    }
 }
 fn normalize(v: &mut [f32]) {
    let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();
    v.iter_mut().for_each(|x| *x /= n);
 }
 struct Data {
    vectors: Vec<Vec<f32>>,
    cluster: Vec<usize>,
    centres: Vec<Vec<f32>>,
 }
 fn data() -> Data {
    let mut rng = Rng(42);
    let centres: Vec<Vec<f32>> = (0..CLUSTERS)
        .map(|_| {
            let mut c: Vec<f32> = (0..DIM).map(|_| rng.unit()).collect();
            normalize(&mut c);
            c
        })
        .collect();
    let mut vectors = Vec::new();
    let mut cluster = Vec::new();
    for i in 0..N {
        let c = i % CLUSTERS;
        let mut v: Vec<f32> = centres[c].iter().map(|x| x + rng.unit() * 0.3).collect();
        normalize(&mut v);
        vectors.push(v);
        cluster.push(c);
    }
    Data {
        vectors,
        cluster,
        centres,
    }
 }
 /// Channel of record `i` for a filter keeping `percent`% of the store at
 /// random (independent of the vectors).
 fn random_channel(i: usize, rng_seed: u64, percent: u64) -> String {
    let mut r = Rng(rng_seed ^ (i as u64 * 7919));
    if r.next() % 100 < percent {
        "keep".into()
    } else {
        "other".into()
    }
 }
 fn build(data: &Data, channel: impl Fn(usize) -> String) -> (TempDir, HDF5Memory) {
    let dir = TempDir::new().unwrap();
    let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
    cfg.hebbian_boost = 0.0; // every query sees the same store
    let mut m = HDF5Memory::create(cfg).unwrap();
    let entries = data
        .vectors
        .iter()
        .enumerate()
        .map(|(i, v)| MemoryEntry {
            chunk: format!("record {i} cluster {}", data.cluster[i]),
            embedding: v.clone(),
            source_channel: channel(i),
            timestamp: i as f64,
            session_id: "s".into(),
            tags: format!("t{i}"),
        })
        .collect();
    m.save_batch(entries).unwrap();
    (dir, m)
 }
 /// Exact top-k by cosine among the records `allowed` keeps.
 fn exact_top(data: &Data, q: &[f32], k: usize, allowed: impl Fn(usize) -> bool) -> Vec<usize> {
    let mut s: Vec<(usize, f32)> = (0..N)
        .filter(|&i| allowed(i))
        .map(|i| (i, data.vectors[i].iter().zip(q).map(|(a, b)| a * b).sum()))
        .collect();
    s.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
    s.into_iter().take(k).map(|(i, _)| i).collect()
 }
 fn query(data: &Data, i: usize) -> Vec<f32> {
    let mut rng = Rng(1000 + i as u64);
    let mut q: Vec<f32> = data.centres[i % CLUSTERS]
        .iter()
        .map(|x| x + rng.unit() * 0.3)
        .collect();
    normalize(&mut q);
    q
 }
 fn vector_only(k: usize) -> SearchOptions {
    SearchOptions::new(k).with_fusion(hybrid::Fusion::Weighted {
        vector: 1.0,
        keyword: 0.0,
    })
 }
 #[test]
 fn source_filter_returns_only_allowed_records_and_a_full_page() {
    let d = data();
    // At N = 3000 and k = 10 the index serves a filter only when that is
    // cheaper than scanning the allowed records: pool = 80 * N / allowed
    // candidates at ~M = 16 distances each, against `allowed` distances. So
    // 90% goes through the index, 50% and 1% to the exact scan.
    for percent in [90, 50, 1] {
        let (_dir, mut m) = build(&d, |i| random_channel(i, 5, percent));
        let allowed = |i: usize| random_channel(i, 5, percent) == "keep";
        let mut hits = 0;
        for qi in 0..40 {
            let q = query(&d, qi);
            let got = m.search(&q, "", &vector_only(10).with_sources(["keep"]));
            assert_eq!(got.len(), 10, "{percent}%: short page");
            assert!(got.iter().all(|r| r.source_channel == "keep"));
            let want: HashSet<usize> = exact_top(&d, &q, 10, allowed).into_iter().collect();
            hits += got.iter().filter(|r| want.contains(&r.index)).count();
        }
        let recall = hits as f64 / 400.0;
        let floor = if percent == 90 { 0.95 } else { 1.0 };
        assert!(recall >= floor, "{percent}%: recall@10 {recall}");
    }
 }
 #[test]
 fn filter_away_from_the_query_falls_back_to_an_exact_scan() {
    // Channel = cluster, and the filter keeps two clusters (10% of the
    // store) that are not the query's: the index's neighbourhood of the
    // query holds none of them. The search must still return the exact
    // top 10 among the allowed records, not a short or empty page.
    let d = data();
    let (_dir, mut m) = build(&d, |i| format!("c{}", d.cluster[i]));
    for qi in 0..20 {
        let q = query(&d, qi);
        let a = format!("c{}", (qi + 7) % CLUSTERS);
        let b = format!("c{}", (qi + 13) % CLUSTERS);
        let got: Vec<usize> = m
            .search(
                &q,
                "",
                &vector_only(10).with_sources([a.clone(), b.clone()]),
            )
            .iter()
            .map(|r| r.index)
            .collect();
        let want = exact_top(&d, &q, 10, |i| {
            let c = format!("c{}", d.cluster[i]);
            c == a || c == b
        });
        assert_eq!(got, want, "query {qi}");
    }
 }
 #[test]
 fn filter_edge_cases() {
    let d = data();
    let (_dir, mut m) = build(&d, |i| random_channel(i, 9, 50));
    let q = query(&d, 0);
    assert!(
        m.search(
            &q,
            "cluster",
            &SearchOptions::new(10).with_sources(Vec::<String>::new())
        )
        .is_empty()
    );
    assert!(
        m.search(
            &q,
            "cluster",
            &SearchOptions::new(10).with_sources(["nope"])
        )
        .is_empty()
    );
    // Keyword matches from other channels are filtered too.
    let got = m.search(
        &q,
        "record cluster",
        &SearchOptions::new(50).with_sources(["keep"]),
    );
    assert_eq!(got.len(), 50);
    assert!(got.iter().all(|r| r.source_channel == "keep"));
    // Deleted records never come back, filtered or not.
    let first = got[0].index;
    m.delete(first).unwrap();
    let again = m.search(
        &q,
        "record cluster",
        &SearchOptions::new(50).with_sources(["keep"]),
    );
    assert!(again.iter().all(|r| r.index != first));
 }
 #[test]
 fn plain_options_equal_hybrid_search_with() {
    // Two identical stores, so neither query sees the other's boosts.
    let d = data();
    let (_a, mut a) = build(&d, |i| random_channel(i, 3, 50));
    let (_b, mut b) = build(&d, |i| random_channel(i, 3, 50));
    for qi in 0..10 {
        let q = query(&d, qi);
        let x: Vec<(usize, u32)> = a
            .search(&q, "record cluster 3", &SearchOptions::new(10))
            .iter()
            .map(|r| (r.index, r.score.to_bits()))
            .collect();
        let y: Vec<(usize, u32)> = b
            .hybrid_search_with(&q, "record cluster 3", hybrid::DEFAULT_FUSION, 10)
            .iter()
            .map(|r| (r.index, r.score.to_bits()))
            .collect();
        assert_eq!(x, y);
    }
 }
 fn small_store(entries: &[(&str, &str, f64)]) -> (TempDir, HDF5Memory) {
    let dir = TempDir::new().unwrap();
    let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("r.h5"), "a", 4)).unwrap();
    m.save_batch(
        entries
            .iter()
            .map(|(chunk, channel, ts)| MemoryEntry {
                chunk: chunk.to_string(),
                embedding: vec![1.0, 0.0, 0.0, 0.0],
                source_channel: channel.to_string(),
                timestamp: *ts,
                session_id: "s".into(),
                tags: String::new(),
            })
            .collect(),
    )
    .unwrap();
    (dir, m)
 }
 #[test]
 fn rerank_breaks_relevance_ties_by_recency() {
    // Identical text and vectors, so retrieval ties; re-ranking must put the
    // newer record first and report the combined score.
    let now = 1_000_000.0;
    let (_d, mut m) = small_store(&[
        ("user prefers dark mode", "chat", now - 30.0 * 86_400.0),
        ("user prefers dark mode", "chat", now - 60.0),
    ]);
    let q = [1.0, 0.0, 0.0, 0.0];
    let plain = m.search(&q, "dark mode", &SearchOptions::new(2));
    assert_eq!(plain[0].index, 0, "ties break by index without re-ranking");
    let reranked = m.search(
        &q,
        "dark mode",
        &SearchOptions::new(2)
            .with_rerank(ReRankConfig::default())
            .at_time(now),
    );
    assert_eq!(reranked[0].index, 1);
    assert!(reranked[0].score > reranked[1].score);
    assert_ne!(reranked[0].score.to_bits(), plain[0].score.to_bits());
 }
 #[test]
 fn confidence_rejects_when_nothing_is_good_enough() {
    let (_d, mut m) = small_store(&[("alpha", "chat", 0.0), ("beta", "chat", 0.0)]);
    let q = [1.0, 0.0, 0.0, 0.0];
    let strict = ConfidenceConfig {
        min_score: 10.0,
        ..ConfidenceConfig::default()
    };
    assert!(
        m.search(&q, "alpha", &SearchOptions::new(2).with_confidence(strict))
            .is_empty()
    );
    let lenient = ConfidenceConfig {
        min_score: 0.0,
        min_gap: f32::INFINITY,
        max_results: 1,
    };
    assert_eq!(
        m.search(&q, "alpha", &SearchOptions::new(2).with_confidence(lenient))
            .len(),
        1
    );
 }
 #[test]
 fn only_returned_results_are_reinforced() {
    // With re-ranking, a pool of max(3k, 10) candidates is retrieved; only
    // the k returned should gain activation.
    let d = data();
    let dir = TempDir::new().unwrap();
    let path = dir.path().join("h.h5");
    let mut m = HDF5Memory::create(MemoryConfig::new(path, "a", DIM)).unwrap();
    m.save_batch(
        (0..200)
            .map(|i| MemoryEntry {
                chunk: format!("record {i}"),
                embedding: d.vectors[i].clone(),
                source_channel: "chat".into(),
                timestamp: i as f64,
                session_id: "s".into(),
                tags: String::new(),
            })
            .collect(),
    )
    .unwrap();
    let q = query(&d, 0);
    let got = m.search(
        &q,
        "record",
        &SearchOptions::new(3).with_rerank(ReRankConfig::default()),
    );
    assert_eq!(got.len(), 3);
    let returned: HashSet<usize> = got.iter().map(|r| r.index).collect();
    // A second plain search reports each record's current activation.
    let all = m.search(&q, "record", &SearchOptions::new(200));
    for r in &all {
        let boosted = r.activation > 1.0;
        assert_eq!(boosted, returned.contains(&r.index), "record {}", r.index);
    }
 }
@@ -64,6 +64,11 @@ use tempfile::TempDir;
 const EMBEDDING_DIM: usize = 384;
 /// `--float16`: build every per-question store with `MemoryConfig::float16`,
 /// so embeddings are rounded to half precision as they are saved — exactly
 /// what such a store searches over.
 static FLOAT16: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
 /// A mode's fusion, as one short string for the reports.
 fn describe(mode: Mode) -> String {
    let fusion = match mode.fusion {
@@ -431,6 +436,7 @@ fn evaluate_question(
    let mut config = MemoryConfig::new(dir.path().join("lme.h5"), "lme-bench", EMBEDDING_DIM);
    config.wal_enabled = false;
    config.compact_threshold = 0.0;
    config.float16 = FLOAT16.load(std::sync::atomic::Ordering::Relaxed);
    let mut memory = HDF5Memory::create(config).expect("failed to create HDF5Memory");
    memory.set_token_filter(mode.tokens);
@@ -940,6 +946,10 @@ fn main() {
                limit = Some(v.parse().expect("--limit must be a positive integer"));
            }
            "--sweep" => sweep = true,
            "--float16" => {
                FLOAT16.store(true, std::sync::atomic::Ordering::Relaxed);
                eprintln!("Stores use MemoryConfig::float16 (half-precision embeddings)");
            }
            "--rerank-sweep" => {
                // Re-ranking needs the vector stage to have candidates worth
                // reordering, so this is an embeddings-only comparison.
@@ -971,6 +981,9 @@ fn main() {
                     --rerank-sweep\n\
                              compare re-ranking off, metadata-only (the old\n\
                              behaviour) and blended at several half-lives.\n\
                     --float16\n\
                              build each store with MemoryConfig::float16, to\n\
                              compare retrieval on half-precision embeddings.\n\
                     --sweep  instead of the three named modes, sweep vector_weight\n\
                              from 0.0 to 1.0 in 0.1 steps. The 0.7/0.3 default was\n\
                              never searched; this is what searches it."
@@ -19,6 +19,8 @@
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --full  # + 100K
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --json out.json
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
 //! ```
 use std::time::{Duration, Instant};
@@ -88,6 +90,9 @@ static UNIFORM: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::n
 /// the memory) instead of f32, to price the recall it costs.
 static INT8: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
 /// `--f16-first`: in `--float16-study`, run the float16 store first.
 static F16_FIRST: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
 /// `--rerank`: re-score the candidate pool against the exact vectors before
 /// taking the top K.
 static RERANK: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
@@ -483,6 +488,319 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
    }));
 }
 // ---------------------------------------------------------------------------
 // Search options study: source filters, re-ranking, confidence rejection
 // ---------------------------------------------------------------------------
 /// `--options-study`: what `HDF5Memory::search`'s options cost and whether a
 /// filtered search finds the right records. Filters keep 50%, 10% or 1% of
 /// the store at random, or two whole clusters away from the query (the case
 /// the index cannot serve, which falls back to an exact scan). Recall is
 /// vector-only against an exact scan of the allowed records; latency is full
 /// hybrid search. Hebbian boosting is off.
 fn options_study(n: usize) {
    use clawhdf5_agent::SearchOptions;
    use clawhdf5_agent::confidence::ConfidenceConfig;
    use clawhdf5_agent::hybrid::Fusion;
    use clawhdf5_agent::reranker::ReRankConfig;
    let data = make_dataset(n, 0x0B7 ^ n as u64);
    let n_clusters = data.cluster_of.iter().max().map_or(1, |m| m + 1);
    let mut rng = Rng(5);
    let bucket_of: Vec<usize> = (0..n).map(|_| rng.below(100)).collect();
    let bucket = &bucket_of;
    let query_texts: Vec<String> = data
        .query_cluster
        .iter()
        .enumerate()
        .map(|(i, c)| text_for(*c, i, &mut rng))
        .collect();
    let exact_top = |q: &[f32], allowed: &dyn Fn(usize) -> bool| -> Vec<usize> {
        let mut s: Vec<(usize, f32)> = (0..n)
            .filter(|&i| allowed(i))
            .map(|i| (i, data.vectors[i].iter().zip(q).map(|(a, b)| a * b).sum()))
            .collect();
        s.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
        s.into_iter().take(K).map(|(i, _)| i).collect()
    };
    // Two stores: channel = random bucket, and channel = cluster.
    let dir = tempfile::tempdir().unwrap();
    let mut stores = Vec::new();
    for by_cluster in [false, true] {
        let mut rng = Rng(3);
        let entries: Vec<MemoryEntry> = data
            .vectors
            .iter()
            .enumerate()
            .map(|(i, v)| MemoryEntry {
                chunk: text_for(data.cluster_of[i], i, &mut rng),
                embedding: v.clone(),
                source_channel: if by_cluster {
                    format!("c{}", data.cluster_of[i])
                } else {
                    format!("b{}", bucket[i])
                },
                timestamp: i as f64,
                session_id: format!("s{}", i % 50),
                tags: format!("t{i}"),
            })
            .collect();
        let mut config = MemoryConfig::new(
            dir.path().join(format!("opt_{by_cluster}.h5")),
            "bench",
            DIM,
        );
        config.hebbian_boost = 0.0;
        let mut mem = HDF5Memory::create(config).unwrap();
        mem.save_batch(entries).unwrap();
        std::hint::black_box(mem.search(&data.queries[0], "", &SearchOptions::new(K)));
        stores.push(mem);
    }
    let vector_only = SearchOptions::new(K).with_fusion(Fusion::Weighted {
        vector: 1.0,
        keyword: 0.0,
    });
    // (label, store, channels for query i, allowed(i, record))
    type Case<'a> = (
        String,
        usize,
        Box<dyn Fn(usize) -> Option<Vec<String>> + 'a>,
        Box<dyn Fn(usize, usize) -> bool + 'a>,
    );
    let mut cases: Vec<Case> = vec![(
        "no filter".into(),
        0,
        Box::new(|_| None),
        Box::new(|_, _| true),
    )];
    for pct in [50usize, 10, 1] {
        cases.push((
            format!("random {pct}%"),
            0,
            Box::new(move |_| Some((0..pct).map(|b| format!("b{b}")).collect())),
            Box::new(move |_, i| bucket[i] < pct),
        ));
    }
    let d = &data;
    let away = move |qi: usize| {
        let qc = d.query_cluster[qi];
        [
            (qc + n_clusters / 3) % n_clusters,
            (qc + 2 * n_clusters / 3) % n_clusters,
        ]
    };
    cases.push((
        "2 clusters away from the query".into(),
        1,
        Box::new(move |qi| Some(away(qi).iter().map(|c| format!("c{c}")).collect())),
        Box::new(move |qi, i| away(qi).contains(&d.cluster_of[i])),
    ));
    for (label, store, channels, allowed) in &cases {
        let mem = &mut stores[*store];
        let mut hits = 0;
        let mut kept = 0;
        for (qi, q) in data.queries.iter().enumerate() {
            let mut opts = vector_only.clone();
            opts.source_channels = channels(qi);
            let got = mem.search(q, "", &opts);
            let want = exact_top(q, &|i| allowed(qi, i));
            kept += want.len();
            hits += got.iter().filter(|r| want.contains(&r.index)).count();
        }
        let latency = summarize(
            (0..N_QUERIES)
                .map(|qi| {
                    let mut opts = SearchOptions::new(K);
                    opts.source_channels = channels(qi);
                    let t = Instant::now();
                    std::hint::black_box(mem.search(&data.queries[qi], &query_texts[qi], &opts));
                    t.elapsed()
                })
                .collect(),
        );
        println!(
            "| {n} | {label} | {:.4} | {:.3} | {:.3} |",
            hits as f64 / kept.max(1) as f64,
            millis(latency.p50),
            millis(latency.p99),
        );
    }
    let mem = &mut stores[0];
    for (label, opts) in [
        (
            "re-rank",
            SearchOptions::new(K).with_rerank(ReRankConfig::default()),
        ),
        (
            "re-rank + confidence",
            SearchOptions::new(K)
                .with_rerank(ReRankConfig::default())
                .with_confidence(ConfidenceConfig::default()),
        ),
    ] {
        let latency = summarize(
            (0..N_QUERIES)
                .map(|qi| {
                    let t = Instant::now();
                    std::hint::black_box(mem.search(&data.queries[qi], &query_texts[qi], &opts));
                    t.elapsed()
                })
                .collect(),
        );
        println!(
            "| {n} | {label} | — | {:.3} | {:.3} |",
            millis(latency.p50),
            millis(latency.p99)
        );
    }
 }
 // ---------------------------------------------------------------------------
 // float16 study: what does half-precision embedding storage cost?
 // ---------------------------------------------------------------------------
 /// `--float16-study`: the same data in an `f32` store and a `float16` store.
 /// Reports file size, checkpoint and open time, vector-search recall@10
 /// against an exact scan of the *original* f32 vectors, how often the two
 /// stores return the same top 10, and `hybrid_search` latency. Hebbian
 /// boosting is off, so every query sees the same store.
 fn float16_study(n: usize) {
    let data = make_dataset(n, 0xF16 ^ n as u64);
    let mut rng = Rng(11);
    let query_texts: Vec<String> = data
        .query_cluster
        .iter()
        .enumerate()
        .map(|(i, c)| text_for(*c, i, &mut rng))
        .collect();
    // Exact top K by cosine (the vectors are unit length) on the f32 inputs.
    let exact: Vec<Vec<usize>> = data
        .queries
        .iter()
        .map(|q| {
            let mut scored: Vec<(usize, f32)> = data
                .vectors
                .iter()
                .enumerate()
                .map(|(i, v)| (i, v.iter().zip(q).map(|(a, b)| a * b).sum()))
                .collect();
            scored.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
            scored.into_iter().take(K).map(|(i, _)| i).collect()
        })
        .collect();
    let dir = tempfile::tempdir().unwrap();
    let mut per_variant: Vec<(bool, Vec<Vec<usize>>)> = Vec::new();
    // `--f16-first` swaps the order, to check the numbers do not depend on
    // which store runs first (page cache, allocator, CPU frequency).
    let order = if F16_FIRST.load(std::sync::atomic::Ordering::Relaxed) {
        [true, false]
    } else {
        [false, true]
    };
    for float16 in order {
        let path = dir.path().join(format!("f16study_{float16}.h5"));
        let mut rng = Rng(3);
        let entries: Vec<MemoryEntry> = data
            .vectors
            .iter()
            .enumerate()
            .map(|(i, v)| MemoryEntry {
                chunk: text_for(data.cluster_of[i], i, &mut rng),
                embedding: v.clone(),
                source_channel: "bench".into(),
                timestamp: i as f64,
                session_id: format!("s{}", i % 50),
                tags: format!("t{i}"),
            })
            .collect();
        let mut config = MemoryConfig::new(path.clone(), "bench", DIM);
        config.float16 = float16;
        config.hebbian_boost = 0.0;
        let mut mem = HDF5Memory::create(config).unwrap();
        mem.save_batch(entries).unwrap();
        // Build the indexes, then time a checkpoint that writes everything.
        std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
        let t = Instant::now();
        mem.flush_wal().unwrap();
        let checkpoint = t.elapsed();
        drop(mem);
        let file_bytes = std::fs::metadata(&path).unwrap().len();
        // Median of three opens.
        let mut opens: Vec<Duration> = (0..3)
            .map(|_| {
                let t = Instant::now();
                let m = HDF5Memory::open(&path).unwrap();
                let d = t.elapsed();
                drop(m);
                d
            })
            .collect();
        opens.sort();
        let mut mem = HDF5Memory::open(&path).unwrap();
        // Vector-only search: empty text, all weight on the vector stage.
        let results: Vec<Vec<usize>> = data
            .queries
            .iter()
            .map(|q| {
                mem.hybrid_search(q, "", 1.0, 0.0, K)
                    .iter()
                    .map(|r| r.index)
                    .collect()
            })
            .collect();
        let hits: usize = results
            .iter()
            .zip(&exact)
            .map(|(got, want)| got.iter().filter(|i| want.contains(i)).count())
            .sum();
        let recall = hits as f64 / (K * data.queries.len()) as f64;
        let latency = summarize(
            (0..N_QUERIES)
                .map(|i| {
                    let t = Instant::now();
                    std::hint::black_box(mem.hybrid_search(
                        &data.queries[i],
                        &query_texts[i],
                        0.4,
                        0.6,
                        K,
                    ));
                    t.elapsed()
                })
                .collect(),
        );
        let overlap = match per_variant.first() {
            Some((_, other)) => {
                let same: usize = results
                    .iter()
                    .zip(other)
                    .map(|(a, b)| a.iter().filter(|i| b.contains(i)).count())
                    .sum();
                format!("{:.4}", same as f64 / (K * data.queries.len()) as f64)
            }
            None => "—".into(),
        };
        println!(
            "| {n} | {} | {:.1} | {:.0} | {:.1} | {recall:.4} | {overlap} | {:.3} |",
            if float16 { "float16" } else { "f32" },
            mib(file_bytes),
            millis(checkpoint),
            millis(opens[1]),
            millis(latency.p50),
        );
        per_variant.push((float16, results));
    }
 }
 // ---------------------------------------------------------------------------
 // Fusion study: does capping the keyword candidate pool change the ranking?
 // ---------------------------------------------------------------------------
@@ -642,6 +960,37 @@ fn main() {
        }
        return;
    }
    if args.iter().any(|a| a == "--options-study") {
        println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
        println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");
        println!("|---:|---|---:|---:|---:|");
        for &n in if full {
            &[10_000, 100_000][..]
        } else {
            &[10_000][..]
        } {
            options_study(n);
        }
        return;
    }
    if args.iter().any(|a| a == "--f16-first") {
        F16_FIRST.store(true, std::sync::atomic::Ordering::Relaxed);
    }
    if args.iter().any(|a| a == "--float16-study") {
        println!("## float16 embedding storage ({DIM}-dim, int8 index, Hebbian boost off)\n");
        println!(
            "| N | embeddings | file MiB | checkpoint ms | open ms | recall@10 | top-10 overlap with the other | hybrid p50 ms |"
        );
        println!("|---:|---|---:|---:|---:|---:|---:|---:|");
        for &n in if full {
            &[1_000, 10_000, 100_000][..]
        } else {
            &[1_000, 10_000][..]
        } {
            float16_study(n);
        }
        return;
    }
    if args.iter().any(|a| a == "--int8") {
        INT8.store(true, std::sync::atomic::Ordering::Relaxed);
        println!("(int8-quantised index vectors)");
@@ -36,6 +36,14 @@ enum Commands {
        /// Accepted for compatibility; int8 is now the default
        #[arg(long, hide = true, conflicts_with = "f32_index")]
        quantized_index: bool,
        /// Store embeddings as full-precision f32 instead of the default
        /// half precision (float16: half the bytes, about three significant
        /// digits, values within ±65504)
        #[arg(long)]
        f32: bool,
        /// Accepted for compatibility; float16 is now the default
        #[arg(long, hide = true, conflicts_with = "f32")]
        float16: bool,
    },
    /// Save a memory entry (reads JSON from stdin or --json)
    Save {
@@ -102,9 +110,16 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
            wal,
            f32_index,
            quantized_index: _,
            f32,
            float16: _,
        } => {
            let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim);
            config.wal_enabled = wal;
            // As with --f32-index: only ever switch the library default off.
            if f32 {
                config.float16 = false;
            }
            let config_float16 = config.float16;
            // Only ever switch *off* the library default: assigning the flag
            // outright would force every CLI-created store back to f32 unless
            // the caller knew to ask for int8.
@@ -120,6 +135,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
                "embedding_dim": dim,
                "wal_enabled": wal,
                "quantized_index": config_quantized,
                "float16": config_float16,
                "count": mem.count(),
            });
            println!("{}", serde_json::to_string_pretty(&j)?);
@@ -24,6 +24,7 @@ libaec-sys = { path = "../libaec-sys", version = "0.1", optional = true }
 pco = { version = "1.0", optional = true }
 [dev-dependencies]
 half = { workspace = true }
 serde_json = "1"
 criterion = { workspace = true }
 clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.7.0" }
@@ -1076,6 +1076,21 @@ pub fn read_as_f32(raw: &[u8], datatype: &Datatype) -> Result<Vec<f32>, FormatEr
    ) {
        return Ok(native_le_to_vec::<f32>(raw, count));
    }
    // Little-endian half precision (numpy float16): widen directly.
    if matches!(
        datatype,
        Datatype::FloatingPoint {
            size: 2,
            byte_order: DatatypeByteOrder::LittleEndian,
            ..
        }
    ) {
        let (halves, _) = raw[..count * 2].as_chunks::<2>();
        return Ok(halves
            .iter()
            .map(|&b| f16_bits_to_f32(u16::from_le_bytes(b)))
            .collect());
    }
    let order = get_byte_order(datatype);
    let mut result = Vec::with_capacity(count);
@@ -1622,36 +1637,7 @@ fn read_f16_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f32 {
    f16_bits_to_f32(u16::from_le_bytes(buf))
 }
-/// Convert the bit pattern of an IEEE-754 half (binary16) to an `f32`.
+use crate::float16::f16_bits_to_f32;
 fn f16_bits_to_f32(h: u16) -> f32 {
    let h = h as u32;
    let sign = (h & 0x8000) << 16;
    let exp = (h >> 10) & 0x1f;
    let mant = h & 0x3ff;
    let bits = if exp == 0 {
        if mant == 0 {
            sign // signed zero
        } else {
            // Subnormal: normalize into an f32 normal.
            let mut e: i32 = -1;
            let mut m = mant;
            loop {
                e += 1;
                m <<= 1;
                if m & 0x400 != 0 {
                    break;
                }
            }
            let m = m & 0x3ff;
            sign | (((127 - 15 - e) as u32) << 23) | (m << 13)
        }
    } else if exp == 0x1f {
        sign | 0x7f80_0000 | (mant << 13) // inf / NaN
    } else {
        sign | ((exp + (127 - 15)) << 23) | (mant << 13)
    };
    f32::from_bits(bits)
 }
 fn read_f32_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f32 {
    let mut buf = [0u8; 4];
@@ -640,7 +640,8 @@ impl Datatype {
                mantissa_size,
                exponent_bias,
            } => {
-                let mut bf0 = 0x20u8; // bit 5: sign location bit (standard IEEE 754)
+                // Bits 4-5: mantissa normalization = 2 (implied leading 1, IEEE 754).
                let mut bf0 = 0x20u8;
                match byte_order {
                    DatatypeByteOrder::BigEndian => {
                        bf0 |= 0x01;
@@ -650,9 +651,14 @@ impl Datatype {
                    }
                    _ => {}
                }
-                // bf[1] bits 0-1: mantissa normalization = 2 (MSB not stored, IEEE 754)
+                // Bits 8-15: the sign bit's position, the top bit of the value.
-                let bf1 = 0x3fu8; // matching what h5py generates
+                // This was hard-coded to 63, which is right only for f64: the
-                let mut buf = Self::build_header(1, 1, [bf0, bf1, 0], *size);
+                // HDF5 library rejects any other float with "sign bit position
                // out of bounds", so every f32 dataset and attribute we wrote
                // was unreadable by h5py and libhdf5.
                let sign_location =
                    (u32::from(*bit_offset) + u32::from(*bit_precision)).saturating_sub(1) as u8;
                let mut buf = Self::build_header(1, 1, [bf0, sign_location, 0], *size);
                buf.extend_from_slice(&bit_offset.to_le_bytes());
                buf.extend_from_slice(&bit_precision.to_le_bytes());
                buf.push(*exponent_location);
@@ -818,6 +824,24 @@ fn build_dt_header(class: u8, version: u8, bf: [u8; 3], size: u32) -> Vec<u8> {
 mod tests {
    use super::*;
    #[test]
    fn float_sign_location_is_the_top_bit_of_the_value() {
        // The HDF5 library rejects a float whose sign position is not inside
        // its precision; this was hard-coded to 63, so every f32 we wrote was
        // unreadable by h5py. Byte 2 of the message is the sign position.
        use crate::type_builders::{make_f16_type, make_f32_type, make_f64_type};
        for (dt, sign) in [
            (make_f16_type(), 15),
            (make_f32_type(), 31),
            (make_f64_type(), 63),
        ] {
            let bytes = dt.serialize();
            assert_eq!(bytes[2], sign, "{dt:?}");
            let (parsed, _) = Datatype::parse(&bytes).unwrap();
            assert_eq!(parsed, dt);
        }
    }
    // Helper to build a fixed-point datatype message
    fn build_fixed_point(
        size: u32,
@@ -86,6 +86,12 @@ pub(crate) fn build_dataset_oh(
    let mut dl = Vec::new();
    dl.push(4); // version
    dl.push(1); // class = contiguous
    // An empty dataset has no storage: its address must be the undefined
    // address, as libhdf5 writes it. A real address with size 0 trips
    // libhdf5's `addr + size <= addr` overflow check, and it refuses the
    // dataset as "invalid dataset size, likely file corruption" — which made
    // every store with no sessions or knowledge graph unreadable by h5py.
    let data_addr = if data_size == 0 { u64::MAX } else { data_addr };
    dl.extend_from_slice(&data_addr.to_le_bytes());
    dl.extend_from_slice(&data_size.to_le_bytes());
    w.add_message(MessageType::DataLayout, dl);
@@ -0,0 +1,155 @@
 //! IEEE-754 half precision (binary16) conversions.
 //!
 //! Pure integer bit manipulation, so it works under `no_std` and needs no
 //! `libm`. The writer ([`crate::type_builders::DatasetBuilder::with_f16_data`]),
 //! the reader and `clawhdf5-agent`'s half-precision embedding store all use
 //! these two functions, so a value rounded in memory is bit-for-bit the value
 //! that reads back from the file.
 /// Largest finite half-precision value. Anything larger in magnitude rounds
 /// to infinity.
 pub const F16_MAX: f32 = 65504.0;
 /// Convert an `f32` to the bit pattern of the nearest half-precision value,
 /// rounding ties to even (the IEEE default, and what numpy and the `half`
 /// crate do).
 ///
 /// Values beyond ±[`F16_MAX`] become ±infinity, values too small for a
 /// subnormal become signed zero, and NaN stays NaN (quiet, payload
 /// truncated).
 pub fn f32_to_f16_bits(value: f32) -> u16 {
    let x = value.to_bits();
    let sign = (x >> 16) & 0x8000;
    let exp = x & 0x7F80_0000;
    let man = x & 0x007F_FFFF;
    // Infinity and NaN.
    if exp == 0x7F80_0000 {
        let quiet_nan = if man == 0 { 0 } else { 0x0200 };
        return (sign | 0x7C00 | quiet_nan | (man >> 13)) as u16;
    }
    let half_exp = ((exp >> 23) as i32) - 127 + 15;
    // Too large: infinity.
    if half_exp >= 0x1F {
        return (sign | 0x7C00) as u16;
    }
    // Subnormal half, or zero.
    if half_exp <= 0 {
        if 14 - half_exp > 24 {
            return sign as u16;
        }
        let man = man | 0x0080_0000; // implicit leading bit
        let shift = (14 - half_exp) as u32;
        let mut half_man = man >> shift;
        let round_bit = 1u32 << (shift - 1);
        // Round half to even: up if above half, or exactly half and odd.
        if (man & round_bit) != 0 && (man & (3 * round_bit - 1)) != 0 {
            half_man += 1;
        }
        return (sign | half_man) as u16;
    }
    // Normal half. A mantissa carry correctly rolls into the exponent (and
    // from the largest finite value into infinity).
    let half = sign | ((half_exp as u32) << 10) | (man >> 13);
    let round_bit = 0x0000_1000;
    if (man & round_bit) != 0 && (man & (3 * round_bit - 1)) != 0 {
        (half + 1) as u16
    } else {
        half as u16
    }
 }
 /// Convert the bit pattern of a half-precision value to `f32` (exact: every
 /// half value is representable as an `f32`).
 pub fn f16_bits_to_f32(h: u16) -> f32 {
    let h = h as u32;
    let sign = (h & 0x8000) << 16;
    let exp = (h >> 10) & 0x1f;
    let mant = h & 0x3ff;
    let bits = if exp == 0 {
        if mant == 0 {
            sign // signed zero
        } else {
            // Subnormal: normalize into an f32 normal.
            let mut e: i32 = -1;
            let mut m = mant;
            loop {
                e += 1;
                m <<= 1;
                if m & 0x400 != 0 {
                    break;
                }
            }
            let m = m & 0x3ff;
            sign | (((127 - 15 - e) as u32) << 23) | (m << 13)
        }
    } else if exp == 0x1f {
        sign | 0x7f80_0000 | (mant << 13) // inf / NaN
    } else {
        sign | ((exp + 127 - 15) << 23) | (mant << 13)
    };
    f32::from_bits(bits)
 }
 /// Round an `f32` to the nearest half-precision value, returned as `f32`.
 pub fn round_to_f16(value: f32) -> f32 {
    f16_bits_to_f32(f32_to_f16_bits(value))
 }
 #[cfg(test)]
 mod tests {
    use super::*;
    #[test]
    fn every_half_value_round_trips() {
        for bits in 0..=u16::MAX {
            let v = f16_bits_to_f32(bits);
            if v.is_nan() {
                assert!(f16_bits_to_f32(f32_to_f16_bits(v)).is_nan(), "{bits:#06x}");
            } else {
                assert_eq!(f32_to_f16_bits(v), bits, "{bits:#06x} -> {v}");
            }
        }
    }
    #[test]
    fn matches_the_half_crate() {
        // Every 257th f32 bit pattern (~16.7M values) covers every exponent,
        // the subnormal range, both signs, ties and the overflow boundary.
        let mut bits: u32 = 0;
        loop {
            let v = f32::from_bits(bits);
            let ours = f32_to_f16_bits(v);
            let theirs = half::f16::from_f32(v);
            if v.is_nan() {
                assert!(theirs.is_nan() && f16_bits_to_f32(ours).is_nan());
            } else {
                assert_eq!(ours, theirs.to_bits(), "{bits:#010x} ({v:e})");
                assert_eq!(f16_bits_to_f32(ours).to_bits(), theirs.to_f32().to_bits());
            }
            match bits.checked_add(257) {
                Some(b) => bits = b,
                None => break,
            }
        }
    }
    #[test]
    fn rounds_ties_to_even_and_saturates_to_infinity() {
        // 1 + 2^-11 is exactly halfway between 1.0 and the next half (1 + 2^-10).
        assert_eq!(round_to_f16(1.0 + 2f32.powi(-11)), 1.0);
        assert_eq!(
            round_to_f16(1.0 + 3.0 * 2f32.powi(-11)),
            1.0 + 2.0 * 2f32.powi(-10)
        );
        assert_eq!(round_to_f16(F16_MAX), F16_MAX);
        assert_eq!(round_to_f16(65520.0), f32::INFINITY); // halfway to 2^16 rounds up
        assert_eq!(round_to_f16(-1e9), f32::NEG_INFINITY);
        assert_eq!(round_to_f16(1e-9).to_bits(), 0);
        assert_eq!(round_to_f16(-1e-9).to_bits(), (-0.0f32).to_bits());
    }
 }
@@ -72,6 +72,7 @@ pub mod filter_pipeline;
 pub mod filters;
 mod filters_szip;
 pub mod fixed_array;
 pub mod float16;
 pub mod fractal_heap;
 pub mod global_heap;
 pub mod group_info;
@@ -56,6 +56,21 @@ pub fn make_f64_type() -> Datatype {
    }
 }
 /// IEEE-754 half precision (binary16), little-endian — numpy's `float16`.
 pub fn make_f16_type() -> Datatype {
    Datatype::FloatingPoint {
        size: 2,
        byte_order: DatatypeByteOrder::LittleEndian,
        bit_offset: 0,
        bit_precision: 16,
        exponent_location: 10,
        exponent_size: 5,
        mantissa_location: 0,
        mantissa_size: 10,
        exponent_bias: 15,
    }
 }
 pub fn make_f32_type() -> Datatype {
    Datatype::FloatingPoint {
        size: 4,
@@ -478,6 +493,24 @@ impl DatasetBuilder {
        self
    }
    /// Store `data` as IEEE half precision (numpy `float16`), rounding each
    /// value to the nearest half ([`crate::float16::f32_to_f16_bits`]).
    /// Half the bytes of [`Self::with_f32_data`], at about three significant
    /// decimal digits; values beyond ±65504 become ±infinity. Reading it back
    /// with `read_f32` yields the rounded values exactly.
    pub fn with_f16_data(&mut self, data: &[f32]) -> &mut Self {
        self.datatype = Some(make_f16_type());
        let mut b = Vec::with_capacity(data.len() * 2);
        for &v in data {
            b.extend_from_slice(&crate::float16::f32_to_f16_bits(v).to_le_bytes());
        }
        self.data = Some(b);
        if self.shape.is_none() {
            self.shape = Some(vec![data.len() as u64]);
        }
        self
    }
    pub fn with_i32_data(&mut self, data: &[i32]) -> &mut Self {
        self.datatype = Some(make_i32_type());
        let mut b = Vec::with_capacity(data.len() * 4);
@@ -1387,3 +1387,166 @@ with h5py.File("{path_str}", "w", libver="latest") as f:
        );
    }
 }
 // ---------------------------------------------------------------------------
 // Half precision (float16) in both directions
 // ---------------------------------------------------------------------------
 /// Values that exercise rounding: ties, subnormals, the overflow boundary and
 /// ordinary embedding-sized components.
 fn f16_probe_values() -> Vec<f32> {
    let mut v = vec![
        0.0,
        -0.0,
        1.0,
        -1.0,
        0.5,
        1.0 + 2f32.powi(-11),
        1.0 + 3.0 * 2f32.powi(-11),
        65504.0,
        65519.0,
        65520.0,
        -70000.0,
        6.0e-8,
        3.0e-8,
        1.0e-9,
        1.0e-5,
        0.1,
        0.333_333,
        1234.567,
        f32::INFINITY,
        f32::NEG_INFINITY,
    ];
    // A deterministic spread of embedding-like values.
    let mut x = 0x2545_F491u32;
    for _ in 0..4000 {
        x ^= x << 13;
        x ^= x >> 17;
        x ^= x << 5;
        v.push((x as f32 / u32::MAX as f32 - 0.5) * 0.4);
    }
    v
 }
 #[test]
 fn clawhdf5_writes_f16_h5py_reads() {
    skip_if_no_python!();
    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("ours_f16.h5");
    let path_str = path.display().to_string();
    let values = f16_probe_values();
    let mut fb = FileBuilder::new();
    fb.create_dataset("plain").with_f16_data(&values);
    fb.create_dataset("chunked")
        .with_f16_data(&values)
        .with_shape(&[values.len() as u64])
        .with_chunks(&[512])
        .with_deflate(6);
    fb.write(&path).unwrap();
    // h5py must see a genuine float16 dataset, and our rounding must agree
    // with numpy's own float32 -> float16 conversion bit for bit.
    let input = values
        .iter()
        .map(|v| format!("{:?}", v.to_bits()))
        .collect::<Vec<_>>()
        .join(",");
    let script = format!(
        r#"
 import h5py, numpy as np
 src = np.array([{input}], dtype=np.uint32).view(np.float32)
 expected = src.astype(np.float16).view(np.uint16)
 with h5py.File("{path_str}", "r") as f:
    for name in ("plain", "chunked"):
        d = f[name]
        assert d.dtype == np.float16, (name, d.dtype)
        got = d[:].view(np.uint16)
        bad = np.nonzero(got != expected)[0]
        assert bad.size == 0, (name, bad[:5], got[bad[:5]], expected[bad[:5]])
 print("ok")
 "#
    );
    assert_eq!(run_python_output(&script), "ok");
 }
 #[test]
 fn h5py_writes_f16_clawhdf5_reads() {
    skip_if_no_python!();
    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("h5py_f16.h5");
    let path_str = path.display().to_string();
    let values = f16_probe_values();
    let input = values
        .iter()
        .map(|v| format!("{:?}", v.to_bits()))
        .collect::<Vec<_>>()
        .join(",");
    let script = format!(
        r#"
 import h5py, numpy as np
 src = np.array([{input}], dtype=np.uint32).view(np.float32).astype(np.float16)
 with h5py.File("{path_str}", "w") as f:
    f.create_dataset("plain", data=src)
    f.create_dataset("chunked", data=src, chunks=(512,), compression="gzip", shuffle=True)
    f.create_dataset("big_endian", data=src.astype(">f2"))
 "#
    );
    run_python(&script);
    let expected: Vec<u32> = values
        .iter()
        .map(|&v| clawhdf5_format::float16::round_to_f16(v).to_bits())
        .collect();
    let file = File::open(&path).unwrap();
    for name in ["plain", "chunked", "big_endian"] {
        let ds = file.dataset(name).unwrap();
        assert_eq!(
            ds.dtype().unwrap(),
            DType::Other("float16".into()),
            "{name}"
        );
        let got: Vec<u32> = ds.read_f32().unwrap().iter().map(|v| v.to_bits()).collect();
        assert_eq!(got, expected, "{name}");
    }
 }
 #[test]
 fn clawhdf5_writes_f32_h5py_reads() {
    // Every f32 dataset used to be unreadable by h5py ("sign bit position out
    // of bounds"): the float datatype's sign position was hard-coded for f64.
    skip_if_no_python!();
    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("ours_f32.h5");
    let path_str = path.display().to_string();
    let values: Vec<f32> = vec![1.5, -2.25, 3.0e-7, 65536.5, f32::MAX, -0.0];
    let mut fb = FileBuilder::new();
    fb.create_dataset("plain").with_f32_data(&values);
    fb.create_dataset("chunked")
        .with_f32_data(&values)
        .with_shape(&[values.len() as u64])
        .with_chunks(&[4])
        .with_deflate(6);
    fb.write(&path).unwrap();
    let bits = values
        .iter()
        .map(|v| v.to_bits().to_string())
        .collect::<Vec<_>>()
        .join(",");
    let script = format!(
        r#"
 import h5py, numpy as np
 expected = np.array([{bits}], dtype=np.uint32)
 with h5py.File("{path_str}", "r") as f:
    for name in ("plain", "chunked"):
        d = f[name]
        assert d.dtype == np.float32, (name, d.dtype)
        assert (d[:].view(np.uint32) == expected).all(), (name, d[:])
 print("ok")
 "#
    );
    assert_eq!(run_python_output(&script), "ok");
 }
@@ -227,3 +227,55 @@ wrong structure. All four are fixed and covered by interop tests against
 HDF5 2.0 at sizes that cross each boundary, including paged data blocks.
 Files written by this crate are unaffected — this was purely a read-path bug.
 ## Every `f32` dataset we wrote was unreadable by h5py / libhdf5
 **Status:** fixed 2026-09-23, after v2.7.0. **Every
 release up to and including v2.7.0 is affected** — the encoder was already
 wrong in v2.1.0.
 The floating-point datatype message carries the position of the sign bit
 (bits 8–15 of its class bit field). `clawhdf5-format` wrote 63 for every
 float, which is correct only for `f64`. libhdf5 validates the field, so opening
 any `f32` dataset written by this crate failed:
 ```
 KeyError: 'Unable to synchronously open object (sign bit position out of bounds)'
 ```
 That covers every agent store (`/memory/embeddings`, `norms` and
 `activation_weights` are `f32`). `clawhdf5` itself ignores the field on read,
 and the interop suites only ever wrote `f64` from our side, so nothing here
 noticed.
 **Fix:** the sign position is computed from the type (`bit_offset +
 bit_precision - 1`: 15, 31, 63 for half, single, double). Regression tests:
 `float_sign_location_is_the_top_bit_of_the_value` (byte level),
 `clawhdf5_writes_f32_h5py_reads` and the agent's
 `h5py_reads_every_dataset_of_an_agent_store`.
 **Existing files:** an agent store is rewritten in full at every checkpoint, so
 it becomes readable by h5py at its next checkpoint with a fixed build. Other
 files with `f32` datasets need to be rewritten.
 ## Empty datasets we wrote were unreadable by h5py / libhdf5
 **Status:** fixed 2026-09-23, after v2.7.0. Every
 release up to and including v2.7.0 is affected.
 A dataset with no elements was written with a real file address and a storage
 size of 0. libhdf5 guards contiguous storage with an overflow check
 (`addr + size <= addr`) that is always true when the size is 0, so it rejected
 the dataset:
 ```
 KeyError: 'Unable to synchronously open object (invalid dataset size, likely file corruption)'
 ```
 In practice: every agent store without sessions or a knowledge graph — the
 `/sessions` and `/knowledge_graph` datasets are empty until something is added
 — could not be read by h5py even once the `f32` bug above was fixed. Found by
 the same agent-store interop test.
 **Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
 which is what libhdf5 itself writes.
Author	SHA1	Message	Date
osobhandClaude Opus 5.5	5c8323cb1e	feat(agent): new stores default to float16 embeddings MemoryConfig::float16 now defaults to true for new stores, on measurement: on the full LongMemEval haystack with real MiniLM embeddings every retrieval metric matched f32 (previous commit), and at 100K the file is 48% smaller with faster checkpoints and opens. Existing stores are unaffected: every agent store has recorded `float16 = false` in /meta and keeps it. A test opens the v2.5.0 fixture, saves and checkpoints, and checks the embeddings are still f32 with the old rows bit-identical; another checks a new store is float16. CLI: `create --f32` opts out; like `--f32-index` it only ever switches the default off. `--float16` is still accepted and now a no-op. Values beyond +-65504 are refused, so f32 remains the choice for unnormalised vectors — the upgrade note says so. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-24 19:39:33 -05:00
osobhandClaude Opus 5.5	dbaf3f505d	bench(longmemeval): --float16, and float16 measured on real embeddings `longmemeval_bench --float16` builds every per-question store with MemoryConfig::float16, so the vector stage searches half-rounded embeddings exactly as such a store holds them. Full longmemeval_s (500 questions, ~494 turns each) with real all-MiniLM-L6-v2 embeddings, f32 vs float16, on tank (CUDA): identical at every Hit@k and MRR, turn and session level, in all eight modes — bar RRF session MRR 0.9253 vs 0.9254 and one or two flips out of ~320 in which gold session ranks first. The f32 run reproduces the published hybrid numbers exactly. The earlier float16 evidence was synthetic clustered data only; this is the real-embedding check. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-24 19:21:14 -05:00
osobhandClaude Opus 5.5	c470244a6f	feat(agent): HDF5Memory::search with source filters, re-ranking, confidence `HDF5Memory::search(query_embedding, query_text, &SearchOptions)` is the store's full search path. `SearchOptions::new(k)` is plain hybrid search with the tuned default fusion; each further stage is opt-in: - `with_sources([..])`: only records from these source channels. The filter applies before ranking, so a filtered search still returns up to k results, normalised over what it can return. The HNSW pool is over-fetched in proportion to what the filter removes, and the allowed records are scanned exactly whenever that costs fewer distance evaluations than the index would (~pool x M) — and as the fallback if the pool comes back short. Keyword matches are filtered too. - `with_rerank(ReRankConfig)` re-ranks a max(3k, 10) candidate pool by relevance, recency, source authority and activation; `with_confidence(ConfidenceConfig)` drops low-confidence results; `at_time(now)` pins the recency clock. These were reachable only through the OpenClaw backend, which is now `search` with both on. Its Hebbian boost now goes to the k results it returns rather than the whole 3k candidate pool. `hybrid_search` and `hybrid_search_with` are wrappers and unchanged (tested bit for bit). Measured on tank (search_harness --options-study --full, 3 runs): at 100K every filter — 50%, 10%, 1% of the store, and records far from the query — returns the exact filtered top 10, and none is slower than an unfiltered search (1%: 2.3 ms vs 4.6 ms). Re-rank + confidence costs about 3%. A first version decided between index and exact scan by pool size vs store size; it measured 0.976 recall at 12.3 ms on the far-from-query filter, which is why the rule compares costs instead. Tests: tests/search_options.rs (filter correctness and full pages via both paths, far-from-query fallback, edge cases, equality with hybrid_search_with, re-rank recency, confidence, boost scope). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-24 16:35:42 -05:00
osobhandClaude Opus 5.5	d0db83812b	feat(agent): MemoryConfig::float16 stores half-precision embeddings CI / test-arm64 (pull_request) Successful in 1m19s Details CI / test (pull_request) Successful in 4m58s Details The setting was persisted in /meta and otherwise ignored: embeddings were always written as f32. It now does what it says. clawhdf5-format: - `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16), rounding to nearest-even, and `make_f16_type`. - `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one implementation the writer, the reader and the agent all use. Checked against the `half` crate on 16.7M f32 values and round-trips all 65536 half values; the h5py interop tests confirm the rounding matches numpy's bit for bit (4020 values incl. ties, subnormals, overflow). - Reading little-endian float16 as f32 has a fast path. clawhdf5-agent: - A float16 store writes /memory/embeddings as half precision, and `MemoryCache::half_precision` rounds each embedding as it enters the cache (save, update, WAL replay, and on load of a store still f32 on disk), so memory and file agree bit for bit and a store searches the same before and after a reopen (tested). - Values beyond +-65504 are refused with the new `MemoryError::InvalidEntry` rather than stored as infinity, on every save path; batches are all or nothing, and a rejected ephemeral entry stays in the ephemeral tier. Breaking for exhaustive matches. - CLI: `create --float16`. Off by default. Measured on tank, 384-dim, six runs alternating order, medians (search_harness --float16-study --full): at 100K the file goes from 154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms; vector recall@10 against an exact scan and hybrid_search latency do not change. At 10K open is 3 ms slower. Also a test that h5py opens a whole agent store, f32 and float16, and decodes every dataset. Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG (including the h5py interop fixes in the previous commit), CLAUDE.md. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-24 12:00:38 -05:00
osobhandClaude Opus 5.5	5e4aa1c6bf	fix(format): files we write now open in h5py and libhdf5 Two write-side bugs, both present in every release (the first at least since v2.1.0), made libhdf5 refuse files written by clawhdf5. Our own reader ignores both fields, and the interop suites only ever wrote f64 from our side, so nothing here caught them. - Every f32 dataset: "sign bit position out of bounds". The float datatype encoder hard-coded the sign bit's position (bits 8-15 of the class bit field) to 63, which is right only for f64. It is now derived from the type: bit_offset + bit_precision - 1. This covered every agent store's embeddings, norms and activation weights. - Every empty dataset: "invalid dataset size, likely file corruption". It was written with a real address and size 0, which trips libhdf5's `addr + size <= addr` overflow check. An empty contiguous dataset now gets the undefined address, as libhdf5 writes it. This covered every agent store without sessions or a knowledge graph. Agent stores are rewritten in full at each checkpoint, so they become readable at their next checkpoint on a fixed build; other files with f32 or empty datasets need rewriting. Both are recorded in docs/known-issues.md. Tests: the sign position byte for f32/f64, and h5py reading our f32 datasets (plain and chunked + deflate) bit for bit. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-24 12:00:26 -05:00