wip-4b4

BENCHMARKS.md

@@ -220,8 +220,11 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).
 >   judge). Retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 > - **Granularity:** turn-level = the returned memory's source turn had `has_answer == true`.
 > - **k = 10**, n = 500.
-> - **Retrieval mode:** BM25 only. The bench passes zero-vector embeddings with
+> - **Retrieval mode:** all three are reported below. Historically the bench passed
->   `vector_weight=0.0`, so the HNSW/vector stage is inert and contributes nothing.
+>   zero-vector embeddings with `vector_weight=0.0`, so the HNSW/vector stage was
+>   inert and every published number was BM25 alone. Real `all-MiniLM-L6-v2`
+>   embeddings are now available via `--features embeddings --embeddings <dir>`,
+>   and BM25-only / vector-only / hybrid are each measured separately.
 
 **Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0`
 
@@ -253,6 +256,53 @@ Per-type, session-level: `single-session-assistant` 100.0% Hit@1 (n=56),
 struggles, since a preference question's evidence rarely shares vocabulary with
 the question.
 
+### Retrieval mode ablation — full haystack, n=500
+
+Real 384-d `all-MiniLM-L6-v2` embeddings, 190,015 unique texts encoded once on an
+RTX 5060 Ti (~13 min; the same work on the 8-core CPU was still unfinished after
+30 minutes, so the GPU path is not a convenience here). Turn-level:
+
+| Mode | Hit@1 | Hit@5 | Hit@10 | MRR |
+|------|-------|-------|--------|-----|
+| BM25 only (`0.0`/`1.0`) | **53.8%** | 75.0% | 81.6% | **0.6320** |
+| Vector only (`1.0`/`0.0`) | 36.0% | 71.8% | 81.6% | 0.5027 |
+| Hybrid (`0.7`/`0.3`) | 44.4% | **79.2%** | **86.0%** | 0.5868 |
+
+Session-level:
+
+| Mode | Hit@1 | Hit@5 | Hit@10 | MRR |
+|------|-------|-------|--------|-----|
+| BM25 only | 86.2% | 93.6% | 96.6% | 0.8948 |
+| Vector only | 85.4% | 94.2% | 96.6% | 0.8901 |
+| Hybrid | **88.2%** | **95.8%** | **97.8%** | **0.9158** |
+
+**The hybrid claim holds for depth, not for precision@1.** Hybrid is the best
+configuration at Hit@5 and Hit@10 at both granularities — turn-level Hit@5 rises
+4.2 points over BM25 and 7.4 over vector-only, which is the result that justifies
+running two stages. But BM25 alone remains the best turn-level Hit@1 (53.8% vs
+44.4%) and MRR (0.6320 vs 0.5868). Fusing at `0.7/0.3` buys deeper recall and
+pays for it at rank 1.
+
+That trade matters for how the index is consumed: a caller assembling five
+memories of context should use hybrid, and a caller taking the single top hit is
+better served by BM25 today. The weights are not tuned — `0.7/0.3` is the
+documented default, not a searched optimum, and a Hit@1-oriented deployment
+should sweep them.
+
+The same pattern shows up independently in omni-cortex's four-signal RRF ablation,
+where adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR.
+Two different codebases, two different fusion schemes, same direction.
+
+Vector-only being *worse* than BM25 at every turn-level cutoff except Hit@10 is
+worth stating plainly rather than hiding: LongMemEval questions share substantial
+vocabulary with their evidence turns, which is close to the best case for lexical
+matching, and MiniLM at 384 dimensions is a small embedding model.
+
+> **Run:** `cargo run --release --bin longmemeval_bench --features embeddings -- \
+>   benchmarks/longmemeval/longmemeval_s_cleaned.json --embeddings weights/all-minilm-l6-v2`
+> Add `--features embeddings-cuda` (and put `nvcc` on `PATH`) for the GPU path.
+> Weights: `huggingface.co/sentence-transformers/all-MiniLM-L6-v2`.
+
 ### Oracle variant — `longmemeval_oracle`, n=500 (easier corpus, kept for continuity)
 
 | Metric | ClawhDF5 (BM25-only, oracle variant) |
@@ -267,6 +317,12 @@ price of the harder corpus, and is the reason oracle-only numbers should not be
 presented as LongMemEval results. Session-level figures on this variant are
 degenerate — see below.
 
+With real embeddings the same oracle corpus gives BM25-only 84.2% / vector-only
+80.4% / hybrid **85.2%** Hit@5 turn-level — hybrid ahead at Hit@5 and Hit@10 and
+behind at Hit@1, matching the full-haystack pattern above. (BM25-only reads 84.2%
+here against 84.4% with zero embedding vectors: one question of 500 changes rank,
+with MRR identical at 0.6597. On the full haystack the two agree exactly.)
+
 ### Retracted: session-level recall and the MemX comparison
 
 Earlier revisions of this file reported session-level Hit@1/5/10 of **100.0%** with

README.md

@@ -103,14 +103,25 @@ Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now
 
 Evaluated against the full **`longmemeval_s`** haystack — all 500 questions, 47.7
 sessions and 493.5 turns each, with only 4.0% of haystack sessions being evidence
-sessions. BM25-only baseline (zero embeddings, so the vector stage is inert). See
+sessions. See [BENCHMARKS.md § LongMemEval
-[BENCHMARKS.md § LongMemEval Results](BENCHMARKS.md#longmemeval-results) for the
+Results](BENCHMARKS.md#longmemeval-results) for the full scoring-target
-full scoring-target declaration:
+declaration:
 
-| Metric | Turn-Level | Session-Level |
+| Mode | Turn-Level Hit@5 | Session-Level Hit@5 |
-|--------|------------|---------------|
+|------|------------------|---------------------|
-| Hit@5 | **75.0%** | **93.6%** |
+| BM25 only | 75.0% | 93.6% |
-| MRR | 0.6320 | 0.8948 |
+| Vector only (MiniLM) | 71.8% | 94.2% |
+| Hybrid (0.7/0.3) | **79.2%** | **95.8%** |
+
+Hybrid is the strongest configuration at Hit@5 and Hit@10, which is what running
+two retrieval stages is for. It is *not* strongest at rank 1 — BM25 alone leads
+turn-level Hit@1 (53.8% vs 44.4%) and MRR (0.6320 vs 0.5868), so a caller taking
+a single top hit is better served by BM25 today. The `0.7/0.3` weights are the
+documented default, not a searched optimum.
+
+Vector embeddings require `--features embeddings`; without it the vector stage is
+inert and only the BM25 row is produced, which is what every previously published
+number here measured.
 
 On the easier `longmemeval_oracle` variant (evidence sessions only) the same
 harness scores 84.4% turn-level Hit@5 / MRR 0.6597, reproduced identically on a