bench: run the full longmemeval_s haystack, and measure the variant (Tier 4a)

The harness only ever ran longmemeval_oracle — evidence sessions only, which is
a substantially easier corpus than the dataset LongMemEval results are normally
quoted on. Worse, the variant was a hardcoded "oracle" string in both the report
header and the JSON summary, so pointing it at longmemeval_s would have produced
full-haystack numbers labelled oracle.

DatasetProfile now measures the corpus instead of asserting it: sessions and
turns per question, and evidence-session density (the mean share of a question's
haystack sessions that are answer sessions). The variant label and the
session-level degeneracy warning are both derived from that density, so a
mislabelled input file cannot produce a mislabelled result. Measured: 100.0%
density on the oracle variant, 4.0% on longmemeval_s.

The full haystack, all 500 questions, 47.7 sessions and 493.5 turns each:

                  turn-level   session-level
    Hit@1            53.8%         86.2%
    Hit@5            75.0%         93.6%
    Hit@10           81.6%         96.6%
    MRR             0.6320        0.8948

Turn-level drops 84.4% -> 75.0% against the oracle variant. That 9.4-point gap
is the price of the real haystack and is exactly why oracle-only numbers should
not be presented as LongMemEval results.

Session-level is now reportable. It was retracted before because at 100% evidence
density every returned document is a hit by construction; at 4.0% density a hit
reflects discrimination, so 93.6% is a real measurement rather than a restatement
of the corpus shape. Per-type it also finally separates: single-session-assistant
100.0% Hit@1 against single-session-preference 33.3% — BM25 has nothing to grip
on a preference question whose evidence shares no vocabulary with the query.

The MemX comparison stays withdrawn. Running the full haystack closes the corpus
half of that mismatch but not the granularity half: MemX measures fact-level over
220,349 records, and this harness measures turn- and session-level.

Two smaller fixes found while running it:

  - --limit samples evenly across the file rather than taking a prefix. The
    dataset is ordered by question type, so `--limit 20` returned 20
    single-session-user questions and nothing else while reading like a
    whole-dataset result.
  - abstention_accuracy emits null rather than 0.0 when a corpus poses no
    abstention questions. longmemeval_s has none, and 0.0000 reads as total
    failure at a task that was never asked.

README.md and BENCHMARKS.md now lead with the full-haystack numbers and keep the
oracle figures alongside, labelled as the easier corpus.

Verified on the Linux dev host: 49 groups / 1659 passed / 0 failed, clippy clean
under -D warnings, fmt clean. The full 500-question run takes ~70 s.

BENCHMARKS.md

@@ -207,8 +207,12 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).
 > which found that changing scoring target alone alters nDCG on 83–94% of queries and
 > can reverse system rankings, this section states its measurement contract explicitly:
 >
-> - **Dataset variant:** `longmemeval_oracle` — evidence sessions only, **not** the full
+> - **Dataset variant:** both are now reported below — the full `longmemeval_s`
->   `longmemeval_s`/`_m` haystack. This is a substantially easier corpus.
+>   haystack (**the headline number**) and `longmemeval_oracle` (evidence sessions
+>   only, a substantially easier corpus, kept for continuity). The harness does not
+>   trust the filename: it measures evidence-session density from the data and
+>   labels the run from that, so a mislabelled input cannot yield a mislabelled
+>   result. Measured density is 4.0% on `longmemeval_s` and 100.0% on the oracle.
 > - **Metric:** *retrieval recall.* A "hit" means the gold-labelled memory appeared in
 >   the top-k. **No answer is generated and none is scored** — the dataset's `answer`
 >   field is deserialized and never read. This is **not** the official LongMemEval
@@ -221,9 +225,35 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).
 
 **Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0`
 
-> **Run:** `cargo run --release --bin longmemeval_bench`
+> **Run:** `cargo run --release --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s_cleaned.json`
+> (~70 s for all 500 questions on the tank reference machine). Omit the path for the
+> oracle variant; add `--limit N` for an evenly-strided subsample.
 
-### Turn-Level Recall (n=500)
+### Full haystack — `longmemeval_s`, n=500 (the number to cite)
+
+47.7 sessions and 493.5 turns per question; 4.0% of haystack sessions are evidence
+sessions, so retrieval has to actually discriminate.
+
+| Metric | Turn-level | Session-level |
+|--------|-----------|---------------|
+| Hit@1 | 53.8% | 86.2% |
+| Hit@5 | **75.0%** | **93.6%** |
+| Hit@10 | 81.6% | 96.6% |
+| MRR | 0.6320 | 0.8948 |
+
+Session-level is reported here because on this corpus it is meaningful — unlike on
+the oracle variant, where it was degenerate and was retracted (below). At 4.0%
+evidence density a session-level hit reflects discrimination rather than corpus
+shape.
+
+Per-type, session-level: `single-session-assistant` 100.0% Hit@1 (n=56),
+`knowledge-update` 96.2% (n=78), `single-session-user` 94.3% (n=70),
+`multi-session` 84.2% (n=133), `temporal-reasoning` 84.2% (n=133), and
+`single-session-preference` 33.3% (n=30) — the one category where BM25 clearly
+struggles, since a preference question's evidence rarely shares vocabulary with
+the question.
+
+### Oracle variant — `longmemeval_oracle`, n=500 (easier corpus, kept for continuity)
 
 | Metric | ClawhDF5 (BM25-only, oracle variant) |
 |--------|--------------------------------------|
@@ -232,8 +262,10 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).
 | Hit@10 | 90.4% |
 | MRR | 0.6597 |
 
-This is the only number in this section that measures retrieval quality. See the
+Turn-level. The 9.4-point gap between this and the full haystack's 75.0% is the
-retracted results below for why the session-level figures do not.
+price of the harder corpus, and is the reason oracle-only numbers should not be
+presented as LongMemEval results. Session-level figures on this variant are
+degenerate — see below.
 
 ### Retracted: session-level recall and the MemX comparison
 
@@ -257,9 +289,11 @@ granularity, and a corpus smaller by orders of magnitude. A higher number on an
 easier corpus at a different granularity is not an outperformance claim, and it
 should not have been presented as one.
 
-A defensible comparison requires re-running against the full `longmemeval_s`
+The full-haystack half of that gap is now closed: the section above reports
-haystack at matched granularity. That work is tracked; until it lands, no
+`longmemeval_s` over all 500 questions. The **granularity** mismatch remains — MemX
-cross-system claim is made here.
+measures fact-level, we measure turn-level and session-level — so no cross-system
+claim is made here even now. Matching granularity would require fact-level
+extraction over the haystack, which this harness does not do.
 
 ### Search Latency (LongMemEval, n=500 queries)
 

README.md

@@ -101,27 +101,33 @@ Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now
 
 ### LongMemEval Retrieval Recall
 
-Evaluated against the **`longmemeval_oracle`** variant (500 questions, evidence
+Evaluated against the full **`longmemeval_s`** haystack — all 500 questions, 47.7
-sessions only — *not* the full `longmemeval_s` haystack). BM25-only baseline
+sessions and 493.5 turns each, with only 4.0% of haystack sessions being evidence
-(zero embeddings, so the vector stage is inert), reproduced identically on a
+sessions. BM25-only baseline (zero embeddings, so the vector stage is inert). See
-second machine (see [BENCHMARKS.md § LongMemEval
+[BENCHMARKS.md § LongMemEval Results](BENCHMARKS.md#longmemeval-results) for the
-Results](BENCHMARKS.md#longmemeval-results) for the full scoring-target
+full scoring-target declaration:
-declaration):
 
-| Metric | Turn-Level |
+| Metric | Turn-Level | Session-Level |
-|--------|------------|
+|--------|------------|---------------|
-| Hit@5 | **84.4%** |
+| Hit@5 | **75.0%** | **93.6%** |
-| MRR | 0.6597 |
+| MRR | 0.6320 | 0.8948 |
+
+On the easier `longmemeval_oracle` variant (evidence sessions only) the same
+harness scores 84.4% turn-level Hit@5 / MRR 0.6597, reproduced identically on a
+second machine. The 9.4-point gap is the cost of the real haystack, and is why the
+full-haystack number is the one quoted here.
 
 This is **retrieval recall** (did the gold memory appear in the top-k), not the
 official LongMemEval QA-accuracy metric — the two are not comparable, and
 retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 
 > **Previously reported here and now retracted:** session-level Hit@5 of 100.0% /
-> MRR 1.0000, and a claim of beating MemX's 51.6%. The session-level figures are
+> MRR 1.0000, and a claim of beating MemX's 51.6%. Those session-level figures were
 > degenerate on the oracle variant (any returned document is a hit by
-> construction), and the MemX comparison was not like-for-like — MemX measures
+> construction); the 93.6% above is a different, real measurement on a corpus where
-> fact-level granularity over 220,349 records. Details in
+> evidence sessions are 4.0% of the haystack. The MemX comparison stays withdrawn —
+> MemX measures fact-level granularity over 220,349 records, which running the full
+> haystack does not fix. Details in
 > [BENCHMARKS.md](BENCHMARKS.md#retracted-session-level-recall-and-the-memx-comparison).
 
 > Enable embeddings via `hybrid_search(query_emb, text, 0.7, 0.3, k)` for substantially higher recall. The vector stage is served by the HNSW index by default (the `hnsw` feature is on by default); build with `--no-default-features --features float16` to fall back to an exact linear cosine scan.

crates/clawhdf5-bench/src/bin/longmemeval_bench.rs

@@ -11,12 +11,18 @@
 //!   field is deserialized and deliberately never read. This is **not** the official
 //!   LongMemEval metric, which is end-to-end QA accuracy (retrieve → generate → LLM
 //!   judge). Reporting retrieval recall as QA accuracy overstates by 20–30 points.
-//! - **Dataset: `longmemeval_oracle`** — evidence sessions only, not the full
+//! - **Dataset: whichever variant you point it at.** Both `longmemeval_oracle`
-//!   `longmemeval_s`/`_m` haystack. Substantially easier corpus.
+//!   (evidence sessions only — a substantially easier corpus) and the full
-//! - **Session-level metrics are degenerate here** and must not be cited: on the
+//!   `longmemeval_s` haystack are supported. The harness does not trust the
-//!   oracle variant the haystack is essentially all-evidence, so any returned
+//!   filename: [`DatasetProfile`] measures evidence-session density from the
-//!   document is a session-level hit at rank 0 by construction. Only turn-level
+//!   data and labels the run from that, so a mislabelled input cannot produce a
-//!   (`has_answer == true` on the source turn) measures the retriever.
+//!   mislabelled result.
+//! - **Session-level metrics are degenerate when evidence density is high**, and
+//!   the report says so per run rather than assuming it. On the oracle variant
+//!   the haystack is essentially all-evidence, so any returned document is a
+//!   session-level hit at rank 0 by construction; only turn-level
+//!   (`has_answer == true` on the source turn) measures the retriever there. On
+//!   the full haystack, session-level recall is meaningful.
 //! - **Not comparable to MemX's Hit@5=51.6% / MRR=0.380**, which is *fact-level*
 //!   granularity over 220,349 records from 19,195 sessions.
 //!
@@ -24,7 +30,13 @@
 //!
 //! # Usage
 //! ```
-//! cargo run --release --bin longmemeval_bench [path/to/longmemeval_oracle.json]
+//! cargo run --release --bin longmemeval_bench [PATH] [--limit N]
+//!
+//! # Usage: full haystack
+//! ```
+//! cargo run --release --bin longmemeval_bench -- \
+//!     benchmarks/longmemeval/longmemeval_s_cleaned.json --limit 50
+//! ```
 //! ```
 //!
 //! # WASM Note
@@ -302,7 +314,77 @@ fn evaluate_question(q: &Question, top_k: usize) -> EvalResult {
 // Report printing
 // ---------------------------------------------------------------------------
 
-fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
+// ---------------------------------------------------------------------------
+// Dataset profile — measured, not assumed
+// ---------------------------------------------------------------------------
+
+/// Shape of the loaded corpus, computed from the data itself.
+///
+/// The variant used to be a hardcoded `"oracle"` string in the report and the
+/// JSON summary, so pointing the harness at `longmemeval_s` would have produced
+/// full-haystack numbers labelled oracle. Everything here is derived from the
+/// questions instead, which means the label cannot drift from the corpus and a
+/// mislabelled input file cannot produce a mislabelled result.
+struct DatasetProfile {
+    n_questions: usize,
+    mean_sessions: f64,
+    mean_turns: f64,
+    /// Mean over questions of `|answer_sessions| / |haystack_sessions|`.
+    ///
+    /// This is what actually decides whether session-level recall means
+    /// anything. At ~1.0 every haystack session is an evidence session, so any
+    /// returned document is a session-level hit by construction.
+    evidence_density: f64,
+}
+
+impl DatasetProfile {
+    fn measure(questions: &[Question]) -> Self {
+        let n = questions.len().max(1) as f64;
+        let mut sessions = 0.0;
+        let mut turns = 0.0;
+        let mut density = 0.0;
+        for q in questions {
+            let n_sess = q.haystack_sessions.len();
+            sessions += n_sess as f64;
+            turns += q.haystack_sessions.iter().map(Vec::len).sum::<usize>() as f64;
+            if n_sess > 0 {
+                let evidence: HashSet<&str> =
+                    q.answer_session_ids.iter().map(String::as_str).collect();
+                let hit = q
+                    .haystack_session_ids
+                    .iter()
+                    .filter(|id| evidence.contains(id.as_str()))
+                    .count();
+                density += hit as f64 / n_sess as f64;
+            }
+        }
+        Self {
+            n_questions: questions.len(),
+            mean_sessions: sessions / n,
+            mean_turns: turns / n,
+            evidence_density: density / n,
+        }
+    }
+
+    /// Above this share of evidence sessions, session-level recall is measuring
+    /// the corpus shape rather than the retriever.
+    const DEGENERACY_THRESHOLD: f64 = 0.9;
+
+    const fn session_level_degenerate(&self) -> bool {
+        self.evidence_density > Self::DEGENERACY_THRESHOLD
+    }
+
+    /// Variant name inferred from evidence density, not from the filename.
+    const fn variant(&self) -> &'static str {
+        if self.session_level_degenerate() {
+            "oracle"
+        } else {
+            "full_haystack"
+        }
+    }
+}
+
+fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>, profile: &DatasetProfile) {
     println!("=================================================================");
     println!("  LongMemEval Benchmark  (BM25-only retrieval, zero embeddings)");
     println!("=================================================================");
@@ -312,8 +394,24 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
     println!("Scoring target: RETRIEVAL RECALL (did the gold memory land in top-k).");
     println!("  No answer is generated or scored. This is NOT the official");
     println!("  LongMemEval metric (QA accuracy via retrieve+generate+judge).");
-    println!("Dataset: longmemeval_oracle — evidence sessions only, NOT the full");
+    println!(
-    println!("  longmemeval_s haystack. This is a substantially easier corpus.");
+        "Dataset: {} — {} questions, {:.1} sessions and {:.0} turns per question,",
+        profile.variant(),
+        profile.n_questions,
+        profile.mean_sessions,
+        profile.mean_turns,
+    );
+    println!(
+        "  {:.1}% of haystack sessions are evidence sessions.",
+        profile.evidence_density * 100.0
+    );
+    if profile.session_level_degenerate() {
+        println!("  This is the evidence-only corpus, NOT the full longmemeval_s");
+        println!("  haystack — a substantially easier retrieval problem.");
+    } else {
+        println!("  This is a full-haystack corpus: evidence sessions are a small");
+        println!("  minority, so retrieval has to actually discriminate.");
+    }
     println!();
     println!("Do NOT compare these to MemX's Hit@5=51.6% / MRR=0.380: that is");
     println!("  fact-level granularity over 220,349 records from 19,195 sessions.");
@@ -321,9 +419,20 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
     println!();
 
     println!("## Session-Level Recall  (n={})", overall.count);
-    println!("  [DEGENERATE on the oracle variant — every returned document belongs");
+    if profile.session_level_degenerate() {
-    println!("   to an answer session by construction. Reported for completeness only;");
+        println!(
-    println!("   this measures the corpus shape, not the retriever. Use turn-level.]");
+            "  [DEGENERATE — {:.1}% of haystack sessions are evidence sessions, so a",
+            profile.evidence_density * 100.0
+        );
+        println!("   returned document is a session-level hit almost by construction.");
+        println!("   This measures the corpus shape, not the retriever. Use turn-level.]");
+    } else {
+        println!(
+            "  [Meaningful on this corpus — only {:.1}% of haystack sessions are",
+            profile.evidence_density * 100.0
+        );
+        println!("   evidence sessions, so a hit reflects the retriever's discrimination.]");
+    }
     println!(
         "  Hit@1:  {:5.1}%   Hit@5:  {:5.1}%   Hit@10: {:5.1}%   MRR: {:.4}",
         overall.hit1_session_pct(),
@@ -408,10 +517,22 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
     println!("{{");
     println!("  \"benchmark\": \"longmemeval\",");
     println!("  \"mode\": \"bm25_only\",");
-    println!("  \"dataset_variant\": \"oracle\",");
+    println!("  \"dataset_variant\": \"{}\",", profile.variant());
     println!("  \"scoring_target\": \"retrieval_recall\",");
     println!("  \"k\": 10,");
-    println!("  \"session_level_degenerate\": true,");
+    println!(
+        "  \"session_level_degenerate\": {},",
+        profile.session_level_degenerate()
+    );
+    println!(
+        "  \"evidence_session_density\": {:.4},",
+        profile.evidence_density
+    );
+    println!(
+        "  \"mean_sessions_per_question\": {:.2},",
+        profile.mean_sessions
+    );
+    println!("  \"mean_turns_per_question\": {:.1},", profile.mean_turns);
     println!(
         "  \"total_questions\": {},",
         overall.count + overall.abstention_total
@@ -434,10 +555,16 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
         overall.mrr_turn()
     );
     println!("  }},");
+    // `null`, not 0.0 — a corpus with no abstention questions has no abstention
+    // accuracy, and emitting 0.0 reads as total failure at a task never posed.
+    if overall.abstention_total > 0 {
         println!(
             "  \"abstention_accuracy\": {:.4},",
             overall.abstention_pct() / 100.0
         );
+    } else {
+        println!("  \"abstention_accuracy\": null,");
+    }
     println!("  \"latency_us\": {{");
     println!(
         "    \"avg\": {:.1}, \"p50\": {:.1}, \"p95\": {:.1}, \"p99\": {:.1}",
@@ -456,17 +583,73 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
 // ---------------------------------------------------------------------------
 
 fn main() {
-    let json_path = std::env::args()
+    let mut json_path: Option<String> = None;
-        .nth(1)
+    let mut limit: Option<usize> = None;
-        .unwrap_or_else(|| "benchmarks/longmemeval/longmemeval_oracle.json".to_string());
+    let mut args = std::env::args().skip(1);
+    while let Some(arg) = args.next() {
+        match arg.as_str() {
+            "--limit" => {
+                let v = args.next().expect("--limit needs a value");
+                limit = Some(v.parse().expect("--limit must be a positive integer"));
+            }
+            "--help" | "-h" => {
+                eprintln!(
+                    "usage: longmemeval_bench [PATH] [--limit N]\n\n\
+                     PATH     dataset JSON; defaults to the oracle variant.\n\
+                              longmemeval_s works too — the harness measures which\n\
+                              variant it was given rather than trusting the filename.\n\
+                     --limit  evaluate N questions, sampled evenly across the file\n\
+                              rather than as a prefix — the dataset is ordered by\n\
+                              question type, so a prefix samples one type only."
+                );
+                return;
+            }
+            other => json_path = Some(other.to_string()),
+        }
+    }
+    let json_path =
+        json_path.unwrap_or_else(|| "benchmarks/longmemeval/longmemeval_oracle.json".to_string());
 
     eprintln!("Loading: {json_path}");
     let data = std::fs::read_to_string(&json_path)
         .unwrap_or_else(|e| panic!("Failed to read {json_path}: {e}"));
-    let questions: Vec<Question> = serde_json::from_str(&data).expect("Failed to parse JSON");
+    let mut questions: Vec<Question> = serde_json::from_str(&data).expect("Failed to parse JSON");
+    if let Some(n) = limit
+        && n < questions.len()
+    {
+        // Stride rather than truncate. The dataset is ordered by question type,
+        // so taking a prefix samples one type: `--limit 20` on longmemeval_s
+        // returns 20 `single-session-user` questions and nothing else, which
+        // reads as a whole-dataset result but is not one.
+        let total = questions.len();
+        let step = total as f64 / n as f64;
+        let keep: HashSet<usize> = (0..n)
+            .map(|i| ((i as f64 * step) as usize).min(total - 1))
+            .collect();
+        questions = questions
+            .into_iter()
+            .enumerate()
+            .filter(|(i, _)| keep.contains(i))
+            .map(|(_, q)| q)
+            .collect();
+        eprintln!(
+            "Sampling {} of {total} questions, evenly strided (--limit)",
+            questions.len()
+        );
+    }
     let total = questions.len();
     eprintln!("Loaded {total} questions");
 
+    let profile = DatasetProfile::measure(&questions);
+    eprintln!(
+        "Corpus: {} variant — {:.1} sessions / {:.0} turns per question, \
+         {:.1}% evidence-session density",
+        profile.variant(),
+        profile.mean_sessions,
+        profile.mean_turns,
+        profile.evidence_density * 100.0,
+    );
+
     let mut overall = Metrics::default();
     let mut by_type: HashMap<String, Metrics> = HashMap::new();
 
@@ -540,5 +723,5 @@ fn main() {
 
     eprintln!("\r  [{total}/{total}] done.          ");
     eprintln!();
-    print_report(&overall, &by_type);
+    print_report(&overall, &by_type, &profile);
 }