docs: retract degenerate LongMemEval session-level numbers and the MemX comparison
CI / test (push) Failing after 4s

A methodology audit found that two benchmark claims published in this repo two
days ago measure the wrong thing. Both are retracted in place rather than
quietly edited, with the reasoning recorded.

1. Session-level LongMemEval recall (100.0% Hit@1/5/10, MRR 1.0000, uniform
   across all six question types) is a degenerate artifact. On the
   longmemeval_oracle variant the ingested haystack for a question is
   essentially only that question's evidence sessions, so every returned
   document belongs to an answer session and session-level hit rate is ~1.0 at
   rank 0 by construction. The uniform 100% across every question type was the
   tell. It measured the shape of the corpus, not the retriever. Only the
   turn-level figure (84.4% Hit@5) carries signal, and it is now the only
   retrieval number cited.

2. The "clawhdf5 outperforms MemX at turn-level retrieval (84.4% vs 51.6%)"
   claim was not like-for-like on two independent axes. Confirmed against
   arxiv:2603.16171: MemX's Hit@5=51.6% / MRR=0.380 is *fact-level*
   granularity over 220,349 fact-level records drawn from 19,195 sessions, and
   the paper explicitly notes fact-level "doubl[es] session-level performance".
   Ours is turn-level on the oracle subset — different granularity, and a
   corpus smaller by orders of magnitude. A higher number on an easier corpus
   at a different granularity is not an outperformance claim.

Also caveats the vector-search "vs MemX" latency ratios, which compare a single
clawhdf5 component (raw vector search) against MemX's end-to-end pipeline
figure (embeddings + FTS5 + four-factor re-ranking). The numbers are real; the
"speedup" framing overstated by an unquantified margin and is now labelled an
order-of-magnitude indication.

Adds an explicit scoring-target declaration to BENCHMARKS.md per arXiv
2605.24060, which found that changing scoring target alone alters nDCG on
83-94% of queries and can reverse system rankings. States dataset variant,
metric (retrieval recall, NOT the official QA-accuracy metric), granularity,
k, and that the vector stage is inert (zero embeddings, vector_weight=0.0).

The harness itself now prints its scoring target, flags the session-level
block as degenerate, warns against the MemX comparison, and emits
dataset_variant/scoring_target/k/session_level_degenerate in its JSON summary,
so the caveats travel with the numbers instead of living only in docs.
This commit is contained in:
Omar Sobh
2026-08-06 16:43:57 -07:00
parent dfae9e2cc1
commit 6f5940d042
3 changed files with 146 additions and 60 deletions
@@ -4,7 +4,23 @@
//! Since no embedding model is available at bench time, all embeddings are zero vectors
//! and `hybrid_search` operates in BM25-only mode (vector_weight=0.0, keyword_weight=1.0).
//!
//! This matches the MemX paper methodology: evaluate retrieval recall, not answer generation.
//! # Scoring target (read before citing any number from this harness)
//!
//! - **Metric: retrieval recall.** A "hit" means the gold-labelled memory appeared in
//! the top-k. No answer is generated and none is scored — the dataset's `answer`
//! field is deserialized and deliberately never read. This is **not** the official
//! LongMemEval metric, which is end-to-end QA accuracy (retrieve → generate → LLM
//! judge). Reporting retrieval recall as QA accuracy overstates by 2030 points.
//! - **Dataset: `longmemeval_oracle`** — evidence sessions only, not the full
//! `longmemeval_s`/`_m` haystack. Substantially easier corpus.
//! - **Session-level metrics are degenerate here** and must not be cited: on the
//! oracle variant the haystack is essentially all-evidence, so any returned
//! document is a session-level hit at rank 0 by construction. Only turn-level
//! (`has_answer == true` on the source turn) measures the retriever.
//! - **Not comparable to MemX's Hit@5=51.6% / MRR=0.380**, which is *fact-level*
//! granularity over 220,349 records from 19,195 sessions.
//!
//! See `BENCHMARKS.md` § "Retracted: session-level recall and the MemX comparison".
//!
//! # Usage
//! ```
@@ -292,11 +308,22 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
println!("=================================================================");
println!();
println!("Mode: vector_weight=0.0 / keyword_weight=1.0 (pure BM25)");
println!("Note: MemX (arxiv:2603.16171) with full system: Hit@5=51.6%, MRR=0.380");
println!(" BM25-only numbers are expected to be lower — honest baseline.");
println!();
println!("Scoring target: RETRIEVAL RECALL (did the gold memory land in top-k).");
println!(" No answer is generated or scored. This is NOT the official");
println!(" LongMemEval metric (QA accuracy via retrieve+generate+judge).");
println!("Dataset: longmemeval_oracle — evidence sessions only, NOT the full");
println!(" longmemeval_s haystack. This is a substantially easier corpus.");
println!();
println!("Do NOT compare these to MemX's Hit@5=51.6% / MRR=0.380: that is");
println!(" fact-level granularity over 220,349 records from 19,195 sessions.");
println!(" Different granularity and a corpus larger by orders of magnitude.");
println!();
println!("## Session-Level Recall (n={})", overall.count);
println!(" [DEGENERATE on the oracle variant — every returned document belongs");
println!(" to an answer session by construction. Reported for completeness only;");
println!(" this measures the corpus shape, not the retriever. Use turn-level.]");
println!(
" Hit@1: {:5.1}% Hit@5: {:5.1}% Hit@10: {:5.1}% MRR: {:.4}",
overall.hit1_session_pct(),
@@ -381,6 +408,10 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
println!("{{");
println!(" \"benchmark\": \"longmemeval\",");
println!(" \"mode\": \"bm25_only\",");
println!(" \"dataset_variant\": \"oracle\",");
println!(" \"scoring_target\": \"retrieval_recall\",");
println!(" \"k\": 10,");
println!(" \"session_level_degenerate\": true,");
println!(
" \"total_questions\": {},",
overall.count + overall.abstention_total