docs: retract degenerate LongMemEval session-level numbers and the MemX comparison
CI / test (push) Failing after 4s
CI / test (push) Failing after 4s
A methodology audit found that two benchmark claims published in this repo two days ago measure the wrong thing. Both are retracted in place rather than quietly edited, with the reasoning recorded. 1. Session-level LongMemEval recall (100.0% Hit@1/5/10, MRR 1.0000, uniform across all six question types) is a degenerate artifact. On the longmemeval_oracle variant the ingested haystack for a question is essentially only that question's evidence sessions, so every returned document belongs to an answer session and session-level hit rate is ~1.0 at rank 0 by construction. The uniform 100% across every question type was the tell. It measured the shape of the corpus, not the retriever. Only the turn-level figure (84.4% Hit@5) carries signal, and it is now the only retrieval number cited. 2. The "clawhdf5 outperforms MemX at turn-level retrieval (84.4% vs 51.6%)" claim was not like-for-like on two independent axes. Confirmed against arxiv:2603.16171: MemX's Hit@5=51.6% / MRR=0.380 is *fact-level* granularity over 220,349 fact-level records drawn from 19,195 sessions, and the paper explicitly notes fact-level "doubl[es] session-level performance". Ours is turn-level on the oracle subset — different granularity, and a corpus smaller by orders of magnitude. A higher number on an easier corpus at a different granularity is not an outperformance claim. Also caveats the vector-search "vs MemX" latency ratios, which compare a single clawhdf5 component (raw vector search) against MemX's end-to-end pipeline figure (embeddings + FTS5 + four-factor re-ranking). The numbers are real; the "speedup" framing overstated by an unquantified margin and is now labelled an order-of-magnitude indication. Adds an explicit scoring-target declaration to BENCHMARKS.md per arXiv 2605.24060, which found that changing scoring target alone alters nDCG on 83-94% of queries and can reverse system rankings. States dataset variant, metric (retrieval recall, NOT the official QA-accuracy metric), granularity, k, and that the vector stage is inert (zero embeddings, vector_weight=0.0). The harness itself now prints its scoring target, flags the session-level block as degenerate, warns against the MemX comparison, and emits dataset_variant/scoring_target/k/session_level_degenerate in its JSON summary, so the caveats travel with the numbers instead of living only in docs.
This commit is contained in:
@@ -4,7 +4,23 @@
|
||||
//! Since no embedding model is available at bench time, all embeddings are zero vectors
|
||||
//! and `hybrid_search` operates in BM25-only mode (vector_weight=0.0, keyword_weight=1.0).
|
||||
//!
|
||||
//! This matches the MemX paper methodology: evaluate retrieval recall, not answer generation.
|
||||
//! # Scoring target (read before citing any number from this harness)
|
||||
//!
|
||||
//! - **Metric: retrieval recall.** A "hit" means the gold-labelled memory appeared in
|
||||
//! the top-k. No answer is generated and none is scored — the dataset's `answer`
|
||||
//! field is deserialized and deliberately never read. This is **not** the official
|
||||
//! LongMemEval metric, which is end-to-end QA accuracy (retrieve → generate → LLM
|
||||
//! judge). Reporting retrieval recall as QA accuracy overstates by 20–30 points.
|
||||
//! - **Dataset: `longmemeval_oracle`** — evidence sessions only, not the full
|
||||
//! `longmemeval_s`/`_m` haystack. Substantially easier corpus.
|
||||
//! - **Session-level metrics are degenerate here** and must not be cited: on the
|
||||
//! oracle variant the haystack is essentially all-evidence, so any returned
|
||||
//! document is a session-level hit at rank 0 by construction. Only turn-level
|
||||
//! (`has_answer == true` on the source turn) measures the retriever.
|
||||
//! - **Not comparable to MemX's Hit@5=51.6% / MRR=0.380**, which is *fact-level*
|
||||
//! granularity over 220,349 records from 19,195 sessions.
|
||||
//!
|
||||
//! See `BENCHMARKS.md` § "Retracted: session-level recall and the MemX comparison".
|
||||
//!
|
||||
//! # Usage
|
||||
//! ```
|
||||
@@ -292,11 +308,22 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
|
||||
println!("=================================================================");
|
||||
println!();
|
||||
println!("Mode: vector_weight=0.0 / keyword_weight=1.0 (pure BM25)");
|
||||
println!("Note: MemX (arxiv:2603.16171) with full system: Hit@5=51.6%, MRR=0.380");
|
||||
println!(" BM25-only numbers are expected to be lower — honest baseline.");
|
||||
println!();
|
||||
println!("Scoring target: RETRIEVAL RECALL (did the gold memory land in top-k).");
|
||||
println!(" No answer is generated or scored. This is NOT the official");
|
||||
println!(" LongMemEval metric (QA accuracy via retrieve+generate+judge).");
|
||||
println!("Dataset: longmemeval_oracle — evidence sessions only, NOT the full");
|
||||
println!(" longmemeval_s haystack. This is a substantially easier corpus.");
|
||||
println!();
|
||||
println!("Do NOT compare these to MemX's Hit@5=51.6% / MRR=0.380: that is");
|
||||
println!(" fact-level granularity over 220,349 records from 19,195 sessions.");
|
||||
println!(" Different granularity and a corpus larger by orders of magnitude.");
|
||||
println!();
|
||||
|
||||
println!("## Session-Level Recall (n={})", overall.count);
|
||||
println!(" [DEGENERATE on the oracle variant — every returned document belongs");
|
||||
println!(" to an answer session by construction. Reported for completeness only;");
|
||||
println!(" this measures the corpus shape, not the retriever. Use turn-level.]");
|
||||
println!(
|
||||
" Hit@1: {:5.1}% Hit@5: {:5.1}% Hit@10: {:5.1}% MRR: {:.4}",
|
||||
overall.hit1_session_pct(),
|
||||
@@ -381,6 +408,10 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
|
||||
println!("{{");
|
||||
println!(" \"benchmark\": \"longmemeval\",");
|
||||
println!(" \"mode\": \"bm25_only\",");
|
||||
println!(" \"dataset_variant\": \"oracle\",");
|
||||
println!(" \"scoring_target\": \"retrieval_recall\",");
|
||||
println!(" \"k\": 10,");
|
||||
println!(" \"session_level_degenerate\": true,");
|
||||
println!(
|
||||
" \"total_questions\": {},",
|
||||
overall.count + overall.abstention_total
|
||||
|
||||
Reference in New Issue
Block a user