docs: retract degenerate LongMemEval session-level numbers and the MemX comparison

A methodology audit found that two benchmark claims published in this repo two days ago measure the wrong thing. Both are retracted in place rather than quietly edited, with the reasoning recorded. 1. Session-level LongMemEval recall (100.0% Hit@1/5/10, MRR 1.0000, uniform across all six question types) is a degenerate artifact. On the longmemeval_oracle variant the ingested haystack for a question is essentially only that question's evidence sessions, so every returned document belongs to an answer session and session-level hit rate is ~1.0 at rank 0 by construction. The uniform 100% across every question type was the tell. It measured the shape of the corpus, not the retriever. Only the turn-level figure (84.4% Hit@5) carries signal, and it is now the only retrieval number cited. 2. The "clawhdf5 outperforms MemX at turn-level retrieval (84.4% vs 51.6%)" claim was not like-for-like on two independent axes. Confirmed against arxiv:2603.16171: MemX's Hit@5=51.6% / MRR=0.380 is *fact-level* granularity over 220,349 fact-level records drawn from 19,195 sessions, and the paper explicitly notes fact-level "doubl[es] session-level performance". Ours is turn-level on the oracle subset — different granularity, and a corpus smaller by orders of magnitude. A higher number on an easier corpus at a different granularity is not an outperformance claim. Also caveats the vector-search "vs MemX" latency ratios, which compare a single clawhdf5 component (raw vector search) against MemX's end-to-end pipeline figure (embeddings + FTS5 + four-factor re-ranking). The numbers are real; the "speedup" framing overstated by an unquantified margin and is now labelled an order-of-magnitude indication. Adds an explicit scoring-target declaration to BENCHMARKS.md per arXiv 2605.24060, which found that changing scoring target alone alters nDCG on 83-94% of queries and can reverse system rankings. States dataset variant, metric (retrieval recall, NOT the official QA-accuracy metric), granularity, k, and that the vector stage is inert (zero embeddings, vector_weight=0.0). The harness itself now prints its scoring target, flags the session-level block as degenerate, warns against the MemX comparison, and emits dataset_variant/scoring_target/k/session_level_degenerate in its JSON summary, so the caveats travel with the numbers instead of living only in docs.
2026-08-06 16:43:57 -07:00
parent dfae9e2cc1
commit 6f5940d042
3 changed files with 146 additions and 60 deletions
@@ -16,6 +16,15 @@
 > Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint,
 > Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is
 > a known, tracked documentation gap, not a claim that those numbers are wrong.
 >
 > **Correctness note (2026-08-06).** Being dated and reproducible is necessary but
 > not sufficient — a number can be perfectly reproducible and still measure the
 > wrong thing. A methodology audit found two such cases and both have been
 > retracted in place: the session-level LongMemEval figures (degenerate on the
 > oracle variant) and the MemX retrieval comparison (mismatched granularity and
 > corpus). Every cross-system comparison in this file now carries an explicit
 > scoping caveat. Where a section states a scoring target, that declaration is the
 > contract — read it before citing the number.
 ---
@@ -35,10 +44,19 @@ Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embed
 MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).
-| Metric | MemX (claimed) | ClawhDF5 | Speedup |
+> **Caveat — not like-for-like.** MemX's `<90 ms` is *end-to-end* search across their
-|--------|----------------|----------|---------|
+> full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5
-| 100K flat search | <90 ms | 11.4 ms | **~8x** |
+> figures below are a *single component* — raw vector search latency, excluding
-| 100K IVF-PQ search | — | 1.19 ms | **~76x** |
+> embedding, keyword, fusion, and re-ranking stages. A component measured against a
 > full pipeline will always look favourable; the "speedup" column overstates the real
 > advantage by an unquantified margin and should be read as an order-of-magnitude
 > indication only, not a benchmark result. Matching MemX's measurement boundary is
 > tracked as follow-up work.
 | Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | Ratio |
 |--------|----------------------------|---------------------------|-------|
 | 100K flat search | <90 ms | 11.4 ms | ~8x |
 | 100K IVF-PQ search | — | 1.19 ms | ~76x |
 | Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable |
 ---
@@ -185,46 +203,63 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).
 ## LongMemEval Results
-**Dataset:** LongMemEval oracle (500 questions, 6 question types, variable-length chat histories)
+> **Scoring target declaration.** Per [arXiv 2605.24060](https://arxiv.org/abs/2605.24060),
 > which found that changing scoring target alone alters nDCG on 83–94% of queries and
 > can reverse system rankings, this section states its measurement contract explicitly:
 >
 > - **Dataset variant:** `longmemeval_oracle` — evidence sessions only, **not** the full
 >   `longmemeval_s`/`_m` haystack. This is a substantially easier corpus.
 > - **Metric:** *retrieval recall.* A "hit" means the gold-labelled memory appeared in
 >   the top-k. **No answer is generated and none is scored** — the dataset's `answer`
 >   field is deserialized and never read. This is **not** the official LongMemEval
 >   leaderboard metric, which is end-to-end QA accuracy (retrieve → generate → LLM
 >   judge). Retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 > - **Granularity:** turn-level = the returned memory's source turn had `has_answer == true`.
 > - **k = 10**, n = 500.
 > - **Retrieval mode:** BM25 only. The bench passes zero-vector embeddings with
 >   `vector_weight=0.0`, so the HNSW/vector stage is inert and contributes nothing.
 **Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0`
 **Reference:** MemX (arxiv:2603.16171) with full embedding system: Hit@5=51.6%, MRR=0.380
 > **Run:** `cargo run --release --bin longmemeval_bench`
 ### Session-Level Recall (n=500)
 | Metric | ClawhDF5 (BM25-only) |
 |--------|---------------------|
 | Hit@1 | **100.0%** |
 | Hit@5 | **100.0%** |
 | Hit@10 | **100.0%** |
 | MRR | **1.0000** |
 Perfect session-level recall across all 500 questions and all 6 question types.
 ### Turn-Level Recall (n=500)
-| Metric | ClawhDF5 (BM25-only) | MemX (full system)¹ |
+| Metric | ClawhDF5 (BM25-only, oracle variant) |
-|--------|---------------------|---------------------|
+|--------|--------------------------------------|
-| Hit@1 | **52.6%** | — |
+| Hit@1 | 52.6% |
-| Hit@5 | **84.4%** | 51.6% |
+| Hit@5 | **84.4%** |
-| Hit@10 | **90.4%** | — |
+| Hit@10 | 90.4% |
-| MRR | **0.6597** | 0.380 |
+| MRR | 0.6597 |
-**clawhdf5 outperforms MemX at turn-level retrieval** — Hit@5 84.4% vs 51.6%, MRR 0.66 vs 0.38 — with BM25 alone, no embeddings needed.
+This is the only number in this section that measures retrieval quality. See the
 retracted results below for why the session-level figures do not.
-> ¹ MemX uses dense embeddings + FTS5 + four-factor re-ranking. Our BM25-only result exceeds their full pipeline.
+### Retracted: session-level recall and the MemX comparison
-### Per-Type Breakdown (session-level)
+Earlier revisions of this file reported session-level Hit@1/5/10 of **100.0%** with
 MRR **1.0000**, uniform across all six question types, and claimed clawhdf5
 "outperforms MemX at turn-level retrieval (84.4% vs 51.6%)". **Both are withdrawn.**
-| Question Type | N | Hit@1 | Hit@5 | Hit@10 | MRR |
+**The session-level numbers are a degenerate artifact.** On the `longmemeval_oracle`
-|---------------|---|-------|-------|--------|-----|
+variant, the ingested haystack for a question consists essentially only of that
-| single-session-user | 70 | 100.0% | 100.0% | 100.0% | 1.0000 |
+question's evidence sessions. Every returned document therefore belongs to an answer
-| single-session-assistant | 56 | 100.0% | 100.0% | 100.0% | 1.0000 |
+session, so session-level hit rate is ≈1.0 at rank 0 *by construction* — which is
-| single-session-preference | 30 | 100.0% | 100.0% | 100.0% | 1.0000 |
+exactly why the result was a uniform 100.0% across every question type. It measured
-| temporal-reasoning | 133 | 100.0% | 100.0% | 100.0% | 1.0000 |
+the shape of the corpus, not the retriever.
-| multi-session | 133 | 100.0% | 100.0% | 100.0% | 1.0000 |
+
-| knowledge-update | 78 | 100.0% | 100.0% | 100.0% | 1.0000 |
+**The MemX comparison was not like-for-like on two independent axes.** MemX
 ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171)) reports Hit@5 = 51.6% /
 MRR = 0.380 at **fact-level granularity over 220,349 fact-level records drawn from
 19,195 sessions**, and explicitly notes that fact-level "doubl[es] session-level
 performance." Our 84.4% is **turn-level, on the oracle subset**. Different retrieval
 granularity, and a corpus smaller by orders of magnitude. A higher number on an
 easier corpus at a different granularity is not an outperformance claim, and it
 should not have been presented as one.
 A defensible comparison requires re-running against the full `longmemeval_s`
 haystack at matched granularity. That work is tracked; until it lands, no
 cross-system claim is made here.
 ### Search Latency (LongMemEval, n=500 queries)
@@ -678,14 +713,19 @@ cargo run --release --bin longmemeval_bench
 ```
 Recall numbers are deterministic (pure BM25 retrieval over a fixed dataset) and
-reproduce exactly:
+reproduce exactly. Scoring target as declared in the LongMemEval section above:
 retrieval recall, turn-level, k=10, `longmemeval_oracle` variant, BM25-only.
-| Metric | Session-Level | Turn-Level |
+| Metric | Turn-Level |
-|--------|---------------|------------|
+|--------|------------|
-| Hit@1 | 100.0% | 52.6% |
+| Hit@1 | 52.6% |
-| Hit@5 | 100.0% | 84.4% |
+| Hit@5 | **84.4%** |
-| Hit@10 | 100.0% | 90.4% |
+| Hit@10 | 90.4% |
-| MRR | 1.0000 | 0.6597 |
+| MRR | 0.6597 |
 Session-level figures are omitted here — they are degenerate on the oracle variant
 and have been retracted; see "Retracted: session-level recall and the MemX
 comparison" above.
 Search latency (hardware-dependent, tank numbers):
@@ -755,10 +795,14 @@ cargo bench -p clawhdf5-agent --bench bench -- "^(vector_search_1k|simd_cosine_1
 (The 1K Pre-norm cell from the original table has no corresponding benchmark
 in the current suite — not re-verified, left blank rather than guessed.)
-| Metric | MemX (claimed) | ClawhDF5 (tank) | Speedup (tank) |
+Same not-like-for-like caveat as the "Comparison to MemX" section at the top of this
-|--------|----------------|------------------|-----------------|
+file applies — MemX's figure is end-to-end, these are a single component. Ratios are
-| 100K flat search | <90 ms | 6.60 ms | **~14x** |
+an order-of-magnitude indication, not a benchmark result.
-| 100K IVF-PQ search | — | 865 µs | **~104x** |
+
 | Metric | MemX (claimed, end-to-end) | ClawhDF5 (tank, component only) | Ratio |
 |--------|----------------------------|----------------------------------|-------|
 | 100K flat search | <90 ms | 6.60 ms | ~14x |
 | 100K IVF-PQ search | — | 865 µs | ~104x |
 | Keyword search 10K | 1,100x improvement over unindexed | 520 µs (BM25) | Comparable |
 Every figure in this subsection is faster than the corresponding i7-12650H
@@ -5,7 +5,7 @@
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![Rust](https://img.shields.io/badge/rust-1.75%2B-orange.svg)](https://www.rust-lang.org)
 [![Tests](https://img.shields.io/badge/tests-1650%2B%20passing-brightgreen.svg)](#performance)
-[![LongMemEval](https://img.shields.io/badge/LongMemEval-Turn--Level%20Hit@5%2084%25%20BM25--only-blue.svg)](BENCHMARKS.md#longmemeval-results)
+[![LongMemEval](https://img.shields.io/badge/LongMemEval%20oracle-Turn--Level%20Hit@5%2084%25%20BM25--only-blue.svg)](BENCHMARKS.md#longmemeval-results)
 [![Footprint](https://img.shields.io/badge/footprint-6.5%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint)
 ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory — all stored in a single portable file.
@@ -64,7 +64,7 @@ Figures below are from an independent reproduction run on a second machine (AMD
 |-------|------|-----------------|--------|----------|
 | 1K | **54 µs** | — | — | — |
 | 10K | 753 µs | **27 µs** | — | — |
-| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | **8–76× faster** |
+| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) |
 > Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected,
 > apples-to-apples SIMD/scalar/parallel comparison methodology — see
@@ -97,21 +97,32 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data):
 Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition.
-> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records.
+> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
 ### LongMemEval Retrieval Recall
-Evaluated against the LongMemEval dataset (500 questions, multi-session haystack).
+Evaluated against the **`longmemeval_oracle`** variant (500 questions, evidence
-BM25-only baseline (no embedding model required at bench time), reproduced
+sessions only — *not* the full `longmemeval_s` haystack). BM25-only baseline
-identically on a second machine (see [BENCHMARKS.md § LongMemEval
+(zero embeddings, so the vector stage is inert), reproduced identically on a
-Results](BENCHMARKS.md#longmemeval-results)):
+second machine (see [BENCHMARKS.md § LongMemEval
 Results](BENCHMARKS.md#longmemeval-results) for the full scoring-target
 declaration):
-| Metric | Session-Level | Turn-Level |
+| Metric | Turn-Level |
-|--------|---------------|------------|
+|--------|------------|
-| Hit@5 | 100.0% | **84.4%** |
+| Hit@5 | **84.4%** |
-| MRR | 1.0000 | 0.6597 |
+| MRR | 0.6597 |
-Turn-level Hit@5 (84.4%) beats MemX's reported 51.6% at the same task.
+This is **retrieval recall** (did the gold memory appear in the top-k), not the
 official LongMemEval QA-accuracy metric — the two are not comparable, and
 retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 > **Previously reported here and now retracted:** session-level Hit@5 of 100.0% /
 > MRR 1.0000, and a claim of beating MemX's 51.6%. The session-level figures are
 > degenerate on the oracle variant (any returned document is a hit by
 > construction), and the MemX comparison was not like-for-like — MemX measures
 > fact-level granularity over 220,349 records. Details in
 > [BENCHMARKS.md](BENCHMARKS.md#retracted-session-level-recall-and-the-memx-comparison).
 > Enable embeddings via `hybrid_search(query_emb, text, 0.7, 0.3, k)` for substantially higher recall. The vector stage is served by the HNSW index by default (the `hnsw` feature is on by default); build with `--no-default-features --features float16` to fall back to an exact linear cosine scan.
@@ -4,7 +4,23 @@
 //! Since no embedding model is available at bench time, all embeddings are zero vectors
 //! and `hybrid_search` operates in BM25-only mode (vector_weight=0.0, keyword_weight=1.0).
 //!
-//! This matches the MemX paper methodology: evaluate retrieval recall, not answer generation.
+//! # Scoring target (read before citing any number from this harness)
 //!
 //! - **Metric: retrieval recall.** A "hit" means the gold-labelled memory appeared in
 //!   the top-k. No answer is generated and none is scored — the dataset's `answer`
 //!   field is deserialized and deliberately never read. This is **not** the official
 //!   LongMemEval metric, which is end-to-end QA accuracy (retrieve → generate → LLM
 //!   judge). Reporting retrieval recall as QA accuracy overstates by 20–30 points.
 //! - **Dataset: `longmemeval_oracle`** — evidence sessions only, not the full
 //!   `longmemeval_s`/`_m` haystack. Substantially easier corpus.
 //! - **Session-level metrics are degenerate here** and must not be cited: on the
 //!   oracle variant the haystack is essentially all-evidence, so any returned
 //!   document is a session-level hit at rank 0 by construction. Only turn-level
 //!   (`has_answer == true` on the source turn) measures the retriever.
 //! - **Not comparable to MemX's Hit@5=51.6% / MRR=0.380**, which is *fact-level*
 //!   granularity over 220,349 records from 19,195 sessions.
 //!
 //! See `BENCHMARKS.md` § "Retracted: session-level recall and the MemX comparison".
 //!
 //! # Usage
 //! ```
@@ -292,11 +308,22 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
    println!("=================================================================");
    println!();
    println!("Mode: vector_weight=0.0 / keyword_weight=1.0 (pure BM25)");
-    println!("Note: MemX (arxiv:2603.16171) with full system: Hit@5=51.6%, MRR=0.380");
+    println!();
-    println!("      BM25-only numbers are expected to be lower — honest baseline.");
+    println!("Scoring target: RETRIEVAL RECALL (did the gold memory land in top-k).");
    println!("  No answer is generated or scored. This is NOT the official");
    println!("  LongMemEval metric (QA accuracy via retrieve+generate+judge).");
    println!("Dataset: longmemeval_oracle — evidence sessions only, NOT the full");
    println!("  longmemeval_s haystack. This is a substantially easier corpus.");
    println!();
    println!("Do NOT compare these to MemX's Hit@5=51.6% / MRR=0.380: that is");
    println!("  fact-level granularity over 220,349 records from 19,195 sessions.");
    println!("  Different granularity and a corpus larger by orders of magnitude.");
    println!();
    println!("## Session-Level Recall  (n={})", overall.count);
    println!("  [DEGENERATE on the oracle variant — every returned document belongs");
    println!("   to an answer session by construction. Reported for completeness only;");
    println!("   this measures the corpus shape, not the retriever. Use turn-level.]");
    println!(
        "  Hit@1:  {:5.1}%   Hit@5:  {:5.1}%   Hit@10: {:5.1}%   MRR: {:.4}",
        overall.hit1_session_pct(),
@@ -381,6 +408,10 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
    println!("{{");
    println!("  \"benchmark\": \"longmemeval\",");
    println!("  \"mode\": \"bm25_only\",");
    println!("  \"dataset_variant\": \"oracle\",");
    println!("  \"scoring_target\": \"retrieval_recall\",");
    println!("  \"k\": 10,");
    println!("  \"session_level_degenerate\": true,");
    println!(
        "  \"total_questions\": {},",
        overall.count + overall.abstention_total