Omar Sobh
c913cd1cbf
CI / test (push) Failing after 2s
Every LongMemEval number this project has published measured BM25 alone. The
bench passed zero-vector embeddings with vector_weight=0.0, so the HNSW/vector
stage — the thing the README credits for retrieval quality — contributed
nothing and was never tested.
An optional `embeddings` feature loads all-MiniLM-L6-v2 via candle and encodes
the corpus for real. It is off by default and nothing in the shipped crates
depends on it, so a project that advertises no heavyweight dependencies keeps
that property; without the feature the bench behaves exactly as before.
Full haystack, n=500, turn-level:
Hit@1 Hit@5 Hit@10 MRR
BM25 only 53.8% 75.0% 81.6% 0.6320
Vector only 36.0% 71.8% 81.6% 0.5027
Hybrid 0.7/0.3 44.4% 79.2% 86.0% 0.5868
Session-level, hybrid leads outright: 88.2 / 95.8 / 97.8 / 0.9158.
The hybrid claim holds for depth and not for precision@1. Hybrid is the best
configuration at Hit@5 and Hit@10 at both granularities — turn-level Hit@5 gains
4.2 points over BM25 and 7.4 over vector-only, which is the result that justifies
running two stages at all. But BM25 alone still leads turn-level Hit@1 and MRR,
so fusing buys deeper recall and pays at rank 1. Callers assembling five memories
of context want hybrid; callers taking a single top hit are better served by BM25
today. The 0.7/0.3 weights are the documented default, not a searched optimum.
omni-cortex's four-signal ablation found the same direction independently — there,
adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR. Two
codebases, two fusion schemes, same trade.
Vector-only trailing BM25 at every turn-level cutoff except Hit@10 is stated
plainly rather than buried: LongMemEval questions share heavy vocabulary with
their evidence turns, which is close to the best case for lexical matching, and
MiniLM at 384-d is a small model.
Implementation notes:
- Texts are deduplicated before encoding. The haystack sessions are drawn from
a shared pool, so 500 questions x 493.5 turns collapses to 190,015 unique
strings — the difference between encoding the corpus once and per question.
- `embeddings-cuda` adds the GPU path, and it is not a convenience: 190k texts
take ~13 min on an RTX 5060 Ti, while the same work on 8 CPU cores was still
unfinished after 30 minutes. The device is selected at runtime with a CPU
fallback, so a machine without CUDA still works.
- Mean-pooling is masked and the output L2-normalised, which is the published
recipe for this checkpoint (not the [CLS] pooler).
One measurement wrinkle, recorded rather than smoothed over: on the oracle
variant BM25-only reads 84.2% Hit@5 with real embedding vectors present against
84.4% with zero vectors — one question of 500 changes rank, MRR identical at
0.6597. On the full haystack the two agree exactly. Weight 0.0 evidently does not
make the vector stage bit-for-bit absent from candidate selection on a small
corpus.
Verified on the Linux dev host: 49 groups / 1659 passed / 0 failed, clippy clean
under -D warnings, fmt clean, with and without the feature.
81 lines
2.5 KiB
TOML
81 lines
2.5 KiB
TOML
[package]
|
|
name = "clawhdf5-bench"
|
|
version = "2.1.0"
|
|
edition = "2024"
|
|
description = "Benchmark harnesses for clawhdf5-agent (Track 8)"
|
|
license = "MIT"
|
|
|
|
[[bin]]
|
|
name = "longmemeval_bench"
|
|
path = "src/bin/longmemeval_bench.rs"
|
|
|
|
[[bin]]
|
|
name = "memory_arena"
|
|
path = "src/bin/memory_arena.rs"
|
|
|
|
[[bin]]
|
|
name = "footprint_bench"
|
|
path = "src/bin/footprint_bench.rs"
|
|
|
|
[[bin]]
|
|
name = "consolidation_efficiency"
|
|
path = "src/bin/consolidation_efficiency.rs"
|
|
|
|
[[bin]]
|
|
name = "ephemeral_perf"
|
|
path = "src/bin/ephemeral_perf.rs"
|
|
|
|
[[bin]]
|
|
name = "mpi_io_bench"
|
|
path = "src/bin/mpi_io_bench.rs"
|
|
required-features = ["mpi-io"]
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# h5bench-equivalent Criterion benchmarks
|
|
# ---------------------------------------------------------------------------
|
|
|
|
[[bench]]
|
|
name = "h5bench_write"
|
|
harness = false
|
|
|
|
[[bench]]
|
|
name = "h5bench_read"
|
|
harness = false
|
|
|
|
[[bench]]
|
|
name = "h5bench_meta"
|
|
harness = false
|
|
|
|
[dependencies]
|
|
clawhdf5-agent = { path = "../clawhdf5-agent" }
|
|
clawhdf5-io = { path = "../clawhdf5-io" }
|
|
mpi = { version = "0.8", optional = true }
|
|
serde = { workspace = true }
|
|
serde_json = "1"
|
|
tempfile = { workspace = true }
|
|
# Optional: libhdf5 C wrapper for side-by-side comparison (requires system libhdf5).
|
|
# Enable with: cargo bench -p clawhdf5-bench --features libhdf5-compare
|
|
# Uses hdf5-metno (fork of hdf5 crate) which supports HDF5 1.14.x.
|
|
hdf5 = { version = "0.12", optional = true, package = "hdf5-metno" }
|
|
# Optional: real sentence embeddings for the LongMemEval bench's vector stage.
|
|
# Enable with: cargo run --release --bin longmemeval_bench --features embeddings
|
|
# Off by default — nothing in the shipped crates depends on these.
|
|
candle-core = { version = "0.9", optional = true }
|
|
candle-nn = { version = "0.9", optional = true }
|
|
candle-transformers = { version = "0.9", optional = true }
|
|
tokenizers = { version = "0.21", optional = true }
|
|
|
|
[dev-dependencies]
|
|
clawhdf5 = { path = "../clawhdf5", features = ["zstd", "pcodec"] }
|
|
criterion = { workspace = true }
|
|
|
|
[features]
|
|
# When enabled, benchmarks add matching libhdf5 variants for side-by-side comparison.
|
|
libhdf5-compare = ["hdf5"]
|
|
mpi-io = ["clawhdf5-io/mpi-io", "mpi"]
|
|
# Real MiniLM embeddings for longmemeval_bench, so the vector stage is not inert.
|
|
embeddings = ["candle-core", "candle-nn", "candle-transformers", "tokenizers"]
|
|
# CUDA-accelerated embedding. MiniLM on a CPU takes hours over the full
|
|
# longmemeval_s haystack; on a GPU it is minutes.
|
|
embeddings-cuda = ["embeddings", "candle-core/cuda", "candle-nn/cuda", "candle-transformers/cuda"]
|