bench: make the vector stage real, and measure BM25 vs vector vs hybrid (Tier 4b)

Every LongMemEval number this project has published measured BM25 alone. The
bench passed zero-vector embeddings with vector_weight=0.0, so the HNSW/vector
stage — the thing the README credits for retrieval quality — contributed
nothing and was never tested.

An optional `embeddings` feature loads all-MiniLM-L6-v2 via candle and encodes
the corpus for real. It is off by default and nothing in the shipped crates
depends on it, so a project that advertises no heavyweight dependencies keeps
that property; without the feature the bench behaves exactly as before.

Full haystack, n=500, turn-level:

                          Hit@1    Hit@5   Hit@10      MRR
    BM25 only             53.8%    75.0%    81.6%   0.6320
    Vector only           36.0%    71.8%    81.6%   0.5027
    Hybrid 0.7/0.3        44.4%    79.2%    86.0%   0.5868

Session-level, hybrid leads outright: 88.2 / 95.8 / 97.8 / 0.9158.

The hybrid claim holds for depth and not for precision@1. Hybrid is the best
configuration at Hit@5 and Hit@10 at both granularities — turn-level Hit@5 gains
4.2 points over BM25 and 7.4 over vector-only, which is the result that justifies
running two stages at all. But BM25 alone still leads turn-level Hit@1 and MRR,
so fusing buys deeper recall and pays at rank 1. Callers assembling five memories
of context want hybrid; callers taking a single top hit are better served by BM25
today. The 0.7/0.3 weights are the documented default, not a searched optimum.

omni-cortex's four-signal ablation found the same direction independently — there,
adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR. Two
codebases, two fusion schemes, same trade.

Vector-only trailing BM25 at every turn-level cutoff except Hit@10 is stated
plainly rather than buried: LongMemEval questions share heavy vocabulary with
their evidence turns, which is close to the best case for lexical matching, and
MiniLM at 384-d is a small model.

Implementation notes:
  - Texts are deduplicated before encoding. The haystack sessions are drawn from
    a shared pool, so 500 questions x 493.5 turns collapses to 190,015 unique
    strings — the difference between encoding the corpus once and per question.
  - `embeddings-cuda` adds the GPU path, and it is not a convenience: 190k texts
    take ~13 min on an RTX 5060 Ti, while the same work on 8 CPU cores was still
    unfinished after 30 minutes. The device is selected at runtime with a CPU
    fallback, so a machine without CUDA still works.
  - Mean-pooling is masked and the output L2-normalised, which is the published
    recipe for this checkpoint (not the [CLS] pooler).

One measurement wrinkle, recorded rather than smoothed over: on the oracle
variant BM25-only reads 84.2% Hit@5 with real embedding vectors present against
84.4% with zero vectors — one question of 500 changes rank, MRR identical at
0.6597. On the full haystack the two agree exactly. Weight 0.0 evidently does not
make the vector stage bit-for-bit absent from candidate selection on a small
corpus.

Verified on the Linux dev host: 49 groups / 1659 passed / 0 failed, clippy clean
under -D warnings, fmt clean, with and without the feature.

crates/clawhdf5-bench/Cargo.toml

@@ -57,6 +57,13 @@ tempfile = { workspace = true }
 # Enable with: cargo bench -p clawhdf5-bench --features libhdf5-compare
 # Uses hdf5-metno (fork of hdf5 crate) which supports HDF5 1.14.x.
 hdf5 = { version = "0.12", optional = true, package = "hdf5-metno" }
+# Optional: real sentence embeddings for the LongMemEval bench's vector stage.
+# Enable with: cargo run --release --bin longmemeval_bench --features embeddings
+# Off by default — nothing in the shipped crates depends on these.
+candle-core = { version = "0.9", optional = true }
+candle-nn = { version = "0.9", optional = true }
+candle-transformers = { version = "0.9", optional = true }
+tokenizers = { version = "0.21", optional = true }
 
 [dev-dependencies]
 clawhdf5 = { path = "../clawhdf5", features = ["zstd", "pcodec"] }
@@ -66,3 +73,8 @@ criterion = { workspace = true }
 # When enabled, benchmarks add matching libhdf5 variants for side-by-side comparison.
 libhdf5-compare = ["hdf5"]
 mpi-io = ["clawhdf5-io/mpi-io", "mpi"]
+# Real MiniLM embeddings for longmemeval_bench, so the vector stage is not inert.
+embeddings = ["candle-core", "candle-nn", "candle-transformers", "tokenizers"]
+# CUDA-accelerated embedding. MiniLM on a CPU takes hours over the full
+# longmemeval_s haystack; on a GPU it is minutes.
+embeddings-cuda = ["embeddings", "candle-core/cuda", "candle-nn/cuda", "candle-transformers/cuda"]