docs: Tier 4e — dated tank re-run for LongMemEval, SIMD, and vector-search sections
CI / test (push) Failing after 2s

Re-ran the three previously-undated sections flagged by the top-of-file
traceability note on tank (Ryzen 7 7800X3D, 2026-08-05), the same machine
already used for the vs-libhdf5 validation:

- LongMemEval Results: recall numbers reproduce exactly (deterministic
  BM25 retrieval), latency numbers are new/hardware-specific and higher
  than the i7 citation with much wider variance — recorded as-is.
- SIMD & Parallelism: found that several of the originally-named
  benchmarks don't actually hold the dataset fixed while varying only
  the SIMD/scalar/parallel axis — several call the same underlying
  function under different names. Used adaptive_benches' strategy_*
  benchmarks instead, which genuinely do isolate that axis via the
  SearchStrategy enum. Real finding: the speedup on tank (~1.5x) is
  smaller than on the i7 (~2.0x), attributed to the Ryzen's large L3
  cache narrowing the scalar-vs-SIMD gap — recorded rather than
  reconciled away.
- Vector Search Latency / Comparison to MemX: re-run with tank numbers,
  all faster than the i7 citation as expected; the 1K Pre-norm cell has
  no corresponding benchmark in the current suite and is left blank
  rather than guessed.

Updated the top-of-file traceability note to reflect that these three
sections (plus Comparison to MemX) now meet the dated/hardware-cited/
reproducible bar, narrowing the list of sections that don't.
This commit is contained in:
Omar Sobh
2026-08-05 14:54:10 -07:00
parent 2013fa94a0
commit 40527be653
+113 -4
View File
@@ -6,12 +6,16 @@
**Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile **Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile
**Date:** 2026-07-01 **Date:** 2026-07-01
> **Traceability note:** the "h5bench-Equivalent I/O Benchmarks" and > **Traceability note:** the "h5bench-Equivalent I/O Benchmarks" and both
> "Independent Validation: tank" sections below meet a dated, > "Independent Validation: tank" sections below meet a dated,
> hardware-cited, reproducible standard (explicit date, machine spec, and a > hardware-cited, reproducible standard (explicit date, machine spec, and a
> runnable command per result). The sections above them do not yet meet > runnable command per result) — this now covers "LongMemEval Results",
> that bar consistently — this is a known, tracked documentation gap, not > "SIMD & Parallelism", "Vector Search Latency", and "Comparison to MemX" via
> a claim that those numbers are wrong. > their tank re-runs. The remaining undated sections above (Hybrid Search,
> Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, Decision
> Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint,
> Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is
> a known, tracked documentation gap, not a claim that those numbers are wrong.
--- ---
@@ -656,3 +660,108 @@ cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta --
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta -- metadata_parse_in_memory cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta -- metadata_parse_in_memory
cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_read -- read_zerocopy_mmap cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_read -- read_zerocopy_mmap
``` ```
## Independent Validation: tank — LongMemEval & Vector Search (Ryzen 7 7800X3D), 2026-08-05
Re-running the "LongMemEval Results" and "SIMD & Parallelism" sections above on
tank (AMD Ryzen 7 7800X3D, 8C/16T, Ubuntu 26.04, same machine as the
vs-libhdf5 validation above) to give both sections the dated, hardware-cited,
reproducible citation the top-of-file traceability note flags them as
missing.
### LongMemEval Results (reproduction)
```bash
cd benchmarks/longmemeval
wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json
cargo run --release --bin longmemeval_bench
```
Recall numbers are deterministic (pure BM25 retrieval over a fixed dataset) and
reproduce exactly:
| Metric | Session-Level | Turn-Level |
|--------|---------------|------------|
| Hit@1 | 100.0% | 52.6% |
| Hit@5 | 100.0% | 84.4% |
| Hit@10 | 100.0% | 90.4% |
| MRR | 1.0000 | 0.6597 |
Search latency (hardware-dependent, tank numbers):
| Metric | avg | p50 | p95 | p99 |
|--------|-----|-----|-----|-----|
| Latency | 2,431 µs | 2,105 µs | 7,250 µs | 12,018 µs |
Higher than the i7-12650H figures at the top of this file (avg 1,004 µs) despite
tank's faster single-core performance elsewhere in this document — BM25 search
latency here scales with per-question haystack size and this run's variance is
wider (p99 is ~5x the mean), suggesting this metric is more sensitive to
momentary scheduling/cache effects than the flat-array vector-search benchmarks.
Recorded as-is rather than smoothed.
### SIMD & Parallelism (reproduction, with a correction)
```bash
cargo bench -p clawhdf5-agent --bench bench -- "^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$"
```
The original 10K table above compares named benchmarks (`vector_search`,
`rayon`, `strategy`) that, on inspection, don't all exercise the same
scalar-vs-SIMD-vs-parallel axis the table implies — several of the
`simd_cosine_10k`/`sequential_cosine_10k`-style benchmarks actually call the
same underlying function under different names. The `adaptive_benches` group's
`strategy_scalar_10k` / `strategy_simd_10k` / `strategy_rayon_10k` benchmarks
are the ones that genuinely hold the dataset fixed and vary only the
`SearchStrategy` enum, so they're the correct apples-to-apples comparison —
used here instead.
| Strategy | Latency (tank) | vs Sequential |
|----------|-----------------|----------------|
| Sequential (scalar) | 502 µs | 1.0x |
| SIMD (auto-vectorized) | 327 µs | **1.53x** |
| Rayon (parallel) | 323 µs | **1.55x** |
| Adaptive (auto-select) | 339 µs | **1.48x** |
Honest finding: the speedup from SIMD/parallelism over scalar is real but
smaller here (~1.5x) than the i7-12650H figures above (~2.0x). The Ryzen 7
7800X3D's large L3 cache (96MB 3D V-Cache) measurably narrows the gap versus a
naive scalar loop compared to the i7 — this is a genuine hardware-dependent
result, not a regression or measurement error, and is recorded rather than
reconciled away.
At 100K, no `strategy_*` benchmark exists in the current suite (`adaptive_benches`
only covers n=10,000), so this row uses the same `simd_cosine_100k`/
`rayon_cosine_100k` benchmarks as the original table — not a true scalar
baseline, so no "vs Sequential" multiple is reported for it:
| Strategy | Latency (tank) |
|----------|-----------------|
| SIMD | 6.60 ms |
| Rayon parallel | 4.73 ms |
### Vector Search Latency & Comparison to MemX (reproduction)
```bash
cargo bench -p clawhdf5-agent --bench bench -- "^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10|bm25_search_10k)$"
```
| Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
|-------|-------------|----------|-----------------|--------|-------|
| **1K** | 47.8 µs | — | — | — | — |
| **10K** | 501 µs | 322 µs | 24.8 µs | — | 109 µs |
| **100K** | 6.60 ms | — | 608 µs | 865 µs | — |
(The 1K Pre-norm cell from the original table has no corresponding benchmark
in the current suite — not re-verified, left blank rather than guessed.)
| Metric | MemX (claimed) | ClawhDF5 (tank) | Speedup (tank) |
|--------|----------------|------------------|-----------------|
| 100K flat search | <90 ms | 6.60 ms | **~14x** |
| 100K IVF-PQ search | — | 865 µs | **~104x** |
| Keyword search 10K | 1,100x improvement over unindexed | 520 µs (BM25) | Comparable |
Every figure in this subsection is faster than the corresponding i7-12650H
number at the top of this file, consistent with the Ryzen 7 7800X3D's higher
single-core throughput and larger cache observed in the vs-libhdf5 validation
above.