Two related trust-boundary gaps, both closed:
1. ConsolidationEngine::add_memory took a plain `source: MemorySource`
parameter, so any caller could claim MemorySource::System/Correction —
which get elevated importance weighting in score_correction — for
content whose actual origin the caller doesn't control or hasn't
verified. Split into add_memory(UntrustedSource) for ordinary
caller-supplied content (User/Tool/Retrieval only, no elevated variant
exists to claim) and add_trusted_memory(TrustedSource) for content whose
elevated trust the caller has independently verified (System/
Correction). Updated the one production consumer outside this crate
(clawhdf5-bench's consolidation_efficiency benchmark) and all tests.
2. The provenance/anomaly wiring added in the previous commit introduced
the same pattern: infer_memory_source mapped source_channel == "system"
or "correction" straight to the elevated MemorySource variants. Since
MemoryEntry.source_channel is unvalidated caller-supplied text, this let
a write dodge check_source_anomaly's User-flood detection by simply
self-labeling source_channel = "system". infer_memory_source now never
returns System/Correction — only Tool/Retrieval (recognized channel
names) or User (everything else, the conservative default).
INT-05
than h5py (5e)
stable-worldmodel (arXiv 2605.21800, LeCun/Balestriero) supports HDF5 as
one of three native formats and measures generic HDF5 at 1,416-1,474
samples/s for per-frame sample loading. This measures clawhdf5 against
that shape, hardware-controlled: clawhdf5 and h5py reading the SAME file
on the SAME machine.
worldmodel_sampling example: mmap an (N,H,W,C) uint8 observation dataset,
read each frame once per pass in shuffled (dataloader) order. The file is
written by h5py (benchmarks/gen_worldmodel_frames.py) — clawhdf5 parsing
an externally-produced HDF5 file is itself the interop result — and read
by both clawhdf5 and the h5py counterpart (benchmarks/bench_worldmodel_h5py.py,
opening exactly stable-worldmodel's HDF5Dataset: swmr + 256 MB cache).
Results (tank, Ryzen 7 7800X3D, 20000x64x64x3 = 246 MB, in page cache,
median of 3):
clawhdf5 zero-copy view 593k samples/sec 8.1x
clawhdf5 materialised copy 518k samples/sec 7.1x
h5py (swmr, 256 MB cache) 73k samples/sec 1.0x
The materialised-copy row is the fair equal-work comparison (to_vec per
frame, matching h5py's numpy materialisation) and is still 7.1x faster;
that the copy costs almost nothing shows the gap is h5py's per-frame call
overhead, not data movement. Honest caveats in BENCHMARKS.md: absolute
numbers are NOT comparable to the paper's (different hardware, smaller
frames, no torch/transform), only the same-machine ratio is; this is an
in-page-cache measurement isolating read-path overhead, not disk
bandwidth.
Adds only an example, two benchmark scripts, and a BENCHMARKS.md section —
no library code. (Workspace clippy has pre-existing toolchain drift
unrelated to this change; tracked separately.)
Tier 4b reported hybrid retrieval at 0.7/0.3 and noted the weights were "the
documented default, not a searched optimum". `--sweep` searches them: 0.0 to 1.0
in 0.1 steps, reusing the one-time embedding table so eleven configurations cost
barely more than three.
The result is not a refinement. 0.7/0.3 is **strictly dominated**:
vector/keyword Hit@1 Hit@5 Hit@10 MRR sHit@5
0.0 / 1.0 53.8% 75.0% 81.6% 0.6320 93.6%
0.3 / 0.7 53.2% 78.8% 87.2% 0.6463 96.0%
0.4 / 0.6 51.6% 81.4% 87.8% 0.6429 96.8%
0.5 / 0.5 48.2% 81.4% 88.2% 0.6234 97.4%
0.7 / 0.3 44.4% 79.2% 86.0% 0.5868 95.8%
1.0 / 0.0 36.0% 71.8% 81.6% 0.5027 94.2%
0.4/0.6 beats 0.7/0.3 on every metric at both granularities — Hit@1 +7.2pp,
Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. No trade is being made; the default simply
sat on the wrong side of the peak. It is now 0.4/0.6, and README's usage snippet
recommends the same.
This corrects a conclusion I published one commit ago. Measuring only 0.7/0.3, I
wrote that fusion "buys deeper recall and pays for it at rank 1" and advised
callers taking a single top hit to prefer BM25. That was an artifact of the bad
weight, not a property of fusion: at 0.3/0.7 hybrid *beats* BM25 on MRR (0.6463
vs 0.6320) and Hit@5 (78.8% vs 75.0%) while giving up 0.6pp of Hit@1. Both
BENCHMARKS.md and README carry the correction rather than a quiet edit, since
the old text told readers to configure their systems a particular way.
The three-mode ablation rows are kept at their original settings — they measure
the shape of each stage in isolation, and the operating point now comes from the
sweep instead.
The GPU path worked but was effectively hidden. cudarc's build script shells out
to `nvcc`, which ships in /usr/local/cuda/bin — a directory the reference host
had installed but never exported to the login shell, so `--features
embeddings-cuda` failed with a bare "`nvcc --version` failed" panic from a
dependency's build script, and the runtime fallback then reported only
"Embedder: CPU (...)" before spending hours on work a GPU does in minutes.
Two changes, both about making the failure legible rather than changing what the
code does:
- The CPU fallback now says why it fell back and what that costs, with the
concrete fix. A run that silently takes two orders of magnitude longer reads
as a hang, not as a configuration choice.
- BENCHMARKS.md states the build-time nvcc requirement, where the toolkit
actually installs, and that a shell file read non-interactively is the place
to export it — `~/.zshenv` rather than `~/.zshrc`, because build scripts do
not run in an interactive shell.
Host-side, the reference machine's CUDA exports lived in ~/.bashrc below its
non-interactive guard while the login shell is zsh, so they never applied to
anything. Moved to ~/.zshenv with duplicate-prepend guards; `nvcc --version`
and `cargo build --features embeddings-cuda` now both work over a plain
non-interactive ssh with no manual export.
Every LongMemEval number this project has published measured BM25 alone. The
bench passed zero-vector embeddings with vector_weight=0.0, so the HNSW/vector
stage — the thing the README credits for retrieval quality — contributed
nothing and was never tested.
An optional `embeddings` feature loads all-MiniLM-L6-v2 via candle and encodes
the corpus for real. It is off by default and nothing in the shipped crates
depends on it, so a project that advertises no heavyweight dependencies keeps
that property; without the feature the bench behaves exactly as before.
Full haystack, n=500, turn-level:
Hit@1 Hit@5 Hit@10 MRR
BM25 only 53.8% 75.0% 81.6% 0.6320
Vector only 36.0% 71.8% 81.6% 0.5027
Hybrid 0.7/0.3 44.4% 79.2% 86.0% 0.5868
Session-level, hybrid leads outright: 88.2 / 95.8 / 97.8 / 0.9158.
The hybrid claim holds for depth and not for precision@1. Hybrid is the best
configuration at Hit@5 and Hit@10 at both granularities — turn-level Hit@5 gains
4.2 points over BM25 and 7.4 over vector-only, which is the result that justifies
running two stages at all. But BM25 alone still leads turn-level Hit@1 and MRR,
so fusing buys deeper recall and pays at rank 1. Callers assembling five memories
of context want hybrid; callers taking a single top hit are better served by BM25
today. The 0.7/0.3 weights are the documented default, not a searched optimum.
omni-cortex's four-signal ablation found the same direction independently — there,
adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR. Two
codebases, two fusion schemes, same trade.
Vector-only trailing BM25 at every turn-level cutoff except Hit@10 is stated
plainly rather than buried: LongMemEval questions share heavy vocabulary with
their evidence turns, which is close to the best case for lexical matching, and
MiniLM at 384-d is a small model.
Implementation notes:
- Texts are deduplicated before encoding. The haystack sessions are drawn from
a shared pool, so 500 questions x 493.5 turns collapses to 190,015 unique
strings — the difference between encoding the corpus once and per question.
- `embeddings-cuda` adds the GPU path, and it is not a convenience: 190k texts
take ~13 min on an RTX 5060 Ti, while the same work on 8 CPU cores was still
unfinished after 30 minutes. The device is selected at runtime with a CPU
fallback, so a machine without CUDA still works.
- Mean-pooling is masked and the output L2-normalised, which is the published
recipe for this checkpoint (not the [CLS] pooler).
One measurement wrinkle, recorded rather than smoothed over: on the oracle
variant BM25-only reads 84.2% Hit@5 with real embedding vectors present against
84.4% with zero vectors — one question of 500 changes rank, MRR identical at
0.6597. On the full haystack the two agree exactly. Weight 0.0 evidently does not
make the vector stage bit-for-bit absent from candidate selection on a small
corpus.
Verified on the Linux dev host: 49 groups / 1659 passed / 0 failed, clippy clean
under -D warnings, fmt clean, with and without the feature.
The harness only ever ran longmemeval_oracle — evidence sessions only, which is
a substantially easier corpus than the dataset LongMemEval results are normally
quoted on. Worse, the variant was a hardcoded "oracle" string in both the report
header and the JSON summary, so pointing it at longmemeval_s would have produced
full-haystack numbers labelled oracle.
DatasetProfile now measures the corpus instead of asserting it: sessions and
turns per question, and evidence-session density (the mean share of a question's
haystack sessions that are answer sessions). The variant label and the
session-level degeneracy warning are both derived from that density, so a
mislabelled input file cannot produce a mislabelled result. Measured: 100.0%
density on the oracle variant, 4.0% on longmemeval_s.
The full haystack, all 500 questions, 47.7 sessions and 493.5 turns each:
turn-level session-level
Hit@1 53.8% 86.2%
Hit@5 75.0% 93.6%
Hit@10 81.6% 96.6%
MRR 0.6320 0.8948
Turn-level drops 84.4% -> 75.0% against the oracle variant. That 9.4-point gap
is the price of the real haystack and is exactly why oracle-only numbers should
not be presented as LongMemEval results.
Session-level is now reportable. It was retracted before because at 100% evidence
density every returned document is a hit by construction; at 4.0% density a hit
reflects discrimination, so 93.6% is a real measurement rather than a restatement
of the corpus shape. Per-type it also finally separates: single-session-assistant
100.0% Hit@1 against single-session-preference 33.3% — BM25 has nothing to grip
on a preference question whose evidence shares no vocabulary with the query.
The MemX comparison stays withdrawn. Running the full haystack closes the corpus
half of that mismatch but not the granularity half: MemX measures fact-level over
220,349 records, and this harness measures turn- and session-level.
Two smaller fixes found while running it:
- --limit samples evenly across the file rather than taking a prefix. The
dataset is ordered by question type, so `--limit 20` returned 20
single-session-user questions and nothing else while reading like a
whole-dataset result.
- abstention_accuracy emits null rather than 0.0 when a corpus poses no
abstention questions. longmemeval_s has none, and 0.0000 reads as total
failure at a task that was never asked.
README.md and BENCHMARKS.md now lead with the full-haystack numbers and keep the
oracle figures alongside, labelled as the easier corpus.
Verified on the Linux dev host: 49 groups / 1659 passed / 0 failed, clippy clean
under -D warnings, fmt clean. The full 500-question run takes ~70 s.
A methodology audit found that two benchmark claims published in this repo two
days ago measure the wrong thing. Both are retracted in place rather than
quietly edited, with the reasoning recorded.
1. Session-level LongMemEval recall (100.0% Hit@1/5/10, MRR 1.0000, uniform
across all six question types) is a degenerate artifact. On the
longmemeval_oracle variant the ingested haystack for a question is
essentially only that question's evidence sessions, so every returned
document belongs to an answer session and session-level hit rate is ~1.0 at
rank 0 by construction. The uniform 100% across every question type was the
tell. It measured the shape of the corpus, not the retriever. Only the
turn-level figure (84.4% Hit@5) carries signal, and it is now the only
retrieval number cited.
2. The "clawhdf5 outperforms MemX at turn-level retrieval (84.4% vs 51.6%)"
claim was not like-for-like on two independent axes. Confirmed against
arxiv:2603.16171: MemX's Hit@5=51.6% / MRR=0.380 is *fact-level*
granularity over 220,349 fact-level records drawn from 19,195 sessions, and
the paper explicitly notes fact-level "doubl[es] session-level performance".
Ours is turn-level on the oracle subset — different granularity, and a
corpus smaller by orders of magnitude. A higher number on an easier corpus
at a different granularity is not an outperformance claim.
Also caveats the vector-search "vs MemX" latency ratios, which compare a single
clawhdf5 component (raw vector search) against MemX's end-to-end pipeline
figure (embeddings + FTS5 + four-factor re-ranking). The numbers are real; the
"speedup" framing overstated by an unquantified margin and is now labelled an
order-of-magnitude indication.
Adds an explicit scoring-target declaration to BENCHMARKS.md per arXiv
2605.24060, which found that changing scoring target alone alters nDCG on
83-94% of queries and can reverse system rankings. States dataset variant,
metric (retrieval recall, NOT the official QA-accuracy metric), granularity,
k, and that the vector stage is inert (zero embeddings, vector_weight=0.0).
The harness itself now prints its scoring target, flags the session-level
block as degenerate, warns against the MemX comparison, and emits
dataset_variant/scoring_target/k/session_level_degenerate in its JSON summary,
so the caveats travel with the numbers instead of living only in docs.
Add [workspace.dependencies] to the root Cargo.toml for the four
duplicated-across-many-crates dependencies flagged by the earlier review:
tempfile (7 crates), criterion (6), half (4 — real version skew, clawhdf5-gpu
pinned 2.7 while others used bare 2), and serde (4). Update every consuming
crate to `dep = { workspace = true }`, preserving crate-local `optional =
true` where it already existed. half now resolves uniformly to 2.7.x
workspace-wide instead of two separate semver ranges.
Also fixed clawhdf5-filters/Cargo.toml's stale "rustyhdf5" description
while touching the file (same class of leftover rename as prior fixes).
Not touching rayon/byteorder/clap (no skew found, lower priority).
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
- README's "HDF5 Core I/O" table claimed 19ns/2,080µs labeled 308× (real ratio
~109,000×) and a 313ns zero-copy mmap figure — neither traced to any dated
benchmark in BENCHMARKS.md. Replaced the table wholesale with the existing
"vs libhdf5 Summary" figures, relabeled from "h5py/C HDF5" to "libhdf5"
(BENCHMARKS.md never benchmarks against h5py, only libhdf5 directly).
- Added two new Criterion benchmarks to close the coverage gaps that produced
the untraceable numbers: metadata_open_from_disk (I/O-inclusive, fair
clawhdf5-vs-libhdf5 file-open comparison) and metadata_parse_in_memory
(clawhdf5-only, explicitly labeled as excluding I/O) in h5bench_meta.rs;
read_zerocopy_mmap in h5bench_read.rs (forces real page-ins by summing
elements rather than just returning a slice length — the mmap path turns
out to be slower than a plain copy at these sizes, an honest, unflattering
but real result now documented instead of a fabricated 313ns).
- Re-ran the full existing benchmark suite plus the two new ones on a second,
independently administered machine (tank: Ryzen 7 7800X3D) to validate the
numbers before publishing them. 5 of 6 rows landed within ~15% of the
original i7-12650H figures; recorded both in BENCHMARKS.md's new
"Independent Validation" section. README now cites the tank numbers.
- Added a short top-of-file README callout naming both halves of the project
(general-purpose HDF5 library vs. agent memory layer) with links to
BENCHMARKS.md and the Crate Map, so a data-infra reader isn't 60% through
a memory-store pitch before finding the part relevant to them.
- Added one factual, no-names line noting benchmark numbers are being
validated in collaboration with HDF5 Group engineers.
- Fixed the same untraceable "2-300x faster than h5py/C HDF5" / "313 ns"
claims in docs/QUICKSTART.md, one click from the README's own "New here?"
link.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Implements Pcodec (filter ID 32023) via the `pco` 1.0.x crate as a new
optional compression codec. Pcodec achieves 30–94% better compression
ratio than Zstd for f32/f64 columnar data at 1–5 GiB/s decompression
speed, making it ideal for write-once/read-many embedding archives.
Write throughput at 512×512: 591 MiB/s (parity with Zstd-3 at 610 MiB/s).
For smaller chunks Zstd-3 remains faster due to Pcodec's fixed per-chunk
distributional analysis overhead.
- Add FILTER_PCODEC = 32023 constant to filter_pipeline.rs
- Add pcodec_compress/pcodec_decompress using pco::standalone API
- Wire into compress_chunk/decompress_chunk dispatch
- Add ChunkOptions.pcodec field and DatasetBuilder.with_pcodec() method
- Enable pcodec as highest-priority codec in build_pipeline()
- Add pco dep (optional, feature = "pcodec") to clawhdf5-format/clawhdf5
- Add write_2d_chunked_pcodec benchmark comparing pcodec vs zstd-3
- Document results in BENCHMARKS.md
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add features = ["zstd"] to clawhdf5-bench dev-dependency so the
write_2d_chunked_zstd benchmark no longer panics with UnsupportedFilter(32015).
Update BENCHMARKS.md and README.md with measured results from the full
h5bench write suite (2026-06-30, post write-performance improvements):
- Zstd-3 hits 593 MiB/s at 512×512 vs deflate-6's 280 MiB/s (2.12×)
- Zstd-3 hits 330 MiB/s at 128×128 vs deflate-6's 132 MiB/s (2.51×)
- Sequential f64 batch write improved ~8-11% from owned-Vec IO path
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Four independent write-path improvements:
1. Cache compressed chunks between Pass 1 and Pass 2 (chunked_write.rs,
file_writer.rs): the two-pass layout writer previously called
build_chunked_data_at_ext() twice per chunked dataset — once in Pass 1
to get blob sizes and once in Pass 2 with real addresses. Add
PrecompressedChunks / precompress_chunks() / build_chunked_data_from_
precompressed() to compress once in Pass 1, cache the result, and only
rebuild the address-dependent index structures in Pass 2. Expected
~2× speedup for chunked+deflate writes (512×512 deflate: 3.33ms → ~1.7ms).
2. SIMD-vectorisable shuffle filter (filters.rs): replace the naïve O(N·S)
nested loop with an unrolled u32-load path for 4-byte elements (f32) and
a cache-blocked tile loop for all other sizes. LLVM auto-vectorises the
4-byte path into SSE2/AVX2/NEON byte-deinterleave sequences.
3. Zstd benchmark variant (h5bench_write.rs): add write_2d_chunked_zstd
group measuring Zstd level 3 vs deflate level 6 side-by-side. Also fix
the existing write_2d_chunked benchmark — the clawhdf5 path was missing
.with_deflate(6), making the comparison apples-to-oranges. Add arXiv-
backed doc recommendation on DatasetBuilder::with_zstd().
4. Zero-copy HNSW save (hnsw.rs, clawhdf5-io/lib.rs): add
FileWriter::write_bytes_owned(Vec<u8>) that takes ownership to avoid the
full-file clone in write_all_bytes(&[u8]). HNSW::save_to_hdf5 uses it.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Adds three Criterion benchmark suites mirroring the h5bench HPC I/O
benchmark workloads in pure Rust — no C libhdf5 required for the default
path, with an optional `libhdf5-compare` feature for side-by-side numbers.
- benches/h5bench_write.rs: write_1d_contiguous, write_2d_chunked,
write_f64_batch, write_multi_dataset, write_with_attrs
- benches/h5bench_read.rs: read_sequential, read_f64_sequential,
read_chunked_2d, read_from_disk, read_hyperslab
- benches/h5bench_meta.rs: metadata_attrs_write, metadata_attrs_read,
metadata_groups_create, metadata_groups_traverse, metadata_string_attrs
All benchmarks pass `cargo bench --bench <name> -- --test` and clippy
reports zero warnings. Run with `cargo bench -p clawhdf5-bench`.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Bumps [workspace.package] and all 17 crate package versions (and internal
path-dependency requirements) from 2.0.0 to 2.1.0 for a coordinated release.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ClawHDF5 Coding Agent