research: implementation brief for perf/security/provenance pass (INT-01..19)

Read against ROADMAP.md/IMPROVEMENT_LOG.md/CLAUDE.md/CHANGELOG.md first so nothing already-merged is re-proposed. 19 numbered INT items grouped by crate (clawhdf5-agent provenance wiring, clawhdf5-format/io untrusted-file parsing, clawhdf5-ann HNSW hot path, clawhdf5-py mutex poisoning), each with file path, problem, and proposed change, plus a suggested implementation order for the coding phase.
bench: world-model sample loading — clawhdf5 reads h5py files 7x faster
2026-08-16 23:57:29 +00:00 · 2026-08-07 22:54:26 -07:00 · 2026-08-07 11:10:12 -07:00 · 2026-08-07 08:13:59 -07:00 · 2026-08-07 07:22:20 -07:00 · 2026-08-07 04:53:39 -07:00
90 changed files with 3754 additions and 792 deletions
@@ -0,0 +1,26 @@
 name: CI
 on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
 jobs:
  test:
    runs-on: ubuntu-latest
    container: rust:latest
    steps:
      - uses: actions/checkout@v4
      - name: Cache cargo registry/target
        uses: actions/cache@v4
        with:
          path: |
            ~/.cargo/registry
            ~/.cargo/git
            target
          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
      - name: Install rustfmt & clippy components
        run: rustup component add rustfmt clippy
      - name: Install thumbv7em-none-eabihf target
        run: rustup target add thumbv7em-none-eabihf
      - name: Run CI script
        run: bash scripts/ci-test.sh
@@ -1,3 +1,6 @@
 /target
 Cargo.lock
 benchmarks/longmemeval/*.json
 # Local model weights (MiniLM etc.) — large, not committed
 weights/
@@ -6,6 +6,26 @@
 **Rust:** 1.96.0-nightly (2026-03-14) · `--release` profile  
 **Date:** 2026-07-01
 > **Traceability note:** the "h5bench-Equivalent I/O Benchmarks" and both
 > "Independent Validation: tank" sections below meet a dated,
 > hardware-cited, reproducible standard (explicit date, machine spec, and a
 > runnable command per result) — this now covers "LongMemEval Results",
 > "SIMD & Parallelism", "Vector Search Latency", and "Comparison to MemX" via
 > their tank re-runs. The remaining undated sections above (Hybrid Search,
 > Knowledge Graph, Memory Consolidation, Temporal Index, Write Path, Decision
 > Gate, Memory Strategy, Multi-Session Benchmark, Memory Footprint,
 > Consolidation Efficiency, Ephemeral Tier) do not yet meet that bar — this is
 > a known, tracked documentation gap, not a claim that those numbers are wrong.
 >
 > **Correctness note (2026-08-06).** Being dated and reproducible is necessary but
 > not sufficient — a number can be perfectly reproducible and still measure the
 > wrong thing. A methodology audit found two such cases and both have been
 > retracted in place: the session-level LongMemEval figures (degenerate on the
 > oracle variant) and the MemX retrieval comparison (mismatched granularity and
 > corpus). Every cross-system comparison in this file now carries an explicit
 > scoping caveat. Where a section states a scoring target, that declaration is the
 > contract — read it before citing the number.
 ---
 ## Vector Search Latency
@@ -24,10 +44,19 @@ Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embed
 MemX claims end-to-end search under 90ms at 100K records (Rust + libSQL + FTS5).
-| Metric | MemX (claimed) | ClawhDF5 | Speedup |
+> **Caveat — not like-for-like.** MemX's `<90 ms` is *end-to-end* search across their
-|--------|----------------|----------|---------|
+> full pipeline (dense embeddings + FTS5 + four-factor re-ranking). The clawhdf5
-| 100K flat search | <90 ms | 11.4 ms | **~8x** |
+> figures below are a *single component* — raw vector search latency, excluding
-| 100K IVF-PQ search | — | 1.19 ms | **~76x** |
+> embedding, keyword, fusion, and re-ranking stages. A component measured against a
 > full pipeline will always look favourable; the "speedup" column overstates the real
 > advantage by an unquantified margin and should be read as an order-of-magnitude
 > indication only, not a benchmark result. Matching MemX's measurement boundary is
 > tracked as follow-up work.
 | Metric | MemX (claimed, end-to-end) | ClawhDF5 (component only) | Ratio |
 |--------|----------------------------|---------------------------|-------|
 | 100K flat search | <90 ms | 11.4 ms | ~8x |
 | 100K IVF-PQ search | — | 1.19 ms | ~76x |
 | Keyword search 10K | 1,100x improvement over unindexed | 583 µs (BM25) | Comparable |
 ---
@@ -174,46 +203,193 @@ _Latency benchmarks generated with Criterion.rs (50-100 samples per benchmark).
 ## LongMemEval Results
-**Dataset:** LongMemEval oracle (500 questions, 6 question types, variable-length chat histories)
+> **Scoring target declaration.** Per [arXiv 2605.24060](https://arxiv.org/abs/2605.24060),
 > which found that changing scoring target alone alters nDCG on 83–94% of queries and
 > can reverse system rankings, this section states its measurement contract explicitly:
 >
 > - **Dataset variant:** both are now reported below — the full `longmemeval_s`
 >   haystack (**the headline number**) and `longmemeval_oracle` (evidence sessions
 >   only, a substantially easier corpus, kept for continuity). The harness does not
 >   trust the filename: it measures evidence-session density from the data and
 >   labels the run from that, so a mislabelled input cannot yield a mislabelled
 >   result. Measured density is 4.0% on `longmemeval_s` and 100.0% on the oracle.
 > - **Metric:** *retrieval recall.* A "hit" means the gold-labelled memory appeared in
 >   the top-k. **No answer is generated and none is scored** — the dataset's `answer`
 >   field is deserialized and never read. This is **not** the official LongMemEval
 >   leaderboard metric, which is end-to-end QA accuracy (retrieve → generate → LLM
 >   judge). Retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 > - **Granularity:** turn-level = the returned memory's source turn had `has_answer == true`.
 > - **k = 10**, n = 500.
 > - **Retrieval mode:** all three are reported below. Historically the bench passed
 >   zero-vector embeddings with `vector_weight=0.0`, so the HNSW/vector stage was
 >   inert and every published number was BM25 alone. Real `all-MiniLM-L6-v2`
 >   embeddings are now available via `--features embeddings --embeddings <dir>`,
 >   and BM25-only / vector-only / hybrid are each measured separately.
 **Mode:** BM25-only retrieval — zero embeddings, `vector_weight=0.0`, `keyword_weight=1.0`
 **Reference:** MemX (arxiv:2603.16171) with full embedding system: Hit@5=51.6%, MRR=0.380
-> **Run:** `cargo run --release --bin longmemeval_bench`
+> **Run:** `cargo run --release --bin longmemeval_bench -- benchmarks/longmemeval/longmemeval_s_cleaned.json`
 > (~70 s for all 500 questions on the tank reference machine). Omit the path for the
 > oracle variant; add `--limit N` for an evenly-strided subsample.
-### Session-Level Recall (n=500)
+### Full haystack — `longmemeval_s`, n=500 (the number to cite)
-| Metric | ClawhDF5 (BM25-only) |
+47.7 sessions and 493.5 turns per question; 4.0% of haystack sessions are evidence
-|--------|---------------------|
+sessions, so retrieval has to actually discriminate.
 | Hit@1 | **100.0%** |
 | Hit@5 | **100.0%** |
 | Hit@10 | **100.0%** |
 | MRR | **1.0000** |
-Perfect session-level recall across all 500 questions and all 6 question types.
+| Metric | Turn-level | Session-level |
 |--------|-----------|---------------|
 | Hit@1 | 53.8% | 86.2% |
 | Hit@5 | **75.0%** | **93.6%** |
 | Hit@10 | 81.6% | 96.6% |
 | MRR | 0.6320 | 0.8948 |
-### Turn-Level Recall (n=500)
+Session-level is reported here because on this corpus it is meaningful — unlike on
 the oracle variant, where it was degenerate and was retracted (below). At 4.0%
 evidence density a session-level hit reflects discrimination rather than corpus
 shape.
-| Metric | ClawhDF5 (BM25-only) | MemX (full system)¹ |
+Per-type, session-level: `single-session-assistant` 100.0% Hit@1 (n=56),
-|--------|---------------------|---------------------|
+`knowledge-update` 96.2% (n=78), `single-session-user` 94.3% (n=70),
-| Hit@1 | **52.6%** | — |
+`multi-session` 84.2% (n=133), `temporal-reasoning` 84.2% (n=133), and
-| Hit@5 | **84.4%** | 51.6% |
+`single-session-preference` 33.3% (n=30) — the one category where BM25 clearly
-| Hit@10 | **90.4%** | — |
+struggles, since a preference question's evidence rarely shares vocabulary with
-| MRR | **0.6597** | 0.380 |
+the question.
-**clawhdf5 outperforms MemX at turn-level retrieval** — Hit@5 84.4% vs 51.6%, MRR 0.66 vs 0.38 — with BM25 alone, no embeddings needed.
+### Retrieval mode ablation — full haystack, n=500
-> ¹ MemX uses dense embeddings + FTS5 + four-factor re-ranking. Our BM25-only result exceeds their full pipeline.
+Real 384-d `all-MiniLM-L6-v2` embeddings, 190,015 unique texts encoded once on an
 RTX 5060 Ti (~13 min; the same work on the 8-core CPU was still unfinished after
 30 minutes, so the GPU path is not a convenience here). Turn-level:
-### Per-Type Breakdown (session-level)
+| Mode | Hit@1 | Hit@5 | Hit@10 | MRR |
 |------|-------|-------|--------|-----|
 | BM25 only (`0.0`/`1.0`) | **53.8%** | 75.0% | 81.6% | **0.6320** |
 | Vector only (`1.0`/`0.0`) | 36.0% | 71.8% | 81.6% | 0.5027 |
 | Hybrid (`0.7`/`0.3`) | 44.4% | **79.2%** | **86.0%** | 0.5868 |
-| Question Type | N | Hit@1 | Hit@5 | Hit@10 | MRR |
+Session-level:
-|---------------|---|-------|-------|--------|-----|
+
-| single-session-user | 70 | 100.0% | 100.0% | 100.0% | 1.0000 |
+| Mode | Hit@1 | Hit@5 | Hit@10 | MRR |
-| single-session-assistant | 56 | 100.0% | 100.0% | 100.0% | 1.0000 |
+|------|-------|-------|--------|-----|
-| single-session-preference | 30 | 100.0% | 100.0% | 100.0% | 1.0000 |
+| BM25 only | 86.2% | 93.6% | 96.6% | 0.8948 |
-| temporal-reasoning | 133 | 100.0% | 100.0% | 100.0% | 1.0000 |
+| Vector only | 85.4% | 94.2% | 96.6% | 0.8901 |
-| multi-session | 133 | 100.0% | 100.0% | 100.0% | 1.0000 |
+| Hybrid | **88.2%** | **95.8%** | **97.8%** | **0.9158** |
-| knowledge-update | 78 | 100.0% | 100.0% | 100.0% | 1.0000 |
+
 ### Weight sweep — full haystack, n=500
 `0.7/0.3` was a documented default, never a searched one. Sweeping
 `vector_weight` from 0.0 to 1.0 (`--sweep`, reusing the one-time embedding
 table) shows it is not merely suboptimal but **strictly dominated**:
 | vector / keyword | Hit@1 | Hit@5 | Hit@10 | MRR | session Hit@5 |
 |---|---|---|---|---|---|
 | 0.0 / 1.0 (BM25) | **53.8%** | 75.0% | 81.6% | 0.6320 | 93.6% |
 | 0.1 / 0.9 | 53.2% | 77.4% | 83.8% | 0.6374 | 95.0% |
 | 0.2 / 0.8 | 53.6% | 78.2% | 85.6% | 0.6440 | 95.4% |
 | 0.3 / 0.7 | 53.2% | 78.8% | 87.2% | **0.6463** | 96.0% |
 | **0.4 / 0.6** | 51.6% | **81.4%** | 87.8% | 0.6429 | 96.8% |
 | 0.5 / 0.5 | 48.2% | **81.4%** | **88.2%** | 0.6234 | **97.4%** |
 | 0.6 / 0.4 | 46.6% | 79.8% | 87.4% | 0.6069 | 96.6% |
 | 0.7 / 0.3 *(old default)* | 44.4% | 79.2% | 86.0% | 0.5868 | 95.8% |
 | 0.8 / 0.2 | 40.6% | 76.2% | 85.4% | 0.5571 | 95.2% |
 | 0.9 / 0.1 | 37.8% | 73.4% | 84.6% | 0.5289 | 94.2% |
 | 1.0 / 0.0 (vector) | 36.0% | 71.8% | 81.6% | 0.5027 | 94.2% |
 **`0.4/0.6` beats `0.7/0.3` on every metric at both granularities** — Hit@1
 +7.2pp, Hit@5 +2.2, Hit@10 +1.8, MRR +0.056. There is no trade being made; the
 old default was simply on the wrong side of the peak. **`0.4/0.6` is the
 recommended setting**, with `0.3/0.7` preferable if rank-1 precision matters
 most (it takes the best MRR in the sweep and gives up only 0.6pp of Hit@1
 against pure BM25).
 **Correction.** An earlier revision of this section, measuring only `0.7/0.3`,
 concluded that fusion "buys deeper recall and pays for it at rank 1" and advised
 callers taking a single top hit to prefer BM25. That was an artifact of the
 badly-chosen weight, not a property of fusion. At `0.3/0.7` hybrid *beats* BM25
 on MRR (0.6463 vs 0.6320) and on Hit@5 (78.8% vs 75.0%) while costing 0.6pp of
 Hit@1. The advice below is corrected accordingly.
 **Hybrid wins, once the weights are right.** At the old `0.7/0.3` the picture
 looked like a trade: best at Hit@5 and Hit@10, worse than BM25 at Hit@1 and MRR.
 The sweep above shows that was the weight, not fusion. At `0.4/0.6` hybrid leads
 Hit@5 and Hit@10 outright; at `0.3/0.7` it also leads MRR and is within 0.6pp of
 BM25 at Hit@1. Both dominate `0.7/0.3`.
 The rows below are kept at the three original settings because they are what the
 mode ablation measured — read them as "the shape of each stage in isolation",
 and take the operating point from the sweep.
 The same pattern shows up independently in omni-cortex's four-signal RRF ablation,
 where adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR.
 Two different codebases, two different fusion schemes, same direction.
 Vector-only being *worse* than BM25 at every turn-level cutoff except Hit@10 is
 worth stating plainly rather than hiding: LongMemEval questions share substantial
 vocabulary with their evidence turns, which is close to the best case for lexical
 matching, and MiniLM at 384 dimensions is a small embedding model.
 > **Run:** `cargo run --release --bin longmemeval_bench --features embeddings -- \
 >   benchmarks/longmemeval/longmemeval_s_cleaned.json --embeddings weights/all-minilm-l6-v2`
 > For the GPU path use `--features embeddings-cuda`. That requires `nvcc` on
 > `PATH` at *build* time — cudarc's build script shells out to it. The toolkit
 > installs to `/usr/local/cuda/bin`, which many distributions do not export;
 > check with `nvcc --version` and, if it is missing, add it somewhere every
 > shell reads (for zsh that is `~/.zshenv`, not `~/.zshrc`, since build tooling
 > runs non-interactively). The device is selected at runtime with a CPU
 > fallback, so a machine without CUDA still produces correct numbers — just far
 > more slowly, and the bench says so on startup.
 >
 > Weights: `huggingface.co/sentence-transformers/all-MiniLM-L6-v2` — place
 > `model.safetensors` and `tokenizer.json` in the `--embeddings` directory.
 ### Oracle variant — `longmemeval_oracle`, n=500 (easier corpus, kept for continuity)
 | Metric | ClawhDF5 (BM25-only, oracle variant) |
 |--------|--------------------------------------|
 | Hit@1 | 52.6% |
 | Hit@5 | **84.4%** |
 | Hit@10 | 90.4% |
 | MRR | 0.6597 |
 Turn-level. The 9.4-point gap between this and the full haystack's 75.0% is the
 price of the harder corpus, and is the reason oracle-only numbers should not be
 presented as LongMemEval results. Session-level figures on this variant are
 degenerate — see below.
 With real embeddings the same oracle corpus gives BM25-only 84.2% / vector-only
 80.4% / hybrid **85.2%** Hit@5 turn-level — hybrid ahead at Hit@5 and Hit@10 and
 behind at Hit@1, matching the full-haystack pattern above. (BM25-only reads 84.2%
 here against 84.4% with zero embedding vectors: one question of 500 changes rank,
 with MRR identical at 0.6597. On the full haystack the two agree exactly.)
 ### Retracted: session-level recall and the MemX comparison
 Earlier revisions of this file reported session-level Hit@1/5/10 of **100.0%** with
 MRR **1.0000**, uniform across all six question types, and claimed clawhdf5
 "outperforms MemX at turn-level retrieval (84.4% vs 51.6%)". **Both are withdrawn.**
 **The session-level numbers are a degenerate artifact.** On the `longmemeval_oracle`
 variant, the ingested haystack for a question consists essentially only of that
 question's evidence sessions. Every returned document therefore belongs to an answer
 session, so session-level hit rate is ≈1.0 at rank 0 *by construction* — which is
 exactly why the result was a uniform 100.0% across every question type. It measured
 the shape of the corpus, not the retriever.
 **The MemX comparison was not like-for-like on two independent axes.** MemX
 ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171)) reports Hit@5 = 51.6% /
 MRR = 0.380 at **fact-level granularity over 220,349 fact-level records drawn from
 19,195 sessions**, and explicitly notes that fact-level "doubl[es] session-level
 performance." Our 84.4% is **turn-level, on the oracle subset**. Different retrieval
 granularity, and a corpus smaller by orders of magnitude. A higher number on an
 easier corpus at a different granularity is not an outperformance claim, and it
 should not have been presented as one.
 The full-haystack half of that gap is now closed: the section above reports
 `longmemeval_s` over all 500 questions. The **granularity** mismatch remains — MemX
 measures fact-level, we measure turn-level and session-level — so no cross-system
 claim is made here even now. Matching granularity would require fact-level
 extraction over the haystack, which this harness does not do.
 ### Search Latency (LongMemEval, n=500 queries)
@@ -368,6 +544,54 @@ No network hop, no serialization — direct HashMap operations.
 ---
 ## World-Model Sample Loading (vs h5py / stable-worldmodel shape)
 Reproduces the access pattern of `stable-worldmodel`'s HDF5 dataloader
 ([arXiv 2605.21800](https://arxiv.org/abs/2605.21800), LeCun/Balestriero
 group), which supports HDF5 as one of three native formats and measures
 generic HDF5 at **1,416-1,474 samples/s** (vs Lance 4,815) for per-frame
 sample loading. This benchmark measures **clawhdf5 vs h5py on the same
 machine and the same file**, so the comparison is hardware-controlled.
 **Absolute numbers are not comparable to the paper's** - different hardware
 (AMD Ryzen 7 7800X3D, local NVMe, warm page cache), smaller frames, and no
 torch-tensor / transform step. Only the clawhdf5-vs-h5py ratio *here* is a
 controlled result. The workload is the dataloader shape: a `(N, H, W, C)`
 uint8 observation dataset (20,000 x 64x64x3 = 246 MB), each frame read once
 per pass in a fixed shuffled (random-access) order, 10 passes.
 Both read a **file written by h5py** - clawhdf5 parsing an
 externally-produced HDF5 file is itself the interop result. h5py opens SWMR
 with a 256 MB chunk cache, exactly `stable-worldmodel`'s `HDF5Dataset`; it
 materialises each frame as a numpy array (`d[i]`) and sums it. clawhdf5
 mmaps once, takes a zero-copy `&[u8]` over the contiguous dataset, and
 indexes frame `i` as a subslice.
 | Reader | samples/sec (median of 3) | vs h5py |
 |--------|---------------------------|---------|
 | **clawhdf5** (zero-copy view) | **593,000** | **8.1x** |
 | **clawhdf5** (materialised copy per frame) | **518,000** | **7.1x** |
 | h5py (swmr, 256 MB cache) | 73,000 | 1.0x |
 The **materialised-copy row is the fair, equal-work comparison** - it
 `to_vec()`s every frame so clawhdf5 pays the same per-frame allocation h5py
 does, and it is still **7.1x faster**. That the copy costs almost nothing
 (518k vs 593k) shows the h5py gap is **per-frame call overhead** (Python +
 library dispatch), not data movement. This is an in-page-cache measurement:
 it isolates the read-path overhead both libraries add on top of the OS,
 which is the thing that differs - not disk bandwidth, which is shared.
 Reproduce (`benchmarks/`):
 ```bash
 python benchmarks/gen_worldmodel_frames.py /tmp/wm_frames.h5 20000
 cargo run --release -p clawhdf5-bench --example worldmodel_sampling -- /tmp/wm_frames.h5 10
 cargo run --release -p clawhdf5-bench --example worldmodel_sampling -- /tmp/wm_frames.h5 10 --copy
 python benchmarks/bench_worldmodel_h5py.py /tmp/wm_frames.h5 10
 ```
 Measured 2026-08-07 on tank (Ryzen 7 7800X3D, 246 MB dataset in page cache).
 ## Cross-Platform Notes
 > **Run:** `./benchmarks/cross_platform.sh [--full] [--output results.json]`
@@ -649,3 +873,117 @@ cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta --
 cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_meta -- metadata_parse_in_memory
 cargo bench -p clawhdf5-bench --features libhdf5-compare --bench h5bench_read -- read_zerocopy_mmap
 ```
 ## Independent Validation: tank — LongMemEval & Vector Search (Ryzen 7 7800X3D), 2026-08-05
 Re-running the "LongMemEval Results" and "SIMD & Parallelism" sections above on
 tank (AMD Ryzen 7 7800X3D, 8C/16T, Ubuntu 26.04, same machine as the
 vs-libhdf5 validation above) to give both sections the dated, hardware-cited,
 reproducible citation the top-of-file traceability note flags them as
 missing.
 ### LongMemEval Results (reproduction)
 ```bash
 cd benchmarks/longmemeval
 wget https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_oracle.json
 cargo run --release --bin longmemeval_bench
 ```
 Recall numbers are deterministic (pure BM25 retrieval over a fixed dataset) and
 reproduce exactly. Scoring target as declared in the LongMemEval section above:
 retrieval recall, turn-level, k=10, `longmemeval_oracle` variant, BM25-only.
 | Metric | Turn-Level |
 |--------|------------|
 | Hit@1 | 52.6% |
 | Hit@5 | **84.4%** |
 | Hit@10 | 90.4% |
 | MRR | 0.6597 |
 Session-level figures are omitted here — they are degenerate on the oracle variant
 and have been retracted; see "Retracted: session-level recall and the MemX
 comparison" above.
 Search latency (hardware-dependent, tank numbers):
 | Metric | avg | p50 | p95 | p99 |
 |--------|-----|-----|-----|-----|
 | Latency | 2,431 µs | 2,105 µs | 7,250 µs | 12,018 µs |
 Higher than the i7-12650H figures at the top of this file (avg 1,004 µs) despite
 tank's faster single-core performance elsewhere in this document — BM25 search
 latency here scales with per-question haystack size and this run's variance is
 wider (p99 is ~5x the mean), suggesting this metric is more sensitive to
 momentary scheduling/cache effects than the flat-array vector-search benchmarks.
 Recorded as-is rather than smoothed.
 ### SIMD & Parallelism (reproduction, with a correction)
 ```bash
 cargo bench -p clawhdf5-agent --bench bench -- "^(strategy_scalar_10k|strategy_simd_10k|strategy_rayon_10k|adaptive_search_10k|simd_cosine_100k|rayon_cosine_100k)$"
 ```
 The original 10K table above compares named benchmarks (`vector_search`,
 `rayon`, `strategy`) that, on inspection, don't all exercise the same
 scalar-vs-SIMD-vs-parallel axis the table implies — several of the
 `simd_cosine_10k`/`sequential_cosine_10k`-style benchmarks actually call the
 same underlying function under different names. The `adaptive_benches` group's
 `strategy_scalar_10k` / `strategy_simd_10k` / `strategy_rayon_10k` benchmarks
 are the ones that genuinely hold the dataset fixed and vary only the
 `SearchStrategy` enum, so they're the correct apples-to-apples comparison —
 used here instead.
 | Strategy | Latency (tank) | vs Sequential |
 |----------|-----------------|----------------|
 | Sequential (scalar) | 502 µs | 1.0x |
 | SIMD (auto-vectorized) | 327 µs | **1.53x** |
 | Rayon (parallel) | 323 µs | **1.55x** |
 | Adaptive (auto-select) | 339 µs | **1.48x** |
 Honest finding: the speedup from SIMD/parallelism over scalar is real but
 smaller here (~1.5x) than the i7-12650H figures above (~2.0x). The Ryzen 7
 7800X3D's large L3 cache (96MB 3D V-Cache) measurably narrows the gap versus a
 naive scalar loop compared to the i7 — this is a genuine hardware-dependent
 result, not a regression or measurement error, and is recorded rather than
 reconciled away.
 At 100K, no `strategy_*` benchmark exists in the current suite (`adaptive_benches`
 only covers n=10,000), so this row uses the same `simd_cosine_100k`/
 `rayon_cosine_100k` benchmarks as the original table — not a true scalar
 baseline, so no "vs Sequential" multiple is reported for it:
 | Strategy | Latency (tank) |
 |----------|-----------------|
 | SIMD | 6.60 ms |
 | Rayon parallel | 4.73 ms |
 ### Vector Search Latency & Comparison to MemX (reproduction)
 ```bash
 cargo bench -p clawhdf5-agent --bench bench -- "^(vector_search_1k|simd_cosine_10k|simd_cosine_100k|prenorm_search_10k|ivf_search_10k_nprobe10|ivf_search_100k_nprobe10|ivf_pq_search_100k|rairs_search_10k_nprobe10|bm25_search_10k)$"
 ```
 | Scale | Flat Search | Pre-norm | IVF (nprobe=10) | IVF-PQ | RAIRS |
 |-------|-------------|----------|-----------------|--------|-------|
 | **1K** | 47.8 µs | — | — | — | — |
 | **10K** | 501 µs | 322 µs | 24.8 µs | — | 109 µs |
 | **100K** | 6.60 ms | — | 608 µs | 865 µs | — |
 (The 1K Pre-norm cell from the original table has no corresponding benchmark
 in the current suite — not re-verified, left blank rather than guessed.)
 Same not-like-for-like caveat as the "Comparison to MemX" section at the top of this
 file applies — MemX's figure is end-to-end, these are a single component. Ratios are
 an order-of-magnitude indication, not a benchmark result.
 | Metric | MemX (claimed, end-to-end) | ClawhDF5 (tank, component only) | Ratio |
 |--------|----------------------------|----------------------------------|-------|
 | 100K flat search | <90 ms | 6.60 ms | ~14x |
 | 100K IVF-PQ search | — | 865 µs | ~104x |
 | Keyword search 10K | 1,100x improvement over unindexed | 520 µs (BM25) | Comparable |
 Every figure in this subsection is faster than the corresponding i7-12650H
 number at the top of this file, consistent with the Ryzen 7 7800X3D's higher
 single-core throughput and larger cache observed in the vs-libhdf5 validation
 above.
@@ -2,6 +2,90 @@
 ## Unreleased
 ### Security
 - `clawhdf5-format`: bounded decompression output (`MAX_DECOMPRESS_SIZE`) for
  deflate/lz4/zstd/pcodec so a crafted compressed chunk can't drive an
  unbounded allocation (memory-exhaustion DoS).
 - `clawhdf5-format`: `chunked_read.rs`/`data_read.rs`/`local_heap.rs` bounds
  audit — added `ensure_len` overflow guards at every plain-arithmetic
  offset+size check, a recursion-depth guard against a crafted
  self-referencing/cyclic B-tree chunk index, a fix for an unguarded
  compound-datatype `byte_offset` overrun in `read_compound_fields`, and an
  `ndims - 1` underflow guard for degenerate zero-dimension chunked layouts.
  Added a new `fuzz_dataset_read` cargo-fuzz target (walks every dataset in a
  parsed file and exercises the contiguous/chunked/compact raw-data read
  paths) which found and fixed 3 real crash bugs — an integer-multiply
  overflow in `copy_chunk_to_output`'s N-D assembly path, the `ndims - 1`
  underflow above, and an overflow in `local_heap.rs` — within the first few
  fuzzing runs.
 - `clawhdf5-format`: `btree_v1.rs` overflow-safe bounds checks via a local
  `ensure_len` helper, closing a `usize`-overflow panic reachable from a
  crafted near-`usize::MAX` B-tree offset.
 - `clawhdf5-agent`: WAL length-prefix caps (`MAX_WAL_FIELD_LEN`, 64 MiB) reject
  a corrupted/truncated length claim before allocating. Followed by a full
  per-entry CRC32 trailer (`WAL_VERSION` bumped to 2) — a bit-flip inside an
  entry now stops replay cleanly instead of silently accepting corrupted
  data. Old-format WAL files are still read correctly and migrated to the new
  format on next open.
 - `clawhdf5-android`: validate `embedding_len`/`query_embedding_len` against
  the handle's configured `embedding_dim` (and reject null pointers) before
  constructing a slice from a raw pointer in `edgehdf5_save` /
  `edgehdf5_hybrid_search`.
 - `clawhdf5-py`: bump pyo3/numpy `0.28` → `0.29`, clearing two RUSTSEC
  advisories (OOB read in `PyList`/`PyTuple` iterator; missing `Sync` bound on
  `PyCFunction::new_closure`).
 - Clarified that the integrity hashes in `clawhdf5-agent::provenance`
  (FNV-1a) and `clawhdf5-format::provenance` (SHA-256) are unkeyed and detect
  only accidental corruption, not tampering — doc-only change, no behavior
  change.
 ### Performance
 - `clawhdf5-format`: chunk cache lookup is now O(1) (`slot_index: HashMap`)
  instead of a linear scan, and cache hits return a shared `Arc` instead of
  cloning the decompressed buffer — the hottest path in chunked reads.
 - `clawhdf5-ann`: optional `parallel` feature (rayon) parallelizes HNSW's
  `prune_connections` neighbor-distance computation. The outer build/insert
  loop is deliberately left sequential — it has genuine cross-iteration data
  dependencies and needs its own correctness-focused design pass.
 - `clawhdf5-format/chunked_read.rs`: removed 12 unnecessary
  `chunk_dimensions[..rank].to_vec()` allocations where callees already
  accept `&[u32]`.
 ### Architecture
 - Added `.gitea/workflows/ci.yml`, actually wiring the long-existing
  `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) into CI on every
  push/PR to `main`. Fixed stale package names in `ci-test.sh`/
  `check-nostd.sh` that had been silently no-op'ing the `clawhdf5-py`
  exclusion and the no_std check.
 - Fixed a genuine no_std build break in `clawhdf5-format` (uncovered once the
  no_std CI check actually started running): `core::sync::atomic::AtomicU64`
  doesn't exist on `thumbv7em-none-eabihf` (switched to `portable-atomic`),
  missing `alloc` imports for `Box`/`Vec`/`format!` on a few no_std paths, and
  `f64::powi` (std/libm-only) replaced with a local exponentiation-by-squaring
  helper in the scale-offset filter.
 - Added `[workspace.dependencies]` for `tempfile`/`criterion`/`half`/`serde`,
  fixing a real version skew on `half` (`2` vs `2.7` across crates).
 - Fixed version skew: `clawhdf5-py` (`pyproject.toml`) and
  `packages/clawhdf5-node` (`package.json`) were both behind the actual crate
  version (2.1.0).
 - Documented that the `mpi-io` feature's read/write paths are root-read
  +broadcast / gather-to-rank-0, not true collective I/O.
 ### Documentation
 - BENCHMARKS.md: re-ran the previously-undated "LongMemEval Results", "SIMD &
  Parallelism", and "Vector Search Latency"/"Comparison to MemX" sections on
  a second machine (tank, Ryzen 7 7800X3D) with explicit dates and reproduce
  commands. Found and corrected a methodology issue in the SIMD/Parallelism
  benchmark selection (several originally-compared benchmarks didn't actually
  isolate the scalar/SIMD/parallel axis).
 - README.md / ROADMAP.md / CLAUDE.md: corrected several stale facts —
  the `clawhdf5-types` crate (removed earlier) was still listed in the
  README crate map; the LongMemEval numbers in the README badge and table
  didn't match the actual (much better) benchmark results in BENCHMARKS.md;
  total line-of-code and test-count figures were stale; `clawhdf5-gpu`'s
  CubeCL→wgpu correction; documented the new `clawhdf5-ann` `parallel`
  feature flag, which had no entry in the Feature Flags table.
 ### New Features
 - `clawhdf5-migrate`: substantial engine improvements:
  - **Real content validation** — the post-migration check now reads the written
@@ -17,7 +17,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
 | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
 | `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
 | `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
-| `clawhdf5-gpu` | GPU-accelerated I/O via CubeCL |
+| `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
 | `clawhdf5-accel` | CPU SIMD acceleration path |
 | `clawhdf5-migrate` | Schema migration engine |
 | `clawhdf5-android` | Android JNI bindings |
@@ -33,7 +33,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
  the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
  the cache and self-heals on drift). Build the agent with
  `--no-default-features --features float16` to force the exact linear cosine scan.
- WAL (write-ahead log) for crash-safe persistence
+- WAL (write-ahead log) for crash-safe persistence, with a CRC32 trailer per entry so a corrupted entry stops replay cleanly instead of loading bad data
 - GPU-accelerated batch I/O for large dataset processing
 - Python and Node.js bindings for cross-language use
 - NetCDF-4 compatibility for scientific data interop
@@ -25,3 +25,9 @@ version = "2.1.0"
 edition = "2024"
 license = "MIT"
 repository = "https://github.com/redclawsystems/clawhdf5"
 [workspace.dependencies]
 tempfile = "3"
 criterion = { version = "0.5", features = ["html_reports"] }
 half = "2.7"
 serde = { version = "1", features = ["derive"] }
@@ -4,8 +4,8 @@
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 [![Rust](https://img.shields.io/badge/rust-1.75%2B-orange.svg)](https://www.rust-lang.org)
-[![Tests](https://img.shields.io/badge/tests-1500%2B%20passing-brightgreen.svg)](#benchmarks)
+[![Tests](https://img.shields.io/badge/tests-1650%2B%20passing-brightgreen.svg)](#performance)
-[![LongMemEval](https://img.shields.io/badge/LongMemEval-Hit@5%2046%25%20BM25--only-blue.svg)](BENCHMARKS.md#longmemeval-results)
+[![LongMemEval](https://img.shields.io/badge/LongMemEval%20oracle-Turn--Level%20Hit@5%2084%25%20BM25--only-blue.svg)](BENCHMARKS.md#longmemeval-results)
 [![Footprint](https://img.shields.io/badge/footprint-6.5%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint)
 ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory — all stored in a single portable file.
@@ -64,7 +64,12 @@ Figures below are from an independent reproduction run on a second machine (AMD
 |-------|------|-----------------|--------|----------|
 | 1K | **54 µs** | — | — | — |
 | 10K | 753 µs | **27 µs** | — | — |
-| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | **8–76× faster** |
+| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) |
 > Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected,
 > apples-to-apples SIMD/scalar/parallel comparison methodology — see
 > [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector
 > Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05).
 ### Agent Memory Operations
@@ -92,20 +97,52 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data):
 Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition.
-> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records.
+> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
 ### LongMemEval Retrieval Recall
-Evaluated against the LongMemEval dataset (500 questions, multi-session haystack).
+Evaluated against the full **`longmemeval_s`** haystack — all 500 questions, 47.7
-BM25-only baseline (no embedding model required at bench time):
+sessions and 493.5 turns each, with only 4.0% of haystack sessions being evidence
 sessions. See [BENCHMARKS.md § LongMemEval
 Results](BENCHMARKS.md#longmemeval-results) for the full scoring-target
 declaration:
-| Metric | BM25-only | Full hybrid¹ |
+| Mode | Turn-Level Hit@5 | Session-Level Hit@5 |
-|--------|-----------|--------------|
+|------|------------------|---------------------|
-| Hit@5 (session) | ~46% | Higher |
+| BM25 only | 75.0% | 93.6% |
-| MRR (session) | ~0.34 | Higher |
+| Vector only (MiniLM) | 71.8% | 94.2% |
-| Abstention accuracy | ~72% | — |
+| Hybrid (0.4/0.6, tuned) | **81.4%** | **96.8%** |
-> ¹ Enable embeddings via `hybrid_search(query_emb, text, 0.7, 0.3, k)` for substantially higher recall. The vector stage is served by the HNSW index by default (the `hnsw` feature is on by default); build with `--no-default-features --features float16` to fall back to an exact linear cosine scan.
+Hybrid is the strongest configuration, which is what running two retrieval stages
 is for. The weights matter more than the stages: a sweep of `vector_weight` from
 0.0 to 1.0 found the long-standing `0.7/0.3` default is **strictly dominated** by
 `0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Use
 `0.4/0.6`, or `0.3/0.7` if rank-1 precision matters most. See
 [BENCHMARKS.md § Weight sweep](BENCHMARKS.md#longmemeval-results).
 Vector embeddings require `--features embeddings`; without it the vector stage is
 inert and only the BM25 row is produced, which is what every previously published
 number here measured.
 On the easier `longmemeval_oracle` variant (evidence sessions only) the same
 harness scores 84.4% turn-level Hit@5 / MRR 0.6597, reproduced identically on a
 second machine. The 9.4-point gap is the cost of the real haystack, and is why the
 full-haystack number is the one quoted here.
 This is **retrieval recall** (did the gold memory appear in the top-k), not the
 official LongMemEval QA-accuracy metric — the two are not comparable, and
 retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 > **Previously reported here and now retracted:** session-level Hit@5 of 100.0% /
 > MRR 1.0000, and a claim of beating MemX's 51.6%. Those session-level figures were
 > degenerate on the oracle variant (any returned document is a hit by
 > construction); the 93.6% above is a different, real measurement on a corpus where
 > evidence sessions are 4.0% of the haystack. The MemX comparison stays withdrawn —
 > MemX measures fact-level granularity over 220,349 records, which running the full
 > haystack does not fix. Details in
 > [BENCHMARKS.md](BENCHMARKS.md#retracted-session-level-recall-and-the-memx-comparison).
 > Enable embeddings via `hybrid_search(query_emb, text, 0.4, 0.6, k)` for substantially higher recall. The vector stage is served by the HNSW index by default (the `hnsw` feature is on by default); build with `--no-default-features --features float16` to fall back to an exact linear cosine scan.
 ### Memory Footprint
@@ -194,7 +231,7 @@ ClawhDF5's agent memory engine implements research from 15+ recent papers on age
 | **`ivf` / `pq`** | IVF-PQ approximate nearest neighbor for billion-scale search |
 | **`bm25`** | BM25 keyword index with TF-IDF scoring |
 | **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph |
-| **`wal`** | Write-ahead log for crash-safe persistence |
+| **`wal`** | Write-ahead log for crash-safe persistence; each entry is CRC32-checked on replay, so a corrupted entry stops replay there instead of loading bad data |
 | **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection |
 | **`decision_gate`** | Sub-microsecond trivial/substantive classification |
 | **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) |
@@ -338,22 +375,22 @@ let exported = backend.export_markdown("MEMORY.md")?;
 ## Crate Map
 ```
-clawhdf5 workspace (17 crates, 84K lines of Rust)
+clawhdf5 workspace (16 crates, ~92K lines of Rust; plus libaec-sys, an
                     internal FFI bindings crate for the optional szip feature)
 │
 ├── Core HDF5
-│   ├── clawhdf5-types      — Type system definitions
+│   ├── clawhdf5-format      — Binary parser/writer (no_std), shared type definitions
 │   ├── clawhdf5-format      — Binary parser/writer (no_std)
 │   ├── clawhdf5-io          — I/O abstraction (buffered, mmap, async)
-│   ├── clawhdf5-filters     — Compression (deflate, lz4, zstd, blosc)
+│   ├── clawhdf5-filters     — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format
 │   ├── clawhdf5-derive      — Proc macros
 │   ├── clawhdf5             — High-level API
 │   ├── clawhdf5-netcdf4     — NetCDF-4 support
 │   ├── clawhdf5-accel       — SIMD (NEON, AVX2, AVX-512)
-│   └── clawhdf5-gpu         — GPU compute (wgpu)
+│   └── clawhdf5-gpu         — GPU compute (wgpu, hand-written WGSL compute shaders)
 │
 ├── Agent Memory
-│   ├── clawhdf5-agent       — Memory engine (20.7K lines, 32 modules)
+│   ├── clawhdf5-agent       — Memory engine (20.9K lines, 32 modules; WAL is CRC32-checked per entry)
-│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor (default backend)
+│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor (default backend; optional `parallel` feature)
 │   ├── clawhdf5-migrate     — SQLite → HDF5 migration
 │   ├── clawhdf5-android     — Android JNI bridge
 │   └── clawhdf5-cli         — CLI tool
@@ -420,6 +457,24 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
 | `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
 | `blake3_hash` | no | BLAKE3 content hashing for provenance |
 ### `clawhdf5-ann`
 | Flag | Default | Description |
 |------|---------|-------------|
 | `parallel` | no | Rayon-parallel neighbor-distance computation during HNSW graph pruning |
 ### `clawhdf5-io`
 | Flag | Default | Description |
 |------|---------|-------------|
 | `mpi-io` | no | MPI-backed I/O via the `mpi` crate |
 > **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read
 > followed by a broadcast, and its write path gathers all ranks' shards to
 > rank 0 before writing — not true collective I/O
 > (`MPI_File_read_at_all`/`write_at_all`). It does not provide I/O bandwidth
 > that scales with rank count; true collective I/O is tracked as future work.
 ---
 ## Building
@@ -435,7 +490,7 @@ cargo build -p clawhdf5-agent --features "agent,float16,parallel,fast-math"
 cargo build -p clawhdf5-agent --features "agent,float16,accelerate,parallel,gpu"
 # Tests
-cargo test --workspace            # all 417+ tests
+cargo test --workspace            # all 1,650+ tests
 cargo test -p clawhdf5-agent      # agent memory tests
 # Benchmarks
@@ -505,7 +560,7 @@ See [ROADMAP.md](ROADMAP.md) for the full implementation tracker.
 - ✅ OpenClaw integration layer
 - ✅ Comprehensive Criterion benchmarks
-**Phase 2** — OpenClaw TypeScript bridge, academic benchmarks (MemoryArena, LongMemEval), cross-platform validation.
+**Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: publish the OpenClaw TypeScript bridge to npm, crates.io/PyPI publishing.
 ---
@@ -523,5 +578,5 @@ MIT
 <p align="center">
  <em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em><br>
-  <em>72,087 lines of Rust. Zero C dependencies. One file to remember everything.</em>
+  <em>~92,000 lines of Rust. Zero C dependencies. One file to remember everything.</em>
 </p>
@@ -145,19 +145,36 @@
 **Phase 3:** ~~Track 6 (multi-modal) + Track 7 (OpenClaw integration)~~ 🟢 Complete
 **Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
-All 8 tracks delivered. 1,546 tests passing, zero clippy warnings.
+All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.
 ---
 ## What's Next
-Verified against current repo state on 2026-08-03 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped):
+Verified against current repo state on 2026-08-05 (see also `docs/superpowers/plans/` for the filter-codec/format-write/MPI-IO work, now shipped):
- [ ] CI/CD pipeline — still no GitHub/Gitea Actions workflow in the repo; automated testing is manual only
+- [ ] TypeScript bridge not wired into CI — `packages/clawhdf5-node/` already has a complete, working napi-rs package (package.json, tsconfig, hand-written TS wrapper matching all 21 `#[napi]` items, Jest test suite, README); it isn't published to npm and has no committed lockfile
 - [ ] Academic benchmark cross-validation — reproduce MemX/LongMemEval under identical conditions
 - [ ] TypeScript bridge — `clawhdf5-napi` has no `package.json`; it's still Rust-only scaffolding, not a publishable npm package
 - [ ] Publish crates to crates.io — no `publish` config anywhere in the workspace yet
 - [ ] Python wheel distribution via maturin — `crates/clawhdf5-py/pyproject.toml` exists (maturin-buildable locally) but wheels aren't published anywhere
 - [ ] `chunked_read.rs`/`data_read.rs` full bounds-check audit + scheduled fuzz campaigns (the new `fuzz_dataset_read` target covers the two files' main entry points; a full manual audit of every indexing site is still open) — see Tier 4 below
 - [ ] WAL per-entry checksum landed as CRC32 (see below); a stronger per-entry format (explicit length prefix, avoiding the read-then-verify restructuring) could still be revisited if profiling shows it matters
 - [ ] HNSW build parallelism is still narrow (only `prune_connections`); the correctness-sensitive outer insert loop needs its own dedicated design pass before parallelizing
 ### Recently closed out (2026-08-05, Tier 3–4 hardening pass)
 - [x] Academic benchmark cross-validation — LongMemEval reproduced against MemX on tank (Ryzen 7 7800X3D): turn-level Hit@5 84.4% vs MemX's 51.6%; recall numbers are deterministic and reproduce exactly across machines. SIMD/Parallelism and Vector Search sections also re-run and dated. See [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05)
 - [x] Android JNI (`clawhdf5-android`): validate `embedding_len`/`query_embedding_len` against the handle's configured `embedding_dim` before constructing a slice from a raw pointer
 - [x] `clawhdf5-py`: bumped pyo3/numpy 0.28 → 0.29, clearing two RUSTSEC advisories
 - [x] WAL (`clawhdf5-agent`): length-prefix caps (`MAX_WAL_FIELD_LEN`) to reject a corrupted length claim before allocating, then a full per-entry CRC32 trailer (`WAL_VERSION` 2) so a bit-flip stops replay cleanly instead of loading corrupted data; old-format WAL files still read correctly and are migrated on next open
 - [x] `chunked_read.rs`/`data_read.rs`/`local_heap.rs` bounds-check audit: added `ensure_len` overflow guards, a recursion-depth guard against cyclic B-trees, and a fix for an unguarded compound-datatype byte-offset overrun. Added a new `fuzz_dataset_read` cargo-fuzz target exercising the contiguous/chunked/compact read paths — it found and we fixed 3 real crash bugs (integer-overflow panics) within the first few runs
 - [x] `clawhdf5-ann`: optional `parallel` feature (rayon) for HNSW's `prune_connections` neighbor-distance computation
 - [x] `[workspace.dependencies]` added for `tempfile`/`criterion`/`half`/`serde`, fixing a real version skew on `half` (2 vs 2.7)
 ### Recently closed out (2026-08-05 hardening pass)
 - [x] CI/CD pipeline — `.gitea/workflows/ci.yml` now runs `scripts/ci-test.sh` (fmt, clippy, tests, no_std check) on push/PR to `main`
 - [x] Fixed no_std build breakage in `clawhdf5-format` (missing alloc imports, `AtomicU64` unsupported on thumbv7em, `f64::powi` requiring std/libm)
 - [x] Fixed version skew: `clawhdf5-py` (pyproject.toml) and `packages/clawhdf5-node` (package.json) were both behind the actual crate version
 ### Recently closed out (2026-08-03 cleanup pass)
@@ -167,4 +184,4 @@ Verified against current repo state on 2026-08-03 (see also `docs/superpowers/pl
 ---
-_Last updated: 2026-08-03_
+_Last updated: 2026-08-05_
@@ -0,0 +1,38 @@
 #!/usr/bin/env python3
 """h5py counterpart to worldmodel_sampling.rs — same file, same shuffled
 per-frame access, same minimal touch (sum the frame bytes). Reports
 samples/sec so the two sit side by side on one machine."""
 import sys, time, numpy as np, h5py
 path = sys.argv[1]
 passes = int(sys.argv[2]) if len(sys.argv) > 2 else 5
 def shuffled(n):
    v = list(range(n))
    state = 0x9E3779B97F4A7C15
    for i in range(n - 1, 0, -1):
        state = (state * 6364136223846793005 + 1442695040888963407) & 0xFFFFFFFFFFFFFFFF
        j = (state >> 33) % (i + 1)
        v[i], v[j] = v[j], v[i]
    return v
 # swmr + a 256 MB chunk cache: exactly stable-worldmodel's HDF5Dataset._open_h5.
 f = h5py.File(path, "r", swmr=True, rdcc_nbytes=256 * 1024 * 1024)
 d = f["observation"]
 n = d.shape[0]
 order = shuffled(n)
 # warm
 sink = 0
 for i in order:
    sink += int(d[i].sum())
 t0 = time.perf_counter()
 sink = 0
 for _ in range(passes):
    for i in order:
        sink += int(d[i].sum())
 elapsed = time.perf_counter() - t0
 total = n * passes
 print(f"h5py: {n} frames x {passes} passes = {total} reads in {elapsed:.3f}s")
 print(f"h5py: {total/elapsed:.0f} samples/sec")
@@ -0,0 +1,27 @@
 #!/usr/bin/env python3
 """Generate a world-model-shaped dataset: N frames of HxWxC uint8 observations,
 contiguous (N,H,W,C), matching stable-worldmodel's per-frame sample-loading
 access pattern. Also emits ep_len/ep_offset like their format."""
 import sys, time, numpy as np, h5py
 path = sys.argv[1]
 N = int(sys.argv[2]) if len(sys.argv) > 2 else 20000
 H = W = 64
 C = 3
 rng = np.random.default_rng(0)
 t0 = time.perf_counter()
 with h5py.File(path, "w", libver="latest") as f:
    # Contiguous (N,H,W,C) uint8 — the fair, both-APIs-support-it layout.
    obs = f.create_dataset("observation", shape=(N, H, W, C), dtype=np.uint8)
    # Write in blocks to bound memory.
    B = 2000
    for i in range(0, N, B):
        n = min(B, N - i)
        obs[i:i+n] = rng.integers(0, 256, size=(n, H, W, C), dtype=np.uint8)
    # Episode metadata like their format: 100-step episodes.
    ep = 100
    n_ep = N // ep
    f.create_dataset("ep_len", data=np.full(n_ep, ep, dtype=np.int32))
    f.create_dataset("ep_offset", data=(np.arange(n_ep) * ep).astype(np.int64))
 print(f"wrote {N} frames {H}x{W}x{C} to {path} in {time.perf_counter()-t0:.1f}s "
      f"({N*H*W*C/1e6:.0f} MB)")
@@ -15,7 +15,7 @@ float16 = ["dep:half"]
 avx512 = []
 [dependencies]
-half = { version = "2", optional = true }
+half = { workspace = true, optional = true }
 [package.metadata.docs.rs]
 features = []
@@ -13,7 +13,8 @@ use std::arch::x86_64::*;
 /// Caller must verify is_x86_feature_detected!("avx512f").
 // SAFETY: Caller must have verified avx512f via is_x86_feature_detected!.
 #[target_feature(enable = "avx512f")]
-pub unsafe fn dot_product(a: &[f32], b: &[f32]) -> f32 { unsafe {
+pub unsafe fn dot_product(a: &[f32], b: &[f32]) -> f32 {
    unsafe {
        assert_eq!(a.len(), b.len());
        let len = a.len();
        let mut i = 0;
@@ -48,7 +49,8 @@ pub unsafe fn dot_product(a: &[f32], b: &[f32]) -> f32 { unsafe {
        }
        sum
-}}
+    }
 }
 /// AVX-512 cosine similarity — fused single pass.
 ///
@@ -56,7 +58,8 @@ pub unsafe fn dot_product(a: &[f32], b: &[f32]) -> f32 { unsafe {
 /// Caller must verify is_x86_feature_detected!("avx512f").
 // SAFETY: Caller must have verified avx512f via is_x86_feature_detected!.
 #[target_feature(enable = "avx512f")]
-pub unsafe fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 { unsafe {
+pub unsafe fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
    unsafe {
        assert_eq!(a.len(), b.len());
        let len = a.len();
        let mut i = 0;
@@ -87,7 +90,8 @@ pub unsafe fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 { unsafe {
        let denom = (norm_a * norm_b).sqrt();
        if denom == 0.0 { 0.0 } else { dot / denom }
-}}
+    }
 }
 /// AVX-512 L2 distance.
 ///
@@ -95,7 +99,8 @@ pub unsafe fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 { unsafe {
 /// Caller must verify is_x86_feature_detected!("avx512f").
 // SAFETY: Caller must have verified avx512f via is_x86_feature_detected!.
 #[target_feature(enable = "avx512f")]
-pub unsafe fn l2_distance(a: &[f32], b: &[f32]) -> f32 { unsafe {
+pub unsafe fn l2_distance(a: &[f32], b: &[f32]) -> f32 {
    unsafe {
        assert_eq!(a.len(), b.len());
        let len = a.len();
        let mut i = 0;
@@ -118,4 +123,5 @@ pub unsafe fn l2_distance(a: &[f32], b: &[f32]) -> f32 { unsafe {
        }
        sum.sqrt()
-}}
+    }
 }
@@ -16,9 +16,9 @@ clawhdf5-io = { path = "../clawhdf5-io", version = "2.1.0", features = ["mmap"]
 clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.1.0" }
 clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.1.0", optional = true }
 clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.1.0", optional = true, default-features = false }
-serde = { version = "1", features = ["derive"] }
+serde = { workspace = true }
 byteorder = "1"
-half = { version = "2", optional = true }
+half = { workspace = true, optional = true }
 rayon = { version = "1", optional = true }
 matrixmultiply = { version = "0.3", optional = true }
 cblas-sys = { version = "0.1", optional = true }
@@ -31,8 +31,8 @@ accelerate-src = { version = "0.3", optional = true }
 openblas-src = { version = "0.10", optional = true, features = ["cblas"] }
 [dev-dependencies]
-tempfile = "3"
+tempfile = { workspace = true }
-criterion = "0.5"
+criterion = { workspace = true }
 rayon = "1"
 tokio = { version = "1", features = ["rt-multi-thread", "sync", "macros"] }
@@ -116,7 +116,8 @@ impl GpuSearchBackend {
            // If we don't have an accelerator but now above threshold, try init
            if vectors.len() >= self.threshold
-                && let Ok(mut accel) = clawhdf5_gpu::GpuAccelerator::new() {
+                && let Ok(mut accel) = clawhdf5_gpu::GpuAccelerator::new()
            {
                let flat: Vec<f32> = vectors.iter().flat_map(|v| v.iter().copied()).collect();
                if accel.upload_vectors(&flat, self.dim).is_ok()
                    && accel.upload_norms(norms).is_ok()
@@ -1,7 +1,9 @@
 //! Memory provenance tracking and integrity verification.
 //!
-//! Records the origin, authorship, and integrity of every memory chunk
+//! Records the origin, authorship, and a content hash of every memory chunk
-//! so the system can detect tampering and trace data lineage.
+//! so the system can detect *accidental* corruption and trace data lineage.
 //! The hash is unkeyed (see [`fnv1a_64`]) — this is not a tamper-evidence or
 //! authenticity guarantee.
 use std::collections::HashMap;
@@ -11,6 +13,10 @@ pub use crate::consolidation::MemorySource;
 // Hash helper (std-only FNV-1a 64-bit)
 // ---------------------------------------------------------------------------
 /// Unkeyed, non-cryptographic FNV-1a hash for detecting accidental content
 /// corruption. It is trivially forgeable by anyone able to modify the stored
 /// data, since they can recompute and overwrite the stored hash alongside
 /// it — do not rely on this as a tamper-evidence or authenticity control.
 fn fnv1a_64(text: &str) -> u64 {
    const OFFSET: u64 = 14_695_981_039_346_656_037;
    const PRIME: u64 = 1_099_511_628_211;
@@ -114,6 +120,11 @@ impl ProvenanceStore {
    /// Re-hash `current_chunk` and compare against the stored hash.
    /// Returns `true` if the content matches (integrity intact).
    ///
    /// This only detects accidental corruption: the hash is unkeyed, so an
    /// actor able to modify the stored chunk can also recompute and
    /// overwrite the stored hash. Do not treat a `true` result as proof the
    /// data hasn't been tampered with.
    pub fn verify_integrity(&self, record_id: u64, current_chunk: &str) -> bool {
        match self.records.get(&record_id) {
            Some(p) => p.content_hash == fnv1a_64(current_chunk),
@@ -26,9 +26,7 @@ impl HDF5Memory {
    ) -> Vec<(usize, f32)> {
        self.ensure_hnsw_fresh();
        match self.hnsw.as_ref() {
-            Some(index)
+            Some(index) if !index.is_empty() && index.dimension() == query_embedding.len() => {
                if !index.is_empty() && index.dimension() == query_embedding.len() =>
            {
                // Over-fetch so the merge sees a useful vector pool; cosine
                // distance from the index converts back to similarity (1 - d).
                let pool = (k * 8).max(64);
@@ -38,7 +36,13 @@ impl HDF5Memory {
                    .map(|(id, dist)| (id, 1.0 - dist))
                    .collect();
                let kw_scores = bm25.search(query_text, self.cache.len());
-                hybrid::merge_vector_keyword(vec_scores, kw_scores, vector_weight, keyword_weight, k)
+                hybrid::merge_vector_keyword(
                    vec_scores,
                    kw_scores,
                    vector_weight,
                    keyword_weight,
                    k,
                )
            }
            _ => hybrid::hybrid_search(
                query_embedding,
@@ -7,10 +7,29 @@ use std::fs::{File, OpenOptions};
 use std::io::{Read, Seek, SeekFrom, Write};
 use std::path::{Path, PathBuf};
 use clawhdf5_format::checksum::crc32;
 use crate::MemoryError;
 const WAL_MAGIC: [u8; 4] = [0x45, 0x48, 0x57, 0x4C]; // "EHWL"
-const WAL_VERSION: u8 = 1;
+
 /// Current WAL format version: every entry ends with a 4-byte CRC32 trailer
 /// (see [`TeeReader`]) so a bit-flip is detected and replay stops there
 /// instead of silently accepting corrupted data.
 const WAL_VERSION: u8 = 2;
 /// The only other WAL version this crate still knows how to *read*: no
 /// per-entry CRC trailer. Written by versions of this crate before the CRC32
 /// hardening. `WalFile::open` migrates a legacy file to [`WAL_VERSION`] by
 /// recreating it fresh — safe because every real call site reads existing
 /// entries via [`WalFile::read_entries`] before calling `open` (see
 /// `HDF5Memory::open`), so no data is lost.
 const WAL_VERSION_LEGACY_NO_CRC: u8 = 1;
 /// Upper bound on a single length-prefixed WAL field (string bytes, or
 /// embedding element count), to reject a corrupted/truncated WAL length
 /// claim before allocating a large buffer for it.
 const MAX_WAL_FIELD_LEN: usize = 64 * 1024 * 1024;
 #[repr(u8)]
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -62,6 +81,11 @@ pub struct WalFile {
 impl WalFile {
    /// Open or create a WAL file. If it exists, read the header and entry count.
    ///
    /// A legacy (pre-CRC) WAL file is migrated to the current format by
    /// recreating it fresh — see [`WAL_VERSION_LEGACY_NO_CRC`]. Callers that
    /// need the legacy file's entries must call [`WalFile::read_entries`]
    /// first, before calling `open`.
    pub fn open(path: &Path) -> Result<Self, MemoryError> {
        if path.exists() {
            // Read existing header
@@ -77,12 +101,8 @@ impl WalFile {
            }
            let mut ver = [0u8; 1];
            f.read_exact(&mut ver)?;
-            if ver[0] != WAL_VERSION {
+            match ver[0] {
-                return Err(MemoryError::Schema(format!(
+                WAL_VERSION => {
                    "unsupported WAL version {}",
                    ver[0]
                )));
            }
                    let mut count_buf = [0u8; 4];
                    f.read_exact(&mut count_buf)?;
                    let entry_count = u32::from_le_bytes(count_buf);
@@ -94,13 +114,21 @@ impl WalFile {
                        entry_count,
                        pending_header_sync: 0,
                    })
                }
                WAL_VERSION_LEGACY_NO_CRC => {
                    drop(f);
                    let f = create_fresh_wal_file(path)?;
                    Ok(Self {
                        path: path.to_path_buf(),
                        file: Some(f),
                        entry_count: 0,
                        pending_header_sync: 0,
                    })
                }
                v => Err(MemoryError::Schema(format!("unsupported WAL version {v}"))),
            }
        } else {
-            // Create new WAL
+            let f = create_fresh_wal_file(path)?;
            let mut f = File::create(path)?;
            f.write_all(&WAL_MAGIC)?;
            f.write_all(&[WAL_VERSION])?;
            f.write_all(&0u32.to_le_bytes())?;
            f.flush()?;
            Ok(Self {
                path: path.to_path_buf(),
                file: Some(f),
@@ -140,6 +168,9 @@ impl WalFile {
        serialize_str(&mut buf, &entry.session_id);
        serialize_str(&mut buf, &entry.tags);
        let crc = crc32(&buf);
        buf.extend_from_slice(&crc.to_le_bytes());
        let f = self
            .file
            .as_mut()
@@ -156,10 +187,12 @@ impl WalFile {
    /// Append a tombstone entry (deletion).
    pub fn append_tombstone(&mut self, index: usize, timestamp: f64) -> Result<(), MemoryError> {
-        let mut buf = [0u8; 1 + 8 + 4]; // type + timestamp + index
+        let mut buf = [0u8; 1 + 8 + 4 + 4]; // type + timestamp + index + crc32
        buf[0] = WalEntryType::Tombstone as u8;
        buf[1..9].copy_from_slice(&timestamp.to_le_bytes());
        buf[9..13].copy_from_slice(&(index as u32).to_le_bytes());
        let crc = crc32(&buf[..13]);
        buf[13..17].copy_from_slice(&crc.to_le_bytes());
        let f = self
            .file
@@ -180,7 +213,9 @@ impl WalFile {
    /// Reads until EOF — the header `entry_count` is used only for pre-allocation
    /// (and may be stale if written with deferred group-commit updates). This
    /// tolerates both truncated files (crash mid-write) and stale header counts
-    /// (crash before the next group-commit header sync).
+    /// (crash before the next group-commit header sync). On a `WAL_VERSION`
    /// file, a CRC32 mismatch on an entry is treated the same way — replay
    /// stops there rather than accepting corrupted data.
    pub fn read_entries(path: &Path) -> Result<Vec<WalEntry>, MemoryError> {
        if !path.exists() {
            return Ok(Vec::new());
@@ -192,81 +227,45 @@ impl WalFile {
        if header[0..4] != WAL_MAGIC {
            return Err(MemoryError::Schema("invalid WAL magic bytes".into()));
        }
        if header[4] != WAL_VERSION {
            return Err(MemoryError::Schema(format!(
                "unsupported WAL version {}",
                header[4]
            )));
        }
        // entry_count is a pre-allocation hint only — we read until EOF.
        let entry_count_hint = u32::from_le_bytes([header[5], header[6], header[7], header[8]]);
        let mut entries = Vec::with_capacity(entry_count_hint as usize);
-        loop {
+        match header[4] {
-            // Read entry type — EOF here is normal end-of-log, not an error
+            WAL_VERSION => loop {
-            let mut type_buf = [0u8; 1];
+                let raw_and_result = {
-            if f.read_exact(&mut type_buf).is_err() {
+                    let mut tee = TeeReader::new(&mut f);
                    let result = read_one_entry(&mut tee);
                    (tee.into_buf(), result)
                };
                let (raw, result) = raw_and_result;
                let entry_opt = match result {
                    Err(()) => break,
                    Ok(v) => v,
                };
                let mut crc_buf = [0u8; 4];
                if f.read_exact(&mut crc_buf).is_err() {
                    break;
                }
-            let entry_type = match WalEntryType::from_u8(type_buf[0]) {
+                let stored_crc = u32::from_le_bytes(crc_buf);
-                Some(et) => et,
+                if crc32(&raw) != stored_crc {
-                None => break,
+                    // Corruption detected — stop replay here, same as a clean
-            };
+                    // truncation/EOF, rather than accepting the bad entry.
            let mut ts_buf = [0u8; 8];
            if f.read_exact(&mut ts_buf).is_err() {
                    break;
                }
-            let timestamp = f64::from_le_bytes(ts_buf);
+                if let Some(entry) = entry_opt {
-
+                    entries.push(entry);
            match entry_type {
                WalEntryType::Save => {
                    let Ok(chunk) = read_len_prefixed_str(&mut f) else {
                        break;
                    };
                    let Ok(embedding) = read_embedding(&mut f) else {
                        break;
                    };
                    let Ok(source_channel) = read_len_prefixed_str(&mut f) else {
                        break;
                    };
                    let Ok(session_id) = read_len_prefixed_str(&mut f) else {
                        break;
                    };
                    let Ok(tags) = read_len_prefixed_str(&mut f) else {
                        break;
                    };
                    entries.push(WalEntry {
                        entry_type,
                        timestamp,
                        chunk,
                        embedding,
                        source_channel,
                        session_id,
                        tags,
                        tombstone_index: None,
                    });
                }
-                WalEntryType::Tombstone => {
+            },
-                    let mut idx_buf = [0u8; 4];
+            WAL_VERSION_LEGACY_NO_CRC => loop {
-                    if f.read_exact(&mut idx_buf).is_err() {
+                match read_one_entry(&mut f) {
-                        break;
+                    Err(()) => break,
-                    }
+                    Ok(Some(entry)) => entries.push(entry),
-                    let idx = u32::from_le_bytes(idx_buf) as usize;
+                    Ok(None) => {}
                    entries.push(WalEntry {
                        entry_type,
                        timestamp,
                        chunk: String::new(),
                        embedding: Vec::new(),
                        source_channel: String::new(),
                        session_id: String::new(),
                        tags: String::new(),
                        tombstone_index: Some(idx),
                    });
                }
                WalEntryType::ActivationUpdate => {
                    // Reserved for future use
                }
            },
            v => {
                return Err(MemoryError::Schema(format!("unsupported WAL version {v}")));
            }
        }
        Ok(entries)
@@ -276,11 +275,7 @@ impl WalFile {
    pub fn truncate(&mut self) -> Result<(), MemoryError> {
        // Close existing handle and recreate
        self.file = None;
-        let mut f = File::create(&self.path)?;
+        let f = create_fresh_wal_file(&self.path)?;
        f.write_all(&WAL_MAGIC)?;
        f.write_all(&[WAL_VERSION])?;
        f.write_all(&0u32.to_le_bytes())?;
        f.flush()?;
        self.file = Some(f);
        self.entry_count = 0;
        self.pending_header_sync = 0;
@@ -345,19 +340,30 @@ fn serialize_str(buf: &mut Vec<u8>, s: &str) {
    buf.extend_from_slice(bytes);
 }
-fn read_len_prefixed_str(f: &mut File) -> Result<String, MemoryError> {
+fn read_len_prefixed_str<R: Read>(f: &mut R) -> Result<String, MemoryError> {
    let mut len_buf = [0u8; 4];
    f.read_exact(&mut len_buf)?;
    let len = u32::from_le_bytes(len_buf) as usize;
    if len > MAX_WAL_FIELD_LEN {
        return Err(MemoryError::Schema(format!(
            "WAL string field length {len} exceeds max {MAX_WAL_FIELD_LEN}"
        )));
    }
    let mut buf = vec![0u8; len];
    f.read_exact(&mut buf)?;
    String::from_utf8(buf).map_err(|e| MemoryError::Schema(format!("invalid UTF-8 in WAL: {e}")))
 }
-fn read_embedding(f: &mut File) -> Result<Vec<f32>, MemoryError> {
+fn read_embedding<R: Read>(f: &mut R) -> Result<Vec<f32>, MemoryError> {
    let mut len_buf = [0u8; 4];
    f.read_exact(&mut len_buf)?;
    let count = u32::from_le_bytes(len_buf) as usize;
    if count > MAX_WAL_FIELD_LEN / 4 {
        return Err(MemoryError::Schema(format!(
            "WAL embedding element count {count} exceeds max {}",
            MAX_WAL_FIELD_LEN / 4
        )));
    }
    let mut vals = Vec::with_capacity(count);
    for _ in 0..count {
        let mut val_buf = [0u8; 4];
@@ -367,6 +373,99 @@ fn read_embedding(f: &mut File) -> Result<Vec<f32>, MemoryError> {
    Ok(vals)
 }
 /// Create a fresh WAL file at `path` with the current-version header,
 /// truncating/overwriting anything already there.
 fn create_fresh_wal_file(path: &Path) -> Result<File, MemoryError> {
    let mut f = File::create(path)?;
    f.write_all(&WAL_MAGIC)?;
    f.write_all(&[WAL_VERSION])?;
    f.write_all(&0u32.to_le_bytes())?;
    f.flush()?;
    Ok(f)
 }
 /// Wraps a [`Read`]er, accumulating every byte actually consumed (including
 /// via `read_exact`, which is implemented in terms of `read`) into an
 /// internal buffer — used to capture a WAL entry's raw bytes for CRC32
 /// verification without needing to know its length up front.
 struct TeeReader<'a, R: Read> {
    inner: &'a mut R,
    buf: Vec<u8>,
 }
 impl<'a, R: Read> TeeReader<'a, R> {
    fn new(inner: &'a mut R) -> Self {
        Self {
            inner,
            buf: Vec::new(),
        }
    }
    fn into_buf(self) -> Vec<u8> {
        self.buf
    }
 }
 impl<R: Read> Read for TeeReader<'_, R> {
    fn read(&mut self, out: &mut [u8]) -> std::io::Result<usize> {
        let n = self.inner.read(out)?;
        self.buf.extend_from_slice(&out[..n]);
        Ok(n)
    }
 }
 /// Read one WAL entry (type + timestamp + type-specific payload) from `r`.
 ///
 /// Returns `Ok(None)` for entry types with no representable `WalEntry` (only
 /// `ActivationUpdate`, reserved for future use). Returns `Err(())` on any
 /// read failure or unrecognized entry type — the caller treats this the same
 /// as a clean end-of-log (crash-mid-write tolerance).
 fn read_one_entry<R: Read>(r: &mut R) -> Result<Option<WalEntry>, ()> {
    let mut type_buf = [0u8; 1];
    r.read_exact(&mut type_buf).map_err(|_| ())?;
    let entry_type = WalEntryType::from_u8(type_buf[0]).ok_or(())?;
    let mut ts_buf = [0u8; 8];
    r.read_exact(&mut ts_buf).map_err(|_| ())?;
    let timestamp = f64::from_le_bytes(ts_buf);
    match entry_type {
        WalEntryType::Save => {
            let chunk = read_len_prefixed_str(r).map_err(|_| ())?;
            let embedding = read_embedding(r).map_err(|_| ())?;
            let source_channel = read_len_prefixed_str(r).map_err(|_| ())?;
            let session_id = read_len_prefixed_str(r).map_err(|_| ())?;
            let tags = read_len_prefixed_str(r).map_err(|_| ())?;
            Ok(Some(WalEntry {
                entry_type,
                timestamp,
                chunk,
                embedding,
                source_channel,
                session_id,
                tags,
                tombstone_index: None,
            }))
        }
        WalEntryType::Tombstone => {
            let mut idx_buf = [0u8; 4];
            r.read_exact(&mut idx_buf).map_err(|_| ())?;
            let idx = u32::from_le_bytes(idx_buf) as usize;
            Ok(Some(WalEntry {
                entry_type,
                timestamp,
                chunk: String::new(),
                embedding: Vec::new(),
                source_channel: String::new(),
                session_id: String::new(),
                tags: String::new(),
                tombstone_index: Some(idx),
            }))
        }
        WalEntryType::ActivationUpdate => Ok(None),
    }
 }
 // --- Tests ---
 #[cfg(test)]
@@ -427,6 +526,40 @@ mod tests {
        assert_eq!(entries[2].embedding, vec![5.0, 6.0]);
    }
    #[test]
    fn read_len_prefixed_str_rejects_oversized_len_claim() {
        let dir = TempDir::new().unwrap();
        let path = dir.path().join("oversized_str.bin");
        {
            let mut f = File::create(&path).unwrap();
            // Claim a length far beyond MAX_WAL_FIELD_LEN; no payload follows.
            f.write_all(&(u32::MAX).to_le_bytes()).unwrap();
        }
        let mut f = File::open(&path).unwrap();
        let result = read_len_prefixed_str(&mut f);
        assert!(
            matches!(result, Err(MemoryError::Schema(_))),
            "expected a clean Schema error, got {result:?}"
        );
    }
    #[test]
    fn read_embedding_rejects_oversized_count_claim() {
        let dir = TempDir::new().unwrap();
        let path = dir.path().join("oversized_embedding.bin");
        {
            let mut f = File::create(&path).unwrap();
            // Claim a count far beyond MAX_WAL_FIELD_LEN / 4; no payload follows.
            f.write_all(&(u32::MAX).to_le_bytes()).unwrap();
        }
        let mut f = File::open(&path).unwrap();
        let result = read_embedding(&mut f);
        assert!(
            matches!(result, Err(MemoryError::Schema(_))),
            "expected a clean Schema error, got {result:?}"
        );
    }
    #[test]
    fn test_wal_truncate() {
        let dir = TempDir::new().unwrap();
@@ -749,6 +882,86 @@ mod tests {
        assert!(err.contains("unsupported WAL version"), "got: {err}");
    }
    #[test]
    fn test_wal_v2_detects_corrupted_payload_and_stops_replay() {
        let dir = TempDir::new().unwrap();
        let wal_path = dir.path().join("test.h5.wal");
        let mut wal = WalFile::open(&wal_path).unwrap();
        wal.append_save(&make_wal_entry("first", &[1.0, 2.0]))
            .unwrap();
        let len_after_first = std::fs::metadata(&wal_path).unwrap().len();
        wal.append_save(&make_wal_entry("second", &[3.0, 4.0]))
            .unwrap();
        drop(wal);
        // Flip one byte inside the second entry's "second" chunk string
        // (well past the header and the first entry, and not touching any
        // length-prefix field) — this must be caught by the CRC32 trailer,
        // not by any length-cap guard.
        let mut bytes = std::fs::read(&wal_path).unwrap();
        let corrupt_at = len_after_first as usize + 15;
        bytes[corrupt_at] ^= 0xFF;
        std::fs::write(&wal_path, &bytes).unwrap();
        let entries = WalFile::read_entries(&wal_path).unwrap();
        assert_eq!(
            entries.len(),
            1,
            "the corrupted second entry must not be returned"
        );
        assert_eq!(entries[0].chunk, "first");
    }
    #[test]
    fn test_wal_reads_legacy_v1_format_without_crc() {
        let dir = TempDir::new().unwrap();
        let wal_path = dir.path().join("legacy.h5.wal");
        let mut buf = Vec::new();
        buf.extend_from_slice(&WAL_MAGIC);
        buf.push(WAL_VERSION_LEGACY_NO_CRC);
        buf.extend_from_slice(&1u32.to_le_bytes());
        // One Save entry in the old format: type + timestamp + fields, with
        // no trailing CRC32.
        buf.push(WalEntryType::Save as u8);
        buf.extend_from_slice(&42.0f64.to_le_bytes());
        serialize_str(&mut buf, "legacy-chunk");
        let embedding = [1.0f32, 2.0];
        buf.extend_from_slice(&(embedding.len() as u32).to_le_bytes());
        for v in embedding {
            buf.extend_from_slice(&v.to_le_bytes());
        }
        serialize_str(&mut buf, "chan");
        serialize_str(&mut buf, "sess");
        serialize_str(&mut buf, "tags");
        std::fs::write(&wal_path, &buf).unwrap();
        let entries = WalFile::read_entries(&wal_path).unwrap();
        assert_eq!(entries.len(), 1);
        assert_eq!(entries[0].chunk, "legacy-chunk");
        assert_eq!(entries[0].embedding, vec![1.0, 2.0]);
    }
    #[test]
    fn test_wal_open_migrates_legacy_v1_to_current_version() {
        let dir = TempDir::new().unwrap();
        let wal_path = dir.path().join("legacy.h5.wal");
        let mut buf = Vec::new();
        buf.extend_from_slice(&WAL_MAGIC);
        buf.push(WAL_VERSION_LEGACY_NO_CRC);
        buf.extend_from_slice(&0u32.to_le_bytes());
        std::fs::write(&wal_path, &buf).unwrap();
        let wal = WalFile::open(&wal_path).unwrap();
        assert!(wal.is_empty());
        drop(wal);
        let bytes = std::fs::read(&wal_path).unwrap();
        assert_eq!(
            bytes[4], WAL_VERSION,
            "legacy file must be migrated to the current version"
        );
    }
    #[test]
    fn test_wal_disabled() {
        let dir = TempDir::new().unwrap();
@@ -80,8 +80,7 @@ fn hnsw_matches_bruteforce_oracle() {
    oracle.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap());
    let oracle_ids: std::collections::HashSet<usize> =
        oracle.iter().take(k).map(|(i, _)| *i).collect();
-    let hnsw_ids: std::collections::HashSet<usize> =
+    let hnsw_ids: std::collections::HashSet<usize> = results.iter().map(|r| r.index).collect();
        results.iter().map(|r| r.index).collect();
    let overlap = oracle_ids.intersection(&hnsw_ids).count();
    assert!(
@@ -127,17 +126,19 @@ fn incremental_inserts_after_search_are_found() {
    // First batch, then a search to force the index to build.
    for i in 0..40 {
        let v = make_vector(&mut seed, dim);
-        mem.save(entry(&format!("a{i}"), v, &format!("a{i}"))).unwrap();
+        mem.save(entry(&format!("a{i}"), v, &format!("a{i}")))
            .unwrap();
    }
    let _ = mem.hybrid_search(&make_vector(&mut seed, dim), "", 1.0, 0.0, 5);
    // Now insert a distinctive vector incrementally and confirm we can find it.
    let needle = vec![10.0f32; dim];
-    let idx = mem
+    let idx = mem.save(entry("needle", needle.clone(), "needle")).unwrap();
        .save(entry("needle", needle.clone(), "needle"))
        .unwrap();
    let hits = mem.hybrid_search(&needle, "", 1.0, 0.0, 1);
-    assert_eq!(hits[0].index, idx, "incrementally inserted vector must be found");
+    assert_eq!(
        hits[0].index, idx,
        "incrementally inserted vector must be found"
    );
 }
 #[test]
@@ -158,6 +159,9 @@ fn save_batch_then_search_is_consistent() {
    // Exact-match queries should resolve to themselves after a batch insert.
    for probe in [0usize, 17, 49] {
        let hits = mem.hybrid_search(&vectors[probe], "", 1.0, 0.0, 1);
-        assert_eq!(hits[0].index, probe, "batch-inserted vector {probe} not found");
+        assert_eq!(
            hits[0].index, probe,
            "batch-inserted vector {probe} not found"
        );
    }
 }
@@ -10,3 +10,6 @@ crate-type = ["cdylib"]
 [dependencies]
 clawhdf5-agent = { path = "../clawhdf5-agent", default-features = false }
 [dev-dependencies]
 tempfile = { workspace = true }
@@ -92,11 +92,18 @@ pub unsafe extern "C" fn edgehdf5_close(handle: Handle) {
 /// Save a memory entry. Returns the entry index, or -1 on failure.
 ///
 /// `embedding_len` is validated against the handle's configured
 /// `embedding_dim` before the input slice is constructed; a mismatch fails
 /// the call with -1 rather than reading out of bounds. This is a length
 /// check only — it cannot detect a same-length buffer that is otherwise
 /// too short or invalid.
 ///
 /// # Safety
 ///
 /// - `handle` must be a valid, non-null handle.
 /// - All `*const c_char` arguments must be valid, null-terminated C strings.
-/// - `embedding_ptr` must point to at least `embedding_len` contiguous `f32` values.
+/// - If `embedding_len` matches the handle's `embedding_dim`, `embedding_ptr`
 ///   must point to at least that many contiguous, valid `f32` values.
 #[unsafe(no_mangle)]
 pub unsafe extern "C" fn edgehdf5_save(
    handle: Handle,
@@ -135,8 +142,14 @@ pub unsafe extern "C" fn edgehdf5_save(
        None => return -1,
    };
    if embedding_ptr.is_null() || embedding_len as usize != mem.config().embedding_dim {
        return -1;
    }
    let embedding =
-        // SAFETY: JNI caller guarantees embedding_ptr points to embedding_len valid f32 values.
+        // SAFETY: embedding_ptr is non-null and embedding_len matches the handle's configured
        // embedding_dim (checked above); JNI caller guarantees it points to that many valid f32
        // values. A mismatched-but-equal-length short buffer is not caught by this length check
        // alone — the caller is still responsible for pointer validity.
        unsafe { std::slice::from_raw_parts(embedding_ptr, embedding_len as usize) }.to_vec();
    let entry = MemoryEntry {
@@ -210,11 +223,18 @@ pub unsafe extern "C" fn edgehdf5_delete(handle: Handle, index: u64) -> i32 {
 /// Performs hybrid search and writes up to `max_results` entries into the
 /// provided output arrays. Returns the number of results written.
 ///
 /// `query_embedding_len` is validated against the handle's configured
 /// `embedding_dim` before the input slice is constructed; a mismatch fails
 /// the call (returns 0) rather than reading out of bounds. This is a length
 /// check only — it cannot detect a same-length buffer that is otherwise too
 /// short or invalid.
 ///
 /// # Safety
 ///
 /// - `handle` must be a valid, non-null handle.
 /// - `query_text` must be a valid, null-terminated C string.
-/// - `query_embedding_ptr` must point to at least `query_embedding_len` `f32` values.
+/// - If `query_embedding_len` matches the handle's `embedding_dim`,
 ///   `query_embedding_ptr` must point to at least that many valid `f32` values.
 /// - `out_indices` and `out_scores` must point to arrays of at least `max_results` elements.
 /// - `out_chunks` must be null or point to an array of at least `max_results` pointers.
 #[unsafe(no_mangle)]
@@ -240,8 +260,14 @@ pub unsafe extern "C" fn edgehdf5_hybrid_search(
        Some(s) => s,
        None => return 0,
    };
    if query_embedding_ptr.is_null() || query_embedding_len as usize != mem.config().embedding_dim {
        return 0;
    }
    let query_embedding =
-        // SAFETY: JNI caller guarantees query_embedding_ptr points to query_embedding_len valid f32 values.
+        // SAFETY: query_embedding_ptr is non-null and query_embedding_len matches the handle's
        // configured embedding_dim (checked above); JNI caller guarantees it points to that many
        // valid f32 values. A mismatched-but-equal-length short buffer is not caught by this
        // length check alone — the caller is still responsible for pointer validity.
        unsafe { std::slice::from_raw_parts(query_embedding_ptr, query_embedding_len as usize) };
    let results = mem.hybrid_search(
@@ -456,3 +482,112 @@ unsafe fn cstr_to_string(ptr: *const c_char) -> Option<String> {
        .ok()
        .map(String::from)
 }
 #[cfg(test)]
 mod tests {
    use super::*;
    const EMBEDDING_DIM: u32 = 4;
    fn open_handle(dir: &tempfile::TempDir) -> Handle {
        let path = CString::new(dir.path().join("mem.h5").to_str().unwrap()).unwrap();
        let agent_id = CString::new("test-agent").unwrap();
        // SAFETY: both C strings are valid and null-terminated.
        unsafe { edgehdf5_create(path.as_ptr(), agent_id.as_ptr(), EMBEDDING_DIM) }
    }
    #[test]
    fn save_rejects_mismatched_embedding_len() {
        let dir = tempfile::tempdir().unwrap();
        let handle = open_handle(&dir);
        assert!(!handle.is_null());
        let embedding = [1.0f32, 2.0, 3.0]; // len 3, dim is 4
        let chunk = CString::new("hello").unwrap();
        let channel = CString::new("test").unwrap();
        let session = CString::new("s1").unwrap();
        let tags = CString::new("").unwrap();
        // SAFETY: handle is valid; all C strings are valid; embedding_len (3) intentionally
        // does not match embedding_dim (4), which edgehdf5_save must reject before touching
        // embedding_ptr.
        let result = unsafe {
            edgehdf5_save(
                handle,
                chunk.as_ptr(),
                embedding.as_ptr(),
                embedding.len() as u32,
                channel.as_ptr(),
                0.0,
                session.as_ptr(),
                tags.as_ptr(),
            )
        };
        assert_eq!(result, -1, "mismatched embedding_len must be rejected");
        unsafe { edgehdf5_close(handle) };
    }
    #[test]
    fn save_rejects_null_embedding_ptr() {
        let dir = tempfile::tempdir().unwrap();
        let handle = open_handle(&dir);
        assert!(!handle.is_null());
        let chunk = CString::new("hello").unwrap();
        let channel = CString::new("test").unwrap();
        let session = CString::new("s1").unwrap();
        let tags = CString::new("").unwrap();
        // SAFETY: handle and C strings are valid; embedding_ptr is intentionally null, which
        // edgehdf5_save must reject before constructing a slice from it.
        let result = unsafe {
            edgehdf5_save(
                handle,
                chunk.as_ptr(),
                ptr::null(),
                EMBEDDING_DIM,
                channel.as_ptr(),
                0.0,
                session.as_ptr(),
                tags.as_ptr(),
            )
        };
        assert_eq!(result, -1, "null embedding_ptr must be rejected");
        unsafe { edgehdf5_close(handle) };
    }
    #[test]
    fn hybrid_search_rejects_mismatched_embedding_len() {
        let dir = tempfile::tempdir().unwrap();
        let handle = open_handle(&dir);
        assert!(!handle.is_null());
        let query_embedding = [1.0f32, 2.0]; // len 2, dim is 4
        let query_text = CString::new("hello").unwrap();
        let mut out_indices = [0u64; 4];
        let mut out_scores = [0.0f32; 4];
        // SAFETY: handle and query_text are valid; query_embedding_len (2) intentionally does
        // not match embedding_dim (4), which edgehdf5_hybrid_search must reject before touching
        // query_embedding_ptr. Output buffers are sized to max_results.
        let count = unsafe {
            edgehdf5_hybrid_search(
                handle,
                query_embedding.as_ptr(),
                query_embedding.len() as u32,
                query_text.as_ptr(),
                0.7,
                0.3,
                4,
                out_indices.as_mut_ptr(),
                out_scores.as_mut_ptr(),
                ptr::null_mut(),
            )
        };
        assert_eq!(count, 0, "mismatched query_embedding_len must be rejected");
        unsafe { edgehdf5_close(handle) };
    }
 }
@@ -12,3 +12,7 @@ categories = ["algorithms", "science"]
 [dependencies]
 clawhdf5-format = { path = "../clawhdf5-format", version = "2.1.0" }
 clawhdf5-io = { path = "../clawhdf5-io", version = "2.1.0" }
 rayon = { version = "1", optional = true }
 [features]
 parallel = ["rayon"]
@@ -379,7 +379,13 @@ impl HnswIndex {
        // Phase 1: greedy descent from the top down to node_level + 1.
        for layer in (node_level + 1..=ep_level).rev() {
-            ep = greedy_closest(&self.vectors, &self.graph[layer], &self.vectors[id], ep, self.metric);
+            ep = greedy_closest(
                &self.vectors,
                &self.graph[layer],
                &self.vectors[id],
                ep,
                self.metric,
            );
        }
        // Phase 2: search and connect from min(node_level, ep_level) down to 0.
@@ -851,6 +857,15 @@ fn prune_connections(
    if neighbors.len() <= max_conn {
        return;
    }
    #[cfg(feature = "parallel")]
    let mut scored: Vec<(usize, f32)> = {
        use rayon::prelude::*;
        neighbors
            .par_iter()
            .map(|&n| (n, compute_distance(&vectors[node], &vectors[n], metric)))
            .collect()
    };
    #[cfg(not(feature = "parallel"))]
    let mut scored: Vec<(usize, f32)> = neighbors
        .iter()
        .map(|&n| (n, compute_distance(&vectors[node], &vectors[n], metric)))
@@ -1012,7 +1027,10 @@ fn get_attr_i64(attrs: &[(String, AttrValue)], name: &str) -> Result<i64, Format
 /// Like [`get_attr_i64`] but returns `None` when the attribute is absent or not
 /// an integer, instead of erroring. Used for optional/back-compat attributes.
 fn get_attr_i64_opt(attrs: &[(String, AttrValue)], name: &str) -> Option<i64> {
-    attrs.iter().find(|(n, _)| n == name).and_then(|(_, v)| match v {
+    attrs
        .iter()
        .find(|(n, _)| n == name)
        .and_then(|(_, v)| match v {
            AttrValue::I64(val) => Some(*val),
            AttrValue::U64(val) => Some(*val as i64),
            _ => None,
@@ -50,19 +50,31 @@ harness = false
 clawhdf5-agent = { path = "../clawhdf5-agent" }
 clawhdf5-io = { path = "../clawhdf5-io" }
 mpi = { version = "0.8", optional = true }
-serde = { version = "1", features = ["derive"] }
+serde = { workspace = true }
 serde_json = "1"
-tempfile = "3"
+tempfile = { workspace = true }
 # Optional: libhdf5 C wrapper for side-by-side comparison (requires system libhdf5).
 # Enable with: cargo bench -p clawhdf5-bench --features libhdf5-compare
 # Uses hdf5-metno (fork of hdf5 crate) which supports HDF5 1.14.x.
 hdf5 = { version = "0.12", optional = true, package = "hdf5-metno" }
 # Optional: real sentence embeddings for the LongMemEval bench's vector stage.
 # Enable with: cargo run --release --bin longmemeval_bench --features embeddings
 # Off by default — nothing in the shipped crates depends on these.
 candle-core = { version = "0.9", optional = true }
 candle-nn = { version = "0.9", optional = true }
 candle-transformers = { version = "0.9", optional = true }
 tokenizers = { version = "0.21", optional = true }
 [dev-dependencies]
 clawhdf5 = { path = "../clawhdf5", features = ["zstd", "pcodec"] }
-criterion = { version = "0.5", features = ["html_reports"] }
+criterion = { workspace = true }
 [features]
 # When enabled, benchmarks add matching libhdf5 variants for side-by-side comparison.
 libhdf5-compare = ["hdf5"]
 mpi-io = ["clawhdf5-io/mpi-io", "mpi"]
 # Real MiniLM embeddings for longmemeval_bench, so the vector stage is not inert.
 embeddings = ["candle-core", "candle-nn", "candle-transformers", "tokenizers"]
 # CUDA-accelerated embedding. MiniLM on a CPU takes hours over the full
 # longmemeval_s haystack; on a GPU it is minutes.
 embeddings-cuda = ["embeddings", "candle-core/cuda", "candle-nn/cuda", "candle-transformers/cuda"]
@@ -41,11 +41,7 @@ fn bench_metadata_attrs_write(c: &mut Criterion) {
            let path = tmp.path().join("attrs_libhdf5.h5");
            b.iter(|| {
                let file = hdf5::File::create(&path).unwrap();
-                let ds = file
+                let ds = file.new_dataset::<f64>().shape([3]).create("data").unwrap();
                    .new_dataset::<f64>()
                    .shape([3])
                    .create("data")
                    .unwrap();
                ds.write(&[1.0f64, 2.0, 3.0]).unwrap();
                for i in 0..k {
                    ds.new_attr::<i64>()
@@ -258,11 +254,7 @@ fn bench_metadata_open_from_disk(c: &mut Criterion) {
        let libhdf5_path = tmp.path().join("open_libhdf5.h5");
        {
            let file = hdf5::File::create(&libhdf5_path).unwrap();
-            let ds = file
+            let ds = file.new_dataset::<f64>().shape([3]).create("data").unwrap();
                .new_dataset::<f64>()
                .shape([3])
                .create("data")
                .unwrap();
            ds.write(&[1.0f64, 2.0, 3.0]).unwrap();
            ds.new_attr::<i64>()
                .create("label")
@@ -307,13 +299,17 @@ fn bench_metadata_parse_in_memory(c: &mut Criterion) {
        fb.finish().unwrap()
    };
-    group.bench_with_input(BenchmarkId::new("clawhdf5", "in_memory"), &bytes, |b, raw| {
+    group.bench_with_input(
        BenchmarkId::new("clawhdf5", "in_memory"),
        &bytes,
        |b, raw| {
            b.iter(|| {
                let file = File::from_bytes(raw.clone()).unwrap();
                let ds = file.dataset("data").unwrap();
                ds.attrs().unwrap()
            });
-    });
+        },
    );
    group.finish();
 }
@@ -74,11 +74,7 @@ fn bench_read_sequential(c: &mut Criterion) {
            let data: Vec<f32> = (0..nn).map(|i| i as f32 * 0.001).collect();
            {
                let lf = hdf5::File::create(&path).unwrap();
-                let lds = lf
+                let lds = lf.new_dataset::<f32>().shape([nn]).create("data").unwrap();
                    .new_dataset::<f32>()
                    .shape([nn])
                    .create("data")
                    .unwrap();
                lds.write(data.as_slice()).unwrap();
            }
            b.iter(|| {
@@ -235,7 +231,10 @@ fn bench_read_zerocopy_mmap(c: &mut Criterion) {
        group.throughput(Throughput::Bytes((n * size_of::<f64>()) as u64));
-        group.bench_with_input(BenchmarkId::new("clawhdf5_mmap_zerocopy", n), &path, |b, p| {
+        group.bench_with_input(
            BenchmarkId::new("clawhdf5_mmap_zerocopy", n),
            &path,
            |b, p| {
                b.iter(|| {
                    let file = MmapFile::open(p).unwrap();
                    let ds = file.dataset("data").unwrap();
@@ -247,7 +246,8 @@ fn bench_read_zerocopy_mmap(c: &mut Criterion) {
                    let sum: f64 = slice.map(|s| s.iter().sum()).unwrap_or(0.0);
                    criterion::black_box(sum)
                });
-        });
+            },
        );
        group.bench_with_input(BenchmarkId::new("clawhdf5_copy", n), &path, |b, p| {
            b.iter(|| {
@@ -63,11 +63,8 @@ fn bench_write_2d_chunked(c: &mut Criterion) {
    let mut group = c.benchmark_group("write_2d_chunked");
    // (rows, cols, chunk_rows, chunk_cols)
-    let configs: &[(usize, usize, u64, u64)] = &[
+    let configs: &[(usize, usize, u64, u64)] =
-        (32, 32, 8, 32),
+        &[(32, 32, 8, 32), (128, 128, 32, 128), (512, 512, 64, 512)];
        (128, 128, 32, 128),
        (512, 512, 64, 512),
    ];
    for &(rows, cols, cr, cc) in configs {
        let n = rows * cols;
@@ -120,11 +117,8 @@ fn bench_write_2d_chunked(c: &mut Criterion) {
 fn bench_write_2d_chunked_zstd(c: &mut Criterion) {
    let mut group = c.benchmark_group("write_2d_chunked_zstd");
-    let configs: &[(usize, usize, u64, u64)] = &[
+    let configs: &[(usize, usize, u64, u64)] =
-        (32, 32, 8, 32),
+        &[(32, 32, 8, 32), (128, 128, 32, 128), (512, 512, 64, 512)];
        (128, 128, 32, 128),
        (512, 512, 64, 512),
    ];
    for &(rows, cols, cr, cc) in configs {
        let n = rows * cols;
@@ -132,7 +126,10 @@ fn bench_write_2d_chunked_zstd(c: &mut Criterion) {
        let label = format!("{rows}x{cols}");
        group.throughput(Throughput::Bytes((n * size_of::<f32>()) as u64));
-        group.bench_with_input(BenchmarkId::new("clawhdf5/zstd-3", &label), &data, |b, d| {
+        group.bench_with_input(
            BenchmarkId::new("clawhdf5/zstd-3", &label),
            &data,
            |b, d| {
                let tmp = TempDir::new().unwrap();
                let path = tmp.path().join("write_2d_chunked_zstd.h5");
                b.iter(|| {
@@ -144,7 +141,8 @@ fn bench_write_2d_chunked_zstd(c: &mut Criterion) {
                        .with_zstd(3);
                    fb.write(&path).unwrap();
                });
-        });
+            },
        );
        group.bench_with_input(
            BenchmarkId::new("clawhdf5/deflate-6", &label),
@@ -178,11 +176,8 @@ fn bench_write_2d_chunked_zstd(c: &mut Criterion) {
 fn bench_write_2d_chunked_pcodec(c: &mut Criterion) {
    let mut group = c.benchmark_group("write_2d_chunked_pcodec");
-    let configs: &[(usize, usize, u64, u64)] = &[
+    let configs: &[(usize, usize, u64, u64)] =
-        (32, 32, 8, 32),
+        &[(32, 32, 8, 32), (128, 128, 32, 128), (512, 512, 64, 512)];
        (128, 128, 32, 128),
        (512, 512, 64, 512),
    ];
    for &(rows, cols, cr, cc) in configs {
        let n = rows * cols;
@@ -0,0 +1,95 @@
 //! World-model sample-loading benchmark — clawhdf5 vs the h5py counterpart.
 //!
 //! Reproduces the access pattern of `stable-worldmodel`'s HDF5 dataloader
 //! (arXiv 2605.21800): a dataset of `(N, H, W, C)` uint8 observation frames,
 //! read one frame at a time in shuffled (dataloader) order. That paper
 //! reports generic HDF5 at 1,416–1,474 samples/s (vs Lance 4,815); this
 //! measures clawhdf5 and h5py on the **same machine and file**, so the
 //! comparison is hardware-controlled. Absolute numbers are not comparable to
 //! the paper's (different box, smaller frames, no torch/transform) — only
 //! clawhdf5-vs-h5py *here* is.
 //!
 //! clawhdf5 mmaps the file once and takes a zero-copy `&[u8]` over the
 //! contiguous observation dataset; frame `i` is a subslice, and the OS pages
 //! it in on access. Two modes, because fairness demands both:
 //!   * default: sum the frame bytes through the zero-copy view — clawhdf5's
 //!     real advantage, no per-frame allocation;
 //!   * `--copy`: `to_vec()` each frame first, matching h5py's unavoidable
 //!     per-frame numpy materialization, so the two do equal work.
 //!
 //! Usage: `... --example worldmodel_sampling -- <file.h5> [passes] [--copy]`
 use std::hint::black_box;
 use std::time::Instant;
 use clawhdf5::MmapFile;
 fn main() {
    let args: Vec<String> = std::env::args().collect();
    let path = args
        .get(1)
        .expect("usage: worldmodel_sampling <file.h5> [passes] [--copy]");
    let passes: usize = args.get(2).and_then(|s| s.parse().ok()).unwrap_or(5);
    let copy = args.iter().any(|a| a == "--copy");
    let file = MmapFile::open(path).expect("open");
    let ds = file.dataset("observation").expect("observation dataset");
    let shape = ds.shape().expect("shape");
    let n = shape[0] as usize;
    let frame_bytes: usize = shape[1..].iter().map(|&d| d as usize).product();
    let raw = ds
        .read_raw_slice()
        .expect("read_raw_slice")
        .expect("contiguous zero-copy slice");
    assert_eq!(raw.len(), n * frame_bytes, "unexpected dataset size");
    let order = shuffled(n);
    let touch = |slice: &[u8]| -> u64 {
        if copy {
            let owned = slice.to_vec();
            owned.iter().map(|&b| u64::from(b)).sum()
        } else {
            slice.iter().map(|&b| u64::from(b)).sum()
        }
    };
    // Warm one pass (page-in), then time.
    let mut sink = 0u64;
    for &i in &order {
        sink = sink.wrapping_add(touch(&raw[i * frame_bytes..(i + 1) * frame_bytes]));
    }
    black_box(sink);
    let t0 = Instant::now();
    let mut sink = 0u64;
    for _ in 0..passes {
        for &i in &order {
            sink = sink.wrapping_add(touch(&raw[i * frame_bytes..(i + 1) * frame_bytes]));
        }
    }
    black_box(sink);
    let elapsed = t0.elapsed().as_secs_f64();
    let total = (n * passes) as f64;
    let mode = if copy {
        "materialized copy"
    } else {
        "zero-copy view"
    };
    println!("clawhdf5 ({mode}): {n} frames x {passes} passes in {elapsed:.3}s");
    println!("clawhdf5 ({mode}): {:.0} samples/sec", total / elapsed);
 }
 fn shuffled(n: usize) -> Vec<usize> {
    let mut v: Vec<usize> = (0..n).collect();
    let mut state: u64 = 0x9E37_79B9_7F4A_7C15;
    for i in (1..n).rev() {
        state = state
            .wrapping_mul(6364136223846793005)
            .wrapping_add(1442695040888963407);
        let j = (state >> 33) as usize % (i + 1);
        v.swap(i, j);
    }
    v
 }
@@ -4,11 +4,39 @@
 //! Since no embedding model is available at bench time, all embeddings are zero vectors
 //! and `hybrid_search` operates in BM25-only mode (vector_weight=0.0, keyword_weight=1.0).
 //!
-//! This matches the MemX paper methodology: evaluate retrieval recall, not answer generation.
+//! # Scoring target (read before citing any number from this harness)
 //!
 //! - **Metric: retrieval recall.** A "hit" means the gold-labelled memory appeared in
 //!   the top-k. No answer is generated and none is scored — the dataset's `answer`
 //!   field is deserialized and deliberately never read. This is **not** the official
 //!   LongMemEval metric, which is end-to-end QA accuracy (retrieve → generate → LLM
 //!   judge). Reporting retrieval recall as QA accuracy overstates by 20–30 points.
 //! - **Dataset: whichever variant you point it at.** Both `longmemeval_oracle`
 //!   (evidence sessions only — a substantially easier corpus) and the full
 //!   `longmemeval_s` haystack are supported. The harness does not trust the
 //!   filename: [`DatasetProfile`] measures evidence-session density from the
 //!   data and labels the run from that, so a mislabelled input cannot produce a
 //!   mislabelled result.
 //! - **Session-level metrics are degenerate when evidence density is high**, and
 //!   the report says so per run rather than assuming it. On the oracle variant
 //!   the haystack is essentially all-evidence, so any returned document is a
 //!   session-level hit at rank 0 by construction; only turn-level
 //!   (`has_answer == true` on the source turn) measures the retriever there. On
 //!   the full haystack, session-level recall is meaningful.
 //! - **Not comparable to MemX's Hit@5=51.6% / MRR=0.380**, which is *fact-level*
 //!   granularity over 220,349 records from 19,195 sessions.
 //!
 //! See `BENCHMARKS.md` § "Retracted: session-level recall and the MemX comparison".
 //!
 //! # Usage
 //! ```
-//! cargo run --release --bin longmemeval_bench [path/to/longmemeval_oracle.json]
+//! cargo run --release --bin longmemeval_bench [PATH] [--limit N]
 //!
 //! # Usage: full haystack
 //! ```
 //! cargo run --release --bin longmemeval_bench -- \
 //!     benchmarks/longmemeval/longmemeval_s_cleaned.json --limit 50
 //! ```
 //! ```
 //!
 //! # WASM Note
@@ -21,12 +49,80 @@
 use std::collections::{HashMap, HashSet};
 use std::time::{Duration, Instant};
 // `#[path]` keeps the module beside its binary without Cargo autodiscovering it
 // as a second bin target (which a bare `src/bin/embedder.rs` would be).
 #[cfg(feature = "embeddings")]
 #[path = "longmemeval_bench/embedder.rs"]
 mod embedder;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
 use serde::Deserialize;
 use tempfile::TempDir;
 const EMBEDDING_DIM: usize = 384;
 /// A retrieval configuration: how much of the score comes from each stage.
 #[derive(Clone, Copy)]
 struct Mode {
    label: &'static str,
    vector_weight: f32,
    keyword_weight: f32,
 }
 /// The only mode available without real embeddings. Passing zero vectors with
 /// `vector_weight = 0.0` is what made the vector stage inert.
 const BM25_ONLY: Mode = Mode {
    label: "BM25 only (vector stage inert)",
    vector_weight: 0.0,
    keyword_weight: 1.0,
 };
 #[cfg(feature = "embeddings")]
 const VECTOR_ONLY: Mode = Mode {
    label: "Vector only (MiniLM + HNSW)",
    vector_weight: 1.0,
    keyword_weight: 0.0,
 };
 /// Tuned by `--sweep` over the full haystack. The former 0.7/0.3 was a
 /// documented default that had never been searched, and the sweep found it
 /// strictly dominated: 0.4/0.6 is better on Hit@1, Hit@5, Hit@10 and MRR at
 /// both granularities.
 #[cfg(feature = "embeddings")]
 const HYBRID: Mode = Mode {
    label: "Hybrid (0.4 vector / 0.6 BM25, tuned)",
    vector_weight: 0.4,
    keyword_weight: 0.6,
 };
 /// Every 0.1 step of vector weight, keyword weight taking the remainder.
 ///
 /// Labels are leaked to `&'static str` because `Mode::label` is a `&'static
 /// str` for the eleven named modes and a sweep is a short-lived process; the
 /// alternative is threading a lifetime through the whole report path for a
 /// diagnostic mode.
 #[cfg(feature = "embeddings")]
 fn sweep_modes() -> Vec<Mode> {
    (0..=10)
        .map(|i| {
            let v = i as f32 / 10.0;
            Mode {
                label: Box::leak(format!("sweep v={v:.1} / k={:.1}", 1.0 - v).into_boxed_str()),
                vector_weight: v,
                keyword_weight: 1.0 - v,
            }
        })
        .collect()
 }
 /// Text -> embedding, built once for the whole corpus.
 type EmbeddingMap = HashMap<String, Vec<f32>>;
 /// Look up a real embedding, falling back to zeros when running BM25-only.
 fn embedding_for(map: Option<&EmbeddingMap>, text: &str) -> Vec<f32> {
    map.and_then(|m| m.get(text))
        .cloned()
        .unwrap_or_else(|| vec![0.0f32; EMBEDDING_DIM])
 }
 // ---------------------------------------------------------------------------
 // JSON data types
 // ---------------------------------------------------------------------------
@@ -168,7 +264,12 @@ struct EvalResult {
    latency: Duration,
 }
-fn evaluate_question(q: &Question, top_k: usize) -> EvalResult {
+fn evaluate_question(
    q: &Question,
    top_k: usize,
    mode: Mode,
    embeddings: Option<&EmbeddingMap>,
 ) -> EvalResult {
    let dir = TempDir::new().expect("failed to create temp dir");
    let mut config = MemoryConfig::new(dir.path().join("lme.h5"), "lme-bench", EMBEDDING_DIM);
    config.wal_enabled = false;
@@ -190,7 +291,7 @@ fn evaluate_question(q: &Question, top_k: usize) -> EvalResult {
        for turn in session {
            entries.push(MemoryEntry {
                chunk: turn.content.clone(),
-                embedding: vec![0.0f32; EMBEDDING_DIM],
+                embedding: embedding_for(embeddings, &turn.content),
                source_channel: "longmemeval".to_string(),
                timestamp: ts,
                session_id: sess_id.to_string(),
@@ -218,10 +319,15 @@ fn evaluate_question(q: &Question, top_k: usize) -> EvalResult {
    // Set of session IDs that contain the answer
    let answer_sess_set: HashSet<&str> = q.answer_session_ids.iter().map(String::as_str).collect();
-    // Run hybrid search (BM25-only: vector_weight=0.0, keyword_weight=1.0)
+    let query_emb = embedding_for(embeddings, &q.question);
    let zero_emb = vec![0.0f32; EMBEDDING_DIM];
    let t0 = Instant::now();
-    let results = memory.hybrid_search(&zero_emb, &q.question, 0.0, 1.0, top_k);
+    let results = memory.hybrid_search(
        &query_emb,
        &q.question,
        mode.vector_weight,
        mode.keyword_weight,
        top_k,
    );
    let latency = t0.elapsed();
    // Session-level recall
@@ -286,17 +392,133 @@ fn evaluate_question(q: &Question, top_k: usize) -> EvalResult {
 // Report printing
 // ---------------------------------------------------------------------------
-fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
+// ---------------------------------------------------------------------------
 // Dataset profile — measured, not assumed
 // ---------------------------------------------------------------------------
 /// Shape of the loaded corpus, computed from the data itself.
 ///
 /// The variant used to be a hardcoded `"oracle"` string in the report and the
 /// JSON summary, so pointing the harness at `longmemeval_s` would have produced
 /// full-haystack numbers labelled oracle. Everything here is derived from the
 /// questions instead, which means the label cannot drift from the corpus and a
 /// mislabelled input file cannot produce a mislabelled result.
 struct DatasetProfile {
    n_questions: usize,
    mean_sessions: f64,
    mean_turns: f64,
    /// Mean over questions of `|answer_sessions| / |haystack_sessions|`.
    ///
    /// This is what actually decides whether session-level recall means
    /// anything. At ~1.0 every haystack session is an evidence session, so any
    /// returned document is a session-level hit by construction.
    evidence_density: f64,
 }
 impl DatasetProfile {
    fn measure(questions: &[Question]) -> Self {
        let n = questions.len().max(1) as f64;
        let mut sessions = 0.0;
        let mut turns = 0.0;
        let mut density = 0.0;
        for q in questions {
            let n_sess = q.haystack_sessions.len();
            sessions += n_sess as f64;
            turns += q.haystack_sessions.iter().map(Vec::len).sum::<usize>() as f64;
            if n_sess > 0 {
                let evidence: HashSet<&str> =
                    q.answer_session_ids.iter().map(String::as_str).collect();
                let hit = q
                    .haystack_session_ids
                    .iter()
                    .filter(|id| evidence.contains(id.as_str()))
                    .count();
                density += hit as f64 / n_sess as f64;
            }
        }
        Self {
            n_questions: questions.len(),
            mean_sessions: sessions / n,
            mean_turns: turns / n,
            evidence_density: density / n,
        }
    }
    /// Above this share of evidence sessions, session-level recall is measuring
    /// the corpus shape rather than the retriever.
    const DEGENERACY_THRESHOLD: f64 = 0.9;
    const fn session_level_degenerate(&self) -> bool {
        self.evidence_density > Self::DEGENERACY_THRESHOLD
    }
    /// Variant name inferred from evidence density, not from the filename.
    const fn variant(&self) -> &'static str {
        if self.session_level_degenerate() {
            "oracle"
        } else {
            "full_haystack"
        }
    }
 }
 fn print_report(
    overall: &Metrics,
    by_type: &HashMap<String, Metrics>,
    profile: &DatasetProfile,
    mode: Mode,
 ) {
    println!("=================================================================");
-    println!("  LongMemEval Benchmark  (BM25-only retrieval, zero embeddings)");
+    println!("  LongMemEval Benchmark  —  {}", mode.label);
    println!("=================================================================");
    println!();
-    println!("Mode: vector_weight=0.0 / keyword_weight=1.0 (pure BM25)");
+    println!(
-    println!("Note: MemX (arxiv:2603.16171) with full system: Hit@5=51.6%, MRR=0.380");
+        "Mode: vector_weight={:.1} / keyword_weight={:.1}",
-    println!("      BM25-only numbers are expected to be lower — honest baseline.");
+        mode.vector_weight, mode.keyword_weight
    );
    println!();
    println!("Scoring target: RETRIEVAL RECALL (did the gold memory land in top-k).");
    println!("  No answer is generated or scored. This is NOT the official");
    println!("  LongMemEval metric (QA accuracy via retrieve+generate+judge).");
    println!(
        "Dataset: {} — {} questions, {:.1} sessions and {:.0} turns per question,",
        profile.variant(),
        profile.n_questions,
        profile.mean_sessions,
        profile.mean_turns,
    );
    println!(
        "  {:.1}% of haystack sessions are evidence sessions.",
        profile.evidence_density * 100.0
    );
    if profile.session_level_degenerate() {
        println!("  This is the evidence-only corpus, NOT the full longmemeval_s");
        println!("  haystack — a substantially easier retrieval problem.");
    } else {
        println!("  This is a full-haystack corpus: evidence sessions are a small");
        println!("  minority, so retrieval has to actually discriminate.");
    }
    println!();
    println!("Do NOT compare these to MemX's Hit@5=51.6% / MRR=0.380: that is");
    println!("  fact-level granularity over 220,349 records from 19,195 sessions.");
    println!("  Different granularity and a corpus larger by orders of magnitude.");
    println!();
    println!("## Session-Level Recall  (n={})", overall.count);
    if profile.session_level_degenerate() {
        println!(
            "  [DEGENERATE — {:.1}% of haystack sessions are evidence sessions, so a",
            profile.evidence_density * 100.0
        );
        println!("   returned document is a session-level hit almost by construction.");
        println!("   This measures the corpus shape, not the retriever. Use turn-level.]");
    } else {
        println!(
            "  [Meaningful on this corpus — only {:.1}% of haystack sessions are",
            profile.evidence_density * 100.0
        );
        println!("   evidence sessions, so a hit reflects the retriever's discrimination.]");
    }
    println!(
        "  Hit@1:  {:5.1}%   Hit@5:  {:5.1}%   Hit@10: {:5.1}%   MRR: {:.4}",
        overall.hit1_session_pct(),
@@ -380,7 +602,26 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
    println!("```json");
    println!("{{");
    println!("  \"benchmark\": \"longmemeval\",");
-    println!("  \"mode\": \"bm25_only\",");
+    println!(
        "  \"mode\": \"vector_{:.1}_keyword_{:.1}\",",
        mode.vector_weight, mode.keyword_weight
    );
    println!("  \"dataset_variant\": \"{}\",", profile.variant());
    println!("  \"scoring_target\": \"retrieval_recall\",");
    println!("  \"k\": 10,");
    println!(
        "  \"session_level_degenerate\": {},",
        profile.session_level_degenerate()
    );
    println!(
        "  \"evidence_session_density\": {:.4},",
        profile.evidence_density
    );
    println!(
        "  \"mean_sessions_per_question\": {:.2},",
        profile.mean_sessions
    );
    println!("  \"mean_turns_per_question\": {:.1},", profile.mean_turns);
    println!(
        "  \"total_questions\": {},",
        overall.count + overall.abstention_total
@@ -403,10 +644,16 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
        overall.mrr_turn()
    );
    println!("  }},");
    // `null`, not 0.0 — a corpus with no abstention questions has no abstention
    // accuracy, and emitting 0.0 reads as total failure at a task never posed.
    if overall.abstention_total > 0 {
        println!(
            "  \"abstention_accuracy\": {:.4},",
            overall.abstention_pct() / 100.0
        );
    } else {
        println!("  \"abstention_accuracy\": null,");
    }
    println!("  \"latency_us\": {{");
    println!(
        "    \"avg\": {:.1}, \"p50\": {:.1}, \"p95\": {:.1}, \"p99\": {:.1}",
@@ -425,17 +672,152 @@ fn print_report(overall: &Metrics, by_type: &HashMap<String, Metrics>) {
 // ---------------------------------------------------------------------------
 fn main() {
-    let json_path = std::env::args()
+    let mut json_path: Option<String> = None;
-        .nth(1)
+    let mut limit: Option<usize> = None;
-        .unwrap_or_else(|| "benchmarks/longmemeval/longmemeval_oracle.json".to_string());
+    let mut weights_dir: Option<String> = None;
    let mut sweep = false;
    let mut args = std::env::args().skip(1);
    while let Some(arg) = args.next() {
        match arg.as_str() {
            "--limit" => {
                let v = args.next().expect("--limit needs a value");
                limit = Some(v.parse().expect("--limit must be a positive integer"));
            }
            "--sweep" => sweep = true,
            "--embeddings" => {
                weights_dir = Some(args.next().expect("--embeddings needs a directory"));
            }
            "--help" | "-h" => {
                eprintln!(
                    "usage: longmemeval_bench [PATH] [--limit N]\n\n\
                     PATH     dataset JSON; defaults to the oracle variant.\n\
                              longmemeval_s works too — the harness measures which\n\
                              variant it was given rather than trusting the filename.\n\
                     --limit  evaluate N questions, sampled evenly across the file\n\
                              rather than as a prefix — the dataset is ordered by\n\
                              question type, so a prefix samples one type only.\n\
                     --embeddings DIR\n\
                              directory holding all-MiniLM-L6-v2's model.safetensors\n\
                              and tokenizer.json. Enables the vector stage and reports\n\
                              BM25-only, vector-only, and hybrid separately. Requires\n\
                              --features embeddings; without it the vector stage is\n\
                              inert and only the BM25 row is produced.\n\
                     --sweep  instead of the three named modes, sweep vector_weight\n\
                              from 0.0 to 1.0 in 0.1 steps. The 0.7/0.3 default was\n\
                              never searched; this is what searches it."
                );
                return;
            }
            other => json_path = Some(other.to_string()),
        }
    }
    let json_path =
        json_path.unwrap_or_else(|| "benchmarks/longmemeval/longmemeval_oracle.json".to_string());
    eprintln!("Loading: {json_path}");
    let data = std::fs::read_to_string(&json_path)
        .unwrap_or_else(|e| panic!("Failed to read {json_path}: {e}"));
-    let questions: Vec<Question> = serde_json::from_str(&data).expect("Failed to parse JSON");
+    let mut questions: Vec<Question> = serde_json::from_str(&data).expect("Failed to parse JSON");
    if let Some(n) = limit
        && n < questions.len()
    {
        // Stride rather than truncate. The dataset is ordered by question type,
        // so taking a prefix samples one type: `--limit 20` on longmemeval_s
        // returns 20 `single-session-user` questions and nothing else, which
        // reads as a whole-dataset result but is not one.
        let total = questions.len();
        let step = total as f64 / n as f64;
        let keep: HashSet<usize> = (0..n)
            .map(|i| ((i as f64 * step) as usize).min(total - 1))
            .collect();
        questions = questions
            .into_iter()
            .enumerate()
            .filter(|(i, _)| keep.contains(i))
            .map(|(_, q)| q)
            .collect();
        eprintln!(
            "Sampling {} of {total} questions, evenly strided (--limit)",
            questions.len()
        );
    }
    let total = questions.len();
    eprintln!("Loaded {total} questions");
    let profile = DatasetProfile::measure(&questions);
    eprintln!(
        "Corpus: {} variant — {:.1} sessions / {:.0} turns per question, \
         {:.1}% evidence-session density",
        profile.variant(),
        profile.mean_sessions,
        profile.mean_turns,
        profile.evidence_density * 100.0,
    );
    // Build the embedding table once for the whole corpus, if asked for.
    let embeddings: Option<EmbeddingMap> = weights_dir
        .as_deref()
        .map(|dir| load_embeddings(dir, &questions));
    if embeddings.is_none() && weights_dir.is_some() {
        eprintln!("warning: --embeddings ignored (build with --features embeddings)");
    }
    let modes: Vec<Mode> = if embeddings.is_some() {
        #[cfg(feature = "embeddings")]
        {
            if sweep {
                sweep_modes()
            } else {
                vec![BM25_ONLY, VECTOR_ONLY, HYBRID]
            }
        }
        #[cfg(not(feature = "embeddings"))]
        {
            vec![BM25_ONLY]
        }
    } else {
        if sweep {
            eprintln!("warning: --sweep needs --embeddings; running BM25 only");
        }
        vec![BM25_ONLY]
    };
    for (mode_idx, mode) in modes.iter().enumerate() {
        eprintln!("[{}/{}] {}", mode_idx + 1, modes.len(), mode.label);
        run_mode(&questions, *mode, embeddings.as_ref(), &profile);
    }
 }
 /// Load and encode the corpus. Returns `None` unless the `embeddings` feature
 /// is compiled in, so the flag degrades to a warning rather than a hard error.
 #[cfg(feature = "embeddings")]
 fn load_embeddings(dir: &str, questions: &[Question]) -> EmbeddingMap {
    let enc = embedder::Embedder::load(std::path::Path::new(dir))
        .unwrap_or_else(|e| panic!("failed to load embedder from {dir}: {e}"));
    let texts = questions.iter().flat_map(|q| {
        q.haystack_sessions
            .iter()
            .flatten()
            .map(|t| t.content.clone())
            .chain(std::iter::once(q.question.clone()))
    });
    enc.encode_unique(texts)
        .unwrap_or_else(|e| panic!("embedding failed: {e}"))
 }
 #[cfg(not(feature = "embeddings"))]
 fn load_embeddings(_dir: &str, _questions: &[Question]) -> EmbeddingMap {
    EmbeddingMap::new()
 }
 /// Evaluate every question under one retrieval mode and print its report.
 fn run_mode(
    questions: &[Question],
    mode: Mode,
    embeddings: Option<&EmbeddingMap>,
    profile: &DatasetProfile,
 ) {
    let total = questions.len();
    let mut overall = Metrics::default();
    let mut by_type: HashMap<String, Metrics> = HashMap::new();
@@ -444,7 +826,7 @@ fn main() {
            eprint!("\r  [{}/{}] evaluating...", i + 1, total);
        }
-        let result = evaluate_question(q, 10);
+        let result = evaluate_question(q, 10, mode, embeddings);
        let is_abs = q.question_type.ends_with("_abs");
        let base_type = if is_abs {
@@ -509,5 +891,5 @@ fn main() {
    eprintln!("\r  [{total}/{total}] done.          ");
    eprintln!();
-    print_report(&overall, &by_type);
+    print_report(&overall, &by_type, profile, mode);
 }
@@ -0,0 +1,170 @@
 //! Optional MiniLM sentence embedder for the LongMemEval bench.
 //!
 //! Compiled only under the `embeddings` feature, so the default build of a
 //! project that prides itself on having no heavyweight dependencies stays
 //! exactly as it was. Without it the bench runs BM25-only, as it always has.
 //!
 //! Loads `sentence-transformers/all-MiniLM-L6-v2` — the same checkpoint
 //! omni-cortex uses — and produces 384-d mean-pooled, L2-normalised sentence
 //! embeddings, which is the published recipe for this model (mean over token
 //! states weighted by the attention mask, *not* the `[CLS]` pooler output).
 use std::collections::HashMap;
 use std::path::Path;
 use candle_core::{DType, Device, Tensor};
 use candle_nn::VarBuilder;
 use candle_transformers::models::bert::{BertModel, Config, HiddenAct};
 use tokenizers::Tokenizer;
 /// Sequences encoded per forward pass. Larger batches amortise the transformer
 /// call; 64 keeps peak memory modest while still saturating a CPU.
 const BATCH: usize = 64;
 /// A loaded MiniLM encoder.
 pub struct Embedder {
    model: BertModel,
    tokenizer: Tokenizer,
    device: Device,
 }
 impl Embedder {
    /// Load from a directory holding `model.safetensors` and `tokenizer.json`.
    ///
    /// `config.json` is read when present; otherwise the published MiniLM-L6-v2
    /// architecture constants are used, which are pinned rather than guessed.
    pub fn load(dir: &Path) -> Result<Self, Box<dyn std::error::Error>> {
        // CUDA when the feature is on and a device is actually present; the CPU
        // path is correct but roughly two orders of magnitude slower, which is
        // the difference between minutes and most of a day on the full haystack.
        let device = match Device::new_cuda(0) {
            Ok(d) => {
                eprintln!("Embedder: CUDA device 0");
                d
            }
            Err(e) => {
                // Loud, because the CPU path is correct but ~100x slower: the
                // full longmemeval_s haystack is minutes on a GPU and most of a
                // day on 8 cores. Silently falling back looks like a hang.
                eprintln!("Embedder: CPU — CUDA unavailable ({e})");
                eprintln!(
                    "  WARNING: CPU embedding is roughly two orders of magnitude slower.\n                       Expect minutes for longmemeval_oracle and many hours for the full\n                       longmemeval_s haystack. For the GPU path, rebuild with\n                       `--features embeddings-cuda` and make sure `nvcc` is on PATH\n                       (it ships in /usr/local/cuda/bin, which is often not exported)."
                );
                Device::Cpu
            }
        };
        let weights = dir.join("model.safetensors");
        let tok_path = dir.join("tokenizer.json");
        let config: Config = match std::fs::read_to_string(dir.join("config.json")) {
            Ok(raw) => serde_json::from_str(&raw)?,
            Err(_) => Config {
                vocab_size: 30_522,
                hidden_size: 384,
                num_hidden_layers: 6,
                num_attention_heads: 12,
                intermediate_size: 1_536,
                hidden_act: HiddenAct::Gelu,
                hidden_dropout_prob: 0.0,
                max_position_embeddings: 512,
                type_vocab_size: 2,
                initializer_range: 0.02,
                layer_norm_eps: 1e-12,
                pad_token_id: 0,
                position_embedding_type: Default::default(),
                use_cache: false,
                classifier_dropout: None,
                model_type: None,
            },
        };
        let vb = unsafe { VarBuilder::from_mmaped_safetensors(&[weights], DType::F32, &device)? };
        let model = BertModel::load(vb, &config)?;
        let tokenizer = Tokenizer::from_file(&tok_path).map_err(|e| e.to_string())?;
        Ok(Self {
            model,
            tokenizer,
            device,
        })
    }
    /// Encode `texts` into 384-d unit vectors, in order.
    fn encode_batch(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>, Box<dyn std::error::Error>> {
        let mut tk = self.tokenizer.clone();
        let tk = tk
            .with_padding(Some(tokenizers::PaddingParams::default()))
            .with_truncation(Some(tokenizers::TruncationParams {
                max_length: 512,
                ..Default::default()
            }))
            .map_err(|e| e.to_string())?;
        let encodings = tk
            .encode_batch(texts.to_vec(), true)
            .map_err(|e| e.to_string())?;
        let ids: Vec<u32> = encodings
            .iter()
            .flat_map(|e| e.get_ids().to_vec())
            .collect();
        let mask: Vec<u32> = encodings
            .iter()
            .flat_map(|e| e.get_attention_mask().to_vec())
            .collect();
        let (b, l) = (encodings.len(), encodings[0].get_ids().len());
        let ids = Tensor::from_vec(ids, (b, l), &self.device)?;
        let mask = Tensor::from_vec(mask, (b, l), &self.device)?;
        let type_ids = ids.zeros_like()?;
        let hidden = self.model.forward(&ids, &type_ids, Some(&mask))?;
        // Mean-pool over real tokens only: sum(hidden * mask) / sum(mask).
        let mask_f = mask.to_dtype(DType::F32)?.unsqueeze(2)?;
        let summed = hidden.broadcast_mul(&mask_f)?.sum(1)?;
        let counts = mask_f.sum(1)?.clamp(1e-9, f32::INFINITY)?;
        let pooled = summed.broadcast_div(&counts)?;
        // L2-normalise so cosine similarity is a plain dot product.
        let norm = pooled
            .sqr()?
            .sum_keepdim(1)?
            .sqrt()?
            .clamp(1e-12, f32::INFINITY)?;
        let normed = pooled.broadcast_div(&norm)?;
        Ok(normed.to_vec2::<f32>()?)
    }
    /// Encode every distinct string in `texts` once, returning a lookup map.
    ///
    /// LongMemEval's haystack sessions are drawn from a shared pool, so the same
    /// turn text recurs across many questions. Deduplicating before encoding is
    /// the difference between encoding the corpus once and encoding it per
    /// question.
    pub fn encode_unique(
        &self,
        texts: impl IntoIterator<Item = String>,
    ) -> Result<HashMap<String, Vec<f32>>, Box<dyn std::error::Error>> {
        let mut unique: Vec<String> = texts.into_iter().collect();
        unique.sort_unstable();
        unique.dedup();
        let total = unique.len();
        eprintln!("Embedding {total} unique texts with MiniLM (batch {BATCH})...");
        let mut out = HashMap::with_capacity(total);
        for (n, chunk) in unique.chunks(BATCH).enumerate() {
            let refs: Vec<&str> = chunk.iter().map(String::as_str).collect();
            let vecs = self.encode_batch(&refs)?;
            for (text, v) in chunk.iter().zip(vecs) {
                out.insert(text.clone(), v);
            }
            if n % 50 == 0 {
                eprint!("\r  [{}/{}] embedded...", (n * BATCH).min(total), total);
            }
        }
        eprintln!("\r  [{total}/{total}] embedded.          ");
        Ok(out)
    }
 }
@@ -17,4 +17,4 @@ path = "src/main.rs"
 clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.1.0" }
 clap = { version = "4", features = ["derive", "env"] }
 serde_json = "1"
-serde = { version = "1", features = ["derive"] }
+serde = { workspace = true }
@@ -2,7 +2,7 @@
 name = "clawhdf5-filters"
 version = "2.1.0"
 edition = "2024"
-description = "Filter and compression pipeline for rustyhdf5"
+description = "Filter and compression pipeline for clawhdf5"
 license = "MIT"
 repository = "https://github.com/redclawsystems/clawhdf5"
 readme = "README.md"
@@ -14,7 +14,7 @@ flate2 = { version = "1", default-features = false, features = ["rust_backend"]
 miniz_oxide = "0.8"
 [dev-dependencies]
-criterion = { version = "0.5", features = ["html_reports"] }
+criterion = { workspace = true }
 [[bench]]
 name = "deflate_bench"
@@ -270,14 +270,29 @@ pub(crate) fn flate2_decompress_preallocated(
    Ok(output)
 }
 /// Absolute ceiling on decompressed output when the caller has no size hint,
 /// preventing unbounded allocation from a hostile/corrupted zlib stream.
 const MAX_DECOMPRESS_SIZE: usize = 256 * 1024 * 1024;
 /// Streaming decompress with dynamic sizing (when output size is unknown).
 ///
 /// Bounded by [`MAX_DECOMPRESS_SIZE`] since there is no chunk-size hint to
 /// validate against here — an unbounded `read_to_end` would let a hostile
 /// zlib stream force arbitrarily large allocation (a "zlib bomb").
 pub(crate) fn flate2_decompress_streaming(data: &[u8]) -> Result<Vec<u8>, String> {
    use std::io::Read;
-    let mut decoder = flate2::read::ZlibDecoder::new(data);
+    let decoder = flate2::read::ZlibDecoder::new(data);
    let mut result = Vec::new();
    decoder
        .take(MAX_DECOMPRESS_SIZE as u64 + 1)
        .read_to_end(&mut result)
        .map_err(|e| e.to_string())?;
    if result.len() > MAX_DECOMPRESS_SIZE {
        return Err(format!(
            "decompressed output exceeds {} MiB limit",
            MAX_DECOMPRESS_SIZE / 1024 / 1024
        ));
    }
    Ok(result)
 }
@@ -11,6 +11,7 @@ categories = ["parser-implementations", "science", "encoding", "no-std"]
 [dependencies]
 byteorder = { version = "1", default-features = false }
 portable-atomic = { version = "1" }
 flate2 = { version = "1", default-features = false, features = ["rust_backend"], optional = true }
 sha2 = { version = "0.10", default-features = false, optional = true }
 rayon = { version = "1", optional = true }
@@ -23,7 +24,7 @@ pco = { version = "1.0", optional = true }
 [dev-dependencies]
 serde_json = "1"
-criterion = { version = "0.5", features = ["html_reports"] }
+criterion = { workspace = true }
 clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.1.0" }
 [[bench]]
@@ -14,6 +14,9 @@ libfuzzer-sys = "0.4"
 path = ".."
 features = ["std", "checksum", "deflate"]
 [dependencies.clawhdf5]
 path = "../../clawhdf5"
 [workspace]
 members = ["."]
@@ -56,3 +59,8 @@ doc = false
 name = "fuzz_full_file"
 path = "fuzz_targets/fuzz_full_file.rs"
 doc = false
 [[bin]]
 name = "fuzz_dataset_read"
 path = "fuzz_targets/fuzz_dataset_read.rs"
 doc = false
@@ -1,4 +1,4 @@
-# Fuzz Testing for rustyhdf5-format
+# Fuzz Testing for clawhdf5-format
 Uses [cargo-fuzz](https://github.com/rust-fuzz/cargo-fuzz) (libFuzzer) to test parser robustness against malformed inputs.
@@ -21,13 +21,14 @@ rustup toolchain install nightly
 | `fuzz_btree_v2` | `BTreeV2Header::parse` | B-tree v2 header parsing |
 | `fuzz_filter_pipeline` | `FilterPipeline::parse` | Filter pipeline messages (v1/v2) |
 | `fuzz_full_file` | signature + superblock + root group | End-to-end file parsing chain |
 | `fuzz_dataset_read` | `Dataset::read_*` (via `clawhdf5`) | Walks every dataset in the parsed file and exercises the contiguous/chunked/compact raw-data read paths (`chunked_read.rs`, `data_read.rs`) that `fuzz_full_file` doesn't reach |
 ## Running
 Run a single target (runs indefinitely until stopped or a crash is found):
 ```bash
-cd crates/rustyhdf5-format
+cd crates/clawhdf5-format
 cargo +nightly fuzz run fuzz_datatype
 ```
@@ -41,12 +42,20 @@ Run all targets for 30 seconds each:
 ```bash
 for target in fuzz_superblock fuzz_object_header fuzz_datatype fuzz_dataspace \
-              fuzz_fractal_heap fuzz_btree_v2 fuzz_filter_pipeline fuzz_full_file; do
+              fuzz_fractal_heap fuzz_btree_v2 fuzz_filter_pipeline fuzz_full_file \
              fuzz_dataset_read; do
    echo "=== $target ==="
    cargo +nightly fuzz run "$target" -- -max_total_time=30 -max_len=4096
 done
 ```
 ## CI
 These targets are **not** run in CI (`.gitea/workflows/ci.yml`) — cargo-fuzz
 requires nightly and each meaningful run takes minutes, which doesn't fit a
 per-PR gate. Run them manually on a schedule (e.g. before a release, or after
 touching parser code) instead.
 ## Reproducing Crashes
 If a crash is found, the input is saved to `fuzz/artifacts/<target>/`. Reproduce with:
@@ -0,0 +1,45 @@
 #![no_main]
 use libfuzzer_sys::fuzz_target;
 const MAX_WALK_DEPTH: usize = 16;
 /// Walk groups/datasets from `group`, exercising every dataset-reading code
 /// path reachable through the public API (contiguous/chunked/compact raw
 /// reads via `chunked_read.rs`/`data_read.rs`). Depth-limited independently
 /// of any parser-level recursion guard, since this is fuzz-harness
 /// bookkeeping, not something under test.
 fn walk_group(group: &clawhdf5::Group, depth: usize) {
    if depth > MAX_WALK_DEPTH {
        return;
    }
    if let Ok(names) = group.datasets() {
        for name in names {
            if let Ok(dataset) = group.dataset(&name) {
                let _ = dataset.shape();
                let _ = dataset.max_dimensions();
                let _ = dataset.dtype();
                let _ = dataset.read_raw_ref();
                let _ = dataset.read_f64();
                let _ = dataset.read_f32();
                let _ = dataset.read_i32();
                let _ = dataset.read_i64();
                let _ = dataset.read_u64();
                let _ = dataset.read_string();
            }
        }
    }
    if let Ok(names) = group.groups() {
        for name in names {
            if let Ok(subgroup) = group.group(&name) {
                walk_group(&subgroup, depth + 1);
            }
        }
    }
 }
 fuzz_target!(|data: &[u8]| {
    let Ok(file) = clawhdf5::File::from_bytes(data.to_vec()) else {
        return;
    };
    walk_group(&file.root(), 0);
 });
@@ -24,6 +24,21 @@ pub struct BTreeV1Node {
    pub children: Vec<u64>,
 }
 /// Checks that `[offset, offset + needed)` fits within `data`, guarding the
 /// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
 fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
    if offset
        .checked_add(needed)
        .is_none_or(|end| end > data.len())
    {
        return Err(FormatError::UnexpectedEof {
            expected: offset.saturating_add(needed),
            available: data.len(),
        });
    }
    Ok(())
 }
 fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
    let s = size as usize;
    if pos.checked_add(s).is_none_or(|end| end > data.len()) {
@@ -45,7 +60,7 @@ fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
 fn is_undefined(data: &[u8], pos: usize, size: u8) -> bool {
    let s = size as usize;
-    if pos + s > data.len() {
+    if ensure_len(data, pos, s).is_err() {
        return false;
    }
    data[pos..pos + s].iter().all(|&b| b == 0xFF)
@@ -65,12 +80,7 @@ impl BTreeV1Node {
        // + left_sibling(offset_size) + right_sibling(offset_size)
        let os = offset_size as usize;
        let header_size = 8 + os * 2;
-        if offset + header_size > file_data.len() {
+        ensure_len(file_data, offset, header_size)?;
            return Err(FormatError::UnexpectedEof {
                expected: offset + header_size,
                available: file_data.len(),
            });
        }
        if &file_data[offset..offset + 4] != b"TREE" {
            return Err(FormatError::InvalidBTreeSignature);
@@ -99,12 +109,7 @@ impl BTreeV1Node {
        let eu = entries_used as usize;
        let key_size = os; // For type 0, key = offset_size
        let needed = eu * (key_size + os) + key_size; // eu children + (eu+1) keys
-        if pos + needed > file_data.len() {
+        ensure_len(file_data, pos, needed)?;
            return Err(FormatError::UnexpectedEof {
                expected: pos + needed,
                available: file_data.len(),
            });
        }
        let mut keys = Vec::with_capacity(eu + 1);
        let mut children = Vec::with_capacity(eu);
@@ -241,6 +246,16 @@ mod tests {
        assert_eq!(node.right_sibling, None);
    }
    #[test]
    fn parse_near_usize_max_offset_rejected_without_overflow() {
        let data = build_btree_node(0, 0, &[0, 5, 10], &[0x100, 0x200], None, None, 8);
        let result = BTreeV1Node::parse(&data, usize::MAX - 4, 8, 8);
        assert!(
            matches!(result, Err(FormatError::UnexpectedEof { .. })),
            "expected a clean UnexpectedEof, got {result:?}"
        );
    }
    #[test]
    fn parse_with_siblings_none() {
        let data = build_btree_node(0, 0, &[0, 8], &[0x300], None, None, 8);
@@ -16,6 +16,8 @@ use core::ops::{Deref, DerefMut};
 use alloc::collections::BTreeMap;
 #[cfg(feature = "std")]
 use std::collections::HashMap;
 #[cfg(feature = "std")]
 use std::sync::Arc;
 use crate::chunk_index::{ChunkIndex, ChunkLayout};
 use crate::chunked_read::ChunkInfo;
@@ -64,6 +66,11 @@ pub struct CacheAlignedBuffer {
 // SAFETY: The raw pointer is exclusively owned — no aliasing.
 unsafe impl Send for CacheAlignedBuffer {}
 // SAFETY: `CacheAlignedBuffer` exposes its contents only via `&[u8]`/`&mut
 // [u8]` through the ordinary borrow-checked `Deref`/`DerefMut` impls below —
 // the same access pattern as `Vec<u8>`, which is `Sync`. Needed so
 // `Arc<CacheAlignedBuffer>` (used by the chunk cache) is itself `Send`.
 unsafe impl Sync for CacheAlignedBuffer {}
 impl CacheAlignedBuffer {
    /// Allocate a new cache-line-aligned buffer of exactly `len` bytes,
@@ -223,7 +230,9 @@ pub const DEFAULT_MAX_SLOTS: usize = 521;
 #[cfg(feature = "std")]
 struct CachedChunk {
    coord: ChunkCoord,
-    data: CacheAlignedBuffer,
+    /// Shared so a cache hit is a refcount bump, not a copy of the whole
    /// (potentially large) decompressed chunk.
    data: Arc<CacheAlignedBuffer>,
    /// Monotonically increasing access counter for LRU ordering.
    last_access: u64,
 }
@@ -267,6 +276,12 @@ struct CacheInner {
    /// LRU cache of decompressed chunk data.
    slots: Vec<CachedChunk>,
    /// Coordinate -> index into `slots`, for O(1) lookup instead of a linear
    /// scan. Kept in sync with `slots` on every insert/evict/clear — in
    /// particular, `slots.swap_remove(i)` moves the last element into slot
    /// `i`, so the moved element's index entry must be updated too.
    slot_index: HashMap<ChunkCoord, usize>,
    /// Current total bytes of cached decompressed data.
    current_bytes: usize,
@@ -344,6 +359,7 @@ impl ChunkCache {
                index: None,
                index_addr: None,
                slots: Vec::with_capacity(max_slots.min(64)),
                slot_index: HashMap::with_capacity(max_slots.min(64)),
                current_bytes: 0,
                max_bytes,
                max_slots,
@@ -375,6 +391,7 @@ impl ChunkCache {
        inner.chunk_index = None;
        inner.chunk_layout = None;
        inner.slots.clear();
        inner.slot_index.clear();
        inner.current_bytes = 0;
        inner.last_coord = None;
        inner.index_addr = Some(addr);
@@ -477,8 +494,20 @@ impl ChunkCache {
    /// Try to get cached decompressed data for a chunk coordinate.
    ///
-    /// Returns a clone of the cache-line-aligned buffer.
+    /// O(1) lookup. Returns an owned copy for API compatibility with callers
    /// that need a `Vec<u8>`; prefer [`Self::get_decompressed_aligned`] when
    /// an `Arc`-shared buffer works for the caller, since that avoids the
    /// copy entirely.
    pub fn get_decompressed(&self, coord: &[u64]) -> Option<Vec<u8>> {
        self.get_decompressed_aligned(coord)
            .map(|arc| arc.as_slice().to_vec())
    }
    /// Try to get a reference-counted clone of the aligned buffer for a chunk.
    ///
    /// O(1) index lookup; the clone is an `Arc` refcount bump, not a copy of
    /// the underlying decompressed data.
    pub fn get_decompressed_aligned(&self, coord: &[u64]) -> Option<Arc<CacheAlignedBuffer>> {
        let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
        inner.tick += 1;
        let tick = inner.tick;
@@ -500,36 +529,12 @@ impl ChunkCache {
        }
        inner.last_coord = Some(coord.to_vec());
-        let mut found = None;
+        let found = if let Some(&idx) = inner.slot_index.get(coord) {
-        for slot in inner.slots.iter_mut() {
+            inner.slots[idx].last_access = tick;
-            if slot.coord.as_slice() == coord {
+            Some(Arc::clone(&inner.slots[idx].data))
                slot.last_access = tick;
                found = Some(slot.data.to_vec());
                break;
            }
        }
        if let Some(ref data) = found {
            inner.stats.hits += 1;
            inner.stats.bytes_read += data.len() as u64;
        } else {
-            inner.stats.misses += 1;
+            None
-        }
+        };
        found
    }
    /// Try to get a reference-counted clone of the aligned buffer for a chunk.
    pub fn get_decompressed_aligned(&self, coord: &[u64]) -> Option<CacheAlignedBuffer> {
        let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
        inner.tick += 1;
        let tick = inner.tick;
        let mut found = None;
        for slot in inner.slots.iter_mut() {
            if slot.coord.as_slice() == coord {
                slot.last_access = tick;
                found = Some(slot.data.clone());
                break;
            }
        }
        if let Some(ref data) = found {
            inner.stats.hits += 1;
            inner.stats.bytes_read += data.len() as u64;
@@ -542,30 +547,39 @@ impl ChunkCache {
    /// Insert decompressed chunk data into the LRU cache.
    ///
    /// The data is stored in a [`CacheAlignedBuffer`] so subsequent reads
-    /// return cache-line-aligned memory.
+    /// return cache-line-aligned memory. Returns the `Arc`-shared buffer that
-    pub fn put_decompressed(&self, coord: ChunkCoord, data: Vec<u8>) {
+    /// is now cached (or already was), so the caller can reuse it directly
-        let aligned = CacheAlignedBuffer::from_slice(&data);
+    /// instead of holding a separate copy of the same data.
-        self.put_decompressed_aligned(coord, aligned);
+    pub fn put_decompressed(&self, coord: ChunkCoord, data: Vec<u8>) -> Arc<CacheAlignedBuffer> {
        let aligned = CacheAlignedBuffer::from_vec(data);
        self.put_decompressed_aligned(coord, aligned)
    }
    /// Insert an already-aligned buffer into the LRU cache.
-    pub fn put_decompressed_aligned(&self, coord: ChunkCoord, data: CacheAlignedBuffer) {
+    ///
    /// Returns the `Arc`-shared buffer now held by the cache (the one just
    /// inserted, or the existing cached copy if `coord` was already present).
    pub fn put_decompressed_aligned(
        &self,
        coord: ChunkCoord,
        data: CacheAlignedBuffer,
    ) -> Arc<CacheAlignedBuffer> {
        let data = Arc::new(data);
        let mut inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
        let data_len = data.len();
-        // Don't cache if single chunk exceeds budget
+        // Don't cache if single chunk exceeds budget — still return the data
        // to the caller, just don't retain it.
        if data_len > inner.max_bytes {
-            return;
+            return data;
        }
        // Check if already present
        inner.tick += 1;
        let tick = inner.tick;
-        for slot in inner.slots.iter_mut() {
+        if let Some(&idx) = inner.slot_index.get(&coord) {
-            if slot.coord == coord {
+            inner.slots[idx].last_access = tick;
-                slot.last_access = tick;
+            return Arc::clone(&inner.slots[idx].data); // already cached
                return; // already cached
            }
        }
        // Evict until we have room
@@ -581,16 +595,26 @@ impl ChunkCache {
                .map(|(i, _)| i)
                .unwrap();
            let removed = inner.slots.swap_remove(lru_idx);
            inner.slot_index.remove(&removed.coord);
            // swap_remove moved the former last element into `lru_idx` (unless
            // it *was* the last element) — fix up that element's index entry.
            if lru_idx < inner.slots.len() {
                let moved_coord = inner.slots[lru_idx].coord.clone();
                inner.slot_index.insert(moved_coord, lru_idx);
            }
            inner.current_bytes -= removed.data.len();
            inner.stats.evictions += 1;
        }
        inner.current_bytes += data_len;
        let new_idx = inner.slots.len();
        inner.slot_index.insert(coord.clone(), new_idx);
        inner.slots.push(CachedChunk {
            coord,
-            data,
+            data: Arc::clone(&data),
            last_access: tick,
        });
        data
    }
    /// Clear the entire cache (index + decompressed data).
@@ -599,6 +623,7 @@ impl ChunkCache {
        inner.index = None;
        inner.index_addr = None;
        inner.slots.clear();
        inner.slot_index.clear();
        inner.current_bytes = 0;
        inner.tick = 0;
        inner.last_coord = None;
@@ -607,11 +632,13 @@ impl ChunkCache {
        inner.chunk_layout = None;
    }
-    /// Hint that the given chunk coordinates will be accessed soon.
+    /// Record that the given chunk coordinates are predicted to be accessed
    /// soon (bookkeeping only).
    ///
-    /// Pre-populates the chunk index for these coordinates so that
+    /// This does **not** prefetch or pre-decompress anything — it only
-    /// subsequent lookups are O(1). This does NOT pre-decompress the
+    /// checks whether each coordinate is already in the chunk index and
-    /// chunks — it only ensures the index entries exist.
+    /// updates access-pattern stats accordingly. Real prefetching (e.g.
    /// background pre-decompression) is not implemented.
    pub fn prefetch_hint(&self, next_coords: &[ChunkCoord]) {
        let inner = self.inner.lock().unwrap_or_else(|e| e.into_inner());
        if inner.index.is_none() {
@@ -785,6 +812,50 @@ mod tests {
        assert_eq!(cache.cached_bytes(), 3);
    }
    #[test]
    fn slot_index_consistent_after_many_evictions() {
        // Force repeated swap_remove evictions (small slot budget, many
        // inserts) and confirm the coord -> slot index stays correct: every
        // remaining coord must still resolve to its own data, not another
        // slot's (which would happen if swap_remove's index fixup were wrong).
        let cache = ChunkCache::with_capacity(1024 * 1024, 4); // max 4 slots
        for i in 0..50u64 {
            cache.put_decompressed(vec![i], vec![(i % 256) as u8; 8]);
            // Interleave reads of a couple of earlier coords to churn LRU
            // order (and thus which slot gets swap_remove'd) beyond simple
            // FIFO eviction.
            if i >= 2 {
                let _ = cache.get_decompressed(&[i - 2]);
            }
        }
        // Whatever remains in the cache (at most 4 slots) must return its
        // own correct data.
        for i in 0..50u64 {
            if let Some(data) = cache.get_decompressed(&[i]) {
                assert_eq!(
                    data,
                    vec![(i % 256) as u8; 8],
                    "coord {i} returned wrong data after eviction churn"
                );
            }
        }
        assert!(cache.cached_chunk_count() <= 4);
    }
    #[test]
    fn get_decompressed_aligned_shares_arc_on_hit() {
        let cache = ChunkCache::new();
        cache.put_decompressed(vec![0, 0], vec![9, 9, 9, 9]);
        let a = cache.get_decompressed_aligned(&[0, 0]).unwrap();
        let b = cache.get_decompressed_aligned(&[0, 0]).unwrap();
        // A cache hit clones the Arc (refcount bump), not the underlying
        // buffer — both handles point at the same allocation.
        assert!(Arc::ptr_eq(&a, &b));
        assert_eq!(a.as_slice(), &[9, 9, 9, 9]);
    }
    // --- CacheAlignedBuffer tests ---
    #[test]
@@ -17,6 +17,8 @@ use crate::extensible_array::{ExtensibleArrayHeader, read_extensible_array_chunk
 use crate::filter_pipeline::FilterPipeline;
 use crate::filters::decompress_chunk;
 use crate::fixed_array::{FixedArrayHeader, read_fixed_array_chunks};
 #[cfg(feature = "std")]
 use std::sync::Arc;
 #[cfg(feature = "parallel")]
 use crate::parallel_read;
@@ -59,12 +61,7 @@ fn decompress_all_chunks(
    for chunk_info in chunks {
        let c_addr = chunk_info.address as usize;
        let size = chunk_info.chunk_size as usize;
-        if c_addr + size > file_data.len() {
+        ensure_len(file_data, c_addr, size)?;
            return Err(FormatError::UnexpectedEof {
                expected: c_addr + size,
                available: file_data.len(),
            });
        }
        let raw_chunk = &file_data[c_addr..c_addr + size];
        let decompressed = if let Some(pl) = pipeline {
@@ -120,6 +117,21 @@ pub struct ChunkInfo {
    pub address: u64,
 }
 /// Checks that `[offset, offset + needed)` fits within `data`, guarding the
 /// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
 fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
    if offset
        .checked_add(needed)
        .is_none_or(|end| end > data.len())
    {
        return Err(FormatError::UnexpectedEof {
            expected: offset.saturating_add(needed),
            available: data.len(),
        });
    }
    Ok(())
 }
 fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
    let s = size as usize;
    if pos.checked_add(s).is_none_or(|end| end > data.len()) {
@@ -148,19 +160,33 @@ pub fn collect_chunk_info(
    btree_address: u64,
    ndims: usize,
    offset_size: u8,
-    _length_size: u8,
+    length_size: u8,
 ) -> Result<Vec<ChunkInfo>, FormatError> {
    collect_chunk_info_inner(file_data, btree_address, ndims, offset_size, length_size, 0)
 }
 /// Maximum recursion depth for chunk B-tree traversal (malformed/cyclic data
 /// protection), matching `btree_v1.rs`'s `MAX_BTREE_DEPTH`.
 const MAX_CHUNK_BTREE_DEPTH: usize = 64;
 fn collect_chunk_info_inner(
    file_data: &[u8],
    btree_address: u64,
    ndims: usize,
    offset_size: u8,
    _length_size: u8,
    depth: usize,
 ) -> Result<Vec<ChunkInfo>, FormatError> {
    if depth > MAX_CHUNK_BTREE_DEPTH {
        return Err(FormatError::NestingDepthExceeded);
    }
    let offset = btree_address as usize;
    let os = offset_size as usize;
    // Parse B-tree v1 header
    let header_size = 8 + os * 2;
-    if offset + header_size > file_data.len() {
+    ensure_len(file_data, offset, header_size)?;
        return Err(FormatError::UnexpectedEof {
            expected: offset + header_size,
            available: file_data.len(),
        });
    }
    if &file_data[offset..offset + 4] != b"TREE" {
        return Err(FormatError::InvalidBTreeSignature);
@@ -183,12 +209,7 @@ pub fn collect_chunk_info(
        // Leaf node: keys and children interleaved
        // key[0], child[0], key[1], child[1], ..., key[N-1], child[N-1], key[N]
        let needed = entries_used * (key_size + os) + key_size;
-        if pos + needed > file_data.len() {
+        ensure_len(file_data, pos, needed)?;
            return Err(FormatError::UnexpectedEof {
                expected: pos + needed,
                available: file_data.len(),
            });
        }
        let mut chunks = Vec::with_capacity(entries_used);
        for _ in 0..entries_used {
@@ -229,12 +250,7 @@ pub fn collect_chunk_info(
    } else {
        // Internal node: recurse into children
        let needed = entries_used * (key_size + os) + key_size;
-        if pos + needed > file_data.len() {
+        ensure_len(file_data, pos, needed)?;
            return Err(FormatError::UnexpectedEof {
                expected: pos + needed,
                available: file_data.len(),
            });
        }
        let mut child_addrs = Vec::with_capacity(entries_used);
        for _ in 0..entries_used {
@@ -246,8 +262,14 @@ pub fn collect_chunk_info(
        let mut all_chunks = Vec::new();
        for child_addr in child_addrs {
-            let child_chunks =
+            let child_chunks = collect_chunk_info_inner(
-                collect_chunk_info(file_data, child_addr, ndims, offset_size, _length_size)?;
+                file_data,
                child_addr,
                ndims,
                offset_size,
                _length_size,
                depth + 1,
            )?;
            all_chunks.extend(child_chunks);
        }
        Ok(all_chunks)
@@ -345,7 +367,9 @@ pub fn read_chunked_data(
    // Both v3 and v4 include element size as last dim (rank+1)
    let ndims = chunk_dimensions.len();
-    let rank = ndims - 1;
+    let rank = ndims
        .checked_sub(1)
        .ok_or_else(|| FormatError::ChunkedReadError("chunked layout has no dimensions".into()))?;
    let chunk_dims: Vec<usize> = chunk_dimensions[..rank]
        .iter()
        .map(|&d| d as usize)
@@ -384,24 +408,24 @@ pub fn read_chunked_data(
        }
        (4, Some(2)) => {
            // Implicit index — use spatial chunk dims only
-            let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+            let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
            generate_implicit_chunks(
                addr,
                &dataspace.dimensions,
-                &spatial_chunk_dims,
+                spatial_chunk_dims,
                elem_size as u32,
            )
        }
        (4, Some(3)) => {
            // Fixed Array — use spatial chunk dims only
-            let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+            let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
            let header =
                FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
            read_fixed_array_chunks(
                file_data,
                &header,
                &dataspace.dimensions,
-                &spatial_chunk_dims,
+                spatial_chunk_dims,
                elem_size as u32,
                offset_size,
                length_size,
@@ -409,14 +433,14 @@ pub fn read_chunked_data(
        }
        (4, Some(4)) => {
            // Extensible Array — use spatial chunk dims only
-            let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+            let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
            let header =
                ExtensibleArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
            read_extensible_array_chunks(
                file_data,
                &header,
                &dataspace.dimensions,
-                &spatial_chunk_dims,
+                spatial_chunk_dims,
                elem_size as u32,
                offset_size,
                length_size,
@@ -459,12 +483,7 @@ pub fn read_chunked_data(
            let c_addr = chunk_info.address as usize;
            let size = chunk_info.chunk_size as usize;
-            if c_addr + size > file_data.len() {
+            ensure_len(file_data, c_addr, size)?;
                return Err(FormatError::UnexpectedEof {
                    expected: c_addr + size,
                    available: file_data.len(),
                });
            }
            let chunk_data = &file_data[c_addr..c_addr + size];
            if rank == 0 {
@@ -577,7 +596,9 @@ pub fn read_chunked_data_cached(
    let elem_size = datatype.type_size() as usize;
    let ndims = chunk_dimensions.len();
-    let rank = ndims - 1;
+    let rank = ndims
        .checked_sub(1)
        .ok_or_else(|| FormatError::ChunkedReadError("chunked layout has no dimensions".into()))?;
    let chunk_dims: Vec<usize> = chunk_dimensions[..rank]
        .iter()
        .map(|&d| d as usize)
@@ -616,30 +637,30 @@ pub fn read_chunked_data_cached(
                }]
            }
            (4, Some(2)) => {
-                let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
                generate_implicit_chunks(
                    addr,
                    &dataspace.dimensions,
-                    &spatial_chunk_dims,
+                    spatial_chunk_dims,
                    elem_size as u32,
                )
            }
            (4, Some(3)) => {
-                let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
                let header =
                    FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
                read_fixed_array_chunks(
                    file_data,
                    &header,
                    &dataspace.dimensions,
-                    &spatial_chunk_dims,
+                    spatial_chunk_dims,
                    elem_size as u32,
                    offset_size,
                    length_size,
                )?
            }
            (4, Some(4)) => {
-                let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
                let header = ExtensibleArrayHeader::parse(
                    file_data,
                    addr as usize,
@@ -650,7 +671,7 @@ pub fn read_chunked_data_cached(
                    file_data,
                    &header,
                    &dataspace.dimensions,
-                    &spatial_chunk_dims,
+                    spatial_chunk_dims,
                    elem_size as u32,
                    offset_size,
                    length_size,
@@ -689,18 +710,13 @@ pub fn read_chunked_data_cached(
        let coord: Vec<u64> = chunk_info.offsets.iter().take(rank).copied().collect();
        // Try decompressed cache first
-        let decompressed = if let Some(cached) = cache.get_decompressed(&coord) {
+        let decompressed = if let Some(cached) = cache.get_decompressed_aligned(&coord) {
            cached
        } else {
            // Decompress from file
            let c_addr = chunk_info.address as usize;
            let size = chunk_info.chunk_size as usize;
-            if c_addr + size > file_data.len() {
+            ensure_len(file_data, c_addr, size)?;
                return Err(FormatError::UnexpectedEof {
                    expected: c_addr + size,
                    available: file_data.len(),
                });
            }
            let raw_chunk = &file_data[c_addr..c_addr + size];
            let dec = if let Some(pl) = pipeline {
                if chunk_info.filter_mask == 0 {
@@ -711,8 +727,7 @@ pub fn read_chunked_data_cached(
            } else {
                raw_chunk.to_vec()
            };
-            cache.put_decompressed(coord, dec.clone());
+            cache.put_decompressed(coord, dec)
            dec
        };
        let chunk_offsets: Vec<usize> = chunk_info
@@ -934,7 +949,9 @@ pub fn read_chunked_data_sweep(
    let elem_size = datatype.type_size() as usize;
    let ndims = chunk_dimensions.len();
-    let rank = ndims - 1;
+    let rank = ndims
        .checked_sub(1)
        .ok_or_else(|| FormatError::ChunkedReadError("chunked layout has no dimensions".into()))?;
    let chunk_dims: Vec<usize> = chunk_dimensions[..rank]
        .iter()
        .map(|&d| d as usize)
@@ -973,30 +990,30 @@ pub fn read_chunked_data_sweep(
                }]
            }
            (4, Some(2)) => {
-                let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
                generate_implicit_chunks(
                    addr,
                    &dataspace.dimensions,
-                    &spatial_chunk_dims,
+                    spatial_chunk_dims,
                    elem_size as u32,
                )
            }
            (4, Some(3)) => {
-                let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
                let header =
                    FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
                read_fixed_array_chunks(
                    file_data,
                    &header,
                    &dataspace.dimensions,
-                    &spatial_chunk_dims,
+                    spatial_chunk_dims,
                    elem_size as u32,
                    offset_size,
                    length_size,
                )?
            }
            (4, Some(4)) => {
-                let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
                let header = ExtensibleArrayHeader::parse(
                    file_data,
                    addr as usize,
@@ -1007,7 +1024,7 @@ pub fn read_chunked_data_sweep(
                    file_data,
                    &header,
                    &dataspace.dimensions,
-                    &spatial_chunk_dims,
+                    spatial_chunk_dims,
                    elem_size as u32,
                    offset_size,
                    length_size,
@@ -1055,18 +1072,13 @@ pub fn read_chunked_data_sweep(
        }
        // Try decompressed cache first
-        let decompressed = if let Some(cached) = cache.get_decompressed(&coord) {
+        let decompressed = if let Some(cached) = cache.get_decompressed_aligned(&coord) {
            cached
        } else {
            // Decompress from file
            let c_addr = chunk_info.address as usize;
            let size = chunk_info.chunk_size as usize;
-            if c_addr + size > file_data.len() {
+            ensure_len(file_data, c_addr, size)?;
                return Err(FormatError::UnexpectedEof {
                    expected: c_addr + size,
                    available: file_data.len(),
                });
            }
            let raw_chunk = &file_data[c_addr..c_addr + size];
            let dec = if let Some(pl) = pipeline {
                if chunk_info.filter_mask == 0 {
@@ -1077,8 +1089,7 @@ pub fn read_chunked_data_sweep(
            } else {
                raw_chunk.to_vec()
            };
-            cache.put_decompressed(coord, dec.clone());
+            cache.put_decompressed(coord, dec)
            dec
        };
        let chunk_offsets: Vec<usize> = chunk_info
@@ -1161,7 +1172,9 @@ pub fn read_chunked_data_indexed(
    let elem_size = datatype.type_size() as usize;
    let ndims = chunk_dimensions.len();
-    let rank = ndims - 1;
+    let rank = ndims
        .checked_sub(1)
        .ok_or_else(|| FormatError::ChunkedReadError("chunked layout has no dimensions".into()))?;
    let chunk_dims: Vec<usize> = chunk_dimensions[..rank]
        .iter()
        .map(|&d| d as usize)
@@ -1200,30 +1213,30 @@ pub fn read_chunked_data_indexed(
                }]
            }
            (4, Some(2)) => {
-                let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
                generate_implicit_chunks(
                    addr,
                    &dataspace.dimensions,
-                    &spatial_chunk_dims,
+                    spatial_chunk_dims,
                    elem_size as u32,
                )
            }
            (4, Some(3)) => {
-                let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
                let header =
                    FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
                read_fixed_array_chunks(
                    file_data,
                    &header,
                    &dataspace.dimensions,
-                    &spatial_chunk_dims,
+                    spatial_chunk_dims,
                    elem_size as u32,
                    offset_size,
                    length_size,
                )?
            }
            (4, Some(4)) => {
-                let spatial_chunk_dims: Vec<u32> = chunk_dimensions[..rank].to_vec();
+                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
                let header = ExtensibleArrayHeader::parse(
                    file_data,
                    addr as usize,
@@ -1234,7 +1247,7 @@ pub fn read_chunked_data_indexed(
                    file_data,
                    &header,
                    &dataspace.dimensions,
-                    &spatial_chunk_dims,
+                    spatial_chunk_dims,
                    elem_size as u32,
                    offset_size,
                    length_size,
@@ -1271,19 +1284,14 @@ pub fn read_chunked_data_indexed(
        .ok_or_else(|| FormatError::ChunkedReadError("chunk layout not available".into()))?;
    // Decompress chunks (using LRU cache where possible)
-    let mut chunk_buffers: Vec<CacheAlignedBuffer> = Vec::with_capacity(mappings_info.len());
+    let mut chunk_buffers: Vec<Arc<CacheAlignedBuffer>> = Vec::with_capacity(mappings_info.len());
    for (coord, file_offset, file_size, filter_mask) in &mappings_info {
        if let Some(cached) = cache.get_decompressed_aligned(coord) {
            chunk_buffers.push(cached);
        } else {
            let c_addr = *file_offset as usize;
            let size = *file_size as usize;
-            if c_addr + size > file_data.len() {
+            ensure_len(file_data, c_addr, size)?;
                return Err(FormatError::UnexpectedEof {
                    expected: c_addr + size,
                    available: file_data.len(),
                });
            }
            let raw_chunk = &file_data[c_addr..c_addr + size];
            let decompressed = if let Some(pl) = pipeline {
                if *filter_mask == 0 {
@@ -1295,8 +1303,8 @@ pub fn read_chunked_data_indexed(
                raw_chunk.to_vec()
            };
            let aligned = CacheAlignedBuffer::from_vec(decompressed);
-            cache.put_decompressed_aligned(coord.clone(), aligned.clone());
+            let arc = cache.put_decompressed_aligned(coord.clone(), aligned);
-            chunk_buffers.push(aligned);
+            chunk_buffers.push(arc);
        }
    }
@@ -1331,9 +1339,18 @@ fn copy_chunk_to_output(
        // Fast path for 1-D: single contiguous copy per chunk
        let global_start = chunk_offsets[0];
        let copy_len = chunk_dims[0].min(ds_dims[0].saturating_sub(global_start));
-        let src_bytes = copy_len * elem_size;
+        let (Some(src_bytes), Some(dst_start)) = (
-        let dst_start = global_start * elem_size;
+            copy_len.checked_mul(elem_size),
-        if src_bytes > 0 && dst_start + src_bytes <= output.len() && src_bytes <= chunk_data.len() {
+            global_start.checked_mul(elem_size),
        ) else {
            return;
        };
        if src_bytes > 0
            && dst_start
                .checked_add(src_bytes)
                .is_some_and(|end| end <= output.len())
            && src_bytes <= chunk_data.len()
        {
            output[dst_start..dst_start + src_bytes].copy_from_slice(&chunk_data[..src_bytes]);
        }
        return;
@@ -1343,19 +1360,29 @@ fn copy_chunk_to_output(
    let inner_dim = rank - 1;
    let inner_chunk_len =
        chunk_dims[inner_dim].min(ds_dims[inner_dim].saturating_sub(chunk_offsets[inner_dim]));
-    let row_bytes = inner_chunk_len * elem_size;
+    let Some(row_bytes) = inner_chunk_len.checked_mul(elem_size) else {
        return;
    };
    if row_bytes == 0 {
        return;
    }
    // Number of rows = product of all outer chunk dimensions
-    let outer_count: usize = chunk_dims[..inner_dim].iter().product();
+    let Some(outer_count) = chunk_dims[..inner_dim]
        .iter()
        .try_fold(1usize, |acc, &d| acc.checked_mul(d))
    else {
        return;
    };
    // Outer strides for iterating chunk-local coordinates
    let mut outer_strides = vec![1usize; inner_dim];
    for i in (0..inner_dim.saturating_sub(1)).rev() {
-        outer_strides[i] = outer_strides[i + 1] * chunk_dims[i + 1];
+        let Some(stride) = outer_strides[i + 1].checked_mul(chunk_dims[i + 1]) else {
            return;
        };
        outer_strides[i] = stride;
    }
    for outer_idx in 0..outer_count {
@@ -1375,13 +1402,29 @@ fn copy_chunk_to_output(
                remaining %= outer_strides[d];
            }
-            let global_coord = chunk_offsets[d] + coord_in_chunk;
+            let Some(global_coord) = chunk_offsets[d].checked_add(coord_in_chunk) else {
                out_of_bounds = true;
                break;
            };
            if global_coord >= ds_dims[d] {
                out_of_bounds = true;
                break;
            }
-            ds_flat += global_coord * ds_strides[d];
+            let (Some(ds_term), Some(src_term)) = (
-            src_flat += coord_in_chunk * chunk_strides[d];
+                global_coord.checked_mul(ds_strides[d]),
                coord_in_chunk.checked_mul(chunk_strides[d]),
            ) else {
                out_of_bounds = true;
                break;
            };
            let (Some(new_ds_flat), Some(new_src_flat)) =
                (ds_flat.checked_add(ds_term), src_flat.checked_add(src_term))
            else {
                out_of_bounds = true;
                break;
            };
            ds_flat = new_ds_flat;
            src_flat = new_src_flat;
        }
        if out_of_bounds {
@@ -1389,12 +1432,27 @@ fn copy_chunk_to_output(
        }
        // Add innermost dimension offset
-        ds_flat += chunk_offsets[inner_dim] * ds_strides[inner_dim];
+        let Some(inner_term) = chunk_offsets[inner_dim].checked_mul(ds_strides[inner_dim]) else {
            continue;
        };
        let Some(ds_flat) = ds_flat.checked_add(inner_term) else {
            continue;
        };
-        let src_start = src_flat * elem_size;
+        let (Some(src_start), Some(dst_start)) = (
-        let dst_start = ds_flat * elem_size;
+            src_flat.checked_mul(elem_size),
            ds_flat.checked_mul(elem_size),
        ) else {
            continue;
        };
-        if src_start + row_bytes <= chunk_data.len() && dst_start + row_bytes <= output.len() {
+        let fits = src_start
            .checked_add(row_bytes)
            .is_some_and(|end| end <= chunk_data.len())
            && dst_start
                .checked_add(row_bytes)
                .is_some_and(|end| end <= output.len());
        if fits {
            output[dst_start..dst_start + row_bytes]
                .copy_from_slice(&chunk_data[src_start..src_start + row_bytes]);
        }
@@ -1639,6 +1697,82 @@ mod tests {
        (file_data, layout, dataspace)
    }
    #[test]
    fn read_chunked_data_rejects_zero_dim_chunk_layout() {
        // Found by fuzzing: chunk_dimensions.len() == 0 caused `ndims - 1` to
        // underflow. A malformed/degenerate chunked layout must error cleanly.
        let layout = DataLayout::Chunked {
            chunk_dimensions: vec![],
            btree_address: Some(0),
            version: 3,
            chunk_index_type: None,
            single_chunk_filtered_size: None,
            single_chunk_filter_mask: None,
        };
        let dataspace = Dataspace {
            space_type: DataspaceType::Simple,
            rank: 1,
            dimensions: vec![10],
            max_dimensions: None,
        };
        let datatype = make_f64_type();
        let file_data = vec![0u8; 64];
        let result = read_chunked_data(&file_data, &layout, &dataspace, &datatype, None, 8, 8);
        assert!(
            matches!(result, Err(FormatError::ChunkedReadError(_))),
            "expected a clean ChunkedReadError, got {result:?}"
        );
    }
    #[test]
    fn copy_chunk_to_output_1d_rejects_overflowing_offset_without_panicking() {
        // Found by fuzzing: `global_start * elem_size` overflowed for a
        // crafted large chunk offset.
        let chunk_data = vec![1u8; 16];
        let mut output = vec![0u8; 16];
        let chunk_offsets = [usize::MAX - 1];
        let chunk_dims = [1usize];
        let ds_dims = [usize::MAX];
        let ds_strides = [1usize];
        let chunk_strides = [1usize];
        copy_chunk_to_output(
            &chunk_data,
            &mut output,
            &chunk_offsets,
            &chunk_dims,
            &ds_dims,
            &ds_strides,
            &chunk_strides,
            8,
            1,
        );
        // No panic; the out-of-range write was skipped, output left untouched.
        assert_eq!(output, vec![0u8; 16]);
    }
    #[test]
    fn copy_chunk_to_output_nd_rejects_overflowing_offset_without_panicking() {
        let chunk_data = vec![1u8; 16];
        let mut output = vec![0u8; 16];
        let chunk_offsets = [usize::MAX - 1, 0];
        let chunk_dims = [1usize, 1usize];
        let ds_dims = [usize::MAX, usize::MAX];
        let ds_strides = [1usize, 1usize];
        let chunk_strides = [1usize, 1usize];
        copy_chunk_to_output(
            &chunk_data,
            &mut output,
            &chunk_offsets,
            &chunk_dims,
            &ds_dims,
            &ds_strides,
            &chunk_strides,
            8,
            2,
        );
        assert_eq!(output, vec![0u8; 16]);
    }
    #[test]
    fn read_1d_two_chunks_no_compression() {
        let values: Vec<f64> = (0..20).map(|i| i as f64).collect();
@@ -1851,6 +1985,54 @@ mod tests {
        assert_eq!(err, FormatError::InvalidBTreeNodeType(0));
    }
    #[test]
    fn collect_chunk_info_rejects_near_usize_max_offset() {
        let file_data = vec![0u8; 64];
        let result = collect_chunk_info(&file_data, u64::MAX - 4, 2, 8, 8);
        assert!(
            matches!(result, Err(FormatError::UnexpectedEof { .. })),
            "expected a clean UnexpectedEof, got {result:?}"
        );
    }
    #[test]
    fn collect_chunk_info_rejects_self_referencing_internal_node() {
        // A type-1 internal node (level 1) whose single child address points
        // back to itself: an infinite-recursion / cyclic B-tree attack.
        let ndims = 2;
        let os: u8 = 8;
        let mut buf = Vec::new();
        buf.extend_from_slice(b"TREE");
        buf.push(1); // node_type = 1 (raw data chunks)
        buf.push(1); // node_level = 1 (internal)
        buf.extend_from_slice(&1u16.to_le_bytes()); // entries_used = 1
        write_offset(&mut buf, u64::MAX, os); // left sibling undefined
        write_offset(&mut buf, u64::MAX, os); // right sibling undefined
        // key[0]: chunk_size(4) + filter_mask(4) + ndims offsets
        buf.extend_from_slice(&0u32.to_le_bytes());
        buf.extend_from_slice(&0u32.to_le_bytes());
        for _ in 0..ndims {
            write_offset(&mut buf, 0, os);
        }
        // child[0]: points back to offset 0 (this same node) — cyclic.
        write_offset(&mut buf, 0, os);
        // final key
        buf.extend_from_slice(&0u32.to_le_bytes());
        buf.extend_from_slice(&0u32.to_le_bytes());
        for _ in 0..ndims {
            write_offset(&mut buf, u64::MAX, os);
        }
        let mut file_data = vec![0u8; 256];
        file_data[..buf.len()].copy_from_slice(&buf);
        let result = collect_chunk_info(&file_data, 0, ndims, os, os);
        assert!(
            matches!(result, Err(FormatError::NestingDepthExceeded)),
            "expected a clean NestingDepthExceeded, got {result:?}"
        );
    }
    // --- Implicit chunk generation tests ---
    #[test]
@@ -65,10 +65,8 @@ impl ChunkOptions {
    pub fn build_pipeline(&self, element_size: u32) -> Option<FilterPipeline> {
        let mut filters = Vec::new();
-        let has_compression = self.deflate_level.is_some()
+        let has_compression =
-            || self.zstd_level.is_some()
+            self.deflate_level.is_some() || self.zstd_level.is_some() || self.lz4 || self.pcodec;
            || self.lz4
            || self.pcodec;
        // Shuffle before compression. Applied if explicitly requested OR if compression
        // is active and the caller hasn't disabled it — matches h5py default behavior
@@ -608,7 +606,7 @@ pub fn precompress_chunks(
    let chunks = raw_chunks
        .into_iter()
-        .zip(compressed.into_iter())
+        .zip(compressed)
        .map(|((_offsets, raw_bytes), c)| (raw_bytes.len() as u64, c))
        .collect();
@@ -755,7 +753,11 @@ pub fn build_chunked_data_at_ext(
    maxshape: Option<&[u64]>,
 ) -> Result<ChunkedDataResult, FormatError> {
    let pre = precompress_chunks(raw_data, shape, chunk_dims, element_size, options)?;
-    Ok(build_chunked_data_from_precompressed(&pre, base_address, maxshape))
+    Ok(build_chunked_data_from_precompressed(
        &pre,
        base_address,
        maxshape,
    ))
 }
 /// Write selected elements into an existing in-memory dataset buffer.
@@ -821,7 +821,8 @@ mod tests {
        let blob = [
            0x00u8, // block version 0
            0x01, 0, 0, 0, 0, 0, 0, 0, // nused = 1
-            0x73, 0x72, 0x63, 0x5f, 0x65, 0x78, 0x74, 0x2e, 0x68, 0x35, 0x00, // "src_ext.h5\0"
+            0x73, 0x72, 0x63, 0x5f, 0x65, 0x78, 0x74, 0x2e, 0x68, 0x35,
            0x00, // "src_ext.h5\0"
            0x64, 0x61, 0x74, 0x61, 0x00, // "data\0"
            0x03, 0, 0, 0, 0x01, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // source sel = ALL
            0x03, 0, 0, 0, 0x01, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, // virtual sel = ALL
@@ -17,6 +17,21 @@ use crate::datatype::{Datatype, DatatypeByteOrder};
 use crate::error::FormatError;
 use crate::filter_pipeline::FilterPipeline;
 /// Checks that `[offset, offset + needed)` fits within `data`, guarding the
 /// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
 fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
    if offset
        .checked_add(needed)
        .is_none_or(|end| end > data.len())
    {
        return Err(FormatError::UnexpectedEof {
            expected: offset.saturating_add(needed),
            available: data.len(),
        });
    }
    Ok(())
 }
 /// Zero-copy read of contiguous raw data, returning a borrowed slice.
 ///
 /// For contiguous layouts, returns a direct `&[u8]` slice into `file_data`.
@@ -47,12 +62,7 @@ pub fn read_raw_data_zerocopy<'a>(
                    actual: sz,
                });
            }
-            if addr + sz > file_data.len() {
+            ensure_len(file_data, addr, sz)?;
                return Err(FormatError::UnexpectedEof {
                    expected: addr + sz,
                    available: file_data.len(),
                });
            }
            Ok(Some(&file_data[addr..addr + sz]))
        }
        _ => Ok(None),
@@ -94,7 +104,14 @@ pub fn read_raw_data_full(
    length_size: u8,
 ) -> Result<Vec<u8>, FormatError> {
    read_raw_data_full_impl(
-        file_data, layout, dataspace, datatype, pipeline, offset_size, length_size, None,
+        file_data,
        layout,
        dataspace,
        datatype,
        pipeline,
        offset_size,
        length_size,
        None,
    )
 }
@@ -112,7 +129,14 @@ pub fn read_raw_data_full_with_resolver(
    resolver: Option<&VdsSourceResolver>,
 ) -> Result<Vec<u8>, FormatError> {
    read_raw_data_full_impl(
-        file_data, layout, dataspace, datatype, pipeline, offset_size, length_size, resolver,
+        file_data,
        layout,
        dataspace,
        datatype,
        pipeline,
        offset_size,
        length_size,
        resolver,
    )
 }
@@ -155,12 +179,7 @@ fn read_raw_data_full_impl(
                    actual: sz,
                });
            }
-            if addr + sz > file_data.len() {
+            ensure_len(file_data, addr, sz)?;
                return Err(FormatError::UnexpectedEof {
                    expected: addr + sz,
                    available: file_data.len(),
                });
            }
            Ok(file_data[addr..addr + sz].to_vec())
        }
        DataLayout::Chunked { .. } => read_chunked_data(
@@ -465,8 +484,8 @@ fn read_virtual_data(
        FormatError::ChunkedReadError("virtual dataset has no mapping global heap".into())
    })?;
    let coll = GlobalHeapCollection::parse(file_data, addr as usize, length_size)?;
-    let obj = coll
+    let obj =
-        .get_object(global_heap_index as u16)
+        coll.get_object(global_heap_index as u16)
            .ok_or(FormatError::GlobalHeapObjectNotFound {
                collection_address: addr,
                index: global_heap_index as u16,
@@ -1204,6 +1223,15 @@ pub fn read_compound_fields(
            for m in members {
                let field_size = m.datatype.type_size() as usize;
                let offset = m.byte_offset as usize;
                if offset
                    .checked_add(field_size)
                    .is_none_or(|end| end > elem_size)
                {
                    return Err(FormatError::Overflow(format!(
                        "compound member '{}': byte_offset({offset}) + field_size({field_size}) exceeds element size({elem_size})",
                        m.name
                    )));
                }
                let mut field_raw = Vec::with_capacity(count * field_size);
                for i in 0..count {
                    let elem_start = i * elem_size + offset;
@@ -1815,13 +1843,19 @@ mod tests {
            0xff, 0xff, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, 0xe8, 0x03, 0x00, 0x00, 0x00, 0x80,
            0x00, 0x00,
        ];
-        assert_eq!(read_as_i32(&raw, &arr).unwrap(), vec![-1, 100, 1000, -32768]);
+        assert_eq!(
            read_as_i32(&raw, &arr).unwrap(),
            vec![-1, 100, 1000, -32768]
        );
        // Nested array-of-array unwraps recursively.
        let nested = Datatype::Array {
            base_type: Box::new(arr),
            dimensions: vec![2],
        };
-        assert_eq!(read_as_i32(&raw, &nested).unwrap(), vec![-1, 100, 1000, -32768]);
+        assert_eq!(
            read_as_i32(&raw, &nested).unwrap(),
            vec![-1, 100, 1000, -32768]
        );
    }
    fn make_f64_le_type() -> Datatype {
@@ -2096,6 +2130,43 @@ mod tests {
        assert_eq!(id_vals, vec![10, 20]);
    }
    #[test]
    fn read_compound_rejects_byte_offset_overrun() {
        use crate::datatype::CompoundMember;
        // Compound declares size=8, but the member's byte_offset(4) + its
        // field_size(8, f64) = 12 > 8 — a crafted out-of-range byte_offset.
        let dt = Datatype::Compound {
            size: 8,
            members: vec![CompoundMember {
                name: "bad".to_string(),
                byte_offset: 4,
                datatype: make_f64_le_type(),
            }],
        };
        let raw = vec![0u8; 8]; // one element, matches declared size
        let result = read_compound_fields(&raw, &dt);
        assert!(
            matches!(result, Err(FormatError::Overflow(_))),
            "expected a clean Overflow error, got {result:?}"
        );
    }
    #[test]
    fn read_raw_data_zerocopy_rejects_near_usize_max_offset() {
        let file_data = vec![0u8; 64];
        let dataspace = make_simple_dataspace(&[4]);
        let datatype = make_i32_le_type();
        let layout = DataLayout::Contiguous {
            address: Some(u64::MAX - 4),
            size: 16,
        };
        let result = read_raw_data_zerocopy(&file_data, &layout, &dataspace, &datatype);
        assert!(
            matches!(result, Err(FormatError::UnexpectedEof { .. })),
            "expected a clean UnexpectedEof, got {result:?}"
        );
    }
    #[test]
    fn read_compound_single_field_by_name() {
        use crate::datatype::CompoundMember;
@@ -1030,14 +1030,8 @@ mod tests {
            Datatype::Compound { size, members } => {
                assert_eq!(size, 20);
                assert_eq!(members.len(), 3);
-                assert_eq!(
+                assert_eq!((members[0].name.as_str(), members[0].byte_offset), ("x", 0));
-                    (members[0].name.as_str(), members[0].byte_offset),
+                assert_eq!((members[1].name.as_str(), members[1].byte_offset), ("y", 8));
                    ("x", 0)
                );
                assert_eq!(
                    (members[1].name.as_str(), members[1].byte_offset),
                    ("y", 8)
                );
                assert_eq!(
                    (members[2].name.as_str(), members[2].byte_offset),
                    ("id", 16)
@@ -1076,7 +1070,10 @@ mod tests {
                dimensions,
            } => {
                assert_eq!(dimensions, vec![3]);
-                assert!(matches!(*base_type, Datatype::FloatingPoint { size: 8, .. }));
+                assert!(matches!(
                    *base_type,
                    Datatype::FloatingPoint { size: 8, .. }
                ));
            }
            other => panic!("expected Array, got {other:?}"),
        }
@@ -20,7 +20,7 @@
 //! ```
 #[cfg(not(feature = "std"))]
-use alloc::{string::String, vec, vec::Vec};
+use alloc::{format, string::String, vec, vec::Vec};
 #[cfg(not(feature = "std"))]
 use alloc::collections::BTreeMap;
@@ -375,7 +375,8 @@ fn build_multiblock_fractal_heap(
    let table_width: u16 = 4;
    let starting_block_size: u64 = 512;
    let dblock_header_size = 4 + 1 + os + block_offset_bytes + 4;
-    let block_capacity = |row: usize| block_size_for_row(starting_block_size, row) - dblock_header_size as u64;
+    let block_capacity =
        |row: usize| block_size_for_row(starting_block_size, row) - dblock_header_size as u64;
    // ---- Pack objects into direct blocks (row-major over the doubling table) ----
    struct Blk {
@@ -542,7 +543,26 @@ fn block_size_for_row(starting_block_size: u64, row: usize) -> u64 {
 /// Size in bytes of the FRHP header for the given offset/length sizes.
 fn frhp_header_size(os: usize, ls: usize) -> usize {
-    4 + 1 + 2 + 2 + 1 + 4 + ls + os + ls + os + ls + ls + ls + ls + ls + ls + ls + ls + 2 + ls + ls
+    4 + 1
        + 2
        + 2
        + 1
        + 4
        + ls
        + os
        + ls
        + os
        + ls
        + ls
        + ls
        + ls
        + ls
        + ls
        + ls
        + ls
        + 2
        + ls
        + ls
        + 2
        + 2
        + os
@@ -670,8 +690,7 @@ pub(crate) fn build_dense_attrs(attrs: &[AttributeMessage], base_address: u64) -
    // Pad to node_size
    btlf.resize(node_size as usize, 0);
-    let mut blob =
+    let mut blob = Vec::with_capacity(heap.blob.len() + bthd.len() + btlf.len());
        Vec::with_capacity(heap.blob.len() + bthd.len() + btlf.len());
    blob.extend_from_slice(&heap.blob);
    blob.extend_from_slice(&bthd);
    blob.extend_from_slice(&btlf);
@@ -762,8 +781,7 @@ pub(crate) fn build_dense_links(links: &[LinkMessage], base_address: u64) -> Den
    btlf.extend_from_slice(&btlf_checksum.to_le_bytes());
    btlf.resize(node_size as usize, 0);
-    let mut blob =
+    let mut blob = Vec::with_capacity(heap.blob.len() + bthd.len() + btlf.len());
        Vec::with_capacity(heap.blob.len() + bthd.len() + btlf.len());
    blob.extend_from_slice(&heap.blob);
    blob.extend_from_slice(&bthd);
    blob.extend_from_slice(&btlf);
@@ -1096,10 +1114,7 @@ impl FileWriter {
            root_attrs.push(build_attr_message(n, v));
        }
-        let is_vds: Vec<bool> = all_ds
+        let is_vds: Vec<bool> = all_ds.iter().map(|d| d.virtual_sources.is_some()).collect();
            .iter()
            .map(|d| d.virtual_sources.is_some())
            .collect();
        let is_chunked: Vec<bool> = all_ds
            .iter()
            .enumerate()
@@ -1198,7 +1213,8 @@ impl FileWriter {
                // Global heap blob size is address-independent; compute it now
                // so pass 2 can place it correctly.
                let vds_mappings = d.virtual_sources.as_deref().unwrap_or(&[]);
-                let gcol_bytes = build_global_heap_collection(&serialize_vds_mappings(vds_mappings));
+                let gcol_bytes =
                    build_global_heap_collection(&serialize_vds_mappings(vds_mappings));
                dummy_blobs.push(DataBlob {
                    data: gcol_bytes, // store heap blob here temporarily
                    oh_bytes: oh,
@@ -1216,8 +1232,11 @@ impl FileWriter {
                    elem_size,
                    &d.chunk_options,
                )?;
-                let result =
+                let result = build_chunked_data_from_precompressed(
-                    build_chunked_data_from_precompressed(&pre, dummy_cursor, d.maxshape.as_deref());
+                    &pre,
                    dummy_cursor,
                    d.maxshape.as_deref(),
                );
                dummy_cursor += result.data_bytes.len() as u64;
                let dense_blob = if ds_dense[i] {
                    Some(build_dense_attrs(&d.attrs, 0))
@@ -1391,7 +1410,10 @@ impl FileWriter {
                // Reuse precompressed chunks from Pass 1 — avoids re-compressing
                // the same data a second time.
                let result = build_chunked_data_from_precompressed(
-                    dummy_blobs[i].precompressed.as_ref().expect("chunked dataset missing precompressed cache"),
+                    dummy_blobs[i]
                        .precompressed
                        .as_ref()
                        .expect("chunked dataset missing precompressed cache"),
                    base_address,
                    d.maxshape.as_deref(),
                );
@@ -1492,7 +1514,9 @@ impl FileWriter {
        // Rebuild the root link blob with real target addresses (same size as
        // the dummy used for layout); its LinkInfo goes in the OH.
        let root_link_blob = root_link_blob_addr.map(|addr| build_dense_links(&root_links, addr));
-        let root_dl = root_link_blob.as_ref().map(|b| b.link_info_message.as_slice());
+        let root_dl = root_link_blob
            .as_ref()
            .map(|b| b.link_info_message.as_slice());
        buf.extend_from_slice(&build_group_oh(
            &root_links,
            root_dl,
@@ -1936,8 +1960,10 @@ mod tests {
        let mut fw = FileWriter::new();
        // Source datasets (real data in this file)
-        fw.create_dataset("src_a").with_f64_data(&[1.0, 2.0, 3.0, 4.0]);
+        fw.create_dataset("src_a")
-        fw.create_dataset("src_b").with_f64_data(&[5.0, 6.0, 7.0, 8.0]);
+            .with_f64_data(&[1.0, 2.0, 3.0, 4.0]);
        fw.create_dataset("src_b")
            .with_f64_data(&[5.0, 6.0, 7.0, 8.0]);
        // Virtual dataset
        fw.create_dataset("vds")
            .with_shape(&[8])
@@ -1950,13 +1976,8 @@ mod tests {
        let sig = signature::find_signature(&bytes).unwrap();
        let sb = Superblock::parse(&bytes, sig).unwrap();
        let vds_addr = resolve_path_any(&bytes, &sb, "vds").unwrap();
-        let hdr = ObjectHeader::parse(
+        let hdr =
-            &bytes,
+            ObjectHeader::parse(&bytes, vds_addr as usize, sb.offset_size, sb.length_size).unwrap();
            vds_addr as usize,
            sb.offset_size,
            sb.length_size,
        )
        .unwrap();
        let dl_data = &hdr
            .messages
@@ -1965,8 +1986,7 @@ mod tests {
            .unwrap()
            .data;
-        let mut layout =
+        let mut layout = DataLayout::parse(dl_data, sb.offset_size, sb.length_size).unwrap();
            DataLayout::parse(dl_data, sb.offset_size, sb.length_size).unwrap();
        // Before resolution, mappings field is empty.
        assert!(
@@ -2033,8 +2053,7 @@ mod tests {
            .find(|m| m.msg_type == MessageType::DataLayout)
            .unwrap()
            .data;
-        let mut layout =
+        let mut layout = DataLayout::parse(dl_data, sb.offset_size, sb.length_size).unwrap();
            DataLayout::parse(dl_data, sb.offset_size, sb.length_size).unwrap();
        layout.resolve_vds_mappings(&bytes, sb.length_size).unwrap();
        match &layout {
@@ -2112,7 +2131,10 @@ mod tests {
            .expect("external link 'remote_temp' not found in group OH");
        match &ext_link.link_target {
-            crate::link_message::LinkTarget::External { filename, object_path } => {
+            crate::link_message::LinkTarget::External {
                filename,
                object_path,
            } => {
                assert_eq!(filename, "other_file.h5");
                assert_eq!(object_path, "/temperature");
            }
@@ -4,14 +4,19 @@
 extern crate alloc;
 #[cfg(not(feature = "std"))]
-use alloc::{vec, vec::Vec};
+use alloc::{boxed::Box, vec, vec::Vec};
 use crate::error::FormatError;
 use crate::filter_pipeline::{
-    FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZ4, FILTER_NBIT, FILTER_PCODEC,
+    FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZ4, FILTER_NBIT, FILTER_PCODEC, FILTER_SCALEOFFSET,
-    FILTER_SCALEOFFSET, FILTER_SHUFFLE, FILTER_SZIP, FILTER_ZSTD, FilterPipeline,
+    FILTER_SHUFFLE, FILTER_SZIP, FILTER_ZSTD, FilterPipeline,
 };
 /// Absolute ceiling on a single decompressed chunk's output size, used only
 /// when the pipeline's declared `chunk_size` is unavailable (0). Prevents
 /// unbounded-allocation DoS from a malicious/corrupted compressed chunk.
 pub(crate) const MAX_DECOMPRESS_SIZE: usize = 256 * 1024 * 1024;
 /// Apply a filter pipeline to decompress a chunk.
 /// Filters are applied in REVERSE order for decompression.
 pub fn decompress_chunk(
@@ -25,16 +30,22 @@ pub fn decompress_chunk(
    for filter in pipeline.filters.iter().rev() {
        data = match filter.filter_id {
            FILTER_SHUFFLE => shuffle_decompress(&data, element_size as usize)?,
-            FILTER_DEFLATE => deflate_decompress(&data)?,
+            // `chunk_size` is the expected decompressed size (shuffle/fletcher32
-            FILTER_LZ4 => lz4_decompress(&data)?,
+            // are size-preserving, so it bounds these too); pass it so these
-            FILTER_ZSTD => zstd_decompress(&data)?,
+            // decoders can't be forced into unbounded allocation by a hostile
            // or corrupted compressed payload.
            FILTER_DEFLATE => deflate_decompress(&data, chunk_size)?,
            FILTER_LZ4 => lz4_decompress(&data, chunk_size)?,
            FILTER_ZSTD => zstd_decompress(&data, chunk_size)?,
            FILTER_FLETCHER32 => fletcher32_verify(&data)?,
-            FILTER_PCODEC => pcodec_decompress(&data, element_size as usize)?,
+            FILTER_PCODEC => pcodec_decompress(&data, element_size as usize, chunk_size)?,
            // `chunk_size` is the expected decompressed size; pass it so these
            // decoders can reject an element count that would over-allocate.
            FILTER_SCALEOFFSET => scaleoffset_decompress(&data, &filter.client_data, chunk_size)?,
            FILTER_NBIT => nbit_decompress(&data, &filter.client_data, chunk_size)?,
-            FILTER_SZIP => crate::filters_szip::szip_decompress(&data, &filter.client_data, chunk_size)?,
+            FILTER_SZIP => {
                crate::filters_szip::szip_decompress(&data, &filter.client_data, chunk_size)?
            }
            other => return Err(FormatError::UnsupportedFilter(other)),
        };
    }
@@ -89,6 +100,27 @@ pub fn compress_chunk(
 /// E-scale, interpreted as i32 for negative exponents), `[2]`=element count,
 /// `[4]`=element size, `[5]`=signed flag, `[6]`=byte order (1 = big-endian),
 /// `[7]`=fill defined, `[8..]`=fill value bits.
 /// `f64::powi` equivalent that works under `no_std` (no libm/std available).
 /// Exponentiation by squaring, matching `powi`'s semantics for negative
 /// exponents via reciprocal.
 fn powi_f64(base: f64, mut exp: i32) -> f64 {
    let neg = exp < 0;
    if neg {
        exp = -exp;
    }
    let mut result = 1.0f64;
    let mut b = base;
    let mut e = exp as u32;
    while e > 0 {
        if e & 1 == 1 {
            result *= b;
        }
        b *= b;
        e >>= 1;
    }
    if neg { 1.0 / result } else { result }
 }
 fn scaleoffset_decompress(
    data: &[u8],
    cd: &[u32],
@@ -207,9 +239,9 @@ fn scaleoffset_decompress(
                if has_fill_code && code == fill_code {
                    fill_value
                } else if is_escale {
-                    minval + code as f64 * 2f64.powi(scale_factor)
+                    minval + code as f64 * powi_f64(2.0, scale_factor)
                } else {
-                    minval + code as f64 / 10f64.powi(scale_factor)
+                    minval + code as f64 / powi_f64(10.0, scale_factor)
                }
            })
            .collect();
@@ -569,24 +601,48 @@ fn nbit_decompress(data: &[u8], cd: &[u32], expected_bytes: usize) -> Result<Vec
 }
 /// Decompress zlib-compressed data.
 ///
 /// `expected_bytes` is the pipeline's declared decompressed chunk size (0 if
 /// unavailable); output is rejected if it exceeds this bound (or, when
 /// unavailable, [`MAX_DECOMPRESS_SIZE`]), preventing a hostile/corrupted
 /// compressed payload from forcing unbounded allocation (a "zlib bomb").
 #[cfg(feature = "deflate")]
-fn deflate_decompress(data: &[u8]) -> Result<Vec<u8>, FormatError> {
+fn deflate_decompress(data: &[u8], expected_bytes: usize) -> Result<Vec<u8>, FormatError> {
    let limit = if expected_bytes != 0 {
        expected_bytes
    } else {
        MAX_DECOMPRESS_SIZE
    };
    // Try system zlib first on macOS (Apple's ARM64-optimized libz is ~1.4x
    // faster at decompression than zlib-ng on Apple Silicon).
    #[cfg(all(target_os = "macos", feature = "system-zlib-decompress"))]
    {
        if let Ok(result) = sysz::decompress(data) {
            if result.len() > limit {
                return Err(FormatError::DecompressionError(
                    "deflate: output exceeds expected chunk size".into(),
                ));
            }
            return Ok(result);
        }
        // Fall through to flate2 on error
    }
    use std::io::Read;
-    let mut decoder = flate2::read::ZlibDecoder::new(data);
+    let decoder = flate2::read::ZlibDecoder::new(data);
-    let mut result = Vec::new();
+    let mut result = Vec::with_capacity(limit.min(1 << 20));
    // Read one byte past the limit so an over-size stream is distinguishable
    // from one that legitimately ends exactly at the limit.
    decoder
        .take(limit as u64 + 1)
        .read_to_end(&mut result)
        .map_err(|e| FormatError::DecompressionError(e.to_string()))?;
    if result.len() > limit {
        return Err(FormatError::DecompressionError(
            "deflate: output exceeds size limit".into(),
        ));
    }
    Ok(result)
 }
@@ -659,7 +715,7 @@ mod sysz {
 }
 #[cfg(not(feature = "deflate"))]
-fn deflate_decompress(_data: &[u8]) -> Result<Vec<u8>, FormatError> {
+fn deflate_decompress(_data: &[u8], _expected_bytes: usize) -> Result<Vec<u8>, FormatError> {
    Err(FormatError::UnsupportedFilter(FILTER_DEFLATE))
 }
@@ -682,20 +738,35 @@ fn deflate_compress(_data: &[u8], _level: u32) -> Result<Vec<u8>, FormatError> {
 }
 /// Decompress LZ4 data. Format: 4 bytes LE original size + LZ4 block data.
 ///
 /// The 4-byte "original size" header is part of the attacker-controlled
 /// compressed payload itself, so it is bounded against `expected_bytes` (the
 /// pipeline's declared chunk size) before being used to size the output
 /// allocation — otherwise a crafted 4-byte value can request up to ~4 GiB.
 #[cfg(feature = "lz4")]
-fn lz4_decompress(data: &[u8]) -> Result<Vec<u8>, FormatError> {
+fn lz4_decompress(data: &[u8], expected_bytes: usize) -> Result<Vec<u8>, FormatError> {
    if data.len() < 4 {
        return Err(FormatError::DecompressionError(
            "lz4: data too short".into(),
        ));
    }
    let orig_size = u32::from_le_bytes([data[0], data[1], data[2], data[3]]) as usize;
    if expected_bytes != 0 && orig_size > expected_bytes {
        return Err(FormatError::DecompressionError(
            "lz4: declared size exceeds chunk size".into(),
        ));
    }
    if orig_size > MAX_DECOMPRESS_SIZE {
        return Err(FormatError::DecompressionError(
            "lz4: declared size exceeds limit".into(),
        ));
    }
    lz4_flex::block::decompress(&data[4..], orig_size)
        .map_err(|e| FormatError::DecompressionError(format!("lz4: {e}")))
 }
 #[cfg(not(feature = "lz4"))]
-fn lz4_decompress(_data: &[u8]) -> Result<Vec<u8>, FormatError> {
+fn lz4_decompress(_data: &[u8], _expected_bytes: usize) -> Result<Vec<u8>, FormatError> {
    Err(FormatError::UnsupportedFilter(FILTER_LZ4))
 }
@@ -715,13 +786,35 @@ fn lz4_compress(_data: &[u8]) -> Result<Vec<u8>, FormatError> {
 }
 /// Decompress zstd data.
 ///
 /// `expected_bytes` bounds the output (or [`MAX_DECOMPRESS_SIZE`] when
 /// unavailable) to guard against a zstd decompression bomb, since zstd's
 /// compression ratio can exceed 1000:1.
 #[cfg(feature = "zstd")]
-fn zstd_decompress(data: &[u8]) -> Result<Vec<u8>, FormatError> {
+fn zstd_decompress(data: &[u8], expected_bytes: usize) -> Result<Vec<u8>, FormatError> {
-    zstd::decode_all(data).map_err(|e| FormatError::DecompressionError(format!("zstd: {e}")))
+    use std::io::Read;
    let limit = if expected_bytes != 0 {
        expected_bytes
    } else {
        MAX_DECOMPRESS_SIZE
    };
    let decoder = zstd::stream::Decoder::new(data)
        .map_err(|e| FormatError::DecompressionError(format!("zstd: {e}")))?;
    let mut out = Vec::with_capacity(limit.min(1 << 20));
    decoder
        .take(limit as u64 + 1)
        .read_to_end(&mut out)
        .map_err(|e| FormatError::DecompressionError(format!("zstd: {e}")))?;
    if out.len() > limit {
        return Err(FormatError::DecompressionError(
            "zstd: output exceeds chunk size".into(),
        ));
    }
    Ok(out)
 }
 #[cfg(not(feature = "zstd"))]
-fn zstd_decompress(_data: &[u8]) -> Result<Vec<u8>, FormatError> {
+fn zstd_decompress(_data: &[u8], _expected_bytes: usize) -> Result<Vec<u8>, FormatError> {
    Err(FormatError::UnsupportedFilter(FILTER_ZSTD))
 }
@@ -982,30 +1075,74 @@ fn pcodec_compress(_data: &[u8], _element_size: usize) -> Result<Vec<u8>, Format
    Err(FormatError::UnsupportedFilter(FILTER_PCODEC))
 }
 /// `expected_bytes` bounds the number of elements decoded: the output buffer
 /// is pre-sized to exactly `expected_bytes / element_size` elements and
 /// `simple_decompress_into` never writes past it, so a corrupted/hostile pco
 /// stream cannot force over-allocation the way an unbounded `simple_decompress`
 /// (which allocates however many elements the stream claims) could.
 #[cfg(feature = "pcodec")]
-fn pcodec_decompress(data: &[u8], element_size: usize) -> Result<Vec<u8>, FormatError> {
+fn pcodec_decompress(
-    use pco::standalone::simple_decompress;
+    data: &[u8],
    element_size: usize,
    expected_bytes: usize,
 ) -> Result<Vec<u8>, FormatError> {
    use pco::standalone::simple_decompress_into;
    let limit_bytes = if expected_bytes != 0 {
        expected_bytes
    } else {
        MAX_DECOMPRESS_SIZE
    };
    let n = if element_size != 0 {
        limit_bytes / element_size
    } else {
        0
    };
    match element_size {
        4 => {
-            let nums = simple_decompress::<f32>(data)
+            let mut buf = vec![0f32; n];
            let progress = simple_decompress_into(data, &mut buf)
                .map_err(|e| FormatError::DecompressionError(format!("pco: {e}")))?;
-            Ok(nums.iter().flat_map(|x| x.to_le_bytes()).collect())
+            if !progress.finished {
                return Err(FormatError::DecompressionError(
                    "pco: stream contains more data than expected chunk size allows".into(),
                ));
            }
            buf.truncate(progress.n_processed);
            Ok(buf.iter().flat_map(|x| x.to_le_bytes()).collect())
        }
        8 => {
-            let nums = simple_decompress::<f64>(data)
+            let mut buf = vec![0f64; n];
            let progress = simple_decompress_into(data, &mut buf)
                .map_err(|e| FormatError::DecompressionError(format!("pco: {e}")))?;
-            Ok(nums.iter().flat_map(|x| x.to_le_bytes()).collect())
+            if !progress.finished {
                return Err(FormatError::DecompressionError(
                    "pco: stream contains more data than expected chunk size allows".into(),
                ));
            }
            buf.truncate(progress.n_processed);
            Ok(buf.iter().flat_map(|x| x.to_le_bytes()).collect())
        }
        _ => {
-            let nums = simple_decompress::<u32>(data)
+            let mut buf = vec![0u32; n];
            let progress = simple_decompress_into(data, &mut buf)
                .map_err(|e| FormatError::DecompressionError(format!("pco: {e}")))?;
-            Ok(nums.iter().flat_map(|x| x.to_le_bytes()).collect())
+            if !progress.finished {
                return Err(FormatError::DecompressionError(
                    "pco: stream contains more data than expected chunk size allows".into(),
                ));
            }
            buf.truncate(progress.n_processed);
            Ok(buf.iter().flat_map(|x| x.to_le_bytes()).collect())
        }
    }
 }
 #[cfg(not(feature = "pcodec"))]
-fn pcodec_decompress(_data: &[u8], _element_size: usize) -> Result<Vec<u8>, FormatError> {
+fn pcodec_decompress(
    _data: &[u8],
    _element_size: usize,
    _expected_bytes: usize,
 ) -> Result<Vec<u8>, FormatError> {
    Err(FormatError::UnsupportedFilter(FILTER_PCODEC))
 }
@@ -1021,7 +1158,7 @@ mod tests {
    fn deflate_compress_decompress_roundtrip() {
        let data: Vec<u8> = (0..256).map(|i| (i % 256) as u8).collect();
        let compressed = deflate_compress(&data, 6).unwrap();
-        let decompressed = deflate_decompress(&compressed).unwrap();
+        let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
        assert_eq!(decompressed, data);
    }
@@ -1034,7 +1171,7 @@ mod tests {
        let compressed: Vec<u8> = vec![
            120, 156, 99, 96, 100, 98, 102, 97, 101, 99, 231, 224, 4, 0, 0, 175, 0, 46,
        ];
-        let decompressed = deflate_decompress(&compressed).unwrap();
+        let decompressed = deflate_decompress(&compressed, 10).unwrap();
        assert_eq!(decompressed, vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9]);
    }
@@ -1045,7 +1182,7 @@ mod tests {
        let data = vec![0u8, 1, 2, 3, 4, 5, 6, 7, 8, 9];
        let compressed = deflate_compress(&data, 6).unwrap();
        assert!(!compressed.is_empty());
-        let decompressed = deflate_decompress(&compressed).unwrap();
+        let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
        assert_eq!(decompressed, data);
    }
@@ -1246,7 +1383,7 @@ mod tests {
    fn lz4_compress_decompress_roundtrip() {
        let data: Vec<u8> = (0..256).map(|i| (i % 256) as u8).collect();
        let compressed = lz4_compress(&data).unwrap();
-        let decompressed = lz4_decompress(&compressed).unwrap();
+        let decompressed = lz4_decompress(&compressed, data.len()).unwrap();
        assert_eq!(decompressed, data);
    }
@@ -1301,7 +1438,7 @@ mod tests {
    fn zstd_compress_decompress_roundtrip() {
        let data: Vec<u8> = (0..256).map(|i| (i % 256) as u8).collect();
        let compressed = zstd_compress(&data, 3).unwrap();
-        let decompressed = zstd_decompress(&compressed).unwrap();
+        let decompressed = zstd_decompress(&compressed, data.len()).unwrap();
        assert_eq!(decompressed, data);
    }
@@ -1372,7 +1509,10 @@ mod tests {
            0x02, 0x00, 0x00, 0x00, 0x08, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xc6, 0x00,
        ];
-        assert_eq!(scaleoffset_decompress(&raw, &cd, 0).unwrap(), i32_le(&[0, 1, 2, 3]));
+        assert_eq!(
            scaleoffset_decompress(&raw, &cd, 0).unwrap(),
            i32_le(&[0, 1, 2, 3])
        );
    }
    #[test]
@@ -1531,8 +1671,12 @@ mod tests {
    fn nbit_compound_with_array_member() {
        // Compound { a: array(2,) of i32 prec 16 @0; b: u32@8 prec 8 }, 2 elements.
        // data = [([-1,100],200), ([1000,-32768],7)].
-        let cd = [20u32, 0, 2, 3, 12, 2, 0, 2, 8, 1, 4, 0, 16, 0, 8, 1, 4, 0, 8, 0];
+        let cd = [
-        let raw = [0xff, 0xff, 0x00, 0x64, 0xc8, 0x03, 0xe8, 0x80, 0x00, 0x07, 0x00];
+            20u32, 0, 2, 3, 12, 2, 0, 2, 8, 1, 4, 0, 16, 0, 8, 1, 4, 0, 8, 0,
        ];
        let raw = [
            0xff, 0xff, 0x00, 0x64, 0xc8, 0x03, 0xe8, 0x80, 0x00, 0x07, 0x00,
        ];
        #[rustfmt::skip]
        let expected: Vec<u8> = vec![
            0xff,0xff,0x00,0x00, 0x64,0x00,0x00,0x00, 0xc8,0x00,0x00,0x00, // ([-1,100], 200)
@@ -1623,4 +1767,79 @@ mod tests {
        // Missing client data entirely.
        assert!(scaleoffset_decompress(&[0u8; 32], &[2, 0], 4).is_err());
    }
    // ----- Decompression-bomb hardening: hostile compressed data must not -----
    // ----- force unbounded allocation.                                    -----
    #[test]
    #[cfg(feature = "lz4")]
    fn lz4_decompress_rejects_oversized_orig_size() {
        // 4-byte LE header claiming ~4 GiB, followed by a few garbage bytes.
        let mut data = u32::MAX.to_le_bytes().to_vec();
        data.extend_from_slice(&[0u8; 8]);
        assert!(lz4_decompress(&data, 64).is_err());
    }
    #[test]
    #[cfg(feature = "lz4")]
    fn lz4_decompress_rejects_size_exceeding_chunk_size() {
        // orig_size (1000) is well under MAX_DECOMPRESS_SIZE but exceeds the
        // pipeline's declared chunk size (64) — must be rejected by the
        // chunk-size check specifically, not just the absolute cap.
        let mut data = 1000u32.to_le_bytes().to_vec();
        data.extend_from_slice(&[0u8; 8]);
        assert!(lz4_decompress(&data, 64).is_err());
    }
    #[test]
    #[cfg(feature = "deflate")]
    fn deflate_decompress_rejects_output_exceeding_chunk_size() {
        // A highly-compressible deflate bomb (1 MiB of zeros compresses to a
        // tiny payload); declared chunk size is far smaller than the real
        // decompressed size, so this must be rejected rather than allocating
        // the full 1 MiB.
        let data = vec![0u8; 1024 * 1024];
        let compressed = deflate_compress(&data, 6).unwrap();
        assert!(deflate_decompress(&compressed, 64).is_err());
    }
    #[test]
    #[cfg(feature = "zstd")]
    fn zstd_decompress_rejects_output_exceeding_chunk_size() {
        let data = vec![0u8; 1024 * 1024];
        let compressed = zstd_compress(&data, 3).unwrap();
        assert!(zstd_decompress(&compressed, 64).is_err());
    }
    #[test]
    #[cfg(feature = "pcodec")]
    fn pcodec_decompress_rejects_element_count_exceeding_chunk_size() {
        let data: Vec<f32> = (0..1000).map(|i| i as f32).collect();
        let raw: Vec<u8> = data.iter().flat_map(|x| x.to_le_bytes()).collect();
        let compressed = pcodec_compress(&raw, 4).unwrap();
        // Declared chunk size only fits 4 f32 elements, far fewer than the
        // 1000 the stream actually contains.
        assert!(pcodec_decompress(&compressed, 4, 16).is_err());
    }
    #[test]
    #[cfg(feature = "lz4")]
    fn decompress_chunk_rejects_hostile_lz4_size_via_public_entrypoint() {
        // The actually-exploited path: a FilterPipeline claiming a small
        // chunk_size, but whose LZ4-compressed data header claims a huge
        // decompressed size.
        use crate::filter_pipeline::{FilterDescription, FilterPipeline};
        let mut data = u32::MAX.to_le_bytes().to_vec();
        data.extend_from_slice(&[0u8; 8]);
        let pipeline = FilterPipeline {
            version: 2,
            filters: vec![FilterDescription {
                filter_id: FILTER_LZ4,
                name: None,
                flags: 0,
                client_data: vec![],
            }],
        };
        assert!(decompress_chunk(&data, &pipeline, 16, 1).is_err());
    }
 }
@@ -2,6 +2,9 @@
 //!
 //! Gated by the `szip` feature which links against the system libaec library.
 #[cfg(not(feature = "std"))]
 use alloc::vec::Vec;
 use crate::error::FormatError;
 /// Decompress SZIP-compressed data using libaec.
@@ -49,9 +52,7 @@ fn szip_decode_impl(data: &[u8], cd: &[u32], chunk_size: usize) -> Result<Vec<u8
        ));
    }
    if data.is_empty() {
-        return Err(FormatError::ChunkedReadError(
+        return Err(FormatError::ChunkedReadError("szip: empty input".into()));
            "szip: empty input".into(),
        ));
    }
    // Map HDF5 option mask to libaec flags.
@@ -112,7 +113,7 @@ mod tests {
    #[cfg(feature = "szip")]
    #[test]
    fn roundtrip_u8_msb_no_nn() {
-        use libaec_sys::{AecStream, AEC_DATA_MSB};
+        use libaec_sys::{AEC_DATA_MSB, AecStream};
        let original: Vec<u8> = (0..1024u32).map(|i| (i % 256) as u8).collect();
@@ -144,7 +145,7 @@ mod tests {
    #[cfg(feature = "szip")]
    #[test]
    fn roundtrip_u8_msb_with_nn() {
-        use libaec_sys::{AecStream, AEC_DATA_MSB, AEC_DATA_PREPROCESS};
+        use libaec_sys::{AEC_DATA_MSB, AEC_DATA_PREPROCESS, AecStream};
        let original: Vec<u8> = (0..1024u32).map(|i| (i % 256) as u8).collect();
@@ -167,6 +168,9 @@ mod tests {
        let cd = [0x20u32, 8, 8, 1024];
        let decoded = szip_decompress(&encoded, &cd, original.len())
            .expect("szip_decompress with NN must succeed");
-        assert_eq!(decoded, original, "NN round-trip must reproduce original data");
+        assert_eq!(
            decoded, original,
            "NN round-trip must reproduce original data"
        );
    }
 }
@@ -186,7 +186,8 @@ pub fn read_fixed_array_chunks(
        chunk_dimensions.iter().map(|&d| d as u64).product::<u64>() * element_size as u64;
    let mut chunks = Vec::new();
-    let push_element = |i: usize, abs: usize, chunks: &mut Vec<ChunkInfo>| -> Result<(), FormatError> {
+    let push_element =
        |i: usize, abs: usize, chunks: &mut Vec<ChunkInfo>| -> Result<(), FormatError> {
            if let Some((address, chunk_size, filter_mask)) = parse_fa_element(
                file_data,
                abs,
@@ -232,9 +233,8 @@ pub fn read_fixed_array_chunks(
    // only the final page holds fewer elements. Uninitialized pages (bit clear)
    // still occupy their slot on disk but are zero-filled, so the bitmap — not a
    // 0xFF sentinel — is what marks a whole page as unallocated.
-    let stride_overflow = || {
+    let stride_overflow =
-        FormatError::ChunkedReadError("Fixed Array page offset overflow".into())
+        || FormatError::ChunkedReadError("Fixed Array page offset overflow".into());
    };
    let npages = num_elements.div_ceil(page_nelmts);
    let bitmap_size = npages.div_ceil(8);
    let bitmap_start = elements_start;
@@ -720,10 +720,7 @@ mod tests {
        // Page 1 (elements 4,5,6,7) is uninitialized => skipped. The remaining
        // 7 chunks (0..4 and 8..11) come back with their original linear index.
        assert_eq!(chunks.len(), 7);
-        let mut got: Vec<(u64, u64)> = chunks
+        let mut got: Vec<(u64, u64)> = chunks.iter().map(|c| (c.offsets[0], c.address)).collect();
            .iter()
            .map(|c| (c.offsets[0], c.address))
            .collect();
        got.sort();
        let expect: Vec<(u64, u64)> = [0usize, 1, 2, 3, 8, 9, 10]
            .iter()
@@ -6,7 +6,7 @@ use alloc::vec::Vec;
 use crate::error::FormatError;
 /// Magic signature for global heap collections.
-const GCOL_SIGNATURE: [u8; 4] = [b'G', b'C', b'O', b'L'];
+const GCOL_SIGNATURE: [u8; 4] = *b"GCOL";
 /// A parsed global heap collection.
 #[derive(Debug, Clone)]
@@ -16,6 +16,21 @@ pub struct LocalHeap {
    pub data_segment_address: u64,
 }
 /// Checks that `[offset, offset + needed)` fits within `data`, guarding the
 /// addition against `usize` overflow from a crafted near-`usize::MAX` offset.
 fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatError> {
    if offset
        .checked_add(needed)
        .is_none_or(|end| end > data.len())
    {
        return Err(FormatError::UnexpectedEof {
            expected: offset.saturating_add(needed),
            available: data.len(),
        });
    }
    Ok(())
 }
 fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
    let s = size as usize;
    if pos.checked_add(s).is_none_or(|end| end > data.len()) {
@@ -47,12 +62,7 @@ impl LocalHeap {
        let ls = length_size as usize;
        let os = offset_size as usize;
        let total = 8 + ls * 2 + os;
-        if offset + total > file_data.len() {
+        ensure_len(file_data, offset, total)?;
            return Err(FormatError::UnexpectedEof {
                expected: offset + total,
                available: file_data.len(),
            });
        }
        if &file_data[offset..offset + 4] != b"HEAP" {
            return Err(FormatError::InvalidLocalHeapSignature);
@@ -172,6 +182,18 @@ mod tests {
        }
    }
    #[test]
    fn parse_rejects_near_usize_max_offset_without_panicking() {
        // Found by fuzzing: `offset + total` overflowed for a crafted
        // near-usize::MAX offset.
        let file = build_heap_file(0, 100, &["hello"], 8, 8);
        let result = LocalHeap::parse(&file, usize::MAX - 4, 8, 8);
        assert!(
            matches!(result, Err(FormatError::UnexpectedEof { .. })),
            "expected a clean UnexpectedEof, got {result:?}"
        );
    }
    #[test]
    fn parse_heap_header() {
        let file = build_heap_file(0, 100, &["hello", "world"], 8, 8);
@@ -9,10 +9,10 @@ use crate::error::FormatError;
 use crate::message_type::MessageType;
 /// OHDR signature for v2 object headers.
-const OHDR_SIGNATURE: [u8; 4] = [b'O', b'H', b'D', b'R'];
+const OHDR_SIGNATURE: [u8; 4] = *b"OHDR";
 /// OCHK signature for v2 continuation chunks.
-const OCHK_SIGNATURE: [u8; 4] = [b'O', b'C', b'H', b'K'];
+const OCHK_SIGNATURE: [u8; 4] = *b"OCHK";
 /// A single parsed header message.
 #[derive(Debug, Clone)]
@@ -555,8 +555,7 @@ mod tests {
        buf.push(2); // version
        buf.push(flags);
-        if has_timestamps
+        if has_timestamps && let Some((at, mt, ct, bt)) = timestamps {
            && let Some((at, mt, ct, bt)) = timestamps {
            buf.extend_from_slice(&at.to_le_bytes());
            buf.extend_from_slice(&mt.to_le_bytes());
            buf.extend_from_slice(&ct.to_le_bytes());
@@ -4,7 +4,7 @@
 //! events. The [`DefaultProfiler`] implementation uses atomic counters for
 //! thread-safe, low-overhead profiling.
-use core::sync::atomic::{AtomicU64, Ordering};
+use portable_atomic::{AtomicU64, Ordering};
 /// Trait for profiling I/O operations.
 ///
@@ -2,6 +2,9 @@
 //! data-integrity verification.
 //!
 //! Enable with the `provenance` Cargo feature (on by default).
 //!
 //! The hash is unkeyed, so this detects accidental corruption only — it is
 //! not a tamper-evidence or authenticity guarantee. See [`verify_dataset`].
 #[cfg(not(feature = "std"))]
 use alloc::{format, string::String, vec::Vec};
@@ -115,6 +118,11 @@ pub enum VerifyResult {
 ///
 /// `file_data` is the entire HDF5 file bytes; `header` is the parsed object
 /// header for the dataset of interest.
 ///
 /// This only detects *accidental* corruption. The hash is unkeyed and stored
 /// alongside the data it protects, so anyone able to modify the dataset can
 /// also recompute and overwrite `_provenance_sha256` — a `VerifyResult::Ok`
 /// is not a tamper-evidence or authenticity guarantee.
 pub fn verify_dataset(
    file_data: &[u8],
    header: &ObjectHeader,
@@ -90,6 +90,16 @@ pub fn make_i64_type() -> Datatype {
    }
 }
 pub fn make_u64_type() -> Datatype {
    Datatype::FixedPoint {
        size: 8,
        byte_order: DatatypeByteOrder::LittleEndian,
        signed: false,
        bit_offset: 0,
        bit_precision: 64,
    }
 }
 pub fn make_u8_type() -> Datatype {
    Datatype::FixedPoint {
        size: 1,
@@ -444,6 +454,25 @@ impl DatasetBuilder {
        self
    }
    /// Write a native unsigned 64-bit integer dataset. Pairs with the
    /// read side's `read_u64`/`read_as_u64`, which already support this
    /// datatype — this was the missing symmetric write-side builder
    /// (callers previously had to bit-cast through `with_i64_data` /
    /// `i64::from_ne_bytes(v.to_ne_bytes())` to round-trip full-range u64
    /// values like timestamps or IDs).
    pub fn with_u64_data(&mut self, data: &[u64]) -> &mut Self {
        self.datatype = Some(make_u64_type());
        let mut b = Vec::with_capacity(data.len() * 8);
        for &v in data {
            b.extend_from_slice(&v.to_le_bytes());
        }
        self.data = Some(b);
        if self.shape.is_none() {
            self.shape = Some(vec![data.len() as u64]);
        }
        self
    }
    pub fn with_u8_data(&mut self, data: &[u8]) -> &mut Self {
        self.datatype = Some(make_u8_type());
        self.data = Some(data.to_vec());
@@ -706,7 +706,10 @@ fn scaleoffset_float_escale_reads_as_raw() {
    let (raw, datatype, _) = read_chunked_dataset(file_data, "x");
    let values = read_as_f64(&raw, &datatype).unwrap();
    let expect: Vec<f64> = (0..20).map(|i| i as f64 * 0.25).collect();
-    assert_eq!(values, expect, "E-scale (raw + masked filter) must read verbatim");
+    assert_eq!(
        values, expect,
        "E-scale (raw + masked filter) must read verbatim"
    );
 }
 #[test]
@@ -717,26 +720,48 @@ fn v4_virtual_dataset_cycle_errors_not_overflow() {
    let offset = find_signature(file_data).unwrap();
    let sb = Superblock::parse(file_data, offset).unwrap();
    let addr = resolve_path_any(file_data, &sb, "virt").unwrap();
-    let hdr = ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
+    let hdr =
        ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
    let ds = Dataspace::parse(
-        &hdr.messages.iter().find(|m| m.msg_type == MessageType::Dataspace).unwrap().data,
+        &hdr.messages
            .iter()
            .find(|m| m.msg_type == MessageType::Dataspace)
            .unwrap()
            .data,
        sb.length_size,
    )
    .unwrap();
    let (dt, _) = Datatype::parse(
-        &hdr.messages.iter().find(|m| m.msg_type == MessageType::Datatype).unwrap().data,
+        &hdr.messages
            .iter()
            .find(|m| m.msg_type == MessageType::Datatype)
            .unwrap()
            .data,
    )
    .unwrap();
    let layout = DataLayout::parse(
-        &hdr.messages.iter().find(|m| m.msg_type == MessageType::DataLayout).unwrap().data,
+        &hdr.messages
            .iter()
            .find(|m| m.msg_type == MessageType::DataLayout)
            .unwrap()
            .data,
        sb.offset_size,
        sb.length_size,
    )
    .unwrap();
    let r = read_raw_data_full(
-        file_data, &layout, &ds, &dt, None, sb.offset_size, sb.length_size,
+        file_data,
        &layout,
        &ds,
        &dt,
        None,
        sb.offset_size,
        sb.length_size,
    );
    assert!(
        r.is_err(),
        "cyclic virtual dataset must error, not overflow"
    );
    assert!(r.is_err(), "cyclic virtual dataset must error, not overflow");
 }
 #[test]
@@ -751,16 +776,28 @@ fn v4_virtual_dataset_external_file_read() {
    let addr = resolve_path_any(virt, &sb, "virt").unwrap();
    let hdr = ObjectHeader::parse(virt, addr as usize, sb.offset_size, sb.length_size).unwrap();
    let ds = Dataspace::parse(
-        &hdr.messages.iter().find(|m| m.msg_type == MessageType::Dataspace).unwrap().data,
+        &hdr.messages
            .iter()
            .find(|m| m.msg_type == MessageType::Dataspace)
            .unwrap()
            .data,
        sb.length_size,
    )
    .unwrap();
    let (dt, _) = Datatype::parse(
-        &hdr.messages.iter().find(|m| m.msg_type == MessageType::Datatype).unwrap().data,
+        &hdr.messages
            .iter()
            .find(|m| m.msg_type == MessageType::Datatype)
            .unwrap()
            .data,
    )
    .unwrap();
    let layout = DataLayout::parse(
-        &hdr.messages.iter().find(|m| m.msg_type == MessageType::DataLayout).unwrap().data,
+        &hdr.messages
            .iter()
            .find(|m| m.msg_type == MessageType::DataLayout)
            .unwrap()
            .data,
        sb.offset_size,
        sb.length_size,
    )
@@ -790,7 +827,14 @@ fn v4_virtual_dataset_external_file_read() {
    // With no resolver, an external source is a clean error (not wrong data).
    let no_resolver = read_raw_data_full_with_resolver(
-        virt, &layout, &ds, &dt, None, sb.offset_size, sb.length_size, None,
+        virt,
        &layout,
        &ds,
        &dt,
        None,
        sb.offset_size,
        sb.length_size,
        None,
    );
    assert!(no_resolver.is_err());
 }
@@ -805,9 +849,18 @@ fn v4_paged_fixed_array_read() {
    let values = read_as_i32(&raw, &datatype).unwrap();
    assert_eq!(values.len(), 1025 * 16);
    for k in 0..1025usize {
-        assert_eq!(values[k * 16], k as i32, "chunk-start mismatch at chunk {k}");
+        assert_eq!(
            values[k * 16],
            k as i32,
            "chunk-start mismatch at chunk {k}"
        );
        for j in 1..16 {
-            assert_eq!(values[k * 16 + j], 0, "non-start element nonzero at {}", k * 16 + j);
+            assert_eq!(
                values[k * 16 + j],
                0,
                "non-start element nonzero at {}",
                k * 16 + j
            );
        }
    }
 }
@@ -11,14 +11,14 @@ categories = ["science", "graphics"]
 [dependencies]
 wgpu = { version = "28", optional = true }
-half = { version = "2.7", optional = true }
+half = { workspace = true, optional = true }
 pollster = { version = "0.4", optional = true }
 bytemuck = { version = "1", features = ["derive"], optional = true }
 thiserror = "2"
 log = "0.4"
 [dev-dependencies]
-criterion = { version = "0.5", features = ["html_reports"] }
+criterion = { workspace = true }
 rand = "0.8"
 approx = "0.5"
 pollster = "0.4"
@@ -15,13 +15,13 @@ memmap2 = { version = "0.9", optional = true }
 libc = { version = "0.2", optional = true }
 tokio = { version = "1", features = ["fs", "io-util"], optional = true }
 reqwest = { version = "0.12", features = ["json"], optional = true }
-serde = { version = "1", features = ["derive"], optional = true }
+serde = { workspace = true, optional = true }
 serde_json = { version = "1", optional = true }
 mpi = { version = "0.8", optional = true }
 [dev-dependencies]
 tokio = { version = "1", features = ["full"] }
-tempfile = "3"
+tempfile = { workspace = true }
 [features]
 default = []
@@ -19,7 +19,7 @@ clawhdf5-format = { path = "../clawhdf5-format", version = "2.1.0" }
 clawhdf5 = { path = "../clawhdf5", version = "2.1.0" }
 rusqlite = { version = "0.31", features = ["bundled"] }
 clap = { version = "4", features = ["derive"] }
-half = "2"
+half = { workspace = true }
 [dev-dependencies]
-tempfile = "3"
+tempfile = { workspace = true }
@@ -309,7 +309,8 @@ mod tests {
        insert_relation(&conn, 1, 1, "self");
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        let opts = hdf5_writer::WriteOptions {
            agent_id: "test-agent".into(),
            embedder: "test-embed".into(),
@@ -319,7 +320,8 @@ mod tests {
        };
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
+        let summary =
            validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
        assert_eq!(summary.chunks, 2);
        assert_eq!(summary.sessions, 1);
        assert_eq!(summary.entities, 1);
@@ -340,7 +342,8 @@ mod tests {
        insert_chunk(&conn, 3, "also active", &make_embedding(4, 3.0), 0);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, true, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, true, None, &SchemaConfig::default()).unwrap();
        assert_eq!(data.chunks.len(), 2);
        let opts = hdf5_writer::WriteOptions {
@@ -352,7 +355,8 @@ mod tests {
        };
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
+        let summary =
            validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
        assert_eq!(summary.chunks, 2);
    }
@@ -367,7 +371,8 @@ mod tests {
        insert_chunk(&conn, 2, "deleted", &make_embedding(4, 2.0), 1);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        assert_eq!(data.chunks.len(), 2);
    }
@@ -381,7 +386,8 @@ mod tests {
        insert_chunk(&conn, 1, "test", &make_embedding(16, 0.5), 0);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        assert_eq!(data.embedding_dim, 16);
    }
@@ -395,7 +401,8 @@ mod tests {
        insert_chunk(&conn, 1, "test", &make_embedding(16, 0.5), 0);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, Some(8), &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, Some(8), &SchemaConfig::default()).unwrap();
        assert_eq!(data.embedding_dim, 8);
        // Embedding truncated to dim 8
        assert_eq!(data.chunks[0].embedding.len(), 8);
@@ -413,7 +420,8 @@ mod tests {
        insert_chunk(&conn, 1, "test", &emb, 0);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        let opts = hdf5_writer::WriteOptions {
            agent_id: "t".into(),
            embedder: "t".into(),
@@ -424,7 +432,8 @@ mod tests {
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
        // Content-validate with the float16 tolerance enabled.
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, true, true).unwrap();
+        let summary =
            validate::validate_hdf5(h5_path.to_str().unwrap(), &data, true, true).unwrap();
        assert_eq!(summary.chunks, 1);
        // Verify float16 values are within tolerance
@@ -453,7 +462,8 @@ mod tests {
        }
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        let opts_compressed = hdf5_writer::WriteOptions {
            agent_id: "t".into(),
@@ -493,7 +503,8 @@ mod tests {
        drop(conn);
        // Simulate dry-run: read data but don't write
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        assert_eq!(data.chunks.len(), 1);
        assert!(!h5_path.exists());
    }
@@ -505,7 +516,8 @@ mod tests {
        let db_path = create_test_db(&dir);
        let h5_path = dir.path().join("out.h5");
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        assert_eq!(data.chunks.len(), 0);
        assert_eq!(data.sessions.len(), 0);
        assert_eq!(data.entities.len(), 0);
@@ -520,7 +532,8 @@ mod tests {
        };
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
+        let summary =
            validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
        assert_eq!(summary.chunks, 0);
    }
@@ -543,7 +556,8 @@ mod tests {
        }
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        assert_eq!(data.chunks.len(), 1000);
        let opts = hdf5_writer::WriteOptions {
@@ -573,7 +587,8 @@ mod tests {
        insert_session(&conn, "session-gamma", 21, 30);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        assert_eq!(data.sessions.len(), 3);
        let opts = hdf5_writer::WriteOptions {
@@ -585,7 +600,8 @@ mod tests {
        };
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
+        let summary =
            validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
        assert_eq!(summary.sessions, 3);
    }
@@ -605,7 +621,8 @@ mod tests {
        insert_relation(&conn, 2, 3, "uses");
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        assert_eq!(data.entities.len(), 3);
        assert_eq!(data.relations.len(), 3);
@@ -618,7 +635,8 @@ mod tests {
        };
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
+        let summary =
            validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
        assert_eq!(summary.entities, 3);
        assert_eq!(summary.relations, 3);
    }
@@ -634,7 +652,8 @@ mod tests {
        insert_chunk(&conn, 1, "test", &make_embedding(4, 1.0), 0);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        let opts = hdf5_writer::WriteOptions {
            agent_id: "t".into(),
            embedder: "t".into(),
@@ -645,8 +664,8 @@ mod tests {
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
        // Validating against a source with an extra (unwritten) chunk must fail.
-        let mut bigger = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default())
+        let mut bigger =
-            .unwrap();
+            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        let mut extra = bigger.chunks[0].clone();
        extra.id = 999;
        bigger.chunks.push(extra);
@@ -666,7 +685,8 @@ mod tests {
        insert_chunk(&conn, 1, "test", &make_embedding(8, 1.0), 0);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        let opts = hdf5_writer::WriteOptions {
            agent_id: "my-agent-42".into(),
            embedder: "openai-ada".into(),
@@ -712,7 +732,8 @@ mod tests {
        insert_chunk(&conn, 1, "test", &emb, 0);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        let opts = hdf5_writer::WriteOptions {
            agent_id: "t".into(),
            embedder: "t".into(),
@@ -758,7 +779,8 @@ mod tests {
        drop(conn);
        // Skip deleted
-        let data = sqlite_reader::read_sqlite(&db_path, true, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, true, None, &SchemaConfig::default()).unwrap();
        assert_eq!(data.chunks.len(), 4); // chunk 3 is deleted
        let opts = hdf5_writer::WriteOptions {
@@ -770,7 +792,8 @@ mod tests {
        };
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
-        let summary = validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
+        let summary =
            validate::validate_hdf5(h5_path.to_str().unwrap(), &data, false, false).unwrap();
        assert_eq!(summary.chunks, 4);
        assert_eq!(summary.sessions, 2);
        assert_eq!(summary.entities, 2);
@@ -789,7 +812,8 @@ mod tests {
        insert_session(&conn, "s1", 0, 10);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        let opts = hdf5_writer::WriteOptions {
            agent_id: "t".into(),
            embedder: "t".into(),
@@ -800,8 +824,8 @@ mod tests {
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
        // Validating against a source whose session content differs must fail.
-        let mut tampered = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default())
+        let mut tampered =
-            .unwrap();
+            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        tampered.sessions[0].summary = "DIFFERENT".into();
        let result = validate::validate_hdf5(h5_path.to_str().unwrap(), &tampered, false, false);
        assert!(result.is_err());
@@ -819,7 +843,8 @@ mod tests {
        insert_chunk(&conn, 1, "hello", &make_embedding(8, 1.0), 0);
        drop(conn);
-        let data = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
+        let data =
            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        let opts = hdf5_writer::WriteOptions {
            agent_id: "t".into(),
            embedder: "t".into(),
@@ -830,8 +855,8 @@ mod tests {
        hdf5_writer::write_hdf5(h5_path.to_str().unwrap(), &data, &opts).unwrap();
        // A source whose embedding differs (but counts match) must fail validation.
-        let mut tampered = sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default())
+        let mut tampered =
-            .unwrap();
+            sqlite_reader::read_sqlite(&db_path, false, None, &SchemaConfig::default()).unwrap();
        tampered.chunks[0].embedding[3] += 9.0;
        let result = validate::validate_hdf5(h5_path.to_str().unwrap(), &tampered, true, false);
        assert!(result.is_err());
@@ -856,7 +881,10 @@ mod tests {
            CREATE TABLE relations (src INTEGER, tgt INTEGER, relation TEXT, weight REAL, timestamp REAL);",
        )
        .unwrap();
-        let blob: Vec<u8> = make_embedding(4, 1.0).iter().flat_map(|v| v.to_le_bytes()).collect();
+        let blob: Vec<u8> = make_embedding(4, 1.0)
            .iter()
            .flat_map(|v| v.to_le_bytes())
            .collect();
        conn.execute(
            "INSERT INTO my_chunks VALUES (1, 'hi', ?1, 'api', 1.0, 's', '', 0)",
            rusqlite::params![blob],
@@ -908,7 +936,8 @@ mod tests {
        let base = hdf5_reader::read_hdf5(h5_path.to_str().unwrap()).unwrap();
        let max_id = base.chunks.iter().map(|c| c.id).max().unwrap_or(0);
        assert_eq!(max_id, 2);
-        let new = sqlite_reader::read_sqlite_filtered(&db_path, false, Some(4), &cfg, max_id).unwrap();
+        let new =
            sqlite_reader::read_sqlite_filtered(&db_path, false, Some(4), &cfg, max_id).unwrap();
        assert_eq!(new.chunks.len(), 2); // only id 3 and 4
        let mut merged = base;
@@ -91,7 +91,14 @@ impl Default for SchemaConfig {
            },
            sessions: TableSchema {
                table: "sessions".into(),
-                columns: vec!["id", "start_idx", "end_idx", "channel", "timestamp", "summary"],
+                columns: vec![
                    "id",
                    "start_idx",
                    "end_idx",
                    "channel",
                    "timestamp",
                    "summary",
                ],
            },
            entities: TableSchema {
                table: "entities".into(),
@@ -77,10 +77,9 @@ pub fn validate_hdf5(
        }
        for (k, (&a, &b)) in s.embedding.iter().zip(g.embedding.iter()).enumerate() {
            if (a - b).abs() > emb_abs + emb_rel * a.abs() {
-                return Err(format!(
+                return Err(
-                    "chunk[{i}].embedding[{k}] mismatch: source {a}, HDF5 {b}"
+                    format!("chunk[{i}].embedding[{k}] mismatch: source {a}, HDF5 {b}").into(),
-                )
+                );
                .into());
            }
        }
        rows_checked += 1;
@@ -108,7 +107,12 @@ pub fn validate_hdf5(
        }
        rows_checked += 1;
    }
-    for (i, (s, g)) in source.relations.iter().zip(got.relations.iter()).enumerate() {
+    for (i, (s, g)) in source
        .relations
        .iter()
        .zip(got.relations.iter())
        .enumerate()
    {
        if s.src != g.src || s.tgt != g.tgt || s.relation != g.relation {
            return Err(format!("relation[{i}] mismatch").into());
        }
@@ -14,4 +14,4 @@ clawhdf5 = { path = "../clawhdf5", version = "2.1.0" }
 clawhdf5-format = { path = "../clawhdf5-format", version = "2.1.0" }
 [dev-dependencies]
-tempfile = "3"
+tempfile = { workspace = true }
@@ -16,8 +16,8 @@ crate-type = ["cdylib", "rlib"]
 [dependencies]
 clawhdf5_rs = { path = "../clawhdf5", version = "2.1.0", package = "clawhdf5" }
 clawhdf5-format = { path = "../clawhdf5-format", version = "2.1.0" }
-pyo3 = "0.28"
+pyo3 = "0.29"
-numpy = "0.28"
+numpy = "0.29"
 [features]
 extension-module = ["pyo3/extension-module"]
@@ -4,7 +4,7 @@ build-backend = "maturin"
 [project]
 name = "rustyhdf5"
-version = "1.93.0"
+version = "2.1.0"
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 requires-python = ">=3.8"
 license = { text = "MIT" }
@@ -15,8 +15,8 @@ clawhdf5-io = { path = "../clawhdf5-io", version = "2.1.0" }
 rayon = { version = "1", optional = true }
 [dev-dependencies]
-tempfile = "3"
+tempfile = { workspace = true }
-criterion = { version = "0.5", features = ["html_reports"] }
+criterion = { workspace = true }
 clawhdf5-io = { path = "../clawhdf5-io", version = "2.1.0", features = ["mmap"] }
 clawhdf5-format = { path = "../clawhdf5-format", version = "2.1.0", features = ["parallel", "fast-checksum"] }
 clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.1.0" }
@@ -494,7 +494,9 @@ mod tests {
        let file = File::open(&path).unwrap();
        let ds = file.dataset("data").unwrap();
-        if let Ok(slice) = ds.read_f32_zerocopy() { assert_eq!(slice, &original[..]) }
+        if let Ok(slice) = ds.read_f32_zerocopy() {
            assert_eq!(slice, &original[..])
        }
        assert_eq!(ds.read_f32().unwrap(), original);
        std::fs::remove_file(&path).ok();
@@ -436,6 +436,14 @@ impl<'f> Dataset<'f> {
        &self,
        selection: &clawhdf5_format::selection::Selection,
    ) -> Result<Vec<u8>, Error> {
        // `Selection::All` is semantically a full read — route it through
        // the same per-file chunk cache `read_raw()` uses instead of the
        // selection path's uncached `read_chunked_data`, so callers get
        // consistent caching behavior regardless of which method they used
        // to ask for "everything".
        if matches!(selection, clawhdf5_format::selection::Selection::All) {
            return self.read_raw();
        }
        let dt = self.datatype()?;
        let ds = self.dataspace()?;
        let dl = self.data_layout()?;
@@ -715,7 +715,9 @@ fn multiple_chunked_datasets_share_file_cache() {
    use clawhdf5_format::datatype::{CharacterSet, Datatype, StringPadding};
    // 1-D chunked + compressed fixed-length strings (payload > compress threshold).
-    let strings: Vec<String> = (0..64).map(|i| format!("entry-{i:06}-{}", "x".repeat(80))).collect();
+    let strings: Vec<String> = (0..64)
        .map(|i| format!("entry-{i:06}-{}", "x".repeat(80)))
        .collect();
    let max_len = strings.iter().map(|s| s.len()).max().unwrap();
    let mut sraw = Vec::new();
    for s in &strings {
@@ -743,7 +745,9 @@ fn multiple_chunked_datasets_share_file_cache() {
    {
        let ds = b.create_dataset("mat");
        ds.with_f32_data(&mat).with_shape(&[n as u64, d as u64]);
-        ds.with_chunks(&[10, d as u64]).with_shuffle().with_deflate(6);
+        ds.with_chunks(&[10, d as u64])
            .with_shuffle()
            .with_deflate(6);
    }
    let bytes = b.finish().unwrap();
    let file = File::from_bytes(bytes).unwrap();
@@ -755,7 +759,10 @@ fn multiple_chunked_datasets_share_file_cache() {
    let got_mat = file.dataset("mat").unwrap().read_f32().unwrap();
    assert_eq!(got_mat, mat);
    // Read the 1-D one again to confirm the cache rebinds back correctly.
-    assert_eq!(file.dataset("strs").unwrap().read_string().unwrap(), strings);
+    assert_eq!(
        file.dataset("strs").unwrap().read_string().unwrap(),
        strings
    );
 }
 #[test]
@@ -850,8 +857,14 @@ fn dense_group_links_roundtrip() {
        );
    }
    // The small (compact) group still works.
-    assert_eq!(file.dataset("small/a").unwrap().read_f64().unwrap(), vec![1.0]);
+    assert_eq!(
-    assert_eq!(file.dataset("small/b").unwrap().read_f64().unwrap(), vec![2.0]);
+        file.dataset("small/a").unwrap().read_f64().unwrap(),
        vec![1.0]
    );
    assert_eq!(
        file.dataset("small/b").unwrap().read_f64().unwrap(),
        vec![2.0]
    );
 }
 #[test]
@@ -908,7 +921,10 @@ fn dense_links_multiblock_fractal_heap_roundtrip() {
    b.add_group(g.finish());
    let file = File::from_bytes(b.finish().unwrap()).unwrap();
-    assert_eq!(file.group("big").unwrap().datasets().unwrap().len(), n as usize);
+    assert_eq!(
        file.group("big").unwrap().datasets().unwrap().len(),
        n as usize
    );
    for i in [0, 1, 1234, n - 1] {
        assert_eq!(
            file.dataset(&format!("big/dataset_number_{i:05}"))
@@ -919,3 +935,52 @@ fn dense_links_multiblock_fractal_heap_roundtrip() {
        );
    }
 }
 #[test]
 fn read_selection_all_matches_read_raw_on_chunked_dataset() {
    // read_selection(&Selection::All) is semantically a full read and must
    // go through the same cached path as read_raw()/read_f64() — not a
    // separate uncached code path that happens to return the same bytes.
    use clawhdf5_format::selection::Selection;
    let data: Vec<f64> = (0..500).map(|i| i as f64 * 0.5).collect();
    let mut b = FileBuilder::new();
    b.create_dataset("chunked")
        .with_f64_data(&data)
        .with_chunks(&[100])
        .with_deflate(6);
    let file = File::from_bytes(b.finish().unwrap()).unwrap();
    let ds = file.dataset("chunked").unwrap();
    let via_read_f64 = ds.read_f64().unwrap();
    let via_selection_bytes = ds.read_selection(&Selection::All).unwrap();
    let via_selection: Vec<f64> = via_selection_bytes
        .chunks_exact(8)
        .map(|c| f64::from_le_bytes(c.try_into().unwrap()))
        .collect();
    assert_eq!(via_read_f64, data);
    assert_eq!(via_selection, data);
 }
 #[test]
 fn u64_data_roundtrip() {
    // Values spanning the full u64 range, including ones with the high bit
    // set that would come back negative (and wrong) if bit-cast through
    // an i64 dataset instead of a native unsigned one.
    let values: Vec<u64> = vec![
        0,
        1,
        u64::MAX,
        u64::MAX / 2,
        1 << 63,
        1_700_000_000_000_000_000,
    ];
    let mut b = FileBuilder::new();
    b.create_dataset("timestamps").with_u64_data(&values);
    let file = File::from_bytes(b.finish().unwrap()).unwrap();
    assert_eq!(
        file.dataset("timestamps").unwrap().read_u64().unwrap(),
        values
    );
 }
@@ -1,6 +1,6 @@
 {
  "name": "@redclaw/clawhdf5",
-  "version": "2.0.0",
+  "version": "2.1.0",
  "description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
  "main": "index.js",
  "types": "index.d.ts",
@@ -0,0 +1,434 @@
 # Implementation Brief — clawhdf5 Performance/Security/Provenance Pass
 **Research date:** 2026-08-16
 **Scope:** `crates/` only. Read against `ROADMAP.md`, `IMPROVEMENT_LOG.md`, `CLAUDE.md`, and
 `CHANGELOG.md` first — those documents record a genuinely large amount of prior hardening
 (WAL CRC32, `chunked_read.rs`/`data_read.rs`/`local_heap.rs`/`btree_v1.rs` bounds audits,
 Android JNI bounds checks, pyo3 bump, HNSW `prune_connections` rayon parallelism, bounded
 decompression, no_std fixes, `cargo-audit`-clean dependency tree). None of that is
 re-proposed here. Every item below was independently verified by reading the current source
 (file path + line numbers cited), not inferred from docs.
 `cargo audit` was run against the current lockfile: **zero vulnerability advisories**, three
 "unmaintained" warnings (`custom_derive` via `mpi`→`conv`, `number_prefix` via `tokenizers`→
 `indicatif`, `paste`) — all transitive through optional deps (`mpi-io` feature, `tokenizers`),
 no upstream fix available, not actionable as a code change. Not filed as an INT item.
 Also checked and found clean (no INT items filed): `clawhdf5-migrate` (zero `unwrap()` outside
 `#[test]` code in `main.rs`; `sqlite_reader.rs`/`hdf5_writer.rs`/`validate.rs` are unwrap-free),
 `clawhdf5-cli`, `clawhdf5-napi` (zero `unwrap()` in `lib.rs`), `clawhdf5-accel` SIMD dispatch
 (`is_x86_feature_detected!`/runtime gating is correct — no illegal-instruction risk),
 `clawhdf5-filters` hot path (slice-based, no byte-by-byte loops of consequence), and TODO/FIXME
 grep across all crates (the only hits are test-fixture bytes literally named `b"XXXX"`, not
 real markers).
 ---
 ## Priority key
 - **P0** — correctness/security bug reachable from untrusted input (crafted file, external
  caller), should block release.
 - **P1** — real functional gap or measurable perf cost on a hot path.
 - **P2** — consistency/hardening/API-quality; safe to defer.
 ---
 ## Group A — `clawhdf5-agent`: provenance/security is unwired (headline finding)
 ### INT-01 — Wire `WriteAnomalyDetector` / `ProvenanceStore` into the actual write path [P0]
 **Files:** `crates/clawhdf5-agent/src/storage.rs` (save/save_batch path), `crates/clawhdf5-agent/src/provenance.rs`, `crates/clawhdf5-agent/src/anomaly.rs`, `crates/clawhdf5-agent/src/lib.rs`
 **Problem:** `ROADMAP.md` Track 5 ("Memory Security & Provenance") is marked 🟢 Complete, listing
 source attribution, write anomaly detection, source isolation, and integrity verification as
 done. The types exist and are unit-tested in isolation — but `grep -rn
 "WriteAnomalyDetector\|ProvenanceStore\|SourceIsolation"` across every file in
 `clawhdf5-agent` *except* `provenance.rs`/`anomaly.rs` themselves returns nothing.
 `storage.rs` (the real save/delete/replay path) never imports or calls into either module.
 Nothing in `HDF5Memory::save`/`save_batch` populates a `ProvenanceStore`, runs a rate/pattern
 check, or routes through `SourceIsolation`. In its current state this is a library the crate
 ships but never uses on itself — every memory write today has **no** rate limiting, no pattern
 detection, and no provenance recorded, contrary to what the roadmap and any consumer relying on
 it would assume.
 **Change:** In `storage.rs`'s save/save_batch entry point(s), construct/thread a
 `WriteAnomalyDetector` and `ProvenanceStore` (or accept them as constructor params on the
 memory-store struct so callers can configure `AnomalyConfig`), call `record_write` +
 `check_rate_anomaly`/`check_pattern_anomaly` before persisting each chunk, and call
 `ProvenanceStore::add` with the resulting `MemoryProvenance` alongside the write. Surface
 anomaly alerts through the existing error/result type rather than silently dropping them
 (decide via a config flag whether pattern/rate hits are hard-rejects or soft warnings — a hard
 reject changes public API behavior, a warning is additive). Add an integration test that writes
 a chunk containing one of the 15 suspicious patterns and asserts the alert actually fires
 through the public save path (not just the unit-level `WriteAnomalyDetector` test).
 ---
 ### INT-02 — `check_pattern_anomaly` is trivially bypassed substring matching [P1]
 **File:** `crates/clawhdf5-agent/src/anomaly.rs:192-211`
 **Problem:**
 ```rust
 let lower = chunk.to_lowercase();
 for pattern in &self.config.suspicious_patterns {
    if lower.contains(pattern.as_str()) { ... }
 }
 ```
 Matching is raw case-folded substring containment against 15 fixed literals (`"ignore
 previous"`, `"system:"`, …). Trivially defeated by inserting extra whitespace/punctuation
 (`"ignore  previous"`), splitting the phrase across two separate writes (checks are per-chunk,
 not per-session-buffer), or any non-ASCII obfuscation. As a poisoning-resistance control this
 currently only stops the laziest attacks.
 **Change:** Normalize input before matching (collapse whitespace/strip zero-width and combining
 characters), and consider word-boundary-tolerant/regex matching instead of raw `contains`.
 Document the remaining limitation (this is a heuristic filter, not a guarantee) rather than
 implying full poisoning resistance.
 ---
 ### INT-03 — `session_counts` grows unbounded and is fully rescanned on every rate check [P1]
 **File:** `crates/clawhdf5-agent/src/anomaly.rs:109, 126-133, 170-181`
 **Problem:** `session_counts: HashMap<String, u32>` is incremented on every `record_write` and
 never pruned — unlike `window` (which has a 60s sliding-window prune). A caller that creates
 many distinct `session_id` values (fully caller-controlled strings) grows this map without
 bound for the process lifetime. `check_rate_anomaly`'s session-level loop
 (`for (session, &count) in &self.session_counts`) then scans the *entire* historical map on
 every single check call, so per-write cost grows with total lifetime session count, not
 current activity.
 **Change:** Bound `session_counts` with an LRU/TTL eviction policy, or track only counts within
 the same rolling window used for `window` (see INT-05, which is closely related — the
 session-level check has its own separate bug on top of this).
 ---
 ### INT-04 — Sliding-window prune only inspects the front of the deque [P1]
 **File:** `crates/clawhdf5-agent/src/anomaly.rs:134-139`
 **Problem:**
 ```rust
 self.window.push_back(event);
 let cutoff = self.last_timestamp - 60.0;
 while self.window.front().is_some_and(|e| e.timestamp < cutoff) {
    self.window.pop_front();
 }
 ```
 `WriteEvent.timestamp` is caller-supplied (not sampled from a clock inside this type), so
 nothing prevents an out-of-order/backdated event from landing behind the front after a more
 recent one. Because eviction only ever looks at `front()`, a single out-of-order event
 permanently corrupts the window — old entries behind it are never pruned, so
 `check_rate_anomaly`'s window-length count over-reports forever (and can be intentionally
 inflated by a caller that varies timestamp ordering).
 **Change:** Prune by retaining only entries `>= cutoff` across the whole deque
 (`self.window.retain(|e| e.timestamp >= cutoff)`), or reject/clamp non-monotonic timestamps in
 `record_write` and document that `WriteEvent.timestamp` must be non-decreasing per detector
 instance.
 ---
 ### INT-05 — Session-level rate check uses a lifetime cumulative counter, not a rate [P1]
 **File:** `crates/clawhdf5-agent/src/anomaly.rs:170-181` (`check_rate_anomaly`)
 **Problem:** `max_writes_per_session` is compared against `session_counts[session]`, which is
 incremented forever and never reset (see INT-03). This measures "how old is this session," not
 "is this session currently abusive" — any long-lived legitimate session (e.g. a persistent
 agent) permanently trips the alert once past the threshold regardless of pace, while a burst of
 writes in a brand-new session under the threshold is missed even if it's the real anomaly.
 **Change:** Make this a rate — either measure session writes within the existing 60s rolling
 window (reuse `window`, filtered by `session_id`) or add a separate per-session rolling window,
 rather than an unbounded lifetime total.
 ---
 ### INT-06 — `bfs_neighbors` re-scans all relations on every queue pop [P1]
 **File:** `crates/clawhdf5-agent/src/knowledge.rs:339-378`, hot loop at 352-365
 **Problem:**
 ```rust
 let neighbours: Vec<u64> = self.relations.iter().filter_map(|r| { ... }).collect();
 ```
 runs once per node dequeued during BFS, giving `O(visited_nodes × total_relations)` total cost.
 `get_subgraph` (`knowledge.rs:387-417`) calls `bfs_neighbors` once per seed node, multiplying
 the cost again. On a graph with a non-trivial relation count this is the dominant cost of any
 graph traversal query — the kind of memory-graph read the whole crate exists to serve
 efficiently.
 **Change:** Build an adjacency `HashMap<u64, Vec<u64>>` once (either eagerly maintained on
 insert/delete, or lazily built and cached with invalidation on mutation) instead of
 linear-scanning `self.relations` per hop.
 ---
 ### INT-07 — Quadratic eviction via `Vec::contains` inside `retain` [P1]
 **File:** `crates/clawhdf5-agent/src/consolidation.rs:345-350`
 **Problem:**
 ```rust
 let evict_ids: Vec<u64> = episodic_indices[..evict_n].iter().map(|&i| self.records[i].id).collect();
 self.records.retain(|r| !evict_ids.contains(&r.id));
 ```
 `retain` invokes the closure once per record; `Vec::contains` is `O(m)`. Worst case this is
 `O(n·m)` per consolidation pass, run periodically over the full record set.
 **Change:** Collect `evict_ids` into a `HashSet<u64>` before the `retain` call — `O(n)` lookup
 per record instead of `O(m)`.
 ---
 ### INT-08 — `MediaRef.checksum` is unkeyed FNV-1a but named/documented as a checksum [P2]
 **File:** `crates/clawhdf5-agent/src/multimodal.rs:96-97, 104, 116, 127`; compare
 `crates/clawhdf5-agent/src/provenance.rs:16-19`
 **Problem:** `provenance.rs` already carries an explicit doc comment (and the CHANGELOG has a
 dedicated "doc-only" entry) clarifying that its FNV-1a content hash is unkeyed and detects only
 accidental corruption, not tampering. `multimodal.rs`'s `MediaRef.checksum` field uses the same
 FNV-1a hash for the same purpose but has no equivalent caveat, and the field name "checksum"
 (vs. "hash") reads as an integrity guarantee to a downstream consumer (e.g. something in
 ZeroClaw deciding whether to trust/reuse a cached media reference).
 **Change:** Either rename the field (e.g. `content_fingerprint`) or add the same
 non-tamper-evidence doc comment already used in `provenance.rs`, so the two unkeyed-hash usages
 in the crate are consistently documented.
 ---
 ## Group B — `clawhdf5-format` / `clawhdf5-io`: untrusted-file parsing gaps
 The 2026-08-05 hardening pass (see CHANGELOG "Security" section) already covers
 `chunked_read.rs`/`data_read.rs`/`local_heap.rs`/`btree_v1.rs` with `ensure_len`-style overflow
 guards, a B-tree recursion-depth guard, and a `fuzz_dataset_read` target. ROADMAP.md explicitly
 flags "a full manual audit of every indexing site is still open" as unfinished — the following
 are concrete gaps found in that follow-up, in files/paths the prior pass did not touch.
 ### INT-09 — `btree_v2.rs` recursive tree-walk has no depth cap (stack-overflow DoS) [P0]
 **File:** `crates/clawhdf5-format/src/btree_v2.rs:264-403` (`collect_internal_records`), entry
 at `176-213` (`collect_btree_v2_records`)
 **Problem:** `BTreeV2Header.depth: u16` (defined at line 21) is parsed straight from file bytes
 with no upper bound. `collect_internal_records` recurses with `child_depth = depth - 1` (line
 299) down to 0 with no depth-remaining cap — unlike the cyclic/self-referencing-index guards
 already added elsewhere in this hardening cycle (`fractal_heap.rs`, `object_header.rs`'s
 `depth_remaining` params, `filters.rs`'s `NBIT_MAX_DEPTH`). A crafted v2 B-tree header claiming
 `depth = 65535` (paired with a matching on-disk `"BTIN"` internal-node chain, or even a node
 that points back into itself since nothing here detects cycles either) drives ~65k stack frames
 of native recursion — an abort/crash from a small crafted file. This is reachable from real
 parse paths: `group_v2.rs:82`, `shared_message.rs:368`, `attribute.rs:384` (dense group/dense
 attribute listings — a realistic file feature, not an obscure one).
 **Change:** Thread a `depth_remaining: u16` (or similar) cap through
 `collect_btree_v2_records`/`collect_internal_records`, capped at some sane bound (e.g. 64,
 consistent with `NBIT_MAX_DEPTH`'s style elsewhere in this codebase), returning a `FormatError`
 instead of recursing past it.
 ---
 ### INT-10 — `fuzz_btree_v2` never exercises the recursive traversal where INT-09 lives [P1]
 **File:** `crates/clawhdf5-format/fuzz/fuzz_targets/fuzz_btree_v2.rs` (or wherever this target
 lives under `crates/clawhdf5-format/fuzz/`)
 **Problem:** The existing target only calls `BTreeV2Header::parse` — it never calls
 `collect_btree_v2_records`, so the actual tree-walk (the code path with the depth-recursion bug
 in INT-09) has zero fuzz coverage today, despite the file being in scope for a target already
 named after it.
 **Change:** Extend `fuzz_btree_v2` to also invoke `collect_btree_v2_records` on the parsed
 header against the fuzz input, so the recursive traversal gets the same adversarial coverage the
 header parse already has. Land this alongside INT-09 so the fix is locked in by the fuzzer, not
 just a manual patch.
 ---
 ### INT-11 — Unchecked multiplication of file-derived sizes in fractal-heap size math [P0]
 **File:** `crates/clawhdf5-format/src/fractal_heap.rs:479-496` (`block_size_for_row`,
 `indirect_block_heap_size`)
 **Problem:**
 ```rust
 sbs * (1u64 << (row - 1))              // line ~484
 total += self.block_size_for_row(row) * tw   // line ~493
 ```
 use plain `*` on `starting_block_size`/`table_width`, both read from the FRHP header with no
 upper-bound validation. A crafted large `starting_block_size` combined with enough rows/columns
 overflows `u64`; under `overflow-checks` (on for debug/fuzz builds, and optionally enabled in
 release) this panics — a DoS abort from a malformed fractal heap, the same bug class the
 2026-08-05 pass already fixed in sibling files.
 **Change:** Replace with `checked_mul`/`saturating_mul` and propagate a `FormatError` on
 overflow, matching the `ensure_len`/checked-arithmetic idiom already used in
 `chunked_read.rs`/`local_heap.rs`.
 ---
 ### INT-12 — Unbounded allocation from an unvalidated length before any data is read [P0]
 **Files:**
 - `crates/clawhdf5-io/src/subfiling.rs:206-210` (`SubfileManager::read_at`) —
  `Vec::with_capacity(length as usize)` where `length: u64` is caller/layout-supplied with no
  cap tied to actual dataset or file size.
 - `crates/clawhdf5-io/src/async_read.rs:84` (`AsyncFileReader::open`) —
  `Vec::with_capacity(len as usize)` sized directly from `file.metadata().len()`, no cap.
 **Problem:** Both allocate a buffer sized from an untrusted/unvalidated length *before*
 validating it against anything (declared dataset size, actual readable bytes, or a configured
 ceiling). A crafted layout-metadata value reaching `subfiling.rs`, or a crafted/sparse file
 opened via `async_read.rs`, can trigger a multi-gigabyte-to-exabyte allocation attempt and an
 OOM abort — the same "bounded allocation" concern the CHANGELOG's `MAX_DECOMPRESS_SIZE` fix
 already addressed for the decompression path, just not yet for these two read paths.
 **Change:** Cap the length against a known-sane bound (file size, or a configurable ceiling
 similar in spirit to `MAX_DECOMPRESS_SIZE`/`MAX_WAL_FIELD_LEN`) before calling
 `Vec::with_capacity`, or use `try_reserve` and return a clean error on failure instead of
 aborting.
 ---
 ### INT-13 — `symbol_table.rs` size arithmetic doesn't use the `checked_*`/`ensure_len` idiom used elsewhere [P2]
 **File:** `crates/clawhdf5-format/src/symbol_table.rs:99-107` (`SymbolTableNode::parse`)
 **Problem:** `let needed = entries_start + num_symbols * entry_size;` uses plain arithmetic.
 Not exploitable to overflow on 64-bit today (`num_symbols` is bounded by its `u16` source
 field), but it's inconsistent with the rest of the audited codebase and becomes a real risk if
 either operand's type widens later.
 **Change:** Route through `checked_mul`/`checked_add` + `ensure_len`, matching the pattern used
 throughout `chunked_read.rs`/`data_read.rs`/`local_heap.rs`/`btree_v1.rs`.
 ---
 ### INT-14 — Filter bit-packing decode loops have no direct fuzz coverage [P2]
 **File:** `crates/clawhdf5-format/src/filters.rs` (scale-offset unpack ~150-270, N-Bit type-tree
 walk ~396-510); fuzz target `fuzz_filter_pipeline`
 **Problem:** `fuzz_filter_pipeline` only fuzzes `FilterPipeline::parse` — the filter-pipeline
 *metadata* message — not the actual decode functions in `filters.rs` that unpack
 attacker-influenced compressed bytes bit-by-bit (scale-offset, N-Bit). This is the most
 bit-twiddling-heavy code in the crate and, per the CHANGELOG, has already had real bugs found
 there in the initial hardening pass (`1 << minbits` overflow, `bit_offset + precision`
 overflow); it's exactly the kind of code that benefits most from fuzzing but currently gets none
 directly.
 **Change:** Add a `fuzz_filter_decode` target that feeds arbitrary bytes through the
 scale-offset and N-Bit decode entry points directly (not just pipeline metadata parsing).
 ---
 ## Group C — `clawhdf5-ann` (HNSW): hot-path performance
 `prune_connections` rayon parallelism (already shipped) is out of scope. The outer
 insert/build loop is intentionally left sequential per ROADMAP's own design note — not
 re-proposed here.
 ### INT-15 — `compute_distance` is scalar-only; `clawhdf5-accel`'s SIMD path is never used [P1]
 **Files:** `crates/clawhdf5-ann/src/hnsw.rs:47-74` (`compute_distance`);
 `crates/clawhdf5-accel/src/lib.rs:125` (`cosine_similarity`), `:173` (`l2_distance`)
 **Problem:** `clawhdf5-ann`'s `Cargo.toml` has no dependency on `clawhdf5-accel` at all.
 `compute_distance` is a hand-written scalar loop for both L2 and cosine, called from every
 candidate-expansion step in `greedy_closest`, `search_layer`, and `prune_connections` — i.e.
 the entire build/insert/search hot path. `clawhdf5-accel` already provides
 runtime-feature-detected, SIMD-accelerated equivalents (AVX2/AVX-512/NEON, correctly gated per
 INT survey — see clean bill of health above) that go completely unused here.
 **Change:** Add a `clawhdf5-accel` dependency to `clawhdf5-ann` and route `compute_distance`
 through `l2_distance`/`cosine_similarity`. This is a drop-in replacement for the scalar
 arithmetic, not a semantic change.
 ---
 ### INT-16 — Best-entry-point distance is discarded and immediately recomputed [P1]
 **File:** `crates/clawhdf5-ann/src/hnsw.rs` — `greedy_closest` (749-772) computes
 `best_dist` at line 756 but returns only the `usize` node id; callers
 (`build_with_metric` 251-253, `insert` 381-389, `search` 504-506) immediately recompute
 `compute_distance(query, &vectors[ep], metric)` for that same `(query, ep)` pair before calling
 `search_layer` (which itself recomputes it again at line 783).
 **Problem:** Every layer transition during insert/search throws away a distance value it just
 computed and recomputes the identical value at least once more. For an L-layer index this wastes
 up to L redundant distance computations per insert/search call — pure waste on what is already
 the hottest path in the crate (compounded by INT-15 if that's not yet fixed).
 **Change:** Change `greedy_closest`'s return type to `(usize, f32)` (node id + its distance) and
 thread that value into the next `greedy_closest`/`search_layer` call instead of recomputing.
 ---
 ### INT-17 — `search_layer`'s visited-set uses `HashSet<usize>` instead of a dense bitset [P1]
 **File:** `crates/clawhdf5-ann/src/hnsw.rs:799, 809-812`
 **Problem:** `let mut visited = HashSet::new();` with `.contains(&neighbor)`/`.insert(neighbor)`
 in the innermost per-candidate-expansion loop, run on every insert and search call. Node ids are
 dense `0..n` integers — a `Vec<bool>` (or bitset) indexed directly by id gives O(1) lookup
 without SipHash overhead, which matters when this loop dominates search cost.
 **Change:** Replace with `vec![false; vectors.len()]` indexed by node id (reset/reused per
 call), or a proper bitset if allocation-per-call cost matters.
 ---
 ### INT-18 — `compact()` clones every surviving vector twice [P1]
 **File:** `crates/clawhdf5-ann/src/hnsw.rs:463-478` (`compact`), `:305`
 (`build_with_metric`'s `vectors: vectors.to_vec()`)
 **Problem:** `compact()` builds an owned `Vec<Vec<f32>>` via `surviving.push(v.clone())` (line
 469), then passes `&surviving` into `build_with_metric`, whose first action clones it again via
 `.to_vec()`. For a large index this doubles the memory-copy cost of an already-`O(n)` rebuild
 operation.
 **Change:** Give `build_with_metric` (or a private variant) an owned-`Vec<Vec<f32>>` entry point
 so `compact` can move `surviving` in directly instead of cloning twice.
 ---
 ## Group D — `clawhdf5-py`: Mutex poisoning bricks write-mode objects
 ### INT-19 — Pervasive `state.lock().unwrap()` on a shared `Mutex` reachable from Python calls [P1]
 **Files:** `crates/clawhdf5-py/src/group.rs` (6 sites, e.g. `:115, :161, :184, :200, :217`),
 `crates/clawhdf5-py/src/attrs.rs` (4 sites, e.g. `:56, :77, :92, :99`),
 `crates/clawhdf5-py/src/file.rs` (1 site, `:282`)
 **Problem:** `PyGroup`/`PyAttrs`/write-mode file state hold a `Mutex<...>` and every method that
 touches it does `state.lock().unwrap()`. If any single call panics while holding the lock (a
 future edge case in `extract_numpy_data`, an allocation failure, anything) the `Mutex` becomes
 permanently poisoned. Every subsequent method call on that same Python object — for the rest of
 its lifetime — then also panics via the same `.unwrap()`, instead of the object cleanly
 returning a `PyErr` and remaining usable. This turns one transient panic into a permanently
 broken object from the caller's perspective, which is a worse failure mode than a single
 raised-and-handled Python exception.
 **Change:** Replace `lock().unwrap()` with a helper that converts a poison error into a
 `PyResult` `PyErr` (e.g. `state.lock().map_err(|_| PyErr::new::<PyRuntimeError, _>("internal state poisoned"))?`,
 or use `parking_lot::Mutex` which doesn't have poisoning at all — likely the simpler fix given
 `clawhdf5-py` doesn't appear to rely on poisoning semantics anywhere). Apply consistently across
 all ~11 call sites.
 ---
 ## Group E — noted, not proposed (checked and found low-priority/out-of-scope)
 - **`clawhdf5-derive`'s generated `from_bytes`** (`crates/clawhdf5-derive/src/lib.rs:106-119`)
  does `assert!(_data.len() >= _required, ...)` before any field-slicing, so it's a documented,
  guarded panic (`# Panics` doc comment already present) rather than an unguarded OOB — and
  `#[derive(H5Type)]` is currently used only in `crates/clawhdf5-format/tests/derive_tests.rs`,
  not in any production code path. Making `from_bytes` return `Result` instead of asserting
  would be a reasonable future API-ergonomics improvement for downstream users of the macro, but
  it's not fixing a reachable bug today — left out as not worth an INT slot this pass.
 - **`cargo audit` unmaintained warnings** (`custom_derive`, `number_prefix`, `paste`) — all
  transitive through optional features (`mpi-io`, and whatever pulls in `tokenizers`), zero
  actual vulnerabilities, no code-level fix available in this repo. FYI only.
 ---
 ## Suggested implementation order for the coding phase
 1. **INT-01** first — it's the load-bearing gap (provenance/anomaly detection is currently
   inert), and INT-02/03/04/05 are bug fixes *inside* the code INT-01 wires up, so fixing them
   before or during the wiring avoids shipping newly-live bugs.
 2. **INT-09 + INT-10 together** (P0, security) and **INT-11, INT-12** (P0, security) — these are
   independent of each other and of Group A, safe to parallelize.
 3. **INT-15/16/17/18** (Group C, HNSW perf) — independent of A/B, safe to parallelize.
 4. **INT-19** (Group D) — independent, small, safe to parallelize.
 5. **INT-06, INT-07, INT-08, INT-13, INT-14** — lower urgency, pick up as time allows.
 All items should land with `cargo test --workspace` (and `cargo clippy --workspace -- -D
 warnings`, per this repo's established gate) passing before being considered done.
@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# CI check: verify rustyhdf5-format compiles under no_std (thumbv7em-none-eabihf).
+# CI check: verify clawhdf5-format compiles under no_std (thumbv7em-none-eabihf).
 #
 # Usage:
 #   ./scripts/check-nostd.sh
@@ -11,7 +11,7 @@ set -euo pipefail
 TARGET="thumbv7em-none-eabihf"
-echo "==> Checking no_std build for rustyhdf5-format (target: $TARGET)"
+echo "==> Checking no_std build for clawhdf5-format (target: $TARGET)"
 # Ensure the target is installed
 if ! rustup target list --installed | grep -q "$TARGET"; then
@@ -20,13 +20,13 @@ if ! rustup target list --installed | grep -q "$TARGET"; then
 fi
 # Build with no default features (no std, no flate2, no sha2)
-cargo build --target "$TARGET" -p rustyhdf5-format --no-default-features
+cargo build --target "$TARGET" -p clawhdf5-format --no-default-features
 echo "==> no_std build succeeded"
 # Also verify the default-features (std) build still works
 echo "==> Checking default-features build"
-cargo build -p rustyhdf5-format
+cargo build -p clawhdf5-format
 echo "==> default-features build succeeded"
 echo "==> All no_std checks passed"
@@ -34,16 +34,16 @@ run_step() {
 # 1. Format check
 run_step "cargo fmt --check" cargo fmt --check
-# 2. Clippy (exclude rustyhdf5-py which needs PyO3/Python)
+# 2. Clippy (exclude clawhdf5-py which needs PyO3/Python)
 run_step "cargo clippy" cargo clippy \
    --workspace \
-    --exclude rustyhdf5-py \
+    --exclude clawhdf5-py \
    -- -D warnings
-# 3. Tests (exclude rustyhdf5-py)
+# 3. Tests (exclude clawhdf5-py)
 run_step "cargo test" cargo test \
    --workspace \
-    --exclude rustyhdf5-py
+    --exclude clawhdf5-py
 # 4. no_std check
 run_step "check-nostd.sh" "$SCRIPT_DIR/check-nostd.sh"