Merge release/v2.7.0

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

.gitea/workflows/ci.yml

@@ -33,8 +33,14 @@ jobs:
           /opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray
           echo "/opt/interop/bin" >> "$GITHUB_PATH"
       - name: Show interop library versions
-        run: python3 -c "import h5py, netCDF4; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'netCDF4', netCDF4.__version__)"
+        run: /opt/interop/bin/python -c "import h5py, netCDF4; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'netCDF4', netCDF4.__version__)"
       - name: Run CI script
         env:
+          # Name the interpreter outright rather than relying on $GITHUB_PATH
+          # reaching the test processes: if `python3` resolved to the system
+          # one instead of the venv, every interop suite would skip.
+          # CLAWHDF5_REQUIRE_INTEROP turns that skip into a failure, so the
+          # two together mean the suites either run or the build goes red.
+          CLAWHDF5_PYTHON: /opt/interop/bin/python
           CLAWHDF5_REQUIRE_INTEROP: "1"
         run: bash scripts/ci-test.sh

.gitignore

@@ -4,3 +4,4 @@ benchmarks/longmemeval/*.json
 
 # Local model weights (MiniLM etc.) — large, not committed
 weights/
+.venv

BENCHMARKS.md

@@ -28,6 +28,210 @@
 
 ---
 
+## Memory footprint
+
+`cargo run --release -p clawhdf5-bench --bin search_harness -- --footprint --full`,
+384-dim `f32`. The figure that matters is **reopened**: a store loaded from
+disk, which is what a long-lived process holds.
+
+Measured with a counting global allocator, not RSS. RSS cannot see this from
+inside one process — freeing a large structure returns its pages to the
+allocator's pool rather than to the OS, so allocating the next one shows no
+change at all. Measured that way a store holding the corpus twice and one
+holding it once came out *identical* (1.00x both), which is how the first
+attempt at this measurement went.
+
+| N | vectors (raw) | reopened, before | reopened, after |
+|---:|---:|---:|---:|
+| 1 000 | 1 MiB | 5 MiB (3.41x) | 4 MiB (2.39x) |
+| 10 000 | 15 MiB | 50 MiB (3.43x) | 35 MiB (2.42x) |
+| 100 000 | 146 MiB | 505 MiB (3.44x) | **357 MiB (2.43x)** |
+
+The cache stored every embedding twice — once as a `Vec<Vec<f32>>` and once
+flattened for the batched kernels, kept in lock-step on every push, update and
+compaction. Storing only the flat buffer and indexing into it gives back
+almost exactly one copy of the corpus (148 MiB at 100k) and one heap
+allocation per entry. Recall and query latency are unchanged.
+
+What remains at 2.43x: the flat vectors (1.0x), the HNSW index's own copy of
+them (1.0x), and text, ids and graph (~0.4x). The index copy is the next
+target — it is what a quantised or borrowed representation would address.
+
+### Quantising the index copy (`quantized_index`)
+
+`MemoryConfig::quantized_index` stores the index's copy as `i8` instead of
+`f32`. Same harness, same binary, `--footprint --full` with and without
+`--int8`:
+
+| N | vectors (raw) | indexes, f32 | indexes, int8 | reopened, f32 | reopened, int8 |
+|---:|---:|---:|---:|---:|---:|
+| 1 000 | 1 MiB | 2 MiB | 1 MiB | 4 MiB (2.40x) | 2 MiB (1.64x) |
+| 10 000 | 15 MiB | 32 MiB | 14 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
+| 100 000 | 146 MiB | 266 MiB | **123 MiB** | 399 MiB (2.72x) | **256 MiB (1.74x)** |
+
+The scale is **per row**, not global. A unit-length row in `d` dimensions has
+components around `1/sqrt(d)`, so a fixed `[-1, 1]` scale spends fewer than 12
+of the 255 levels on a 128-dimensional vector: measured against an exact
+ranking that gives 0.35 top-10 overlap — unusable. Scaling each row by its own
+largest component brings the same measurement to 0.99.
+
+Quantised distances still cost recall on their own, and **`ef` does not buy it
+back**, because the loss is in the distances rather than in the graph
+(`--ann-only --full`, N = 100 000):
+
+| ef | recall@10, f32 | recall@10, int8 | recall@10, int8 + re-score |
+|---:|---:|---:|---:|
+| 32 | 0.9775 | 0.9415 | 0.9785 |
+| 64 | 0.9945 | 0.9625 | 0.9940 |
+| 128 | 0.9995 | 0.9670 | 0.9990 |
+| 256 | 0.9995 | 0.9670 (ceiling) | 0.9990 |
+
+Re-scoring closes the gap: the store already holds the exact embeddings, so
+the query path re-scores the candidate pool against them before fusion. That
+is done automatically whenever the index is quantised.
+
+**On AVX2 this costs nothing — it pays.** The first measurement of this put
+the cost at ~13% of QPS and ~16% of build time, but that compared a scalar
+int8 loop against `clawhdf5-accel`'s hand-written AVX2 kernels for `f32`:
+the gap was a missing kernel, not a property of int8. With
+`clawhdf5_accel::dot_i8` (AVX2: sign-extend to `i16`, then `madd_epi16`),
+medians of three alternating runs at N = 100 000, same binary:
+
+| | f32 | int8 | int8 + re-score |
+|---|---:|---:|---:|
+| build | 3197 ms | **1778 ms** | 1826 ms |
+| QPS at ef = 64 | 13 399 | 29 195 | **21 848** |
+| recall@10 at ef = 64 | 0.9945 | 0.9625 | **0.9940** |
+
+So at equal recall the quantised index answers **1.63x as many queries per
+second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare
+only at equal `ef`: with re-scoring the harness raises `ef` to at least the
+candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.)
+
+It is still **off by default**, for portability rather than performance: the
+int8 kernel is AVX2-only, and on aarch64 — including `clawhdf5-android` — it
+falls back to the scalar loop, where the original trade still applies. A NEON
+kernel would remove that caveat. On an x86-64 deployment, turning it on is a
+win on every axis measured.
+
+A measurement trap worth recording: the synthetic `clustered` generator in the
+`clawhdf5-ann` tests draws clusters far tighter than any real embedding, so
+neighbours there sit closer together than the quantisation error and top-10
+*identity* is noise. Scored on that fixture int8 looks catastrophic (0.57
+overlap) — a fact about the fixture, not the storage. The tests use random
+vectors, and recall is measured against brute-force ground truth rather than
+against the f32 index, whose own approximation errors a re-scored search is
+entitled to get right.
+
+### Opening a store (`read_from_disk`)
+
+`HDF5Memory::open` memory-mapped the file, copied the whole mapping into a
+`Vec`, and handed that to `File::from_bytes` — while `File::open` memory-maps
+the file itself. Dropping the copy takes **store open from 455 ms to 327 ms**
+at 100 000 x 384 (`--e2e-only --full`; two runs after the change, 326.8 and
+328.1 ms).
+
+It does **not** lower the process's peak memory, which is worth stating
+precisely because it is the obvious thing to assume. The harness now reports a
+high-water mark alongside the retained figure:
+
+| N | reopened MiB | peak during open MiB |
+|---:|---:|---:|
+| 1 000 | 4 | 5 |
+| 10 000 | 44 | 61 |
+| 100 000 | 399 | 562 |
+
+The peak is set *after* the parse, by the index build, so a buffer allocated
+and freed during the parse never reaches the high-water mark. Holding a
+deliberate extra copy of the file across the whole parse leaves the peak
+unmoved, which is how this was confirmed rather than assumed. What the change
+saves is the copy itself: a full-file memcpy on every open, and the transient
+that goes with it.
+
+## Read harness
+
+Produced by `cargo run --release -p clawhdf5-bench --bin read_harness`: a 4096 x
+2048 `f64` dataset (64 MB) written three ways, read in full and through four
+hyperslab selections, each from a fresh file handle. The last column is the
+point: does a selection cost what the *selection* costs?
+
+### Baseline (v2.4.0): every selection decodes the whole dataset
+
+4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
+
+| layout | read | selected | time ms | MB/s of selection | vs full read |
+|---|---|---:|---:|---:|---:|
+| chunked + deflate | full (first) | 64 MB | 181.8 | 352 | |
+| chunked + deflate | full (repeat) | 64 MB | 162.1 | 395 | 1.00x |
+| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 104.89 | 0 | 0.577x |
+| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 110.26 | 18 | 0.606x |
+| chunked + deflate | one row | 0.02 MB | 105.81 | 0 | 0.582x |
+| chunked + deflate | one column | 0.03 MB | 108.37 | 0 | 0.596x |
+| chunked | full (first) | 64 MB | 97.4 | 657 | |
+| chunked | full (repeat) | 64 MB | 86.7 | 738 | 1.00x |
+| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 40.97 | 1 | 0.420x |
+| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 44.66 | 45 | 0.458x |
+| chunked | one row | 0.02 MB | 30.88 | 1 | 0.317x |
+| chunked | one column | 0.03 MB | 30.27 | 1 | 0.311x |
+| contiguous | full (first) | 64 MB | 57.6 | 1112 | |
+| contiguous | full (repeat) | 64 MB | 53.5 | 1195 | 1.00x |
+| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 30.97 | 1 | 0.538x |
+| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 31.64 | 63 | 0.550x |
+| contiguous | one row | 0.02 MB | 31.90 | 0 | 0.554x |
+| contiguous | one column | 0.03 MB | 29.36 | 1 | 0.510x |
+
+### After: partial reads
+
+Only the rows of a contiguous dataset, or the chunks, that overlap the
+selection's bounding box are read/decoded. A 64 x 64 window of the compressed
+dataset: **105 -> 0.39 ms**; one row: **106 -> 2.7 ms**; one column:
+**108 -> 5.2 ms**. (Absolute full-read times differ between the two runs
+because the machine's speed drifted; compare the *vs full read* column.)
+
+4096 x 2048 f64 (64 MB per dataset), chunks 256 x 256, file 129 MB
+
+| layout | read | selected | time ms | MB/s of selection | vs full read |
+|---|---|---:|---:|---:|---:|
+| chunked + deflate | full (first) | 64 MB | 112.5 | 569 | |
+| chunked + deflate | full (repeat) | 64 MB | 104.5 | 612 | 1.00x |
+| chunked + deflate | 64 x 64 window (1 chunk) | 0.03 MB | 0.39 | 81 | 0.003x |
+| chunked + deflate | 512 x 512 window (4-9 chunks) | 2.00 MB | 4.85 | 412 | 0.043x |
+| chunked + deflate | one row | 0.02 MB | 2.69 | 6 | 0.024x |
+| chunked + deflate | one column | 0.03 MB | 5.23 | 6 | 0.046x |
+| chunked | full (first) | 64 MB | 70.0 | 915 | |
+| chunked | full (repeat) | 64 MB | 61.7 | 1037 | 1.00x |
+| chunked | 64 x 64 window (1 chunk) | 0.03 MB | 0.06 | 541 | 0.001x |
+| chunked | 512 x 512 window (4-9 chunks) | 2.00 MB | 1.99 | 1005 | 0.028x |
+| chunked | one row | 0.02 MB | 0.05 | 285 | 0.001x |
+| chunked | one column | 0.03 MB | 0.45 | 69 | 0.006x |
+| contiguous | full (first) | 64 MB | 60.3 | 1062 | |
+| contiguous | full (repeat) | 64 MB | 56.4 | 1134 | 1.00x |
+| contiguous | 64 x 64 window (1 chunk) | 0.03 MB | 0.08 | 396 | 0.001x |
+| contiguous | 512 x 512 window (4-9 chunks) | 2.00 MB | 2.12 | 944 | 0.035x |
+| contiguous | one row | 0.02 MB | 0.03 | 576 | 0.000x |
+| contiguous | one column | 0.03 MB | 2.55 | 12 | 0.042x |
+
+### After: parallel cached decode, fewer copies (full reads)
+
+Full-read times, old and new binaries run alternately at the same moment (this
+machine's absolute speed drifts over a long session, so only same-moment
+comparisons mean anything):
+
+| layout (64 MB `f64`) | before | after |
+|---|---:|---:|
+| chunked + deflate | 110 ms | 69 ms |
+| chunked | 72 ms | 60 ms |
+| contiguous | 56 ms | 30 ms |
+
+What changed: the facade's cached read path decompressed chunks one at a time
+(only the uncached reader was parallel) and pushed every chunk through a 16 MiB
+cache that a 64 MB read simply churns; it now decodes cache misses in parallel
+batches and caches only datasets that fit. Unfiltered chunks are copied
+straight from the file bytes instead of via two intermediate buffers. A
+contiguous dataset is converted straight from the file bytes (one copy instead
+of two), and the native-endian conversions no longer zero a buffer they are
+about to overwrite.
+
 ## Search harness baseline (v2.3.0)
 
 Produced by `cargo run --release -p clawhdf5-bench --bin search_harness -- --full`
@@ -243,6 +447,28 @@ results.
 | 10000 | 104 | 1487 | 33.8 | 13.7 | 13.9 | 0.49 | 0.51 | 2020.9 |
 | 100000 | 1376 | 20285 | 728.9 | 353.1 | 142.2 | 4.65 | 4.78 | 214.7 |
 
+### After: batched bulk build (optionally parallel); deletions handled in search
+
+Profiling showed **90% of a build's distance evaluations are in back-link
+pruning**. The bulk build now inserts in batches: plan each node's neighbours
+against the graph as it stood at the start of the batch, link, then prune every
+overflowing list once. That is less work even single-threaded (a node gaining
+several back-links in a batch is pruned once), and with the `parallel` feature
+planning and pruning run on a thread pool. The graph is deterministic and the
+same with or without the feature. Parallelising *within* one insert was tried
+first and gave only 1.45x on 16 cores (tasks too small).
+
+| build | 1K | 10K | 100K |
+|---|---:|---:|---:|
+| v2.4.0 | 116 ms | 1676 ms | ~21 s |
+| batched | 83 ms | 1074 ms | 19.2 s |
+| batched + `parallel` (16 cores) | 34 ms | 388 ms | 5.9 s |
+
+Recall on clustered data is unchanged or slightly better (100K, `ef = 64`:
+0.984 -> 0.9945). On uniform random data it dips slightly (10K, `ef = 64`:
+0.474 -> 0.444), the cost of batch members not seeing each other while
+planning; batches are capped at 1/16 of the graph and 512 nodes.
+
 ## Vector Search Latency
 
 Brute-force cosine similarity over 384-dimensional embeddings (OpenAI text-embedding-3-small size).
@@ -491,6 +717,102 @@ Session-level:
 | Vector only | 85.4% | 94.2% | 96.6% | 0.8901 |
 | Hybrid | **88.2%** | **95.8%** | **97.8%** | **0.9158** |
 
+### Fusion method — weighted vs. RRF, full haystack, n=500
+
+Reciprocal rank fusion has been in the codebase since early on but was only
+reachable as a free function over a linear scan, so it had never been compared
+with the weighted sum on equal terms. `HDF5Memory::hybrid_search_with` now
+takes a `Fusion`, and both run over the same HNSW + BM25 candidates:
+
+| Mode | turn Hit@1 | Hit@5 | Hit@10 | MRR | session Hit@1 | session MRR |
+|---|---|---|---|---|---|---|
+| BM25 only | **53.8%** | 75.0% | 81.6% | 0.6320 | 86.2% | 0.8948 |
+| Vector only | 36.0% | 71.8% | 81.6% | 0.5031 | 85.4% | 0.8901 |
+| **Weighted 0.4 / 0.6** | 51.6% | **81.4%** | **87.8%** | **0.6430** | **91.0%** | **0.9347** |
+| RRF (k=60) | 45.0% | 78.8% | 87.6% | 0.5967 | 89.6% | 0.9253 |
+
+**RRF loses to the tuned weighted sum** — 6.6pp of turn Hit@1 and 0.046 of MRR
+— and lands almost exactly where the old `0.7/0.3` weighting did (44.2% / 
+0.5856). That is not a coincidence: RRF combines the two stages by rank with
+*equal* influence, and on this corpus the stages are not equally good. BM25
+alone beats the vector stage by 17.8pp at Hit@1, so any scheme that treats them
+as peers gives up rank-1 accuracy, and RRF discards the score magnitudes that
+would say which stage to believe.
+
+This is a property of the corpus, not a defect in RRF: its selling point is
+robustness when the two stages' scores are not comparable and there is no
+labelled data to tune against. Here there is, so the weighted sum is kept as
+the default. `Fusion::Rrf` remains available for callers whose stages are more
+evenly matched.
+
+### Keyword tokenizer — stemming, full haystack, n=500
+
+The keyword stage lowercases and splits on non-alphanumerics, with no stemming,
+so "training" and "trains" are unrelated terms. `TokenFilter::Stemmed` strips
+common English inflections (plurals, `-ing`/`-ed`, with consonant un-doubling)
+from documents and queries alike. Turn-level:
+
+| Mode | Hit@1 | Hit@5 | Hit@10 | MRR | session Hit@1 |
+|---|---|---|---|---|---|
+| BM25 only | **53.8%** | 75.0% | 81.6% | 0.6320 | 86.2% |
+| BM25 only, stemmed | 52.0% | 77.8% | 84.0% | 0.6320 | 88.0% |
+| Hybrid 0.4/0.6 | 51.6% | **81.4%** | 87.8% | **0.6430** | 91.0% |
+| Hybrid 0.4/0.6, stemmed | 50.2% | **81.4%** | **88.2%** | 0.6394 | **91.4%** |
+
+**Stemming is a trade, not a win, and the default stays off.** It reliably buys
+depth and costs the top rank: on BM25 alone, +2.8pp Hit@5 and +2.4pp Hit@10 for
+−1.8pp Hit@1, with MRR unchanged to four decimal places — the gains deeper down
+exactly offset the loss at rank 1. That is what conflation does: merging
+"train"/"training"/"trains" surfaces documents an exact-match query would never
+reach, and also lets a near-miss outrank the exact hit.
+
+On the configuration that actually ships (hybrid 0.4/0.6) the trade is
+narrower still — Hit@5 identical, Hit@10 +0.4pp, Hit@1 −1.4pp, MRR −0.004 —
+because the vector stage already supplies much of the recall stemming would
+add. There is no case here for changing the default; `TokenFilter::Stemmed`
+is available via `HDF5Memory::set_token_filter` for callers who want Hit@5/@10
+over rank-1 precision.
+
+### Re-ranking and recency — full haystack, n=500
+
+`reranker::rerank` combines temporal decay, source authority and Hebbian
+activation. Until now its combined score contained **no relevance term at
+all** — `RerankInput` did not carry the retrieval score — so a caller that
+re-ranked its candidates threw the retriever's ordering away and returned them
+ordered by age. The OpenClaw backend did exactly that on every search.
+
+Measuring that is unambiguous. "Recency" below is the share of
+`knowledge-update` questions where the newest gold session outranked the stale
+one (see `newest_gold_first`); ~45% is chance.
+
+| Mode | Hit@1 | Hit@5 | Hit@10 | MRR | recency |
+|---|---|---|---|---|---|
+| Hybrid 0.4/0.6, no re-rank | 51.6% | **81.4%** | 87.8% | 0.6430 | 45.0% |
+| + re-rank, **metadata only** (pre-fix) | 11.0% | 24.8% | 43.8% | 0.1829 | **87.5%** |
+| + re-rank, relevance-led, half-life 1 day | **52.0%** | 79.8% | 87.8% | 0.6403 | 51.7% |
+| + re-rank, relevance-led, half-life 7 days | 51.8% | 80.8% | 87.6% | **0.6437** | **52.2%** |
+| + re-rank, relevance-led, half-life 30 days | 51.8% | 81.0% | 87.8% | 0.6427 | 51.4% |
+| + re-rank, relevance-led, half-life 90 days | **52.0%** | 80.4% | 87.8% | 0.6425 | 50.8% |
+
+**The pre-fix row is the finding.** Ordering candidates by recency alone costs
+40.6pp of Hit@1 and two thirds of MRR: the results are the newest memories in
+the pool rather than the ones that answer the question. It does ace the recency
+metric, which is exactly what makes that metric worth having — a number that
+only goes up when a change is good would not have caught this.
+
+With relevance leading, retrieval is preserved (Hit@1 +0.4pp, MRR −0.003
+against no re-ranking) and recency discrimination gains 6–7pp. That is a real
+improvement but not a solved problem: recency only breaks near-ties, so it
+cannot reach the 87.5% the degenerate ordering gets. Those two rows are the
+ends of a trade-off, and the default sits deliberately near the relevance end.
+
+**Half-life is not a sensitive knob.** Across 1, 7, 30 and 90 days recency
+moves 1.4pp and MRR 0.003 — inside the noise of a 500-question run — because
+the temporal term is capped by its weight (0.3) while relevance differences
+between candidates are larger. The 24-hour default is kept; there is no
+measured reason to change it, and a corpus-matched value is not the lever it
+looks like.
+
 ### Weight sweep — full haystack, n=500
 
 `0.7/0.3` was a documented default, never a searched one. Sweeping
@@ -533,7 +855,8 @@ BM25 at Hit@1. Both dominate `0.7/0.3`.
 
 The rows below are kept at the three original settings because they are what the
 mode ablation measured — read them as "the shape of each stage in isolation",
-and take the operating point from the sweep.
+and take the operating point from the sweep. `0.4/0.6` is now the shipped
+default (`hybrid::DEFAULT_FUSION`).
 
 The same pattern shows up independently in omni-cortex's four-signal RRF ablation,
 where adding BM25 to a dense retriever raised nDCG@5 while lowering Hit@1 and MRR.

CHANGELOG.md

@@ -1,5 +1,318 @@
 # Changelog
 
+## v2.7.0 (2026-09-20)
+
+### Upgrade Notes
+- **Two read-path bugs fixed, one of them silent.** Datasets indexed by an
+  Extensible Array (any dataset with one unlimited dimension) returned data
+  from the wrong chunks past their first few dozen. If you have readings taken
+  from such a dataset with an earlier release, they may be wrong; re-read them.
+- **A corrupt chunk index is now an error.** Fixed and Extensible Array
+  structures carry checksums that were previously ignored, so damage surfaced
+  as plausible data from the wrong offset. Code that read a damaged file and
+  got numbers will now get `ChecksumMismatch` instead. That is the point.
+- **Breaking:** `MemoryConfig` gained `hnsw_m`, `hnsw_ef_construction` and
+  `hnsw_ef_search`, so literal constructions need updating;
+  `..Default::default()` does not. All three default to the previous
+  behaviour.
+
+### Correctness
+- `clawhdf5-format`: **datasets indexed by an Extensible Array returned wrong
+  data beyond their first few dozen chunks.** One unlimited dimension gives a
+  dataset an Extensible Array chunk index, whose first elements (4 by default)
+  sit inline in the index block and whose rest live in data blocks sized by a
+  formula the reader got wrong. In the default layout everything through the
+  36th chunk happened to line up and the 37th onwards did not: a 400-chunk
+  dataset silently returned wrong values from chunk 37, and datasets past
+  about a thousand chunks failed outright with "invalid Extensible Array data
+  block signature". **Reads were wrong, not
+  merely refused** — the caller got plausible numbers from the wrong chunks.
+  Four separate layout errors, each checked against files written by HDF5 2.0
+  and against the library source:
+  - the number of data blocks in super block `u` is `2^(u/2)`, not `2^u`;
+  - each holds `2^((u+1)/2) * data_blk_min_elmts` elements, which doubles
+    every *other* level rather than every level;
+  - a super block carries a block-offset field before its data block
+    addresses, which was not skipped;
+  - the page-init bitmap belongs to the super block, one bit per page packed
+    across all its data blocks (MSB first), and was being read from inside the
+    data block instead; a paged data block also ends its prefix with a
+    checksum before the first page.
+  Covered now by interop tests at 4, 37, 400, 5 000 and 200 000 chunks (the
+  last large enough for paged data blocks), plus sparse, gzip-filtered and
+  2-D cases. Writing is unaffected; this is a read-path bug.
+- `clawhdf5-format`: the sibling Fixed Array index (fixed dimensions written
+  with `libver='latest'`) was checked against the same range and is correct,
+  including paged data blocks and sparse datasets — it really does keep its
+  page-init bitmap in the data block, where the Extensible Array does not.
+  It had no real-file coverage above the inline sizes either, so it now has
+  the same tests.
+
+### Security
+- `clawhdf5-format`: **a crafted file could crash any reader through B-tree v2
+  traversal.** Recursion was bounded only by the depth the file claimed (a
+  `u16`), and child addresses were never checked for sharing. A node listing
+  itself as its own child under a header claiming 65 535 levels — under 100
+  bytes — overflowed the stack and **aborted the process** (SIGABRT, not a
+  catchable error). Levels whose children all point at one shared node below
+  reached it fan-out^depth times: 29.5 million records from ~5 KB, and one
+  more level would exhaust memory. Both are now errors, returned in under a
+  millisecond: depth is capped at 64 (as the fractal heap already was), and
+  traversal stops once it has produced more records than the file has bytes
+  to hold. Every B-tree v2 user goes through this path — dense attributes,
+  v2 groups, shared messages and chunk indexes. Valid files are unaffected,
+  including a depth-2 HDF5 2.0 chunk index with 40 000 records, now covered by
+  an interop test.
+
+### Integrity
+- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
+  their checksums** (the `checksum` feature, on by default). Every structure
+  in both — header, index block, super block, data block and each data block
+  page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
+  The consequence of skipping it is not a missing warning but wrong data: a
+  single flipped bit in a chunk address still parses, still points inside the
+  file, and the reader hands back whatever bytes now sit there as the chunk's
+  contents. Verified in both directions — the checksums accept files written
+  by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
+  and an interop test corrupts an address to confirm the read now fails
+  instead of returning data (it does return data when the check is removed).
+
+### Performance
+- `clawhdf5-agent`: **opening a store is ~28% faster** (455 ms -> 327 ms at
+  100k x 384). `read_from_disk` memory-mapped the file and then copied the
+  entire mapping into a `Vec` for `File::from_bytes`, when `File::open`
+  memory-maps it directly — so every open paid a full-file memcpy for nothing.
+  Process peak memory is unchanged: the peak falls after the parse, during the
+  index build, so the transient never reached the high-water mark. The
+  footprint harness now reports that peak next to the retained figure, which
+  is how this was checked rather than assumed.
+- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
+  sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
+  elsewhere). The quantised HNSW index used a scalar loop while the `f32` path
+  it was measured against ran AVX2, so the ~13% throughput cost recorded for
+  `MemoryConfig::quantized_index` was a missing kernel rather than a property
+  of int8. With the kernel, at N = 100 000 x 384 and equal recall, the
+  quantised index answers **1.63x as many queries per second** (21 848 vs
+  13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778
+  vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three
+  alternating runs. It remains off by default only because the kernel is
+  AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so
+  the SIMD path is tested to agree with scalar bit for bit.
+
+### Tuning
+- `clawhdf5-agent`: **the HNSW parameters are configurable** —
+  `MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search`
+  (defaults 16, 64, and 0 meaning "scale with `k`", i.e. today's behaviour).
+  They were constants, so a deployment could not trade recall against memory
+  or query speed at all. All three are persisted with the store. Values are
+  clamped where the index requires it: `clawhdf5-ann` asserts a graph degree
+  of at least 2, so a configured 0 — from a file, or from a caller who took 0
+  to mean "default" — used to abort the process inside the builder. Lowering
+  `ef_search` also no longer narrows the candidate pool that fusion sees.
+  **Breaking:** `MemoryConfig` gained fields, so literal constructions need
+  updating; `..Default::default()` does not.
+
+### Documentation
+- `clawhdf5-agent`: `BM25Index::search` claimed to use Block-Max WAND for early
+  termination. It never did; it scores every match exhaustively. It now says
+  so, and why no pruning would help the store: `hybrid_search` uses `scores()`,
+  since fusion normalises over every match.
+
+## v2.6.0 (2026-09-20)
+
+### Upgrade Notes
+- **Re-ranked results change, substantially for the better.** `RerankInput`
+  and `ReRankConfig` gained fields (`relevance`, `relevance_weight`), so
+  literal constructions need updating; `..Default::default()` does not. Any
+  caller that re-ranked was previously getting results ordered by age with the
+  retrieval score discarded — see below.
+- **Breaking:** `MemoryCache::embeddings` is a `cache::Embeddings` rather than
+  a `Vec<Vec<f32>>` (indexing still yields a `&[f32]` row); `embeddings_flat`
+  is gone, replaced by `flat_embeddings()`; `rebuild_flat()` is a deprecated
+  no-op.
+- `MemoryConfig` gained `quantized_index` (default `false`, so behaviour is
+  unchanged unless you opt in); literal constructions need the field.
+
+### Retrieval quality
+- `clawhdf5-agent`: **re-ranking discarded the retrieval score.**
+  `reranker::rerank` built its combined score from temporal decay, source
+  authority and Hebbian activation only — `RerankInput` had no relevance field
+  — so re-ranking a candidate pool reordered it by age and threw the
+  retriever's ordering away. The OpenClaw backend re-ranked every search, so
+  this was its shipping behaviour: measured over the full LongMemEval haystack
+  it cost **40.6pp of Hit@1** (11.0% vs 51.6%) and two thirds of MRR (0.183 vs
+  0.643). `RerankInput::relevance` and `ReRankConfig::relevance_weight` (1.0 by
+  default) fix it: relevance leads and the metadata signals break near-ties,
+  which restores retrieval (Hit@1 +0.4pp vs no re-ranking) and improves
+  recency discrimination by 6–7pp. **Breaking:** `RerankInput` and
+  `ReRankConfig` gained fields, so literal constructions need updating;
+  `..Default::default()` does not.
+- `clawhdf5-bench`: the LongMemEval harness feeds the dataset's real session
+  dates to the store instead of a synthetic counter (decay needs true
+  intervals, not just the right order), and reports `newest_gold_first` — on a
+  `knowledge-update` question, did the newest gold session outrank the stale
+  one it supersedes? Plain recall cannot see this, because both are labelled
+  gold. New `--rerank-sweep`.
+
+### Memory
+- `clawhdf5-agent`: **`MemoryConfig::quantized_index`** stores the vector
+  index's own copy of the embeddings as `i8` rather than `f32`, which at 100k
+  384-dim entries takes the index from 266 to 123 MiB and the whole reopened
+  store from 399 to 256 MiB (2.72x -> **1.74x** the raw vectors). Quantised
+  distances are approximate and `ef` cannot compensate — recall@10 tops out at
+  0.967 against f32's 0.9995 — so the query path re-scores the candidate pool
+  against the exact embeddings the store already holds, which restores recall
+  (0.9940 vs 0.9945 at ef=64) for about 13% of QPS. **Off by default**: it
+  trades query speed for memory, and which side is worth more depends on the
+  deployment. The setting is persisted, so a reopened store does not silently
+  revert to four times the index memory.
+- `clawhdf5-ann`: `Storage::Int8` and the `build_with` / `new_with` /
+  `from_graph_bytes_with` constructors that select it. The scale is per row,
+  not global — a fixed `[-1, 1]` scale spends fewer than 12 of the 255 levels
+  on a unit-length 128-dim vector and is unusable (0.35 top-10 overlap against
+  an exact ranking, versus 0.99 per row). `compact()` keeps the storage it was
+  given; serialized indexes still carry f32 vectors, so a quantised index is
+  rebuilt rather than loaded.
+- `clawhdf5-agent`: **a loaded store holds ~30% less memory** (100k 384-dim
+  entries: 505 -> 357 MiB, 3.44x -> 2.43x the raw vectors). The cache kept
+  every embedding twice — a `Vec<Vec<f32>>` and a flattened copy for the
+  batched kernels, maintained in lock-step — so it now stores only the flat
+  buffer and indexes into it. Recall and query latency are unchanged.
+  **Breaking:** `MemoryCache::embeddings` is a `cache::Embeddings` rather than
+  a `Vec<Vec<f32>>` (indexing still yields a `&[f32]` row); `embeddings_flat`
+  is gone, replaced by `flat_embeddings()`; `rebuild_flat()` is a deprecated
+  no-op. Rows are now always exactly `dim` long — shorter ones are
+  zero-padded — which makes the ragged-row case that used to silently
+  misalign the flattened copy unrepresentable.
+- `clawhdf5-bench`: `search_harness --footprint` reports live heap use per
+  stage, measured with a counting allocator (RSS cannot see a structure freed
+  into the allocator's own pool).
+
+### Testing
+- The Python interop suites honour **`CLAWHDF5_PYTHON`**, and `ci-test.sh`
+  picks up a `.venv/bin/python` automatically. On a PEP 668 "externally
+  managed" system h5py cannot be installed into the system interpreter at all,
+  so every interop suite — the h5py writer round-trips, the facade, netCDF4
+  and the reference files — was skipping silently. A silent skip here is
+  exactly how the v5 compound-datatype bug reached a release.
+  `CLAWHDF5_REQUIRE_INTEROP=1` still turns a skip into a failure.
+
+## v2.5.0 (2026-09-19)
+
+### Upgrade Notes
+- **Retrieval rankings change, for the better.** The default fusion weights
+  move from `0.7/0.3` to `0.4/0.6` (`hybrid::DEFAULT_FUSION`), measured over the
+  full LongMemEval haystack: turn-level Hit@1 51.6% vs 44.2%, MRR 0.643 vs
+  0.586. `unified_search` and the OpenClaw backend pick this up automatically;
+  callers passing weights to `hybrid_search` explicitly are unaffected.
+- **Out-of-range selections are now errors.** `read_*_selection` used to return
+  data for a selection that ran past a dataset edge — a hyperslab came back
+  zero-padded, and a point with an out-of-range coordinate wrapped into the
+  next row. Both are now `FormatError::SelectionOutOfBounds`. Code relying on
+  the old (wrong) values will start seeing errors.
+- **Large compressed datasets written without explicit chunk dimensions get a
+  different layout.** They used to be stored as one chunk; they are now split
+  to ~1 MiB chunks. The files stay standard and h5py-readable, and explicit
+  `with_chunks` is unaffected.
+- `rayon` is now a default dependency of `clawhdf5-agent` (the parallel index
+  build). Opt out with `--no-default-features --features float16,hnsw`.
+- `clawhdf5-ann` search results no longer shrink when records near the query
+  have been deleted, so a search that previously returned fewer than `k`
+  results now returns `k`.
+
+### Retrieval quality
+- `clawhdf5-agent`: optional keyword stemming — `bm25::TokenFilter::Stemmed`
+  and `HDF5Memory::set_token_filter`, so "training" and "trains" match. **Off
+  by default**, on measurement rather than principle: over the full LongMemEval
+  haystack it buys depth and costs the top rank (BM25 alone: Hit@5 +2.8pp,
+  Hit@10 +2.4pp, Hit@1 −1.8pp, MRR unchanged), and on the shipping hybrid
+  configuration the trade is narrower still. See `BENCHMARKS.md`.
+- `clawhdf5-agent`: **`QueryExpander::expand` panicked on ordinary non-ASCII
+  input** — `"İ AI"` was enough. It searched a lowercased copy of the query and
+  then sliced the *original* with those offsets, which only works while
+  lowercasing preserves byte length (Turkish `İ` is 2 bytes and lowercases to
+  3). Depending on where the offsets drifted it either corrupted the output
+  ("İstanbul AI trip" lost a character) or panicked. Matching now walks the
+  original string.
+- `clawhdf5-agent`: query expansion no longer rewrites text inside words.
+  `replace_word_case_insensitive` did a plain substring replace despite its
+  name, so "training" became "trArtificial Intelligencening" and "programming"
+  became "Pull Requestogramming" — every acronym expansion of ordinary prose
+  was corrupt. Matches now require word boundaries; genuine acronyms
+  (`API`, `database`) still expand.
+- `clawhdf5-agent`: **the default fusion weights are now the measured ones.**
+  A sweep of every 0.1 step over the full LongMemEval haystack (500 questions,
+  real MiniLM embeddings) shows the long-standing `0.7/0.3` default is
+  *strictly dominated* by `0.4/0.6` — turn-level Hit@1 51.6% vs 44.2%, Hit@5
+  81.4% vs 79.2%, Hit@10 87.8% vs 85.8%, MRR 0.643 vs 0.586, and better at
+  session level too. The finding was recorded in `BENCHMARKS.md` but had never
+  been applied: `unified_search` and the OpenClaw backend both hardcoded
+  `0.7/0.3`. They now use `hybrid::DEFAULT_FUSION`. **Callers passing weights
+  to `hybrid_search` explicitly are unaffected** — pass `0.4`/`0.6` (or use
+  `hybrid_search_with`) to get the tuned behaviour.
+- `clawhdf5-agent`: fusion is now selectable. New `hybrid::Fusion`
+  (`Weighted { vector, keyword }` or `Rrf { k }`), `hybrid::fuse`,
+  `hybrid::hybrid_search_fused` and `HDF5Memory::hybrid_search_with`.
+  Reciprocal rank fusion existed but was unreachable from the store, so it had
+  never been measured against the weighted sum; the LongMemEval bench now has
+  an `RRF` mode.
+
+### HDF5 Read Path
+- **Selection reads cost what the selection costs.** `read_*_selection` decoded
+  the *entire* dataset and then picked elements out, so a 64 x 64 window of a
+  64 MB compressed dataset took 105 ms - about as long as reading all of it.
+  Now only the rows (contiguous) or chunks that overlap the selection's
+  bounding box are read and decompressed: that window takes 0.39 ms, one row
+  2.7 ms, one column 5.2 ms. Results are identical to the full-read path
+  (equivalence-tested over random hyperslabs and point lists, ranks 1-3,
+  contiguous / chunked / deflate). New `read_harness` bench binary.
+- **Faster full reads** (same-moment A/B, 64 MB `f64`): chunked + deflate
+  110 -> 69 ms, chunked 72 -> 60 ms, contiguous 56 -> 30 ms. The facade's
+  cached read path now decompresses cache misses in parallel batches (it was
+  sequential; only the uncached reader was parallel) and caches only datasets
+  that fit the chunk cache; unfiltered chunks are copied straight from the file
+  bytes; a contiguous dataset is converted straight from the file bytes; and
+  the native-endian conversions no longer zero a buffer before overwriting it.
+- **Datasets indexed by a version-2 B-tree now read** (layout v4, chunk index
+  type 5 — what `libver='latest'` uses for two or more unlimited dimensions;
+  previously "unsupported chunked layout"). The four copies of the chunk-index
+  dispatch are now one shared function, so every read path gets it.
+- **`H5T_STD_REF` references** (HDF5 1.12+, datatype message version 4) parse:
+  `ReferenceType` gains `Object2`, `DatasetRegion2` and `Attribute`, and
+  `read_object_references` decodes the new object references. Previously any
+  dataset of this type failed with `InvalidReferenceType(2)`. Tested against a
+  file written by HDF5 2.0 itself (fixture + generator script committed).
+- **Automatic chunk sizes.** Asking for compression (or any filter) without
+  `with_chunks` used to store the whole dataset as one chunk, so any read had
+  to decompress everything and nothing could be decoded in parallel. Datasets up
+  to 1 MiB stay a single chunk, as before; larger ones are split by halving the
+  dimensions in turn until a chunk is at most 1 MiB (the approach h5py takes).
+  **Behaviour change:** large compressed datasets written without explicit
+  chunk dimensions get a different (standard, h5py-readable) layout. Explicit
+  `with_chunks` is unaffected.
+- **Out-of-range selections are errors.** They used to return data: a hyperslab
+  past an edge came back padded with zeros, and a point whose column was out of
+  range wrapped into the next row and returned that element. Now
+  `FormatError::SelectionOutOfBounds` (also for a rank mismatch or overlapping
+  blocks).
+
+### Search
+- `clawhdf5-ann`: **faster index builds.** Back-link pruning is 90% of a
+  build's distance evaluations; the bulk build now inserts in batches and
+  prunes each overflowing neighbour list once per batch (10K: 1676 -> 1074 ms).
+  With the `parallel` feature, planning and pruning run on a thread pool (10K:
+  388 ms, 100K: ~21 s -> 5.9 s on 16 cores). The graph is deterministic and
+  identical with or without the feature. `clawhdf5-agent`'s `parallel` feature
+  enables it for the agent's index and is now **on by default** (adds `rayon`
+  to the default dependency set; build with `--no-default-features --features
+  float16,hnsw` to opt out).
+- `clawhdf5-ann`: `HnswIndex::search` returned fewer than `k` results — often
+  none — when the records nearest the query had been deleted: it collected `ef`
+  candidates, *then* dropped the deleted ones, *then* took `k`. Deleted nodes
+  are now traversed as waypoints but never occupy a result slot, so a search
+  returns the `k` nearest live records. Matters for any store that deletes or
+  supersedes memories without compacting straight away.
+
 ## v2.4.0 (2026-09-19)
 
 ### Upgrade Notes

CLAUDE.md

@@ -33,11 +33,21 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
   the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
   the cache and self-heals on drift). Build the agent with
   `--no-default-features --features float16` to force the exact linear cosine scan.
+  The agent's `parallel` feature (also default) builds the index on a thread
+  pool; the graph is identical with or without it.
   The index uses the HNSW paper's diversity heuristic for neighbour selection
   (plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its
   graph is saved to `<store>.h5.ann` at each checkpoint and reloaded by `open()`
   (tied to the checkpoint by a generation id; stale/damaged sidecars are
-  ignored and the index rebuilt). `hybrid_search` keeps one incremental BM25
+  ignored and the index rebuilt). `MemoryConfig::quantized_index` (off by
+  default, persisted) stores the index's own copy of the embeddings as `i8`,
+  which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors
+  at 100K); because quantised distances are approximate and `ef` cannot
+  compensate, the query path then re-scores the candidate pool against the
+  exact embeddings, which holds recall at the f32 index's level. On AVX2 it is
+  also 1.63x the QPS and 1.8x the build speed (`clawhdf5_accel::dot_i8`); it
+  stays off by default only because that kernel is AVX2-only and aarch64 falls
+  back to scalar. `hybrid_search` keeps one incremental BM25
   index for the life of the store and never writes the store: Hebbian
   activation boosts are persisted by the next checkpoint (or on drop), not per
   query. Measure any search-path change with

Cargo.toml

@@ -21,7 +21,7 @@ members = [
 resolver = "2"
 
 [workspace.package]
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"

README.md

@@ -432,6 +432,20 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
 | `agent` | no | Full agent memory layer |
 | `float16` | **yes** | Half-precision embedding storage (2× compression) |
 | `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
+
+`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search` tune the
+vector index (16 / 64 / scale-with-`k` by default) and are stored with the
+file.
+
+`MemoryConfig::quantized_index` (off by default) stores the HNSW index's own
+copy of the embeddings as `i8`, roughly halving a loaded store's memory
+(2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are
+approximate, so the query path re-scores the candidate pool against the exact
+embeddings the store already holds, which keeps recall at the `f32` index's
+level. On AVX2 it is also **faster** — 1.63x the queries per second and 1.8x
+the build speed at equal recall — because the int8 kernel is SIMD too. It
+stays off by default only because that kernel is AVX2-only and aarch64 falls
+back to a scalar loop. See `BENCHMARKS.md`, "Quantising the index copy".
 | `parallel` | no | Rayon parallel search |
 | `fast-math` | no | BLAS matrix-vector multiply |
 | `accelerate` | no | Apple Accelerate / AMX (macOS) |
@@ -492,6 +506,14 @@ cargo build -p clawhdf5-agent --features "agent,float16,accelerate,parallel,gpu"
 # Tests
 cargo test --workspace            # all 1,650+ tests
 cargo test -p clawhdf5-agent      # agent memory tests
+scripts/ci-test.sh                # what CI runs: fmt, clippy matrix, tests,
+                                  # h5py/netCDF4 interop, no_std
+
+# The interop suites need a Python with h5py; on a PEP 668 system that has to
+# be a virtualenv. `ci-test.sh` finds `.venv` on its own, or set
+# CLAWHDF5_PYTHON. Without one they skip — set CLAWHDF5_REQUIRE_INTEROP=1 to
+# make that a failure instead.
+python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4 xarray
 
 # Benchmarks
 cargo bench -p clawhdf5-agent               # agent memory suite

crates/clawhdf5-accel/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-accel"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "SIMD-accelerated operations for rustyhdf5"
 license = "MIT"

crates/clawhdf5-accel/src/avx2.rs

@@ -25,6 +25,55 @@ unsafe fn hsum_256(v: __m256) -> f32 {
     _mm_cvtss_f32(result)
 }
 
+/// AVX2 dot product of two `i8` slices, widened to `i32`.
+///
+/// Each 16-byte half is sign-extended to sixteen `i16` lanes and multiplied
+/// pairwise with `madd_epi16`, which sums adjacent products straight into
+/// eight `i32` lanes — the widening that an autovectorised scalar loop does
+/// in several shuffles is one instruction here. A pair sum is at most
+/// `2 * 127 * 127`, far inside `i32`.
+///
+/// # Safety
+/// Caller must verify is_x86_feature_detected!("avx2").
+// SAFETY: Caller must have verified AVX2 via is_x86_feature_detected!.
+#[target_feature(enable = "avx2")]
+pub unsafe fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
+    // SAFETY: Caller guarantees AVX2 is available per the # Safety contract;
+    // every load reads 32 bytes at an index checked against `len` first.
+    unsafe {
+        assert_eq!(a.len(), b.len());
+        let len = a.len();
+        let mut i = 0;
+        let mut acc0 = _mm256_setzero_si256();
+        let mut acc1 = _mm256_setzero_si256();
+
+        while i + 32 <= len {
+            let va = _mm256_loadu_si256(a.as_ptr().add(i).cast());
+            let vb = _mm256_loadu_si256(b.as_ptr().add(i).cast());
+            let a_lo = _mm256_cvtepi8_epi16(_mm256_castsi256_si128(va));
+            let b_lo = _mm256_cvtepi8_epi16(_mm256_castsi256_si128(vb));
+            let a_hi = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(va, 1));
+            let b_hi = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(vb, 1));
+            acc0 = _mm256_add_epi32(acc0, _mm256_madd_epi16(a_lo, b_lo));
+            acc1 = _mm256_add_epi32(acc1, _mm256_madd_epi16(a_hi, b_hi));
+            i += 32;
+        }
+
+        // Horizontal sum of the eight i32 lanes.
+        let v = _mm256_add_epi32(acc0, acc1);
+        let s128 = _mm_add_epi32(_mm256_castsi256_si128(v), _mm256_extracti128_si256(v, 1));
+        let s64 = _mm_add_epi32(s128, _mm_unpackhi_epi64(s128, s128));
+        let s32 = _mm_add_epi32(s64, _mm_shuffle_epi32(s64, 0b01));
+        let mut sum = _mm_cvtsi128_si32(s32);
+
+        while i < len {
+            sum += i32::from(a[i]) * i32::from(b[i]);
+            i += 1;
+        }
+        sum
+    }
+}
+
 /// AVX2 dot product for f32 slices.
 ///
 /// # Safety

crates/clawhdf5-accel/src/lib.rs

@@ -122,6 +122,23 @@ pub fn dot_product(a: &[f32], b: &[f32]) -> f32 {
     }
 }
 
+/// Dot product of two `i8` slices, widened to `i32`.
+///
+/// The kernel behind int8-quantised vector search. Uses the AVX2 path
+/// whenever AVX2 is present — including on AVX-512 machines, where it is
+/// what the f32 kernels use too on a default build.
+pub fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
+    match detect_backend() {
+        #[cfg(target_arch = "x86_64")]
+        // SAFETY: both variants imply AVX2 was detected at runtime (the
+        // AVX-512 backend is only selected on CPUs that also have AVX2).
+        Backend::Avx2 | Backend::Avx512 if is_x86_feature_detected!("avx2") => unsafe {
+            avx2::dot_i8(a, b)
+        },
+        _ => scalar::dot_i8(a, b),
+    }
+}
+
 /// Compute the L2 norm (magnitude) of a vector.
 pub fn vector_norm(v: &[f32]) -> f32 {
     dot_product(v, v).sqrt()
@@ -713,3 +730,42 @@ mod tests {
         }
     }
 }
+
+#[cfg(test)]
+mod dot_i8_tests {
+    use super::*;
+
+    fn codes(n: usize, seed: u64) -> Vec<i8> {
+        let mut state = seed;
+        (0..n)
+            .map(|_| {
+                state = state
+                    .wrapping_mul(6_364_136_223_846_793_005)
+                    .wrapping_add(1_442_695_040_888_963_407);
+                // Full range, including the extremes.
+                ((state >> 56) as u8) as i8
+            })
+            .collect()
+    }
+
+    #[test]
+    fn dispatched_kernel_matches_scalar_exactly() {
+        // Integer arithmetic: the SIMD path must agree bit for bit, at every
+        // length — including ones that are not multiples of the 32-byte block,
+        // which exercise the tail.
+        for len in [0, 1, 7, 31, 32, 33, 63, 64, 100, 384, 385, 1536] {
+            let a = codes(len, 1 + len as u64);
+            let b = codes(len, 1000 + len as u64);
+            assert_eq!(dot_i8(&a, &b), scalar::dot_i8(&a, &b), "len {len}");
+        }
+    }
+
+    #[test]
+    fn extremes_do_not_overflow() {
+        // -128 * -128 is the largest product; a long run of it must still fit.
+        let a = vec![-128i8; 4096];
+        assert_eq!(dot_i8(&a, &a), 4096 * 128 * 128);
+        let b = vec![127i8; 4096];
+        assert_eq!(dot_i8(&a, &b), -4096 * 128 * 127);
+    }
+}

crates/clawhdf5-accel/src/scalar.rs

@@ -140,3 +140,33 @@ fn f16_to_f32_soft(h: u16) -> f32 {
 
     f32::from_bits(f32_bits)
 }
+
+/// Dot product of two `i8` slices, widened to `i32`.
+///
+/// `dim` terms of at most `127 * 127` fit an `i32` for any realistic
+/// dimension (over 130 000 terms before overflow is possible).
+pub fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
+    assert_eq!(a.len(), b.len());
+    // Four independent accumulators over 32-lane blocks: the widening product
+    // has to sit in a fixed-length chunk for the vectoriser to see it, and the
+    // separate accumulators keep it off one dependency chain.
+    const LANE: usize = 8;
+    let (a_blocks, a_tail) = a.as_chunks::<{ LANE * 4 }>();
+    let (b_blocks, b_tail) = b.as_chunks::<{ LANE * 4 }>();
+    let mut acc = [0i32; 4];
+    for (x, y) in a_blocks.iter().zip(b_blocks) {
+        for (lane, slot) in acc.iter_mut().enumerate() {
+            let mut sum = 0i32;
+            for k in 0..LANE {
+                sum += i32::from(x[lane * LANE + k]) * i32::from(y[lane * LANE + k]);
+            }
+            *slot += sum;
+        }
+    }
+    let tail: i32 = a_tail
+        .iter()
+        .zip(b_tail)
+        .map(|(&x, &y)| i32::from(x) * i32::from(y))
+        .sum();
+    acc[0] + acc[1] + acc[2] + acc[3] + tail
+}

crates/clawhdf5-agent/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-agent"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "HDF5-backed persistent memory store for on-device AI agents"
 license = "MIT"
@@ -10,12 +10,12 @@ keywords = ["agent", "memory", "hdf5", "vector-search", "embedding"]
 categories = ["database", "science", "algorithms"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0", features = ["parallel", "fast-checksum"] }
-clawhdf5 = { path = "../clawhdf5", version = "2.4.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.4.0", features = ["mmap"] }
-clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.4.0" }
-clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.4.0", optional = true }
-clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.4.0", optional = true, default-features = false }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0", features = ["parallel", "fast-checksum"] }
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0", features = ["mmap"] }
+clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.7.0" }
+clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true }
+clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
 serde = { workspace = true }
 byteorder = "1"
 half = { workspace = true, optional = true }
@@ -45,9 +45,11 @@ name = "memory_bench"
 harness = false
 
 [features]
-default = ["float16", "hnsw"]
+default = ["float16", "hnsw", "parallel"]
 float16 = ["half"]
-parallel = ["rayon"]
+# Rayon-parallel brute-force search strategies, and a parallel bulk build of
+# the HNSW index (same graph, several times faster on a multi-core machine).
+parallel = ["rayon", "clawhdf5-ann?/parallel"]
 # Compress embeddings with Zstd instead of deflate when
 # `MemoryConfig::compression` is on. Off by default: it links libzstd (C).
 zstd = ["clawhdf5/zstd"]

crates/clawhdf5-agent/src/agents_md.rs

@@ -118,6 +118,10 @@ mod tests {
             created_at: "2025-01-01T00:00:00Z".to_string(),
             wal_enabled: false,
             wal_max_entries: 500,
+            quantized_index: false,
+            hnsw_m: 16,
+            hnsw_ef_construction: 64,
+            hnsw_ef_search: 0,
         }
     }
 

crates/clawhdf5-agent/src/async_memory.rs

@@ -37,7 +37,7 @@
 //! let mem = AsyncHDF5Memory::open_with(path, config).await?;
 //! mem.save(entry).await?;           // buffered → background writer
 //! mem.save_batch(entries).await?;    // also buffered
-//! let results = mem.hybrid_search(emb, "query".into(), 0.7, 0.3, 5).await;
+//! let results = mem.hybrid_search(emb, "query".into(), 0.4, 0.6, 5).await;
 //! mem.shutdown().await?;             // final flush + stop
 //! ```
 

crates/clawhdf5-agent/src/bm25.rs

@@ -59,11 +59,18 @@ pub struct BM25Index {
     k1: f32,
     /// BM25 b parameter.
     b: f32,
+    /// Applied to every document and query token, so the two always agree.
+    filter: TokenFilter,
 }
 
 impl BM25Index {
     /// Build a BM25 index from a set of documents, excluding tombstoned entries.
     pub fn build(documents: &[String], tombstones: &[u8]) -> Self {
+        Self::build_with(documents, tombstones, TokenFilter::default())
+    }
+
+    /// [`BM25Index::build`] with the token filter chosen explicitly.
+    pub fn build_with(documents: &[String], tombstones: &[u8], filter: TokenFilter) -> Self {
         let mut index = Self {
             inverted: HashMap::new(),
             doc_lengths: vec![0; documents.len()],
@@ -72,6 +79,7 @@ impl BM25Index {
             num_docs: 0,
             k1: DEFAULT_K1,
             b: DEFAULT_B,
+            filter,
         };
         index.index_documents(documents, tombstones);
         index
@@ -80,8 +88,11 @@ impl BM25Index {
     /// Search the index for a query, returning the top `k` results
     /// as `(doc_id, score)` pairs sorted by score descending.
     ///
-    /// Uses Block-Max WAND for early termination when remaining documents
-    /// cannot beat the current top-k threshold.
+    /// Scores every matching document exhaustively, then keeps the top `k`.
+    /// There is no early termination (WAND, MaxScore): the store's hot path
+    /// is [`scores`](Self::scores), because score fusion normalises over the
+    /// whole matching set and so needs every score, which no pruning scheme
+    /// can skip. This method is for BM25-only callers.
     pub fn search(&self, query: &str, k: usize) -> Vec<(usize, f32)> {
         if k == 0 {
             return Vec::new();
@@ -120,7 +131,7 @@ impl BM25Index {
         // add/remove, and costs one `ln` per query term.
         let mut acc = vec![0.0f32; self.doc_lengths.len()];
         let mut matched = false;
-        for token in tokenize(query) {
+        for token in tokenize_with(query, self.filter) {
             let Some(postings) = self.inverted.get(token.as_str()) else {
                 continue;
             };
@@ -146,6 +157,11 @@ impl BM25Index {
             .collect()
     }
 
+    /// The token filter this index was built with.
+    pub fn token_filter(&self) -> TokenFilter {
+        self.filter
+    }
+
     /// Number of document slots (live or not) the index covers. Ids are
     /// positions in the document list it mirrors.
     pub fn len(&self) -> usize {
@@ -168,7 +184,7 @@ impl BM25Index {
         }
         debug_assert_eq!(self.doc_lengths[doc_id], 0, "slot {doc_id} is occupied");
 
-        let tokens = tokenize(text);
+        let tokens = tokenize_with(text, self.filter);
         let mut term_freqs: HashMap<&str, u32> = HashMap::new();
         for token in &tokens {
             *term_freqs.entry(token).or_insert(0) += 1;
@@ -201,7 +217,7 @@ impl BM25Index {
     /// Remove document `doc_id`, whose indexed text was `text`. The text is
     /// needed to find its postings; pass exactly what was added.
     pub fn remove_document(&mut self, doc_id: usize, text: &str) {
-        let tokens = tokenize(text);
+        let tokens = tokenize_with(text, self.filter);
         let mut seen: std::collections::HashSet<&str> = std::collections::HashSet::new();
         for token in &tokens {
             if !seen.insert(token) {
@@ -252,7 +268,7 @@ impl BM25Index {
                 continue;
             }
 
-            let tokens = tokenize(doc);
+            let tokens = tokenize_with(doc, self.filter);
             let doc_len = tokens.len() as u32;
             self.doc_lengths[i] = doc_len;
             total_length += doc_len as u64;
@@ -285,11 +301,86 @@ impl BM25Index {
 
 /// Tokenize a string: lowercase, split on non-alphanumeric characters,
 /// filter empty tokens.
+/// What [`tokenize_with`] does to each token after splitting.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum TokenFilter {
+    /// Lowercase and split only — the original behaviour.
+    #[default]
+    Plain,
+    /// Also strip common English inflections, so "running" and "runs" match
+    /// "run". Conservative on purpose: only plural and past/continuous verb
+    /// endings, and only on tokens long enough that stripping leaves a real
+    /// stem. A stemmer earns its keep by conflating *related* words; an
+    /// aggressive one also conflates unrelated ones ("universe"/"university"),
+    /// which costs precision.
+    Stemmed,
+}
+
+/// Strip common English inflections from an already-lowercased token.
+///
+/// Applied identically to documents and queries, so the pair only has to agree
+/// with itself — the stem need not be a real word.
+fn stem(token: &str) -> &str {
+    // Below this, stripping does more harm than good ("bed" -> "b").
+    const MIN_STEM: usize = 4;
+    let strip = |suffix: &str, min_len: usize| -> Option<&str> {
+        let stem = token.strip_suffix(suffix)?;
+        (stem.len() >= min_len).then_some(stem)
+    };
+
+    // Plurals first: "studies" -> "studi", "classes" -> "class", "cats" -> "cat".
+    // "ies" keeps its "i" so the result meets "-ied" ("studied" -> "studi").
+    if let Some(stem) = strip("ies", 2) {
+        return &token[..stem.len() + 1];
+    }
+    for suffix in ["sses", "shes", "ches", "xes", "zes"] {
+        if let Some(stem) = strip(suffix, MIN_STEM - 1) {
+            // Keep the sibilant: "classes" -> "class", not "clas".
+            return &token[..stem.len() + 2];
+        }
+    }
+    // Verb endings before the bare plural, so "raced" doesn't become "raced".
+    if let Some(stem) = strip("ing", MIN_STEM - 1).or_else(|| strip("ed", MIN_STEM - 1)) {
+        return undouble(stem);
+    }
+    if !token.ends_with("ss")
+        && !token.ends_with("us")
+        && !token.ends_with("is")
+        && let Some(stem) = strip("s", MIN_STEM - 1)
+    {
+        return stem;
+    }
+    token
+}
+
+/// "runn" -> "run": undo the consonant doubling that "-ing"/"-ed" introduce.
+fn undouble(stem: &str) -> &str {
+    let mut chars = stem.chars().rev();
+    let (Some(last), Some(prev)) = (chars.next(), chars.next()) else {
+        return stem;
+    };
+    let doubled = last == prev && !"aeiou".contains(last) && last.is_ascii_alphabetic();
+    if doubled && stem.len() > 3 {
+        &stem[..stem.len() - 1]
+    } else {
+        stem
+    }
+}
+
+#[cfg(test)]
 fn tokenize(text: &str) -> Vec<String> {
+    tokenize_with(text, TokenFilter::Plain)
+}
+
+/// Split `text` into scoring tokens under `filter`.
+pub fn tokenize_with(text: &str, filter: TokenFilter) -> Vec<String> {
     text.to_lowercase()
         .split(|c: char| !c.is_alphanumeric())
         .filter(|s| !s.is_empty())
-        .map(|s| s.to_string())
+        .map(|token| match filter {
+            TokenFilter::Plain => token.to_string(),
+            TokenFilter::Stemmed => stem(token).to_string(),
+        })
         .collect()
 }
 
@@ -473,8 +564,9 @@ mod tests {
     }
 
     #[test]
-    fn wand_returns_same_results_as_exhaustive() {
-        // WAND-style search should produce same scores as exhaustive
+    fn top_k_search_matches_ranking_every_score() {
+        // `search` must agree with ranking the full `scores` set — the
+        // bounded heap is an optimisation over sorting, not an approximation.
         let docs: Vec<String> = (0..100)
             .map(|i| {
                 if i % 3 == 0 {
@@ -599,6 +691,53 @@ mod tests {
         }
     }
 
+    #[test]
+    fn stemming_conflates_inflections_of_the_same_word() {
+        let stem_of = |w: &str| tokenize_with(w, TokenFilter::Stemmed).pop().unwrap();
+        // Pairs that should meet.
+        for (a, b) in [
+            ("running", "runs"),
+            ("trained", "training"),
+            ("miles", "mile"),
+            ("studies", "studied"),
+            ("mentioned", "mentioning"),
+            ("classes", "class"),
+            ("planned", "planning"),
+        ] {
+            assert_eq!(stem_of(a), stem_of(b), "{a} / {b} should share a stem");
+        }
+        // Pairs that must stay apart. Note which pairs are deliberately absent:
+        // "bed"/"bedding" and "gas"/"gassed" both collapse to one stem, which
+        // is what Porter does too and is right — they are related words.
+        for (a, b) in [
+            ("universe", "university"),
+            ("business", "busy"),
+            ("this", "thing"),
+        ] {
+            assert_ne!(stem_of(a), stem_of(b), "{a} / {b} must not be conflated");
+        }
+        // Short words and non-inflections are left alone.
+        for word in ["run", "bus", "is", "his", "data", "gas"] {
+            assert_eq!(stem_of(word), word, "{word} should be untouched");
+        }
+    }
+
+    #[test]
+    fn stemming_is_off_by_default_and_applied_consistently() {
+        assert_eq!(tokenize("Running miles"), ["running", "miles"]);
+        assert_eq!(
+            tokenize_with("Running miles", TokenFilter::Stemmed),
+            ["run", "mile"]
+        );
+
+        // A query inflected differently from the document still matches.
+        let docs = vec!["I ran while training for the marathon".to_string()];
+        let plain = BM25Index::build_with(&docs, &[0], TokenFilter::Plain);
+        let stemmed = BM25Index::build_with(&docs, &[0], TokenFilter::Stemmed);
+        assert!(plain.search("trains", 1).is_empty());
+        assert_eq!(stemmed.search("trains", 1).len(), 1);
+    }
+
     #[test]
     fn ties_break_towards_the_lower_doc_id() {
         let docs: Vec<String> = (0..6).map(|_| "same text".to_string()).collect();

crates/clawhdf5-agent/src/cache.rs

@@ -2,16 +2,143 @@
 
 use crate::vector_search;
 
+/// Every entry's embedding, in one contiguous `[N x dim]` buffer.
+///
+/// Rows are always exactly `dim` long: a shorter one is zero-padded, a longer
+/// one truncated. The previous `Vec<Vec<f32>>` allowed ragged rows, which
+/// silently misaligned the flattened copy that the batched kernels read — a
+/// single wrong-length embedding shifted every row after it. Padding makes
+/// that unrepresentable. A record stored without an embedding therefore holds
+/// a zero row, and is told apart by its norm being zero rather than by length.
+///
+/// This used to be two fields — a `Vec<Vec<f32>>` and a flattened copy kept in
+/// lock-step — which stored the whole corpus twice and cost one heap
+/// allocation per entry on top. At 100k 384-dim entries that duplicate was
+/// ~150 MiB. Indexing yields a `&[f32]` row, so `embeddings[i]` still reads
+/// the same way.
+#[derive(Debug, Clone, Default)]
+pub struct Embeddings {
+    flat: Vec<f32>,
+    dim: usize,
+}
+
+impl Embeddings {
+    pub fn new(dim: usize) -> Self {
+        Self {
+            flat: Vec::new(),
+            dim,
+        }
+    }
+
+    /// Number of embeddings.
+    pub fn len(&self) -> usize {
+        self.flat.len().checked_div(self.dim).unwrap_or(0)
+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.len() == 0
+    }
+
+    /// The whole buffer, `[N x dim]` row-major — what batched kernels read.
+    pub fn as_flat(&self) -> &[f32] {
+        &self.flat
+    }
+
+    pub fn dim(&self) -> usize {
+        self.dim
+    }
+
+    /// Row `i`, or `None` if out of range.
+    pub fn get(&self, i: usize) -> Option<&[f32]> {
+        let start = i.checked_mul(self.dim)?;
+        self.flat.get(start..start.checked_add(self.dim)?)
+    }
+
+    pub fn iter(&self) -> impl ExactSizeIterator<Item = &[f32]> {
+        self.flat.chunks_exact(self.dim.max(1))
+    }
+
+    /// Append one embedding. A row whose length doesn't match `dim` is padded
+    /// or truncated, so the buffer stays rectangular whatever a caller passes.
+    pub fn push(&mut self, embedding: &[f32]) {
+        if self.dim == 0 {
+            return;
+        }
+        let take = embedding.len().min(self.dim);
+        self.flat.extend_from_slice(&embedding[..take]);
+        self.flat.resize(self.flat.len() + (self.dim - take), 0.0);
+    }
+
+    /// Replace row `i`. Out-of-range indices are ignored.
+    pub fn set(&mut self, i: usize, embedding: &[f32]) {
+        let Some(start) = i.checked_mul(self.dim) else {
+            return;
+        };
+        if start + self.dim > self.flat.len() {
+            return;
+        }
+        let take = embedding.len().min(self.dim);
+        self.flat[start..start + take].copy_from_slice(&embedding[..take]);
+        self.flat[start + take..start + self.dim].fill(0.0);
+    }
+
+    /// Keep only the rows `keep` returns true for, preserving order.
+    pub fn retain(&mut self, mut keep: impl FnMut(usize) -> bool) {
+        if self.dim == 0 {
+            return;
+        }
+        let mut write = 0usize;
+        for read in 0..self.len() {
+            if keep(read) {
+                if write != read {
+                    let (dst, src) = (write * self.dim, read * self.dim);
+                    self.flat.copy_within(src..src + self.dim, dst);
+                }
+                write += 1;
+            }
+        }
+        self.flat.truncate(write * self.dim);
+    }
+
+    /// Replace the contents with `rows`.
+    pub fn reset_from(&mut self, dim: usize, rows: impl IntoIterator<Item = Vec<f32>>) {
+        self.dim = dim;
+        self.flat.clear();
+        for row in rows {
+            self.push(&row);
+        }
+    }
+
+    /// Adopt an already-flat buffer, trimming any partial trailing row.
+    pub fn set_flat(&mut self, dim: usize, mut flat: Vec<f32>) {
+        self.dim = dim;
+        match flat.len().checked_div(dim) {
+            Some(rows) => flat.truncate(rows * dim),
+            None => flat.clear(),
+        }
+        self.flat = flat;
+    }
+}
+
+impl PartialEq for Embeddings {
+    fn eq(&self, other: &Self) -> bool {
+        self.dim == other.dim && self.flat == other.flat
+    }
+}
+
+impl std::ops::Index<usize> for Embeddings {
+    type Output = [f32];
+
+    fn index(&self, i: usize) -> &[f32] {
+        self.get(i).expect("embedding index out of range")
+    }
+}
+
 /// In-memory cache for the /memory group data.
 #[derive(Debug, Clone)]
 pub struct MemoryCache {
     pub chunks: Vec<String>,
-    pub embeddings: Vec<Vec<f32>>,
-    /// `embeddings` flattened into one contiguous `[N × embedding_dim]`
-    /// buffer, maintained incrementally alongside `embeddings` (push/update/
-    /// compact) so BLAS/Accelerate batch search can read it directly instead
-    /// of re-flattening the whole corpus on every query.
-    pub embeddings_flat: Vec<f32>,
+    pub embeddings: Embeddings,
     pub source_channels: Vec<String>,
     pub timestamps: Vec<f64>,
     pub session_ids: Vec<String>,
@@ -28,8 +155,7 @@ impl MemoryCache {
     pub fn new(embedding_dim: usize) -> Self {
         Self {
             chunks: Vec::new(),
-            embeddings: Vec::new(),
-            embeddings_flat: Vec::new(),
+            embeddings: Embeddings::new(embedding_dim),
             source_channels: Vec::new(),
             timestamps: Vec::new(),
             session_ids: Vec::new(),
@@ -41,15 +167,14 @@ impl MemoryCache {
         }
     }
 
-    /// Rebuild `embeddings_flat` from `embeddings` from scratch. Callers that
-    /// populate `embeddings` directly (bulk loads) must call this afterward.
-    pub fn rebuild_flat(&mut self) {
-        self.embeddings_flat.clear();
-        self.embeddings_flat
-            .reserve(self.embeddings.len() * self.embedding_dim);
-        for emb in &self.embeddings {
-            self.embeddings_flat.extend_from_slice(emb);
-        }
+    /// Kept for callers that used to have to re-flatten after a bulk load.
+    /// The buffer is always flat now, so there is nothing to rebuild.
+    #[deprecated(note = "embeddings are stored flat; this is a no-op")]
+    pub fn rebuild_flat(&mut self) {}
+
+    /// The embeddings as one contiguous `[N x dim]` buffer.
+    pub fn flat_embeddings(&self) -> &[f32] {
+        self.embeddings.as_flat()
     }
 
     /// Total number of entries (including tombstoned).
@@ -79,8 +204,7 @@ impl MemoryCache {
         let idx = self.chunks.len();
         let norm = vector_search::compute_norm(&embedding);
         self.chunks.push(chunk);
-        self.embeddings_flat.extend_from_slice(&embedding);
-        self.embeddings.push(embedding);
+        self.embeddings.push(&embedding);
         self.source_channels.push(source_channel);
         self.timestamps.push(timestamp);
         self.session_ids.push(session_id);
@@ -118,20 +242,7 @@ impl MemoryCache {
         if idx < self.chunks.len() {
             let norm = vector_search::compute_norm(&embedding);
             self.chunks[idx] = chunk;
-            let dim = self.embedding_dim;
-            let flat_start = idx * dim;
-            let matches_dim =
-                embedding.len() == dim && flat_start + dim <= self.embeddings_flat.len();
-            self.embeddings[idx] = embedding;
-            if matches_dim {
-                self.embeddings_flat[flat_start..flat_start + dim]
-                    .copy_from_slice(&self.embeddings[idx]);
-            } else {
-                // Embedding length doesn't match embedding_dim (shouldn't
-                // happen in practice) — fall back to a full rebuild rather
-                // than leave embeddings_flat misaligned with embeddings.
-                self.rebuild_flat();
-            }
+            self.embeddings.set(idx, &embedding);
             self.source_channels[idx] = source_channel;
             self.timestamps[idx] = timestamp;
             self.session_ids[idx] = session_id;
@@ -183,7 +294,7 @@ impl MemoryCache {
                 new_idx += 1;
                 let norm = vector_search::compute_norm(&self.embeddings[i]);
                 new_chunks.push(self.chunks[i].clone());
-                new_embeddings.push(self.embeddings[i].clone());
+                new_embeddings.push(self.embeddings[i].to_vec());
                 new_source_channels.push(self.source_channels[i].clone());
                 new_timestamps.push(self.timestamps[i]);
                 new_session_ids.push(self.session_ids[i].clone());
@@ -196,7 +307,8 @@ impl MemoryCache {
 
         let removed = old_len - new_chunks.len();
         self.chunks = new_chunks;
-        self.embeddings = new_embeddings;
+        self.embeddings
+            .reset_from(self.embedding_dim, new_embeddings);
         self.source_channels = new_source_channels;
         self.timestamps = new_timestamps;
         self.session_ids = new_session_ids;
@@ -204,16 +316,14 @@ impl MemoryCache {
         self.tombstones = new_tombstones;
         self.norms = new_norms;
         self.activation_weights = new_activation_weights;
-        self.rebuild_flat();
 
         (removed, index_map)
     }
 
-    /// Flatten all embeddings into a single Vec<f32> for HDF5 storage.
-    /// `embeddings_flat` is already maintained incrementally, so this just
-    /// clones it — kept as a method for callers that want an owned copy.
-    pub fn flat_embeddings(&self) -> Vec<f32> {
-        self.embeddings_flat.clone()
+    /// All embeddings as one owned `[N x dim]` buffer, for HDF5 storage.
+    /// Prefer [`MemoryCache::flat_embeddings`] where a borrow will do.
+    pub fn flat_embeddings_owned(&self) -> Vec<f32> {
+        self.embeddings.as_flat().to_vec()
     }
 }
 
@@ -224,7 +334,7 @@ mod tests {
     /// `embeddings_flat` must always equal a from-scratch flatten of `embeddings`.
     fn assert_flat_in_sync(cache: &MemoryCache) {
         let expected: Vec<f32> = cache.embeddings.iter().flatten().copied().collect();
-        assert_eq!(cache.embeddings_flat, expected);
+        assert_eq!(cache.embeddings.as_flat(), expected);
     }
 
     #[test]
@@ -247,7 +357,10 @@ mod tests {
             String::new(),
         );
         assert_flat_in_sync(&cache);
-        assert_eq!(cache.embeddings_flat, vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
+        assert_eq!(
+            cache.embeddings.as_flat(),
+            vec![1.0, 2.0, 3.0, 4.0, 5.0, 6.0]
+        );
     }
 
     #[test]
@@ -279,7 +392,7 @@ mod tests {
         );
         assert_flat_in_sync(&cache);
         assert_eq!(
-            cache.embeddings_flat,
+            cache.embeddings.as_flat(),
             vec![7.0, 8.0, 9.0, 4.0, 5.0, 6.0],
             "update must overwrite the correct flat slice, not just append"
         );
@@ -315,14 +428,15 @@ mod tests {
         cache.mark_deleted(1);
         cache.compact();
         assert_flat_in_sync(&cache);
-        assert_eq!(cache.embeddings_flat, vec![1.0, 1.0, 3.0, 3.0]);
+        assert_eq!(cache.embeddings.as_flat(), vec![1.0, 1.0, 3.0, 3.0]);
     }
 
     #[test]
     fn rebuild_flat_matches_manual_flatten() {
         let mut cache = MemoryCache::new(2);
-        cache.embeddings = vec![vec![1.0, 2.0], vec![3.0, 4.0]];
-        cache.rebuild_flat();
-        assert_eq!(cache.embeddings_flat, vec![1.0, 2.0, 3.0, 4.0]);
+        cache
+            .embeddings
+            .reset_from(2, vec![vec![1.0, 2.0], vec![3.0, 4.0]]);
+        assert_eq!(cache.embeddings.as_flat(), vec![1.0, 2.0, 3.0, 4.0]);
     }
 }

crates/clawhdf5-agent/src/hybrid.rs

@@ -28,13 +28,40 @@ use crate::vector_search;
 pub fn hybrid_search(
     query_embedding: &[f32],
     query_text: &str,
-    vectors: &[Vec<f32>],
-    _chunks: &[String],
+    vectors: &(impl crate::vector_search::VectorSet + Sync + ?Sized),
+    chunks: &[String],
     tombstones: &[u8],
     bm25_index: &BM25Index,
     vector_weight: f32,
     keyword_weight: f32,
     k: usize,
+) -> Vec<(usize, f32)> {
+    hybrid_search_fused(
+        query_embedding,
+        query_text,
+        vectors,
+        chunks,
+        tombstones,
+        bm25_index,
+        Fusion::Weighted {
+            vector: vector_weight,
+            keyword: keyword_weight,
+        },
+        k,
+    )
+}
+
+/// [`hybrid_search`] with the fusion method chosen explicitly.
+#[allow(clippy::too_many_arguments)]
+pub fn hybrid_search_fused(
+    query_embedding: &[f32],
+    query_text: &str,
+    vectors: &(impl crate::vector_search::VectorSet + Sync + ?Sized),
+    _chunks: &[String],
+    tombstones: &[u8],
+    bm25_index: &BM25Index,
+    fusion: Fusion,
+    k: usize,
 ) -> Vec<(usize, f32)> {
     // Get raw scores from both systems. Request all results so normalization
     // covers the full distribution.
@@ -42,12 +69,12 @@ pub fn hybrid_search(
     let vec_scores = {
         #[cfg(feature = "parallel")]
         {
-            if vectors.len() > 10_000 {
+            if vectors.count() > 10_000 {
                 vector_search::parallel_cosine_batch(
                     query_embedding,
                     vectors,
                     tombstones,
-                    vectors.len(),
+                    vectors.count(),
                 )
             } else {
                 vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
@@ -60,7 +87,7 @@ pub fn hybrid_search(
     };
     let kw_scores = bm25_index.scores(query_text);
 
-    merge_vector_keyword(vec_scores, kw_scores, vector_weight, keyword_weight, k)
+    fuse(vec_scores, kw_scores, fusion, k)
 }
 
 /// Merge pre-computed vector-similarity and keyword scores into a single ranking.
@@ -76,18 +103,92 @@ pub fn merge_vector_keyword(
     keyword_weight: f32,
     k: usize,
 ) -> Vec<(usize, f32)> {
-    // Normalize each set to [0, 1].
-    let vec_normalized = normalize_scores(&vec_scores);
-    let kw_normalized = normalize_scores(&kw_scores);
+    fuse(
+        vec_scores,
+        kw_scores,
+        Fusion::Weighted {
+            vector: vector_weight,
+            keyword: keyword_weight,
+        },
+        k,
+    )
+}
 
-    // Merge scores with weights.
-    let mut merged: HashMap<usize, f32> = HashMap::new();
+/// How the vector and keyword stages are combined into one ranking.
+#[derive(Debug, Clone, Copy, PartialEq)]
+pub enum Fusion {
+    /// Min-max normalise each stage over its own candidates, then take a
+    /// weighted sum. Uses the *scores*, so a stage that separates its
+    /// candidates sharply keeps that separation — and a stage whose candidates
+    /// are all near-identical contributes little.
+    Weighted {
+        /// Weight on the vector stage.
+        vector: f32,
+        /// Weight on the keyword stage.
+        keyword: f32,
+    },
+    /// Reciprocal rank fusion: each stage contributes `1 / (k + rank)`,
+    /// ignoring score magnitudes entirely. Robust when the two stages'
+    /// scores aren't comparable, at the cost of discarding confidence.
+    Rrf {
+        /// The rank-damping constant; 60 is the value from the original paper.
+        k: f32,
+    },
+}
 
-    for (idx, score) in &vec_normalized {
-        *merged.entry(*idx).or_insert(0.0) += vector_weight * score;
+impl Default for Fusion {
+    fn default() -> Self {
+        DEFAULT_FUSION
+    }
+}
+
+/// The fusion `hybrid_search` uses unless told otherwise.
+///
+/// The weights are not a guess: a sweep of every 0.1 step over the full
+/// LongMemEval haystack (500 questions, real MiniLM embeddings) found the
+/// long-standing 0.7/0.3 default *strictly dominated* — 0.4/0.6 is better at
+/// Hit@1, Hit@5, Hit@10 and MRR, at both turn and session granularity. See
+/// `BENCHMARKS.md`, "Weight sweep".
+pub const DEFAULT_FUSION: Fusion = Fusion::Weighted {
+    vector: 0.4,
+    keyword: 0.6,
+};
+
+/// Combine one ranked candidate list from each stage into a single top-`k`.
+///
+/// Neither list need be sorted; both are consumed.
+pub fn fuse(
+    vec_scores: Vec<(usize, f32)>,
+    kw_scores: Vec<(usize, f32)>,
+    fusion: Fusion,
+    k: usize,
+) -> Vec<(usize, f32)> {
+    let mut merged: HashMap<usize, f32> = HashMap::new();
+    match fusion {
+        Fusion::Weighted { vector, keyword } => {
+            // Normalize each set to [0, 1].
+            for (idx, score) in &normalize_scores(&vec_scores) {
+                *merged.entry(*idx).or_insert(0.0) += vector * score;
+            }
+            for (idx, score) in &normalize_scores(&kw_scores) {
+                *merged.entry(*idx).or_insert(0.0) += keyword * score;
+            }
+        }
+        Fusion::Rrf { k: damping } => {
+            for mut stage in [vec_scores, kw_scores] {
+                // Rank 1 is the best score. Ties break by index so a stage's
+                // contribution doesn't depend on the candidate order it
+                // happened to be produced in.
+                stage.sort_by(|a, b| {
+                    b.1.partial_cmp(&a.1)
+                        .unwrap_or(std::cmp::Ordering::Equal)
+                        .then(a.0.cmp(&b.0))
+                });
+                for (rank, (idx, _)) in stage.iter().enumerate() {
+                    *merged.entry(*idx).or_insert(0.0) += 1.0 / (damping + (rank + 1) as f32);
+                }
+            }
         }
-    for (idx, score) in &kw_normalized {
-        *merged.entry(*idx).or_insert(0.0) += keyword_weight * score;
     }
 
     let mut results: Vec<(usize, f32)> = merged.into_iter().collect();
@@ -169,7 +270,7 @@ fn normalize_scores(scores: &[(usize, f32)]) -> Vec<(usize, f32)> {
 pub fn rrf_hybrid_search(
     query_embedding: &[f32],
     query_text: &str,
-    vectors: &[Vec<f32>],
+    vectors: &(impl crate::vector_search::VectorSet + Sync + ?Sized),
     _chunks: &[String],
     tombstones: &[u8],
     bm25_index: &BM25Index,
@@ -181,12 +282,12 @@ pub fn rrf_hybrid_search(
     let mut vec_scores = {
         #[cfg(feature = "parallel")]
         {
-            if vectors.len() > 10_000 {
+            if vectors.count() > 10_000 {
                 vector_search::parallel_cosine_batch(
                     query_embedding,
                     vectors,
                     tombstones,
-                    vectors.len(),
+                    vectors.count(),
                 )
             } else {
                 vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
@@ -197,7 +298,7 @@ pub fn rrf_hybrid_search(
             vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
         }
     };
-    let mut kw_scores = bm25_index.search(query_text, vectors.len());
+    let mut kw_scores = bm25_index.search(query_text, vectors.count());
 
     // Sort both lists descending so rank 1 = best.
     vec_scores.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
@@ -353,6 +454,49 @@ mod tests {
         assert_eq!(result[0].1, 1.0);
     }
 
+    #[test]
+    fn default_fusion_is_the_tuned_operating_point() {
+        // A sweep over the full LongMemEval haystack found 0.7/0.3 strictly
+        // dominated by 0.4/0.6 (BENCHMARKS.md). This guards the finding
+        // against being quietly undone.
+        assert_eq!(
+            DEFAULT_FUSION,
+            Fusion::Weighted {
+                vector: 0.4,
+                keyword: 0.6
+            }
+        );
+    }
+
+    #[test]
+    fn rrf_rewards_agreement_between_the_stages_and_ignores_magnitudes() {
+        // Doc 1 is second-best in both stages; doc 0 is best in one and absent
+        // from the other. RRF prefers the doc both stages liked.
+        let vec_scores = vec![(0, 100.0), (1, 0.9)];
+        let kw_scores = vec![(2, 5.0), (1, 4.9)];
+        let ranked = fuse(vec_scores, kw_scores, Fusion::Rrf { k: 60.0 }, 3);
+        assert_eq!(ranked[0].0, 1, "{ranked:?}");
+
+        // Scaling one stage's scores cannot change an RRF ranking, only the
+        // order within that stage can.
+        let a = fuse(
+            vec![(0, 1.0), (1, 0.5)],
+            vec![(1, 2.0), (0, 1.0)],
+            Fusion::Rrf { k: 60.0 },
+            2,
+        );
+        let b = fuse(
+            vec![(0, 1e6), (1, -3.0)],
+            vec![(1, 0.002), (0, 0.001)],
+            Fusion::Rrf { k: 60.0 },
+            2,
+        );
+        assert_eq!(
+            a.iter().map(|r| r.0).collect::<Vec<_>>(),
+            b.iter().map(|r| r.0).collect::<Vec<_>>()
+        );
+    }
+
     #[test]
     fn merge_top_k_matches_a_full_sort() {
         // Many ties (scores repeat) so the index tie-break is exercised.

crates/clawhdf5-agent/src/lib.rs

@@ -62,15 +62,9 @@ use std::path::{Path, PathBuf};
 
 use cache::MemoryCache;
 #[cfg(feature = "hnsw")]
-use clawhdf5_ann::{DistanceMetric, HnswIndex};
+use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};
 use ephemeral::{EphemeralConfig, EphemeralStore};
 
-/// HNSW construction parameters used for the agent's vector index. Cosine is the
-/// agent's similarity metric, so the index is built with cosine distance.
-#[cfg(feature = "hnsw")]
-const HNSW_M: usize = 16;
-#[cfg(feature = "hnsw")]
-const HNSW_EF_CONSTRUCTION: usize = 64;
 // EphemeralEntry and EphemeralStats are part of the crate public API via
 // the `ephemeral` module; they are not needed directly in lib.rs internals.
 #[allow(unused_imports)]
@@ -139,6 +133,34 @@ pub struct MemoryConfig {
     pub created_at: String,
     pub wal_enabled: bool,
     pub wal_max_entries: usize,
+    /// Store the vector index's own copy of the embeddings as int8 rather than
+    /// f32, a quarter of the memory.
+    ///
+    /// The index's copy is the single largest part of a loaded store's
+    /// footprint. Quantised distances are approximate, so the candidate pool
+    /// is re-scored against the cache's exact embeddings before fusion, which
+    /// restores recall; what it costs is throughput — roughly 13% of queries
+    /// per second and 16% of build time at 100K x 384. See `BENCHMARKS.md`.
+    ///
+    /// Has no effect without the `hnsw` feature.
+    pub quantized_index: bool,
+    /// HNSW graph degree. Higher means a denser graph: better recall, more
+    /// memory and slower builds. Clamped to at least 2 when the index is
+    /// built, since a graph with fewer connections is not one.
+    ///
+    /// Has no effect without the `hnsw` feature.
+    pub hnsw_m: usize,
+    /// Candidate list size while building the HNSW graph. Higher means a
+    /// better graph and a slower build; it does not affect query cost.
+    ///
+    /// Has no effect without the `hnsw` feature.
+    pub hnsw_ef_construction: usize,
+    /// Candidate list size for a query, trading throughput for recall. `0`
+    /// keeps the default, which scales with the requested `k`
+    /// (`max(k * 8, 64)`) so that fusion still sees a useful pool.
+    ///
+    /// Has no effect without the `hnsw` feature.
+    pub hnsw_ef_search: usize,
 }
 
 impl MemoryConfig {
@@ -160,6 +182,10 @@ impl MemoryConfig {
             created_at,
             wal_enabled: true,
             wal_max_entries: 500,
+            quantized_index: false,
+            hnsw_m: 16,
+            hnsw_ef_construction: 64,
+            hnsw_ef_search: 0,
         }
     }
 }
@@ -258,6 +284,9 @@ pub struct HDF5Memory {
     /// every record, on every single query. Built lazily on first use; see
     /// [`HDF5Memory::ensure_bm25_fresh`] for how it stays in sync.
     bm25: Option<bm25::BM25Index>,
+    /// Token filter the keyword index is built with. Changing it drops the
+    /// index; it is not persisted, because the index is not either.
+    bm25_filter: bm25::TokenFilter,
     /// Activation weights changed since the last checkpoint (searches boost
     /// the records they return). Cleared by `flush`.
     activations_dirty: bool,
@@ -314,6 +343,7 @@ impl HDF5Memory {
             anomaly: anomaly::WriteAnomalyDetector::new(anomaly::AnomalyConfig::default()),
             anomaly_alerts: Vec::new(),
             bm25: None,
+            bm25_filter: bm25::TokenFilter::default(),
             activations_dirty: false,
             read_only: false,
             quarantined_wal: None,
@@ -440,7 +470,17 @@ impl HDF5Memory {
 
         #[cfg(feature = "hnsw")]
         let loaded_index = if replay_only_appended {
-            Self::load_vector_index(path, checkpoint.ann_generation, &cache, n_checkpoint)
+            Self::load_vector_index(
+                path,
+                checkpoint.ann_generation,
+                &cache,
+                n_checkpoint,
+                if config.quantized_index {
+                    Storage::Int8
+                } else {
+                    Storage::Float32
+                },
+            )
         } else {
             None
         };
@@ -481,6 +521,7 @@ impl HDF5Memory {
             anomaly: anomaly::WriteAnomalyDetector::new(anomaly::AnomalyConfig::default()),
             anomaly_alerts: Vec::new(),
             bm25: None,
+            bm25_filter: bm25::TokenFilter::default(),
             activations_dirty: false,
             read_only,
             quarantined_wal,
@@ -553,6 +594,7 @@ impl HDF5Memory {
         generation: Option<u64>,
         cache: &MemoryCache,
         n_checkpoint: usize,
+        storage: Storage,
     ) -> Option<HnswIndex> {
         let generation = generation?;
         let bytes = std::fs::read(Self::vector_index_path(store)).ok()?;
@@ -560,15 +602,17 @@ impl HDF5Memory {
         if u64::from_le_bytes(stamp.try_into().ok()?) != generation {
             return None;
         }
-        let vectors = cache.embeddings.get(..n_checkpoint)?.to_vec();
-        let mut index = HnswIndex::from_graph_bytes(graph, vectors).ok()?;
+        let vectors: Vec<Vec<f32>> = (0..n_checkpoint)
+            .map(|i| cache.embeddings.get(i).map(<[f32]>::to_vec))
+            .collect::<Option<_>>()?;
+        let mut index = HnswIndex::from_graph_bytes_with(graph, vectors, storage).ok()?;
         if index.dimension() != cache.embedding_dim {
             return None;
         }
         // Records appended since (replayed from the WAL) join incrementally.
         for id in n_checkpoint..cache.embeddings.len() {
             if cache.embeddings[id].len() != index.dimension()
-                || index.insert(cache.embeddings[id].clone()) != id
+                || index.insert(cache.embeddings[id].to_vec()) != id
             {
                 return None;
             }
@@ -591,7 +635,7 @@ impl HDF5Memory {
     pub(crate) fn ensure_bm25_fresh(&mut self) -> &bm25::BM25Index {
         let n = self.cache.chunks.len();
         let bm25 = match self.bm25.take() {
-            Some(index) if index.len() <= n => {
+            Some(index) if index.len() <= n && index.token_filter() == self.bm25_filter => {
                 let mut index = index;
                 for id in index.len()..n {
                     if self.cache.tombstones[id] == 0 {
@@ -601,11 +645,26 @@ impl HDF5Memory {
                 index.pad_to(n);
                 index
             }
-            _ => bm25::BM25Index::build(&self.cache.chunks, &self.cache.tombstones),
+            _ => bm25::BM25Index::build_with(
+                &self.cache.chunks,
+                &self.cache.tombstones,
+                self.bm25_filter,
+            ),
         };
         self.bm25.insert(bm25)
     }
 
+    /// Choose how keyword-search tokens are normalised, rebuilding the index
+    /// on next use. [`bm25::TokenFilter::Stemmed`] matches inflections of the
+    /// same word at some cost in precision; measure before adopting it (see
+    /// `BENCHMARKS.md`).
+    pub fn set_token_filter(&mut self, filter: bm25::TokenFilter) {
+        if filter != self.bm25_filter {
+            self.bm25_filter = filter;
+            self.bm25 = None;
+        }
+    }
+
     /// Record `id` was tombstoned; its text is still in the cache.
     fn bm25_on_delete(&mut self, id: usize) {
         if let Some(index) = self.bm25.as_mut()
@@ -781,6 +840,42 @@ impl HDF5Memory {
     // the index length drifts from the cache length (covering any mutation path
     // that doesn't call a hook, e.g. consolidation pushes).
 
+    /// Graph degree for the index, never below the 2 the builder requires:
+    /// a config value of 0 or 1 would otherwise panic inside `clawhdf5-ann`.
+    #[cfg(feature = "hnsw")]
+    fn hnsw_m(&self) -> usize {
+        self.config.hnsw_m.max(2)
+    }
+
+    /// Build-time candidate list size, never below the graph degree — a
+    /// smaller one cannot fill a node's connections.
+    #[cfg(feature = "hnsw")]
+    fn hnsw_ef_construction(&self) -> usize {
+        self.config.hnsw_ef_construction.max(self.hnsw_m())
+    }
+
+    /// Query-time candidate list size for a `k`-result search. `0` means the
+    /// default, which scales with `k`.
+    #[cfg(feature = "hnsw")]
+    pub(crate) fn hnsw_ef_search(&self, k: usize) -> usize {
+        let default = (k * 8).max(64);
+        if self.config.hnsw_ef_search == 0 {
+            default
+        } else {
+            self.config.hnsw_ef_search.max(k)
+        }
+    }
+
+    /// How the index should store its copy of the vectors, per the config.
+    #[cfg(feature = "hnsw")]
+    fn index_storage(&self) -> Storage {
+        if self.config.quantized_index {
+            Storage::Int8
+        } else {
+            Storage::Float32
+        }
+    }
+
     /// Build an HNSW index over the entire cache, re-applying tombstones as
     /// soft-deletions so node ids stay aligned with cache indices.
     ///
@@ -796,11 +891,15 @@ impl HDF5Memory {
         if self.cache.embeddings.iter().any(|e| e.len() != dim) {
             return None;
         }
-        let mut index = HnswIndex::build_with_metric(
-            &self.cache.embeddings,
-            HNSW_M,
-            HNSW_EF_CONSTRUCTION,
+        // The index owns its vectors, so it needs rows rather than the cache's
+        // flat buffer. This copy is the index's own; the cache keeps one.
+        let rows: Vec<Vec<f32>> = self.cache.embeddings.iter().map(<[f32]>::to_vec).collect();
+        let mut index = HnswIndex::build_with(
+            &rows,
+            self.hnsw_m(),
+            self.hnsw_ef_construction(),
             DistanceMetric::Cosine,
+            self.index_storage(),
         );
         for (i, &t) in self.cache.tombstones.iter().enumerate() {
             if t != 0 {
@@ -826,7 +925,7 @@ impl HDF5Memory {
             let dim = index.dimension();
             let appended = (self.hnsw_synced_len..n).all(|id| {
                 self.cache.embeddings[id].len() == dim
-                    && index.insert(self.cache.embeddings[id].clone()) == id
+                    && index.insert(self.cache.embeddings[id].to_vec()) == id
             });
             if appended {
                 for id in self.hnsw_synced_len..n {
@@ -857,7 +956,7 @@ impl HDF5Memory {
         let emb_len = self.cache.embeddings[idx].len();
         match self.hnsw.as_mut() {
             Some(index) if emb_len == index.dimension() => {
-                let id = index.insert(self.cache.embeddings[idx].clone());
+                let id = index.insert(self.cache.embeddings[idx].to_vec());
                 if id == idx {
                     self.hnsw_synced_len = self.cache.embeddings.len();
                 } else {
@@ -1350,7 +1449,8 @@ impl HDF5Memory {
         k: usize,
     ) -> Vec<SearchResult> {
         // Persistent tier.
-        let persistent = self.hybrid_search(query_embedding, query_text, 0.7, 0.3, k);
+        let persistent =
+            self.hybrid_search_with(query_embedding, query_text, hybrid::DEFAULT_FUSION, k);
         const EPHEMERAL_BOOST: f32 = 1.2;
         let mut results = persistent;
 
@@ -1954,6 +2054,34 @@ mod tests {
         assert_eq!(top_ids(&mut reopened, &q), expected_after);
     }
 
+    #[test]
+    fn set_token_filter_rebuilds_the_keyword_index() {
+        let dir = TempDir::new().unwrap();
+        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
+        mem.save(make_entry(
+            "I was training for a marathon",
+            &[1.0, 0.0, 0.0, 0.0],
+        ))
+        .unwrap();
+
+        // Count only genuine keyword matches: `hybrid_search` also returns
+        // zero-score filler when fewer than k records are relevant.
+        let hits = |mem: &mut HDF5Memory| {
+            mem.hybrid_search(&[0.0, 0.0, 0.0, 0.0], "trains", 0.0, 1.0, 5)
+                .iter()
+                .filter(|r| r.score > 0.0)
+                .count()
+        };
+        assert_eq!(hits(&mut mem), 0);
+
+        mem.set_token_filter(bm25::TokenFilter::Stemmed);
+        assert_eq!(hits(&mut mem), 1, "index should have been rebuilt stemmed");
+
+        // And back, rebuilding again.
+        mem.set_token_filter(bm25::TokenFilter::Plain);
+        assert_eq!(hits(&mut mem), 0);
+    }
+
     #[test]
     fn keyword_index_stays_in_sync_through_every_mutation() {
         let dir = TempDir::new().unwrap();

crates/clawhdf5-agent/src/openclaw.rs

@@ -466,7 +466,7 @@ impl ClawhdfBackend {
             let record = MemoryRecord {
                 id: i as u64,
                 chunk: cache.chunks[i].clone(),
-                embedding: cache.embeddings[i].clone(),
+                embedding: cache.embeddings[i].to_vec(),
                 tier: MemoryTier::Working,
                 importance: cache.activation_weights[i],
                 access_count: 0,
@@ -531,11 +531,14 @@ impl MemoryBackend for ClawhdfBackend {
         query_embedding: &[f32],
         k: usize,
     ) -> Vec<MemorySearchResult> {
-        // 1. Hybrid retrieval (RRF-blended vector + BM25).
+        // 1. Hybrid retrieval (vector + BM25, fused by score).
         let candidates = k.saturating_mul(3).max(10);
-        let raw = self
-            .memory
-            .hybrid_search(query_embedding, query_text, 0.7, 0.3, candidates);
+        let raw = self.memory.hybrid_search_with(
+            query_embedding,
+            query_text,
+            crate::hybrid::DEFAULT_FUSION,
+            candidates,
+        );
 
         if raw.is_empty() {
             return Vec::new();
@@ -551,6 +554,7 @@ impl MemoryBackend for ClawhdfBackend {
                 timestamp: r.timestamp,
                 source_channel: r.source_channel.clone(),
                 raw_activation: r.activation,
+                relevance: r.score,
             })
             .collect();
 
@@ -713,11 +717,13 @@ impl MemoryBackend for ClawhdfBackend {
 
         let total_records = cache.count_active();
 
+        // A record saved without an embedding occupies a zero row, so "has an
+        // embedding" is "has a non-zero norm" rather than "row is non-empty".
         let total_embeddings = cache
-            .embeddings
+            .norms
             .iter()
             .enumerate()
-            .filter(|(i, emb)| cache.tombstones[*i] == 0 && !emb.is_empty())
+            .filter(|(i, norm)| cache.tombstones[*i] == 0 && **norm > 0.0)
             .count();
 
         let file_size_bytes = std::fs::metadata(&self.hdf5_path)

crates/clawhdf5-agent/src/query_expand.rs

@@ -6,6 +6,11 @@
 //! - Temporal expansion (time-related rewrites)
 //! - Morphological variants (stemming-like transforms)
 //! - Knowledge graph expansion (entity aliases and neighbors)
+//!
+//! The morphological rules are crude suffix swaps, so some variants are not
+//! words ("during" -> "dured"). That is tolerable for a BM25 stage, which
+//! simply finds no postings for a nonsense term, but it means expansion is not
+//! free: measure before enabling it on a retrieval path.
 
 use crate::knowledge::KnowledgeCache;
 
@@ -340,18 +345,85 @@ fn contains_phrase(text: &str, phrase: &str) -> bool {
 
 /// Replace a phrase in `text` case-insensitively, preserving surrounding case.
 fn replace_word_case_insensitive(text: &str, from: &str, to: &str) -> String {
-    case_insensitive_replace(text, from, to)
+    replace_first(text, from, to, MatchKind::WholeWord)
 }
 
 fn case_insensitive_replace(text: &str, from: &str, to: &str) -> String {
-    let lower = text.to_lowercase();
-    let lower_from = from.to_lowercase();
-    if let Some(pos) = lower.find(&lower_from) {
-        let end = pos + from.len();
-        format!("{}{}{}", &text[..pos], to, &text[end..])
-    } else {
-        text.to_string()
+    replace_first(text, from, to, MatchKind::Substring)
+}
+
+/// Whether a match may fall inside a larger word.
+#[derive(Clone, Copy, PartialEq)]
+enum MatchKind {
+    /// Match anywhere, including inside another word.
+    Substring,
+    /// Match only when both ends sit on a word boundary.
+    WholeWord,
+}
+
+/// Replace the first case-insensitive match of `from` in `text` with `to`.
+///
+/// Matching walks the *original* string rather than a lowercased copy. The
+/// previous implementation searched `text.to_lowercase()` and then sliced
+/// `text` with the offsets it found, which only holds while lowercasing
+/// preserves byte length. It does not: Turkish `İ` (2 bytes) lowercases to
+/// `i` + U+0307 (3 bytes), so every later offset was wrong — silently
+/// corrupting the output, or panicking when an offset landed inside a
+/// character or past the end. `"İ AI"` was enough to panic.
+fn replace_first(text: &str, from: &str, to: &str, kind: MatchKind) -> String {
+    match find_case_insensitive(text, from, kind) {
+        Some((start, end)) => {
+            let mut out = String::with_capacity(text.len() - (end - start) + to.len());
+            out.push_str(&text[..start]);
+            out.push_str(to);
+            out.push_str(&text[end..]);
+            out
         }
+        None => text.to_string(),
+    }
+}
+
+/// Byte range of the first case-insensitive match of `needle` in `haystack`.
+fn find_case_insensitive(haystack: &str, needle: &str, kind: MatchKind) -> Option<(usize, usize)> {
+    if needle.is_empty() {
+        return None;
+    }
+    let lowered: Vec<char> = needle.chars().flat_map(char::to_lowercase).collect();
+    let is_word = |c: char| c.is_alphanumeric() || c == '_';
+
+    for (start, _) in haystack.char_indices() {
+        if kind == MatchKind::WholeWord
+            && haystack[..start].chars().next_back().is_some_and(is_word)
+        {
+            continue; // mid-word: "ai" inside "training"
+        }
+        let mut matched = 0usize;
+        let mut end = start;
+        for (offset, ch) in haystack[start..].char_indices() {
+            if matched == lowered.len() {
+                break;
+            }
+            let mut consumed_all = true;
+            for lc in ch.to_lowercase() {
+                if lowered.get(matched) != Some(&lc) {
+                    consumed_all = false;
+                    break;
+                }
+                matched += 1;
+            }
+            if !consumed_all {
+                break;
+            }
+            end = start + offset + ch.len_utf8();
+        }
+        if matched == lowered.len()
+            && !(kind == MatchKind::WholeWord
+                && haystack[end..].chars().next().is_some_and(is_word))
+        {
+            return Some((start, end));
+        }
+    }
+    None
 }
 
 /// Simple whitespace/punctuation tokenizer.
@@ -637,4 +709,86 @@ mod tests {
             expanded.iter().map(|x| &x.text).collect::<Vec<_>>()
         );
     }
+    #[test]
+    fn acronyms_only_match_whole_words() {
+        let ex = QueryExpander::new(QueryExpansionConfig::default());
+        // "training" contains "ai", "programming" contains "pr". These used to
+        // be rewritten to "trArtificial Intelligencening" and
+        // "Pull Requestogramming".
+        for query in [
+            "How many miles during my marathon training?",
+            "Which programming language did I pick?",
+            "I updated the maintainer list",
+        ] {
+            for expansion in ex.expand(query) {
+                assert!(
+                    expansion.expansion_type != "acronym",
+                    "{query:?} produced {expansion:?}"
+                );
+            }
+        }
+        // A real acronym still expands, in both directions.
+        let texts: Vec<String> = ex
+            .expand("What about the API and the database?")
+            .into_iter()
+            .filter(|e| e.expansion_type == "acronym")
+            .map(|e| e.text)
+            .collect();
+        assert!(
+            texts
+                .iter()
+                .any(|t| t.contains("Application Programming Interface")),
+            "{texts:?}"
+        );
+        assert!(texts.iter().any(|t| t.contains("DB")), "{texts:?}");
+    }
+
+    #[test]
+    fn non_ascii_queries_do_not_panic_or_corrupt() {
+        let ex = QueryExpander::new(QueryExpansionConfig::default());
+        // Turkish 'İ' is 2 bytes but lowercases to 3, so offsets taken from a
+        // lowercased copy no longer line up with the original. `"İ AI"` used
+        // to panic; `"İstanbul AI trip"` used to silently eat a character.
+        for query in ["İ AI", "İé AI", "İİ ML", "İstanbul AI trip", "ǰ ML notes"] {
+            for expansion in ex.expand(query) {
+                assert!(
+                    expansion.text.contains('İ') || expansion.text.contains('ǰ'),
+                    "{query:?} lost its leading character: {expansion:?}"
+                );
+            }
+        }
+        let expanded = ex.expand("İstanbul AI trip");
+        assert!(
+            expanded
+                .iter()
+                .any(|e| e.text == "İstanbul Artificial Intelligence trip"),
+            "{expanded:?}"
+        );
+    }
+
+    #[test]
+    fn whole_word_matching_handles_string_edges_and_case() {
+        assert_eq!(
+            replace_word_case_insensitive("ai tools", "AI", "Artificial Intelligence"),
+            "Artificial Intelligence tools"
+        );
+        assert_eq!(
+            replace_word_case_insensitive("tools for ai", "AI", "Artificial Intelligence"),
+            "tools for Artificial Intelligence"
+        );
+        assert_eq!(
+            replace_word_case_insensitive("the aim", "AI", "Artificial Intelligence"),
+            "the aim",
+            "must not match inside a word"
+        );
+        assert_eq!(
+            replace_word_case_insensitive("no match here", "xyz", "abc"),
+            "no match here"
+        );
+        // Only the first occurrence is replaced, as before.
+        assert_eq!(
+            replace_word_case_insensitive("ai and ai", "ai", "ML"),
+            "ML and ai"
+        );
+    }
 }

crates/clawhdf5-agent/src/reranker.rs

@@ -4,8 +4,10 @@
 //! into a single composite score for each retrieved result.
 
 /// Configuration for the multi-factor re-ranker.
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, Copy)]
 pub struct ReRankConfig {
+    /// Weight applied to the retrieval score the candidate arrived with.
+    pub relevance_weight: f32,
     /// Weight applied to the temporal decay score (0.0–1.0).
     pub temporal_weight: f32,
     /// Weight applied to the source authority score (0.0–1.0).
@@ -20,6 +22,9 @@ pub struct ReRankConfig {
 impl Default for ReRankConfig {
     fn default() -> Self {
         Self {
+            // Relevance leads: the metadata signals break ties and nudge, they
+            // do not decide. See `BENCHMARKS.md`, "Recency discrimination".
+            relevance_weight: 1.0,
             temporal_weight: 0.3,
             authority_weight: 0.2,
             activation_weight: 0.5,
@@ -41,6 +46,8 @@ pub struct ReRankResult {
     pub authority_score: f32,
     /// Normalised Hebbian activation score in [0, 1].
     pub activation_score: f32,
+    /// The retrieval score carried through from the input.
+    pub relevance_score: f32,
 }
 
 /// Compute an exponential decay temporal score.
@@ -105,6 +112,15 @@ pub struct RerankInput {
     pub source_channel: String,
     /// Raw Hebbian activation weight for this entry.
     pub raw_activation: f32,
+    /// The retrieval score that put this entry in the candidate list.
+    ///
+    /// Re-ranking is meant to *adjust* the retriever's ordering with signals
+    /// it does not have, not to replace it. Without this the combined score
+    /// was made of recency, authority and activation alone, so a candidate
+    /// pool came back ordered by age with its relevance ordering discarded.
+    /// Callers with no meaningful score can pass the same value for every
+    /// entry, which reduces to the old behaviour.
+    pub relevance: f32,
 }
 
 /// Re-rank a list of retrieval results using multi-factor scoring.
@@ -138,7 +154,8 @@ pub fn rerank(
             let auth = source_authority_score(&inp.source_channel);
             let act = activation_score(inp.raw_activation);
 
-            let combined = config.temporal_weight * ts
+            let combined = config.relevance_weight * inp.relevance
+                + config.temporal_weight * ts
                 + config.authority_weight * auth
                 + config.activation_weight * act;
 
@@ -148,6 +165,7 @@ pub fn rerank(
                 temporal_score: ts,
                 authority_score: auth,
                 activation_score: act,
+                relevance_score: inp.relevance,
             }
         })
         .collect();
@@ -253,22 +271,51 @@ mod tests {
                 timestamp: 0.0, // very old
                 source_channel: "other".to_string(),
                 raw_activation: 0.1,
+                relevance: 0.0,
             },
             RerankInput {
                 index: 1,
                 timestamp: 86_400.0, // one day ago
                 source_channel: "conversation".to_string(),
                 raw_activation: 0.5,
+                relevance: 0.0,
             },
             RerankInput {
                 index: 2,
                 timestamp: 172_800.0, // "now"
                 source_channel: "user_correction".to_string(),
                 raw_activation: 1.0,
+                relevance: 0.0,
             },
         ]
     }
 
+    #[test]
+    fn relevance_leads_but_recency_breaks_near_ties() {
+        let entry = |index, timestamp, relevance| RerankInput {
+            index,
+            timestamp,
+            source_channel: "conversation".to_string(),
+            raw_activation: 1.0,
+            relevance,
+        };
+        let now = 10.0 * 86_400.0;
+        let config = ReRankConfig::default();
+
+        // A clearly better match wins despite being much older. Before
+        // `relevance` existed the combined score ignored it entirely, so this
+        // returned the newer, irrelevant entry.
+        let ranked = rerank(&[entry(0, 0.0, 1.0), entry(1, now, 0.1)], &config, now);
+        assert_eq!(ranked[0].index, 0, "{ranked:?}");
+
+        // Between near-equal matches, the newer one wins.
+        let ranked = rerank(&[entry(0, 0.0, 0.80), entry(1, now, 0.79)], &config, now);
+        assert_eq!(ranked[0].index, 1, "{ranked:?}");
+
+        // The breakdown carries the relevance through.
+        assert_eq!(ranked[0].relevance_score, 0.79);
+    }
+
     #[test]
     fn rerank_returns_all_entries() {
         let inputs = make_inputs();
@@ -302,6 +349,7 @@ mod tests {
     #[test]
     fn rerank_score_breakdown_matches_manual_calculation() {
         let config = ReRankConfig {
+            relevance_weight: 0.0,
             temporal_weight: 1.0,
             authority_weight: 0.0,
             activation_weight: 0.0,
@@ -312,6 +360,7 @@ mod tests {
             timestamp: 0.0,
             source_channel: "other".to_string(),
             raw_activation: 0.5,
+            relevance: 0.0,
         }];
         let now = 3600.0_f64; // exactly one half-life later
         let results = rerank(&inputs, &config, now);

crates/clawhdf5-agent/src/schema.rs

@@ -104,6 +104,19 @@ pub fn build_hdf5_file_with_meta(
         "wal_max_entries",
         AttrValue::I64(config.wal_max_entries as i64),
     );
+    meta.set_attr(
+        "quantized_index",
+        AttrValue::I64(config.quantized_index.into()),
+    );
+    meta.set_attr("hnsw_m", AttrValue::I64(config.hnsw_m as i64));
+    meta.set_attr(
+        "hnsw_ef_construction",
+        AttrValue::I64(config.hnsw_ef_construction as i64),
+    );
+    meta.set_attr(
+        "hnsw_ef_search",
+        AttrValue::I64(config.hnsw_ef_search as i64),
+    );
     meta.set_attr(
         "edgehdf5_version",
         AttrValue::String(ZEROCLAW_VERSION.into()),
@@ -153,7 +166,7 @@ fn build_memory_group(
     {
         let ds = group
             .create_dataset("embeddings")
-            .with_f32_data(&flat)
+            .with_f32_data(flat)
             .with_shape(&[n, d]);
 
         // Chunk size tuning: target ~256KB per chunk for optimal I/O
@@ -484,6 +497,16 @@ pub fn validate_and_load(
         wal_max_entries: optional_i64_attr(&attrs, "wal_max_entries")
             .and_then(|v| usize::try_from(v).ok())
             .unwrap_or(500),
+        quantized_index: optional_bool_attr(&attrs, "quantized_index", false),
+        hnsw_m: optional_i64_attr(&attrs, "hnsw_m")
+            .and_then(|v| usize::try_from(v).ok())
+            .unwrap_or(16),
+        hnsw_ef_construction: optional_i64_attr(&attrs, "hnsw_ef_construction")
+            .and_then(|v| usize::try_from(v).ok())
+            .unwrap_or(64),
+        hnsw_ef_search: optional_i64_attr(&attrs, "hnsw_ef_search")
+            .and_then(|v| usize::try_from(v).ok())
+            .unwrap_or(0),
     };
 
     // Load /memory group
@@ -563,12 +586,7 @@ fn load_memory_group(
             .collect(),
     };
 
-    // Unflatten embeddings
-    let embeddings: Vec<Vec<f32>> = flat_embeddings
-        .chunks(embedding_dim)
-        .map(|c| c.to_vec())
-        .collect();
-
+    // No unflattening: the cache stores the buffer as it is on disk.
     // Read activation_weights if present, default to vec![1.0; N] for backward compat
     let activation_weights = match read_f32_dataset(&group, "activation_weights") {
         Ok(w) if w.len() == n => w,
@@ -576,7 +594,7 @@ fn load_memory_group(
     };
 
     cache.chunks = chunks;
-    cache.embeddings = embeddings;
+    cache.embeddings.set_flat(embedding_dim, flat_embeddings);
     cache.source_channels = source_channels;
     cache.timestamps = timestamps;
     cache.session_ids = session_ids;
@@ -584,7 +602,6 @@ fn load_memory_group(
     cache.tombstones = tombstones;
     cache.norms = norms;
     cache.activation_weights = activation_weights;
-    cache.rebuild_flat();
 
     Ok(cache)
 }

crates/clawhdf5-agent/src/search.rs

@@ -20,8 +20,7 @@ impl HDF5Memory {
         query_embedding: &[f32],
         query_text: &str,
         bm25: &bm25::BM25Index,
-        vector_weight: f32,
-        keyword_weight: f32,
+        fusion: hybrid::Fusion,
         k: usize,
     ) -> Vec<(usize, f32)> {
         self.ensure_hnsw_fresh();
@@ -29,32 +28,45 @@ impl HDF5Memory {
             Some(index) if !index.is_empty() && index.dimension() == query_embedding.len() => {
                 // Over-fetch so the merge sees a useful vector pool; cosine
                 // distance from the index converts back to similarity (1 - d).
+                // `ef` is configurable, but the pool the fusion stage sees is
+                // not tied to it: a caller lowering `ef` for speed should not
+                // silently narrow what fusion has to work with.
                 let pool = (k * 8).max(64);
-                let vec_scores: Vec<(usize, f32)> = index
-                    .search(query_embedding, pool, pool)
+                let ef = self.hnsw_ef_search(k).max(pool);
+                let candidates = index.search(query_embedding, pool, ef);
+                // A quantised index returns approximate distances, and no
+                // amount of `ef` fixes that — the loss is in the distances,
+                // not the graph. Re-score the pool against the cache's exact
+                // embeddings, which cost nothing extra to keep: recall then
+                // matches an f32 index. See `BENCHMARKS.md`.
+                let exact = index.storage() == clawhdf5_ann::Storage::Int8;
+                let vec_scores: Vec<(usize, f32)> = candidates
                     .into_iter()
-                    .map(|(id, dist)| (id, 1.0 - dist))
+                    .map(|(id, dist)| {
+                        let score = if exact {
+                            crate::vector_search::cosine_similarity(
+                                query_embedding,
+                                &self.cache.embeddings[id],
+                            )
+                        } else {
+                            1.0 - dist
+                        };
+                        (id, score)
+                    })
                     .collect();
                 // Fusion normalises over every keyword match, so it needs all
                 // the scores — but not ranked.
                 let kw_scores = bm25.scores(query_text);
-                hybrid::merge_vector_keyword(
-                    vec_scores,
-                    kw_scores,
-                    vector_weight,
-                    keyword_weight,
-                    k,
-                )
+                hybrid::fuse(vec_scores, kw_scores, fusion, k)
             }
-            _ => hybrid::hybrid_search(
+            _ => hybrid::hybrid_search_fused(
                 query_embedding,
                 query_text,
                 &self.cache.embeddings,
                 &self.cache.chunks,
                 &self.cache.tombstones,
                 bm25,
-                vector_weight,
-                keyword_weight,
+                fusion,
                 k,
             ),
         }
@@ -66,19 +78,17 @@ impl HDF5Memory {
         query_embedding: &[f32],
         query_text: &str,
         bm25: &bm25::BM25Index,
-        vector_weight: f32,
-        keyword_weight: f32,
+        fusion: hybrid::Fusion,
         k: usize,
     ) -> Vec<(usize, f32)> {
-        hybrid::hybrid_search(
+        hybrid::hybrid_search_fused(
             query_embedding,
             query_text,
             &self.cache.embeddings,
             &self.cache.chunks,
             &self.cache.tombstones,
             bm25,
-            vector_weight,
-            keyword_weight,
+            fusion,
             k,
         )
     }
@@ -91,20 +101,36 @@ impl HDF5Memory {
         vector_weight: f32,
         keyword_weight: f32,
         k: usize,
+    ) -> Vec<SearchResult> {
+        self.hybrid_search_with(
+            query_embedding,
+            query_text,
+            hybrid::Fusion::Weighted {
+                vector: vector_weight,
+                keyword: keyword_weight,
+            },
+            k,
+        )
+    }
+
+    /// [`HDF5Memory::hybrid_search`] with the fusion method chosen explicitly.
+    ///
+    /// [`hybrid::DEFAULT_FUSION`] is what the weighted form defaults to;
+    /// [`hybrid::Fusion::Rrf`] combines the two stages by rank instead of by
+    /// score.
+    pub fn hybrid_search_with(
+        &mut self,
+        query_embedding: &[f32],
+        query_text: &str,
+        fusion: hybrid::Fusion,
+        k: usize,
     ) -> Vec<SearchResult> {
         // The keyword index lives for the life of the store and is updated
         // incrementally. Take it out for the duration of the call so the
         // vector stage can borrow `self` mutably, then put it back.
         self.ensure_bm25_fresh();
         let bm25 = self.bm25.take().expect("ensure_bm25_fresh leaves an index");
-        let scored = self.vector_keyword_search(
-            query_embedding,
-            query_text,
-            &bm25,
-            vector_weight,
-            keyword_weight,
-            k,
-        );
+        let scored = self.vector_keyword_search(query_embedding, query_text, &bm25, fusion, k);
         let mut results: Vec<SearchResult> = scored
             .into_iter()
             .map(|(idx, score)| {

crates/clawhdf5-agent/src/storage.rs

@@ -113,13 +113,11 @@ pub type StoreState = (MemoryConfig, MemoryCache, SessionCache, KnowledgeCache);
 /// [`read_from_disk`], plus the checkpoint's [`WalMark`] (if any) so the
 /// caller can skip WAL entries this file already contains.
 pub fn read_from_disk_with_mark(path: &Path) -> Result<(StoreState, Option<WalMark>), MemoryError> {
-    let mmap = clawhdf5_io::MmapReader::open(path).map_err(MemoryError::Io)?;
-
-    // Advise the OS we'll need the whole file for parsing
-    mmap.advise_willneed(0, mmap.len());
-
-    // Parse the HDF5 file from the mmap'd bytes
-    let file = clawhdf5::File::from_bytes(mmap.as_bytes().to_vec())
+    // `File::open` memory-maps the file itself (the facade's `mmap` feature is
+    // on by default). Mapping it here and handing over `as_bytes().to_vec()`
+    // did the same work and then copied the whole store — a second full copy
+    // of the file, live for the whole parse, on top of the mapping.
+    let file = clawhdf5::File::open(path)
         .map_err(|e| MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
 
     let (mut config, cache, sessions, knowledge) = schema::validate_and_load(&file)?;
@@ -133,9 +131,7 @@ pub fn read_from_disk_with_mark(path: &Path) -> Result<(StoreState, Option<WalMa
 pub fn read_from_disk_with_meta(
     path: &Path,
 ) -> Result<(StoreState, schema::CheckpointMeta), MemoryError> {
-    let mmap = clawhdf5_io::MmapReader::open(path).map_err(MemoryError::Io)?;
-    mmap.advise_willneed(0, mmap.len());
-    let file = clawhdf5::File::from_bytes(mmap.as_bytes().to_vec())
+    let file = clawhdf5::File::open(path)
         .map_err(|e| MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
     let (mut config, cache, sessions, knowledge) = schema::validate_and_load(&file)?;
     config.path = path.to_path_buf();

crates/clawhdf5-agent/src/vector_search.rs

@@ -4,6 +4,44 @@
 //! `clawhdf5_accel`, with optional float16 support via the `half` crate.
 //! Supports pre-computed norms for eliminating redundant norm computations.
 
+/// A corpus of equal-length embeddings addressable by index.
+///
+/// Lets the batch kernels read either the cache's flat `[N x dim]` buffer or a
+/// plain `Vec<Vec<f32>>` without either side owning a second copy.
+pub trait VectorSet {
+    /// Number of embeddings.
+    fn count(&self) -> usize;
+    /// Embedding `i`; callers only index below [`VectorSet::count`].
+    fn row(&self, i: usize) -> &[f32];
+}
+
+impl VectorSet for [Vec<f32>] {
+    fn count(&self) -> usize {
+        self.len()
+    }
+    fn row(&self, i: usize) -> &[f32] {
+        &self[i]
+    }
+}
+
+impl VectorSet for Vec<Vec<f32>> {
+    fn count(&self) -> usize {
+        self.len()
+    }
+    fn row(&self, i: usize) -> &[f32] {
+        &self[i]
+    }
+}
+
+impl VectorSet for crate::cache::Embeddings {
+    fn count(&self) -> usize {
+        self.len()
+    }
+    fn row(&self, i: usize) -> &[f32] {
+        &self[i]
+    }
+}
+
 /// Compute cosine similarity between two f32 slices.
 ///
 /// Returns 0.0 if either vector has zero magnitude.
@@ -22,7 +60,7 @@ pub fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
 /// Returns `(index, score)` pairs sorted by score descending.
 pub fn cosine_similarity_batch(
     query: &[f32],
-    vectors: &[Vec<f32>],
+    vectors: &(impl VectorSet + ?Sized),
     tombstones: &[u8],
 ) -> Vec<(usize, f32)> {
     let query_norm = clawhdf5_accel::vector_norm(query);
@@ -30,7 +68,7 @@ pub fn cosine_similarity_batch(
         return Vec::new();
     }
 
-    let n = vectors.len();
+    let n = vectors.count();
     let mut results: Vec<(usize, f32)> = Vec::with_capacity(n);
 
     // Process 4 vectors at a time where possible
@@ -42,8 +80,9 @@ pub fn cosine_similarity_batch(
             if i < tombstones.len() && tombstones[i] != 0 {
                 continue;
             }
-            let vec_norm = clawhdf5_accel::vector_norm(&vectors[i]);
-            let score = crate::cosine_similarity_prenorm(query, query_norm, &vectors[i], vec_norm);
+            let vec_norm = clawhdf5_accel::vector_norm(vectors.row(i));
+            let score =
+                crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), vec_norm);
             results.push((i, score));
         }
     }
@@ -53,8 +92,8 @@ pub fn cosine_similarity_batch(
         if i < tombstones.len() && tombstones[i] != 0 {
             continue;
         }
-        let vec_norm = clawhdf5_accel::vector_norm(&vectors[i]);
-        let score = crate::cosine_similarity_prenorm(query, query_norm, &vectors[i], vec_norm);
+        let vec_norm = clawhdf5_accel::vector_norm(vectors.row(i));
+        let score = crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), vec_norm);
         results.push((i, score));
     }
 
@@ -68,7 +107,7 @@ pub fn cosine_similarity_batch(
 /// collections. Uses `score = dot(query, vec) / (query_norm * stored_norm)`.
 pub fn cosine_similarity_batch_prenorm(
     query: &[f32],
-    vectors: &[Vec<f32>],
+    vectors: &(impl VectorSet + ?Sized),
     norms: &[f32],
     tombstones: &[u8],
 ) -> Vec<(usize, f32)> {
@@ -77,7 +116,7 @@ pub fn cosine_similarity_batch_prenorm(
         return Vec::new();
     }
 
-    let n = vectors.len();
+    let n = vectors.count();
     let mut results: Vec<(usize, f32)> = Vec::with_capacity(n);
 
     for i in 0..n {
@@ -85,7 +124,7 @@ pub fn cosine_similarity_batch_prenorm(
             continue;
         }
         let vec_norm = norms[i];
-        let score = crate::cosine_similarity_prenorm(query, query_norm, &vectors[i], vec_norm);
+        let score = crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), vec_norm);
         results.push((i, score));
     }
 
@@ -162,7 +201,7 @@ pub fn cosine_similarity_f16(
 #[cfg(feature = "parallel")]
 pub fn parallel_cosine_batch(
     query: &[f32],
-    vectors: &[Vec<f32>],
+    vectors: &(impl VectorSet + Sync + ?Sized),
     tombstones: &[u8],
     k: usize,
 ) -> Vec<(usize, f32)> {
@@ -174,24 +213,27 @@ pub fn parallel_cosine_batch(
     }
 
     let num_cores = rayon::current_num_threads().max(1);
-    let chunk_size = vectors.len().div_ceil(num_cores);
+    let chunk_size = vectors.count().div_ceil(num_cores);
     if chunk_size == 0 {
         return Vec::new();
     }
 
-    let mut all_results: Vec<(usize, f32)> = vectors
-        .par_chunks(chunk_size)
-        .enumerate()
-        .flat_map(|(chunk_idx, chunk)| {
+    // Chunk over index ranges: the corpus may be one flat buffer rather than
+    // a slice of rows, so there is nothing to `par_chunks` over.
+    let n = vectors.count();
+    let mut all_results: Vec<(usize, f32)> = (0..n.div_ceil(chunk_size))
+        .into_par_iter()
+        .flat_map(|chunk_idx| {
             let base = chunk_idx * chunk_size;
-            let mut local: Vec<(usize, f32)> = Vec::with_capacity(chunk.len());
-            for (j, vec) in chunk.iter().enumerate() {
-                let i = base + j;
+            let end = (base + chunk_size).min(n);
+            let mut local: Vec<(usize, f32)> = Vec::with_capacity(end - base);
+            for i in base..end {
                 if i < tombstones.len() && tombstones[i] != 0 {
                     continue;
                 }
-                let vec_norm = clawhdf5_accel::vector_norm(vec);
-                let score = crate::cosine_similarity_prenorm(query, query_norm, vec, vec_norm);
+                let vec_norm = clawhdf5_accel::vector_norm(vectors.row(i));
+                let score =
+                    crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), vec_norm);
                 local.push((i, score));
             }
             local.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
@@ -209,7 +251,7 @@ pub fn parallel_cosine_batch(
 #[cfg(feature = "parallel")]
 pub fn parallel_cosine_batch_prenorm(
     query: &[f32],
-    vectors: &[Vec<f32>],
+    vectors: &(impl VectorSet + Sync + ?Sized),
     norms: &[f32],
     tombstones: &[u8],
     k: usize,
@@ -222,23 +264,26 @@ pub fn parallel_cosine_batch_prenorm(
     }
 
     let num_cores = rayon::current_num_threads().max(1);
-    let chunk_size = vectors.len().div_ceil(num_cores);
+    let chunk_size = vectors.count().div_ceil(num_cores);
     if chunk_size == 0 {
         return Vec::new();
     }
 
-    let mut all_results: Vec<(usize, f32)> = vectors
-        .par_chunks(chunk_size)
-        .enumerate()
-        .flat_map(|(chunk_idx, chunk)| {
+    // Chunk over index ranges: the corpus may be one flat buffer rather than
+    // a slice of rows, so there is nothing to `par_chunks` over.
+    let n = vectors.count();
+    let mut all_results: Vec<(usize, f32)> = (0..n.div_ceil(chunk_size))
+        .into_par_iter()
+        .flat_map(|chunk_idx| {
             let base = chunk_idx * chunk_size;
-            let mut local: Vec<(usize, f32)> = Vec::with_capacity(chunk.len());
-            for (j, vec) in chunk.iter().enumerate() {
-                let i = base + j;
+            let end = (base + chunk_size).min(n);
+            let mut local: Vec<(usize, f32)> = Vec::with_capacity(end - base);
+            for i in base..end {
                 if i < tombstones.len() && tombstones[i] != 0 {
                     continue;
                 }
-                let score = crate::cosine_similarity_prenorm(query, query_norm, vec, norms[i]);
+                let score =
+                    crate::cosine_similarity_prenorm(query, query_norm, vectors.row(i), norms[i]);
                 local.push((i, score));
             }
             local.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));

crates/clawhdf5-agent/tests/hnsw_integration.rs

@@ -165,3 +165,122 @@ fn save_batch_then_search_is_consistent() {
         );
     }
 }
+
+#[test]
+fn quantized_index_matches_the_f32_index_after_re_scoring() {
+    // A quantised index holds approximate vectors, but the store still has the
+    // exact ones, so the query path re-scores the candidate pool before
+    // fusion. The results a caller sees should therefore be the same.
+    let dim = 64;
+    let n = 400;
+    let mut seed = 0x5EED_1234_5678_9ABC;
+    let vectors: Vec<Vec<f32>> = (0..n).map(|_| make_vector(&mut seed, dim)).collect();
+    let queries: Vec<Vec<f32>> = (0..20).map(|_| make_vector(&mut seed, dim)).collect();
+
+    let build = |dir: &TempDir, quantized: bool| {
+        let mut config = MemoryConfig::new(dir.path().join("mem.h5"), "agent", dim);
+        config.quantized_index = quantized;
+        let mut mem = HDF5Memory::create(config).unwrap();
+        for (i, v) in vectors.iter().enumerate() {
+            mem.save(entry(&format!("chunk {i}"), v.clone(), &format!("k{i}")))
+                .unwrap();
+        }
+        mem
+    };
+
+    let exact_dir = TempDir::new().unwrap();
+    let quant_dir = TempDir::new().unwrap();
+    let mut exact = build(&exact_dir, false);
+    let mut quantized = build(&quant_dir, true);
+
+    let k = 10;
+    let mut agree = 0;
+    for q in &queries {
+        let want: Vec<usize> = exact
+            .hybrid_search(q, "", 1.0, 0.0, k)
+            .iter()
+            .map(|r| r.index)
+            .collect();
+        agree += quantized
+            .hybrid_search(q, "", 1.0, 0.0, k)
+            .iter()
+            .filter(|r| want.contains(&r.index))
+            .count();
+    }
+    let overlap = agree as f64 / (k * queries.len()) as f64;
+    assert!(
+        overlap >= 0.95,
+        "quantised store should match the f32 one: {overlap}"
+    );
+}
+
+#[test]
+fn quantized_index_setting_survives_a_reopen() {
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("mem.h5");
+    let mut config = MemoryConfig::new(path.clone(), "agent", 8);
+    config.quantized_index = true;
+    let mut mem = HDF5Memory::create(config).unwrap();
+    let mut seed = 7;
+    for i in 0..30 {
+        mem.save(entry(&format!("c{i}"), make_vector(&mut seed, 8), "t"))
+            .unwrap();
+    }
+    mem.flush_wal().unwrap();
+    drop(mem);
+
+    // Reopening must not silently quadruple the index's memory, so the flag
+    // is part of the stored config rather than a per-session choice.
+    let reopened = HDF5Memory::open(&path).unwrap();
+    assert!(reopened.config().quantized_index);
+}
+
+#[test]
+fn hnsw_parameters_are_configurable_and_persisted() {
+    // The graph degree and both candidate-list sizes used to be constants, so
+    // a deployment could not trade recall against memory or speed at all.
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("mem.h5");
+    let mut config = MemoryConfig::new(path.clone(), "agent", 16);
+    config.hnsw_m = 8;
+    config.hnsw_ef_construction = 32;
+    config.hnsw_ef_search = 128;
+    let mut mem = HDF5Memory::create(config).unwrap();
+
+    let mut seed = 99;
+    let vectors: Vec<Vec<f32>> = (0..300).map(|_| make_vector(&mut seed, 16)).collect();
+    for (i, v) in vectors.iter().enumerate() {
+        mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap();
+    }
+    // Still correct with a smaller graph: an exact match must rank first.
+    let top = mem.hybrid_search(&vectors[42], "", 1.0, 0.0, 1);
+    assert_eq!(top[0].index, 42);
+
+    mem.flush_wal().unwrap();
+    drop(mem);
+    let reopened = HDF5Memory::open(&path).unwrap();
+    assert_eq!(reopened.config().hnsw_m, 8);
+    assert_eq!(reopened.config().hnsw_ef_construction, 32);
+    assert_eq!(reopened.config().hnsw_ef_search, 128);
+}
+
+#[test]
+fn degenerate_hnsw_parameters_do_not_panic() {
+    // `clawhdf5-ann` asserts m >= 2, so a zero from a config file — or from a
+    // caller who assumed 0 meant "default" — would abort the process inside
+    // the index builder. The store clamps instead.
+    let dir = TempDir::new().unwrap();
+    let mut config = MemoryConfig::new(dir.path().join("mem.h5"), "agent", 8);
+    config.hnsw_m = 0;
+    config.hnsw_ef_construction = 0;
+    config.hnsw_ef_search = 1;
+    let mut mem = HDF5Memory::create(config).unwrap();
+
+    let mut seed = 5;
+    let vectors: Vec<Vec<f32>> = (0..50).map(|_| make_vector(&mut seed, 8)).collect();
+    for (i, v) in vectors.iter().enumerate() {
+        mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap();
+    }
+    let results = mem.hybrid_search(&vectors[7], "", 1.0, 0.0, 5);
+    assert_eq!(results[0].index, 7, "exact match should still rank first");
+}

crates/clawhdf5-android/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-android"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "Android JNI bridge for edgehdf5-memory HDF5 backend"
 license = "MIT"

crates/clawhdf5-ann/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-ann"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "HNSW approximate nearest neighbor index stored as HDF5"
 license = "MIT"
@@ -10,9 +10,9 @@ keywords = ["hdf5", "ann", "hnsw", "nearest-neighbor"]
 categories = ["algorithms", "science"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.4.0" }
-clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.4.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0" }
+clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.7.0" }
 rayon = { version = "1", optional = true }
 
 [features]

crates/clawhdf5-ann/src/lib.rs

@@ -5,4 +5,4 @@
 
 mod hnsw;
 
-pub use hnsw::{DistanceMetric, HnswIndex};
+pub use hnsw::{DistanceMetric, HnswIndex, Storage};

crates/clawhdf5-bench/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-bench"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "Benchmark harnesses for clawhdf5-agent (Track 8)"
 license = "MIT"
@@ -13,6 +13,10 @@ path = "src/bin/longmemeval_bench.rs"
 name = "memory_arena"
 path = "src/bin/memory_arena.rs"
 
+[[bin]]
+name = "read_harness"
+path = "src/bin/read_harness.rs"
+
 [[bin]]
 name = "search_harness"
 path = "src/bin/search_harness.rs"
@@ -53,6 +57,8 @@ harness = false
 [dependencies]
 clawhdf5-agent = { path = "../clawhdf5-agent" }
 clawhdf5-ann = { path = "../clawhdf5-ann" }
+clawhdf5 = { path = "../clawhdf5" }
+clawhdf5-format = { path = "../clawhdf5-format" }
 clawhdf5-io = { path = "../clawhdf5-io" }
 mpi = { version = "0.8", optional = true }
 serde = { workspace = true }

crates/clawhdf5-bench/src/bin/longmemeval_bench.rs

@@ -55,44 +55,150 @@ use std::time::{Duration, Instant};
 #[path = "longmemeval_bench/embedder.rs"]
 mod embedder;
 
-use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
+use clawhdf5_agent::bm25::TokenFilter;
+use clawhdf5_agent::hybrid::Fusion;
+use clawhdf5_agent::reranker::{ReRankConfig, RerankInput, rerank};
+use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchResult};
 use serde::Deserialize;
 use tempfile::TempDir;
 
 const EMBEDDING_DIM: usize = 384;
 
+/// A mode's fusion, as one short string for the reports.
+fn describe(mode: Mode) -> String {
+    let fusion = match mode.fusion {
+        Fusion::Weighted { vector, keyword } => format!("vector_{vector:.1}_keyword_{keyword:.1}"),
+        Fusion::Rrf { k } => format!("rrf_k{k:.0}"),
+    };
+    let tokens = match mode.tokens {
+        TokenFilter::Plain => fusion,
+        TokenFilter::Stemmed => format!("{fusion}_stemmed"),
+    };
+    match mode.rerank {
+        None => tokens,
+        Some(cfg) if cfg.relevance_weight == 0.0 => format!("{tokens}_rerank_metadata"),
+        Some(cfg) => format!(
+            "{tokens}_rerank_blended_hl{:.0}d",
+            cfg.temporal_half_life_secs / 86_400.0
+        ),
+    }
+}
+
 /// A retrieval configuration: how much of the score comes from each stage.
 #[derive(Clone, Copy)]
 struct Mode {
     label: &'static str,
-    vector_weight: f32,
-    keyword_weight: f32,
+    /// How the two retrieval stages are combined into one ranking.
+    fusion: Fusion,
+    /// How keyword tokens are normalised before indexing and querying.
+    tokens: TokenFilter,
+    /// Re-rank the retrieved candidates with recency and friends, relative to
+    /// the question's own date.
+    rerank: Option<ReRankConfig>,
+}
+
+impl Mode {
+    const fn weighted(label: &'static str, vector: f32, keyword: f32) -> Self {
+        Self {
+            label,
+            fusion: Fusion::Weighted { vector, keyword },
+            tokens: TokenFilter::Plain,
+            rerank: None,
+        }
+    }
+
+    #[cfg_attr(not(feature = "embeddings"), allow(dead_code))]
+    fn reranked(mut self, label: &'static str, rerank: ReRankConfig) -> Self {
+        self.label = label;
+        self.rerank = Some(rerank);
+        self
+    }
+
+    const fn stemmed(mut self, label: &'static str) -> Self {
+        self.label = label;
+        self.tokens = TokenFilter::Stemmed;
+        self
+    }
 }
 
 /// The only mode available without real embeddings. Passing zero vectors with
 /// `vector_weight = 0.0` is what made the vector stage inert.
-const BM25_ONLY: Mode = Mode {
-    label: "BM25 only (vector stage inert)",
-    vector_weight: 0.0,
-    keyword_weight: 1.0,
-};
+const BM25_ONLY: Mode = Mode::weighted("BM25 only (vector stage inert)", 0.0, 1.0);
 #[cfg(feature = "embeddings")]
-const VECTOR_ONLY: Mode = Mode {
-    label: "Vector only (MiniLM + HNSW)",
-    vector_weight: 1.0,
-    keyword_weight: 0.0,
-};
+const VECTOR_ONLY: Mode = Mode::weighted("Vector only (MiniLM + HNSW)", 1.0, 0.0);
 /// Tuned by `--sweep` over the full haystack. The former 0.7/0.3 was a
 /// documented default that had never been searched, and the sweep found it
 /// strictly dominated: 0.4/0.6 is better on Hit@1, Hit@5, Hit@10 and MRR at
 /// both granularities.
 #[cfg(feature = "embeddings")]
-const HYBRID: Mode = Mode {
-    label: "Hybrid (0.4 vector / 0.6 BM25, tuned)",
-    vector_weight: 0.4,
-    keyword_weight: 0.6,
+const HYBRID: Mode = Mode::weighted("Hybrid (0.4 vector / 0.6 BM25, tuned)", 0.4, 0.6);
+
+/// Reciprocal rank fusion, the documented alternative to the weighted sum.
+/// It ignores score magnitudes, so there is nothing to tune — which is the
+/// claim being tested.
+#[cfg(feature = "embeddings")]
+const RRF: Mode = Mode {
+    label: "Hybrid (reciprocal rank fusion, k=60)",
+    fusion: Fusion::Rrf { k: 60.0 },
+    tokens: TokenFilter::Plain,
+    rerank: None,
 };
 
+/// The same two configurations with stemmed keyword tokens, so the tokenizer's
+/// effect is isolated from everything else.
+const BM25_STEMMED: Mode = BM25_ONLY.stemmed("BM25 only, stemmed tokens");
+
+/// Re-ranking as it behaved before `relevance` was an input: the combined
+/// score was recency + authority + activation only, so the retriever's own
+/// ordering was discarded.
+#[cfg(feature = "embeddings")]
+fn hybrid_rerank_metadata_only() -> Mode {
+    HYBRID.reranked(
+        "Hybrid + rerank (metadata only, pre-fix)",
+        ReRankConfig {
+            relevance_weight: 0.0,
+            ..ReRankConfig::default()
+        },
+    )
+}
+
+/// Re-ranking as it behaves now: relevance leads, recency nudges.
+#[cfg(feature = "embeddings")]
+fn hybrid_rerank_blended() -> Mode {
+    HYBRID.reranked(
+        "Hybrid + rerank (relevance + recency)",
+        ReRankConfig::default(),
+    )
+}
+
+/// The same blend at several half-lives. Decay is `2^(-age / half_life)`, so a
+/// half-life far shorter than the gaps between memories sends every score to
+/// zero and the signal vanishes; far longer and everything scores ~1 and it
+/// vanishes the other way. The right value tracks how far apart the memories
+/// actually are.
+#[cfg(feature = "embeddings")]
+fn hybrid_rerank_half_lives() -> Vec<Mode> {
+    [
+        ("1 day", 86_400.0),
+        ("7 days", 7.0 * 86_400.0),
+        ("30 days", 30.0 * 86_400.0),
+        ("90 days", 90.0 * 86_400.0),
+    ]
+    .into_iter()
+    .map(|(label, half_life)| {
+        HYBRID.reranked(
+            Box::leak(format!("Hybrid + rerank, half-life {label}").into_boxed_str()),
+            ReRankConfig {
+                temporal_half_life_secs: half_life,
+                ..ReRankConfig::default()
+            },
+        )
+    })
+    .collect()
+}
+#[cfg(feature = "embeddings")]
+const HYBRID_STEMMED: Mode = HYBRID.stemmed("Hybrid 0.4/0.6, stemmed tokens");
+
 /// Every 0.1 step of vector weight, keyword weight taking the remainder.
 ///
 /// Labels are leaked to `&'static str` because `Mode::label` is a `&'static
@@ -104,11 +210,11 @@ fn sweep_modes() -> Vec<Mode> {
     (0..=10)
         .map(|i| {
             let v = i as f32 / 10.0;
-            Mode {
-                label: Box::leak(format!("sweep v={v:.1} / k={:.1}", 1.0 - v).into_boxed_str()),
-                vector_weight: v,
-                keyword_weight: 1.0 - v,
-            }
+            Mode::weighted(
+                Box::leak(format!("sweep v={v:.1} / k={:.1}", 1.0 - v).into_boxed_str()),
+                v,
+                1.0 - v,
+            )
         })
         .collect()
 }
@@ -181,6 +287,37 @@ struct Question {
     haystack_session_ids: Vec<String>,
     haystack_sessions: Vec<Vec<Turn>>,
     answer_session_ids: Vec<String>,
+    /// One timestamp per haystack session, e.g. "2023/05/25 (Thu) 20:21".
+    #[serde(default)]
+    haystack_dates: Vec<String>,
+}
+
+/// Seconds since the epoch for a LongMemEval session date, which looks like
+/// `2023/05/25 (Thu) 20:21`. Sessions are stored in chronological order, so a
+/// date that cannot be parsed falls back to its position — order is preserved
+/// even if the interval is not.
+fn session_time(date: &str, position: usize) -> f64 {
+    let stamp = |y: i64, mo: i64, d: i64, h: i64, mi: i64| -> f64 {
+        // Days since 1970-01-01 via the civil-from-days algorithm.
+        let (y, mo) = if mo <= 2 { (y - 1, mo + 12) } else { (y, mo) };
+        let era = y.div_euclid(400);
+        let yoe = y - era * 400;
+        let doy = (153 * (mo - 3) + 2) / 5 + d - 1;
+        let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy;
+        let days = era * 146_097 + doe - 719_468;
+        (days * 86_400 + h * 3_600 + mi * 60) as f64
+    };
+    let parse = || -> Option<f64> {
+        let (ymd, rest) = date.split_once(' ')?;
+        let mut ymd = ymd.split('/');
+        let y = ymd.next()?.parse().ok()?;
+        let mo = ymd.next()?.parse().ok()?;
+        let d = ymd.next()?.parse().ok()?;
+        let hm = rest.rsplit(' ').next()?;
+        let (h, mi) = hm.split_once(':')?;
+        Some(stamp(y, mo, d, h.parse().ok()?, mi.parse().ok()?))
+    };
+    parse().unwrap_or(1_000_000.0 + position as f64 * 86_400.0)
 }
 
 // ---------------------------------------------------------------------------
@@ -199,11 +336,21 @@ struct Metrics {
     rr_turn: f64,
     abstention_correct: u32,
     abstention_total: u32,
+    /// Questions where the newest gold session outranked the older ones, out
+    /// of those with more than one gold session and at least one retrieved.
+    newest_gold_first: u32,
+    newest_gold_total: u32,
     latency_ns: Vec<u64>,
     count: u32,
 }
 
 impl Metrics {
+    /// `None` when no question in this bucket had multiple gold sessions.
+    fn newest_gold_first_pct(&self) -> Option<f64> {
+        (self.newest_gold_total > 0)
+            .then(|| self.newest_gold_first as f64 / self.newest_gold_total as f64 * 100.0)
+    }
+
     fn hit1_session_pct(&self) -> f64 {
         self.hit1_session as f64 / self.count.max(1) as f64 * 100.0
     }
@@ -261,6 +408,16 @@ struct EvalResult {
     hit5_turn: bool,
     hit10_turn: bool,
     rr_turn: Option<f64>,
+    /// For a question whose evidence spans several dated sessions (a
+    /// `knowledge-update`, where an earlier fact is superseded by a later
+    /// one): did the *newest* gold session outrank every older gold session
+    /// that was returned? `None` when the question has one gold session, or
+    /// when none were retrieved, so there is nothing to discriminate.
+    ///
+    /// Plain recall cannot see this. LongMemEval labels *both* the stale and
+    /// the updated session as gold, so returning either counts as a hit — yet
+    /// only one of them answers the question correctly.
+    newest_gold_first: Option<bool>,
     latency: Duration,
 }
 
@@ -276,19 +433,26 @@ fn evaluate_question(
     config.compact_threshold = 0.0;
 
     let mut memory = HDF5Memory::create(config).expect("failed to create HDF5Memory");
+    memory.set_token_filter(mode.tokens);
 
     // Build MemoryEntry list from all haystack sessions
     let mut entries: Vec<MemoryEntry> = Vec::new();
     let mut turn_has_answer: Vec<bool> = Vec::new();
-    let mut ts = 1_000_000.0f64;
-
     for (sess_idx, session) in q.haystack_sessions.iter().enumerate() {
         let sess_id = q
             .haystack_session_ids
             .get(sess_idx)
             .map(String::as_str)
             .unwrap_or("unknown");
-        for turn in session {
+        // Real session dates, not a synthetic counter: anything that decays
+        // with age needs true intervals, not just the right order.
+        let session_start = q
+            .haystack_dates
+            .get(sess_idx)
+            .map_or(sess_idx as f64 * 86_400.0, |d| session_time(d, sess_idx));
+        for (turn_idx, turn) in session.iter().enumerate() {
+            // Spread a session's turns over the minutes following its start.
+            let ts = session_start + turn_idx as f64 * 60.0;
             entries.push(MemoryEntry {
                 chunk: turn.content.clone(),
                 embedding: embedding_for(embeddings, &turn.content),
@@ -302,7 +466,6 @@ fn evaluate_question(
                 },
             });
             turn_has_answer.push(turn.has_answer);
-            ts += 1.0;
         }
     }
 
@@ -319,17 +482,87 @@ fn evaluate_question(
     // Set of session IDs that contain the answer
     let answer_sess_set: HashSet<&str> = q.answer_session_ids.iter().map(String::as_str).collect();
 
+    // When each gold session was recorded, so "newest" is by date rather than
+    // by position (the two agree in this dataset, but the metric should not
+    // depend on that).
+    let gold_times: HashMap<&str, f64> = q
+        .haystack_session_ids
+        .iter()
+        .enumerate()
+        .filter(|(_, sid)| answer_sess_set.contains(sid.as_str()))
+        .map(|(i, sid)| {
+            let t = q
+                .haystack_dates
+                .get(i)
+                .map_or(i as f64 * 86_400.0, |d| session_time(d, i));
+            (sid.as_str(), t)
+        })
+        .collect();
+
     let query_emb = embedding_for(embeddings, &q.question);
     let t0 = Instant::now();
-    let results = memory.hybrid_search(
-        &query_emb,
-        &q.question,
-        mode.vector_weight,
-        mode.keyword_weight,
-        top_k,
-    );
+    // Re-ranking only reorders; it needs a candidate pool larger than `top_k`
+    // to have anything to promote.
+    let pool = if mode.rerank.is_some() {
+        top_k * 4
+    } else {
+        top_k
+    };
+    let mut results = memory.hybrid_search_with(&query_emb, &q.question, mode.fusion, pool);
+    if let Some(config) = mode.rerank {
+        // "Now" is the moment the question was asked, so decay measures how
+        // stale each memory was at that point.
+        let now = session_time(&q.question_date, q.haystack_sessions.len());
+        let inputs: Vec<RerankInput> = results
+            .iter()
+            .map(|r| RerankInput {
+                index: r.index,
+                timestamp: r.timestamp,
+                source_channel: r.source_channel.clone(),
+                raw_activation: r.activation,
+                relevance: r.score,
+            })
+            .collect();
+        let order: Vec<usize> = rerank(&inputs, &config, now)
+            .into_iter()
+            .map(|r| r.index)
+            .collect();
+        let by_index: HashMap<usize, SearchResult> =
+            results.into_iter().map(|r| (r.index, r)).collect();
+        results = order
+            .into_iter()
+            .filter_map(|i| by_index.get(&i).cloned())
+            .collect();
+    }
+    results.truncate(top_k);
     let latency = t0.elapsed();
 
+    // Rank of the best-placed result from each gold session.
+    let mut first_rank: HashMap<&str, usize> = HashMap::new();
+    for (rank, result) in results.iter().enumerate() {
+        let sid = memory.cache.session_ids[result.index].as_str();
+        if let Some((gold_sid, _)) = gold_times.get_key_value(sid) {
+            first_rank.entry(gold_sid).or_insert(rank);
+        }
+    }
+    let newest_gold_first = if gold_times.len() < 2 || first_rank.is_empty() {
+        None
+    } else {
+        // The newest gold session must be retrieved, and no older gold session
+        // may outrank it.
+        let newest = gold_times
+            .iter()
+            .max_by(|a, b| a.1.total_cmp(b.1))
+            .map(|(sid, _)| *sid)
+            .expect("at least two gold sessions");
+        Some(match first_rank.get(newest) {
+            Some(&newest_rank) => first_rank
+                .iter()
+                .all(|(sid, &rank)| *sid == newest || rank > newest_rank),
+            None => false,
+        })
+    };
+
     // Session-level recall
     let mut hit1_session = false;
     let mut hit5_session = false;
@@ -384,6 +617,7 @@ fn evaluate_question(
         hit5_turn,
         hit10_turn,
         rr_turn,
+        newest_gold_first,
         latency,
     }
 }
@@ -472,10 +706,7 @@ fn print_report(
     println!("  LongMemEval Benchmark  —  {}", mode.label);
     println!("=================================================================");
     println!();
-    println!(
-        "Mode: vector_weight={:.1} / keyword_weight={:.1}",
-        mode.vector_weight, mode.keyword_weight
-    );
+    println!("Mode: {}", describe(mode));
     println!();
     println!("Scoring target: RETRIEVAL RECALL (did the gold memory land in top-k).");
     println!("  No answer is generated or scored. This is NOT the official");
@@ -538,6 +769,24 @@ fn print_report(
     );
     println!();
 
+    if let Some(pct) = overall.newest_gold_first_pct() {
+        println!(
+            "## Recency Discrimination  (n={})",
+            overall.newest_gold_total
+        );
+        println!(
+            "  Newest gold session ranked first: {}/{} ({pct:.1}%)",
+            overall.newest_gold_first, overall.newest_gold_total
+        );
+        println!(
+            "  Questions whose evidence spans several dated sessions — a fact and\n  \
+             its later correction. Both sessions are labelled gold, so recall\n  \
+             scores either as a hit; this asks whether the *current* one came\n  \
+             first. A retriever with no sense of time scores near chance."
+        );
+        println!();
+    }
+
     if overall.abstention_total > 0 {
         println!("## Abstention Accuracy");
         println!(
@@ -602,10 +851,7 @@ fn print_report(
     println!("```json");
     println!("{{");
     println!("  \"benchmark\": \"longmemeval\",");
-    println!(
-        "  \"mode\": \"vector_{:.1}_keyword_{:.1}\",",
-        mode.vector_weight, mode.keyword_weight
-    );
+    println!("  \"mode\": \"{}\",", describe(mode));
     println!("  \"dataset_variant\": \"{}\",", profile.variant());
     println!("  \"scoring_target\": \"retrieval_recall\",");
     println!("  \"k\": 10,");
@@ -654,6 +900,14 @@ fn print_report(
     } else {
         println!("  \"abstention_accuracy\": null,");
     }
+    match overall.newest_gold_first_pct() {
+        Some(pct) => println!(
+            "  \"newest_gold_first\": {:.4}, \"newest_gold_n\": {},",
+            pct / 100.0,
+            overall.newest_gold_total
+        ),
+        None => println!("  \"newest_gold_first\": null,"),
+    }
     println!("  \"latency_us\": {{");
     println!(
         "    \"avg\": {:.1}, \"p50\": {:.1}, \"p95\": {:.1}, \"p99\": {:.1}",
@@ -676,6 +930,8 @@ fn main() {
     let mut limit: Option<usize> = None;
     let mut weights_dir: Option<String> = None;
     let mut sweep = false;
+    #[cfg_attr(not(feature = "embeddings"), allow(unused_mut, unused_variables))]
+    let mut rerank_sweep = false;
     let mut args = std::env::args().skip(1);
     while let Some(arg) = args.next() {
         match arg.as_str() {
@@ -684,6 +940,16 @@ fn main() {
                 limit = Some(v.parse().expect("--limit must be a positive integer"));
             }
             "--sweep" => sweep = true,
+            "--rerank-sweep" => {
+                // Re-ranking needs the vector stage to have candidates worth
+                // reordering, so this is an embeddings-only comparison.
+                #[cfg(feature = "embeddings")]
+                {
+                    rerank_sweep = true;
+                }
+                #[cfg(not(feature = "embeddings"))]
+                eprintln!("warning: --rerank-sweep needs --features embeddings; ignoring");
+            }
             "--embeddings" => {
                 weights_dir = Some(args.next().expect("--embeddings needs a directory"));
             }
@@ -702,6 +968,9 @@ fn main() {
                               BM25-only, vector-only, and hybrid separately. Requires\n\
                               --features embeddings; without it the vector stage is\n\
                               inert and only the BM25 row is produced.\n\
+                     --rerank-sweep\n\
+                              compare re-ranking off, metadata-only (the old\n\
+                              behaviour) and blended at several half-lives.\n\
                      --sweep  instead of the three named modes, sweep vector_weight\n\
                               from 0.0 to 1.0 in 0.1 steps. The 0.7/0.3 default was\n\
                               never searched; this is what searches it."
@@ -767,19 +1036,34 @@ fn main() {
         {
             if sweep {
                 sweep_modes()
+            } else if rerank_sweep {
+                let mut modes = vec![HYBRID, hybrid_rerank_metadata_only()];
+                modes.extend(hybrid_rerank_half_lives());
+                modes
             } else {
-                vec![BM25_ONLY, VECTOR_ONLY, HYBRID]
+                vec![
+                    BM25_ONLY,
+                    VECTOR_ONLY,
+                    HYBRID,
+                    RRF,
+                    BM25_STEMMED,
+                    HYBRID_STEMMED,
+                    hybrid_rerank_metadata_only(),
+                    hybrid_rerank_blended(),
+                ]
             }
         }
         #[cfg(not(feature = "embeddings"))]
         {
-            vec![BM25_ONLY]
+            vec![BM25_ONLY, BM25_STEMMED]
         }
     } else {
         if sweep {
             eprintln!("warning: --sweep needs --embeddings; running BM25 only");
         }
-        vec![BM25_ONLY]
+        // Stemming is a property of the keyword stage, so it can be compared
+        // without a model.
+        vec![BM25_ONLY, BM25_STEMMED]
     };
 
     for (mode_idx, mode) in modes.iter().enumerate() {
@@ -882,6 +1166,14 @@ fn run_mode(
                 entry.rr_turn += rr;
                 overall.rr_turn += rr;
             }
+            if let Some(newest_first) = result.newest_gold_first {
+                entry.newest_gold_total += 1;
+                overall.newest_gold_total += 1;
+                if newest_first {
+                    entry.newest_gold_first += 1;
+                    overall.newest_gold_first += 1;
+                }
+            }
 
             let ns = result.latency.as_nanos() as u64;
             entry.latency_ns.push(ns);
@@ -893,3 +1185,30 @@ fn run_mode(
     eprintln!();
     print_report(&overall, &by_type, profile, mode);
 }
+
+#[cfg(test)]
+mod tests {
+    use super::session_time;
+
+    #[test]
+    fn session_dates_parse_to_the_right_instant() {
+        // Reference values from Python's datetime, UTC.
+        for (date, expected) in [
+            ("2023/05/25 (Thu) 20:21", 1_685_046_060.0),
+            ("1970/01/01 (Thu) 00:00", 0.0),
+            ("2000/02/29 (Tue) 12:00", 951_825_600.0),
+            ("2023/12/31 (Sun) 23:59", 1_704_067_140.0),
+            ("2024/03/01 (Fri) 00:00", 1_709_251_200.0),
+        ] {
+            assert_eq!(session_time(date, 0), expected, "{date}");
+        }
+    }
+
+    #[test]
+    fn unparseable_dates_fall_back_to_position_order() {
+        let a = session_time("not a date", 0);
+        let b = session_time("", 1);
+        let c = session_time("2023/13/99 (???) 99:99", 2);
+        assert!(a < b && b < c, "fallback must preserve session order");
+    }
+}

crates/clawhdf5-bench/src/bin/read_harness.rs

@@ -0,0 +1,176 @@
+//! HDF5 read-path measurement harness: full reads vs. hyperslab selections on
+//! a chunked 2-D dataset, compressed and uncompressed, plus a contiguous one.
+//!
+//! The question it answers for every read-path change: does the cost of a
+//! selection scale with the *selection*, or with the whole dataset?
+//!
+//! ```text
+//! cargo run --release -p clawhdf5-bench --bin read_harness
+//! cargo run --release -p clawhdf5-bench --bin read_harness -- --large   # 512 MB
+//! ```
+
+use std::time::{Duration, Instant};
+
+use clawhdf5::{File, FileBuilder};
+use clawhdf5_format::selection::Selection;
+
+const CHUNK: u64 = 256;
+
+struct Layout {
+    name: &'static str,
+    chunked: bool,
+    deflate: bool,
+}
+
+const LAYOUTS: [Layout; 3] = [
+    Layout {
+        name: "chunked + deflate",
+        chunked: true,
+        deflate: true,
+    },
+    Layout {
+        name: "chunked",
+        chunked: true,
+        deflate: false,
+    },
+    Layout {
+        name: "contiguous",
+        chunked: false,
+        deflate: false,
+    },
+];
+
+/// Smooth-ish, compressible data whose value encodes its position, so a read
+/// can be verified exactly.
+fn value(row: u64, col: u64) -> f64 {
+    (row * 100_003 + col) as f64 * 0.5
+}
+
+fn write_file(path: &std::path::Path, rows: u64, cols: u64) {
+    let data: Vec<f64> = (0..rows)
+        .flat_map(|r| (0..cols).map(move |c| value(r, c)))
+        .collect();
+    let mut builder = FileBuilder::new();
+    for (i, layout) in LAYOUTS.iter().enumerate() {
+        let ds = builder.create_dataset(&format!("d{i}"));
+        ds.with_f64_data(&data).with_shape(&[rows, cols]);
+        if layout.chunked {
+            ds.with_chunks(&[CHUNK, CHUNK]);
+        }
+        if layout.deflate {
+            ds.with_deflate(4);
+        }
+    }
+    builder.write(path).unwrap();
+}
+
+fn median(mut samples: Vec<Duration>) -> Duration {
+    samples.sort();
+    samples[samples.len() / 2]
+}
+
+fn time<T>(reps: usize, mut f: impl FnMut() -> T) -> Duration {
+    median(
+        (0..reps)
+            .map(|_| {
+                let t = Instant::now();
+                std::hint::black_box(f());
+                t.elapsed()
+            })
+            .collect(),
+    )
+}
+
+fn slab(start: [u64; 2], count: [u64; 2]) -> Selection {
+    Selection::Hyperslab {
+        start: start.to_vec(),
+        stride: vec![1, 1],
+        count: count.to_vec(),
+        block: vec![1, 1],
+    }
+}
+
+fn main() {
+    let large = std::env::args().any(|a| a == "--large");
+    let (rows, cols) = if large { (8192, 8192) } else { (4096, 2048) };
+    let total_mb = (rows * cols * 8) as f64 / (1 << 20) as f64;
+    if cfg!(debug_assertions) {
+        eprintln!("warning: debug build — numbers are meaningless. Use --release.");
+    }
+
+    let dir = tempfile::TempDir::new().unwrap();
+    let path = dir.path().join("read_harness.h5");
+    write_file(&path, rows, cols);
+    let file_mb = std::fs::metadata(&path).unwrap().len() as f64 / (1 << 20) as f64;
+
+    println!("## Read harness");
+    println!(
+        "\n{rows} x {cols} f64 ({total_mb:.0} MB per dataset), chunks {CHUNK} x {CHUNK}, file {file_mb:.0} MB\n"
+    );
+
+    // (label, selection, elements selected)
+    let selections: Vec<(&str, Selection, u64)> = vec![
+        (
+            "64 x 64 window (1 chunk)",
+            slab([300, 300], [64, 64]),
+            64 * 64,
+        ),
+        (
+            "512 x 512 window (4-9 chunks)",
+            slab([1000, 700], [512, 512]),
+            512 * 512,
+        ),
+        ("one row", slab([rows / 2, 0], [1, cols]), cols),
+        ("one column", slab([0, cols / 2], [rows, 1]), rows),
+    ];
+
+    println!("| layout | read | selected | time ms | MB/s of selection | vs full read |");
+    println!("|---|---|---:|---:|---:|---:|");
+    for (i, layout) in LAYOUTS.iter().enumerate() {
+        // Fresh handle per layout so one dataset's cached chunks don't help
+        // (or evict) another's.
+        let file = File::open(&path).unwrap();
+        let ds = file.dataset(&format!("d{i}")).unwrap();
+
+        let full_cold = time(1, || ds.read_f64().unwrap());
+        let full = time(3, || ds.read_f64().unwrap());
+        println!(
+            "| {} | full (first) | {total_mb:.0} MB | {:.1} | {:.0} | |",
+            layout.name,
+            full_cold.as_secs_f64() * 1e3,
+            total_mb / full_cold.as_secs_f64()
+        );
+        println!(
+            "| {} | full (repeat) | {total_mb:.0} MB | {:.1} | {:.0} | 1.00x |",
+            layout.name,
+            full.as_secs_f64() * 1e3,
+            total_mb / full.as_secs_f64()
+        );
+
+        for (label, selection, elements) in &selections {
+            // A fresh handle again: measure the selection on its own, not
+            // served from chunks the full read just cached.
+            let file = File::open(&path).unwrap();
+            let ds = file.dataset(&format!("d{i}")).unwrap();
+            let got = ds.read_f64_selection(selection).unwrap();
+            assert_eq!(got.len() as u64, *elements, "{label}");
+            if let Selection::Hyperslab { start, .. } = selection {
+                assert_eq!(got[0], value(start[0], start[1]), "{label}: wrong data");
+            }
+            let took = time(5, || {
+                let file = File::open(&path).unwrap();
+                let ds = file.dataset(&format!("d{i}")).unwrap();
+                ds.read_f64_selection(selection).unwrap()
+            });
+            let mb = (*elements * 8) as f64 / (1 << 20) as f64;
+            println!(
+                "| {} | {label} | {:.2} MB | {:.2} | {:.0} | {:.3}x |",
+                layout.name,
+                mb,
+                took.as_secs_f64() * 1e3,
+                mb / took.as_secs_f64(),
+                took.as_secs_f64() / full_cold.as_secs_f64()
+            );
+        }
+    }
+}

crates/clawhdf5-bench/src/bin/search_harness.rs

@@ -24,7 +24,7 @@
 use std::time::{Duration, Instant};
 
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
-use clawhdf5_ann::{DistanceMetric, HnswIndex};
+use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};
 
 const DIM: usize = 384;
 const K: usize = 10;
@@ -84,6 +84,22 @@ struct Dataset {
 /// that appears only on clustered data points at graph connectivity.
 static UNIFORM: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
 
+/// `--int8`: build the HNSW index over int8-quantised vectors (a quarter of
+/// the memory) instead of f32, to price the recall it costs.
+static INT8: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
+
+/// `--rerank`: re-score the candidate pool against the exact vectors before
+/// taking the top K.
+static RERANK: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
+
+fn storage() -> Storage {
+    if INT8.load(std::sync::atomic::Ordering::Relaxed) {
+        Storage::Int8
+    } else {
+        Storage::Float32
+    }
+}
+
 fn make_dataset(n: usize, seed: u64) -> Dataset {
     let mut rng = Rng(seed);
     if UNIFORM.load(std::sync::atomic::Ordering::Relaxed) {
@@ -169,6 +185,11 @@ fn text_for(cluster: usize, i: usize, rng: &mut Rng) -> String {
 // Measurement helpers
 // ---------------------------------------------------------------------------
 
+/// Exact cosine distance between unit-length vectors.
+fn exact_dist(a: &[f32], b: &[f32]) -> f32 {
+    1.0 - a.iter().zip(b).map(|(x, y)| x * y).sum::<f32>()
+}
+
 fn exact_top_k(vectors: &[Vec<f32>], query: &[f32], k: usize) -> Vec<usize> {
     // Vectors are unit length, so cosine order == dot-product order.
     let mut scored: Vec<(usize, f32)> = vectors
@@ -198,6 +219,83 @@ fn summarize(mut samples: Vec<Duration>) -> Latency {
     }
 }
 
+/// Counts live heap bytes, so a structure's cost can be measured by
+/// difference.
+///
+/// RSS cannot do this from inside one process: freeing a large structure
+/// returns its pages to the allocator's pool rather than to the OS, so
+/// allocating the next one shows no change. Measured that way, a store that
+/// holds the corpus twice and one that holds it once look identical.
+struct CountingAllocator;
+
+static LIVE_BYTES: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0);
+
+/// High-water mark of [`LIVE_BYTES`] since it was last reset.
+///
+/// Live bytes at a checkpoint cannot see a buffer that was allocated and
+/// freed in between, and that is exactly the shape of a transient copy —
+/// which still has to fit in memory while it exists.
+static PEAK_BYTES: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0);
+
+fn note_peak(live: i64) {
+    PEAK_BYTES.fetch_max(live, std::sync::atomic::Ordering::Relaxed);
+}
+
+// SAFETY: every method forwards to the system allocator with the same layout
+// it was given, and only adds bookkeeping around it.
+unsafe impl std::alloc::GlobalAlloc for CountingAllocator {
+    unsafe fn alloc(&self, layout: std::alloc::Layout) -> *mut u8 {
+        let ptr = unsafe { std::alloc::System.alloc(layout) };
+        if !ptr.is_null() {
+            let live = LIVE_BYTES
+                .fetch_add(layout.size() as i64, std::sync::atomic::Ordering::Relaxed)
+                + layout.size() as i64;
+            note_peak(live);
+        }
+        ptr
+    }
+
+    unsafe fn dealloc(&self, ptr: *mut u8, layout: std::alloc::Layout) {
+        LIVE_BYTES.fetch_sub(layout.size() as i64, std::sync::atomic::Ordering::Relaxed);
+        unsafe { std::alloc::System.dealloc(ptr, layout) }
+    }
+
+    unsafe fn realloc(&self, ptr: *mut u8, layout: std::alloc::Layout, new_size: usize) -> *mut u8 {
+        let new_ptr = unsafe { std::alloc::System.realloc(ptr, layout, new_size) };
+        if !new_ptr.is_null() {
+            let delta = new_size as i64 - layout.size() as i64;
+            let live = LIVE_BYTES.fetch_add(delta, std::sync::atomic::Ordering::Relaxed) + delta;
+            note_peak(live);
+        }
+        new_ptr
+    }
+}
+
+#[global_allocator]
+static ALLOCATOR: CountingAllocator = CountingAllocator;
+
+/// Live heap bytes right now.
+fn heap_bytes() -> u64 {
+    LIVE_BYTES.load(std::sync::atomic::Ordering::Relaxed).max(0) as u64
+}
+
+/// Start watching for a new high-water mark from the current live total.
+fn reset_peak() {
+    PEAK_BYTES.store(
+        LIVE_BYTES.load(std::sync::atomic::Ordering::Relaxed),
+        std::sync::atomic::Ordering::Relaxed,
+    );
+}
+
+/// The highest live total seen since [`reset_peak`].
+fn peak_bytes() -> u64 {
+    PEAK_BYTES.load(std::sync::atomic::Ordering::Relaxed).max(0) as u64
+}
+
+fn mib(bytes: u64) -> f64 {
+    bytes as f64 / (1 << 20) as f64
+}
+
 fn micros(d: Duration) -> f64 {
     d.as_secs_f64() * 1e6
 }
@@ -219,11 +317,12 @@ fn bench_ann(n: usize, json: &mut Vec<serde_json::Value>) {
         .collect();
 
     let started = Instant::now();
-    let index = HnswIndex::build_with_metric(
+    let index = HnswIndex::build_with(
         &data.vectors,
         HNSW_M,
         HNSW_EF_CONSTRUCTION,
         DistanceMetric::Cosine,
+        storage(),
     );
     let build = started.elapsed();
 
@@ -240,7 +339,8 @@ fn bench_ann(n: usize, json: &mut Vec<serde_json::Value>) {
     );
 
     println!(
-        "\n### HNSW, N = {n}, dim = {DIM}, M = {HNSW_M}, ef_construction = {HNSW_EF_CONSTRUCTION}\n"
+        "\n### HNSW, N = {n}, dim = {DIM}, M = {HNSW_M}, ef_construction = {HNSW_EF_CONSTRUCTION}, storage = {:?}\n",
+        index.storage()
     );
     println!(
         "build: {:.1} ms ({:.0} vectors/s) · exact scan: {:.0} QPS, p50 {:.0} µs\n",
@@ -251,12 +351,26 @@ fn bench_ann(n: usize, json: &mut Vec<serde_json::Value>) {
     );
     println!("| ef | recall@{K} | QPS | p50 µs | p99 µs |");
     println!("|---:|---:|---:|---:|---:|");
+    // With a quantised index the distances it returns are approximate, so
+    // the candidates are re-scored against the exact vectors the caller
+    // already holds (in the agent, the embedding cache) before taking the
+    // top K. `--rerank` prices that: it costs one exact distance per
+    // candidate and is what decides whether int8 is usable.
+    let rerank = RERANK.load(std::sync::atomic::Ordering::Relaxed);
+    let pool = if rerank { K * 4 } else { K };
     for ef in EF_VALUES {
         let mut hits = 0usize;
         let mut samples = Vec::with_capacity(data.queries.len());
         for (q, want) in data.queries.iter().zip(&truth) {
             let t = Instant::now();
-            let got = index.search(q, K, ef);
+            let mut got = index.search(q, pool, ef.max(pool));
+            if rerank {
+                for cand in &mut got {
+                    cand.1 = exact_dist(&data.vectors[cand.0], q);
+                }
+                got.select_nth_unstable_by(K - 1, |a, b| a.1.total_cmp(&b.1));
+                got.truncate(K);
+            }
             samples.push(t.elapsed());
             hits += got.iter().filter(|(id, _)| want.contains(id)).count();
         }
@@ -394,11 +508,12 @@ fn fusion_study(n: usize) {
         .map(|(i, c)| text_for(*c, i, &mut rng))
         .collect();
     let bm25 = BM25Index::build(&texts, &vec![0u8; n]);
-    let index = HnswIndex::build_with_metric(
+    let index = HnswIndex::build_with(
         &data.vectors,
         HNSW_M,
         HNSW_EF_CONSTRUCTION,
         DistanceMetric::Cosine,
+        storage(),
     );
 
     let vec_pool = (K * 8).max(64);
@@ -450,6 +565,69 @@ fn fusion_study(n: usize) {
     }
 }
 
+/// What an in-memory store costs, stage by stage. The vectors are the floor:
+/// everything above it is bookkeeping that could in principle be shared.
+fn bench_footprint(n: usize) {
+    let data = make_dataset(n, 0xF007 ^ n as u64);
+    let mut rng = Rng(11);
+    let dir = tempfile::TempDir::new().unwrap();
+    let path = dir.path().join("footprint.h5");
+
+    let base = heap_bytes();
+    let entries: Vec<MemoryEntry> = data
+        .vectors
+        .iter()
+        .enumerate()
+        .map(|(i, v)| MemoryEntry {
+            chunk: text_for(data.cluster_of[i], i, &mut rng),
+            embedding: v.clone(),
+            source_channel: "bench".into(),
+            timestamp: i as f64,
+            session_id: format!("s{}", i % 50),
+            tags: format!("t{i}"),
+        })
+        .collect();
+    let after_entries = heap_bytes();
+
+    let mut config = MemoryConfig::new(path, "bench", DIM);
+    config.quantized_index = INT8.load(std::sync::atomic::Ordering::Relaxed);
+    let mut mem = HDF5Memory::create(config).unwrap();
+    mem.save_batch(entries).unwrap();
+    let after_store = heap_bytes();
+
+    // First query builds the vector and keyword indexes.
+    std::hint::black_box(mem.hybrid_search(&data.queries[0], "record", 0.7, 0.3, K));
+    let after_indexes = heap_bytes();
+
+    // Reopening is the figure that matters for a long-lived process, and the
+    // only one RSS reports honestly: memory freed when the ingest buffers went
+    // away stays in the allocator's pool, so the stage deltas above understate
+    // what was given back.
+    let path = mem.config().path.clone();
+    drop(mem);
+    let before_open = heap_bytes();
+    reset_peak();
+    let reopened = HDF5Memory::open(&path).unwrap();
+    let after_open = heap_bytes();
+    let loaded = after_open.saturating_sub(before_open);
+    // Peak over the open, not just what it leaves behind: a buffer allocated
+    // and freed during the parse never shows up in the live total.
+    let peak = peak_bytes().saturating_sub(before_open);
+    drop(reopened);
+
+    let raw = (n * DIM * 4) as u64;
+    println!(
+        "| {n} | {:.0} | {:.0} | {:.0} | {:.0} | {:.0} | {:.0} | {:.2}x |",
+        mib(raw),
+        mib(after_entries.saturating_sub(base)),
+        mib(after_store.saturating_sub(after_entries)),
+        mib(after_indexes.saturating_sub(after_store)),
+        mib(loaded),
+        mib(peak),
+        loaded as f64 / raw as f64,
+    );
+}
+
 fn main() {
     let args: Vec<String> = std::env::args().skip(1).collect();
     let full = args.iter().any(|a| a == "--full");
@@ -464,6 +642,14 @@ fn main() {
         }
         return;
     }
+    if args.iter().any(|a| a == "--int8") {
+        INT8.store(true, std::sync::atomic::Ordering::Relaxed);
+        println!("(int8-quantised index vectors)");
+    }
+    if args.iter().any(|a| a == "--rerank") {
+        RERANK.store(true, std::sync::atomic::Ordering::Relaxed);
+        println!("(candidates re-scored against exact vectors)");
+    }
     if args.iter().any(|a| a == "--uniform") {
         UNIFORM.store(true, std::sync::atomic::Ordering::Relaxed);
         println!("(uniform random data)");
@@ -485,9 +671,25 @@ fn main() {
 
     let mut json = Vec::new();
     println!("## Search harness");
+
+    if args.iter().any(|a| a == "--footprint") {
+        println!("\n### Resident memory, {DIM}-dim f32\n");
+        println!(
+            "| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw |"
+        );
+        println!("|---:|---:|---:|---:|---:|---:|---:|---:|");
+        for &n in sizes {
+            bench_footprint(n);
+        }
+        return;
+    }
+    // `--e2e-only` skips the index benchmarks, so the end-to-end section runs
+    // in a process that has not already spun up a thread pool.
+    if !args.iter().any(|a| a == "--e2e-only") {
         for &n in sizes {
             bench_ann(n, &mut json);
         }
+    }
 
     if ann_only {
         return;

crates/clawhdf5-cli/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-cli"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 license = "MIT"
 description = "CLI for clawhdf5 agent memory — create, save, search, recall, stats"
@@ -14,7 +14,7 @@ name = "clawhdf5"
 path = "src/main.rs"
 
 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.4.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
 clap = { version = "4", features = ["derive", "env"] }
 serde_json = "1"
 serde = { workspace = true }

crates/clawhdf5-cli/src/main.rs

@@ -28,6 +28,10 @@ enum Commands {
         /// Enable write-ahead log
         #[arg(long)]
         wal: bool,
+        /// Store the vector index's copy of the embeddings as int8, roughly
+        /// halving a loaded store's memory at about 13% fewer queries/second
+        #[arg(long)]
+        quantized_index: bool,
     },
     /// Save a memory entry (reads JSON from stdin or --json)
     Save {
@@ -88,9 +92,15 @@ fn main() {
 
 fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
     match cli.command {
-        Commands::Create { agent_id, dim, wal } => {
+        Commands::Create {
+            agent_id,
+            dim,
+            wal,
+            quantized_index,
+        } => {
             let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim);
             config.wal_enabled = wal;
+            config.quantized_index = quantized_index;
             let mem = HDF5Memory::create(config)?;
             let j = serde_json::json!({
                 "status": "created",
@@ -98,6 +108,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
                 "agent_id": agent_id,
                 "embedding_dim": dim,
                 "wal_enabled": wal,
+                "quantized_index": quantized_index,
                 "count": mem.count(),
             });
             println!("{}", serde_json::to_string_pretty(&j)?);

crates/clawhdf5-derive/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-derive"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "Derive macros for rustyhdf5 HDF5 traits"
 license = "MIT"

crates/clawhdf5-filters/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-filters"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "Filter and compression pipeline for clawhdf5"
 license = "MIT"

crates/clawhdf5-format/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-format"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "Pure-Rust HDF5 binary format parsing and writing — no C dependencies"
 license = "MIT"
@@ -25,7 +25,7 @@ pco = { version = "1.0", optional = true }
 [dev-dependencies]
 serde_json = "1"
 criterion = { workspace = true }
-clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.4.0" }
+clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.7.0" }
 
 [[bench]]
 name = "bench"

crates/clawhdf5-format/fuzz/.gitignore

@@ -1 +1,4 @@
 target/
+corpus/
+artifacts/
+coverage/

crates/clawhdf5-format/fuzz/fuzz_targets/fuzz_btree_v2.rs

@@ -1,15 +1,36 @@
 #![no_main]
+use clawhdf5_format::btree_v2::{BTreeV2Header, collect_btree_v2_records};
 use libfuzzer_sys::fuzz_target;
 
 fuzz_target!(|data: &[u8]| {
     for &offset_size in &[4u8, 8] {
         for &length_size in &[4u8, 8] {
-            let _ = clawhdf5_format::btree_v2::BTreeV2Header::parse(
-                data,
-                0,
-                offset_size,
-                length_size,
-            );
+            if let Ok(header) = BTreeV2Header::parse(data, 0, offset_size, length_size) {
+                let _ = collect_btree_v2_records(data, &header, offset_size, length_size);
             }
         }
+    }
+
+    // Parsing a header requires a valid checksum, which random input almost
+    // never has, so the traversal behind it went unfuzzed — and that is where
+    // a node listing itself as its own child overflowed the stack. Take the
+    // header fields straight from the input instead and walk the rest.
+    let Some((fields, file)) = data.split_first_chunk::<20>() else {
+        return;
+    };
+    let header = BTreeV2Header {
+        tree_type: fields[0],
+        node_size: u32::from_le_bytes([fields[1], fields[2], fields[3], fields[4]]),
+        record_size: u16::from_le_bytes([fields[5], fields[6]]),
+        depth: u16::from_le_bytes([fields[7], fields[8]]),
+        root_node_address: u64::from(u32::from_le_bytes([
+            fields[9], fields[10], fields[11], fields[12],
+        ])),
+        num_records_in_root: u16::from_le_bytes([fields[13], fields[14]]),
+        total_records: u64::from(u32::from_le_bytes([
+            fields[15], fields[16], fields[17], fields[18],
+        ])),
+    };
+    let offset_size = if fields[19] & 1 == 0 { 4 } else { 8 };
+    let _ = collect_btree_v2_records(file, &header, offset_size, 8);
 });

crates/clawhdf5-format/src/btree_v2.rs

@@ -172,6 +172,17 @@ fn max_records_leaf(node_size: u32, record_size: u16) -> u64 {
     ((node_size - overhead) / record_size as u32) as u64
 }
 
+/// Deepest B-tree v2 accepted. See [`collect_btree_v2_records`].
+const MAX_DEPTH: u16 = 64;
+
+/// Take `n` records from the traversal's budget, or refuse the tree.
+fn spend(budget: &mut usize, n: usize) -> Result<(), FormatError> {
+    *budget = budget
+        .checked_sub(n)
+        .ok_or(FormatError::NestingDepthExceeded)?;
+    Ok(())
+}
+
 /// Collect all records from a B-tree v2 by traversing from the root.
 pub fn collect_btree_v2_records(
     file_data: &[u8],
@@ -182,6 +193,22 @@ pub fn collect_btree_v2_records(
     if header.total_records == 0 || header.num_records_in_root == 0 {
         return Ok(Vec::new());
     }
+    // Recursion is one frame per level, and the depth is read from the file:
+    // a crafted header claiming 65 535 levels over a node that is its own
+    // child overflowed the stack. 64 matches the fractal heap's guard, and no
+    // real tree comes close — even at the minimum fan-out of two it would
+    // hold more than 2^64 records.
+    if header.depth > MAX_DEPTH {
+        return Err(FormatError::NestingDepthExceeded);
+    }
+    // A valid tree stores each record once, in its own bytes, so it cannot
+    // hold more records than the file has room for. Children are addresses,
+    // though, and nothing makes them distinct: levels whose children all
+    // point at one shared node below reach it fan-out^depth times, which is
+    // millions of records from a few kilobytes. Counting against what the
+    // file could physically contain bounds that without trusting the
+    // header's own `total_records`.
+    let mut budget = file_data.len() / usize::from(header.record_size.max(1));
 
     let max_leaf_nrec = max_records_leaf(header.node_size, header.record_size);
 
@@ -206,6 +233,7 @@ pub fn collect_btree_v2_records(
             offset_size,
             length_size,
             max_leaf_nrec,
+            &mut budget,
             &mut records,
         )?;
         Ok(records)
@@ -273,6 +301,7 @@ fn collect_internal_records(
     offset_size: u8,
     length_size: u8,
     max_leaf_nrec: u64,
+    budget: &mut usize,
     out: &mut Vec<BTreeV2Record>,
 ) -> Result<(), FormatError> {
     // signature(4) + version(1) + type(1) = 6
@@ -350,6 +379,8 @@ fn collect_internal_records(
     // We collect child[0] records, then record[0], then child[1], etc.
     for (i, &(child_addr, child_nrec)) in children.iter().enumerate() {
         if child_depth == 0 {
+            // Before parsing, so a refused tree is not also a large allocation.
+            spend(budget, usize::from(child_nrec))?;
             let leaf_recs =
                 parse_leaf_records(file_data, child_addr as usize, child_nrec, record_size)?;
             out.extend(leaf_recs);
@@ -364,6 +395,7 @@ fn collect_internal_records(
                 offset_size,
                 length_size,
                 max_leaf_nrec,
+                budget,
                 out,
             )?;
         }
@@ -393,6 +425,7 @@ fn collect_internal_records(
                     available: file_data.len(),
                 });
             }
+            spend(budget, 1)?;
             out.push(BTreeV2Record {
                 data: file_data[rec_start..rec_end].to_vec(),
             });
@@ -466,6 +499,124 @@ mod tests {
         buf
     }
 
+    /// An internal node laid out exactly as `collect_internal_records` will
+    /// read it at `depth`: `records` zeroed records, then `children` pointers,
+    /// all to `child_addr` claiming `child_nrec` records.
+    fn internal_node(
+        depth: u16,
+        node_size: u32,
+        record_size: u16,
+        records: usize,
+        children: usize,
+        child_addr: u64,
+        child_nrec: u64,
+    ) -> Vec<u8> {
+        let max_leaf = max_records_leaf(node_size, record_size);
+        let nrec_width = bytes_for_max_records(if depth == 1 { max_leaf } else { max_leaf * 2 });
+        let total_width = if depth > 1 {
+            bytes_for_max_records(header_max_total_records(max_leaf, depth - 1))
+        } else {
+            0
+        };
+        let mut buf = b"BTIN".to_vec();
+        buf.extend_from_slice(&[0, 5]);
+        buf.resize(buf.len() + records * record_size as usize, 0);
+        for _ in 0..children {
+            buf.extend_from_slice(&child_addr.to_le_bytes());
+            buf.extend_from_slice(&child_nrec.to_le_bytes()[..nrec_width]);
+            buf.resize(buf.len() + total_width, 0);
+        }
+        buf
+    }
+
+    fn header(depth: u16, root: u64, root_nrec: u16, total: u64) -> BTreeV2Header {
+        BTreeV2Header {
+            tree_type: 5,
+            node_size: 512,
+            record_size: 8,
+            depth,
+            root_node_address: root,
+            num_records_in_root: root_nrec,
+            total_records: total,
+        }
+    }
+
+    #[test]
+    fn a_node_that_is_its_own_child_is_rejected_not_recursed() {
+        // One internal node whose two children are itself, under a header
+        // claiming the deepest tree a u16 allows. The layout stops depending
+        // on depth once the subtree-total width saturates, so every level
+        // parses cleanly and recursion runs ~65 000 frames deep: before the
+        // cap this overflowed the stack and aborted the process, from a file
+        // of under 100 bytes.
+        let mut data = internal_node(u16::MAX, 512, 8, 1, 2, 0, 1);
+        data.resize(4096, 0);
+        let result = collect_btree_v2_records(&data, &header(u16::MAX, 0, 1, 1), 8, 8);
+        assert!(result.is_err(), "{result:?}");
+    }
+
+    #[test]
+    fn a_shared_subtree_cannot_multiply_the_work() {
+        // A chain of distinct levels, each node's children all pointing at the
+        // single node below, ending in a real leaf. Every node parses and
+        // nothing is cyclic, yet the leaf is reached fan-out^depth times: 62
+        // children over 4 levels is ~15 million leaf visits from a few
+        // kilobytes. A valid tree cannot hold more records than the file has
+        // room for, so that bounds the traversal instead.
+        let (node_size, record_size) = (512u32, 8u16);
+        let fanout = 62usize;
+        let depth = 4u16;
+        let leaf = build_leaf_node(5, &[&[0u8; 8][..]]);
+
+        // Lay out root first, then each lower level, then the leaf.
+        let mut nodes: Vec<Vec<u8>> = Vec::new();
+        let mut addrs = Vec::new();
+        let mut at = 0u64;
+        let mut sizes = Vec::new();
+        for d in (1..=depth).rev() {
+            let n = internal_node(d, node_size, record_size, fanout - 1, fanout, 0, 0);
+            sizes.push(n.len());
+        }
+        for size in &sizes {
+            addrs.push(at);
+            at += *size as u64;
+        }
+        let leaf_addr = at;
+        for (i, d) in (1..=depth).rev().enumerate() {
+            let (child, child_nrec) = if d == 1 {
+                (leaf_addr, 1)
+            } else {
+                (addrs[i + 1], fanout as u64 - 1)
+            };
+            nodes.push(internal_node(
+                d,
+                node_size,
+                record_size,
+                fanout - 1,
+                fanout,
+                child,
+                child_nrec,
+            ));
+        }
+        let mut data: Vec<u8> = nodes.concat();
+        data.extend_from_slice(&leaf);
+        data.resize(data.len() + 64, 0);
+
+        let started = std::time::Instant::now();
+        let result =
+            collect_btree_v2_records(&data, &header(depth, 0, fanout as u16 - 1, u64::MAX), 8, 8);
+        assert!(
+            result.is_err(),
+            "expected a refusal, got {} records",
+            result.map_or(0, |r| r.len())
+        );
+        assert!(
+            started.elapsed() < std::time::Duration::from_secs(2),
+            "took {:?}",
+            started.elapsed()
+        );
+    }
+
     #[test]
     fn parse_header() {
         let data = build_btree_v2_header(5, 512, 11, 0, 0x1000, 3, 3, 8, 8);

crates/clawhdf5-format/src/chunk_cache.rs

@@ -374,6 +374,11 @@ impl ChunkCache {
 
     // ----- Index operations -----
 
+    /// The most decompressed bytes this cache will hold.
+    pub fn max_bytes(&self) -> usize {
+        self.inner.lock().map(|g| g.max_bytes).unwrap_or(0)
+    }
+
     /// Bind the cache to the dataset at chunk-index address `addr`.
     ///
     /// The cache is shared per file across all of its datasets. If the cache

crates/clawhdf5-format/src/chunked_read.rs

@@ -165,12 +165,29 @@ pub(crate) fn checked_chunk_byte_len(
 /// process when the allocation fails; a size taken from the file must surface
 /// as an error instead.
 pub(crate) fn alloc_output(len: usize) -> Result<Vec<u8>, FormatError> {
-    let mut out = Vec::new();
-    out.try_reserve_exact(len).map_err(|_| {
-        FormatError::Overflow(format!("cannot allocate {len} bytes for dataset output"))
-    })?;
-    out.resize(len, 0);
-    Ok(out)
+    if len == 0 {
+        return Ok(Vec::new());
+    }
+    let failed =
+        || FormatError::Overflow(format!("cannot allocate {len} bytes for dataset output"));
+    let layout = core::alloc::Layout::array::<u8>(len).map_err(|_| failed())?;
+    // Ask the allocator for zeroed memory instead of reserving and then
+    // writing zeros: for a large buffer the OS hands out already-zero pages
+    // lazily, where an explicit fill touches every page up front — and most of
+    // the buffer is about to be overwritten with chunk data anyway.
+    //
+    // SAFETY (both arms): `layout` has non-zero size (len > 0) and alignment 1.
+    #[cfg(feature = "std")]
+    let ptr = unsafe { std::alloc::alloc_zeroed(layout) };
+    #[cfg(not(feature = "std"))]
+    let ptr = unsafe { alloc::alloc::alloc_zeroed(layout) };
+    if ptr.is_null() {
+        return Err(failed());
+    }
+    // SAFETY: `ptr` came from the global allocator with the layout of
+    // `[u8; len]`, which is exactly what `Vec<u8>` with capacity `len` frees;
+    // all `len` bytes are initialised (zero).
+    Ok(unsafe { Vec::from_raw_parts(ptr, len, len) })
 }
 
 fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
@@ -362,6 +379,116 @@ pub fn generate_implicit_chunks(
 }
 
 /// Read a chunked dataset, decompressing chunks as needed.
+/// Chunks decompressed together before being copied out, bounding the extra
+/// memory a parallel full read holds at once.
+const DECODE_BATCH: usize = 128;
+
+/// B-tree v2 record types used for chunk indexing.
+const BT2_CHUNK_UNFILTERED: u8 = 10;
+const BT2_CHUNK_FILTERED: u8 = 11;
+
+/// Chunks indexed by a version-2 B-tree (layout v4, index type 5).
+///
+/// Record layouts (all little endian):
+/// * type 10, unfiltered: address, then one 8-byte *scaled* offset per
+///   dimension (offset / chunk dimension);
+/// * type 11, filtered: address, stored chunk size (a variable number of
+///   bytes), 4-byte filter mask, then the scaled offsets.
+///
+/// The width of the stored-size field depends on the largest possible chunk;
+/// rather than re-derive the library's formula it is taken from the record
+/// size the tree header declares, which is what actually governs the bytes.
+fn read_btree_v2_chunks(
+    file_data: &[u8],
+    addr: u64,
+    chunk_dims: &[usize],
+    elem_size: usize,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<ChunkInfo>, FormatError> {
+    use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
+
+    let bad = |what: &str| FormatError::ChunkedReadError(format!("B-tree v2 chunk index: {what}"));
+    let header = BTreeV2Header::parse(file_data, addr as usize, offset_size, length_size)?;
+    let rank = chunk_dims.len();
+    let os = offset_size as usize;
+    let record_size = header.record_size as usize;
+    let size_len = match header.tree_type {
+        BT2_CHUNK_UNFILTERED => {
+            if record_size != os + 8 * rank {
+                return Err(bad("unexpected record size for unfiltered chunks"));
+            }
+            0
+        }
+        BT2_CHUNK_FILTERED => {
+            let fixed = os + 4 + 8 * rank;
+            let size_len = record_size
+                .checked_sub(fixed)
+                .ok_or_else(|| bad("record too small"))?;
+            if !(1..=8).contains(&size_len) {
+                return Err(bad("implausible chunk-size field width"));
+            }
+            size_len
+        }
+        _ => return Err(bad("tree is not a chunk index")),
+    };
+    let unfiltered_bytes = checked_chunk_byte_len(chunk_dims, elem_size)?;
+    let unfiltered_bytes =
+        u32::try_from(unfiltered_bytes).map_err(|_| bad("chunk larger than 4 GiB"))?;
+
+    let records = collect_btree_v2_records(file_data, &header, offset_size, length_size)?;
+    let mut chunks = Vec::with_capacity(records.len());
+    for record in &records {
+        let data = record.data.as_slice();
+        if data.len() < record_size {
+            return Err(bad("truncated record"));
+        }
+        let address = read_offset(data, 0, offset_size)?;
+        let mut pos = os;
+        let (chunk_size, filter_mask) = if size_len == 0 {
+            (unfiltered_bytes, 0)
+        } else {
+            let mut size = 0u64;
+            for (i, &b) in data[pos..pos + size_len].iter().enumerate() {
+                size |= u64::from(b) << (8 * i);
+            }
+            pos += size_len;
+            let mask = u32::from_le_bytes([data[pos], data[pos + 1], data[pos + 2], data[pos + 3]]);
+            pos += 4;
+            (
+                u32::try_from(size).map_err(|_| bad("stored chunk larger than 4 GiB"))?,
+                mask,
+            )
+        };
+        let mut offsets = Vec::with_capacity(rank);
+        for &dim in chunk_dims {
+            let scaled = u64::from_le_bytes([
+                data[pos],
+                data[pos + 1],
+                data[pos + 2],
+                data[pos + 3],
+                data[pos + 4],
+                data[pos + 5],
+                data[pos + 6],
+                data[pos + 7],
+            ]);
+            pos += 8;
+            offsets.push(
+                scaled
+                    .checked_mul(dim as u64)
+                    .ok_or_else(|| bad("chunk offset overflows"))?,
+            );
+        }
+        chunks.push(ChunkInfo {
+            chunk_size,
+            filter_mask,
+            offsets,
+            address,
+        });
+    }
+    Ok(chunks)
+}
+
 /// Every allocated chunk of a chunked dataset, for any supported chunk index,
 /// plus the spatial chunk dimensions. Chunks the file never allocated (sparse
 /// datasets) are simply absent from the list.
@@ -487,6 +614,18 @@ pub fn list_chunks(
                 length_size,
             )?
         }
+        (4, Some(5)) => {
+            // Version-2 B-tree: what the library uses for a dataset with two
+            // or more unlimited dimensions.
+            read_btree_v2_chunks(
+                file_data,
+                addr,
+                &chunk_dims,
+                elem_size,
+                offset_size,
+                length_size,
+            )?
+        }
         (v, idx) => {
             return Err(FormatError::ChunkedReadError(format!(
                 "unsupported chunked layout version={v}, index_type={idx:?}"
@@ -629,29 +768,12 @@ pub fn read_chunked_data_cached(
     length_size: u8,
     cache: &ChunkCache,
 ) -> Result<Vec<u8>, FormatError> {
-    let (
-        chunk_dimensions,
-        version,
-        chunk_index_type,
-        addr_opt,
-        single_filtered_size,
-        single_filter_mask,
-    ) = match layout {
+    let (chunk_dimensions, addr_opt) = match layout {
         DataLayout::Chunked {
             chunk_dimensions,
             btree_address,
-            version,
-            chunk_index_type,
-            single_chunk_filtered_size,
-            single_chunk_filter_mask,
-        } => (
-            chunk_dimensions,
-            *version,
-            *chunk_index_type,
-            *btree_address,
-            *single_chunk_filtered_size,
-            *single_chunk_filter_mask,
-        ),
+            ..
+        } => (chunk_dimensions, *btree_address),
         _ => {
             return Err(FormatError::ChunkedReadError(
                 "expected chunked layout".into(),
@@ -688,69 +810,14 @@ pub fn read_chunked_data_cached(
 
     // Populate chunk index on first access
     if !cache.has_index() {
-        let chunks = match (version, chunk_index_type) {
-            (3, _) => collect_chunk_info(file_data, addr, ndims, offset_size, length_size)?,
-            (4, Some(1)) => {
-                let chunk_byte_size = checked_chunk_byte_len(&chunk_dims, elem_size)?;
-                let (csize, fmask) = if let Some(fs) = single_filtered_size {
-                    (fs as u32, single_filter_mask.unwrap_or(0))
-                } else {
-                    (chunk_byte_size as u32, 0)
-                };
-                vec![ChunkInfo {
-                    chunk_size: csize,
-                    filter_mask: fmask,
-                    offsets: vec![0u64; rank],
-                    address: addr,
-                }]
-            }
-            (4, Some(2)) => {
-                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-                generate_implicit_chunks(
-                    addr,
-                    &dataspace.dimensions,
-                    spatial_chunk_dims,
-                    elem_size as u32,
-                )
-            }
-            (4, Some(3)) => {
-                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-                let header =
-                    FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
-                read_fixed_array_chunks(
+        let (chunks, _) = list_chunks(
             file_data,
-                    &header,
-                    &dataspace.dimensions,
-                    spatial_chunk_dims,
-                    elem_size as u32,
-                    offset_size,
-                    length_size,
-                )?
-            }
-            (4, Some(4)) => {
-                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-                let header = ExtensibleArrayHeader::parse(
-                    file_data,
-                    addr as usize,
+            layout,
+            dataspace,
+            elem_size,
             offset_size,
             length_size,
         )?;
-                read_extensible_array_chunks(
-                    file_data,
-                    &header,
-                    &dataspace.dimensions,
-                    spatial_chunk_dims,
-                    elem_size as u32,
-                    offset_size,
-                    length_size,
-                )?
-            }
-            (v, idx) => {
-                return Err(FormatError::ChunkedReadError(format!(
-                    "unsupported chunked layout version={v}, index_type={idx:?}"
-                )));
-            }
-        };
         cache.populate_index(&chunks, rank);
     }
 
@@ -777,43 +844,20 @@ pub fn read_chunked_data_cached(
 
     let chunk_total_bytes = checked_chunk_byte_len(&chunk_dims, elem_size)?;
 
-    for chunk_info in &chunks {
-        let coord: Vec<u64> = chunk_info.offsets.iter().take(rank).copied().collect();
-
-        // Try decompressed cache first
-        let decompressed = if let Some(cached) = cache.get_decompressed_aligned(&coord) {
-            cached
-        } else {
-            // Decompress from file
-            let c_addr = chunk_info.address as usize;
-            let size = chunk_info.chunk_size as usize;
-            ensure_len(file_data, c_addr, size)?;
-            let raw_chunk = &file_data[c_addr..c_addr + size];
-            let dec = if let Some(pl) = pipeline {
-                if chunk_info.filter_mask == 0 {
-                    decompress_chunk(raw_chunk, pl, chunk_total_bytes, elem_size as u32)?
-                } else {
-                    raw_chunk.to_vec()
+    let mut place = |data: &[u8], chunk_info: &ChunkInfo| {
+        if rank == 0 {
+            let copy_len = data.len().min(output.len());
+            output[..copy_len].copy_from_slice(&data[..copy_len]);
+            return;
         }
-            } else {
-                raw_chunk.to_vec()
-            };
-            cache.put_decompressed(coord, dec)
-        };
-
         let chunk_offsets: Vec<usize> = chunk_info
             .offsets
             .iter()
             .take(rank)
             .map(|&o| o as usize)
             .collect();
-
-        if rank == 0 {
-            let copy_len = decompressed.len().min(output.len());
-            output[..copy_len].copy_from_slice(&decompressed[..copy_len]);
-        } else {
         copy_chunk_to_output(
-                &decompressed,
+            data,
             &mut output,
             &chunk_offsets,
             &chunk_dims,
@@ -823,6 +867,63 @@ pub fn read_chunked_data_cached(
             elem_size,
             rank,
         );
+    };
+    let raw_bytes = |chunk_info: &ChunkInfo| -> Result<&[u8], FormatError> {
+        let c_addr = chunk_info.address as usize;
+        let size = chunk_info.chunk_size as usize;
+        ensure_len(file_data, c_addr, size)?;
+        Ok(&file_data[c_addr..c_addr + size])
+    };
+
+    // Chunks stored as-is (no pipeline, or the filter mask says this chunk
+    // skipped it) are copied straight from the file bytes: they are already in
+    // memory, so routing them through a Vec and then an aligned cache buffer
+    // was two extra copies of the whole dataset for nothing.
+    let stored_raw = |c: &ChunkInfo| pipeline.is_none() || c.filter_mask != 0;
+    let mut misses: Vec<&ChunkInfo> = Vec::new();
+    for chunk_info in &chunks {
+        if stored_raw(chunk_info) {
+            place(raw_bytes(chunk_info)?, chunk_info);
+            continue;
+        }
+        let coord: Vec<u64> = chunk_info.offsets.iter().take(rank).copied().collect();
+        match cache.get_decompressed_aligned(&coord) {
+            Some(cached) => place(&cached, chunk_info),
+            None => misses.push(chunk_info),
+        }
+    }
+
+    // Decompress what the cache didn't have, a bounded batch at a time — in
+    // parallel with the `parallel` feature (this path, the one the facade
+    // uses, was sequential; only the uncached reader was parallel). Chunks are
+    // cached only when the whole dataset fits: pushing a larger dataset
+    // through the cache just evicts each chunk moments after inserting it.
+    let cache_them = total_bytes <= cache.max_bytes();
+    if let Some(pl) = pipeline {
+        let decode = |c: &&ChunkInfo| -> Result<Vec<u8>, FormatError> {
+            decompress_chunk(raw_bytes(c)?, pl, chunk_total_bytes, elem_size as u32)
+        };
+        for batch in misses.chunks(DECODE_BATCH) {
+            #[cfg(feature = "parallel")]
+            let decoded: Vec<Result<Vec<u8>, FormatError>> = if batch.len() >= 4 {
+                use rayon::prelude::*;
+                batch.par_iter().map(decode).collect()
+            } else {
+                batch.iter().map(decode).collect()
+            };
+            #[cfg(not(feature = "parallel"))]
+            let decoded: Vec<Result<Vec<u8>, FormatError>> = batch.iter().map(decode).collect();
+
+            for (chunk_info, data) in batch.iter().zip(decoded) {
+                let data = data?;
+                if cache_them {
+                    let coord: Vec<u64> = chunk_info.offsets.iter().take(rank).copied().collect();
+                    let cached = cache.put_decompressed(coord, data);
+                    place(&cached, chunk_info);
+                } else {
+                    place(&data, chunk_info);
+                }
+            }
         }
     }
 
@@ -985,29 +1086,12 @@ pub fn read_chunked_data_sweep(
     cache: &ChunkCache,
     sweep: &mut SweepContext,
 ) -> Result<Vec<u8>, FormatError> {
-    let (
-        chunk_dimensions,
-        version,
-        chunk_index_type,
-        addr_opt,
-        single_filtered_size,
-        single_filter_mask,
-    ) = match layout {
+    let (chunk_dimensions, addr_opt) = match layout {
         DataLayout::Chunked {
             chunk_dimensions,
             btree_address,
-            version,
-            chunk_index_type,
-            single_chunk_filtered_size,
-            single_chunk_filter_mask,
-        } => (
-            chunk_dimensions,
-            *version,
-            *chunk_index_type,
-            *btree_address,
-            *single_chunk_filtered_size,
-            *single_chunk_filter_mask,
-        ),
+            ..
+        } => (chunk_dimensions, *btree_address),
         _ => {
             return Err(FormatError::ChunkedReadError(
                 "expected chunked layout".into(),
@@ -1044,69 +1128,14 @@ pub fn read_chunked_data_sweep(
 
     // Populate chunk index on first access
     if !cache.has_index() {
-        let chunks = match (version, chunk_index_type) {
-            (3, _) => collect_chunk_info(file_data, addr, ndims, offset_size, length_size)?,
-            (4, Some(1)) => {
-                let chunk_byte_size = checked_chunk_byte_len(&chunk_dims, elem_size)?;
-                let (csize, fmask) = if let Some(fs) = single_filtered_size {
-                    (fs as u32, single_filter_mask.unwrap_or(0))
-                } else {
-                    (chunk_byte_size as u32, 0)
-                };
-                vec![ChunkInfo {
-                    chunk_size: csize,
-                    filter_mask: fmask,
-                    offsets: vec![0u64; rank],
-                    address: addr,
-                }]
-            }
-            (4, Some(2)) => {
-                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-                generate_implicit_chunks(
-                    addr,
-                    &dataspace.dimensions,
-                    spatial_chunk_dims,
-                    elem_size as u32,
-                )
-            }
-            (4, Some(3)) => {
-                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-                let header =
-                    FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
-                read_fixed_array_chunks(
+        let (chunks, _) = list_chunks(
             file_data,
-                    &header,
-                    &dataspace.dimensions,
-                    spatial_chunk_dims,
-                    elem_size as u32,
-                    offset_size,
-                    length_size,
-                )?
-            }
-            (4, Some(4)) => {
-                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-                let header = ExtensibleArrayHeader::parse(
-                    file_data,
-                    addr as usize,
+            layout,
+            dataspace,
+            elem_size,
             offset_size,
             length_size,
         )?;
-                read_extensible_array_chunks(
-                    file_data,
-                    &header,
-                    &dataspace.dimensions,
-                    spatial_chunk_dims,
-                    elem_size as u32,
-                    offset_size,
-                    length_size,
-                )?
-            }
-            (v, idx) => {
-                return Err(FormatError::ChunkedReadError(format!(
-                    "unsupported chunked layout version={v}, index_type={idx:?}"
-                )));
-            }
-        };
         cache.populate_index(&chunks, rank);
     }
 
@@ -1211,29 +1240,12 @@ pub fn read_chunked_data_indexed(
     length_size: u8,
     cache: &ChunkCache,
 ) -> Result<Vec<u8>, FormatError> {
-    let (
-        chunk_dimensions,
-        version,
-        chunk_index_type,
-        addr_opt,
-        single_filtered_size,
-        single_filter_mask,
-    ) = match layout {
+    let (chunk_dimensions, addr_opt) = match layout {
         DataLayout::Chunked {
             chunk_dimensions,
             btree_address,
-            version,
-            chunk_index_type,
-            single_chunk_filtered_size,
-            single_chunk_filter_mask,
-        } => (
-            chunk_dimensions,
-            *version,
-            *chunk_index_type,
-            *btree_address,
-            *single_chunk_filtered_size,
-            *single_chunk_filter_mask,
-        ),
+            ..
+        } => (chunk_dimensions, *btree_address),
         _ => {
             return Err(FormatError::ChunkedReadError(
                 "expected chunked layout".into(),
@@ -1270,69 +1282,14 @@ pub fn read_chunked_data_indexed(
 
     // Build chunk index on first access
     if !cache.has_chunk_index() {
-        let chunks = match (version, chunk_index_type) {
-            (3, _) => collect_chunk_info(file_data, addr, ndims, offset_size, length_size)?,
-            (4, Some(1)) => {
-                let chunk_byte_size = checked_chunk_byte_len(&chunk_dims, elem_size)?;
-                let (csize, fmask) = if let Some(fs) = single_filtered_size {
-                    (fs as u32, single_filter_mask.unwrap_or(0))
-                } else {
-                    (chunk_byte_size as u32, 0)
-                };
-                vec![ChunkInfo {
-                    chunk_size: csize,
-                    filter_mask: fmask,
-                    offsets: vec![0u64; rank],
-                    address: addr,
-                }]
-            }
-            (4, Some(2)) => {
-                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-                generate_implicit_chunks(
-                    addr,
-                    &dataspace.dimensions,
-                    spatial_chunk_dims,
-                    elem_size as u32,
-                )
-            }
-            (4, Some(3)) => {
-                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-                let header =
-                    FixedArrayHeader::parse(file_data, addr as usize, offset_size, length_size)?;
-                read_fixed_array_chunks(
+        let (chunks, _) = list_chunks(
             file_data,
-                    &header,
-                    &dataspace.dimensions,
-                    spatial_chunk_dims,
-                    elem_size as u32,
-                    offset_size,
-                    length_size,
-                )?
-            }
-            (4, Some(4)) => {
-                let spatial_chunk_dims: &[u32] = &chunk_dimensions[..rank];
-                let header = ExtensibleArrayHeader::parse(
-                    file_data,
-                    addr as usize,
+            layout,
+            dataspace,
+            elem_size,
             offset_size,
             length_size,
         )?;
-                read_extensible_array_chunks(
-                    file_data,
-                    &header,
-                    &dataspace.dimensions,
-                    spatial_chunk_dims,
-                    elem_size as u32,
-                    offset_size,
-                    length_size,
-                )?
-            }
-            (v, idx) => {
-                return Err(FormatError::ChunkedReadError(format!(
-                    "unsupported chunked layout version={v}, index_type={idx:?}"
-                )));
-            }
-        };
         cache.populate_chunk_index(&chunks, rank);
         // Also populate the legacy index for compatibility
         if !cache.has_index() {
@@ -2284,21 +2241,23 @@ mod tests {
     }
 
     #[test]
-    fn cached_read_second_call_uses_cache() {
+    fn cached_read_second_call_reuses_the_index() {
         let values: Vec<f64> = (0..20).map(|i| i as f64).collect();
         let (file_data, layout, dataspace) = build_1d_chunked_file(&values, 10);
         let datatype = make_f64_type();
         let cache = ChunkCache::new();
 
-        // First read — populates index + decompressed cache
+        // First read — populates the chunk index. These chunks are stored
+        // unfiltered, so they are copied straight from the file bytes and the
+        // decompressed-chunk cache is (deliberately) not involved.
         let raw1 = read_chunked_data_cached(
             &file_data, &layout, &dataspace, &datatype, None, 8, 8, &cache,
         )
         .unwrap();
         assert!(cache.has_index());
-        assert!(cache.cached_chunk_count() > 0);
+        assert_eq!(cache.cached_chunk_count(), 0);
 
-        // Second read — should hit the decompressed cache
+        // Second read — reuses the cached index
         let raw2 = read_chunked_data_cached(
             &file_data, &layout, &dataspace, &datatype, None, 8, 8, &cache,
         )

crates/clawhdf5-format/src/chunked_write.rs

@@ -15,7 +15,6 @@ use crate::filter_pipeline::{
     FilterDescription, FilterPipeline,
 };
 use crate::filters::compress_chunk;
-
 /// Round a file offset up to the next cache-line boundary.
 ///
 /// This ensures chunk data starts at an address that is a multiple of the
@@ -49,6 +48,38 @@ pub struct ChunkOptions {
     pub pcodec: bool,
 }
 
+/// Largest chunk the automatic choice produces, in bytes.
+const AUTO_CHUNK_TARGET_BYTES: u64 = 1 << 20;
+
+/// Extent assumed for a dimension that is currently empty (an unlimited
+/// dimension not yet written to) — the same stand-in h5py uses.
+const AUTO_CHUNK_EMPTY_DIM: u64 = 1024;
+
+/// Choose chunk dimensions for a dataset nobody specified them for.
+///
+/// Asking for compression (or any filter) without chunk dimensions used to
+/// make the whole dataset one chunk. That defeats the point of chunking: any
+/// read — even a single row — must decompress everything, and a large dataset
+/// cannot be decompressed in parallel. Datasets up to the target size stay a
+/// single chunk, exactly as before; larger ones are split by halving the
+/// dimensions in turn (so chunks keep roughly the dataset's proportions, the
+/// approach h5py takes) until a chunk fits the target.
+pub fn auto_chunk_dims(shape: &[u64], elem_size: usize) -> Vec<u64> {
+    let mut dims: Vec<u64> = shape
+        .iter()
+        .map(|&d| if d == 0 { AUTO_CHUNK_EMPTY_DIM } else { d })
+        .collect();
+    let elem = elem_size.max(1) as u64;
+    let bytes = |dims: &[u64]| dims.iter().fold(elem, |acc, &d| acc.saturating_mul(d));
+    let mut axis = 0;
+    while bytes(&dims) > AUTO_CHUNK_TARGET_BYTES && dims.iter().any(|&d| d > 1) {
+        let i = axis % dims.len();
+        dims[i] = dims[i].div_ceil(2);
+        axis += 1;
+    }
+    dims
+}
+
 impl ChunkOptions {
     /// Whether any chunking option is enabled.
     pub fn is_chunked(&self) -> bool {
@@ -135,11 +166,17 @@ impl ChunkOptions {
 
     /// Determine chunk dimensions, using user-specified or auto-computing.
     pub fn resolve_chunk_dims(&self, shape: &[u64]) -> Vec<u64> {
-        if let Some(ref dims) = self.chunk_dims {
-            dims.clone()
-        } else {
-            // Auto chunk: use the full dataset shape (single chunk)
-            shape.to_vec()
+        // Without the element size, assume 8 bytes (the widest common scalar);
+        // the writer uses `resolve_chunk_dims_for`.
+        self.resolve_chunk_dims_for(shape, 8)
+    }
+
+    /// Chunk dimensions for a dataset of `shape` whose elements are `elem_size`
+    /// bytes: the caller's if given, otherwise chosen automatically.
+    pub fn resolve_chunk_dims_for(&self, shape: &[u64], elem_size: usize) -> Vec<u64> {
+        match self.chunk_dims {
+            Some(ref dims) => dims.clone(),
+            None => auto_chunk_dims(shape, elem_size),
         }
     }
 }
@@ -890,6 +927,7 @@ pub fn write_selection_to_buffer(
 
 #[cfg(test)]
 mod tests {
+
     use super::*;
     use crate::chunked_read::read_chunked_data;
     use crate::data_layout::DataLayout;
@@ -1143,6 +1181,45 @@ mod tests {
         assert_eq!(dims, vec![100, 50]);
     }
 
+    #[test]
+    fn auto_chunking_splits_only_large_datasets() {
+        let bytes = |dims: &[u64], elem: u64| dims.iter().product::<u64>() * elem;
+        // Up to the target: one chunk, as before.
+        assert_eq!(auto_chunk_dims(&[100, 50], 8), [100, 50]);
+        assert_eq!(auto_chunk_dims(&[131_072], 8), [131_072]); // exactly 1 MiB
+        // Larger: split, keeping proportions, never above the target.
+        let big = auto_chunk_dims(&[4096, 2048], 8);
+        assert!(bytes(&big, 8) <= AUTO_CHUNK_TARGET_BYTES, "{big:?}");
+        assert!(bytes(&big, 8) > AUTO_CHUNK_TARGET_BYTES / 4, "{big:?}");
+        assert_eq!(big[0] / big[1], 2, "proportions kept: {big:?}");
+        // Every dimension stays within the dataset and at least 1.
+        for shape in [
+            vec![10_000_000u64],
+            vec![3, 5_000_000],
+            vec![1, 1, 9_000_000],
+            vec![7; 9],
+        ] {
+            let dims = auto_chunk_dims(&shape, 4);
+            assert!(
+                dims.iter().zip(&shape).all(|(c, s)| *c >= 1 && c <= s),
+                "{shape:?} -> {dims:?}"
+            );
+            assert!(
+                bytes(&dims, 4) <= AUTO_CHUNK_TARGET_BYTES,
+                "{shape:?} -> {dims:?}"
+            );
+        }
+        // An empty (unlimited, unwritten) dimension still gets a usable chunk.
+        let growable = auto_chunk_dims(&[0, 128], 8);
+        assert!(growable[0] >= 1 && bytes(&growable, 8) <= AUTO_CHUNK_TARGET_BYTES);
+        // Explicit dimensions always win.
+        let explicit = ChunkOptions {
+            chunk_dims: Some(vec![10, 10]),
+            ..Default::default()
+        };
+        assert_eq!(explicit.resolve_chunk_dims_for(&[4096, 2048], 8), [10, 10]);
+    }
+
     #[test]
     fn chunk_options_pipeline_deflate() {
         // Auto-shuffle is applied before compression by default (matches h5py).
@@ -1435,9 +1512,20 @@ mod tests {
 
     // ---- h5py round-trip tests for chunked writes ----
 
+    /// The Python interpreter to drive interop checks with.
+    ///
+    /// `CLAWHDF5_PYTHON` lets these run against a virtualenv holding h5py,
+    /// which on a PEP 668 "externally managed" system is the only place it
+    /// can be installed. Without it the suite silently skips, and a silent
+    /// skip here is how a datatype bug once reached a release.
+    #[cfg(feature = "std")]
+    fn python() -> String {
+        std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+    }
+
     #[cfg(feature = "std")]
     fn h5py_available() -> bool {
-        std::process::Command::new("python3")
+        std::process::Command::new(python())
             .args(["-c", "import h5py"])
             .output()
             .map(|o| o.status.success())
@@ -1449,10 +1537,10 @@ mod tests {
         if !h5py_available() {
             panic!("h5py not installed — skipping interop test");
         }
-        let o = std::process::Command::new("python3")
+        let o = std::process::Command::new(python())
             .args(["-c", script])
             .output()
-            .expect("python3");
+            .expect("python interpreter");
         if !o.status.success() {
             panic!("h5py: {}", String::from_utf8_lossy(&o.stderr));
         }

crates/clawhdf5-format/src/data_read.rs

@@ -307,6 +307,24 @@ pub fn read_raw_data_selection(
 ) -> Result<Vec<u8>, FormatError> {
     use crate::selection::Selection;
 
+    crate::partial_read::validate(selection, &dataspace.dimensions)?;
+
+    // Read only what the selection's bounding box touches when that is
+    // possible; everything below is the decode-everything-then-pick path,
+    // kept for the cases `partial_read` declines.
+    if let Some(selected) = crate::partial_read::read_selection(
+        file_data,
+        layout,
+        dataspace,
+        datatype.type_size() as usize,
+        pipeline,
+        offset_size,
+        length_size,
+        selection,
+    )? {
+        return Ok(selected);
+    }
+
     match selection {
         Selection::All => {
             return read_raw_data_full(
@@ -858,6 +876,30 @@ fn get_size(dt: &Datatype) -> usize {
     dt.type_size() as usize
 }
 
+/// Reinterpret little-endian bytes as `count` native values of `T` on a
+/// little-endian target, in one copy.
+///
+/// The buffer is allocated uninitialised and filled by the copy. It used to be
+/// `vec![0; count]` first, which for a large dataset meant writing every page
+/// twice (zero it, then overwrite it) — about as expensive as the copy itself.
+#[cfg(target_endian = "little")]
+fn native_le_to_vec<T: Copy>(raw: &[u8], count: usize) -> Vec<T> {
+    let bytes = count * core::mem::size_of::<T>();
+    debug_assert!(bytes <= raw.len());
+    let mut result: Vec<T> = Vec::with_capacity(count);
+    // SAFETY: `result` has capacity for `count` values of `T`, i.e. `bytes`
+    // bytes; `raw` holds at least `bytes` bytes (callers derive `count` from
+    // `raw.len() / size_of::<T>()`); the regions cannot overlap because
+    // `result` was just allocated. Every `T` used here (f32/f64/i32/i64) is
+    // valid for any bit pattern, so after the copy all `count` values are
+    // initialised and `set_len` is sound.
+    unsafe {
+        core::ptr::copy_nonoverlapping(raw.as_ptr(), result.as_mut_ptr().cast::<u8>(), bytes);
+        result.set_len(count);
+    }
+    result
+}
+
 /// Convert raw bytes to `f64` values.
 pub fn read_as_f64(raw: &[u8], datatype: &Datatype) -> Result<Vec<f64>, FormatError> {
     // Array datatypes (e.g. an array-typed compound member) are read as a flat
@@ -885,14 +927,7 @@ pub fn read_as_f64(raw: &[u8], datatype: &Datatype) -> Result<Vec<f64>, FormatEr
             ..
         }
     ) {
-        let mut result = vec![0.0f64; count];
-        // SAFETY: On LE platforms, f64 in-memory representation matches LE bytes.
-        // We copy raw bytes directly into the f64 buffer.
-        // SAFETY: The byte slice is properly aligned for this type and the length is divisible by size_of::<T>().
-        unsafe {
-            core::ptr::copy_nonoverlapping(raw.as_ptr(), result.as_mut_ptr() as *mut u8, raw.len());
-        }
-        return Ok(result);
+        return Ok(native_le_to_vec::<f64>(raw, count));
     }
 
     let order = get_byte_order(datatype);
@@ -975,12 +1010,7 @@ pub fn read_as_i64(raw: &[u8], datatype: &Datatype) -> Result<Vec<i64>, FormatEr
             }
         )
     {
-        let mut result = vec![0i64; count];
-        // SAFETY: The byte slice is properly aligned for this type and the length is divisible by size_of::<T>().
-        unsafe {
-            core::ptr::copy_nonoverlapping(raw.as_ptr(), result.as_mut_ptr() as *mut u8, raw.len());
-        }
-        return Ok(result);
+        return Ok(native_le_to_vec::<i64>(raw, count));
     }
 
     let order = get_byte_order(datatype);
@@ -1044,12 +1074,7 @@ pub fn read_as_f32(raw: &[u8], datatype: &Datatype) -> Result<Vec<f32>, FormatEr
             ..
         }
     ) {
-        let mut result = vec![0.0f32; count];
-        // SAFETY: The byte slice is properly aligned for this type and the length is divisible by size_of::<T>().
-        unsafe {
-            core::ptr::copy_nonoverlapping(raw.as_ptr(), result.as_mut_ptr() as *mut u8, raw.len());
-        }
-        return Ok(result);
+        return Ok(native_le_to_vec::<f32>(raw, count));
     }
 
     let order = get_byte_order(datatype);
@@ -1126,12 +1151,7 @@ pub fn read_as_i32(raw: &[u8], datatype: &Datatype) -> Result<Vec<i32>, FormatEr
             }
         )
     {
-        let mut result = vec![0i32; count];
-        // SAFETY: The byte slice is properly aligned for this type and the length is divisible by size_of::<T>().
-        unsafe {
-            core::ptr::copy_nonoverlapping(raw.as_ptr(), result.as_mut_ptr() as *mut u8, raw.len());
-        }
-        return Ok(result);
+        return Ok(native_le_to_vec::<i32>(raw, count));
     }
 
     let order = get_byte_order(datatype);
@@ -1407,6 +1427,26 @@ pub fn read_object_references(
             }
             Ok(result)
         }
+        Datatype::Reference {
+            ref_type: crate::datatype::ReferenceType::Object2,
+            size,
+        } => {
+            let elem_size = *size as usize;
+            if elem_size == 0 {
+                return Ok(Vec::new());
+            }
+            if !raw.len().is_multiple_of(elem_size) {
+                return Err(FormatError::DataSizeMismatch {
+                    expected: 0,
+                    actual: raw.len(),
+                });
+            }
+            raw.chunks_exact(elem_size)
+                .map(|element| {
+                    decode_std_object_ref(element).map(|address| ObjectReference { address })
+                })
+                .collect()
+        }
         _ => Err(FormatError::TypeMismatch {
             expected: "Reference(Object)",
             actual: datatype_name(datatype),
@@ -1414,6 +1454,46 @@ pub fn read_object_references(
     }
 }
 
+/// Decode one `H5T_STD_REF` object reference as stored in a dataset:
+/// `type(1) flags(1) token_size(1) token(token_size)`, zero-padded to the
+/// element size. For a reference within the same file the token is the target
+/// object's header address. An all-zero element is a null reference and
+/// decodes to the undefined address (`u64::MAX`).
+fn decode_std_object_ref(element: &[u8]) -> Result<u64, FormatError> {
+    const STD_REF_OBJECT: u8 = 2;
+    const FLAG_EXTERNAL: u8 = 0x01;
+    if element.iter().all(|&b| b == 0) {
+        return Ok(u64::MAX);
+    }
+    let [ref_type, flags, token_size, token @ ..] = element else {
+        return Err(FormatError::UnexpectedEof {
+            expected: 3,
+            available: element.len(),
+        });
+    };
+    if *ref_type != STD_REF_OBJECT {
+        return Err(FormatError::InvalidReferenceType(*ref_type));
+    }
+    if flags & FLAG_EXTERNAL != 0 {
+        // Carries a file name as well; nothing here follows those.
+        return Err(FormatError::TypeMismatch {
+            expected: "object reference within this file",
+            actual: "external object reference",
+        });
+    }
+    let n = *token_size as usize;
+    if n == 0 || n > 8 || n > token.len() {
+        return Err(FormatError::UnexpectedEof {
+            expected: 3 + n,
+            available: element.len(),
+        });
+    }
+    Ok(token[..n]
+        .iter()
+        .rev()
+        .fold(0u64, |addr, &byte| (addr << 8) | u64::from(byte)))
+}
+
 /// Read region references from raw bytes.
 ///
 /// Region references encode a dataset selection (hyperslab, point list, etc.)

crates/clawhdf5-format/src/datatype.rs

@@ -36,8 +36,18 @@ pub enum CharacterSet {
 /// Reference type.
 #[derive(Debug, Clone, PartialEq)]
 pub enum ReferenceType {
+    /// Legacy object reference: the target's object header address.
     Object,
+    /// Legacy dataset region reference.
     DatasetRegion,
+    /// `H5T_STD_REF` object reference (HDF5 1.12+, datatype message version
+    /// 4): a small header followed by an object token. Decoded by
+    /// `data_read::read_object_references`.
+    Object2,
+    /// `H5T_STD_REF` dataset region reference.
+    DatasetRegion2,
+    /// `H5T_STD_REF` attribute reference.
+    Attribute,
 }
 
 /// A member of a compound datatype.
@@ -424,9 +434,15 @@ impl Datatype {
             7 => {
                 // Reference
                 let ref_type_val = bf0 & 0x0F;
-                let ref_type = match ref_type_val {
-                    0 => ReferenceType::Object,
-                    1 => ReferenceType::DatasetRegion,
+                // Datatype message version 4 (HDF5 1.12) revised this class:
+                // types 2-4 are the new `H5T_STD_REF` references, and the high
+                // nibble of the first flag byte carries their encoding version.
+                let ref_type = match (ref_type_val, version) {
+                    (0, _) => ReferenceType::Object,
+                    (1, _) => ReferenceType::DatasetRegion,
+                    (2, 4..) => ReferenceType::Object2,
+                    (3, 4..) => ReferenceType::DatasetRegion2,
+                    (4, 4..) => ReferenceType::Attribute,
                     _ => return Err(FormatError::InvalidReferenceType(ref_type_val)),
                 };
                 Ok((Datatype::Reference { size, ref_type }, pos))
@@ -1563,6 +1579,28 @@ mod tests {
         assert_eq!(err, FormatError::InvalidCharacterSet(2));
     }
 
+    #[test]
+    fn test_reference_v4_std_ref_from_hdf5_2_0() {
+        // Datatype message of an H5T_STD_REF dataset written by HDF5 2.0:
+        // class 7, version 4, type 2 (object), encoding version 1, 18 bytes.
+        let bytes = [0x47, 0x12, 0x00, 0x00, 0x12, 0x00, 0x00, 0x00];
+        let (dt, consumed) = Datatype::parse(&bytes).unwrap();
+        assert_eq!(consumed, 8);
+        assert_eq!(
+            dt,
+            Datatype::Reference {
+                size: 18,
+                ref_type: ReferenceType::Object2
+            }
+        );
+        // The new types are only valid from datatype version 4.
+        let old_version = [0x37, 0x12, 0x00, 0x00, 0x12, 0x00, 0x00, 0x00];
+        assert_eq!(
+            Datatype::parse(&old_version).unwrap_err(),
+            FormatError::InvalidReferenceType(2)
+        );
+    }
+
     #[test]
     fn test_error_invalid_reference_type() {
         let buf = build_dt_header(7, 1, [5, 0, 0], 8);

crates/clawhdf5-format/src/error.rs

@@ -117,6 +117,9 @@ pub enum FormatError {
     /// A message is marked shared but was parsed without access to the file,
     /// so the reference to the real message could not be followed.
     UnresolvedSharedMessage,
+    /// A selection does not fit the dataset it was applied to (wrong rank, or
+    /// it reaches past a dimension's extent).
+    SelectionOutOfBounds(String),
     /// The dataset's raw data is stored in external files (External Data
     /// Files message), which this reader does not follow.
     ExternalDataFilesUnsupported,
@@ -333,6 +336,9 @@ impl fmt::Display for FormatError {
                 f,
                 "dataset raw data is stored in external file(s), which is not supported"
             ),
+            FormatError::SelectionOutOfBounds(msg) => {
+                write!(f, "selection out of bounds: {msg}")
+            }
             FormatError::UnresolvedSharedMessage => write!(
                 f,
                 "message is shared but no file data was available to resolve it"

crates/clawhdf5-format/src/extensible_array.rs

@@ -12,6 +12,31 @@ use alloc::{format, vec, vec::Vec};
 use crate::chunked_read::ChunkInfo;
 use crate::error::FormatError;
 
+/// Verify the Jenkins lookup3 checksum stored immediately after
+/// `data[start..end]`, as every Extensible Array structure carries one.
+///
+/// A corrupt chunk index yields addresses pointing at the wrong bytes, so a
+/// mismatch is an error: otherwise the damage surfaces as plausible data read
+/// from the wrong chunk.
+#[cfg(feature = "checksum")]
+fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatError> {
+    ensure_len(data, end, 4)?;
+    let stored = u32::from_le_bytes([data[end], data[end + 1], data[end + 2], data[end + 3]]);
+    let computed = crate::checksum::jenkins_lookup3(&data[start..end]);
+    if computed != stored {
+        return Err(FormatError::ChecksumMismatch {
+            expected: stored,
+            computed,
+        });
+    }
+    Ok(())
+}
+
+#[cfg(not(feature = "checksum"))]
+fn verify_checksum(_data: &[u8], _start: usize, _end: usize) -> Result<(), FormatError> {
+    Ok(())
+}
+
 /// Parsed Extensible Array header (AEHD).
 #[derive(Debug, Clone)]
 pub struct ExtensibleArrayHeader {
@@ -145,6 +170,8 @@ impl ExtensibleArrayHeader {
         pos += ls; // skip nelmts
         pos += ls; // skip max_idx_set (6th stats field)
         let index_block_address = read_offset(d, pos, offset_size)?;
+        pos += offset_size as usize;
+        verify_checksum(file_data, offset, offset + pos)?;
 
         Ok(ExtensibleArrayHeader {
             client_id,
@@ -270,6 +297,40 @@ fn index_to_chunk_offsets(
 
 /// Collect elements from a data block at the given offset.
 #[allow(clippy::too_many_arguments)]
+/// Layout of super block `u`, per the HDF5 spec: the number of data blocks it
+/// owns and how many elements each of them holds.
+///
+/// `ndblks` and `dblk_nelmts` each double every *other* level, a half-step
+/// apart, so the blocks grow as 1x16, 1x32, 2x32, 2x64, 4x64 ... for a
+/// 16-element minimum. Treating either as doubling every level (the previous
+/// implementation) puts every element after the first data block at the wrong
+/// index.
+fn sblk_info(u: usize, data_blk_min_elmts: usize) -> Option<(usize, usize)> {
+    let ndblks = 1usize.checked_shl((u / 2) as u32)?;
+    let dblk_nelmts = 1usize
+        .checked_shl(u.div_ceil(2) as u32)?
+        .checked_mul(data_blk_min_elmts)?;
+    Some((ndblks, dblk_nelmts))
+}
+
+/// Width of the "offset of the block in the array" field carried by super and
+/// data blocks (`hdr->arr_off_size`).
+fn arr_off_size(header: &ExtensibleArrayHeader) -> usize {
+    (header.max_nelmts_bits as usize).div_ceil(8)
+}
+
+/// Elements per data block page, once a data block is large enough to be paged.
+fn page_nelmts(header: &ExtensibleArrayHeader) -> Option<usize> {
+    1usize.checked_shl(u32::from(header.max_dblk_nelmts_bits))
+}
+
+/// Read the elements of one data block (EADB).
+///
+/// `page_init` is the owning super block's page-init bitmap and `first_page`
+/// this block's first bit in it; both are only consulted when the block is
+/// paged. The bitmap lives in the super block, not here — a paged data block
+/// stores only its prefix, then one slot per page.
+#[allow(clippy::too_many_arguments)]
 fn read_data_block_elements(
     file_data: &[u8],
     db_offset: usize,
@@ -280,117 +341,101 @@ fn read_data_block_elements(
     start_index: usize,
     num_chunks_per_dim: &[u64],
     chunk_dimensions: &[u32],
+    page_init: &[u8],
+    first_page: usize,
 ) -> Result<Vec<ChunkInfo>, FormatError> {
-    // AEDB: signature(4) + version(1) + client_id(1) + header_address(offset_size)
-    let db_header_size = 4 + 1 + 1 + offset_size as usize;
+    // EADB: signature(4) + version(1) + client_id(1) + header_address(offset_size)
+    //       + block offset(arr_off_size)
+    let db_header_size = 4 + 1 + 1 + offset_size as usize + arr_off_size(header);
     ensure_len(file_data, db_offset, db_header_size)?;
 
-    let d = &file_data[db_offset..];
-    if &d[0..4] != b"EADB" {
+    if &file_data[db_offset..db_offset + 4] != b"EADB" {
         return Err(FormatError::ChunkedReadError(
             "invalid Extensible Array data block signature".into(),
         ));
     }
-    // Skip version(1) + client_id(1) + header_address(offset_size) + block_offset
-    // Block offset is encoded in ceil(max_nelmts_bits/8) bytes
-    let blk_off_size = (header.max_nelmts_bits as usize).div_ceil(8);
-    let mut pos = db_offset + db_header_size + blk_off_size;
 
-    // Check if paged
-    if header.max_nelmts_bits >= usize::BITS as u8 {
-        return Err(FormatError::Overflow(
-            "max_nelmts_bits exceeds usize bit width".into(),
-        ));
-    }
-    let page_nelmts = 1usize << header.max_nelmts_bits;
-    let is_paged = nelmts > page_nelmts;
+    let mut pos = db_offset + db_header_size;
+    let page = page_nelmts(header).ok_or_else(|| {
+        FormatError::Overflow("Extensible Array page element count overflows usize".into())
+    })?;
 
     let mut chunks = Vec::new();
-
-    if !is_paged {
-        for i in 0..nelmts {
+    let read_run = |from: usize,
+                    count: usize,
+                    first_index: usize,
+                    chunks: &mut Vec<ChunkInfo>|
+     -> Result<usize, FormatError> {
+        let mut p = from;
+        for i in 0..count {
             let (info, consumed) = read_element(
                 file_data,
-                pos,
+                p,
                 header.client_id,
                 header.element_size,
                 offset_size,
                 chunk_byte_size,
-                start_index + i,
+                first_index + i,
                 num_chunks_per_dim,
                 chunk_dimensions,
             )?;
             if let Some(ci) = info {
                 chunks.push(ci);
             }
-            pos += consumed;
+            p += consumed;
         }
-    } else {
-        // Paged: elements are split into pages of page_nelmts.
-        // After the data block header comes a page bitmap, then each page
-        // has page_nelmts elements followed by a 4-byte checksum.
-        let npages = nelmts.div_ceil(page_nelmts);
-        // Page bitmap: ceil(npages / 8) bytes
-        let bitmap_size = npages.div_ceil(8);
-        // Read bitmap
-        if pos + bitmap_size > file_data.len() {
-            return Err(FormatError::UnexpectedEof {
-                expected: pos + bitmap_size,
-                available: file_data.len(),
-            });
-        }
-        let bitmap = &file_data[pos..pos + bitmap_size];
-        pos += bitmap_size;
+        Ok(p)
+    };
 
+    if nelmts <= page {
+        // Prefix and elements are covered by one checksum.
         let elem_bytes = if header.client_id == 0 {
             offset_size as usize
         } else {
             header.element_size as usize
         };
-
-        let mut global_idx = start_index;
-        for page_idx in 0..npages {
-            let byte_idx = page_idx / 8;
-            let bit_idx = page_idx % 8;
-            let page_has_data = (bitmap[byte_idx] >> bit_idx) & 1 != 0;
-
-            let elems_this_page = if page_idx == npages - 1 {
-                let remainder = nelmts % page_nelmts;
-                if remainder == 0 {
-                    page_nelmts
-                } else {
-                    remainder
+        let end = nelmts
+            .checked_mul(elem_bytes)
+            .and_then(|b| pos.checked_add(b))
+            .ok_or_else(|| FormatError::Overflow("Extensible Array data block span".into()))?;
+        verify_checksum(file_data, db_offset, end)?;
+        read_run(pos, nelmts, start_index, &mut chunks)?;
+        return Ok(chunks);
     }
-            } else {
-                page_nelmts
-            };
 
-            if page_has_data {
-                for i in 0..elems_this_page {
-                    let (info, consumed) = read_element(
-                        file_data,
-                        pos,
-                        header.client_id,
-                        header.element_size,
-                        offset_size,
-                        chunk_byte_size,
-                        global_idx + i,
-                        num_chunks_per_dim,
-                        chunk_dimensions,
-                    )?;
-                    if let Some(ci) = info {
-                        chunks.push(ci);
-                    }
-                    pos += consumed;
-                }
-                // Skip page checksum (4 bytes)
+    // Paged: the prefix ends with its own checksum, then one slot per page,
+    // each holding `page` elements followed by a checksum. Pages whose bit is
+    // clear were never written; their slot still occupies the file, so stride
+    // over it rather than reading zeros as addresses.
+    verify_checksum(file_data, db_offset, pos)?;
     pos += 4;
+    let elem_bytes = if header.client_id == 0 {
+        offset_size as usize
     } else {
-                // Empty page: skip all elements + checksum
-                pos += elems_this_page * elem_bytes + 4;
-            }
-            global_idx += elems_this_page;
+        header.element_size as usize
+    };
+    let page_stride = page
+        .checked_mul(elem_bytes)
+        .and_then(|b| b.checked_add(4))
+        .ok_or_else(|| FormatError::Overflow("Extensible Array page stride".into()))?;
+    let npages = nelmts.div_ceil(page);
+    for p in 0..npages {
+        // One bit per page across the whole super block, packed contiguously
+        // and MSB-first within each byte, as H5VM_bit_get reads it.
+        let bit = first_page + p;
+        let initialised = page_init
+            .get(bit / 8)
+            .is_some_and(|byte| byte & (0x80 >> (bit % 8)) != 0);
+        if initialised {
+            let count = core::cmp::min(page, nelmts - p * page);
+            // Each page carries its own checksum, over a full page's worth of
+            // slots even when the last one holds fewer live elements.
+            verify_checksum(file_data, pos, pos + page * elem_bytes)?;
+            read_run(pos, count, start_index + p * page, &mut chunks)?;
         }
+        pos = pos
+            .checked_add(page_stride)
+            .ok_or_else(|| FormatError::Overflow("Extensible Array page offset".into()))?;
     }
 
     Ok(chunks)
@@ -427,30 +472,83 @@ pub fn read_extensible_array_chunks(
     let chunk_byte_size: u64 =
         chunk_dimensions.iter().map(|&d| d as u64).product::<u64>() * element_size as u64;
 
-    // Parse index block (AEIB)
+    // Parse index block (EAIB): signature(4) + version(1) + client_id(1)
+    // + header address(offset_size), then the inline elements, then the
+    // direct data block addresses, then the super block addresses.
     let ib_offset = header.index_block_address as usize;
-    let ib_header_size = 4 + 1 + 1 + offset_size as usize; // sig + ver + client + hdr_addr
+    let ib_header_size = 4 + 1 + 1 + os;
     ensure_len(file_data, ib_offset, ib_header_size)?;
 
-    let ib = &file_data[ib_offset..];
-    if &ib[0..4] != b"EAIB" {
+    if &file_data[ib_offset..ib_offset + 4] != b"EAIB" {
         return Err(FormatError::ChunkedReadError(
             "invalid Extensible Array index block signature".into(),
         ));
     }
-    // Skip version(1) + client_id(1) + header_address(offset_size)
     let mut pos = ib_offset + ib_header_size;
 
     let mut chunks = Vec::new();
-    let mut global_index = 0usize;
     let total_elements = header.num_elements as usize;
 
-    // 1. Read inline elements in index block
-    let n_inline = header.idx_blk_elmts as usize;
-    for i in 0..n_inline {
-        if global_index + i >= total_elements {
-            break;
+    let dmin = header.min_dblk_nelmts as usize;
+    if dmin == 0 || !dmin.is_power_of_two() {
+        return Err(FormatError::ChunkedReadError(
+            "Extensible Array data block minimum is not a power of two".into(),
+        ));
     }
+    // nsblks = 1 + (max_nelmts_bits - log2(data_blk_min_elmts)), and the index
+    // block holds 2 * (sup_blk_min_data_ptrs - 1) data block addresses.
+    let log2_dmin = dmin.trailing_zeros() as usize;
+    let nsblks = 1 + (header.max_nelmts_bits as usize).saturating_sub(log2_dmin);
+    let ndblk_addrs = 2 * (header.super_blk_min_nelmts as usize).saturating_sub(1);
+
+    // The data blocks listed directly in the index block are the first
+    // `ndblk_addrs` in super-block order, each sized by the level it belongs
+    // to; the super block addresses that follow resume at the next level.
+    let mut direct: Vec<usize> = Vec::with_capacity(ndblk_addrs);
+    let mut level = 0usize;
+    while direct.len() < ndblk_addrs {
+        if level >= nsblks {
+            return Err(FormatError::ChunkedReadError(
+                "Extensible Array index block claims more data blocks than the array has".into(),
+            ));
+        }
+        let (ndblks, dblk_nelmts) = sblk_info(level, dmin).ok_or_else(|| {
+            FormatError::Overflow("Extensible Array super block layout overflows usize".into())
+        })?;
+        for _ in 0..ndblks {
+            direct.push(dblk_nelmts);
+        }
+        level += 1;
+    }
+    if direct.len() != ndblk_addrs {
+        // A partial level in the index block is not a layout HDF5 produces,
+        // and guessing where the super blocks resume would misplace elements.
+        return Err(FormatError::ChunkedReadError(
+            "Extensible Array index block ends mid super block".into(),
+        ));
+    }
+
+    // One checksum covers the prefix, every inline element slot, and every
+    // data block and super block address.
+    let elem_bytes = if header.client_id == 0 {
+        os
+    } else {
+        header.element_size as usize
+    };
+    let ib_end = (header.idx_blk_elmts as usize)
+        .checked_mul(elem_bytes)
+        .and_then(|b| pos.checked_add(b))
+        .and_then(|p| {
+            ndblk_addrs
+                .checked_add(nsblks - level)
+                .and_then(|n| n.checked_mul(os).and_then(|b| p.checked_add(b)))
+        })
+        .ok_or_else(|| FormatError::Overflow("Extensible Array index block span".into()))?;
+    verify_checksum(file_data, ib_offset, ib_end)?;
+
+    // 1. Elements stored inline in the index block.
+    let n_inline = (header.idx_blk_elmts as usize).min(total_elements);
+    for i in 0..n_inline {
         let (info, consumed) = read_element(
             file_data,
             pos,
@@ -458,7 +556,7 @@ pub fn read_extensible_array_chunks(
             header.element_size,
             offset_size,
             chunk_byte_size,
-            global_index + i,
+            i,
             &num_chunks_per_dim,
             chunk_dimensions,
         )?;
@@ -467,154 +565,90 @@ pub fn read_extensible_array_chunks(
         }
         pos += consumed;
     }
-    global_index += n_inline.min(total_elements);
-
-    // If all elements were inline, we're done
+    let mut global_index = n_inline;
     if global_index >= total_elements {
         return Ok(chunks);
     }
 
-    // Compute data block and super block counts
-    let min_dblk = header.min_dblk_nelmts as usize;
-    let sblk_min = header.super_blk_min_nelmts as usize;
-
-    // The first sblk_min super block levels have their data blocks listed directly
-    // in the index block. Compute their sizes.
-    let mut n_direct_dblks = 0usize;
-    let mut dblk_sizes: Vec<usize> = Vec::new();
-    {
-        let mut nelmts = min_dblk;
-        for sb_level in 0..sblk_min {
-            if sb_level >= usize::BITS as usize {
-                return Err(FormatError::Overflow(
-                    "sb_level exceeds usize bit width".into(),
+    // 2. Data blocks listed directly in the index block.
+    for &dblk_nelmts in &direct {
+        if global_index >= total_elements {
+            return Ok(chunks);
+        }
+        ensure_len(file_data, pos, os)?;
+        let addr = read_offset(file_data, pos, offset_size)?;
+        pos += os;
+        if !is_undefined_addr(addr, offset_size) {
+            if dblk_nelmts > page_nelmts(header).unwrap_or(usize::MAX) {
+                // Would need a page-init bitmap, which only a super block
+                // carries. HDF5 never pages these small early blocks.
+                return Err(FormatError::ChunkedReadError(
+                    "Extensible Array index block references a paged data block".into(),
                 ));
             }
-            let ndblks = 1usize << sb_level;
-            for _ in 0..ndblks {
-                dblk_sizes.push(nelmts);
-                n_direct_dblks += 1;
-            }
-            if sb_level > 0 {
-                nelmts *= 2;
-            }
-        }
-    }
-
-    // Read direct data block addresses from index block
-    let mut dblk_addrs: Vec<u64> = Vec::with_capacity(n_direct_dblks);
-    for _ in 0..n_direct_dblks {
-        if pos + os > file_data.len() {
-            break;
-        }
-        let addr = read_offset(file_data, pos, offset_size)?;
-        dblk_addrs.push(addr);
-        pos += os;
-    }
-
-    // Read elements from direct data blocks
-    for (i, &addr) in dblk_addrs.iter().enumerate() {
-        if i >= dblk_sizes.len() {
-            break;
-        }
-        let nelmts = dblk_sizes[i];
-        if is_undefined_addr(addr, offset_size) {
-            global_index += nelmts;
-            continue;
-        }
-        let block_chunks = read_data_block_elements(
+            chunks.extend(read_data_block_elements(
                 file_data,
                 addr as usize,
-            nelmts,
+                dblk_nelmts,
                 header,
                 offset_size,
                 chunk_byte_size,
                 global_index,
                 &num_chunks_per_dim,
                 chunk_dimensions,
-        )?;
-        chunks.extend(block_chunks);
-        global_index += nelmts;
+                &[],
+                0,
+            )?);
+        }
+        global_index += dblk_nelmts;
     }
 
-    // Remaining elements are in super blocks
-    let total_in_ib_and_direct: usize = n_inline + dblk_sizes.iter().sum::<usize>();
-    if total_elements <= total_in_ib_and_direct {
-        return Ok(chunks);
-    }
-    let remaining_elements = total_elements - total_in_ib_and_direct;
-
-    // Compute super block layout
-    let mut sb_addrs: Vec<u64> = Vec::new();
-    let mut sb_infos: Vec<(usize, usize)> = Vec::new();
-    {
-        let mut covered = 0usize;
-        let mut sb_level = sblk_min;
-        let mut nelmts_per_dblk = min_dblk;
-        for lev in 0..sblk_min {
-            if lev > 0 {
-                nelmts_per_dblk *= 2;
-            }
-        }
-
-        while covered < remaining_elements {
-            if sb_level >= usize::BITS as usize {
-                return Err(FormatError::Overflow(
-                    "sb_level exceeds usize bit width".into(),
-                ));
-            }
-            let ndblks = 1usize << sb_level;
-            nelmts_per_dblk *= 2;
-            let total_in_sb = ndblks * nelmts_per_dblk;
-            sb_infos.push((ndblks, nelmts_per_dblk));
-            covered += total_in_sb;
-            sb_level += 1;
-        }
-    }
-
-    // Read super block addresses from index block
-    for _ in 0..sb_infos.len() {
-        if pos + os > file_data.len() {
+    // 3. Everything else lives in super blocks, one address per remaining
+    //    level, starting at the level after the direct data blocks.
+    for u in level..nsblks {
+        if global_index >= total_elements {
             break;
         }
-        let addr = read_offset(file_data, pos, offset_size)?;
-        sb_addrs.push(addr);
+        ensure_len(file_data, pos, os)?;
+        let sb_addr = read_offset(file_data, pos, offset_size)?;
         pos += os;
-    }
-
-    // Process each super block
-    for (sb_idx, &sb_addr) in sb_addrs.iter().enumerate() {
-        let (ndblks, nelmts_per_dblk) = sb_infos[sb_idx];
-        if is_undefined_addr(sb_addr, offset_size) {
-            global_index += ndblks * nelmts_per_dblk;
-            continue;
-        }
-        let sb_chunks = read_super_block(
+        let (ndblks, dblk_nelmts) = sblk_info(u, dmin).ok_or_else(|| {
+            FormatError::Overflow("Extensible Array super block layout overflows usize".into())
+        })?;
+        if !is_undefined_addr(sb_addr, offset_size) {
+            chunks.extend(read_super_block(
                 file_data,
                 sb_addr as usize,
                 ndblks,
-            nelmts_per_dblk,
+                dblk_nelmts,
                 header,
                 offset_size,
                 chunk_byte_size,
                 global_index,
                 &num_chunks_per_dim,
                 chunk_dimensions,
-        )?;
-        chunks.extend(sb_chunks);
-        global_index += ndblks * nelmts_per_dblk;
+            )?);
+        }
+        global_index =
+            global_index.saturating_add(ndblks.checked_mul(dblk_nelmts).ok_or_else(|| {
+                FormatError::Overflow("Extensible Array super block span".into())
+            })?);
     }
 
     Ok(chunks)
 }
 
-/// Read a super block (AESB) and its data blocks.
+/// Read a super block (EASB) and the data blocks it owns.
+///
+/// On disk: signature(4) + version(1) + client_id(1) + header address
+/// + block offset + the page-init bitmap for every data block it owns
+/// + one address per data block + checksum.
 #[allow(clippy::too_many_arguments)]
 fn read_super_block(
     file_data: &[u8],
     sb_offset: usize,
     ndblks: usize,
-    nelmts_per_dblk: usize,
+    dblk_nelmts: usize,
     header: &ExtensibleArrayHeader,
     offset_size: u8,
     chunk_byte_size: u64,
@@ -623,9 +657,7 @@ fn read_super_block(
     chunk_dimensions: &[u32],
 ) -> Result<Vec<ChunkInfo>, FormatError> {
     let os = offset_size as usize;
-
-    // AESB: signature(4) + version(1) + client_id(1) + header_address(offset_size)
-    let sb_header_size = 4 + 1 + 1 + os;
+    let sb_header_size = 4 + 1 + 1 + os + arr_off_size(header);
     ensure_len(file_data, sb_offset, sb_header_size)?;
 
     if &file_data[sb_offset..sb_offset + 4] != b"EASB" {
@@ -634,43 +666,57 @@ fn read_super_block(
         ));
     }
 
-    let mut pos = sb_offset + sb_header_size;
-
-    // Read data block addresses
-    let mut dblk_addrs: Vec<u64> = Vec::with_capacity(ndblks);
-    for _ in 0..ndblks {
-        if pos + os > file_data.len() {
-            return Err(FormatError::UnexpectedEof {
-                expected: pos + os,
-                available: file_data.len(),
-            });
-        }
-        let addr = read_offset(file_data, pos, offset_size)?;
-        dblk_addrs.push(addr);
-        pos += os;
-    }
+    // Page-init bitmap: one bit per page, `npages` bits per data block, packed
+    // contiguously. HDF5 sizes the buffer `ndblks * ceil(npages / 8)`, which
+    // is bigger than the bits need when `npages` is not a multiple of eight.
+    // Zero-sized unless this level's data blocks are paged.
+    let page = page_nelmts(header).ok_or_else(|| {
+        FormatError::Overflow("Extensible Array page element count overflows usize".into())
+    })?;
+    let npages = if dblk_nelmts > page {
+        dblk_nelmts / page
+    } else {
+        0
+    };
+    let per_dblk_bitmap = npages.div_ceil(8);
+    let bitmap_bytes = per_dblk_bitmap
+        .checked_mul(ndblks)
+        .ok_or_else(|| FormatError::Overflow("Extensible Array page bitmap size".into()))?;
+    let bitmap_start = sb_offset + sb_header_size;
+    ensure_len(file_data, bitmap_start, bitmap_bytes)?;
+    let bitmap = &file_data[bitmap_start..bitmap_start + bitmap_bytes];
 
+    let mut pos = bitmap_start + bitmap_bytes;
     let mut chunks = Vec::new();
     let mut global_idx = start_index;
 
-    for &addr in &dblk_addrs {
-        if is_undefined_addr(addr, offset_size) {
-            global_idx += nelmts_per_dblk;
-            continue;
-        }
-        let block_chunks = read_data_block_elements(
+    // One checksum covers the prefix, the bitmap and every data block address.
+    let sb_end = ndblks
+        .checked_mul(os)
+        .and_then(|b| pos.checked_add(b))
+        .ok_or_else(|| FormatError::Overflow("Extensible Array super block span".into()))?;
+    verify_checksum(file_data, sb_offset, sb_end)?;
+
+    for i in 0..ndblks {
+        ensure_len(file_data, pos, os)?;
+        let addr = read_offset(file_data, pos, offset_size)?;
+        pos += os;
+        if !is_undefined_addr(addr, offset_size) {
+            chunks.extend(read_data_block_elements(
                 file_data,
                 addr as usize,
-            nelmts_per_dblk,
+                dblk_nelmts,
                 header,
                 offset_size,
                 chunk_byte_size,
                 global_idx,
                 num_chunks_per_dim,
                 chunk_dimensions,
-        )?;
-        chunks.extend(block_chunks);
-        global_idx += nelmts_per_dblk;
+                bitmap,
+                i * npages,
+            )?);
+        }
+        global_idx += dblk_nelmts;
     }
 
     Ok(chunks)
@@ -679,6 +725,14 @@ fn read_super_block(
 #[cfg(test)]
 mod tests {
     use super::*;
+
+    /// Stamp the Jenkins checksum a real file would carry over
+    /// `data[start..end]`, writing it at `end`. Hand-built fixtures need this
+    /// now that the reader validates it, exactly as HDF5 writes it.
+    fn stamp_checksum(data: &mut [u8], start: usize, end: usize) {
+        let sum = crate::checksum::jenkins_lookup3(&data[start..end]);
+        data[end..end + 4].copy_from_slice(&sum.to_le_bytes());
+    }
     #[test]
     fn index_to_offsets_1d() {
         let num_chunks = vec![5u64];
@@ -734,6 +788,7 @@ mod tests {
         buf[44..52].copy_from_slice(&5u64.to_le_bytes()); // stat[4] = num_elements
         buf[52..60].copy_from_slice(&0u64.to_le_bytes()); // stat[5]
         buf[60..68].copy_from_slice(&0x1000u64.to_le_bytes()); // index_block_address
+        stamp_checksum(&mut buf, 0, 68);
 
         let hdr = ExtensibleArrayHeader::parse(&buf, 0, os, ls).unwrap();
         assert_eq!(hdr.client_id, 0);
@@ -819,6 +874,7 @@ mod tests {
             .copy_from_slice(&(num_chunks as u64).to_le_bytes());
         file_data[aehd_offset + 60..aehd_offset + 68]
             .copy_from_slice(&(aeib_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, aehd_offset, aehd_offset + 68);
         // checksum (4 bytes at +68) — not validated
 
         // Build AEIB at aeib_offset
@@ -836,6 +892,23 @@ mod tests {
             let p = elem_start + i * osv;
             file_data[p..p + osv].copy_from_slice(&addr.to_le_bytes());
         }
+        // The index block's checksum covers its prefix, every inline element
+        // slot, and every data block and super block address slot:
+        // ndblk_addrs = 2 * (sup_blk_min_data_ptrs - 1), and the super block
+        // pointers make up the rest of nsblks levels.
+        let sup_ptrs = file_data[aehd_offset + 10] as usize;
+        let dmin = file_data[aehd_offset + 9] as usize;
+        let nsblks = 1 + 10 - dmin.trailing_zeros() as usize;
+        let ndblk_addrs = 2 * (sup_ptrs - 1);
+        // Levels consumed by those direct data blocks (1, 1, 2, 2, ... per level).
+        let mut consumed = 0usize;
+        let mut levels = 0usize;
+        while consumed < ndblk_addrs {
+            consumed += 1 << (levels / 2);
+            levels += 1;
+        }
+        let ib_end = elem_start + num_chunks * osv + (ndblk_addrs + nsblks - levels) * osv;
+        stamp_checksum(&mut file_data, aeib_offset, ib_end);
 
         let header = ExtensibleArrayHeader::parse(&file_data, aehd_offset, os, ls).unwrap();
         let ds_dims = vec![40u64]; // 2 chunks × 20 elements
@@ -885,6 +958,7 @@ mod tests {
         // idx_blk_addr at offset 12 + 6*8 = 60
         file_data[aehd_offset + 60..aehd_offset + 68]
             .copy_from_slice(&(aeib_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, aehd_offset, aehd_offset + 68);
 
         // AEIB
         file_data[aeib_offset..aeib_offset + 4].copy_from_slice(b"EAIB");
@@ -903,42 +977,48 @@ mod tests {
             pos += osv;
         }
 
-        // Direct data block addresses: first sb_level=0 has 1 dblk, sb_level=1 has 1 dblk
-        // Total direct dblks for sblk_min=2: 2^0 + 2^1 = 1 + 2 = 3 (oops)
-        // Actually: sblk_min levels. level 0: 2^0=1 dblk, level 1: 2^1=2 dblks => 3 dblks
-        // But we only have 2 remaining elements.
-        // dblk sizes: level 0: 1 dblk of min_dblk=2; level 1: 2 dblks of 2 each (nelmts doubles at level > 0)
-        // Wait, re-reading the code: at level 0, nelmts=min_dblk=2, 1 dblk.
-        // At level 1, 1 dblk, nelmts still 2 (doubles only at level > 0... but the code says
-        // `if sb_level > 0 { nelmts *= 2 }` after pushing). Let me re-check.
-        // After push at level 0: nelmts=2. Then if 0>0 false, no double. Push 1 dblk of 2.
-        // Level 1: ndblks=2. Push 2 dblks of 2. Then 1>0 true, nelmts=4.
-        // Total: 3 dblks with sizes [2, 2, 2]. Total = 6.
-        // We only need 2 more elements. So only the first dblk has data.
-        let n_direct_dblks = 3;
+        // Direct data block addresses. With sup_blk_min_data_ptrs = 2 the index
+        // block holds 2 * (2 - 1) = 2 of them, which are the data blocks of
+        // super block levels 0 and 1: one of `min_dblk_nelmts` elements, then
+        // one of twice that (ndblks = 2^(u/2), dblk_nelmts = 2^((u+1)/2) * min).
+        // Only the first is allocated here; the rest of the array is empty.
+        let ndblk_addrs = 2 * (sblk_min as usize - 1);
         file_data[pos..pos + osv].copy_from_slice(&(aedb_offset as u64).to_le_bytes());
         pos += osv;
-        // 2 more dblk addresses - undefined
-        for _ in 1..n_direct_dblks {
+        for _ in 1..ndblk_addrs {
             file_data[pos..pos + osv].copy_from_slice(&u64::MAX.to_le_bytes());
             pos += osv;
         }
+        // Super block addresses fill the remaining levels; all unallocated.
+        let nsblks = 1 + 10 - (min_dblk_nelmts as usize).trailing_zeros() as usize;
+        let mut consumed = 0usize;
+        let mut levels = 0usize;
+        while consumed < ndblk_addrs {
+            consumed += 1 << (levels / 2);
+            levels += 1;
+        }
+        for _ in 0..(nsblks - levels) {
+            file_data[pos..pos + osv].copy_from_slice(&u64::MAX.to_le_bytes());
+            pos += osv;
+        }
+        stamp_checksum(&mut file_data, aeib_offset, pos);
 
-        // EADB at aedb_offset (min_dblk_nelmts elements)
+        // EADB holding the first data block's `min_dblk_nelmts` elements.
         file_data[aedb_offset..aedb_offset + 4].copy_from_slice(b"EADB");
         file_data[aedb_offset + 4] = 0;
         file_data[aedb_offset + 5] = 0;
         file_data[aedb_offset + 6..aedb_offset + 14]
             .copy_from_slice(&(aehd_offset as u64).to_le_bytes());
-        // block_offset: ceil(max_nelmts_bits/8) = ceil(10/8) = 2 bytes
-        // block_offset = 0 for first data block
-        let blk_off_size = (10usize).div_ceil(8); // max_nelmts_bits=10
-        let mut dbpos = aedb_offset + 6 + osv + blk_off_size;
+        // Block offset field: ceil(max_nelmts_bits / 8) bytes, zero here.
+        let blk_off_size = (10usize).div_ceil(8);
+        let db_elems = aedb_offset + 6 + osv + blk_off_size;
+        let mut dbpos = db_elems;
         for i in 0..min_dblk_nelmts as usize {
             let addr = base_addr + (idx_blk_elmts as u64 + i as u64) * chunk_byte_size;
             file_data[dbpos..dbpos + osv].copy_from_slice(&addr.to_le_bytes());
             dbpos += osv;
         }
+        stamp_checksum(&mut file_data, aedb_offset, dbpos);
 
         let header = ExtensibleArrayHeader::parse(&file_data, aehd_offset, os, ls).unwrap();
         let ds_dims = vec![40u64];

crates/clawhdf5-format/src/file_writer.rs

@@ -1221,8 +1221,10 @@ impl FileWriter {
                     precompressed: None,
                 });
             } else if is_chunked[i] {
-                let chunk_dims = d.chunk_options.resolve_chunk_dims(&d.ds.dimensions);
                 let elem_size = d.dt.type_size() as usize;
+                let chunk_dims = d
+                    .chunk_options
+                    .resolve_chunk_dims_for(&d.ds.dimensions, elem_size);
                 // Compress once in Pass 1; cache the result so Pass 2 can skip
                 // re-compression and just rebuild the index with real addresses.
                 let pre = precompress_chunks(

crates/clawhdf5-format/src/filters.rs

@@ -845,9 +845,33 @@ fn shuffle_decompress(data: &[u8], element_size: usize) -> Result<Vec<u8>, Forma
     let num_elements = data.len() / element_size;
     let mut result = vec![0u8; data.len()];
 
-    for i in 0..num_elements {
-        for j in 0..element_size {
-            result[i * element_size + j] = data[j * num_elements + i];
+    // The shuffled stream is `element_size` byte planes of `num_elements`
+    // bytes each; un-shuffling interleaves them. This is on the read path of
+    // every compressed dataset (shuffle is applied automatically before
+    // compression). The naive `result[i * es + j] = data[j * n + i]` form does
+    // a multiply and two bounds checks per byte and defeats vectorisation;
+    // fixed-width plane arrays sliced to a common length let the compiler
+    // hoist the checks and emit interleaves for the common 4- and 8-byte
+    // element sizes.
+    fn interleave<const W: usize>(data: &[u8], n: usize, out: &mut [u8]) {
+        let planes: [&[u8]; W] = core::array::from_fn(|j| &data[j * n..(j + 1) * n]);
+        for (i, element) in out.as_chunks_mut::<W>().0.iter_mut().enumerate() {
+            for (byte, plane) in element.iter_mut().zip(&planes) {
+                *byte = plane[i];
+            }
+        }
+    }
+    match element_size {
+        2 => interleave::<2>(data, num_elements, &mut result),
+        4 => interleave::<4>(data, num_elements, &mut result),
+        8 => interleave::<8>(data, num_elements, &mut result),
+        16 => interleave::<16>(data, num_elements, &mut result),
+        _ => {
+            for (i, element) in result.chunks_exact_mut(element_size).enumerate() {
+                for (j, byte) in element.iter_mut().enumerate() {
+                    *byte = data[j * num_elements + i];
+                }
+            }
         }
     }
 
@@ -1848,4 +1872,20 @@ mod tests {
         };
         assert!(decompress_chunk(&data, &pipeline, 16, 1).is_err());
     }
+    #[test]
+    fn unshuffle_inverts_shuffle_for_every_element_size() {
+        for element_size in [1usize, 2, 3, 4, 5, 8, 12, 16, 24] {
+            for elements in [0usize, 1, 2, 7, 64, 1000] {
+                let original: Vec<u8> = (0..element_size * elements)
+                    .map(|i| (i * 31 + 7) as u8)
+                    .collect();
+                let shuffled = shuffle_compress(&original, element_size).unwrap();
+                assert_eq!(
+                    shuffle_decompress(&shuffled, element_size).unwrap(),
+                    original,
+                    "element_size {element_size}, {elements} elements"
+                );
+            }
+        }
+    }
 }

crates/clawhdf5-format/src/fixed_array.rs

@@ -9,6 +9,31 @@ use alloc::{format, vec, vec::Vec};
 use crate::chunked_read::ChunkInfo;
 use crate::error::FormatError;
 
+/// Verify the Jenkins lookup3 checksum stored immediately after
+/// `data[start..end]`, as every Fixed Array structure carries one.
+///
+/// A corrupt chunk index silently yields addresses pointing at the wrong
+/// bytes, so a mismatch has to be an error rather than a shrug: without this
+/// the damage surfaces as plausible-looking data from the wrong chunk.
+#[cfg(feature = "checksum")]
+fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatError> {
+    ensure_len(data, end, 4)?;
+    let stored = u32::from_le_bytes([data[end], data[end + 1], data[end + 2], data[end + 3]]);
+    let computed = crate::checksum::jenkins_lookup3(&data[start..end]);
+    if computed != stored {
+        return Err(FormatError::ChecksumMismatch {
+            expected: stored,
+            computed,
+        });
+    }
+    Ok(())
+}
+
+#[cfg(not(feature = "checksum"))]
+fn verify_checksum(_data: &[u8], _start: usize, _end: usize) -> Result<(), FormatError> {
+    Ok(())
+}
+
 /// Parsed Fixed Array header (FAHD).
 #[derive(Debug, Clone)]
 pub struct FixedArrayHeader {
@@ -103,6 +128,8 @@ impl FixedArrayHeader {
         let num_elements = read_length(d, pos, length_size)?;
         pos += length_size as usize;
         let data_block_address = read_offset(d, pos, offset_size)?;
+        pos += offset_size as usize;
+        verify_checksum(file_data, offset, offset + pos)?;
 
         Ok(FixedArrayHeader {
             client_id,
@@ -223,7 +250,8 @@ pub fn read_fixed_array_chunks(
 
     if !is_paged {
         // Non-paged: prefix, then `num_elements` elements packed directly,
-        // then a trailing checksum (which we don't validate).
+        // then a checksum over both.
+        verify_checksum(file_data, db_offset, elem_at(elements_start, num_elements)?)?;
         for i in 0..num_elements {
             push_element(i, elem_at(elements_start, i)?, &mut chunks)?;
         }
@@ -254,6 +282,9 @@ pub fn read_fixed_array_chunks(
             available: file_data.len(),
         });
     }
+    // The prefix and page bitmap are covered by their own checksum, and each
+    // initialised page by one of its own.
+    verify_checksum(file_data, db_offset, bitmap_start + bitmap_size)?;
 
     for p in 0..npages {
         let page_first = p * page_nelmts; // < num_elements, cannot overflow
@@ -270,6 +301,7 @@ pub fn read_fixed_array_chunks(
             .checked_mul(page_stride)
             .and_then(|o| pages_start.checked_add(o))
             .ok_or_else(stride_overflow)?;
+        verify_checksum(file_data, page_off, elem_at(page_off, page_count)?)?;
         for e in 0..page_count {
             push_element(page_first + e, elem_at(page_off, e)?, &mut chunks)?;
         }
@@ -374,6 +406,14 @@ fn read_variable_length(data: &[u8], size: usize) -> Result<u64, FormatError> {
 mod tests {
     use super::*;
 
+    /// Stamp the Jenkins checksum a real file would carry over
+    /// `data[start..end]`, writing it at `end`. Fixtures built by hand need
+    /// this now that the reader validates it — as every HDF5 writer does.
+    fn stamp_checksum(data: &mut [u8], start: usize, end: usize) {
+        let sum = crate::checksum::jenkins_lookup3(&data[start..end]);
+        data[end..end + 4].copy_from_slice(&sum.to_le_bytes());
+    }
+
     #[test]
     fn index_to_offsets_1d() {
         let num_chunks = vec![5u64];
@@ -439,7 +479,7 @@ mod tests {
         buf[8..16].copy_from_slice(&5u64.to_le_bytes());
         // data_block_address (offset_size=8)
         buf[16..24].copy_from_slice(&0x1000u64.to_le_bytes());
-        // checksum (4 bytes, we don't validate in parse)
+        stamp_checksum(&mut buf, 0, 24);
 
         let header = FixedArrayHeader::parse(&buf, 0, 8, 8).unwrap();
         assert_eq!(header.client_id, 1);
@@ -449,6 +489,54 @@ mod tests {
         assert_eq!(header.data_block_address, 0x1000);
     }
 
+    /// Corruption anywhere in the index must be an error, not a wrong
+    /// address. Every structure carries a checksum; flipping a bit in each in
+    /// turn must be caught, because the alternative is reading a chunk from
+    /// the wrong offset and returning it as data.
+    #[test]
+    fn corrupting_any_fixed_array_structure_is_detected() {
+        let build = || -> (Vec<u8>, usize) {
+            let (os, fahd, db) = (8usize, 0x100usize, 0x200usize);
+            let mut f = vec![0u8; 0x3000];
+            f[fahd..fahd + 4].copy_from_slice(b"FAHD");
+            f[fahd + 6] = os as u8;
+            f[fahd + 7] = 10;
+            f[fahd + 8..fahd + 16].copy_from_slice(&3u64.to_le_bytes());
+            f[fahd + 16..fahd + 24].copy_from_slice(&(db as u64).to_le_bytes());
+            stamp_checksum(&mut f, fahd, fahd + 24);
+            f[db..db + 4].copy_from_slice(b"FADB");
+            f[db + 6..db + 14].copy_from_slice(&(fahd as u64).to_le_bytes());
+            let elems = db + 6 + os;
+            for i in 0..3usize {
+                let addr = 0x1000u64 + i as u64 * 0x100;
+                f[elems + i * os..elems + (i + 1) * os].copy_from_slice(&addr.to_le_bytes());
+            }
+            stamp_checksum(&mut f, db, elems + 3 * os);
+            (f, fahd)
+        };
+
+        let read = |f: &[u8], fahd: usize| -> Result<Vec<ChunkInfo>, FormatError> {
+            let h = FixedArrayHeader::parse(f, fahd, 8, 8)?;
+            read_fixed_array_chunks(f, &h, &[60], &[20], 8, 8, 8)
+        };
+
+        let (clean, fahd) = build();
+        assert!(read(&clean, fahd).is_ok(), "the intact fixture must read");
+
+        // A byte inside the header, and one inside a data block element.
+        for &at in &[0x108usize, 0x210usize] {
+            let (mut damaged, fahd) = build();
+            damaged[at] ^= 0x01;
+            assert!(
+                matches!(
+                    read(&damaged, fahd),
+                    Err(FormatError::ChecksumMismatch { .. })
+                ),
+                "corruption at {at:#x} went undetected"
+            );
+        }
+    }
+
     #[test]
     fn parse_fixed_array_header_invalid_signature() {
         let mut buf = vec![0u8; 256];
@@ -469,6 +557,7 @@ mod tests {
         buf[fahd + 7] = 200; // max_nelmts_bits — absurd, would overflow a shift
         buf[fahd + 8..fahd + 16].copy_from_slice(&3u64.to_le_bytes()); // num_elements
         buf[fahd + 16..fahd + 24].copy_from_slice(&0x100u64.to_le_bytes());
+        stamp_checksum(&mut buf, fahd, fahd + 24);
         // FADB so parsing reaches the paged check
         let db = 0x100usize;
         buf[db..db + 4].copy_from_slice(b"FADB");
@@ -486,6 +575,8 @@ mod tests {
         buf[fahd + 7] = 10;
         buf[fahd + 8..fahd + 16].copy_from_slice(&u64::MAX.to_le_bytes()); // absurd count
         buf[fahd + 16..fahd + 24].copy_from_slice(&0x80u64.to_le_bytes());
+        // Valid checksum, so it is the element count that must be rejected.
+        stamp_checksum(&mut buf, fahd, fahd + 24);
         buf[0x80..0x84].copy_from_slice(b"FADB");
         let header = FixedArrayHeader::parse(&buf, fahd, 8, 8).unwrap();
         let r = read_fixed_array_chunks(&buf, &header, &[100], &[20], 8, 8, 8);
@@ -545,6 +636,7 @@ mod tests {
         file_data[fahd_offset + 8..fahd_offset + 16].copy_from_slice(&num_chunks.to_le_bytes());
         file_data[fahd_offset + 16..fahd_offset + 24]
             .copy_from_slice(&(db_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, fahd_offset, fahd_offset + 24);
 
         // Build FADB at db_offset
         file_data[db_offset..db_offset + 4].copy_from_slice(b"FADB");
@@ -562,6 +654,7 @@ mod tests {
             let pos = elem_start + i * os;
             file_data[pos..pos + os].copy_from_slice(&addr.to_le_bytes());
         }
+        stamp_checksum(&mut file_data, db_offset, elem_start + 5 * os);
 
         let header =
             FixedArrayHeader::parse(&file_data, fahd_offset, offset_size, length_size).unwrap();
@@ -611,6 +704,7 @@ mod tests {
         file_data[fahd_offset + 8..fahd_offset + 16].copy_from_slice(&num_chunks.to_le_bytes());
         file_data[fahd_offset + 16..fahd_offset + 24]
             .copy_from_slice(&(db_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, fahd_offset, fahd_offset + 24);
 
         file_data[db_offset..db_offset + 4].copy_from_slice(b"FADB");
         file_data[db_offset + 4] = 0;
@@ -632,6 +726,11 @@ mod tests {
             file_data[pos + os..pos + os + 4].copy_from_slice(&csize.to_le_bytes());
             file_data[pos + os + 4..pos + os + 8].copy_from_slice(&fmask.to_le_bytes());
         }
+        stamp_checksum(
+            &mut file_data,
+            db_offset,
+            elem_start + test_chunks.len() * elem_size,
+        );
 
         let header =
             FixedArrayHeader::parse(&file_data, fahd_offset, offset_size, length_size).unwrap();
@@ -696,6 +795,7 @@ mod tests {
         file_data[fahd_offset + 8..fahd_offset + 16].copy_from_slice(&num_elements.to_le_bytes());
         file_data[fahd_offset + 16..fahd_offset + 24]
             .copy_from_slice(&(db_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, fahd_offset, fahd_offset + 24);
 
         // FADB prefix
         file_data[db_offset..db_offset + 4].copy_from_slice(b"FADB");
@@ -715,6 +815,9 @@ mod tests {
         let base_addr = 0x1000u64;
         // Page 0 (elements 0..4) and page 2 (elements 8..11) carry addresses;
         // page 1's slot is left zero-filled and must be skipped.
+        // The prefix and bitmap carry one checksum, each initialised page
+        // another — as a real file does.
+        stamp_checksum(&mut file_data, db_offset, bitmap_off + bitmap_size);
         for &p in &[0usize, 2usize] {
             let page_off = pages_start + p * page_total;
             let count = core::cmp::min(page_nelmts, num_elements as usize - p * page_nelmts);
@@ -724,6 +827,7 @@ mod tests {
                 let pos = page_off + e * os;
                 file_data[pos..pos + os].copy_from_slice(&addr.to_le_bytes());
             }
+            stamp_checksum(&mut file_data, page_off, page_off + count * os);
         }
 
         let header =

crates/clawhdf5-format/src/lib.rs

@@ -89,6 +89,7 @@ pub mod object_header;
 pub mod object_header_writer;
 #[cfg(feature = "parallel")]
 pub mod parallel_read;
+pub mod partial_read;
 pub mod profiling;
 pub mod property_list;
 pub mod selection;

crates/clawhdf5-format/src/partial_read.rs

@@ -0,0 +1,359 @@
+//! Selection reads that cost what the selection costs, not what the dataset
+//! costs.
+//!
+//! [`crate::data_read::read_raw_data_selection`] used to decode the *entire*
+//! dataset and then pick elements out of it, so reading a 64x64 window of a
+//! large dataset took about as long as reading all of it. Here the selection's
+//! bounding box is materialised instead — only the rows of a contiguous
+//! dataset, or only the chunks, that overlap it — and the existing extractor
+//! runs over that small buffer with the selection translated to the box's
+//! origin. Extraction semantics are therefore exactly the full-read ones.
+
+#[cfg(not(feature = "std"))]
+use alloc::string as alloc_or_std;
+#[cfg(not(feature = "std"))]
+use alloc::{format, vec, vec::Vec};
+#[cfg(feature = "std")]
+use std::string as alloc_or_std;
+
+use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks};
+use crate::data_layout::DataLayout;
+use crate::data_read::extract_selection_from_buffer;
+use crate::dataspace::Dataspace;
+use crate::error::FormatError;
+use crate::filter_pipeline::FilterPipeline;
+use crate::filters::decompress_chunk;
+use crate::selection::Selection;
+
+/// The smallest axis-aligned box containing every selected element, as
+/// `(start, extent)` per dimension. `None` when there is nothing to gain or
+/// the selection is not valid for `dims` (the caller's full path then reports
+/// the error exactly as before).
+fn bounding_box(selection: &Selection, dims: &[u64]) -> Option<(Vec<u64>, Vec<u64>)> {
+    match selection {
+        Selection::Hyperslab {
+            start,
+            stride,
+            count,
+            block,
+        } => {
+            let rank = dims.len();
+            if [start.len(), stride.len(), count.len(), block.len()] != [rank; 4] {
+                return None;
+            }
+            let mut extent = Vec::with_capacity(rank);
+            for d in 0..rank {
+                if count[d] == 0 || block[d] == 0 {
+                    return None;
+                }
+                // Last selected index + 1, relative to start.
+                let span = (count[d] - 1)
+                    .checked_mul(stride[d])?
+                    .checked_add(block[d])?;
+                if start[d].checked_add(span)? > dims[d] {
+                    return None;
+                }
+                extent.push(span);
+            }
+            Some((start.clone(), extent))
+        }
+        Selection::Points(points) => {
+            let rank = dims.len();
+            let first = points.first()?;
+            if first.len() != rank {
+                return None;
+            }
+            let (mut lo, mut hi) = (first.clone(), first.clone());
+            for p in points {
+                if p.len() != rank {
+                    return None;
+                }
+                for d in 0..rank {
+                    if p[d] >= dims[d] {
+                        return None;
+                    }
+                    lo[d] = lo[d].min(p[d]);
+                    hi[d] = hi[d].max(p[d]);
+                }
+            }
+            let extent = lo.iter().zip(&hi).map(|(l, h)| h - l + 1).collect();
+            Some((lo, extent))
+        }
+        Selection::All | Selection::None => None,
+    }
+}
+
+/// Check that `selection` addresses only elements that exist in a dataset of
+/// shape `dims`. Without this an out-of-range selection read *something*: a
+/// hyperslab past the edge came back padded with zeros, and a point whose
+/// column was out of range wrapped into the next row.
+pub fn validate(selection: &Selection, dims: &[u64]) -> Result<(), FormatError> {
+    let rank = dims.len();
+    let bad = |msg: alloc_or_std::String| Err(FormatError::SelectionOutOfBounds(msg));
+    match selection {
+        Selection::All | Selection::None => Ok(()),
+        Selection::Hyperslab {
+            start,
+            stride,
+            count,
+            block,
+        } => {
+            if [start.len(), stride.len(), count.len(), block.len()] != [rank; 4] {
+                return bad(format!("hyperslab rank does not match dataset rank {rank}"));
+            }
+            for d in 0..rank {
+                if count[d] == 0 || block[d] == 0 {
+                    continue; // selects nothing along this dimension
+                }
+                let end = (count[d] - 1)
+                    .checked_mul(stride[d])
+                    .and_then(|v| v.checked_add(block[d]))
+                    .and_then(|v| v.checked_add(start[d]));
+                if !end.is_some_and(|end| end <= dims[d]) {
+                    return bad(format!(
+                        "dimension {d}: start {} stride {} count {} block {} exceeds extent {}",
+                        start[d], stride[d], count[d], block[d], dims[d]
+                    ));
+                }
+                if block[d] > stride[d] && count[d] > 1 {
+                    return bad(format!(
+                        "dimension {d}: block {} larger than stride {} (overlapping blocks)",
+                        block[d], stride[d]
+                    ));
+                }
+            }
+            Ok(())
+        }
+        Selection::Points(points) => {
+            for p in points {
+                if p.len() != rank {
+                    return bad(format!("point {p:?} does not match dataset rank {rank}"));
+                }
+                if let Some(d) = (0..rank).find(|&d| p[d] >= dims[d]) {
+                    return bad(format!(
+                        "point {p:?}: coordinate {} exceeds extent {} of dimension {d}",
+                        p[d], dims[d]
+                    ));
+                }
+            }
+            Ok(())
+        }
+    }
+}
+
+/// The same selection expressed relative to `origin`.
+fn translate(selection: &Selection, origin: &[u64]) -> Selection {
+    match selection {
+        Selection::Hyperslab {
+            start,
+            stride,
+            count,
+            block,
+        } => Selection::Hyperslab {
+            start: start.iter().zip(origin).map(|(s, o)| s - o).collect(),
+            stride: stride.clone(),
+            count: count.clone(),
+            block: block.clone(),
+        },
+        Selection::Points(points) => Selection::Points(
+            points
+                .iter()
+                .map(|p| p.iter().zip(origin).map(|(c, o)| c - o).collect())
+                .collect(),
+        ),
+        other => other.clone(),
+    }
+}
+
+/// Copy the part of a source region that overlaps the box into `out` (which
+/// is the box, row-major).
+///
+/// The source region starts at `src_origin` in dataset coordinates, has shape
+/// `src_shape`, and its elements are in `src` row-major. One `memcpy` per
+/// overlapping row of the last dimension.
+#[allow(clippy::too_many_arguments)]
+fn copy_overlap(
+    src: &[u8],
+    src_origin: &[u64],
+    src_shape: &[u64],
+    out: &mut [u8],
+    box_start: &[u64],
+    box_extent: &[u64],
+    elem_size: usize,
+) {
+    let rank = box_start.len();
+    // Overlap in dataset coordinates.
+    let mut lo = vec![0u64; rank];
+    let mut hi = vec![0u64; rank];
+    for d in 0..rank {
+        lo[d] = src_origin[d].max(box_start[d]);
+        hi[d] = (src_origin[d] + src_shape[d]).min(box_start[d] + box_extent[d]);
+        if lo[d] >= hi[d] {
+            return;
+        }
+    }
+    let strides = |shape: &[u64]| {
+        let mut s = vec![1u64; rank];
+        for d in (0..rank.saturating_sub(1)).rev() {
+            s[d] = s[d + 1] * shape[d + 1];
+        }
+        s
+    };
+    let (src_strides, out_strides) = (strides(src_shape), strides(box_extent));
+    let last = rank - 1;
+    let run = ((hi[last] - lo[last]) as usize) * elem_size;
+
+    let mut idx = lo.clone();
+    loop {
+        let src_at: u64 = (0..rank)
+            .map(|d| (idx[d] - src_origin[d]) * src_strides[d])
+            .sum();
+        let out_at: u64 = (0..rank)
+            .map(|d| (idx[d] - box_start[d]) * out_strides[d])
+            .sum();
+        let (s, o) = (src_at as usize * elem_size, out_at as usize * elem_size);
+        if let (Some(from), Some(to)) = (src.get(s..s + run), out.get_mut(o..o + run)) {
+            to.copy_from_slice(from);
+        }
+        // Advance over every dimension but the last.
+        let mut d = last;
+        loop {
+            if d == 0 {
+                return;
+            }
+            d -= 1;
+            idx[d] += 1;
+            if idx[d] < hi[d] {
+                break;
+            }
+            idx[d] = lo[d];
+        }
+    }
+}
+
+/// Read `selection` without materialising the whole dataset, when that is
+/// possible and worthwhile. `Ok(None)` means "use the full-read path": an
+/// `All`/`None`/invalid selection, a layout this doesn't handle (compact,
+/// virtual, storage-less), or a bounding box covering most of the dataset.
+#[allow(clippy::too_many_arguments)]
+pub fn read_selection(
+    file_data: &[u8],
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+    selection: &Selection,
+) -> Result<Option<Vec<u8>>, FormatError> {
+    let dims = &dataspace.dimensions;
+    if dims.is_empty() || elem_size == 0 {
+        return Ok(None);
+    }
+    let Some((box_start, box_extent)) = bounding_box(selection, dims) else {
+        return Ok(None);
+    };
+    let total = dataspace.checked_num_elements()?;
+    let box_elements = box_extent
+        .iter()
+        .try_fold(1u64, |acc, &e| acc.checked_mul(e))
+        .ok_or_else(|| FormatError::Overflow("selection bounding box overflows".into()))?;
+    // A box covering most of the dataset gains nothing over the full path.
+    if box_elements.saturating_mul(2) > total {
+        return Ok(None);
+    }
+    let mut boxed = alloc_output(checked_byte_len(box_elements, elem_size)?)?;
+
+    match layout {
+        DataLayout::Contiguous {
+            address: Some(address),
+            ..
+        } => {
+            let base = usize::try_from(*address)
+                .map_err(|_| FormatError::Overflow("data address exceeds usize".into()))?;
+            let data = file_data
+                .get(base..)
+                .and_then(|d| d.get(..checked_byte_len(total, elem_size).ok()?))
+                .ok_or(FormatError::UnexpectedEof {
+                    expected: base,
+                    available: file_data.len(),
+                })?;
+            let origin = vec![0u64; dims.len()];
+            copy_overlap(
+                data,
+                &origin,
+                dims,
+                &mut boxed,
+                &box_start,
+                &box_extent,
+                elem_size,
+            );
+        }
+        DataLayout::Chunked {
+            btree_address: Some(_),
+            ..
+        } => {
+            let (chunks, chunk_dims) = list_chunks(
+                file_data,
+                layout,
+                dataspace,
+                elem_size,
+                offset_size,
+                length_size,
+            )?;
+            let rank = dims.len();
+            let chunk_shape: Vec<u64> = chunk_dims.iter().map(|&d| d as u64).collect();
+            let chunk_bytes = crate::chunked_read::checked_chunk_byte_len(&chunk_dims, elem_size)?;
+            for chunk in &chunks {
+                if chunk.offsets.len() < rank || chunk.address == u64::MAX {
+                    continue;
+                }
+                let origin = &chunk.offsets[..rank];
+                let overlaps = (0..rank).all(|d| {
+                    origin[d] < box_start[d] + box_extent[d]
+                        && origin[d].saturating_add(chunk_shape[d]) > box_start[d]
+                });
+                if !overlaps {
+                    continue;
+                }
+                let at = usize::try_from(chunk.address)
+                    .map_err(|_| FormatError::Overflow("chunk address exceeds usize".into()))?;
+                let raw = at
+                    .checked_add(chunk.chunk_size as usize)
+                    .and_then(|end| file_data.get(at..end))
+                    .ok_or(FormatError::UnexpectedEof {
+                        expected: at.saturating_add(chunk.chunk_size as usize),
+                        available: file_data.len(),
+                    })?;
+                // Mirrors the full-read path: a non-zero filter mask means the
+                // chunk was stored unfiltered.
+                let decoded;
+                let data: &[u8] = match pipeline {
+                    Some(pl) if chunk.filter_mask == 0 => {
+                        decoded = decompress_chunk(raw, pl, chunk_bytes, elem_size as u32)?;
+                        &decoded
+                    }
+                    _ => raw,
+                };
+                copy_overlap(
+                    data,
+                    origin,
+                    &chunk_shape,
+                    &mut boxed,
+                    &box_start,
+                    &box_extent,
+                    elem_size,
+                );
+            }
+        }
+        _ => return Ok(None),
+    }
+
+    extract_selection_from_buffer(
+        &boxed,
+        &box_extent,
+        elem_size,
+        &translate(selection, &box_start),
+    )
+    .map(Some)
+}

crates/clawhdf5-format/tests/fixtures/gen_std_ref.py

@@ -0,0 +1,44 @@
+"""Generate std_ref_hdf5_2_0.h5: a dataset of H5T_STD_REF (the reference
+datatype introduced in HDF5 1.12, datatype message version 4) holding two
+object references — to /target (a dataset) and /grp (a group).
+
+h5py has no API for this type, so the file is written by calling the libhdf5
+bundled in the h5py wheel directly through ctypes. Written with h5py 3.16.0 /
+HDF5 2.0.0. Re-run only if the fixture ever needs regenerating:
+
+    python gen_std_ref.py std_ref_hdf5_2_0.h5
+"""
+import ctypes
+import glob
+import os
+import sys
+
+import h5py
+import numpy as np
+
+libdir = os.path.join(os.path.dirname(os.path.dirname(h5py.__file__)), "h5py.libs")
+libs = [p for p in glob.glob(os.path.join(libdir, "libhdf5*.so*")) if "_hl" not in os.path.basename(p)]
+lib = ctypes.CDLL(libs[0])
+lib.H5open()
+hid = ctypes.c_int64
+std_ref = hid.in_dll(lib, "H5T_STD_REF_g").value
+
+lib.H5Screate_simple.restype = hid
+lib.H5Screate_simple.argtypes = [ctypes.c_int, ctypes.POINTER(ctypes.c_uint64), ctypes.POINTER(ctypes.c_uint64)]
+lib.H5Dcreate2.restype = hid
+lib.H5Dcreate2.argtypes = [hid, ctypes.c_char_p, hid, hid, hid, hid, hid]
+lib.H5Rcreate_object.argtypes = [hid, ctypes.c_char_p, hid, ctypes.c_void_p]
+lib.H5Dwrite.argtypes = [hid, hid, hid, hid, hid, ctypes.c_void_p]
+lib.H5Dclose.argtypes = [hid]
+
+with h5py.File(sys.argv[1], "w", libver="latest") as f:
+    f.create_dataset("target", data=np.arange(5, dtype="<i4"))
+    f.create_group("grp")
+    fid = f.id.id
+    sid = lib.H5Screate_simple(1, (ctypes.c_uint64 * 1)(2), None)
+    did = lib.H5Dcreate2(fid, b"refs", std_ref, sid, 0, 0, 0)
+    refs = ((ctypes.c_ubyte * 64) * 2)()  # H5R_ref_t is a 64-byte buffer
+    assert lib.H5Rcreate_object(fid, b"/target", 0, ctypes.byref(refs[0])) == 0
+    assert lib.H5Rcreate_object(fid, b"/grp", 0, ctypes.byref(refs[1])) == 0
+    assert lib.H5Dwrite(did, std_ref, 0, 0, 0, ctypes.byref(refs)) == 0
+    lib.H5Dclose(did)

crates/clawhdf5-format/tests/reference_tests.rs

@@ -2,6 +2,15 @@
 
 use clawhdf5_format::data_read::{read_object_references, read_region_references};
 use clawhdf5_format::datatype::{Datatype, ReferenceType};
+/// The Python interpreter to drive interop checks with.
+///
+/// `CLAWHDF5_PYTHON` lets these run against a virtualenv holding h5py, which
+/// on a PEP 668 "externally managed" system is the only place it can be
+/// installed. Without it the suite silently skips, and a silent skip here is
+/// how a datatype bug once reached a release.
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
 
 #[test]
 fn object_ref_single_valid() {
@@ -173,7 +182,7 @@ print('ok')
 "#,
         path.display()
     );
-    let output = std::process::Command::new("python3")
+    let output = std::process::Command::new(python())
         .args(["-c", &script])
         .output();
 
@@ -316,3 +325,97 @@ print('ok')
     // Clean up
     let _ = std::fs::remove_file(&path);
 }
+
+// ---------------------------------------------------------------------------
+// H5T_STD_REF (HDF5 1.12+ references, datatype message version 4)
+// ---------------------------------------------------------------------------
+
+/// `fixtures/std_ref_hdf5_2_0.h5` (see `gen_std_ref.py`) holds a dataset of
+/// `H5T_STD_REF` with two object references, written by HDF5 2.0 itself. The
+/// datatype used to be rejected with `InvalidReferenceType(2)`.
+#[test]
+fn std_ref_object_references_from_hdf5_2_0() {
+    use clawhdf5_format::data_layout::DataLayout;
+    use clawhdf5_format::dataspace::Dataspace;
+    use clawhdf5_format::group_v2::resolve_path_any;
+    use clawhdf5_format::message_type::MessageType;
+    use clawhdf5_format::object_header::ObjectHeader;
+    use clawhdf5_format::signature::find_signature;
+    use clawhdf5_format::superblock::Superblock;
+
+    let bytes: &[u8] = include_bytes!("fixtures/std_ref_hdf5_2_0.h5");
+    let sb = Superblock::parse(bytes, find_signature(bytes).unwrap()).unwrap();
+    let (os, ls) = (sb.offset_size, sb.length_size);
+
+    let refs_addr = resolve_path_any(bytes, &sb, "refs").unwrap();
+    let header = ObjectHeader::parse(bytes, refs_addr as usize, os, ls).unwrap();
+    let message = |t: MessageType| {
+        &header
+            .messages
+            .iter()
+            .find(|m| m.msg_type == t)
+            .unwrap()
+            .data
+    };
+
+    let (datatype, _) = Datatype::parse(message(MessageType::Datatype)).unwrap();
+    assert_eq!(
+        datatype,
+        Datatype::Reference {
+            size: 18,
+            ref_type: ReferenceType::Object2
+        }
+    );
+    let dataspace = Dataspace::parse(message(MessageType::Dataspace), ls).unwrap();
+    let layout = DataLayout::parse(message(MessageType::DataLayout), os, ls).unwrap();
+    let raw =
+        clawhdf5_format::data_read::read_raw_data(bytes, &layout, &dataspace, &datatype).unwrap();
+    assert_eq!(raw.len(), 2 * 18);
+
+    // The references point at the objects they were created from.
+    let refs = read_object_references(&raw, &datatype, os).unwrap();
+    let addresses: Vec<u64> = refs.iter().map(|r| r.address).collect();
+    assert_eq!(
+        addresses,
+        [
+            resolve_path_any(bytes, &sb, "target").unwrap(),
+            resolve_path_any(bytes, &sb, "grp").unwrap(),
+        ]
+    );
+    // And what they point at is a real object header.
+    for address in addresses {
+        ObjectHeader::parse(bytes, address as usize, os, ls).unwrap();
+    }
+}
+
+#[test]
+fn std_ref_decoding_rejects_malformed_elements() {
+    let dt = Datatype::Reference {
+        size: 18,
+        ref_type: ReferenceType::Object2,
+    };
+    let mut good = vec![0u8; 18];
+    good[..4].copy_from_slice(&[2, 0, 8, 0xb3]);
+    assert_eq!(
+        read_object_references(&good, &dt, 8).unwrap()[0].address,
+        0xb3
+    );
+
+    // Null reference.
+    assert_eq!(
+        read_object_references(&[0u8; 18], &dt, 8).unwrap()[0].address,
+        u64::MAX
+    );
+    for (what, patch) in [
+        ("wrong reference type", (0usize, 3u8)),
+        ("external flag", (1, 1)),
+        ("token longer than the element", (2, 200)),
+        ("zero-length token", (2, 0)),
+    ] {
+        let mut bad = good.clone();
+        bad[patch.0] = patch.1;
+        assert!(read_object_references(&bad, &dt, 8).is_err(), "{what}");
+    }
+    // Not a whole number of elements.
+    assert!(read_object_references(&good[..17], &dt, 8).is_err());
+}

crates/clawhdf5-format/tests/writer_h5py_tests.rs

@@ -4,9 +4,18 @@
 //! (and vice versa). They require python3 + h5py to be installed.
 
 use clawhdf5_format::file_writer::{AttrValue, CompoundTypeBuilder, EnumTypeBuilder, FileWriter};
+/// The Python interpreter to drive interop checks with.
+///
+/// `CLAWHDF5_PYTHON` lets these run against a virtualenv holding h5py, which
+/// on a PEP 668 "externally managed" system is the only place it can be
+/// installed. Without it the suite silently skips, and a silent skip here is
+/// how a datatype bug once reached a release.
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
 
 fn h5py_available() -> bool {
-    std::process::Command::new("python3")
+    std::process::Command::new(python())
         .args(["-c", "import h5py"])
         .output()
         .map(|o| o.status.success())
@@ -17,10 +26,10 @@ fn h5py_read(_path: &std::path::Path, script: &str) -> String {
     if !h5py_available() {
         panic!("h5py not installed — skipping interop test");
     }
-    let o = std::process::Command::new("python3")
+    let o = std::process::Command::new(python())
         .args(["-c", script])
         .output()
-        .expect("python3");
+        .expect("python interpreter");
     if !o.status.success() {
         panic!("h5py: {}", String::from_utf8_lossy(&o.stderr));
     }

crates/clawhdf5-gpu/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-gpu"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "GPU-accelerated vector operations for rustyhdf5 using wgpu compute shaders"
 license = "MIT"

crates/clawhdf5-io/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-io"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "I/O abstraction layer for rustyhdf5"
 license = "MIT"
@@ -10,7 +10,7 @@ keywords = ["hdf5", "io", "science", "data"]
 categories = ["filesystem", "science"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 memmap2 = { version = "0.9", optional = true }
 libc = { version = "0.2", optional = true }
 tokio = { version = "1", features = ["fs", "io-util"], optional = true }

crates/clawhdf5-migrate/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-migrate"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "CLI to migrate SQLite agent memory databases to HDF5 format"
 license = "MIT"
@@ -14,9 +14,9 @@ name = "clawhdf5-migrate"
 path = "src/main.rs"
 
 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.4.0" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0" }
-clawhdf5 = { path = "../clawhdf5", version = "2.4.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
 rusqlite = { version = "0.31", features = ["bundled"] }
 clap = { version = "4", features = ["derive"] }
 half = { workspace = true }

crates/clawhdf5-napi/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-napi"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "Node.js native addon (napi-rs) exposing clawhdf5-agent to TypeScript/JavaScript"
 license = "MIT"
@@ -10,7 +10,7 @@ repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
 crate-type = ["cdylib"]
 
 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.4.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
 napi = { version = "2", default-features = false, features = ["napi9"] }
 napi-derive = "2"
 

crates/clawhdf5-netcdf4/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-netcdf4"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "NetCDF-4 read support built on rustyhdf5 — pure Rust, no C dependencies"
 license = "MIT"
@@ -10,8 +10,8 @@ keywords = ["netcdf", "netcdf4", "hdf5", "science", "climate"]
 categories = ["parser-implementations", "science"]
 
 [dependencies]
-clawhdf5 = { path = "../clawhdf5", version = "2.4.0" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0" }
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 
 [dev-dependencies]
 tempfile = { workspace = true }

crates/clawhdf5-netcdf4/tests/interop_tests.rs

@@ -9,6 +9,15 @@ use clawhdf5_netcdf4::{AttrValue, NetCDF4File};
 // ---------------------------------------------------------------------------
 // Helpers
 // ---------------------------------------------------------------------------
+/// The Python interpreter to drive interop checks with.
+///
+/// `CLAWHDF5_PYTHON` lets these run against a virtualenv holding h5py, which
+/// on a PEP 668 "externally managed" system is the only place it can be
+/// installed. Without it the suite silently skips, and a silent skip here is
+/// how a datatype bug once reached a release.
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
 
 /// When `CLAWHDF5_REQUIRE_INTEROP=1` (set in CI), a missing Python dependency
 /// is a test failure instead of a silent skip.
@@ -17,7 +26,7 @@ fn interop_required() -> bool {
 }
 
 fn netcdf4_python_available() -> bool {
-    Command::new("python3")
+    Command::new(python())
         .args(["-c", "import netCDF4; print(netCDF4.__version__)"])
         .output()
         .map(|o| o.status.success())
@@ -25,7 +34,7 @@ fn netcdf4_python_available() -> bool {
 }
 
 fn xarray_available() -> bool {
-    Command::new("python3")
+    Command::new(python())
         .args(["-c", "import xarray; print(xarray.__version__)"])
         .output()
         .map(|o| o.status.success())
@@ -59,7 +68,7 @@ macro_rules! skip_if_no_xarray {
 }
 
 fn run_python(script: &str) {
-    let output = Command::new("python3")
+    let output = Command::new(python())
         .args(["-c", script])
         .output()
         .expect("failed to run python3");

crates/clawhdf5-py/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-py"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 license = "MIT"
@@ -14,8 +14,8 @@ name = "clawhdf5"
 crate-type = ["cdylib", "rlib"]
 
 [dependencies]
-clawhdf5_rs = { path = "../clawhdf5", version = "2.4.0", package = "clawhdf5" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0" }
+clawhdf5_rs = { path = "../clawhdf5", version = "2.7.0", package = "clawhdf5" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 pyo3 = "0.29"
 numpy = "0.29"
 

crates/clawhdf5-py/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "rustyhdf5"
-version = "2.4.0"
+version = "2.7.0"
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 requires-python = ">=3.8"
 license = { text = "MIT" }

crates/clawhdf5/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5"
-version = "2.4.0"
+version = "2.7.0"
 edition = "2024"
 description = "Pure-Rust HDF5 reader/writer — no C dependencies"
 license = "MIT"
@@ -10,16 +10,16 @@ keywords = ["hdf5", "science", "data", "binary"]
 categories = ["parser-implementations", "science", "encoding"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.4.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0" }
 rayon = { version = "1", optional = true }
 
 [dev-dependencies]
 tempfile = { workspace = true }
 criterion = { workspace = true }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.4.0", features = ["mmap"] }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.4.0", features = ["parallel", "fast-checksum"] }
-clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.4.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0", features = ["mmap"] }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0", features = ["parallel", "fast-checksum"] }
+clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.7.0" }
 
 [[bench]]
 name = "mmap_bench"

crates/clawhdf5/src/reader.rs

@@ -381,8 +381,13 @@ impl<'f> Dataset<'f> {
 
     /// Read all data as `f64` values.
     pub fn read_f64(&self) -> Result<Vec<f64>, Error> {
-        let raw = self.read_raw()?;
         let dt = self.datatype()?;
+        // A contiguous dataset is converted straight from the file bytes; going
+        // through `read_raw` first copied the whole dataset an extra time.
+        if let Ok(Some(bytes)) = self.read_raw_ref() {
+            return Ok(data_read::read_as_f64(bytes, &dt)?);
+        }
+        let raw = self.read_raw()?;
         Ok(data_read::read_as_f64(&raw, &dt)?)
     }
 
@@ -393,29 +398,49 @@ impl<'f> Dataset<'f> {
     ///
     /// Read all data as `f32` values.
     pub fn read_f32(&self) -> Result<Vec<f32>, Error> {
-        let raw = self.read_raw()?;
         let dt = self.datatype()?;
+        // A contiguous dataset is converted straight from the file bytes; going
+        // through `read_raw` first copied the whole dataset an extra time.
+        if let Ok(Some(bytes)) = self.read_raw_ref() {
+            return Ok(data_read::read_as_f32(bytes, &dt)?);
+        }
+        let raw = self.read_raw()?;
         Ok(data_read::read_as_f32(&raw, &dt)?)
     }
 
     /// Read all data as `i32` values.
     pub fn read_i32(&self) -> Result<Vec<i32>, Error> {
-        let raw = self.read_raw()?;
         let dt = self.datatype()?;
+        // A contiguous dataset is converted straight from the file bytes; going
+        // through `read_raw` first copied the whole dataset an extra time.
+        if let Ok(Some(bytes)) = self.read_raw_ref() {
+            return Ok(data_read::read_as_i32(bytes, &dt)?);
+        }
+        let raw = self.read_raw()?;
         Ok(data_read::read_as_i32(&raw, &dt)?)
     }
 
     /// Read all data as `i64` values.
     pub fn read_i64(&self) -> Result<Vec<i64>, Error> {
-        let raw = self.read_raw()?;
         let dt = self.datatype()?;
+        // A contiguous dataset is converted straight from the file bytes; going
+        // through `read_raw` first copied the whole dataset an extra time.
+        if let Ok(Some(bytes)) = self.read_raw_ref() {
+            return Ok(data_read::read_as_i64(bytes, &dt)?);
+        }
+        let raw = self.read_raw()?;
         Ok(data_read::read_as_i64(&raw, &dt)?)
     }
 
     /// Read all data as `u64` values.
     pub fn read_u64(&self) -> Result<Vec<u64>, Error> {
-        let raw = self.read_raw()?;
         let dt = self.datatype()?;
+        // A contiguous dataset is converted straight from the file bytes; going
+        // through `read_raw` first copied the whole dataset an extra time.
+        if let Ok(Some(bytes)) = self.read_raw_ref() {
+            return Ok(data_read::read_as_u64(bytes, &dt)?);
+        }
+        let raw = self.read_raw()?;
         Ok(data_read::read_as_u64(&raw, &dt)?)
     }
 
@@ -458,6 +483,7 @@ impl<'f> Dataset<'f> {
             || (matches!(dl, DataLayout::Chunked { .. })
                 && !clawhdf5_format::fill_value::is_default(fill.as_deref()));
         if fill_matters {
+            clawhdf5_format::partial_read::validate(selection, &ds.dimensions)?;
             let full = self.read_raw()?;
             return Ok(data_read::extract_selection_from_buffer(
                 &full,

crates/clawhdf5/tests/h5py_interop_tests.rs

@@ -9,6 +9,15 @@ use clawhdf5::{AttrValue, CompoundTypeBuilder, DType, File, FileBuilder};
 // ---------------------------------------------------------------------------
 // Helpers
 // ---------------------------------------------------------------------------
+/// The Python interpreter to drive interop checks with.
+///
+/// `CLAWHDF5_PYTHON` lets these run against a virtualenv holding h5py, which
+/// on a PEP 668 "externally managed" system is the only place it can be
+/// installed. Without it the suite silently skips, and a silent skip here is
+/// how a datatype bug once reached a release.
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
 
 /// When `CLAWHDF5_REQUIRE_INTEROP=1` (set in CI), a missing Python dependency
 /// is a test failure instead of a silent skip.
@@ -17,7 +26,7 @@ fn interop_required() -> bool {
 }
 
 fn python_available() -> bool {
-    Command::new("python3")
+    Command::new(python())
         .args(["-c", "import h5py; print(h5py.__version__)"])
         .output()
         .map(|o| o.status.success())
@@ -39,7 +48,7 @@ macro_rules! skip_if_no_python {
 
 /// Run a Python script and panic if it fails.
 fn run_python(script: &str) {
-    let output = Command::new("python3")
+    let output = Command::new(python())
         .args(["-c", script])
         .output()
         .expect("failed to run python3");
@@ -52,7 +61,7 @@ fn run_python(script: &str) {
 
 /// Run a Python script and return stdout as a trimmed string.
 fn run_python_output(script: &str) -> String {
-    let output = Command::new("python3")
+    let output = Command::new(python())
         .args(["-c", script])
         .output()
         .expect("failed to run python3");
@@ -913,3 +922,468 @@ with h5py.File("{dst_str}", "r") as f:
         "[(1, 2.5), (3, 4.5)] ('a', 'b') [18446744073709551615, 0, 9223372036854775808] uint64"
     );
 }
+
+// ---------------------------------------------------------------------------
+// h5py writes datasets indexed by a version-2 B-tree -> clawhdf5 reads
+// ---------------------------------------------------------------------------
+
+/// With `libver='latest'`, a chunked dataset with two or more unlimited
+/// dimensions indexes its chunks with a version-2 B-tree (layout v4, index
+/// type 5). These used to fail with "unsupported chunked layout".
+#[test]
+fn h5py_btree_v2_chunk_index_clawhdf5_reads() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("bt2.h5");
+    let path_str = path.display().to_string();
+    let script = format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    a = np.arange(60 * 45, dtype="<i4").reshape(60, 45)
+    f.create_dataset("plain", data=a, chunks=(7, 8), maxshape=(None, None))
+    f.create_dataset("gz", data=a, chunks=(7, 8), maxshape=(None, None), compression="gzip", shuffle=True)
+    # Enough chunks (2500) that the tree has internal nodes.
+    big = np.arange(200 * 200, dtype="<i4").reshape(200, 200)
+    f.create_dataset("deep", data=big, chunks=(4, 4), maxshape=(None, None))
+    s = f.create_dataset("sparse", shape=(30, 30), dtype="<i4", chunks=(5, 5), maxshape=(None, None), fillvalue=-9)
+    s[10:15, 20:25] = 4
+    s[29, 29] = 1
+with h5py.File("{path_str}", "r") as f:
+    print("sparse", f["sparse"][...].ravel().tolist())
+    print("slab", f["deep"][37:141:13, 5:190:31].ravel().tolist())
+"#
+    );
+    let out = run_python_output(&script);
+    let expected: std::collections::HashMap<&str, Vec<i32>> = out
+        .lines()
+        .map(|l| {
+            let (name, list) = l.split_once(' ').unwrap();
+            (name, parse_int_list(list))
+        })
+        .collect();
+
+    let file = File::open(&path).unwrap();
+    let small: Vec<i32> = (0..60 * 45).collect();
+    assert_eq!(file.dataset("plain").unwrap().read_i32().unwrap(), small);
+    assert_eq!(file.dataset("gz").unwrap().read_i32().unwrap(), small);
+    let deep: Vec<i32> = (0..200 * 200).collect();
+    assert_eq!(file.dataset("deep").unwrap().read_i32().unwrap(), deep);
+    assert_eq!(
+        file.dataset("sparse").unwrap().read_i32().unwrap(),
+        expected["sparse"]
+    );
+    // Partial read through the same index: rows 37,50,..,128 x cols 5,36,..,160.
+    let slab = clawhdf5_format::selection::Selection::Hyperslab {
+        start: vec![37, 5],
+        stride: vec![13, 31],
+        count: vec![8, 6],
+        block: vec![1, 1],
+    };
+    assert_eq!(
+        file.dataset("deep")
+            .unwrap()
+            .read_i32_selection(&slab)
+            .unwrap(),
+        expected["slab"]
+    );
+}
+
+// ---------------------------------------------------------------------------
+// clawhdf5 auto-chunks a large compressed dataset -> h5py reads
+// ---------------------------------------------------------------------------
+
+/// Compression without explicit chunk dimensions used to store the whole
+/// dataset as a single chunk. Large datasets are now split automatically;
+/// h5py must read the result and see sensibly sized chunks.
+#[test]
+fn clawhdf5_auto_chunked_dataset_h5py_reads() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("auto_chunk.h5");
+    let path_str = path.display().to_string();
+
+    let (rows, cols) = (1500u64, 1100u64); // 13.2 MB of f64
+    let data: Vec<f64> = (0..rows * cols).map(|i| (i % 9973) as f64 * 0.25).collect();
+    let mut builder = FileBuilder::new();
+    builder
+        .create_dataset("big")
+        .with_f64_data(&data)
+        .with_shape(&[rows, cols])
+        .with_deflate(4);
+    builder
+        .create_dataset("small")
+        .with_f64_data(&data[..600])
+        .with_shape(&[20, 30])
+        .with_deflate(4);
+    builder.write(&path).unwrap();
+
+    let out = run_python_output(&format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "r") as f:
+    big, small = f["big"], f["small"]
+    expect = (np.arange(1500 * 1100) % 9973) * 0.25
+    ok = bool(np.array_equal(big[...].ravel(), expect)) and bool(np.array_equal(small[...].ravel(), expect[:600]))
+    chunk_bytes = int(np.prod(big.chunks)) * 8
+    print(ok, chunk_bytes <= 1 << 20, chunk_bytes >= 1 << 17, small.chunks == (20, 30), big.compression)
+"#
+    ));
+    assert_eq!(out.trim(), "True True True True gzip");
+
+    // And it reads back here, in full and partially.
+    let file = File::open(&path).unwrap();
+    let ds = file.dataset("big").unwrap();
+    assert_eq!(ds.read_f64().unwrap(), data);
+    let row = clawhdf5_format::selection::Selection::Hyperslab {
+        start: vec![777, 0],
+        stride: vec![1, 1],
+        count: vec![1, cols],
+        block: vec![1, 1],
+    };
+    let start = (777 * cols) as usize;
+    assert_eq!(
+        ds.read_f64_selection(&row).unwrap(),
+        data[start..start + cols as usize]
+    );
+}
+
+#[test]
+fn h5py_deep_btree_v2_chunk_index_clawhdf5_reads() {
+    // Two unlimited dimensions give a B-tree v2 chunk index, and 2x2 chunks
+    // over 400x400 give 40 000 index records — enough for HDF5 to build a
+    // tree of depth 2. Small h5py files only ever produce depth-0 trees, so
+    // this is the one fixture that walks internal nodes: the path where the
+    // traversal's record budget (the guard against crafted shared-subtree
+    // trees) is spent, which must never refuse a real file.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("deep_btree.h5");
+    let path_str = path.display().to_string();
+
+    let script = format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=(400, 400), maxshape=(None, None),
+                         chunks=(2, 2), dtype="i4")
+    d[...] = np.arange(160000, dtype="i4").reshape(400, 400)
+"#
+    );
+    run_python(&script);
+
+    // The fixture is only meaningful if HDF5 really built internal nodes.
+    let bytes = std::fs::read(&path).unwrap();
+    let at = bytes
+        .windows(4)
+        .position(|w| w == b"BTHD")
+        .expect("expected a B-tree v2 chunk index");
+    let depth = u16::from_le_bytes([bytes[at + 12], bytes[at + 13]]);
+    assert!(
+        depth >= 1,
+        "fixture tree has depth {depth}; it tests nothing"
+    );
+
+    let file = File::open(&path).unwrap();
+    let values = file.dataset("x").unwrap().read_i32().unwrap();
+    assert_eq!(values.len(), 160_000);
+    for (i, &v) in values.iter().enumerate() {
+        assert_eq!(v, i as i32, "element {i}");
+    }
+}
+
+#[test]
+fn h5py_extensible_array_chunk_index_clawhdf5_reads() {
+    // One unlimited dimension means an Extensible Array chunk index. Only its
+    // first few elements live inline in the index block (4 by default), and
+    // every other fixture here is small enough to stop there — which is how
+    // the data block and super block layouts came to be wrong without a test
+    // noticing. The counts below step over each boundary in turn:
+    //   4      inline elements only
+    //   37     past the first direct data block
+    //   400    into the first super block
+    //   5000   several super block levels
+    //   200000 data blocks large enough to be paged
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+
+    for n in [4usize, 37, 400, 5_000, 200_000] {
+        let path = dir.path().join(format!("ea_{n}.h5"));
+        let path_str = path.display().to_string();
+        run_python(&format!(
+            r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=({n},), maxshape=(None,), chunks=(1,), dtype="i4")
+    d[...] = np.arange({n}, dtype="i4")
+"#
+        ));
+
+        let bytes = std::fs::read(&path).unwrap();
+        assert!(
+            bytes.windows(4).any(|w| w == b"EAHD"),
+            "n={n}: fixture is not indexed by an Extensible Array"
+        );
+
+        let file = File::open(&path).unwrap();
+        let values = file.dataset("x").unwrap().read_i32().unwrap();
+        assert_eq!(values.len(), n, "n={n}");
+        let wrong = values
+            .iter()
+            .enumerate()
+            .filter(|&(i, &v)| v != i as i32)
+            .count();
+        assert_eq!(wrong, 0, "n={n}: {wrong} of {n} elements read back wrong");
+    }
+}
+
+#[test]
+fn h5py_sparse_extensible_array_leaves_pages_uninitialised() {
+    // Writing a scattered subset leaves whole pages of a paged data block
+    // never initialised. Those pages still occupy their slot on disk, so the
+    // reader has to skip them by stride and take the fill value instead —
+    // driven by the page-init bitmap, which is packed one bit per page across
+    // the whole super block, MSB first.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("ea_sparse.h5");
+    let path_str = path.display().to_string();
+    let n = 200_000usize;
+    let step = 997usize;
+
+    run_python(&format!(
+        r#"
+import h5py
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=({n},), maxshape=(None,), chunks=(1,),
+                         dtype="i4", fillvalue=-1)
+    for i in list(range(0, {n}, {step})) + list(range(0, 40)):
+        d[i] = i
+"#
+    ));
+
+    let file = File::open(&path).unwrap();
+    let values = file.dataset("x").unwrap().read_i32().unwrap();
+    assert_eq!(values.len(), n);
+    let wrong = values
+        .iter()
+        .enumerate()
+        .filter(|&(i, &v)| {
+            let expected = if i % step == 0 || i < 40 {
+                i as i32
+            } else {
+                -1
+            };
+            v != expected
+        })
+        .count();
+    assert_eq!(wrong, 0, "{wrong} of {n} elements read back wrong");
+}
+
+#[test]
+fn h5py_filtered_and_2d_extensible_array_clawhdf5_reads() {
+    // Filtered elements carry a size and filter mask beside the address, and
+    // a second (fixed) dimension changes how a linear index maps back to
+    // chunk offsets. Both run through the same traversal.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+
+    let gz = dir.path().join("ea_gzip.h5");
+    let gz_str = gz.display().to_string();
+    run_python(&format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{gz_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=(5000,), maxshape=(None,), chunks=(1,),
+                         dtype="i4", compression="gzip", compression_opts=4)
+    d[...] = np.arange(5000, dtype="i4")
+"#
+    ));
+    let values = File::open(&gz)
+        .unwrap()
+        .dataset("x")
+        .unwrap()
+        .read_i32()
+        .unwrap();
+    assert_eq!(values.len(), 5000);
+    assert_eq!(
+        values
+            .iter()
+            .enumerate()
+            .filter(|&(i, &v)| v != i as i32)
+            .count(),
+        0
+    );
+
+    let two_d = dir.path().join("ea_2d.h5");
+    let two_d_str = two_d.display().to_string();
+    run_python(&format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{two_d_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=(3000, 4), maxshape=(None, 4), chunks=(1, 4), dtype="i4")
+    d[...] = np.arange(12000, dtype="i4").reshape(3000, 4)
+"#
+    ));
+    let values = File::open(&two_d)
+        .unwrap()
+        .dataset("x")
+        .unwrap()
+        .read_i32()
+        .unwrap();
+    assert_eq!(values.len(), 12_000);
+    assert_eq!(
+        values
+            .iter()
+            .enumerate()
+            .filter(|&(i, &v)| v != i as i32)
+            .count(),
+        0
+    );
+}
+
+#[test]
+fn h5py_fixed_array_chunk_index_clawhdf5_reads() {
+    // Fixed dimensions plus libver='latest' give a Fixed Array chunk index.
+    // Its data blocks are paged above 2^page_bits elements (1024 by default),
+    // and unlike the Extensible Array it keeps the page-init bitmap in the
+    // data block itself — a difference worth pinning down, since assuming
+    // otherwise is exactly what made the Extensible Array reader wrong. The
+    // sparse case leaves whole pages uninitialised so the bitmap is actually
+    // consulted rather than being all ones.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+
+    for n in [100usize, 5_000, 200_000] {
+        let path = dir.path().join(format!("fa_{n}.h5"));
+        let path_str = path.display().to_string();
+        run_python(&format!(
+            r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=({n},), chunks=(1,), dtype="i4")
+    d[...] = np.arange({n}, dtype="i4")
+"#
+        ));
+        let bytes = std::fs::read(&path).unwrap();
+        assert!(
+            bytes.windows(4).any(|w| w == b"FAHD"),
+            "n={n}: fixture is not indexed by a Fixed Array"
+        );
+        let values = File::open(&path)
+            .unwrap()
+            .dataset("x")
+            .unwrap()
+            .read_i32()
+            .unwrap();
+        assert_eq!(values.len(), n, "n={n}");
+        let wrong = values
+            .iter()
+            .enumerate()
+            .filter(|&(i, &v)| v != i as i32)
+            .count();
+        assert_eq!(wrong, 0, "n={n}: {wrong} elements read back wrong");
+    }
+
+    let sparse = dir.path().join("fa_sparse.h5");
+    let sparse_str = sparse.display().to_string();
+    let (n, step) = (200_000usize, 997usize);
+    run_python(&format!(
+        r#"
+import h5py
+with h5py.File("{sparse_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=({n},), chunks=(1,), dtype="i4", fillvalue=-1)
+    for i in list(range(0, {n}, {step})) + list(range(0, 40)):
+        d[i] = i
+"#
+    ));
+    let values = File::open(&sparse)
+        .unwrap()
+        .dataset("x")
+        .unwrap()
+        .read_i32()
+        .unwrap();
+    assert_eq!(values.len(), n);
+    let wrong = values
+        .iter()
+        .enumerate()
+        .filter(|&(i, &v)| {
+            let expected = if i % step == 0 || i < 40 {
+                i as i32
+            } else {
+                -1
+            };
+            v != expected
+        })
+        .count();
+    assert_eq!(wrong, 0, "sparse: {wrong} of {n} elements read back wrong");
+}
+
+#[test]
+fn corrupting_a_chunk_index_is_an_error_not_wrong_data() {
+    // Every Fixed/Extensible Array structure carries a Jenkins checksum, and
+    // the reader now verifies it. The point is not the checksum itself but
+    // what it prevents: a damaged index otherwise yields addresses pointing
+    // at the wrong bytes, and the caller receives another chunk's data as if
+    // it were the one asked for.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+
+    for (name, maxshape) in [("fixed", "None"), ("extensible", "(None,)")] {
+        let path = dir.path().join(format!("{name}.h5"));
+        let path_str = path.display().to_string();
+        let shape_arg = if maxshape == "None" {
+            String::new()
+        } else {
+            format!(", maxshape={maxshape}")
+        };
+        run_python(&format!(
+            r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=(400,), chunks=(1,), dtype="i4"{shape_arg})
+    d[...] = np.arange(400, dtype="i4")
+"#
+        ));
+
+        let clean = std::fs::read(&path).unwrap();
+        assert_eq!(
+            File::open(&path)
+                .unwrap()
+                .dataset("x")
+                .unwrap()
+                .read_i32()
+                .unwrap()
+                .len(),
+            400,
+            "{name}: the intact file must read"
+        );
+
+        // Flip a low bit of a chunk address inside a data block. Structurally
+        // everything still parses — the index still has the right shape and
+        // the address still lands inside the file — so nothing but the
+        // checksum can notice. Without it the read succeeds and hands back
+        // whatever bytes now sit at that address.
+        let sig: &[u8] = if name == "fixed" { b"FADB" } else { b"EADB" };
+        let block = clean
+            .windows(4)
+            .position(|w| w == sig)
+            .unwrap_or_else(|| panic!("{name}: no data block in the fixture"));
+        // Past the prefix (signature, version, client id, header address, and
+        // for the Extensible Array a block offset), into the first address.
+        let at = block + 4 + 1 + 1 + 8 + if name == "fixed" { 0 } else { 4 } + 1;
+        let mut damaged = clean.clone();
+        damaged[at] ^= 0x10;
+        let damaged_path = dir.path().join(format!("{name}_damaged.h5"));
+        std::fs::write(&damaged_path, &damaged).unwrap();
+
+        let result = File::open(&damaged_path)
+            .unwrap()
+            .dataset("x")
+            .and_then(|d| d.read_i32());
+        assert!(
+            result.is_err(),
+            "{name}: corruption produced data instead of an error"
+        );
+    }
+}

crates/clawhdf5/tests/partial_read_equivalence.rs

@@ -0,0 +1,197 @@
+//! Selection reads must return exactly what a full read followed by element
+//! extraction returns — for every layout, rank and selection shape — while
+//! touching only what the selection needs.
+
+use clawhdf5::{File, FileBuilder};
+use clawhdf5_format::selection::Selection;
+
+struct Rng(u64);
+impl Rng {
+    fn next(&mut self) -> u64 {
+        self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
+        let mut z = self.0;
+        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+        z ^ (z >> 31)
+    }
+    fn below(&mut self, n: u64) -> u64 {
+        self.next() % n.max(1)
+    }
+}
+
+/// Row-major reference extraction from a full read.
+fn reference(full: &[i32], dims: &[u64], selection: &Selection) -> Vec<i32> {
+    let strides: Vec<u64> = (0..dims.len())
+        .map(|d| dims[d + 1..].iter().product())
+        .collect();
+    let at =
+        |coord: &[u64]| full[coord.iter().zip(&strides).map(|(c, s)| c * s).sum::<u64>() as usize];
+    match selection {
+        Selection::Points(points) => points.iter().map(|p| at(p)).collect(),
+        Selection::Hyperslab {
+            start,
+            stride,
+            count,
+            block,
+        } => {
+            // Selected indices per dimension, then their cartesian product.
+            let per_dim: Vec<Vec<u64>> = (0..dims.len())
+                .map(|d| {
+                    (0..count[d])
+                        .flat_map(|c| (0..block[d]).map(move |b| (c, b)))
+                        .map(|(c, b)| start[d] + c * stride[d] + b)
+                        .collect()
+                })
+                .collect();
+            let mut out = Vec::new();
+            let mut idx = vec![0usize; dims.len()];
+            loop {
+                let coord: Vec<u64> = idx
+                    .iter()
+                    .enumerate()
+                    .map(|(d, &i)| per_dim[d][i])
+                    .collect();
+                out.push(at(&coord));
+                let mut d = dims.len();
+                loop {
+                    if d == 0 {
+                        return out;
+                    }
+                    d -= 1;
+                    idx[d] += 1;
+                    if idx[d] < per_dim[d].len() {
+                        break;
+                    }
+                    idx[d] = 0;
+                }
+            }
+        }
+        _ => unreachable!(),
+    }
+}
+
+fn random_hyperslab(rng: &mut Rng, dims: &[u64]) -> Selection {
+    let mut start = Vec::new();
+    let mut stride = Vec::new();
+    let mut count = Vec::new();
+    let mut block = Vec::new();
+    for &dim in dims {
+        let b = 1 + rng.below(3);
+        let st = b + rng.below(4); // stride >= block: no overlap
+        let s = rng.below(dim - b + 1);
+        let max_count = (dim - s - b) / st + 1;
+        let c = 1 + rng.below(max_count.min(6));
+        start.push(s);
+        stride.push(st);
+        count.push(c);
+        block.push(b);
+    }
+    Selection::Hyperslab {
+        start,
+        stride,
+        count,
+        block,
+    }
+}
+
+#[test]
+fn selection_reads_match_full_reads_for_every_layout() {
+    let dir = tempfile::tempdir().unwrap();
+    let mut rng = Rng(7);
+    // (dims, chunk dims)
+    let shapes: [(&[u64], &[u64]); 3] = [
+        (&[97], &[10]),
+        (&[41, 53], &[8, 9]),
+        (&[11, 13, 17], &[4, 5, 6]),
+    ];
+    for (dims, chunks) in shapes {
+        let n: u64 = dims.iter().product();
+        let data: Vec<i32> = (0..n as i32).map(|v| v * 3 - 7).collect();
+
+        let path = dir.path().join(format!("r{}.h5", dims.len()));
+        let mut builder = FileBuilder::new();
+        builder
+            .create_dataset("contiguous")
+            .with_i32_data(&data)
+            .with_shape(dims);
+        builder
+            .create_dataset("chunked")
+            .with_i32_data(&data)
+            .with_shape(dims)
+            .with_chunks(chunks);
+        builder
+            .create_dataset("deflated")
+            .with_i32_data(&data)
+            .with_shape(dims)
+            .with_chunks(chunks)
+            .with_deflate(3);
+        builder.write(&path).unwrap();
+
+        let file = File::open(&path).unwrap();
+        for name in ["contiguous", "chunked", "deflated"] {
+            let ds = file.dataset(name).unwrap();
+            let full = ds.read_i32().unwrap();
+            assert_eq!(full, data, "{name} full read");
+
+            for case in 0..60 {
+                let selection = if case % 5 == 4 {
+                    let points = (0..1 + rng.below(12))
+                        .map(|_| dims.iter().map(|&d| rng.below(d)).collect())
+                        .collect();
+                    Selection::Points(points)
+                } else {
+                    random_hyperslab(&mut rng, dims)
+                };
+                assert_eq!(
+                    ds.read_i32_selection(&selection).unwrap(),
+                    reference(&full, dims, &selection),
+                    "{name} rank {} case {case}: {selection:?}",
+                    dims.len()
+                );
+            }
+        }
+    }
+}
+
+#[test]
+fn out_of_bounds_selections_are_errors() {
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("oob.h5");
+    let mut builder = FileBuilder::new();
+    builder
+        .create_dataset("d")
+        .with_i32_data(&(0..100).collect::<Vec<i32>>())
+        .with_shape(&[10, 10])
+        .with_chunks(&[4, 4]);
+    builder.write(&path).unwrap();
+    let file = File::open(&path).unwrap();
+    let ds = file.dataset("d").unwrap();
+    let beyond = Selection::Hyperslab {
+        start: vec![8, 8],
+        stride: vec![1, 1],
+        count: vec![5, 5],
+        block: vec![1, 1],
+    };
+    use clawhdf5::Error;
+    use clawhdf5_format::error::FormatError;
+    let is_oob = |s: &Selection| {
+        matches!(
+            ds.read_i32_selection(s),
+            Err(Error::Format(FormatError::SelectionOutOfBounds(_)))
+        )
+    };
+    // Used to come back padded with zeros.
+    assert!(is_oob(&beyond));
+    // Row out of range.
+    assert!(is_oob(&Selection::Points(vec![vec![10, 0]])));
+    // Column out of range: used to wrap into the next row and return its value.
+    assert!(is_oob(&Selection::Points(vec![vec![0, 12]])));
+    // Wrong rank.
+    assert!(is_oob(&Selection::Points(vec![vec![3]])));
+    // In range is fine.
+    assert_eq!(
+        ds.read_i32_selection(&Selection::Points(vec![vec![9, 9]]))
+            .unwrap(),
+        [99]
+    );
+}

docs/QUICKSTART.md

@@ -364,6 +364,11 @@ cargo install --path crates/clawhdf5-cli
 clawhdf5 --path agent.h5 create --agent-id my-agent --dim 384 --wal
 ```
 
+Add `--quantized-index` to store the vector index's copy of the embeddings as
+int8. That roughly halves a loaded store's memory at about 13% fewer queries
+per second, with recall unchanged — the query path re-scores candidates
+against the exact embeddings. The setting is recorded in the file.
+
 Output:
 ```json
 {

docs/known-issues.md

@@ -62,13 +62,21 @@ only files using the native type through the C API / h5py low-level API hit this
 
 ## Revised reference datatype (class 7, version 4) is not parsed
 
-**Status:** open, unconfirmed against a real file.
+**Status:** fixed 2026-09-19 for object references; region and attribute
+references are recognised but not decoded.
 
-**Summary:** HDF5 1.12+ `H5T_STD_REF` references use datatype version 4 with
-reference types 2–4 (object2 / region2 / attribute), which `Datatype::parse`
-rejects with `InvalidReferenceType`. h5py still writes the legacy v1
-object/region references, which read correctly, so no reproducing file has been
-generated yet; one written with the C API (`H5T_STD_REF`) is needed.
+**Summary:** HDF5 1.12+ `H5T_STD_REF` references use datatype message version 4
+with reference types 2-4 (object / region / attribute), which `Datatype::parse`
+rejected with `InvalidReferenceType`. h5py still writes the legacy references,
+so no file had been available to test against.
+
+**Fix:** a real file was produced by driving the libhdf5 bundled in the h5py
+wheel through ctypes (`tests/fixtures/gen_std_ref.py` ->
+`std_ref_hdf5_2_0.h5`). The three new types parse as
+`ReferenceType::{Object2, DatasetRegion2, Attribute}`, and
+`read_object_references` decodes `Object2` elements (type, flags, token size,
+token = target object header address). External references (flag bit 0) and
+the region/attribute payloads are errors rather than misreads.
 
 ## `clawhdf5-gpu` `gpu_tests` can hang under the default parallel test runner
 
@@ -116,16 +124,17 @@ not reported).
 
 ## B-tree v2 chunk index (layout v4, index type 5) is not supported
 
-**Status:** open.
+**Status:** fixed 2026-09-19.
 
 **Summary:** a chunked dataset with **two or more unlimited dimensions** written
-with `libver='latest'` indexes its chunks with a version-2 B-tree. Reading it
-fails with `ChunkedReadError("unsupported chunked layout version=4,
-index_type=Some(5)")`. Single-chunk, implicit, fixed-array and
-extensible-array indexes (and the v3 B-tree v1) are supported.
+with `libver='latest'` indexes its chunks with a version-2 B-tree, and reading it
+failed with `unsupported chunked layout version=4, index_type=Some(5)`.
 
-**Repro:** `f.create_dataset("d", shape=(5, 7), chunks=(2, 3), maxshape=(None, None))`
-with `h5py.File(..., libver='latest')`.
+**Fix:** record types 10 (unfiltered) and 11 (filtered) are decoded — address,
+stored size, filter mask, scaled offsets — through the shared chunk-listing
+function, so full reads, cached reads, partial reads and fill-value handling
+all work. Covered by an h5py interop test (plain, gzip+shuffle, a 2500-chunk
+tree with internal nodes, a sparse dataset with a fill value, a hyperslab).
 
 ## External links and external raw data are not followed
 
@@ -137,3 +146,84 @@ created with `external=[...]` storage returns
 `FormatError::ExternalDataFilesUnsupported`. Neither is resolved. If support is
 added, file names must be confined to the opened file's directory, as the
 virtual-dataset resolver now does.
+
+---
+
+## Python interop suites skip silently when no interpreter has h5py
+
+**Status:** fixed on `main` in `a29c1b2` (2026-09-19).
+
+On a system where `python3` is a PEP 668 "externally managed" interpreter,
+h5py cannot be installed into it at all, and every interop suite — the h5py
+writer round-trips, the facade suite, netCDF4, and the reference files —
+returned `false` from its availability probe and skipped without failing. CI
+reported `SKIP` and a green run. This is the same class of gap that let the
+compound-datatype v5 bug above reach a release.
+
+The probes now read `CLAWHDF5_PYTHON`, and `scripts/ci-test.sh` picks up
+`.venv/bin/python` automatically. To restore the coverage on a fresh checkout:
+
+```bash
+python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4
+```
+
+Set `CLAWHDF5_REQUIRE_INTEROP=1` in any automated runner so a missing
+interpreter is a failure rather than a skip.
+
+
+---
+
+## Crafted B-tree v2 structures crash or exhaust the reader
+
+**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
+and including v2.6.0 is affected.**
+
+B-tree v2 traversal (`clawhdf5-format`, `btree_v2::collect_btree_v2_records`)
+recursed one frame per level with the depth taken from the file, and followed
+child addresses without checking whether they were shared. Two consequences
+for anyone reading untrusted files:
+
+- A node that is its own child, under a header claiming 65 535 levels, overflows
+  the stack and aborts the process. The file is under 100 bytes.
+- Levels whose children all point at one node below make the traversal visit it
+  fan-out^depth times: ~30 million records from ~5 KB, and memory exhaustion one
+  level deeper.
+
+B-tree v2 backs dense attribute storage, v2 groups, shared object header
+messages and chunk indexes, so opening an object that uses any of them is
+enough. Both are now errors: depth is capped at 64, and traversal stops once it
+has produced more records than the file could physically hold.
+
+---
+
+## Extensible Array chunk indexes read back wrong data past the inline elements
+
+**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
+and including v2.6.0 is affected.**
+
+A dataset created with exactly one unlimited dimension (`maxshape=(None, ...)`,
+the usual append-only/resizable case) is indexed by an Extensible Array. Its
+index block holds the first `idx_blk_elmts` chunk entries inline — 4 by
+default — and everything after that lives in data blocks and super blocks whose
+layout `clawhdf5-format` computed incorrectly.
+
+Consequences, by dataset size (1 chunk per element):
+
+| chunks | result before the fix |
+|---|---|
+| <= 36 | correct (inline, plus two data blocks that happened to line up) |
+| 37 | 1 element wrong |
+| 400 | 364 elements wrong |
+| >= ~1000 | `invalid Extensible Array data block signature` |
+
+The dangerous case is the middle one: values were returned from the wrong
+chunks rather than an error being raised. Any reader that accepted the data at
+face value saw plausible but incorrect numbers.
+
+The root causes were the super block sizing formulas (`ndblks` and
+`dblk_nelmts` each double every *other* level, a half-step apart), a missing
+block-offset field in the super block, and a page-init bitmap read from the
+wrong structure. All four are fixed and covered by interop tests against
+HDF5 2.0 at sizes that cross each boundary, including paged data blocks.
+
+Files written by this crate are unaffected — this was purely a read-path bug.

packages/clawhdf5-node/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@redclaw/clawhdf5",
-  "version": "2.4.0",
+  "version": "2.7.0",
   "description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
   "main": "index.js",
   "types": "index.d.ts",

scripts/ci-test.sh

@@ -20,6 +20,13 @@
 set -uo pipefail
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+# Interop suites drive a Python interpreter. On a PEP 668 "externally managed"
+# system h5py can only live in a virtualenv, so pick one up here — before any
+# test step, since the non-ignored interop suites read the same variable.
+if [ -z "${CLAWHDF5_PYTHON:-}" ] && [ -x "$SCRIPT_DIR/../.venv/bin/python" ]; then
+    export CLAWHDF5_PYTHON="$SCRIPT_DIR/../.venv/bin/python"
+fi
 PASS=0
 FAIL=0
 STEPS=()
@@ -63,6 +70,13 @@ run_step "cargo clippy (format feature matrix)" cargo clippy \
     --features parallel,lz4,zstd,pcodec,fast-checksum \
     -- -D warnings
 
+# The HNSW index's parallel bulk build is feature-gated too.
+run_step "cargo clippy (ann parallel)" cargo clippy \
+    -p clawhdf5-ann \
+    --all-targets \
+    --features parallel \
+    -- -D warnings
+
 # 4. Tests (exclude clawhdf5-py)
 run_step "cargo test" cargo test \
     --workspace \
@@ -72,14 +86,24 @@ run_step "cargo test (format feature matrix)" cargo test \
     -p clawhdf5-format \
     --features parallel,lz4,zstd,pcodec,fast-checksum
 
+run_step "cargo test (ann parallel)" cargo test \
+    -p clawhdf5-ann \
+    --features parallel
+
 # 5. Python interop suites. The h5py writer tests are #[ignore]d so a plain
 #    `cargo test` stays hermetic; run them explicitly here.
-if python3 -c "import h5py" >/dev/null 2>&1 || [ "${CLAWHDF5_REQUIRE_INTEROP:-0}" = "1" ]; then
+#    On a PEP 668 "externally managed" system h5py can only live in a
+#    virtualenv, so honour CLAWHDF5_PYTHON (and a local .venv) rather than
+#    skipping — the tests read the same variable.
+PYTHON="${CLAWHDF5_PYTHON:-python3}"
+if "$PYTHON" -c "import h5py" >/dev/null 2>&1 || [ "${CLAWHDF5_REQUIRE_INTEROP:-0}" = "1" ]; then
     run_step "h5py interop (format, ignored tests)" cargo test \
         -p clawhdf5-format --test writer_h5py_tests -- --include-ignored
 else
     echo ""
-    echo "==> [h5py interop] SKIPPED: python3 with h5py not available"
+    echo "==> [h5py interop] SKIPPED: no h5py in $PYTHON"
+    echo "    (set CLAWHDF5_PYTHON=/path/to/venv/bin/python, or create .venv;"
+    echo "     CLAWHDF5_REQUIRE_INTEROP=1 makes this a failure instead)"
     STEPS+=("SKIP: h5py interop (format, ignored tests)")
 fi