Merge release/v2.3.0

Co-Authored-By: Claude Fable 5.1 <[email protected]>

.gitea/workflows/ci.yml

@@ -22,5 +22,19 @@ jobs:
         run: rustup component add rustfmt clippy
       - name: Install thumbv7em-none-eabihf target
         run: rustup target add thumbv7em-none-eabihf
+      - name: Install Python interop dependencies
+        # The interop suites used to skip silently when python3/h5py were
+        # missing, so they never ran in CI. Install them and make a missing
+        # dependency a failure (CLAWHDF5_REQUIRE_INTEROP below).
+        run: |
+          apt-get update
+          apt-get install -y --no-install-recommends python3 python3-venv
+          python3 -m venv /opt/interop
+          /opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray
+          echo "/opt/interop/bin" >> "$GITHUB_PATH"
+      - name: Show interop library versions
+        run: python3 -c "import h5py, netCDF4; print('h5py', h5py.__version__, 'HDF5', h5py.version.hdf5_version, 'netCDF4', netCDF4.__version__)"
       - name: Run CI script
+        env:
+          CLAWHDF5_REQUIRE_INTEROP: "1"
         run: bash scripts/ci-test.sh

CHANGELOG.md

@@ -1,5 +1,155 @@
 # Changelog
 
+## v2.3.0 (2026-09-19)
+
+### Upgrade Notes
+- **A memory store now has a single writer.** `HDF5Memory::create`/`open` take
+  an exclusive lock (`<store>.h5.lock`); a second open of the same store — in
+  the same or another process — returns `MemoryError::Locked`. Code that opened
+  a second handle just to read should use `HDF5Memory::open_read_only`.
+- **Unsigned array attributes arrive as `AttrValue::U64Array`**, not
+  `I64Array`, and `attrs()` may now return `AttrValue::Raw`. Exhaustive matches
+  on `AttrValue` need the two new arms.
+- **WAL header version 3 → 4.** v3 files are read and upgraded in place, but a
+  store written by 2.3.0 with a pending WAL cannot be opened by 2.2.0 or
+  earlier (it is refused, not corrupted). Checkpoint first
+  (`flush_wal`) if you need to downgrade.
+- `MemoryConfig::compression` now uses deflate unless the agent's new `zstd`
+  feature is enabled; it previously failed outright in a default build.
+- `MemoryError` gained `Locked`; `FormatError` gained `UnresolvedSharedMessage`,
+  `ExternalDataFilesUnsupported` and `ExternalLinkUnsupported`; `MessageType`
+  gained `ExternalDataFiles`.
+
+### Bug Fixes
+- `clawhdf5-format`: compound datatypes written with **default libver bounds**
+  (datatype message version 1 — what plain `h5py.File(path, 'w')` produces)
+  were mis-parsed. The v1 member layout carries 28 bytes of legacy array
+  fields after the byte offset (the parser skipped 24), and v2 pads member
+  names to 8 bytes and has no array fields at all (the parser did neither), so
+  every member after the first byte offset was read from the wrong position —
+  typically surfacing as `Overflow("compound member ...")` on read. Found by
+  adding a default-libver axis to the h5py interop tests; byte-level regression
+  tests for v1 and v2 added.
+- `clawhdf5-gpu`: `gpu_tests` could hang forever under the default parallel
+  test runner — every test created its own wgpu instance and device at once.
+  Tests now serialise GPU access, and GPU→CPU readback waits are bounded
+  (30 s) so a wedged driver returns `GpuError::BufferMap` instead of blocking.
+- `clawhdf5-agent`: `benches/bench.rs` and `benches/memory_bench.rs` no longer
+  compiled against the current `strategy`/`consolidation` APIs.
+
+### HDF5 Compatibility
+- `clawhdf5-format`/`clawhdf5`: datasets and attributes that use a **committed
+  (named) datatype** now read correctly. They store a shared-message reference;
+  the facade parsed the reference bytes as the datatype (`Time { size: 0 }`,
+  unreadable data) and silently dropped such attributes. The shared-reference
+  parser itself was wrong for real files: version 2 has no reserved bytes, and
+  the version 3 types were inverted (1 = SOHM heap, 2 = committed).
+- **Fill values are applied on read.** There was no Fill Value message parser:
+  the holes of a sparse chunked dataset read as zeros even when the fill value
+  was not zero (silently wrong data), and a dataset that was created but never
+  written failed with `NoDataAllocated` where h5py returns a filled array.
+  Messages v1–v3 and the old 0x0004 form are parsed; the fill value is written
+  into exactly the chunk-grid cells missing from the chunk index.
+- **Soft links are followed** during path resolution, in old- and new-style
+  groups (absolute/relative targets, links to groups, links through links),
+  with a depth limit so a link cycle is an error rather than a hang. A dangling
+  link reports the target it could not find.
+- Things the reader does not follow are now explicit errors instead of wrong
+  answers: an external link is `ExternalLinkUnsupported { filename,
+  object_path }` (was `PathNotFound`), and a dataset whose raw data lives in
+  external files (message 0x0007, now a known `MessageType`) is
+  `ExternalDataFilesUnsupported` (it would otherwise read as fill values).
+- **`attrs()` no longer drops attributes.** Any attribute whose datatype had
+  no `AttrValue` variant was omitted with no error — including every Python
+  `bool` (h5py stores `attrs["flag"] = True` as an enum), complex numbers,
+  compound values and object references. Now:
+  - numpy/h5py-style booleans (an enum of exactly `FALSE`=0 / `TRUE`=1) decode
+    as `I64` / `I64Array` of 0/1;
+  - new `AttrValue::U64Array` keeps unsigned arrays unsigned (they were cast to
+    `I64Array`, so values above `i64::MAX` came back negative). **Behaviour
+    change:** code matching `I64Array` for an unsigned attribute must also
+    match `U64Array` (the netCDF-4 CF helpers and Python bindings do);
+  - new `AttrValue::Raw { datatype, shape, data }` carries everything else
+    verbatim, decodable with `clawhdf5_format::data_read` against `datatype`.
+  Both new variants are writable, so an attribute can be copied between files
+  unchanged. Python receives `Raw` as `{"dtype", "shape", "data"}`.
+- All of the above are covered by h5py interop tests under both default and
+  `libver='latest'` bounds, compared against h5py's own readback.
+
+### Security
+- `clawhdf5`: virtual-dataset source file names are untrusted input but were
+  joined straight onto the opened file's directory, so a crafted file could
+  make the reader open any path the process can reach (absolute path, or `..`
+  components). Only plain relative paths inside that directory are accepted.
+
+### Durability & Integrity
+- `clawhdf5-agent`: a crash between writing a checkpoint and truncating the WAL
+  no longer **duplicates every pending entry** on the next open. Each
+  checkpoint records a `WalMark` (byte length + chained CRC of the WAL prefix it
+  folded in) in `/meta`; `open()` skips exactly that prefix when it is still
+  present. No WAL format change for this; older files behave as before.
+- `clawhdf5-agent`: checkpoints and snapshots are durable as a unit — the temp
+  file is synced before the rename and the directory after it. Individual WAL
+  appends remain unsynced by design (documented in `CLAUDE.md`).
+- `clawhdf5-agent`: `save_or_update` hits are logged as a new `Update` WAL
+  record, so replay updates in place instead of appending a duplicate. WAL
+  header version 3 → 4 (so older builds refuse the file rather than truncating
+  a record they can't parse); v3 files are read and upgraded in place.
+- `clawhdf5-agent`: loading validates every per-record dataset length (a
+  truncated store is now `MemoryError::Schema`, not a later panic), fixes the
+  `n.len() == n.len()` tautology that trusted a norms dataset of any length,
+  and rejects `embedding_dim == 0` with records present.
+- `clawhdf5-agent`: eight behavioural `MemoryConfig` fields are now persisted in
+  `/meta`. Previously they reset to defaults on every open — a compressed store
+  was rewritten uncompressed, `wal_enabled = false` flipped back to `true`.
+- `clawhdf5-agent`: `compression = true` never worked in a default build (it
+  requested Zstd without enabling the feature, so every checkpoint failed with
+  `unsupported filter: 32015`). Default builds now use deflate; Zstd is the new
+  opt-in `zstd` feature.
+- `clawhdf5-agent`: **single-writer lock** (`<store>.h5.lock`,
+  `MemoryError::Locked`) — two handles on one store used to silently destroy
+  each other's data. New `HDF5Memory::open_read_only` gives a lock-free,
+  never-writing view; the CLI's read-only subcommands use it.
+- `clawhdf5-agent`: an unreadable WAL (torn header / bad magic) is quarantined
+  (`HDF5Memory::quarantined_wal()`) instead of blocking `open()` of a healthy
+  store. A WAL from an unknown newer version still fails and is left intact.
+- `clawhdf5-agent`: provenance records are renumbered on compaction (they
+  weren't, so every later `save_or_update` raised a false High integrity
+  alert); pending anomaly alerts and tracked sessions are bounded;
+  `snapshot()` includes entries still in the WAL.
+- `clawhdf5-agent`: hybrid ranking is deterministic (index tie-breaks instead
+  of `HashMap` order); a set of identical positive scores — including a single
+  candidate — normalises to 1.0 rather than 0.0; the Hebbian boost no longer
+  reinforces zero-score filler results.
+- `clawhdf5-format`: chunked/VDS/hyperslab reads size their buffers with
+  overflow-checked arithmetic and fallible allocation, so crafted dimensions
+  are `FormatError::Overflow` instead of a wrapped size or a process abort;
+  `parallel_read` bounds checks use `checked_add`.
+- `clawhdf5`: a malformed filter-pipeline message is an error instead of being
+  treated as "no filters" (which returned compressed bytes as data);
+  `FileBuilder::write` is atomic and synced instead of truncating the
+  destination first.
+
+### CI / Testing
+- CI now lints every target (`cargo clippy --all-targets`) plus
+  `clawhdf5-format`'s optional features, compiles all benches, and tests the
+  format feature matrix. Previously test/bench code and feature-gated modules
+  were never linted; the accumulated clippy backlog is fixed.
+- CI installs python3 + h5py/numpy/netCDF4/xarray and sets
+  `CLAWHDF5_REQUIRE_INTEROP=1`, which turns a missing interop dependency into a
+  test **failure**. Until now every h5py/netCDF4 interop test silently skipped
+  in CI, which is how the HDF5 2.0 compound bug fixed in v2.2.0 reached a user.
+  The `#[ignore]`d `writer_h5py_tests` suite is run explicitly.
+- h5py-generated-file tests now cover default libver bounds as well as
+  `libver='latest'` (HDF5 2.0 raised the default low bound to 1.8).
+- `clawhdf5-agent`: WAL property tests (round trip; after any corruption the
+  entries read back are an exact prefix of what was written — 1500 seeded
+  cases), a crash-recovery matrix (an on-disk image after every operation, the
+  checkpoint window, and the WAL torn at every byte length, each reopened and
+  checked against a model), and a WAL fuzz target.
+- Optional fuzz smoke run (`CLAWHDF5_FUZZ_SECONDS=N scripts/ci-test.sh`); new
+  datatype corpus seeds for v1 compound and native complex messages.
+
 ## v2.2.0 (2026-09-18)
 
 ### Security

CLAUDE.md

@@ -40,6 +40,25 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
   format (v2) is still fully readable; the oldest no-CRC format (v1) is only
   reachable through the one-time migration path in `HDF5Memory::open`, not
   through the public `WalFile::read_entries`.
+  **What the WAL guarantees:** integrity, ordering, and recovery from a
+  *process* crash at any point — including between a checkpoint and the WAL
+  truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips
+  the WAL prefix the `.h5` already contains, so entries are never applied
+  twice). Checkpoints and snapshots are made durable as a unit (temp file
+  synced, renamed, directory synced). **What it does not guarantee:**
+  individual WAL appends are *not* fsynced (a deliberate latency trade-off), so
+  saves made since the last checkpoint can be lost on power failure or kernel
+  panic. Current header version is 4 (adds the `Update` record used by
+  `save_or_update`); v3 files are read and upgraded in place.
+- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive
+  advisory lock on `<store>.h5.lock` and a second opener gets
+  `MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free,
+  never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/
+  `export` do). An unreadable WAL (torn header, bad magic) is quarantined to
+  `<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
+  unknown *newer* version still fails and is left untouched.
+- `MemoryConfig::compression` uses deflate by default; enable the agent's
+  `zstd` feature to compress embeddings with Zstd instead (links libzstd).
 - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
   default) recomputes a dataset's SHA-256 and compares it against the
   `_provenance_sha256` attribute written automatically on save when

Cargo.toml

@@ -21,7 +21,7 @@ members = [
 resolver = "2"
 
 [workspace.package]
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"

crates/clawhdf5-accel/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-accel"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "SIMD-accelerated operations for rustyhdf5"
 license = "MIT"

crates/clawhdf5-accel/src/avx2.rs

@@ -111,7 +111,11 @@ pub unsafe fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
         }
 
         let denom = (norm_a * norm_b).sqrt();
-        if denom < f32::EPSILON { 0.0 } else { dot / denom }
+        if denom < f32::EPSILON {
+            0.0
+        } else {
+            dot / denom
+        }
     }
 }
 

crates/clawhdf5-accel/src/avx512.rs

@@ -89,7 +89,11 @@ pub unsafe fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
         }
 
         let denom = (norm_a * norm_b).sqrt();
-        if denom < f32::EPSILON { 0.0 } else { dot / denom }
+        if denom < f32::EPSILON {
+            0.0
+        } else {
+            dot / denom
+        }
     }
 }
 

crates/clawhdf5-accel/src/neon.rs

@@ -94,7 +94,11 @@ pub unsafe fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
     }
 
     let denom = (norm_a * norm_b).sqrt();
-    if denom < f32::EPSILON { 0.0 } else { dot / denom }
+    if denom < f32::EPSILON {
+        0.0
+    } else {
+        dot / denom
+    }
 }
 
 /// NEON L2 distance.

crates/clawhdf5-accel/src/scalar.rs

@@ -21,7 +21,11 @@ pub fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
         norm_b += y * y;
     }
     let denom = (norm_a * norm_b).sqrt();
-    if denom < f32::EPSILON { 0.0 } else { dot / denom }
+    if denom < f32::EPSILON {
+        0.0
+    } else {
+        dot / denom
+    }
 }
 
 pub fn batch_cosine(query: &[f32], vectors: &[&[f32]], results: &mut [(usize, f32)]) {

crates/clawhdf5-agent/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-agent"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "HDF5-backed persistent memory store for on-device AI agents"
 license = "MIT"
@@ -10,12 +10,12 @@ keywords = ["agent", "memory", "hdf5", "vector-search", "embedding"]
 categories = ["database", "science", "algorithms"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.2.0", features = ["parallel", "fast-checksum"] }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.3.0", features = ["parallel", "fast-checksum"] }
-clawhdf5 = { path = "../clawhdf5", version = "2.2.0" }
+clawhdf5 = { path = "../clawhdf5", version = "2.3.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.2.0", features = ["mmap"] }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.3.0", features = ["mmap"] }
-clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.2.0" }
+clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.3.0" }
-clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.2.0", optional = true }
+clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.3.0", optional = true }
-clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.2.0", optional = true, default-features = false }
+clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.3.0", optional = true, default-features = false }
 serde = { workspace = true }
 byteorder = "1"
 half = { workspace = true, optional = true }
@@ -48,6 +48,9 @@ harness = false
 default = ["float16", "hnsw"]
 float16 = ["half"]
 parallel = ["rayon"]
+# Compress embeddings with Zstd instead of deflate when
+# `MemoryConfig::compression` is on. Off by default: it links libzstd (C).
+zstd = ["clawhdf5/zstd"]
 # HNSW approximate-nearest-neighbour acceleration for the vector stage of
 # hybrid_search. On by default; the index is rebuilt from the cache on demand
 # and stays self-consistent with the persisted memory store. Disable with

crates/clawhdf5-agent/benches/bench.rs

@@ -483,7 +483,7 @@ fn rayon_benches(c: &mut Criterion) {
                 use rayon::prelude::*;
                 let query_norm = vector_search::compute_norm(&query);
                 let num_cores = rayon::current_num_threads().max(1);
-                let chunk_size = (n + num_cores - 1) / num_cores;
+                let chunk_size = n.div_ceil(num_cores);
                 let mut results: Vec<(usize, f32)> = vectors
                     .par_chunks(chunk_size)
                     .enumerate()
@@ -537,7 +537,7 @@ fn rayon_benches(c: &mut Criterion) {
                 use rayon::prelude::*;
                 let query_norm = vector_search::compute_norm(&query);
                 let num_cores = rayon::current_num_threads().max(1);
-                let chunk_size = (n + num_cores - 1) / num_cores;
+                let chunk_size = n.div_ceil(num_cores);
                 let mut results: Vec<(usize, f32)> = vectors
                     .par_chunks(chunk_size)
                     .enumerate()
@@ -766,12 +766,22 @@ fn adaptive_benches(c: &mut Criterion) {
         .map(|v| vector_search::compute_norm(v))
         .collect();
     let tombstones = vec![0u8; n];
+    let flat: Vec<f32> = vectors.iter().flatten().copied().collect();
 
     c.bench_function("adaptive_search_10k", |b| {
         let hw = HardwareCapabilities::detect();
         let strat = strategy::auto_select_strategy(n, &hw);
         b.iter(|| {
-            strategy::search_with_metrics(&query, &vectors, &norms, &tombstones, 10, strat, None)
+            strategy::search_with_metrics(
+                &query,
+                &vectors,
+                &flat,
+                &norms,
+                &tombstones,
+                10,
+                strat,
+                None,
+            )
         });
     });
 
@@ -781,6 +791,7 @@ fn adaptive_benches(c: &mut Criterion) {
             strategy::search_with_metrics(
                 &query,
                 &vectors,
+                &flat,
                 &norms,
                 &tombstones,
                 10,
@@ -795,6 +806,7 @@ fn adaptive_benches(c: &mut Criterion) {
             strategy::search_with_metrics(
                 &query,
                 &vectors,
+                &flat,
                 &norms,
                 &tombstones,
                 10,
@@ -809,6 +821,7 @@ fn adaptive_benches(c: &mut Criterion) {
             strategy::search_with_metrics(
                 &query,
                 &vectors,
+                &flat,
                 &norms,
                 &tombstones,
                 10,

crates/clawhdf5-agent/benches/memory_bench.rs

@@ -1,6 +1,7 @@
 use clawhdf5_agent::bm25::BM25Index;
 use clawhdf5_agent::consolidation::{
     ConsolidationConfig, ConsolidationEngine, ImportanceScorer, ImportanceWeights, MemorySource,
+    UntrustedSource,
 };
 use clawhdf5_agent::hybrid::{hybrid_search, rrf_hybrid_search};
 use clawhdf5_agent::knowledge::KnowledgeCache;
@@ -285,7 +286,12 @@ fn consolidation_benches(c: &mut Criterion) {
                         for i in 0..n {
                             let embedding = make_vec(&mut rng, DIM);
                             let chunk = format!("memory record {i} with some content");
-                            engine.add_memory(chunk, embedding, MemorySource::User, now + i as f64);
+                            engine.add_memory(
+                                chunk,
+                                embedding,
+                                UntrustedSource::User,
+                                now + i as f64,
+                            );
                         }
                         engine
                     },
@@ -307,9 +313,10 @@ fn consolidation_benches(c: &mut Criterion) {
         for i in 0..50usize {
             let embedding = make_vec(&mut rng, DIM);
             let chunk = format!("existing record {i}");
-            engine.add_memory(chunk, embedding, MemorySource::User, now + i as f64);
+            engine.add_memory(chunk, embedding, UntrustedSource::User, now + i as f64);
         }
         let records = engine.records().to_vec();
+        let record_refs: Vec<&_> = records.iter().collect();
         let weights = ImportanceWeights::default();
         let query_embedding = make_vec(&mut rng, DIM);
         let sample_text =
@@ -317,7 +324,7 @@ fn consolidation_benches(c: &mut Criterion) {
 
         group.bench_function("bench_importance_scoring", |b| {
             b.iter(|| {
-                let surprise = ImportanceScorer::score_surprise(&query_embedding, &records);
+                let surprise = ImportanceScorer::score_surprise(&query_embedding, &record_refs);
                 let correction = ImportanceScorer::score_correction(&MemorySource::Correction);
                 let length = ImportanceScorer::score_length(sample_text);
                 ImportanceScorer::score_combined(surprise, correction, length, &weights)
@@ -354,7 +361,7 @@ fn temporal_benches(c: &mut Criterion) {
     // Insert benchmark: measure time to insert 10k timestamps one by one
     group.bench_function("bench_temporal_insert_10k", |b| {
         b.iter_batched(
-            || TemporalIndex::new(),
+            TemporalIndex::new,
             |mut idx| {
                 for i in 0..N {
                     // Shuffle insertion order slightly using a simple offset pattern
@@ -442,7 +449,8 @@ fn large_consolidation_benches(c: &mut Criterion) {
     let mut group = c.benchmark_group("consolidation_large");
     group.sample_size(10);
 
-    for (label, n) in [("10k", 10_000usize)] {
+    {
+        let (label, n) = ("10k", 10_000usize);
         group.bench_with_input(
             BenchmarkId::new("bench_consolidation_cycle", label),
             &n,
@@ -459,7 +467,12 @@ fn large_consolidation_benches(c: &mut Criterion) {
                         for i in 0..n {
                             let embedding = make_vec(&mut rng, DIM);
                             let chunk = format!("memory record {i} with content");
-                            engine.add_memory(chunk, embedding, MemorySource::User, now + i as f64);
+                            engine.add_memory(
+                                chunk,
+                                embedding,
+                                UntrustedSource::User,
+                                now + i as f64,
+                            );
                         }
                         engine
                     },

crates/clawhdf5-agent/fuzz/.gitignore

@@ -0,0 +1,3 @@
+target/
+artifacts/
+coverage/

crates/clawhdf5-agent/fuzz/Cargo.toml

@@ -0,0 +1,23 @@
+[package]
+name = "clawhdf5-agent-fuzz"
+version = "0.0.0"
+publish = false
+edition = "2024"
+
+[package.metadata]
+cargo-fuzz = true
+
+[dependencies]
+libfuzzer-sys = "0.4"
+tempfile = "3"
+
+[dependencies.clawhdf5-agent]
+path = ".."
+
+[workspace]
+members = ["."]
+
+[[bin]]
+name = "fuzz_wal_replay"
+path = "fuzz_targets/fuzz_wal_replay.rs"
+doc = false

crates/clawhdf5-agent/fuzz/fuzz_targets/fuzz_wal_replay.rs

@@ -0,0 +1,36 @@
+#![no_main]
+//! Arbitrary bytes as a WAL file. Reading, and opening for append (which scans
+//! the chain and truncates an unverifiable tail), must never panic, hang, or
+//! allocate without bound — and after `open` repairs the file, everything
+//! `read_entries` returned before must still be returned.
+//!
+//! The deterministic counterpart that runs in ordinary CI is
+//! `tests/wal_properties.rs`; this target explores inputs it cannot reach.
+
+use std::io::Write as _;
+
+use clawhdf5_agent::wal::WalFile;
+use libfuzzer_sys::fuzz_target;
+
+fuzz_target!(|data: &[u8]| {
+    let Ok(mut tmp) = tempfile::NamedTempFile::new() else {
+        return;
+    };
+    if tmp.write_all(data).and_then(|()| tmp.flush()).is_err() {
+        return;
+    }
+    let before = WalFile::read_entries(tmp.path()).map(|e| e.len());
+    // Only the chained formats (header versions 3 and 4) are repaired in
+    // place. `open` deliberately recreates a legacy-format file from scratch:
+    // `HDF5Memory::open` has already replayed its entries by then.
+    let chained = matches!(data.get(4), Some(3 | 4));
+    let opened = WalFile::open(tmp.path());
+    if !chained {
+        return;
+    }
+    if let (Ok(before), Ok(wal)) = (before, opened) {
+        drop(wal);
+        let after = WalFile::read_entries(tmp.path()).map(|e| e.len());
+        assert_eq!(after.ok(), Some(before), "open() changed what is replayable");
+    }
+});

crates/clawhdf5-agent/src/anomaly.rs

@@ -161,6 +161,9 @@ pub struct WriteEvent {
 // WriteAnomalyDetector
 // ---------------------------------------------------------------------------
 
+/// Upper bound on distinct session ids the detector tracks at once.
+const MAX_TRACKED_SESSIONS: usize = 4096;
+
 /// Tracks write events and raises alerts for suspicious behaviour.
 #[derive(Debug)]
 pub struct WriteAnomalyDetector {
@@ -189,6 +192,23 @@ impl WriteAnomalyDetector {
         if event.timestamp > self.last_timestamp {
             self.last_timestamp = event.timestamp;
         }
+        // Bound the per-session map: a long-lived process sees an unbounded
+        // number of distinct session ids. When it overflows, forget the
+        // sessions with the fewest writes (they are furthest from the limit
+        // this map exists to enforce); the current one is re-added below.
+        if self.session_counts.len() >= MAX_TRACKED_SESSIONS
+            && !self.session_counts.contains_key(&event.session_id)
+        {
+            let mut counts: Vec<u32> = self.session_counts.values().copied().collect();
+            let keep_from = counts.len() / 2;
+            counts.select_nth_unstable(keep_from);
+            let threshold = counts[keep_from];
+            self.session_counts.retain(|_, c| *c >= threshold);
+            if self.session_counts.len() >= MAX_TRACKED_SESSIONS {
+                // Every session had the same count: drop them all.
+                self.session_counts.clear();
+            }
+        }
         *self
             .session_counts
             .entry(event.session_id.clone())
@@ -437,7 +457,11 @@ mod tests {
     fn rate_anomaly_names_offending_session() {
         let mut det = WriteAnomalyDetector::new(cfg());
         for i in 0..11 {
-            det.record_write(event(1.0 + i as f64 * 0.1, "flood-session", MemorySource::User));
+            det.record_write(event(
+                1.0 + i as f64 * 0.1,
+                "flood-session",
+                MemorySource::User,
+            ));
         }
         let alert = det.check_rate_anomaly().unwrap();
         assert!(
@@ -454,11 +478,19 @@ mod tests {
         let mut det = WriteAnomalyDetector::new(cfg());
         // 5 sessions with 1 write each (below any per-session limit)...
         for i in 0..5 {
-            det.record_write(event(1.0 + i as f64 * 0.1, "minor-session", MemorySource::User));
+            det.record_write(event(
+                1.0 + i as f64 * 0.1,
+                "minor-session",
+                MemorySource::User,
+            ));
         }
         // ...plus one session responsible for the majority of the flood.
         for i in 0..8 {
-            det.record_write(event(2.0 + i as f64 * 0.1, "major-session", MemorySource::User));
+            det.record_write(event(
+                2.0 + i as f64 * 0.1,
+                "major-session",
+                MemorySource::User,
+            ));
         }
         let alert = det.check_rate_anomaly().unwrap();
         assert!(

crates/clawhdf5-agent/src/async_memory.rs

@@ -408,6 +408,10 @@ impl AsyncHDF5Memory {
         let (tx, rx) = oneshot::channel();
         let _ = self.write_tx.send(WriteCmd::Shutdown(tx)).await;
         let _ = rx.await;
+        // The writer task has stopped, so nothing can write through this
+        // handle any more: release the single-writer lock now rather than at
+        // drop, so the store can be reopened while `self` is still in scope.
+        self.inner.lock().await.release_store_lock();
         Ok(())
     }
 }

crates/clawhdf5-agent/src/consolidation.rs

@@ -563,7 +563,7 @@ mod tests {
     #[test]
     fn test_importance_scorer_surprise_identical() {
         let emb = unit_vec(4, 0);
-        let existing = vec![MemoryRecord {
+        let existing = [MemoryRecord {
             id: 0,
             chunk: "existing".to_string(),
             embedding: emb.clone(),
@@ -603,23 +603,20 @@ mod tests {
     fn test_importance_scorer_length() {
         assert!((ImportanceScorer::score_length("")).abs() < f32::EPSILON);
         // 50 words → 0.5
-        let fifty_words = std::iter::repeat("word")
+        let fifty_words = std::iter::repeat_n("word", 50)
-            .take(50)
             .collect::<Vec<_>>()
             .join(" ");
         let s50 = ImportanceScorer::score_length(&fifty_words);
         assert!((s50 - 0.5).abs() < 1e-5, "expected 0.5, got {s50}");
 
         // 100 words → 1.0
-        let hundred_words = std::iter::repeat("word")
+        let hundred_words = std::iter::repeat_n("word", 100)
-            .take(100)
             .collect::<Vec<_>>()
             .join(" ");
         assert_eq!(ImportanceScorer::score_length(&hundred_words), 1.0);
 
         // 200 words → still 1.0 (clamped)
-        let two_hundred = std::iter::repeat("word")
+        let two_hundred = std::iter::repeat_n("word", 200)
-            .take(200)
             .collect::<Vec<_>>()
             .join(" ");
         assert_eq!(ImportanceScorer::score_length(&two_hundred), 1.0);
@@ -693,9 +690,11 @@ mod tests {
     // ---------------------------------------------------------------------------
     #[test]
     fn test_consolidate_eviction_working() {
-        let mut cfg = ConsolidationConfig::default();
+        let cfg = ConsolidationConfig {
-        cfg.working_capacity = 3;
+            working_capacity: 3,
-        cfg.working_to_episodic_threshold = 2.0; // never promote in this test
+            working_to_episodic_threshold: 2.0, // never promote in this test
+            ..Default::default()
+        };
         let mut engine = ConsolidationEngine::new(cfg);
 
         // Add 5 records; all have very low importance so none get promoted.
@@ -800,7 +799,12 @@ mod tests {
     #[test]
     fn test_access_memory_reactivation() {
         let mut engine = ConsolidationEngine::new(ConsolidationConfig::default());
-        let id = engine.add_memory("chunk".to_string(), unit_vec(4, 0), UntrustedSource::User, 0.0);
+        let id = engine.add_memory(
+            "chunk".to_string(),
+            unit_vec(4, 0),
+            UntrustedSource::User,
+            0.0,
+        );
 
         engine.access_memory(id, 5000.0);
         let rec = engine.get_by_id(id).unwrap();
@@ -853,9 +857,11 @@ mod tests {
     // ---------------------------------------------------------------------------
     #[test]
     fn test_consolidate_episodic_eviction() {
-        let mut cfg = ConsolidationConfig::default();
+        let cfg = ConsolidationConfig {
-        cfg.episodic_capacity = 3;
+            episodic_capacity: 3,
-        cfg.working_to_episodic_threshold = 2.0; // never auto-promote from Working
+            working_to_episodic_threshold: 2.0, // never auto-promote from Working
+            ..Default::default()
+        };
         let mut engine = ConsolidationEngine::new(cfg);
 
         // Seed 5 records directly in Episodic.

crates/clawhdf5-agent/src/entity_extract.rs

@@ -777,8 +777,10 @@ mod tests {
 
     #[test]
     fn test_tech_disabled() {
-        let mut config = ExtractorConfig::default();
+        let config = ExtractorConfig {
-        config.extract_technology = false;
+            extract_technology: false,
+            ..Default::default()
+        };
         let e = EntityExtractor::new(config);
         let entities = e.extract("We use Rust and Docker.");
         assert!(
@@ -847,8 +849,10 @@ mod tests {
 
     #[test]
     fn test_date_disabled() {
-        let mut config = ExtractorConfig::default();
+        let config = ExtractorConfig {
-        config.extract_dates = false;
+            extract_dates: false,
+            ..Default::default()
+        };
         let e = EntityExtractor::new(config);
         let entities = e.extract("Released on 2024-03-19.");
         assert!(
@@ -981,8 +985,10 @@ mod tests {
 
     #[test]
     fn test_confidence_filter() {
-        let mut config = ExtractorConfig::default();
+        let config = ExtractorConfig {
-        config.min_confidence = 0.95;
+            min_confidence: 0.95,
+            ..Default::default()
+        };
         let e = EntityExtractor::new(config);
         // Only dates (0.95) and techs (0.9) should survive; 0.9 < 0.95 filters techs.
         let entities = e.extract("We use Rust since 2024-01-01.");
@@ -1002,7 +1008,7 @@ mod tests {
     fn test_batch_dedup() {
         let e = default_extractor();
         let texts = ["We use Rust.", "Rust is fast.", "Also Rust for safety."];
-        let entities = e.extract_batch(&texts.iter().map(|s| *s).collect::<Vec<_>>());
+        let entities = e.extract_batch(&texts);
         let rust_count = entities.iter().filter(|x| x.text == "Rust").count();
         assert_eq!(rust_count, 1, "Rust should appear exactly once after dedup");
     }
@@ -1011,7 +1017,7 @@ mod tests {
     fn test_batch_multiple_types() {
         let e = default_extractor();
         let texts = ["Deploy with Docker.", "We merged last week."];
-        let entities = e.extract_batch(&texts.iter().map(|s| *s).collect::<Vec<_>>());
+        let entities = e.extract_batch(&texts);
         assert!(
             entities
                 .iter()

crates/clawhdf5-agent/src/hybrid.rs

@@ -91,14 +91,22 @@ pub fn merge_vector_keyword(
     }
 
     let mut results: Vec<(usize, f32)> = merged.into_iter().collect();
-    results.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));
+    // Index tie-break: `merged` is a HashMap, so without it the ties that
+    // survive `truncate` differ from run to run.
+    results.sort_by(|a, b| {
+        b.1.partial_cmp(&a.1)
+            .unwrap_or(std::cmp::Ordering::Equal)
+            .then(a.0.cmp(&b.0))
+    });
     results.truncate(k);
     results
 }
 
 /// Normalize a set of scores to the [0, 1] range using min-max normalization.
 ///
-/// If all scores are identical, returns 0.0 for each entry.
+/// If all scores are identical there is no spread to normalise: each entry
+/// gets 1.0 when that score is positive (all equally the best match) and 0.0
+/// otherwise (nothing matched).
 fn normalize_scores(scores: &[(usize, f32)]) -> Vec<(usize, f32)> {
     if scores.is_empty() {
         return Vec::new();
@@ -112,7 +120,13 @@ fn normalize_scores(scores: &[(usize, f32)]) -> Vec<(usize, f32)> {
 
     let range = max - min;
     if range == 0.0 {
-        return scores.iter().map(|(idx, _)| (*idx, 0.0)).collect();
+        // All candidates scored the same (including the single-candidate
+        // case), so min-max has no spread to work with. They are all equally
+        // the best match if that score is positive, and all non-matches
+        // otherwise. This used to return 0.0 unconditionally, which erased a
+        // lone perfect match from the fused score.
+        let level = if max > 0.0 { 1.0 } else { 0.0 };
+        return scores.iter().map(|(idx, _)| (*idx, level)).collect();
     }
 
     scores
@@ -324,10 +338,18 @@ mod tests {
 
     #[test]
     fn normalize_scores_single() {
+        // A lone positive score is the best match there is, not a non-match.
         let result = normalize_scores(&[(0, 5.0)]);
         assert_eq!(result.len(), 1);
-        // Single score normalizes to 0.0 (range is 0)
+        assert_eq!(result[0].1, 1.0);
-        assert_eq!(result[0].1, 0.0);
+    }
+
+    #[test]
+    fn normalize_scores_all_equal() {
+        let matched = normalize_scores(&[(0, 0.4), (1, 0.4)]);
+        assert!(matched.iter().all(|(_, s)| *s == 1.0));
+        let unmatched = normalize_scores(&[(0, 0.0), (1, 0.0)]);
+        assert!(unmatched.iter().all(|(_, s)| *s == 0.0));
     }
 
     #[test]

crates/clawhdf5-agent/src/lib.rs

@@ -37,6 +37,7 @@ pub mod schema;
 pub mod search;
 pub mod session;
 pub mod storage;
+mod store_lock;
 pub mod temporal;
 pub mod wal;
 
@@ -86,6 +87,8 @@ pub enum MemoryError {
     Hdf5(String),
     Schema(String),
     NotFound(String),
+    /// Another `HDF5Memory` (in this or another process) has the store open.
+    Locked(String),
 }
 
 impl std::fmt::Display for MemoryError {
@@ -95,6 +98,7 @@ impl std::fmt::Display for MemoryError {
             MemoryError::Hdf5(e) => write!(f, "HDF5 error: {e}"),
             MemoryError::Schema(e) => write!(f, "schema error: {e}"),
             MemoryError::NotFound(e) => write!(f, "not found: {e}"),
+            MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
         }
     }
 }
@@ -201,6 +205,9 @@ pub trait AgentMemory {
     fn get_session_summary(&self, session_id: &str) -> Result<Option<String>>;
 }
 
+/// Most anomaly alerts kept between `take_anomaly_alerts` calls.
+const MAX_PENDING_ALERTS: usize = 1024;
+
 // --- HDF5Memory ---
 
 pub struct HDF5Memory {
@@ -240,6 +247,15 @@ pub struct HDF5Memory {
     /// via [`HDF5Memory::take_anomaly_alerts`]. Saves are never blocked on
     /// these — surfacing is opt-in for callers that want to act on them.
     anomaly_alerts: Vec<anomaly::AnomalyAlert>,
+    /// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk.
+    read_only: bool,
+    /// A WAL that `open()` could not read and moved aside; see
+    /// [`HDF5Memory::quarantined_wal`].
+    quarantined_wal: Option<PathBuf>,
+    /// Single-writer guard. Declared last so it is released only after the
+    /// WAL and everything else has been dropped. `None` once a wrapper that
+    /// has stopped all writes released it early (see `release_store_lock`).
+    _lock: Option<store_lock::StoreLock>,
 }
 
 impl std::fmt::Debug for HDF5Memory {
@@ -251,6 +267,7 @@ impl std::fmt::Debug for HDF5Memory {
 impl HDF5Memory {
     /// Create a new HDF5 memory file with the given configuration.
     pub fn create(config: MemoryConfig) -> Result<Self> {
+        let lock = store_lock::StoreLock::acquire(&config.path)?;
         let cache = MemoryCache::new(config.embedding_dim);
         let sessions = SessionCache::new();
         let knowledge = KnowledgeCache::new();
@@ -282,20 +299,110 @@ impl HDF5Memory {
             provenance: provenance::ProvenanceStore::new(),
             anomaly: anomaly::WriteAnomalyDetector::new(anomaly::AnomalyConfig::default()),
             anomaly_alerts: Vec::new(),
+            read_only: false,
+            quarantined_wal: None,
+            _lock: Some(lock),
         })
     }
 
     /// Open an existing HDF5 memory file.
+    /// If the WAL at `wal_path` can't possibly be replayed — its header is
+    /// torn (crash while the file was being created) or isn't a WAL header at
+    /// all — move it aside so a healthy `.h5` still opens, and return where it
+    /// went. A well-formed header with an *unknown version* is left alone and
+    /// still fails `open()`: that WAL was most likely written by a newer
+    /// build, and discarding it would lose data this binary merely can't read.
+    fn quarantine_unreadable_wal(wal_path: &Path) -> Result<Option<PathBuf>> {
+        if !wal_path.exists() {
+            return Ok(None);
+        }
+        let reason = match wal::wal_header_status(wal_path)? {
+            wal::WalHeaderStatus::Readable | wal::WalHeaderStatus::UnknownVersion(_) => {
+                return Ok(None);
+            }
+            wal::WalHeaderStatus::Torn => "truncated header",
+            wal::WalHeaderStatus::BadMagic => "bad magic bytes",
+        };
+        let ts = std::time::SystemTime::now()
+            .duration_since(std::time::UNIX_EPOCH)
+            .unwrap_or_default()
+            .as_secs();
+        let dest = wal_path.with_extension(format!("wal.corrupt-{ts}"));
+        std::fs::rename(wal_path, &dest)?;
+        eprintln!(
+            "clawhdf5-agent: WAL {} is unreadable ({reason}); moved to {} and continuing \
+             from the last checkpoint",
+            wal_path.display(),
+            dest.display()
+        );
+        Ok(Some(dest))
+    }
+
+    /// Give up the single-writer lock before this value is dropped. Only for
+    /// wrappers that have already stopped every write path but keep the handle
+    /// alive (`AsyncHDF5Memory::shutdown`), so the store can be reopened.
+    #[cfg_attr(not(feature = "async"), allow(dead_code))]
+    pub(crate) fn release_store_lock(&mut self) {
+        self._lock = None;
+    }
+
+    /// Where `open()` moved an unreadable WAL, if it had to. Entries that were
+    /// only in that WAL are not in this store; the file is kept for forensics.
+    pub fn quarantined_wal(&self) -> Option<&Path> {
+        self.quarantined_wal.as_deref()
+    }
+
     pub fn open(path: &Path) -> Result<Self> {
-        let (config, mut cache, sessions, knowledge) = storage::read_from_disk(path)?;
+        Self::open_impl(path, false)
+    }
+
+    /// Open a store for reading only, without taking the single-writer lock —
+    /// so it works while another `HDF5Memory` (in this or another process)
+    /// has the store open for writing, e.g. to inspect what is on disk.
+    ///
+    /// It loads the last checkpoint plus whatever the WAL held at that
+    /// moment; it is a point-in-time view and does not follow later writes.
+    /// Nothing is written: the WAL file is not repaired, upgraded or moved,
+    /// and every operation that would persist state returns an error.
+    pub fn open_read_only(path: &Path) -> Result<Self> {
+        Self::open_impl(path, true)
+    }
+
+    fn open_impl(path: &Path, read_only: bool) -> Result<Self> {
+        let lock = if read_only {
+            None
+        } else {
+            Some(store_lock::StoreLock::acquire(path)?)
+        };
+        let ((config, mut cache, sessions, knowledge), wal_applied) =
+            storage::read_from_disk_with_mark(path)?;
 
         // Replay WAL if present
         let wal_path = path.with_extension("h5.wal");
-        let wal = if wal_path.exists() {
+        let quarantined_wal = if read_only {
+            None
+        } else {
+            Self::quarantine_unreadable_wal(&wal_path)?
+        };
+        let wal = if read_only {
+            // Replay in memory only. `WalFile::open` would truncate a torn
+            // tail and may rewrite the header — both belong to the writer. An
+            // unreadable WAL is simply skipped: the writer will deal with it.
+            if wal_path.exists()
+                && let Ok(entries) =
+                    wal::WalFile::read_entries_for_migration(&wal_path, wal_applied)
+            {
+                wal::replay_into_cache(&entries, &mut cache);
+            }
+            None
+        } else if wal_path.exists() {
             // Uses the migration-only reader since this is the one legitimate
             // path that may need to read a legacy (pre-CRC) WAL file — see
             // WalFile::read_entries_for_migration.
-            let entries = wal::WalFile::read_entries_for_migration(&wal_path)?;
+            // `wal_applied` drops the prefix a checkpoint already folded in,
+            // in case the process died between writing the .h5 and
+            // truncating the WAL.
+            let entries = wal::WalFile::read_entries_for_migration(&wal_path, wal_applied)?;
             wal::replay_into_cache(&entries, &mut cache);
             Some(wal::WalFile::open(&wal_path)?)
         } else if config.wal_enabled {
@@ -327,6 +434,9 @@ impl HDF5Memory {
             provenance: provenance::ProvenanceStore::new(),
             anomaly: anomaly::WriteAnomalyDetector::new(anomaly::AnomalyConfig::default()),
             anomaly_alerts: Vec::new(),
+            read_only,
+            quarantined_wal,
+            _lock: lock,
         })
     }
 
@@ -336,12 +446,22 @@ impl HDF5Memory {
     /// also clear the WAL, otherwise `open()` will replay stale entries
     /// on top of the already-persisted data, duplicating them.
     fn flush(&mut self) -> Result<()> {
-        storage::write_to_disk(
+        if self.read_only {
+            return Err(MemoryError::Io(std::io::Error::new(
+                std::io::ErrorKind::PermissionDenied,
+                "store was opened read-only",
+            )));
+        }
+        // Record which WAL prefix this checkpoint contains, so a crash before
+        // the truncate below can't replay those entries a second time.
+        let wal_applied = self.wal.as_ref().map(|w| w.mark());
+        storage::write_to_disk_with_mark(
             &self.config.path,
             &self.config,
             &self.cache,
             &self.sessions,
             &self.knowledge,
+            wal_applied,
         )?;
         if let Some(ref mut w) = self.wal {
             w.truncate()?;
@@ -410,7 +530,7 @@ impl HDF5Memory {
         .into_iter()
         .flatten()
         {
-            self.anomaly_alerts.push(alert);
+            self.push_anomaly_alert(alert);
         }
     }
 
@@ -427,8 +547,11 @@ impl HDF5Memory {
         if self.provenance.get(record_id as u64).is_none() {
             return; // nothing recorded yet this session — nothing to check
         }
-        if !self.provenance.verify_integrity(record_id as u64, current_chunk) {
+        if !self
-            self.anomaly_alerts.push(anomaly::AnomalyAlert {
+            .provenance
+            .verify_integrity(record_id as u64, current_chunk)
+        {
+            self.push_anomaly_alert(anomaly::AnomalyAlert {
                 severity: anomaly::Severity::High,
                 message: format!(
                     "provenance integrity mismatch for record {record_id}: stored content no \
@@ -439,6 +562,18 @@ impl HDF5Memory {
         }
     }
 
+    /// Queue an alert, keeping only the most recent [`MAX_PENDING_ALERTS`].
+    /// Alerts never block a save, so a caller that never drains them — or a
+    /// session stuck over its write limit, which alerts on every write —
+    /// must not be able to grow this without bound.
+    fn push_anomaly_alert(&mut self, alert: anomaly::AnomalyAlert) {
+        if self.anomaly_alerts.len() >= MAX_PENDING_ALERTS {
+            let excess = self.anomaly_alerts.len() + 1 - MAX_PENDING_ALERTS;
+            self.anomaly_alerts.drain(..excess);
+        }
+        self.anomaly_alerts.push(alert);
+    }
+
     /// Alerts raised by anomaly detection / provenance checks since the last
     /// call, draining the internal queue.
     pub fn take_anomaly_alerts(&mut self) -> Vec<anomaly::AnomalyAlert> {
@@ -618,7 +753,7 @@ impl HDF5Memory {
         if let Some(existing_idx) = self.cache.find_by_tags(&entry.tags) {
             if let Some(ref mut w) = self.wal {
                 let wal_entry = wal::WalEntry {
-                    entry_type: wal::WalEntryType::Save,
+                    entry_type: wal::WalEntryType::Update,
                     timestamp: entry.timestamp,
                     chunk: entry.chunk.clone(),
                     embedding: entry.embedding.clone(),
@@ -626,6 +761,7 @@ impl HDF5Memory {
                     session_id: entry.session_id.clone(),
                     tags: entry.tags.clone(),
                     tombstone_index: None,
+                    update_index: Some(existing_idx),
                 };
                 w.append_save(&wal_entry)?;
             }
@@ -657,9 +793,6 @@ impl HDF5Memory {
                 .is_none_or(|w| w.pending_count() as usize > self.config.wal_max_entries);
             if needs_flush {
                 self.flush()?;
-                if let Some(ref mut w) = self.wal {
-                    w.truncate()?;
-                }
             }
             return Ok(existing_idx);
         }
@@ -680,6 +813,7 @@ impl AgentMemory for HDF5Memory {
                 session_id: entry.session_id.clone(),
                 tags: entry.tags.clone(),
                 tombstone_index: None,
+                update_index: None,
             };
             w.append_save(&wal_entry)?;
         }
@@ -705,9 +839,6 @@ impl AgentMemory for HDF5Memory {
             .is_none_or(|w| w.pending_count() as usize > self.config.wal_max_entries);
         if needs_flush {
             self.flush()?;
-            if let Some(ref mut w) = self.wal {
-                w.truncate()?;
-            }
         }
         Ok(idx)
     }
@@ -758,8 +889,10 @@ impl AgentMemory for HDF5Memory {
     }
 
     fn compact(&mut self) -> Result<usize> {
-        let (removed, _index_map) = self.cache.compact();
+        let (removed, index_map) = self.cache.compact();
         if removed > 0 {
+            // Record ids are cache indices, which compaction just renumbered.
+            self.provenance.remap(&index_map);
             // Compaction renumbers cache indices; rebuild the index to match.
             self.hnsw_mark_dirty();
             self.flush()?;
@@ -776,7 +909,18 @@ impl AgentMemory for HDF5Memory {
     }
 
     fn snapshot(&self, dest: &Path) -> Result<PathBuf> {
-        storage::snapshot_file(&self.config.path, dest)
+        let snapshot = storage::snapshot_file(&self.config.path, dest)?;
+        // Entries saved since the last checkpoint live only in the WAL. Copy
+        // it alongside (where `open()` looks for it) so the snapshot is the
+        // store as it is now, not as of the last checkpoint. The .h5 is
+        // copied first: if a checkpoint lands in between, the WAL copy is
+        // empty or its prefix is skipped via the checkpoint mark — never
+        // applied twice.
+        let wal_path = self.config.path.with_extension("h5.wal");
+        if self.wal.as_ref().is_some_and(|w| !w.is_empty()) && wal_path.exists() {
+            storage::snapshot_file(&wal_path, &snapshot.with_extension("h5.wal"))?;
+        }
+        Ok(snapshot)
     }
 
     fn add_session(
@@ -839,6 +983,193 @@ fn is_leap(y: i64) -> bool {
     (y % 4 == 0 && y % 100 != 0) || y % 400 == 0
 }
 
+impl HDF5Memory {
+    pub fn set_strategy(&mut self, s: Box<dyn MemoryStrategy>) {
+        self.strategy = Some(s);
+    }
+    pub fn record(&mut self, exchange: Exchange) -> Result<StrategyOutput> {
+        let strat = self.strategy.as_ref().ok_or_else(|| {
+            MemoryError::Schema(
+                "strategy not initialized: call set_strategy() before record()".to_owned(),
+            )
+        })?;
+        let view = memory_strategy::CacheStoreView::new(&self.cache, &self.knowledge);
+        let output = strat.evaluate(&exchange, &view);
+        for e in &output.entries {
+            self.cache.push(
+                e.chunk.clone(),
+                e.embedding.clone(),
+                e.source_channel.clone(),
+                e.timestamp,
+                e.session_id.clone(),
+                e.tags.clone(),
+            );
+        }
+        for eu in &output.entity_updates {
+            let id = self.knowledge.add_entity(&eu.name, &eu.entity_type, -1);
+            for a in &eu.aliases {
+                self.knowledge.add_alias(a, id as i64);
+            }
+        }
+        if !output.entries.is_empty() || !output.entity_updates.is_empty() {
+            self.flush()?;
+        }
+        Ok(output)
+    }
+}
+
+impl HDF5Memory {
+    pub fn tick_session(&mut self) -> Result<()> {
+        let d = self.config.decay_factor;
+        for w in self.cache.activation_weights.iter_mut() {
+            *w *= d;
+        }
+        self.flush()?;
+        Ok(())
+    }
+
+    /// Number of pending WAL entries (0 if WAL disabled).
+    pub fn wal_pending_count(&self) -> usize {
+        self.wal.as_ref().map_or(0, |w| w.pending_count() as usize)
+    }
+
+    /// Explicit WAL merge: flush .h5, truncate WAL.
+    pub fn flush_wal(&mut self) -> Result<()> {
+        self.flush()?;
+        Ok(())
+    }
+}
+
+// ─────────────────────────────────────────────────────────────────────────────
+// Ephemeral tier integration
+// ─────────────────────────────────────────────────────────────────────────────
+
+impl HDF5Memory {
+    /// Enable the ephemeral working memory tier with the given configuration.
+    pub fn enable_ephemeral(&mut self, config: EphemeralConfig) {
+        self.ephemeral = Some(EphemeralStore::new(config));
+    }
+
+    /// Return a shared reference to the ephemeral store, if enabled.
+    pub fn ephemeral(&self) -> Option<&EphemeralStore> {
+        self.ephemeral.as_ref()
+    }
+
+    /// Return a mutable reference to the ephemeral store, if enabled.
+    pub fn ephemeral_mut(&mut self) -> Option<&mut EphemeralStore> {
+        self.ephemeral.as_mut()
+    }
+
+    /// Promote frequently-accessed ephemeral entries into the persistent cache.
+    ///
+    /// Every entry whose `access_count >= min_access_count` is removed from the
+    /// ephemeral store and written to the HDF5 cache, then the file is flushed.
+    /// Returns the number of entries promoted.
+    pub fn promote_ephemeral(&mut self, min_access_count: u32) -> Result<usize> {
+        let candidates = match &self.ephemeral {
+            None => return Ok(0),
+            Some(s) => s.promotion_candidates(min_access_count),
+        };
+
+        if candidates.is_empty() {
+            return Ok(0);
+        }
+
+        let dim = self.config.embedding_dim;
+        let mut promoted = 0;
+
+        for key in candidates {
+            let entry = match self
+                .ephemeral
+                .as_mut()
+                .and_then(|s| s.take_for_promotion(&key))
+            {
+                Some(e) => e,
+                None => continue,
+            };
+
+            let chunk = entry
+                .text
+                .clone()
+                .unwrap_or_else(|| String::from_utf8_lossy(&entry.value).into_owned());
+            let embedding = entry.embedding.clone().unwrap_or_else(|| vec![0.0f32; dim]);
+
+            self.cache.push(
+                chunk,
+                embedding,
+                format!("ephemeral::{key}"),
+                entry.created_at,
+                String::new(),
+                entry.tags.join(","),
+            );
+            promoted += 1;
+        }
+
+        if promoted > 0 {
+            self.flush()?;
+        }
+        Ok(promoted)
+    }
+
+    /// Search both the persistent HDF5 tier and the ephemeral tier, returning
+    /// the top `k` results sorted by score descending.
+    ///
+    /// Ephemeral results are boosted by a factor of 1.2 to surface recent
+    /// in-context information above older persisted data.
+    pub fn unified_search(
+        &mut self,
+        query_embedding: &[f32],
+        query_text: &str,
+        k: usize,
+    ) -> Vec<SearchResult> {
+        // Persistent tier.
+        let persistent = self.hybrid_search(query_embedding, query_text, 0.7, 0.3, k);
+        const EPHEMERAL_BOOST: f32 = 1.2;
+        let mut results = persistent;
+
+        if self.ephemeral.is_none() {
+            return results;
+        }
+
+        let eph = self.ephemeral.as_mut().unwrap();
+
+        // Collect (key, score) pairs from ephemeral — borrow ends before we
+        // access entries again below.
+        let eph_hits: Vec<(String, f32)> = if !query_embedding.is_empty() {
+            eph.search_embedding(query_embedding, k)
+        } else if !query_text.is_empty() {
+            eph.search_text(query_text, k)
+        } else {
+            Vec::new()
+        };
+
+        for (key, score) in &eph_hits {
+            if let Some(entry) = eph.get_entry(key) {
+                let chunk = entry
+                    .text
+                    .clone()
+                    .unwrap_or_else(|| String::from_utf8_lossy(&entry.value).into_owned());
+                results.push(SearchResult {
+                    score: score * EPHEMERAL_BOOST,
+                    chunk,
+                    index: usize::MAX,
+                    timestamp: entry.created_at,
+                    source_channel: format!("ephemeral::{key}"),
+                    activation: 1.0,
+                });
+            }
+        }
+
+        results.sort_by(|a, b| {
+            b.score
+                .partial_cmp(&a.score)
+                .unwrap_or(std::cmp::Ordering::Equal)
+        });
+        results.truncate(k);
+        results
+    }
+}
+
 // --- Tests ---
 
 #[cfg(test)]
@@ -1205,6 +1536,149 @@ mod tests {
         assert_eq!(mem.count(), 1);
     }
 
+    #[test]
+    fn compaction_does_not_cause_false_provenance_alerts() {
+        let dir = TempDir::new().unwrap();
+        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
+        for name in ["a", "b", "c"] {
+            let mut e = make_entry(name, &[1.0, 0.0, 0.0, 0.0]);
+            e.tags = format!("tag-{name}");
+            mem.save(e).unwrap();
+        }
+        // 1 of 3 tombstoned exceeds compact_threshold, so delete() compacts.
+        mem.delete(0).unwrap();
+        assert_eq!(mem.cache.chunks, ["b", "c"]);
+        mem.take_anomaly_alerts();
+
+        // "c" moved from id 2 to id 1. Its recorded hash must have moved too,
+        // or this update is checked against "b"'s hash and flagged.
+        let mut update = make_entry("c2", &[0.0, 1.0, 0.0, 0.0]);
+        update.tags = "tag-c".into();
+        assert_eq!(mem.save_or_update(update).unwrap(), 1);
+        let alerts = mem.take_anomaly_alerts();
+        assert!(
+            !alerts.iter().any(|a| a.message.contains("provenance")),
+            "{alerts:?}"
+        );
+    }
+
+    #[test]
+    fn pending_alerts_are_bounded() {
+        let dir = TempDir::new().unwrap();
+        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
+        for i in 0..(MAX_PENDING_ALERTS + 50) {
+            mem.push_anomaly_alert(anomaly::AnomalyAlert {
+                severity: anomaly::Severity::Low,
+                message: format!("alert {i}"),
+                timestamp: i as f64,
+            });
+        }
+        let alerts = mem.take_anomaly_alerts();
+        assert_eq!(alerts.len(), MAX_PENDING_ALERTS);
+        assert_eq!(alerts[0].message, "alert 50", "oldest are dropped first");
+    }
+
+    #[test]
+    fn snapshot_includes_entries_still_in_the_wal() {
+        let dir = TempDir::new().unwrap();
+        let mut config = make_config(&dir);
+        config.wal_enabled = true;
+        let mut mem = HDF5Memory::create(config).unwrap();
+        mem.save(make_entry("checkpointed", &[1.0, 0.0, 0.0, 0.0]))
+            .unwrap();
+        mem.flush_wal().unwrap();
+        mem.save(make_entry("wal-only", &[0.0, 1.0, 0.0, 0.0]))
+            .unwrap();
+
+        let snap = mem.snapshot(&dir.path().join("snap.h5")).unwrap();
+        let restored = HDF5Memory::open(&snap).unwrap();
+        assert_eq!(restored.cache.chunks, ["checkpointed", "wal-only"]);
+    }
+
+    #[test]
+    fn store_has_a_single_writer() {
+        let dir = TempDir::new().unwrap();
+        let config = make_config(&dir);
+        let path = config.path.clone();
+        let mem = HDF5Memory::create(config).unwrap();
+        assert!(matches!(
+            HDF5Memory::open(&path),
+            Err(MemoryError::Locked(_))
+        ));
+        drop(mem);
+        HDF5Memory::open(&path).unwrap();
+    }
+
+    #[test]
+    fn read_only_open_coexists_with_a_writer_and_never_writes() {
+        let dir = TempDir::new().unwrap();
+        let mut config = make_config(&dir);
+        config.wal_enabled = true;
+        let path = config.path.clone();
+        let wal_path = path.with_extension("h5.wal");
+        let mut writer = HDF5Memory::create(config).unwrap();
+        writer
+            .save(make_entry("pending", &[1.0, 0.0, 0.0, 0.0]))
+            .unwrap();
+        let wal_before = std::fs::read(&wal_path).unwrap();
+        let h5_before = std::fs::read(&path).unwrap();
+
+        // Sees the checkpoint plus the writer's un-checkpointed WAL entry.
+        let mut reader = HDF5Memory::open_read_only(&path).unwrap();
+        assert_eq!(reader.cache.chunks, ["pending"]);
+        assert!(reader.save(make_entry("nope", &[0.0; 4])).is_err());
+        assert!(reader.flush_wal().is_err());
+        drop(reader);
+
+        assert_eq!(std::fs::read(&wal_path).unwrap(), wal_before);
+        assert_eq!(std::fs::read(&path).unwrap(), h5_before);
+        // The writer is unaffected.
+        writer
+            .save(make_entry("more", &[0.0, 1.0, 0.0, 0.0]))
+            .unwrap();
+    }
+
+    #[test]
+    fn unreadable_wal_is_quarantined_not_fatal() {
+        let dir = TempDir::new().unwrap();
+        let config = make_config(&dir);
+        let path = config.path.clone();
+        let wal_path = path.with_extension("h5.wal");
+        {
+            let mut mem = HDF5Memory::create(config).unwrap();
+            mem.save(make_entry("kept", &[1.0, 0.0, 0.0, 0.0])).unwrap();
+            mem.flush_wal().unwrap();
+        }
+        std::fs::write(&wal_path, b"not a wal at all").unwrap();
+
+        let mem = HDF5Memory::open(&path).unwrap();
+        assert_eq!(mem.cache.chunks, ["kept"]);
+        let moved = mem.quarantined_wal().expect("WAL should be quarantined");
+        assert_eq!(std::fs::read(moved).unwrap(), b"not a wal at all");
+        // A fresh, valid WAL took its place.
+        assert!(wal::WalFile::read_entries(&wal_path).unwrap().is_empty());
+    }
+
+    #[test]
+    fn wal_from_a_newer_build_is_refused_not_discarded() {
+        let dir = TempDir::new().unwrap();
+        let mut config = make_config(&dir);
+        config.wal_enabled = true;
+        let path = config.path.clone();
+        let wal_path = path.with_extension("h5.wal");
+        drop(HDF5Memory::create(config).unwrap());
+        let mut bytes = std::fs::read(&wal_path).unwrap();
+        bytes[4] = 200; // a version this build has never heard of
+        std::fs::write(&wal_path, &bytes).unwrap();
+
+        assert!(HDF5Memory::open(&path).is_err());
+        assert_eq!(
+            std::fs::read(&wal_path).unwrap(),
+            bytes,
+            "WAL left untouched"
+        );
+    }
+
     #[test]
     fn empty_file_operations() {
         let dir = TempDir::new().unwrap();
@@ -1213,6 +1687,7 @@ mod tests {
         let mem = HDF5Memory::create(config).unwrap();
         assert_eq!(mem.count(), 0);
         assert_eq!(mem.count_active(), 0);
+        drop(mem); // a store has a single writer; release it before reopening
 
         let mem2 = HDF5Memory::open(&path).unwrap();
         assert_eq!(mem2.count(), 0);
@@ -1676,196 +2151,3 @@ mod tests {
         assert!((mem.cache.tombstone_fraction() - 0.50).abs() < 0.01);
     }
 }
-
-impl HDF5Memory {
-    pub fn set_strategy(&mut self, s: Box<dyn MemoryStrategy>) {
-        self.strategy = Some(s);
-    }
-    pub fn record(&mut self, exchange: Exchange) -> Result<StrategyOutput> {
-        let strat = self.strategy.as_ref().ok_or_else(|| {
-            MemoryError::Schema(
-                "strategy not initialized: call set_strategy() before record()".to_owned(),
-            )
-        })?;
-        let view = memory_strategy::CacheStoreView::new(&self.cache, &self.knowledge);
-        let output = strat.evaluate(&exchange, &view);
-        for e in &output.entries {
-            self.cache.push(
-                e.chunk.clone(),
-                e.embedding.clone(),
-                e.source_channel.clone(),
-                e.timestamp,
-                e.session_id.clone(),
-                e.tags.clone(),
-            );
-        }
-        for eu in &output.entity_updates {
-            let id = self.knowledge.add_entity(&eu.name, &eu.entity_type, -1);
-            for a in &eu.aliases {
-                self.knowledge.add_alias(a, id as i64);
-            }
-        }
-        if !output.entries.is_empty() || !output.entity_updates.is_empty() {
-            self.flush()?;
-        }
-        Ok(output)
-    }
-}
-
-impl HDF5Memory {
-    pub fn tick_session(&mut self) -> Result<()> {
-        let d = self.config.decay_factor;
-        for w in self.cache.activation_weights.iter_mut() {
-            *w *= d;
-        }
-        self.flush()?;
-        if let Some(ref mut w) = self.wal {
-            w.truncate()?;
-        }
-        Ok(())
-    }
-
-    /// Number of pending WAL entries (0 if WAL disabled).
-    pub fn wal_pending_count(&self) -> usize {
-        self.wal.as_ref().map_or(0, |w| w.pending_count() as usize)
-    }
-
-    /// Explicit WAL merge: flush .h5, truncate WAL.
-    pub fn flush_wal(&mut self) -> Result<()> {
-        self.flush()?;
-        if let Some(ref mut w) = self.wal {
-            w.truncate()?;
-        }
-        Ok(())
-    }
-}
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Ephemeral tier integration
-// ─────────────────────────────────────────────────────────────────────────────
-
-impl HDF5Memory {
-    /// Enable the ephemeral working memory tier with the given configuration.
-    pub fn enable_ephemeral(&mut self, config: EphemeralConfig) {
-        self.ephemeral = Some(EphemeralStore::new(config));
-    }
-
-    /// Return a shared reference to the ephemeral store, if enabled.
-    pub fn ephemeral(&self) -> Option<&EphemeralStore> {
-        self.ephemeral.as_ref()
-    }
-
-    /// Return a mutable reference to the ephemeral store, if enabled.
-    pub fn ephemeral_mut(&mut self) -> Option<&mut EphemeralStore> {
-        self.ephemeral.as_mut()
-    }
-
-    /// Promote frequently-accessed ephemeral entries into the persistent cache.
-    ///
-    /// Every entry whose `access_count >= min_access_count` is removed from the
-    /// ephemeral store and written to the HDF5 cache, then the file is flushed.
-    /// Returns the number of entries promoted.
-    pub fn promote_ephemeral(&mut self, min_access_count: u32) -> Result<usize> {
-        let candidates = match &self.ephemeral {
-            None => return Ok(0),
-            Some(s) => s.promotion_candidates(min_access_count),
-        };
-
-        if candidates.is_empty() {
-            return Ok(0);
-        }
-
-        let dim = self.config.embedding_dim;
-        let mut promoted = 0;
-
-        for key in candidates {
-            let entry = match self
-                .ephemeral
-                .as_mut()
-                .and_then(|s| s.take_for_promotion(&key))
-            {
-                Some(e) => e,
-                None => continue,
-            };
-
-            let chunk = entry
-                .text
-                .clone()
-                .unwrap_or_else(|| String::from_utf8_lossy(&entry.value).into_owned());
-            let embedding = entry.embedding.clone().unwrap_or_else(|| vec![0.0f32; dim]);
-
-            self.cache.push(
-                chunk,
-                embedding,
-                format!("ephemeral::{key}"),
-                entry.created_at,
-                String::new(),
-                entry.tags.join(","),
-            );
-            promoted += 1;
-        }
-
-        if promoted > 0 {
-            self.flush()?;
-        }
-        Ok(promoted)
-    }
-
-    /// Search both the persistent HDF5 tier and the ephemeral tier, returning
-    /// the top `k` results sorted by score descending.
-    ///
-    /// Ephemeral results are boosted by a factor of 1.2 to surface recent
-    /// in-context information above older persisted data.
-    pub fn unified_search(
-        &mut self,
-        query_embedding: &[f32],
-        query_text: &str,
-        k: usize,
-    ) -> Vec<SearchResult> {
-        // Persistent tier.
-        let persistent = self.hybrid_search(query_embedding, query_text, 0.7, 0.3, k);
-        const EPHEMERAL_BOOST: f32 = 1.2;
-        let mut results = persistent;
-
-        if self.ephemeral.is_none() {
-            return results;
-        }
-
-        let eph = self.ephemeral.as_mut().unwrap();
-
-        // Collect (key, score) pairs from ephemeral — borrow ends before we
-        // access entries again below.
-        let eph_hits: Vec<(String, f32)> = if !query_embedding.is_empty() {
-            eph.search_embedding(query_embedding, k)
-        } else if !query_text.is_empty() {
-            eph.search_text(query_text, k)
-        } else {
-            Vec::new()
-        };
-
-        for (key, score) in &eph_hits {
-            if let Some(entry) = eph.get_entry(key) {
-                let chunk = entry
-                    .text
-                    .clone()
-                    .unwrap_or_else(|| String::from_utf8_lossy(&entry.value).into_owned());
-                results.push(SearchResult {
-                    score: score * EPHEMERAL_BOOST,
-                    chunk,
-                    index: usize::MAX,
-                    timestamp: entry.created_at,
-                    source_channel: format!("ephemeral::{key}"),
-                    activation: 1.0,
-                });
-            }
-        }
-
-        results.sort_by(|a, b| {
-            b.score
-                .partial_cmp(&a.score)
-                .unwrap_or(std::cmp::Ordering::Equal)
-        });
-        results.truncate(k);
-        results
-    }
-}

crates/clawhdf5-agent/src/openclaw.rs

@@ -748,6 +748,69 @@ impl MemoryBackend for ClawhdfBackend {
     }
 }
 
+// ─────────────────────────────────────────────────────────────────────────────
+// Ephemeral tier methods on ClawhdfBackend
+// ─────────────────────────────────────────────────────────────────────────────
+
+impl ClawhdfBackend {
+    /// Enable the ephemeral (in-memory only) working memory tier.
+    pub fn enable_ephemeral(&mut self, config: crate::ephemeral::EphemeralConfig) {
+        self.memory.enable_ephemeral(config);
+    }
+
+    /// Store a text value in ephemeral memory.
+    ///
+    /// Returns an error string if the ephemeral tier has not been enabled.
+    pub fn ephemeral_set(
+        &mut self,
+        key: &str,
+        value: &str,
+        ttl_secs: Option<f64>,
+    ) -> Result<(), String> {
+        match self.memory.ephemeral_mut() {
+            Some(s) => {
+                s.set_text(key, value, ttl_secs);
+                Ok(())
+            }
+            None => Err("ephemeral tier not enabled".to_string()),
+        }
+    }
+
+    /// Retrieve a text value from ephemeral memory.
+    ///
+    /// Returns `None` if the tier is disabled, the key is absent, or the
+    /// entry has expired.
+    pub fn ephemeral_get(&mut self, key: &str) -> Option<String> {
+        self.memory
+            .ephemeral_mut()?
+            .get_text(key)
+            .map(|s| s.to_string())
+    }
+
+    /// Delete a key from ephemeral memory.
+    ///
+    /// Returns `true` if the key existed and was removed.
+    pub fn ephemeral_delete(&mut self, key: &str) -> bool {
+        self.memory.ephemeral_mut().is_some_and(|s| s.delete(key))
+    }
+
+    /// Return a snapshot of ephemeral tier statistics, or `None` if the tier
+    /// is not enabled.
+    pub fn ephemeral_stats(&self) -> Option<crate::ephemeral::EphemeralStats> {
+        self.memory.ephemeral().map(|s| s.stats())
+    }
+
+    /// Promote frequently-accessed ephemeral entries to persistent HDF5 storage.
+    ///
+    /// Entries with `access_count >= min_access_count` are moved from the
+    /// ephemeral store into the persistent cache.  Returns the count promoted.
+    pub fn promote_ephemeral(&mut self, min_access_count: u32) -> Result<usize, String> {
+        self.memory
+            .promote_ephemeral(min_access_count)
+            .map_err(|e| e.to_string())
+    }
+}
+
 // ─────────────────────────────────────────────────────────────────────────────
 // Tests
 // ─────────────────────────────────────────────────────────────────────────────
@@ -1333,66 +1396,3 @@ mod tests {
         assert!(out.starts_with("# Title"));
     }
 }
-
-// ─────────────────────────────────────────────────────────────────────────────
-// Ephemeral tier methods on ClawhdfBackend
-// ─────────────────────────────────────────────────────────────────────────────
-
-impl ClawhdfBackend {
-    /// Enable the ephemeral (in-memory only) working memory tier.
-    pub fn enable_ephemeral(&mut self, config: crate::ephemeral::EphemeralConfig) {
-        self.memory.enable_ephemeral(config);
-    }
-
-    /// Store a text value in ephemeral memory.
-    ///
-    /// Returns an error string if the ephemeral tier has not been enabled.
-    pub fn ephemeral_set(
-        &mut self,
-        key: &str,
-        value: &str,
-        ttl_secs: Option<f64>,
-    ) -> Result<(), String> {
-        match self.memory.ephemeral_mut() {
-            Some(s) => {
-                s.set_text(key, value, ttl_secs);
-                Ok(())
-            }
-            None => Err("ephemeral tier not enabled".to_string()),
-        }
-    }
-
-    /// Retrieve a text value from ephemeral memory.
-    ///
-    /// Returns `None` if the tier is disabled, the key is absent, or the
-    /// entry has expired.
-    pub fn ephemeral_get(&mut self, key: &str) -> Option<String> {
-        self.memory
-            .ephemeral_mut()?
-            .get_text(key)
-            .map(|s| s.to_string())
-    }
-
-    /// Delete a key from ephemeral memory.
-    ///
-    /// Returns `true` if the key existed and was removed.
-    pub fn ephemeral_delete(&mut self, key: &str) -> bool {
-        self.memory.ephemeral_mut().is_some_and(|s| s.delete(key))
-    }
-
-    /// Return a snapshot of ephemeral tier statistics, or `None` if the tier
-    /// is not enabled.
-    pub fn ephemeral_stats(&self) -> Option<crate::ephemeral::EphemeralStats> {
-        self.memory.ephemeral().map(|s| s.stats())
-    }
-
-    /// Promote frequently-accessed ephemeral entries to persistent HDF5 storage.
-    ///
-    /// Entries with `access_count >= min_access_count` are moved from the
-    /// ephemeral store into the persistent cache.  Returns the count promoted.
-    pub fn promote_ephemeral(&mut self, min_access_count: u32) -> Result<usize, String> {
-        self.memory
-            .promote_ephemeral(min_access_count)
-            .map_err(|e| e.to_string())
-    }
-}

crates/clawhdf5-agent/src/provenance.rs

@@ -105,6 +105,23 @@ impl ProvenanceStore {
         self.records.insert(provenance.record_id, provenance);
     }
 
+    /// Renumber records after the store was compacted. `index_map[old]` is
+    /// the record's new id, or `None` if it was removed. Without this, every
+    /// surviving record's hash ends up filed under some other record's id and
+    /// the next integrity check reports a bogus mismatch.
+    pub fn remap(&mut self, index_map: &[Option<usize>]) {
+        let old = std::mem::take(&mut self.records);
+        for (old_id, mut prov) in old {
+            let new_id = usize::try_from(old_id)
+                .ok()
+                .and_then(|i| index_map.get(i).copied().flatten());
+            if let Some(new_id) = new_id {
+                prov.record_id = new_id as u64;
+                self.records.insert(new_id as u64, prov);
+            }
+        }
+    }
+
     /// Retrieve by record ID.
     pub fn get(&self, record_id: u64) -> Option<&MemoryProvenance> {
         self.records.get(&record_id)

crates/clawhdf5-agent/src/schema.rs

@@ -12,16 +12,36 @@ use crate::MemoryError;
 use crate::cache::MemoryCache;
 use crate::knowledge::KnowledgeCache;
 use crate::session::SessionCache;
+use crate::wal::WalMark;
 
 pub const SCHEMA_VERSION: &str = "1.0";
 pub const ZEROCLAW_VERSION: &str = "0.8.0";
 
+/// `/meta` attributes holding the [`WalMark`] of the WAL prefix already folded
+/// into this file. Absent on files written before the mark existed, and when
+/// the checkpoint was taken with an empty WAL.
+const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
+const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
+
 /// Build a complete HDF5 file from the in-memory state.
 pub fn build_hdf5_file(
     config: &MemoryConfig,
     cache: &MemoryCache,
     sessions: &SessionCache,
     knowledge: &KnowledgeCache,
+) -> Result<Vec<u8>, MemoryError> {
+    build_hdf5_file_with_mark(config, cache, sessions, knowledge, None)
+}
+
+/// [`build_hdf5_file`], recording which WAL prefix this state already
+/// contains (see [`WalMark`]) so a crash before the WAL is truncated doesn't
+/// replay those entries a second time.
+pub fn build_hdf5_file_with_mark(
+    config: &MemoryConfig,
+    cache: &MemoryCache,
+    sessions: &SessionCache,
+    knowledge: &KnowledgeCache,
+    wal_applied: Option<WalMark>,
 ) -> Result<Vec<u8>, MemoryError> {
     let mut builder = clawhdf5::FileBuilder::new();
 
@@ -34,10 +54,35 @@ pub fn build_hdf5_file(
     meta.set_attr("embedding_dim", AttrValue::I64(config.embedding_dim as i64));
     meta.set_attr("chunk_size", AttrValue::I64(config.chunk_size as i64));
     meta.set_attr("overlap", AttrValue::I64(config.overlap as i64));
+    // Behavioural settings. These used to live only in memory, so reopening a
+    // store silently reset them to defaults — e.g. a compressed store was
+    // rewritten uncompressed by the first checkpoint after a reopen. Loaders
+    // treat each one as optional so older files keep opening.
+    meta.set_attr("float16", AttrValue::I64(config.float16.into()));
+    meta.set_attr("compression", AttrValue::I64(config.compression.into()));
+    meta.set_attr(
+        "compression_level",
+        AttrValue::I64(config.compression_level.into()),
+    );
+    meta.set_attr(
+        "compact_threshold",
+        AttrValue::F64(config.compact_threshold.into()),
+    );
+    meta.set_attr("hebbian_boost", AttrValue::F64(config.hebbian_boost.into()));
+    meta.set_attr("decay_factor", AttrValue::F64(config.decay_factor.into()));
+    meta.set_attr("wal_enabled", AttrValue::I64(config.wal_enabled.into()));
+    meta.set_attr(
+        "wal_max_entries",
+        AttrValue::I64(config.wal_max_entries as i64),
+    );
     meta.set_attr(
         "edgehdf5_version",
         AttrValue::String(ZEROCLAW_VERSION.into()),
     );
+    if let Some(mark) = wal_applied.filter(|m| m.len > 0) {
+        meta.set_attr(WAL_APPLIED_LEN_ATTR, AttrValue::I64(mark.len as i64));
+        meta.set_attr(WAL_APPLIED_CRC_ATTR, AttrValue::I64(i64::from(mark.crc)));
+    }
     // Need at least one dataset in the group for it to be a proper group
     meta.create_dataset("_marker").with_u8_data(&[1]).compact();
     let finished_meta = meta.finish();
@@ -83,16 +128,34 @@ fn build_memory_group(
             let rows_per_chunk = (target_chunk_bytes / (d * 4)).max(1).min(n);
             ds.with_chunks(&[rows_per_chunk, d]);
 
-            // Compression: Zstd for embeddings — faster than deflate at same ratio.
+            // Compression. Shuffle is applied automatically (auto-shuffle
-            // Shuffle is applied automatically (auto-shuffle pre-filter).
+            // pre-filter). Zstd is faster than deflate at the same ratio but
+            // pulls in libzstd, so it is opt-in via the `zstd` feature; the
+            // default build uses deflate, which is always available. (This
+            // used to call `with_zstd` unconditionally, so without the
+            // feature every checkpoint of a compressed store failed with
+            // "unsupported filter: 32015".) Both are standard HDF5 filters;
+            // reading a zstd-compressed store needs a zstd-enabled build.
             if config.compression {
+                #[cfg(feature = "zstd")]
+                {
                     let level = if config.compression_level > 0 {
                         config.compression_level.min(22)
                     } else {
-                    3 // Zstd level 3: fast + good ratio for f32 embeddings
+                        3 // fast + good ratio for f32 embeddings
                     };
                     ds.with_zstd(level);
                 }
+                #[cfg(not(feature = "zstd"))]
+                {
+                    let level = if config.compression_level > 0 {
+                        config.compression_level.min(9)
+                    } else {
+                        4
+                    };
+                    ds.with_deflate(level);
+                }
+            }
         }
 
         // Skip fill-value initialization — embeddings are fully written
@@ -309,6 +372,20 @@ fn write_string_dataset(
 }
 
 /// Validate an HDF5 file has the correct schema and load all data.
+/// Read the checkpoint's [`WalMark`] from `/meta`, if it has one.
+pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
+    let attrs = file.group("meta").ok()?.attrs().ok()?;
+    let len = match attrs.get(WAL_APPLIED_LEN_ATTR)? {
+        AttrValue::I64(v) => u64::try_from(*v).ok()?,
+        _ => return None,
+    };
+    let crc = match attrs.get(WAL_APPLIED_CRC_ATTR)? {
+        AttrValue::I64(v) => u32::try_from(*v).ok()?,
+        _ => return None,
+    };
+    Some(WalMark { len, crc })
+}
+
 pub fn validate_and_load(
     file: &clawhdf5::File,
 ) -> Result<(MemoryConfig, MemoryCache, SessionCache, KnowledgeCache), MemoryError> {
@@ -344,15 +421,19 @@ pub fn validate_and_load(
         embedding_dim,
         chunk_size,
         overlap,
-        float16: false,
+        float16: optional_bool_attr(&attrs, "float16", false),
-        compression: false,
+        compression: optional_bool_attr(&attrs, "compression", false),
-        compression_level: 0,
+        compression_level: optional_i64_attr(&attrs, "compression_level")
-        compact_threshold: 0.3,
+            .and_then(|v| u32::try_from(v).ok())
-        hebbian_boost: 0.15,
+            .unwrap_or(0),
-        decay_factor: 0.98,
+        compact_threshold: optional_f32_attr(&attrs, "compact_threshold", 0.3),
+        hebbian_boost: optional_f32_attr(&attrs, "hebbian_boost", 0.15),
+        decay_factor: optional_f32_attr(&attrs, "decay_factor", 0.98),
         created_at,
-        wal_enabled: true,
+        wal_enabled: optional_bool_attr(&attrs, "wal_enabled", true),
-        wal_max_entries: 500,
+        wal_max_entries: optional_i64_attr(&attrs, "wal_max_entries")
+            .and_then(|v| usize::try_from(v).ok())
+            .unwrap_or(500),
     };
 
     // Load /memory group
@@ -391,19 +472,45 @@ fn load_memory_group(
     let tags = read_string_dataset_from_group(&group, "tags")?;
     let tombstones = read_u8_dataset(&group, "tombstones")?;
 
-    // Read norms if present, otherwise compute from embeddings
+    // Every per-record dataset must describe exactly `n` records. Without
+    // this, a truncated or hand-edited file loads "successfully" and then
+    // panics on the first out-of-bounds index during search/delete.
+    if embedding_dim == 0 {
+        return Err(MemoryError::Schema(format!(
+            "/memory has {n} records but embedding_dim is 0"
+        )));
+    }
+    let expected_flat = n.checked_mul(embedding_dim).ok_or_else(|| {
+        MemoryError::Schema(format!("/memory size overflow: {n} x {embedding_dim}"))
+    })?;
+    let check_len = |name: &str, actual: usize, expected: usize| {
+        if actual == expected {
+            Ok(())
+        } else {
+            Err(MemoryError::Schema(format!(
+                "/memory/{name} has {actual} entries, expected {expected} \
+                 ({n} records)"
+            )))
+        }
+    };
+    check_len("embeddings", flat_embeddings.len(), expected_flat)?;
+    check_len("source_channel", source_channels.len(), n)?;
+    check_len("timestamps", timestamps.len(), n)?;
+    check_len("session_ids", session_ids.len(), n)?;
+    check_len("tags", tags.len(), n)?;
+    check_len("tombstones", tombstones.len(), n)?;
+
+    // Norms are derived data: use the stored ones only if they are present
+    // and the right length, otherwise recompute from the embeddings.
     let norms = match read_f32_dataset(&group, "norms") {
-        Ok(n) if n.len() == n.len() => n,
+        Ok(stored) if stored.len() == n => stored,
-        _ => {
+        _ => flat_embeddings
-            // Compute norms from flat embeddings
-            flat_embeddings
             .chunks(embedding_dim)
             .map(|chunk| {
                 let sq_sum: f32 = chunk.iter().map(|x| x * x).sum();
                 sq_sum.sqrt()
             })
-                .collect()
+            .collect(),
-        }
     };
 
     // Unflatten embeddings
@@ -531,6 +638,27 @@ fn extract_string_attr(
     }
 }
 
+type MetaAttrs = std::collections::HashMap<String, AttrValue>;
+
+fn optional_i64_attr(attrs: &MetaAttrs, name: &str) -> Option<i64> {
+    match attrs.get(name) {
+        Some(AttrValue::I64(v)) => Some(*v),
+        _ => None,
+    }
+}
+
+fn optional_bool_attr(attrs: &MetaAttrs, name: &str, default: bool) -> bool {
+    optional_i64_attr(attrs, name).map_or(default, |v| v != 0)
+}
+
+/// Finite values only: a NaN threshold/decay would poison every comparison.
+fn optional_f32_attr(attrs: &MetaAttrs, name: &str, default: f32) -> f32 {
+    match attrs.get(name) {
+        Some(AttrValue::F64(v)) if v.is_finite() => *v as f32,
+        _ => default,
+    }
+}
+
 fn extract_i64_attr(
     attrs: &std::collections::HashMap<String, AttrValue>,
     name: &str,
@@ -616,3 +744,108 @@ fn read_u8_dataset(group: &clawhdf5::Group<'_>, name: &str) -> Result<Vec<u8>, M
         .map_err(|e| MemoryError::Hdf5(format!("cannot read u8 from {name}: {e}")))?;
     Ok(data.into_iter().map(|v| v as u8).collect())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn config() -> MemoryConfig {
+        MemoryConfig::new(std::path::PathBuf::from("unused.h5"), "agent", 4)
+    }
+
+    fn cache_with(n: usize) -> MemoryCache {
+        let mut cache = MemoryCache::new(4);
+        for i in 0..n {
+            cache.push(
+                format!("chunk {i}"),
+                vec![i as f32 + 1.0, 0.0, 0.0, 0.0],
+                "user".into(),
+                i as f64,
+                "s".into(),
+                "t".into(),
+            );
+        }
+        cache
+    }
+
+    fn roundtrip(cache: &MemoryCache) -> Result<MemoryCache, MemoryError> {
+        let bytes = build_hdf5_file(
+            &config(),
+            cache,
+            &SessionCache::new(),
+            &KnowledgeCache::new(),
+        )?;
+        let file =
+            clawhdf5::File::from_bytes(bytes).map_err(|e| MemoryError::Hdf5(e.to_string()))?;
+        validate_and_load(&file).map(|(_, cache, _, _)| cache)
+    }
+
+    #[test]
+    fn behavioural_config_survives_a_reopen() {
+        let mut cfg = config();
+        cfg.compression = true;
+        cfg.compression_level = 7;
+        cfg.compact_threshold = 0.5;
+        cfg.hebbian_boost = 0.25;
+        cfg.decay_factor = 0.9;
+        cfg.wal_enabled = false;
+        cfg.wal_max_entries = 42;
+        let bytes = build_hdf5_file(
+            &cfg,
+            &cache_with(2),
+            &SessionCache::new(),
+            &KnowledgeCache::new(),
+        )
+        .unwrap();
+        let file = clawhdf5::File::from_bytes(bytes).unwrap();
+        let (loaded, loaded_cache, ..) = validate_and_load(&file).unwrap();
+        // The compressed embeddings must also read back intact.
+        assert_eq!(loaded_cache.embeddings, cache_with(2).embeddings);
+        assert!(loaded.compression);
+        assert_eq!(loaded.compression_level, 7);
+        assert_eq!(loaded.compact_threshold, 0.5);
+        assert_eq!(loaded.hebbian_boost, 0.25);
+        assert_eq!(loaded.decay_factor, 0.9);
+        assert!(!loaded.wal_enabled);
+        assert_eq!(loaded.wal_max_entries, 42);
+    }
+
+    #[test]
+    fn consistent_store_loads() {
+        let loaded = roundtrip(&cache_with(3)).unwrap();
+        assert_eq!(loaded.chunks.len(), 3);
+        assert_eq!(loaded.norms, vec![1.0, 2.0, 3.0]);
+    }
+
+    #[test]
+    fn wrong_length_norms_are_recomputed_not_trusted() {
+        // Regression: the guard used to be `n.len() == n.len()`, so a norms
+        // dataset of any length was accepted and corrupted every cosine score.
+        let mut cache = cache_with(3);
+        cache.norms = vec![99.0];
+        let loaded = roundtrip(&cache).unwrap();
+        assert_eq!(loaded.norms, vec![1.0, 2.0, 3.0]);
+    }
+
+    #[test]
+    fn mismatched_per_record_datasets_are_schema_errors() {
+        type Corrupt = fn(&mut MemoryCache);
+        let cases: [(&str, Corrupt); 5] = [
+            ("tombstones", |c| c.tombstones.truncate(1)),
+            ("timestamps", |c| c.timestamps.truncate(1)),
+            ("tags", |c| c.tags.truncate(1)),
+            ("session_ids", |c| c.session_ids.truncate(1)),
+            ("source_channel", |c| c.source_channels.truncate(1)),
+        ];
+        for (name, corrupt) in cases {
+            let mut cache = cache_with(3);
+            corrupt(&mut cache);
+            match roundtrip(&cache) {
+                Err(MemoryError::Schema(msg)) => {
+                    assert!(msg.contains(name), "{name}: unexpected message {msg}")
+                }
+                other => panic!("{name}: expected Schema error, got {:?}", other.map(|_| ())),
+            }
+        }
+    }
+}

crates/clawhdf5-agent/src/search.rs

@@ -113,13 +113,24 @@ impl HDF5Memory {
                 }
             })
             .collect();
+        // Ties broken by index so results (and therefore which records get
+        // boosted) don't depend on HashMap iteration order upstream.
         results.sort_by(|a, b| {
             b.score
                 .partial_cmp(&a.score)
                 .unwrap_or(std::cmp::Ordering::Equal)
+                .then(a.index.cmp(&b.index))
         });
 
-        let hit_indices: Vec<usize> = results.iter().map(|r| r.index).collect();
+        // Only reinforce records that actually matched. When fewer than `k`
+        // records are relevant, the rest of the list is zero-score filler;
+        // boosting it would teach the store that arbitrary records are
+        // important just because they were nearby in iteration order.
+        let hit_indices: Vec<usize> = results
+            .iter()
+            .filter(|r| r.score > 0.0)
+            .map(|r| r.index)
+            .collect();
         self.apply_hebbian_boost(&hit_indices);
         self.flush().ok();
 

crates/clawhdf5-agent/src/storage.rs

@@ -11,6 +11,7 @@ use crate::cache::MemoryCache;
 use crate::knowledge::KnowledgeCache;
 use crate::schema;
 use crate::session::SessionCache;
+use crate::wal::WalMark;
 
 /// Write all in-memory state to an HDF5 file on disk.
 pub fn write_to_disk(
@@ -20,7 +21,20 @@ pub fn write_to_disk(
     sessions: &SessionCache,
     knowledge: &KnowledgeCache,
 ) -> Result<(), MemoryError> {
-    let bytes = schema::build_hdf5_file(config, cache, sessions, knowledge)?;
+    write_to_disk_with_mark(path, config, cache, sessions, knowledge, None)
+}
+
+/// [`write_to_disk`] for a checkpoint: `wal_applied` is the mark of the WAL
+/// prefix whose entries `cache` already contains.
+pub fn write_to_disk_with_mark(
+    path: &Path,
+    config: &MemoryConfig,
+    cache: &MemoryCache,
+    sessions: &SessionCache,
+    knowledge: &KnowledgeCache,
+    wal_applied: Option<WalMark>,
+) -> Result<(), MemoryError> {
+    let bytes = schema::build_hdf5_file_with_mark(config, cache, sessions, knowledge, wal_applied)?;
 
     if bytes.is_empty() {
         return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into()));
@@ -28,9 +42,41 @@ pub fn write_to_disk(
 
     // Write to a temp file first, then rename for atomicity
     let tmp_path = path.with_extension("h5.tmp");
-    std::fs::write(&tmp_path, &bytes).map_err(MemoryError::Io)?;
+    write_synced(&tmp_path, &bytes)?;
-    std::fs::rename(&tmp_path, path).map_err(MemoryError::Io)?;
+    rename_synced(&tmp_path, path)
+}
 
+/// Write `bytes` to `path` and flush them to stable storage.
+fn write_synced(path: &Path, bytes: &[u8]) -> Result<(), MemoryError> {
+    use std::io::Write;
+    let mut f = std::fs::File::create(path).map_err(MemoryError::Io)?;
+    f.write_all(bytes).map_err(MemoryError::Io)?;
+    f.sync_all().map_err(MemoryError::Io)
+}
+
+/// Rename `from` over `to`, then sync the parent directory so the rename
+/// itself survives a power loss. `from` must already be synced: without that,
+/// the rename can reach disk before the data and leave an empty or partial
+/// file under the final name.
+///
+/// This is per-checkpoint/snapshot cost only (each is already a full file
+/// write). Individual WAL appends are deliberately not synced — see the
+/// durability notes in the crate docs.
+fn rename_synced(from: &Path, to: &Path) -> Result<(), MemoryError> {
+    std::fs::rename(from, to).map_err(MemoryError::Io)?;
+    #[cfg(unix)]
+    if let Some(dir) = to.parent() {
+        let dir = if dir.as_os_str().is_empty() {
+            Path::new(".")
+        } else {
+            dir
+        };
+        // Directory fsync is best-effort: some filesystems refuse it, and the
+        // rename has already happened.
+        if let Ok(d) = std::fs::File::open(dir) {
+            let _ = d.sync_all();
+        }
+    }
     Ok(())
 }
 
@@ -42,6 +88,15 @@ pub fn write_to_disk(
 pub fn read_from_disk(
     path: &Path,
 ) -> Result<(MemoryConfig, MemoryCache, SessionCache, KnowledgeCache), MemoryError> {
+    read_from_disk_with_mark(path).map(|(state, _mark)| state)
+}
+
+/// Everything [`read_from_disk`] returns.
+pub type StoreState = (MemoryConfig, MemoryCache, SessionCache, KnowledgeCache);
+
+/// [`read_from_disk`], plus the checkpoint's [`WalMark`] (if any) so the
+/// caller can skip WAL entries this file already contains.
+pub fn read_from_disk_with_mark(path: &Path) -> Result<(StoreState, Option<WalMark>), MemoryError> {
     let mmap = clawhdf5_io::MmapReader::open(path).map_err(MemoryError::Io)?;
 
     // Advise the OS we'll need the whole file for parsing
@@ -53,8 +108,9 @@ pub fn read_from_disk(
 
     let (mut config, cache, sessions, knowledge) = schema::validate_and_load(&file)?;
     config.path = path.to_path_buf();
+    let wal_applied = schema::read_wal_mark(&file);
 
-    Ok((config, cache, sessions, knowledge))
+    Ok(((config, cache, sessions, knowledge), wal_applied))
 }
 
 /// Copy an HDF5 file atomically to a destination.
@@ -78,7 +134,10 @@ pub fn snapshot_file(src: &Path, dest: &Path) -> Result<std::path::PathBuf, Memo
     // Atomic copy: write to temp, then rename
     let tmp_path = dest_file.with_extension("h5.tmp");
     std::fs::copy(src, &tmp_path).map_err(MemoryError::Io)?;
-    std::fs::rename(&tmp_path, &dest_file).map_err(MemoryError::Io)?;
+    std::fs::File::open(&tmp_path)
+        .and_then(|f| f.sync_all())
+        .map_err(MemoryError::Io)?;
+    rename_synced(&tmp_path, &dest_file)?;
 
     Ok(dest_file)
 }

crates/clawhdf5-agent/src/store_lock.rs

@@ -0,0 +1,79 @@
+//! Single-writer guard for a memory store.
+//!
+//! `HDF5Memory` keeps the whole store in memory and rewrites the `.h5` file at
+//! every checkpoint, so two handles on one store (two processes, or two opens
+//! in one process) silently destroy each other's data: whoever checkpoints
+//! last wins, and both append to the same WAL with independent CRC chains.
+//! The lock turns that into an immediate, explicit error.
+
+use std::fs::{File, OpenOptions, TryLockError};
+use std::path::{Path, PathBuf};
+
+use crate::MemoryError;
+
+const LOCK_RETRIES: u32 = 25;
+const LOCK_RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(10);
+
+/// An exclusive advisory lock on `<store>.h5.lock`, held for the lifetime of
+/// the owning `HDF5Memory` and released when it is dropped (or when the
+/// process dies — the OS drops the lock with the file descriptor, so a crash
+/// never leaves a stale lock behind; the empty lock file itself is harmless).
+#[derive(Debug)]
+pub(crate) struct StoreLock {
+    _file: File,
+}
+
+impl StoreLock {
+    pub(crate) fn lock_path(store: &Path) -> PathBuf {
+        store.with_extension("h5.lock")
+    }
+
+    pub(crate) fn acquire(store: &Path) -> Result<Self, MemoryError> {
+        let path = Self::lock_path(store);
+        let file = OpenOptions::new()
+            .create(true)
+            .truncate(false)
+            .write(true)
+            .open(&path)?;
+        // A previous owner may be mid-teardown (e.g. an `AsyncHDF5Memory`
+        // dropped without `shutdown()`: its background task releases the
+        // store a moment later), so give the lock a short, bounded grace
+        // period before reporting a genuine second writer.
+        let mut attempts_left = LOCK_RETRIES;
+        loop {
+            match file.try_lock() {
+                Ok(()) => return Ok(Self { _file: file }),
+                Err(TryLockError::WouldBlock) if attempts_left > 0 => {
+                    attempts_left -= 1;
+                    std::thread::sleep(LOCK_RETRY_DELAY);
+                }
+                Err(TryLockError::WouldBlock) => {
+                    return Err(MemoryError::Locked(format!(
+                        "{} is already open in this or another process (lock file {})",
+                        store.display(),
+                        path.display()
+                    )));
+                }
+                Err(TryLockError::Error(e)) => return Err(MemoryError::Io(e)),
+            }
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn second_acquire_fails_until_first_is_dropped() {
+        let dir = tempfile::TempDir::new().unwrap();
+        let store = dir.path().join("s.h5");
+        let first = StoreLock::acquire(&store).unwrap();
+        assert!(matches!(
+            StoreLock::acquire(&store),
+            Err(MemoryError::Locked(_))
+        ));
+        drop(first);
+        StoreLock::acquire(&store).unwrap();
+    }
+}

crates/clawhdf5-agent/src/wal.rs

@@ -28,7 +28,18 @@ const WAL_HEADER_LEN: u64 = WAL_MAGIC.len() as u64 + 1 + 4;
 /// its target Save) — the moved/inserted entry's stored CRC was computed
 /// against a different predecessor than the one now in front of it on disk,
 /// so the chain breaks at that point and replay stops there.
-const WAL_VERSION: u8 = 3;
+const WAL_VERSION: u8 = 4;
+
+/// The chained-CRC format before [`WalEntryType::Update`] records existed.
+/// Byte-for-byte the same framing as [`WAL_VERSION`], so it is read by the
+/// same code, and `WalFile::open` upgrades it in place by rewriting the
+/// header's version byte (the header is not covered by the CRC chain).
+///
+/// The bump exists for *older binaries*: they don't know record type 0x04,
+/// would treat it as a torn tail, and would truncate it — and everything
+/// after it — away. An unknown header version makes them refuse the file
+/// with a clear error instead.
+const WAL_VERSION_CHAINED_NO_UPDATE: u8 = 3;
 
 /// The previous WAL format version: still a CRC32 per entry (so a bit-flip
 /// within one entry is caught), but not chained to the previous entry's CRC
@@ -67,6 +78,10 @@ pub enum WalEntryType {
     Save = 0x01,
     Tombstone = 0x02,
     ActivationUpdate = 0x03,
+    /// Replace the record at `update_index` in place (`save_or_update` hit).
+    /// Logged as a plain `Save` before this existed, so replay appended a
+    /// duplicate instead of updating.
+    Update = 0x04,
 }
 
 impl WalEntryType {
@@ -75,6 +90,7 @@ impl WalEntryType {
             0x01 => Some(Self::Save),
             0x02 => Some(Self::Tombstone),
             0x03 => Some(Self::ActivationUpdate),
+            0x04 => Some(Self::Update),
             _ => None,
         }
     }
@@ -91,6 +107,8 @@ pub struct WalEntry {
     pub tags: String,
     /// For tombstone entries: the index of the entry to delete.
     pub tombstone_index: Option<usize>,
+    /// For update entries: the index of the record to replace.
+    pub update_index: Option<usize>,
 }
 
 /// How many entries to accumulate before updating the header entry_count.
@@ -112,6 +130,62 @@ pub struct WalFile {
     /// Reset to 0 by `truncate()`/`create_fresh_wal_file`, and re-derived by
     /// scanning existing entries when `open()` attaches to a non-empty file.
     running_crc: u32,
+    /// Bytes of verified entries after the header (the length of the chain
+    /// `running_crc` covers). Together they form the [`WalMark`].
+    chain_len: u64,
+}
+
+/// What a WAL file's 9-byte header looks like, without reading any entries.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum WalHeaderStatus {
+    /// A version this build can read (current or legacy).
+    Readable,
+    /// Shorter than a header — e.g. a crash while the file was being created.
+    /// It cannot contain entries.
+    Torn,
+    /// Not a WAL file at all.
+    BadMagic,
+    /// Well-formed header from a version this build doesn't know — most
+    /// likely written by a *newer* build. Never discard this: the entries are
+    /// probably fine, this binary just can't read them.
+    UnknownVersion(u8),
+}
+
+/// Classify the header of the WAL at `path`.
+pub fn wal_header_status(path: &Path) -> std::io::Result<WalHeaderStatus> {
+    let mut header = [0u8; WAL_HEADER_LEN as usize];
+    let mut f = File::open(path)?;
+    let mut filled = 0;
+    while filled < header.len() {
+        match f.read(&mut header[filled..])? {
+            0 => return Ok(WalHeaderStatus::Torn),
+            n => filled += n,
+        }
+    }
+    if header[0..4] != WAL_MAGIC {
+        return Ok(WalHeaderStatus::BadMagic);
+    }
+    Ok(match header[4] {
+        WAL_VERSION
+        | WAL_VERSION_CHAINED_NO_UPDATE
+        | WAL_VERSION_CRC_UNCHAINED
+        | WAL_VERSION_LEGACY_NO_CRC => WalHeaderStatus::Readable,
+        v => WalHeaderStatus::UnknownVersion(v),
+    })
+}
+
+/// A position in a WAL's CRC chain: `len` bytes of entries after the header,
+/// whose chained CRC is `crc`.
+///
+/// A checkpoint stores the mark of the WAL prefix it folded into the `.h5`
+/// file. If the process dies after the new `.h5` is in place but before the
+/// WAL is truncated, the next `open()` finds that exact prefix still in the
+/// WAL and skips it instead of replaying it on top of data that already
+/// contains it (which used to duplicate every pending entry).
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct WalMark {
+    pub len: u64,
+    pub crc: u32,
 }
 
 impl WalFile {
@@ -138,7 +212,15 @@ impl WalFile {
             let mut ver = [0u8; 1];
             f.read_exact(&mut ver)?;
             match ver[0] {
-                WAL_VERSION => {
+                WAL_VERSION | WAL_VERSION_CHAINED_NO_UPDATE => {
+                    if ver[0] == WAL_VERSION_CHAINED_NO_UPDATE {
+                        // Same framing; stamp the current version so an older
+                        // binary refuses this file rather than truncating an
+                        // Update record it can't parse. See the constant.
+                        f.seek(SeekFrom::Start(4))?;
+                        f.write_all(&[WAL_VERSION])?;
+                        f.seek(SeekFrom::Start(5))?;
+                    }
                     let mut count_buf = [0u8; 4];
                     f.read_exact(&mut count_buf)?;
                     let header_count = u32::from_le_bytes(count_buf);
@@ -147,7 +229,8 @@ impl WalFile {
                     // be stale from deferred group-commit sync, same
                     // tolerance `read_entries` already has, so the scanned
                     // count is also the more accurate of the two).
-                    let (entries, running_crc, verified_bytes) = read_chained_entries(&mut f, 0);
+                    let (entries, running_crc, verified_bytes) =
+                        read_chained_entries(&mut f, 0, None);
                     let entry_count = if entries.is_empty() {
                         header_count
                     } else {
@@ -190,6 +273,7 @@ impl WalFile {
                         entry_count,
                         pending_header_sync: 0,
                         running_crc,
+                        chain_len: verified_bytes,
                     })
                 }
                 WAL_VERSION_CRC_UNCHAINED | WAL_VERSION_LEGACY_NO_CRC => {
@@ -201,6 +285,7 @@ impl WalFile {
                         entry_count: 0,
                         pending_header_sync: 0,
                         running_crc: 0,
+                        chain_len: 0,
                     })
                 }
                 v => Err(MemoryError::Schema(format!("unsupported WAL version {v}"))),
@@ -213,6 +298,7 @@ impl WalFile {
                 entry_count: 0,
                 pending_header_sync: 0,
                 running_crc: 0,
+                chain_len: 0,
             })
         }
     }
@@ -236,8 +322,20 @@ impl WalFile {
             4 + entry.session_id.len() +
             4 + entry.tags.len(),
         );
+        match entry.update_index {
+            Some(index) => {
+                let index = u32::try_from(index).map_err(|_| {
+                    MemoryError::Schema(format!("WAL update index {index} exceeds u32"))
+                })?;
+                buf.push(WalEntryType::Update as u8);
+                buf.extend_from_slice(&entry.timestamp.to_le_bytes());
+                buf.extend_from_slice(&index.to_le_bytes());
+            }
+            None => {
                 buf.push(WalEntryType::Save as u8);
                 buf.extend_from_slice(&entry.timestamp.to_le_bytes());
+            }
+        }
         serialize_str(&mut buf, &entry.chunk);
         buf.extend_from_slice(&(emb_len as u32).to_le_bytes());
         for &val in &entry.embedding {
@@ -258,6 +356,7 @@ impl WalFile {
             .as_mut()
             .ok_or_else(|| MemoryError::Io(std::io::Error::other("WAL file not open")))?;
         f.write_all(&buf)?;
+        self.chain_len += buf.len() as u64;
 
         self.running_crc = crc;
         self.entry_count += 1;
@@ -282,6 +381,7 @@ impl WalFile {
             .as_mut()
             .ok_or_else(|| MemoryError::Io(std::io::Error::other("WAL file not open")))?;
         f.write_all(&buf)?;
+        self.chain_len += buf.len() as u64;
 
         self.running_crc = crc;
         self.entry_count += 1;
@@ -311,7 +411,7 @@ impl WalFile {
     /// legacy-no-CRC file returns a typed error instead of silently
     /// downgrading to the unverified parser.
     pub fn read_entries(path: &Path) -> Result<Vec<WalEntry>, MemoryError> {
-        Self::read_entries_impl(path, false)
+        Self::read_entries_impl(path, false, None)
     }
 
     /// Like [`WalFile::read_entries`], but also accepts
@@ -320,13 +420,23 @@ impl WalFile {
     /// legitimate caller is `HDF5Memory::open`'s one-time migration of a
     /// pre-CRC WAL file, which immediately recreates it in the current
     /// format afterward. Do not use this for anything else.
-    pub(crate) fn read_entries_for_migration(path: &Path) -> Result<Vec<WalEntry>, MemoryError> {
+    ///
-        Self::read_entries_impl(path, true)
+    /// `applied` is the checkpoint mark read from the `.h5` file, if any: if
+    /// the WAL's chain passes through it (same byte length, same chained
+    /// CRC), everything up to that point is already in the `.h5` and is
+    /// dropped. If it never does — the normal case, because the WAL was
+    /// truncated after the checkpoint — every entry is returned.
+    pub(crate) fn read_entries_for_migration(
+        path: &Path,
+        applied: Option<WalMark>,
+    ) -> Result<Vec<WalEntry>, MemoryError> {
+        Self::read_entries_impl(path, true, applied)
     }
 
     fn read_entries_impl(
         path: &Path,
         allow_legacy_no_crc: bool,
+        applied: Option<WalMark>,
     ) -> Result<Vec<WalEntry>, MemoryError> {
         if !path.exists() {
             return Ok(Vec::new());
@@ -342,8 +452,9 @@ impl WalFile {
         let entry_count_hint = u32::from_le_bytes([header[5], header[6], header[7], header[8]]);
 
         match header[4] {
-            WAL_VERSION => {
+            WAL_VERSION | WAL_VERSION_CHAINED_NO_UPDATE => {
-                let (entries, _final_crc, _verified_bytes) = read_chained_entries(&mut f, 0);
+                let (entries, _final_crc, _verified_bytes) =
+                    read_chained_entries(&mut f, 0, applied);
                 Ok(entries)
             }
             WAL_VERSION_CRC_UNCHAINED => {
@@ -406,9 +517,19 @@ impl WalFile {
         self.entry_count = 0;
         self.pending_header_sync = 0;
         self.running_crc = 0;
+        self.chain_len = 0;
         Ok(())
     }
 
+    /// The mark covering every entry currently in this WAL. Store it with a
+    /// checkpoint taken from the state those entries produced.
+    pub fn mark(&self) -> WalMark {
+        WalMark {
+            len: self.chain_len,
+            crc: self.running_crc,
+        }
+    }
+
     /// Number of pending entries.
     pub fn pending_count(&self) -> u32 {
         self.entry_count
@@ -448,6 +569,28 @@ pub fn replay_into_cache(entries: &[WalEntry], cache: &mut crate::cache::MemoryC
                     entry.tags.clone(),
                 );
             }
+            WalEntryType::Update => match entry.update_index {
+                // The index was valid when the record was written; if the
+                // store no longer has it, keep the data rather than drop it.
+                Some(idx) if idx < cache.len() => cache.update(
+                    idx,
+                    entry.chunk.clone(),
+                    entry.embedding.clone(),
+                    entry.source_channel.clone(),
+                    entry.timestamp,
+                    entry.session_id.clone(),
+                ),
+                _ => {
+                    cache.push(
+                        entry.chunk.clone(),
+                        entry.embedding.clone(),
+                        entry.source_channel.clone(),
+                        entry.timestamp,
+                        entry.session_id.clone(),
+                        entry.tags.clone(),
+                    );
+                }
+            },
             WalEntryType::Tombstone => {
                 if let Some(idx) = entry.tombstone_index {
                     cache.mark_deleted(idx);
@@ -524,7 +667,16 @@ fn chained_crc(entry_bytes: &[u8], prev_crc: u32) -> u32 {
 /// The byte count is what lets `open()` position an append at the end of the
 /// VERIFIED prefix rather than at end-of-file. Appending past a torn tail
 /// writes entries that replay can never reach — see `open`.
-fn read_chained_entries<R: Read>(f: &mut R, start_crc: u32) -> (Vec<WalEntry>, u32, u64) {
+///
+/// `applied`, when given, is a checkpoint mark: once the chain reaches exactly
+/// that position, the entries collected so far are discarded (they are
+/// already in the `.h5` file). A zero-length mark matches nothing.
+fn read_chained_entries<R: Read>(
+    f: &mut R,
+    start_crc: u32,
+    applied: Option<WalMark>,
+) -> (Vec<WalEntry>, u32, u64) {
+    let applied = applied.filter(|m| m.len > 0);
     let mut entries = Vec::new();
     let mut running_crc = start_crc;
     let mut verified_bytes: u64 = 0;
@@ -554,6 +706,14 @@ fn read_chained_entries<R: Read>(f: &mut R, start_crc: u32) -> (Vec<WalEntry>, u
         if let Some(entry) = entry_opt {
             entries.push(entry);
         }
+        if applied
+            == Some(WalMark {
+                len: verified_bytes,
+                crc: running_crc,
+            })
+        {
+            entries.clear();
+        }
     }
     (entries, running_crc, verified_bytes)
 }
@@ -615,7 +775,14 @@ fn read_one_entry<R: Read>(r: &mut R) -> Result<Option<WalEntry>, ()> {
     let timestamp = f64::from_le_bytes(ts_buf);
 
     match entry_type {
-        WalEntryType::Save => {
+        WalEntryType::Save | WalEntryType::Update => {
+            let update_index = if entry_type == WalEntryType::Update {
+                let mut idx_buf = [0u8; 4];
+                r.read_exact(&mut idx_buf).map_err(|_| ())?;
+                Some(u32::from_le_bytes(idx_buf) as usize)
+            } else {
+                None
+            };
             let chunk = read_len_prefixed_str(r).map_err(|_| ())?;
             let embedding = read_embedding(r).map_err(|_| ())?;
             let source_channel = read_len_prefixed_str(r).map_err(|_| ())?;
@@ -630,6 +797,7 @@ fn read_one_entry<R: Read>(r: &mut R) -> Result<Option<WalEntry>, ()> {
                 session_id,
                 tags,
                 tombstone_index: None,
+                update_index,
             }))
         }
         WalEntryType::Tombstone => {
@@ -645,6 +813,7 @@ fn read_one_entry<R: Read>(r: &mut R) -> Result<Option<WalEntry>, ()> {
                 session_id: String::new(),
                 tags: String::new(),
                 tombstone_index: Some(idx),
+                update_index: None,
             }))
         }
         WalEntryType::ActivationUpdate => Ok(None),
@@ -668,6 +837,7 @@ mod tests {
             session_id: "sess-001".to_string(),
             tags: "tag1,tag2".to_string(),
             tombstone_index: None,
+            update_index: None,
         }
     }
 
@@ -786,7 +956,7 @@ mod tests {
         let dir = TempDir::new().unwrap();
         let wal_path = dir.path().join("test.h5.wal");
         let unicode_chunk = "Hello 世界! 🌍 émojis & ünïcödé";
-        let embedding = vec![0.1, -0.2, 3.14159, f32::MAX, f32::MIN_POSITIVE];
+        let embedding = vec![0.1, -0.2, 3.4567, f32::MAX, f32::MIN_POSITIVE];
         {
             let mut wal = WalFile::open(&wal_path).unwrap();
             let entry = WalEntry {
@@ -798,6 +968,7 @@ mod tests {
                 session_id: "sess-öö-123".to_string(),
                 tags: "α,β,γ".to_string(),
                 tombstone_index: None,
+                update_index: None,
             };
             wal.append_save(&entry).unwrap();
         }
@@ -932,6 +1103,148 @@ mod tests {
         assert!(entries.is_empty());
     }
 
+    /// Reopen `path` and return the stored chunks in order.
+    fn reopen_chunks(path: &std::path::Path) -> Vec<String> {
+        let mem = HDF5Memory::open(path).unwrap();
+        mem.cache.chunks.clone()
+    }
+
+    #[test]
+    fn crash_between_checkpoint_and_wal_truncate_does_not_duplicate() {
+        // flush() writes the new .h5 and only then truncates the WAL. Dying in
+        // between leaves BOTH a .h5 that contains the pending entries and a
+        // WAL that still lists them; replaying blindly used to double them.
+        let dir = TempDir::new().unwrap();
+        let config = make_config(&dir);
+        let h5_path = config.path.clone();
+        let wal_path = h5_path.with_extension("h5.wal");
+        let stale_wal = dir.path().join("stale.wal");
+
+        {
+            let mut mem = HDF5Memory::create(config).unwrap();
+            for name in ["a", "b", "c"] {
+                mem.save(make_entry(name, &[1.0, 0.0, 0.0, 0.0])).unwrap();
+            }
+            assert_eq!(mem.wal_pending_count(), 3);
+            std::fs::copy(&wal_path, &stale_wal).unwrap();
+            mem.flush_wal().unwrap();
+        }
+        // Undo the truncate: this is the on-disk state right after the crash.
+        std::fs::copy(&stale_wal, &wal_path).unwrap();
+        assert_eq!(WalFile::read_entries(&wal_path).unwrap().len(), 3);
+
+        assert_eq!(reopen_chunks(&h5_path), ["a", "b", "c"]);
+
+        // Entries appended to that same WAL after recovery are still replayed.
+        {
+            let mut mem = HDF5Memory::open(&h5_path).unwrap();
+            mem.save(make_entry("d", &[0.0, 1.0, 0.0, 0.0])).unwrap();
+        }
+        assert_eq!(reopen_chunks(&h5_path), ["a", "b", "c", "d"]);
+    }
+
+    #[test]
+    fn entries_written_after_a_completed_checkpoint_are_all_replayed() {
+        // Normal case: the checkpoint's mark refers to a WAL that has since
+        // been truncated, so it must not suppress anything in the new one —
+        // including when the new WAL grows past the old mark's length.
+        let dir = TempDir::new().unwrap();
+        let config = make_config(&dir);
+        let h5_path = config.path.clone();
+        {
+            let mut mem = HDF5Memory::create(config).unwrap();
+            mem.save(make_entry("a", &[1.0, 0.0, 0.0, 0.0])).unwrap();
+            mem.flush_wal().unwrap();
+            for name in ["b", "c", "d"] {
+                mem.save(make_entry(name, &[1.0, 0.0, 0.0, 0.0])).unwrap();
+            }
+        }
+        assert_eq!(reopen_chunks(&h5_path), ["a", "b", "c", "d"]);
+    }
+
+    #[test]
+    fn save_or_update_replays_as_update_not_duplicate() {
+        let dir = TempDir::new().unwrap();
+        let config = make_config(&dir);
+        let h5_path = config.path.clone();
+        {
+            let mut mem = HDF5Memory::create(config).unwrap();
+            let mut first = make_entry("v1", &[1.0, 0.0, 0.0, 0.0]);
+            first.tags = "key".into();
+            let mut second = make_entry("v2", &[0.0, 1.0, 0.0, 0.0]);
+            second.tags = "key".into();
+            let a = mem.save_or_update(first).unwrap();
+            mem.save(make_entry("other", &[0.0, 0.0, 1.0, 0.0]))
+                .unwrap();
+            let b = mem.save_or_update(second).unwrap();
+            assert_eq!(a, b);
+            assert_eq!(mem.cache.chunks, ["v2", "other"]);
+            // Dropped without a checkpoint: all three records live in the WAL.
+        }
+        let mem = HDF5Memory::open(&h5_path).unwrap();
+        assert_eq!(mem.cache.chunks, ["v2", "other"]);
+        assert_eq!(mem.cache.embeddings[0], [0.0, 1.0, 0.0, 0.0]);
+    }
+
+    #[test]
+    fn v3_wal_is_read_and_upgraded_in_place() {
+        let dir = TempDir::new().unwrap();
+        let wal_path = dir.path().join("old.wal");
+        {
+            let mut wal = WalFile::open(&wal_path).unwrap();
+            wal.append_save(&make_wal_entry("kept", &[1.0])).unwrap();
+        }
+        // Rewrite the header as the pre-Update chained format.
+        let mut bytes = std::fs::read(&wal_path).unwrap();
+        bytes[4] = WAL_VERSION_CHAINED_NO_UPDATE;
+        std::fs::write(&wal_path, &bytes).unwrap();
+
+        assert_eq!(WalFile::read_entries(&wal_path).unwrap().len(), 1);
+        {
+            let mut wal = WalFile::open(&wal_path).unwrap();
+            assert_eq!(wal.pending_count(), 1);
+            wal.append_save(&make_wal_entry("new", &[2.0])).unwrap();
+        }
+        assert_eq!(std::fs::read(&wal_path).unwrap()[4], WAL_VERSION);
+        let chunks: Vec<_> = WalFile::read_entries(&wal_path)
+            .unwrap()
+            .into_iter()
+            .map(|e| e.chunk)
+            .collect();
+        assert_eq!(chunks, ["kept", "new"]);
+    }
+
+    #[test]
+    fn mark_matching_is_exact() {
+        let dir = TempDir::new().unwrap();
+        let wal_path = dir.path().join("m.wal");
+        let mut wal = WalFile::open(&wal_path).unwrap();
+        wal.append_save(&make_wal_entry("one", &[1.0])).unwrap();
+        let after_one = wal.mark();
+        wal.append_save(&make_wal_entry("two", &[2.0])).unwrap();
+        let after_two = wal.mark();
+        drop(wal);
+
+        let read = |m| {
+            WalFile::read_entries_for_migration(&wal_path, m)
+                .unwrap()
+                .into_iter()
+                .map(|e| e.chunk)
+                .collect::<Vec<_>>()
+        };
+        assert_eq!(read(None), ["one", "two"]);
+        assert_eq!(read(Some(after_one)), ["two"]);
+        assert!(read(Some(after_two)).is_empty());
+        // Right length, wrong CRC (a different WAL generation): skip nothing.
+        let foreign = WalMark {
+            crc: after_one.crc ^ 1,
+            ..after_one
+        };
+        assert_eq!(read(Some(foreign)), ["one", "two"]);
+        // Reopening resumes the same mark.
+        assert_eq!(WalFile::open(&wal_path).unwrap().mark(), after_two);
+    }
+
     #[test]
     fn test_wal_replay_on_open() {
         // Test WAL replay using read_entries + replay_into_cache directly,
@@ -1228,8 +1541,7 @@ mod tests {
         drop(wal); // simulate a restart without ever truncating the WAL
 
         let mut wal2 = WalFile::open(&wal_path).unwrap();
-        wal2.append_save(&make_wal_entry("second", &[2.0]))
+        wal2.append_save(&make_wal_entry("second", &[2.0])).unwrap();
-            .unwrap();
         drop(wal2);
 
         let entries = WalFile::read_entries(&wal_path).unwrap();
@@ -1270,7 +1582,7 @@ mod tests {
         std::fs::write(&wal_path, build_legacy_v1_wal_bytes()).unwrap();
 
         // Only the migration-only reader may read a legacy no-CRC file.
-        let entries = WalFile::read_entries_for_migration(&wal_path).unwrap();
+        let entries = WalFile::read_entries_for_migration(&wal_path, None).unwrap();
         assert_eq!(entries.len(), 1);
         assert_eq!(entries[0].chunk, "legacy-chunk");
         assert_eq!(entries[0].embedding, vec![1.0, 2.0]);

crates/clawhdf5-agent/tests/crash_recovery.rs

@@ -0,0 +1,187 @@
+//! Crash-recovery matrix for `HDF5Memory`.
+//!
+//! A process crash leaves whatever reached the OS on disk. These tests build
+//! the on-disk images such a crash can leave behind — after every operation,
+//! inside the checkpoint window (new `.h5` in place, WAL not yet truncated),
+//! and with the WAL torn at every possible length — then reopen each image
+//! and check the recovered store against a model of what was acknowledged.
+//!
+//! Invariants:
+//! * never a duplicated or invented record;
+//! * an image taken between operations recovers *exactly* the acknowledged
+//!   state;
+//! * a torn WAL recovers the last checkpoint plus a prefix of the operations
+//!   logged since.
+
+use std::path::{Path, PathBuf};
+
+use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
+use tempfile::TempDir;
+
+struct Rng(u64);
+
+impl Rng {
+    fn next(&mut self) -> u64 {
+        self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
+        let mut z = self.0;
+        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+        z ^ (z >> 31)
+    }
+    fn below(&mut self, n: usize) -> usize {
+        (self.next() % n.max(1) as u64) as usize
+    }
+}
+
+fn entry(chunk: &str, tags: &str) -> MemoryEntry {
+    MemoryEntry {
+        chunk: chunk.to_string(),
+        embedding: vec![1.0, 0.0, 0.0, 0.0],
+        source_channel: "test".into(),
+        timestamp: 1.0,
+        session_id: "s".into(),
+        tags: tags.to_string(),
+    }
+}
+
+fn wal_path(h5: &Path) -> PathBuf {
+    h5.with_extension("h5.wal")
+}
+
+/// Copy the store (`.h5` + WAL) into a fresh directory, as a crash image.
+fn image(h5: &Path, into: &TempDir, name: &str) -> PathBuf {
+    let dest = into.path().join(format!("{name}.h5"));
+    std::fs::copy(h5, &dest).unwrap();
+    if wal_path(h5).exists() {
+        std::fs::copy(wal_path(h5), wal_path(&dest)).unwrap();
+    }
+    dest
+}
+
+fn recovered(h5: &Path) -> Vec<String> {
+    // Read-only: the image must not be modified, and no lock is needed.
+    HDF5Memory::open_read_only(h5).unwrap().cache.chunks.clone()
+}
+
+/// Apply one random operation to the store and to the model.
+fn step(mem: &mut HDF5Memory, model: &mut Vec<String>, rng: &mut Rng, n: usize) {
+    match rng.below(6) {
+        0 => mem.flush_wal().unwrap(),
+        1 if !model.is_empty() => {
+            // Update an existing record in place, addressed by its tag.
+            let idx = rng.below(model.len());
+            let chunk = format!("u{n}");
+            assert_eq!(
+                mem.save_or_update(entry(&chunk, &format!("tag{idx}")))
+                    .unwrap(),
+                idx
+            );
+            model[idx] = chunk;
+        }
+        _ => {
+            let chunk = format!("c{n}");
+            mem.save(entry(&chunk, &format!("tag{}", model.len())))
+                .unwrap();
+            model.push(chunk);
+        }
+    }
+}
+
+#[test]
+fn image_after_every_operation_recovers_the_acknowledged_state() {
+    for seed in 0..40u64 {
+        let mut rng = Rng(seed);
+        let dir = TempDir::new().unwrap();
+        let images = TempDir::new().unwrap();
+        let mut config = MemoryConfig::new(dir.path().join("store.h5"), "agent", 4);
+        config.wal_enabled = true;
+        config.wal_max_entries = 1 + rng.below(6); // force frequent checkpoints
+        let h5 = config.path.clone();
+        let mut mem = HDF5Memory::create(config).unwrap();
+        let mut model = Vec::new();
+
+        for n in 0..30 {
+            step(&mut mem, &mut model, &mut rng, n);
+            let img = image(&h5, &images, &format!("s{seed}-{n}"));
+            assert_eq!(recovered(&img), model, "seed {seed}, after op {n}");
+        }
+    }
+}
+
+#[test]
+fn crash_inside_the_checkpoint_window_never_duplicates() {
+    for seed in 0..40u64 {
+        let mut rng = Rng(seed ^ 0xABCD);
+        let dir = TempDir::new().unwrap();
+        let images = TempDir::new().unwrap();
+        let mut config = MemoryConfig::new(dir.path().join("store.h5"), "agent", 4);
+        config.wal_enabled = true;
+        config.wal_max_entries = 1000; // checkpoints only when we ask
+        let h5 = config.path.clone();
+        let mut mem = HDF5Memory::create(config).unwrap();
+        let mut model = Vec::new();
+
+        for round in 0..4 {
+            for n in 0..(1 + rng.below(6)) {
+                step(&mut mem, &mut model, &mut rng, round * 100 + n);
+            }
+            // The WAL as it is just before the checkpoint...
+            let stale_wal = images.path().join(format!("stale-{seed}-{round}.wal"));
+            if wal_path(&h5).exists() {
+                std::fs::copy(wal_path(&h5), &stale_wal).unwrap();
+            }
+            mem.flush_wal().unwrap();
+            // ...put back next to the NEW .h5: the crash-in-the-window image.
+            let img = image(&h5, &images, &format!("w{seed}-{round}"));
+            if stale_wal.exists() {
+                std::fs::copy(&stale_wal, wal_path(&img)).unwrap();
+            }
+            assert_eq!(recovered(&img), model, "seed {seed}, round {round}");
+        }
+    }
+}
+
+#[test]
+fn torn_wal_recovers_checkpoint_plus_a_prefix() {
+    let dir = TempDir::new().unwrap();
+    let images = TempDir::new().unwrap();
+    let mut config = MemoryConfig::new(dir.path().join("store.h5"), "agent", 4);
+    config.wal_enabled = true;
+    config.wal_max_entries = 1000;
+    let h5 = config.path.clone();
+    let mut mem = HDF5Memory::create(config).unwrap();
+
+    for name in ["a", "b"] {
+        mem.save(entry(name, name)).unwrap();
+    }
+    mem.flush_wal().unwrap();
+    let checkpointed = vec!["a".to_string(), "b".to_string()];
+
+    // States the store passes through as each later op is logged.
+    let mut states = vec![checkpointed.clone()];
+    let mut model = checkpointed.clone();
+    mem.save(entry("c", "c")).unwrap();
+    model.push("c".into());
+    states.push(model.clone());
+    mem.save_or_update(entry("a2", "a")).unwrap();
+    model[0] = "a2".into();
+    states.push(model.clone());
+    mem.save(entry("d", "d")).unwrap();
+    model.push("d".into());
+    states.push(model.clone());
+
+    let full_wal = std::fs::read(wal_path(&h5)).unwrap();
+    let mut seen = std::collections::BTreeSet::new();
+    for len in 0..=full_wal.len() {
+        let img = image(&h5, &images, &format!("t{len}"));
+        std::fs::write(wal_path(&img), &full_wal[..len]).unwrap();
+        let got = recovered(&img);
+        let which = states
+            .iter()
+            .position(|s| *s == got)
+            .unwrap_or_else(|| panic!("WAL torn at {len} bytes recovered {got:?}"));
+        seen.insert(which);
+    }
+    // Every intermediate state is reachable, and the full WAL gives the last.
+    assert_eq!(seen.into_iter().collect::<Vec<_>>(), [0, 1, 2, 3]);
+}

crates/clawhdf5-agent/tests/e2e_tests.rs

@@ -196,7 +196,7 @@ fn test_migration_round_trip() {
     mem.add_relation(e1, e2, "discusses", 0.8).unwrap();
 
     // Verify all data transferred by reopening
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.count(), 500);
 
     // Verify sessions
@@ -266,7 +266,7 @@ fn test_knowledge_graph_workflow() {
     assert_eq!(entity.entity_type, "library");
 
     // Persistence
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.knowledge().entities.len(), 4);
     assert_eq!(reopened.knowledge().relations.len(), 4);
 
@@ -316,7 +316,7 @@ fn test_multi_session_workflow() {
     assert_eq!(mem.count(), 100); // 5 sessions * 20 entries
 
     // Reopen and verify sessions
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     for sess in 0..5 {
         let summary = reopened
             .get_session_summary(&format!("sess_{sess}"))
@@ -460,7 +460,7 @@ fn test_snapshot_and_continue() {
     assert_eq!(snap_mem.count(), 50);
 
     // Original should have 100
-    let orig_mem = HDF5Memory::open(&path).unwrap();
+    let orig_mem = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(orig_mem.count(), 100);
 }
 
@@ -483,7 +483,7 @@ fn test_config_persistence_across_ops() {
     mem.add_session("s1", 0, 0, "ch", "summary").unwrap();
     mem.add_entity("Entity", "type", -1).unwrap();
 
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.config().embedding_dim, 128);
     assert_eq!(reopened.config().embedder, "custom:my-embedder-v2");
     assert_eq!(reopened.config().chunk_size, 2048);
@@ -695,7 +695,7 @@ fn test_large_text_chunks() {
     mem.save_batch(entries).unwrap();
 
     // Reopen and verify
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.count(), 10);
 
     let (_, cache, _, _) = read_cache(&path);
@@ -752,7 +752,7 @@ fn test_interleaved_sessions_entries() {
     mem.flush_wal().unwrap();
 
     // Verify
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.count(), 6);
     assert_eq!(
         reopened.get_session_summary("s1").unwrap().as_deref(),
@@ -806,7 +806,7 @@ fn test_knowledge_graph_with_embeddings() {
     mem.add_relation(e_python, e_hdf5, "reads", 0.9).unwrap();
 
     // Verify entity-embedding linkage persists
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     let rust_entity = reopened.knowledge().get_entity(e_rust).unwrap();
     assert_eq!(rust_entity.embedding_idx, idx0 as i64);
 
@@ -1048,7 +1048,7 @@ fn test_gpu_l2_fallback_works() {
     let tombstones = vec![0u8; 3];
 
     let gpu = clawhdf5_agent::gpu_search::GpuSearchBackend::try_init(&vectors, &norms, 2, 1);
-    let results = gpu.search_l2(&vec![0.0, 0.0], &vectors, &tombstones, 3);
+    let results = gpu.search_l2(&[0.0, 0.0], &vectors, &tombstones, 3);
 
     assert_eq!(results.len(), 3);
     assert_eq!(results[0].0, 0);
@@ -1099,7 +1099,7 @@ fn test_mmap_reader_direct_access() {
 
     // Open via MmapReader directly
     let mmap = clawhdf5_io::MmapReader::open(&path).unwrap();
-    assert!(mmap.len() > 0);
+    assert!(!mmap.is_empty());
     // Verify we can read bytes at specific offsets
     let bytes = mmap.read_at(0, 8);
     assert!(bytes.is_some());

crates/clawhdf5-agent/tests/memx_comparison.rs

@@ -137,12 +137,12 @@ fn bench_hit_at_1_1014_records() {
             0.3,
             1,
         );
-        if let Some((top_idx, _)) = results.first() {
+        if let Some((top_idx, _)) = results.first()
-            if *top_idx == target_indices[qi] {
+            && *top_idx == target_indices[qi]
+        {
             hits += 1;
         }
     }
-    }
 
     let hit_at_1 = hits as f64 / NUM_QUERIES as f64;
     println!(

crates/clawhdf5-agent/tests/stress_tests.rs

@@ -105,7 +105,7 @@ fn test_heavy_tombstoning() {
     assert_eq!(mem.count_active(), 5000);
 
     // Verify persistence
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.count(), 5000);
 }
 
@@ -163,7 +163,7 @@ fn test_large_embeddings_1536() {
     assert_eq!(mem.count(), 10_000);
 
     // Verify persistence
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.count(), 10_000);
 
     // Verify search works on large dims
@@ -545,7 +545,7 @@ fn test_delete_all_entries() {
     assert_eq!(mem.count(), 0);
 
     // Verify persistence
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.count(), 0);
 }
 
@@ -639,7 +639,7 @@ fn test_unicode_content() {
     ];
     mem.save_batch(entries).unwrap();
 
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.count(), 3);
 
     let (_, cache, _, _) = clawhdf5_agent::storage::read_from_disk(&path).unwrap();
@@ -685,6 +685,6 @@ fn test_rapid_save_delete_cycles() {
     assert_eq!(removed, 250);
     assert_eq!(mem.count(), 250);
 
-    let reopened = HDF5Memory::open(&path).unwrap();
+    let reopened = HDF5Memory::open_read_only(&path).unwrap();
     assert_eq!(reopened.count(), 250);
 }

crates/clawhdf5-agent/tests/wal_properties.rs

@@ -0,0 +1,213 @@
+//! Property tests for the write-ahead log.
+//!
+//! A deterministic generator (no external crates, reproducible from the seed
+//! printed on failure) drives thousands of cases through two properties:
+//!
+//! 1. **Round trip** — whatever was appended is read back, in order, intact.
+//! 2. **Prefix under corruption** — after *any* damage to the file (bit flips,
+//!    truncation, inserted or deleted bytes, duplicated or reordered regions),
+//!    reading never panics and yields an exact *prefix* of what was written.
+//!    This is the guarantee the chained CRC exists to provide: replay may stop
+//!    early, but it never returns a corrupted, reordered, or invented entry.
+
+use clawhdf5_agent::wal::{WalEntry, WalEntryType, WalFile};
+
+/// SplitMix64: tiny, well-distributed, and fully determined by its seed.
+struct Rng(u64);
+
+impl Rng {
+    fn next(&mut self) -> u64 {
+        self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
+        let mut z = self.0;
+        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+        z ^ (z >> 31)
+    }
+
+    fn below(&mut self, n: usize) -> usize {
+        (self.next() % n.max(1) as u64) as usize
+    }
+
+    fn string(&mut self, max_len: usize) -> String {
+        const ALPHABET: &[char] = &['a', 'Z', '0', ' ', '\n', '\0', 'é', '漢', '🦀', '"'];
+        (0..self.below(max_len + 1))
+            .map(|_| ALPHABET[self.below(ALPHABET.len())])
+            .collect()
+    }
+}
+
+/// What a test appended, in a form comparable with what is read back.
+#[derive(Debug, Clone, PartialEq)]
+enum Logged {
+    Save(String, Vec<u32>, String, String, String, u64),
+    Update(usize, String, Vec<u32>, u64),
+    Tombstone(usize, u64),
+}
+
+fn logged(entry: &WalEntry) -> Logged {
+    // Compare floats by bit pattern so NaN payloads and -0.0 count as intact.
+    let bits: Vec<u32> = entry.embedding.iter().map(|f| f.to_bits()).collect();
+    let ts = entry.timestamp.to_bits();
+    match entry.entry_type {
+        WalEntryType::Save => Logged::Save(
+            entry.chunk.clone(),
+            bits,
+            entry.source_channel.clone(),
+            entry.session_id.clone(),
+            entry.tags.clone(),
+            ts,
+        ),
+        WalEntryType::Update => {
+            Logged::Update(entry.update_index.unwrap(), entry.chunk.clone(), bits, ts)
+        }
+        WalEntryType::Tombstone => Logged::Tombstone(entry.tombstone_index.unwrap(), ts),
+        WalEntryType::ActivationUpdate => unreachable!("never written by these tests"),
+    }
+}
+
+/// Append a random mix of records; return what was written.
+fn write_random_wal(path: &std::path::Path, rng: &mut Rng) -> Vec<Logged> {
+    let mut wal = WalFile::open(path).unwrap();
+    let mut written = Vec::new();
+    for _ in 0..rng.below(12) {
+        let timestamp = f64::from_bits(rng.next());
+        if rng.below(5) == 0 {
+            let index = rng.below(1000);
+            wal.append_tombstone(index, timestamp).unwrap();
+            written.push(Logged::Tombstone(index, timestamp.to_bits()));
+            continue;
+        }
+        let update_index = (rng.below(4) == 0).then(|| rng.below(1000));
+        let entry = WalEntry {
+            entry_type: if update_index.is_some() {
+                WalEntryType::Update
+            } else {
+                WalEntryType::Save
+            },
+            timestamp,
+            chunk: rng.string(40),
+            embedding: (0..rng.below(9))
+                .map(|_| f32::from_bits(rng.next() as u32))
+                .collect(),
+            source_channel: rng.string(8),
+            session_id: rng.string(8),
+            tags: rng.string(8),
+            tombstone_index: None,
+            update_index,
+        };
+        wal.append_save(&entry).unwrap();
+        written.push(logged(&entry));
+    }
+    written
+}
+
+fn read_back(path: &std::path::Path) -> Option<Vec<Logged>> {
+    WalFile::read_entries(path)
+        .ok()
+        .map(|entries| entries.iter().map(logged).collect())
+}
+
+#[test]
+fn everything_appended_is_read_back_intact() {
+    let dir = tempfile::TempDir::new().unwrap();
+    for seed in 0..300u64 {
+        let path = dir.path().join(format!("rt-{seed}.wal"));
+        let written = write_random_wal(&path, &mut Rng(seed));
+        assert_eq!(read_back(&path).unwrap(), written, "seed {seed}");
+        // Reopening (which scans and repositions) must not disturb anything.
+        drop(WalFile::open(&path).unwrap());
+        assert_eq!(
+            read_back(&path).unwrap(),
+            written,
+            "seed {seed} after reopen"
+        );
+    }
+}
+
+/// Damage `bytes` in one of several ways.
+fn corrupt(bytes: &mut Vec<u8>, rng: &mut Rng) {
+    if bytes.is_empty() {
+        return;
+    }
+    match rng.below(7) {
+        0 => {
+            let i = rng.below(bytes.len());
+            bytes[i] ^= 1 << rng.below(8);
+        }
+        1 => bytes.truncate(rng.below(bytes.len())),
+        2 => {
+            let i = rng.below(bytes.len() + 1);
+            bytes.insert(i, rng.next() as u8);
+        }
+        3 => {
+            let i = rng.below(bytes.len());
+            bytes.remove(i);
+        }
+        4 => {
+            // Duplicate a region in place (a replayed/duplicated entry).
+            let a = rng.below(bytes.len());
+            let b = a + rng.below(bytes.len() - a);
+            let region = bytes[a..b].to_vec();
+            let at = rng.below(bytes.len() + 1);
+            bytes.splice(at..at, region);
+        }
+        5 => {
+            // Swap two regions (reordered entries).
+            let mid = rng.below(bytes.len());
+            bytes.rotate_left(mid);
+        }
+        _ => {
+            let i = rng.below(bytes.len());
+            let n = rng.below(bytes.len() - i + 1);
+            for b in &mut bytes[i..i + n] {
+                *b = rng.next() as u8;
+            }
+        }
+    }
+}
+
+#[test]
+fn any_corruption_yields_a_prefix_never_a_wrong_entry() {
+    let dir = tempfile::TempDir::new().unwrap();
+    let mut shortened = 0u32;
+    for seed in 0..1500u64 {
+        let mut rng = Rng(seed ^ 0xC0FF_EE00);
+        let path = dir.path().join("c.wal");
+        let _ = std::fs::remove_file(&path);
+        let written = write_random_wal(&path, &mut rng);
+
+        let mut bytes = std::fs::read(&path).unwrap();
+        for _ in 0..=rng.below(3) {
+            corrupt(&mut bytes, &mut rng);
+        }
+        std::fs::write(&path, &bytes).unwrap();
+
+        // An unreadable header is a clean error; anything else is a prefix.
+        if let Some(read) = read_back(&path) {
+            assert!(
+                read.len() <= written.len() && read[..] == written[..read.len()],
+                "seed {seed}: read {read:?}\nis not a prefix of {written:?}"
+            );
+            if read.len() < written.len() {
+                shortened += 1;
+            }
+            // Opening for append repairs the tail; what was readable stays so,
+            // and a new entry lands right after it.
+            if let Ok(mut wal) = WalFile::open(&path) {
+                wal.append_tombstone(7, 1.0).unwrap();
+                drop(wal);
+                let mut expected = read.clone();
+                expected.push(Logged::Tombstone(7, 1.0f64.to_bits()));
+                assert_eq!(
+                    read_back(&path).unwrap(),
+                    expected,
+                    "seed {seed} after repair"
+                );
+            }
+        }
+    }
+    assert!(
+        shortened > 100,
+        "corruption rarely took effect: {shortened}"
+    );
+}

crates/clawhdf5-android/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-android"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "Android JNI bridge for edgehdf5-memory HDF5 backend"
 license = "MIT"

crates/clawhdf5-ann/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-ann"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "HNSW approximate nearest neighbor index stored as HDF5"
 license = "MIT"
@@ -10,9 +10,9 @@ keywords = ["hdf5", "ann", "hnsw", "nearest-neighbor"]
 categories = ["algorithms", "science"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.2.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.3.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.2.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.3.0" }
-clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.2.0" }
+clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.3.0" }
 rayon = { version = "1", optional = true }
 
 [features]

crates/clawhdf5-bench/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-bench"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "Benchmark harnesses for clawhdf5-agent (Track 8)"
 license = "MIT"

crates/clawhdf5-bench/src/bin/consolidation_efficiency.rs

@@ -242,7 +242,12 @@ fn run_quality_benchmark() {
     for i in 0..990 {
         let chunk = make_noise_content(i);
         let embedding = make_embedding(i + 100);
-        engine.add_trusted_memory(chunk, embedding, TrustedSource::System, now + i as f64 * 0.1);
+        engine.add_trusted_memory(
+            chunk,
+            embedding,
+            TrustedSource::System,
+            now + i as f64 * 0.1,
+        );
     }
 
     println!("  → Inserted {} records total", engine.records().len());

crates/clawhdf5-cli/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-cli"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 license = "MIT"
 description = "CLI for clawhdf5 agent memory — create, save, search, recall, stats"
@@ -14,7 +14,7 @@ name = "clawhdf5"
 path = "src/main.rs"
 
 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.2.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.3.0" }
 clap = { version = "4", features = ["derive", "env"] }
 serde_json = "1"
 serde = { workspace = true }

crates/clawhdf5-cli/src/main.rs

@@ -146,7 +146,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
         }
 
         Commands::Recall { index } => {
-            let mem = HDF5Memory::open(&cli.path)?;
+            let mem = HDF5Memory::open_read_only(&cli.path)?;
             match mem.get_chunk(index) {
                 Some(content) => {
                     let j = serde_json::json!({ "index": index, "chunk": content });
@@ -160,7 +160,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
         }
 
         Commands::Stats => {
-            let mem = HDF5Memory::open(&cli.path)?;
+            let mem = HDF5Memory::open_read_only(&cli.path)?;
             let cfg = mem.config();
             let j = serde_json::json!({
                 "path": cli.path.display().to_string(),
@@ -187,7 +187,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
         }
 
         Commands::AgentsMd { output } => {
-            let mem = HDF5Memory::open(&cli.path)?;
+            let mem = HDF5Memory::open_read_only(&cli.path)?;
             let md = mem.generate_agents_md();
             match output {
                 Some(p) => {
@@ -199,7 +199,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
         }
 
         Commands::Export => {
-            let mem = HDF5Memory::open(&cli.path)?;
+            let mem = HDF5Memory::open_read_only(&cli.path)?;
             for i in 0..mem.count() {
                 if let Some(chunk) = mem.get_chunk(i) {
                     let j = serde_json::json!({ "index": i, "chunk": chunk });

crates/clawhdf5-derive/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-derive"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "Derive macros for rustyhdf5 HDF5 traits"
 license = "MIT"

crates/clawhdf5-filters/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-filters"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "Filter and compression pipeline for clawhdf5"
 license = "MIT"

crates/clawhdf5-format/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-format"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "Pure-Rust HDF5 binary format parsing and writing — no C dependencies"
 license = "MIT"
@@ -25,7 +25,7 @@ pco = { version = "1.0", optional = true }
 [dev-dependencies]
 serde_json = "1"
 criterion = { workspace = true }
-clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.2.0" }
+clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.3.0" }
 
 [[bench]]
 name = "bench"

crates/clawhdf5-format/src/attribute.rs

@@ -1,7 +1,9 @@
 //! HDF5 Attribute message parsing (message type 0x000C).
 
 #[cfg(not(feature = "std"))]
-use alloc::{string::String, vec::Vec};
+use alloc::{borrow::Cow, string::String, vec::Vec};
+#[cfg(feature = "std")]
+use std::borrow::Cow;
 
 use crate::attribute_info::AttributeInfoMessage;
 use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
@@ -48,17 +50,64 @@ impl AttributeMessage {
     ///
     /// `length_size` is needed for dataspace dimension parsing.
     pub fn parse(data: &[u8], length_size: u8) -> Result<AttributeMessage, FormatError> {
+        Self::parse_impl(data, length_size, None)
+    }
+
+    /// [`AttributeMessage::parse`] with access to the rest of the file, which
+    /// is needed when the attribute's datatype or dataspace is *shared* (v2/v3
+    /// flag bits 0/1) — e.g. an attribute created with a committed datatype.
+    /// In that case the embedded bytes are a reference to the real message,
+    /// not the message. Without file access such an attribute is an error
+    /// rather than a garbage datatype.
+    pub fn parse_in_file(
+        data: &[u8],
+        file_data: &[u8],
+        offset_size: u8,
+        length_size: u8,
+    ) -> Result<AttributeMessage, FormatError> {
+        Self::parse_impl(data, length_size, Some((file_data, offset_size)))
+    }
+
+    fn parse_impl(
+        data: &[u8],
+        length_size: u8,
+        file: Option<(&[u8], u8)>,
+    ) -> Result<AttributeMessage, FormatError> {
         ensure_len(data, 0, 2)?;
         let version = data[0];
 
         match version {
             1 => Self::parse_v1(data, length_size),
-            2 => Self::parse_v2(data, length_size),
+            2 => Self::parse_v2(data, length_size, file),
-            3 => Self::parse_v3(data, length_size),
+            3 => Self::parse_v3(data, length_size, file),
             _ => Err(FormatError::InvalidAttributeVersion(version)),
         }
     }
 
+    /// The bytes of an embedded datatype/dataspace message, following the
+    /// shared-message reference when `shared` is set.
+    fn embedded_message<'a>(
+        bytes: &'a [u8],
+        shared: bool,
+        msg_type: MessageType,
+        length_size: u8,
+        file: Option<(&[u8], u8)>,
+    ) -> Result<Cow<'a, [u8]>, FormatError> {
+        if !shared {
+            return Ok(Cow::Borrowed(bytes));
+        }
+        let (file_data, offset_size) = file.ok_or(FormatError::UnresolvedSharedMessage)?;
+        let shared_ref = shared_message::parse_shared_ref(bytes, offset_size)?;
+        shared_message::resolve_shared_message(
+            file_data,
+            &shared_ref,
+            msg_type,
+            offset_size,
+            length_size,
+        )
+        .map(Cow::Owned)
+    }
+
     fn parse_v1(data: &[u8], length_size: u8) -> Result<AttributeMessage, FormatError> {
         // version(1) + reserved(1) + name_size(2) + datatype_size(2) + dataspace_size(2) = 8
         ensure_len(data, 0, 8)?;
@@ -94,7 +143,13 @@ impl AttributeMessage {
         })
     }
 
-    fn parse_v2(data: &[u8], length_size: u8) -> Result<AttributeMessage, FormatError> {
+    fn parse_v2(
+        data: &[u8],
+        length_size: u8,
+        file: Option<(&[u8], u8)>,
+    ) -> Result<AttributeMessage, FormatError> {
+        // Flags: bit 0 = datatype is shared, bit 1 = dataspace is shared.
+        let flags = data.get(1).copied().unwrap_or(0);
         // version(1) + flags(1) + name_size(2) + datatype_size(2) + dataspace_size(2) = 8
         ensure_len(data, 0, 8)?;
         let name_size = u16::from_le_bytes([data[2], data[3]]) as usize;
@@ -110,12 +165,26 @@ impl AttributeMessage {
 
         // Datatype (NO padding)
         ensure_len(data, pos, datatype_size)?;
-        let (datatype, _) = Datatype::parse(&data[pos..pos + datatype_size])?;
+        let dt_bytes = Self::embedded_message(
+            &data[pos..pos + datatype_size],
+            flags & 0x01 != 0,
+            MessageType::Datatype,
+            length_size,
+            file,
+        )?;
+        let (datatype, _) = Datatype::parse(&dt_bytes)?;
         pos += datatype_size;
 
         // Dataspace (NO padding)
         ensure_len(data, pos, dataspace_size)?;
-        let dataspace = Dataspace::parse(&data[pos..pos + dataspace_size], length_size)?;
+        let ds_bytes = Self::embedded_message(
+            &data[pos..pos + dataspace_size],
+            flags & 0x02 != 0,
+            MessageType::Dataspace,
+            length_size,
+            file,
+        )?;
+        let dataspace = Dataspace::parse(&ds_bytes, length_size)?;
         pos += dataspace_size;
 
         let raw_data = compute_raw_data(data, pos, &dataspace, &datatype);
@@ -128,7 +197,13 @@ impl AttributeMessage {
         })
     }
 
-    fn parse_v3(data: &[u8], length_size: u8) -> Result<AttributeMessage, FormatError> {
+    fn parse_v3(
+        data: &[u8],
+        length_size: u8,
+        file: Option<(&[u8], u8)>,
+    ) -> Result<AttributeMessage, FormatError> {
+        // Flags: bit 0 = datatype is shared, bit 1 = dataspace is shared.
+        let flags = data.get(1).copied().unwrap_or(0);
         // version(1) + flags(1) + name_size(2) + datatype_size(2) + dataspace_size(2) + encoding(1) = 9
         ensure_len(data, 0, 9)?;
         let name_size = u16::from_le_bytes([data[2], data[3]]) as usize;
@@ -145,12 +220,26 @@ impl AttributeMessage {
 
         // Datatype (NO padding)
         ensure_len(data, pos, datatype_size)?;
-        let (datatype, _) = Datatype::parse(&data[pos..pos + datatype_size])?;
+        let dt_bytes = Self::embedded_message(
+            &data[pos..pos + datatype_size],
+            flags & 0x01 != 0,
+            MessageType::Datatype,
+            length_size,
+            file,
+        )?;
+        let (datatype, _) = Datatype::parse(&dt_bytes)?;
         pos += datatype_size;
 
         // Dataspace (NO padding)
         ensure_len(data, pos, dataspace_size)?;
-        let dataspace = Dataspace::parse(&data[pos..pos + dataspace_size], length_size)?;
+        let ds_bytes = Self::embedded_message(
+            &data[pos..pos + dataspace_size],
+            flags & 0x02 != 0,
+            MessageType::Dataspace,
+            length_size,
+            file,
+        )?;
+        let dataspace = Dataspace::parse(&ds_bytes, length_size)?;
         pos += dataspace_size;
 
         let raw_data = compute_raw_data(data, pos, &dataspace, &datatype);
@@ -326,10 +415,20 @@ pub fn extract_attributes_full(
                     offset_size,
                     length_size,
                 )?;
-                let attr = AttributeMessage::parse(&resolved_data, length_size)?;
+                let attr = AttributeMessage::parse_in_file(
+                    &resolved_data,
+                    file_data,
+                    offset_size,
+                    length_size,
+                )?;
                 attrs.push(attr);
             } else {
-                let attr = AttributeMessage::parse(&msg.data, length_size)?;
+                let attr = AttributeMessage::parse_in_file(
+                    &msg.data,
+                    file_data,
+                    offset_size,
+                    length_size,
+                )?;
                 attrs.push(attr);
             }
         }
@@ -399,7 +498,8 @@ fn extract_dense_attributes(
         let attr_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
 
         // The data in the heap is a complete attribute message
-        let attr = AttributeMessage::parse(&attr_data, length_size)?;
+        let attr =
+            AttributeMessage::parse_in_file(&attr_data, file_data, offset_size, length_size)?;
         attrs.push(attr);
     }
 
@@ -472,14 +572,13 @@ mod tests {
 
         // Name padded to 8 bytes
         data.extend_from_slice(name);
-        while data.len() % 8 != 0 || data.len() == 8 {
+        if data.len() % 8 != 0 || data.len() == 8 {
             // Pad name to 8-byte boundary from start of name
             let name_start = 8;
             let name_padded = pad8(name_size);
             while data.len() < name_start + name_padded {
                 data.push(0);
             }
-            break;
         }
 
         // Datatype padded to 8 bytes
@@ -749,11 +848,11 @@ mod tests {
         data.extend_from_slice(name);
         data.extend_from_slice(&dt_bytes);
         data.extend_from_slice(&ds_bytes);
-        data.extend_from_slice(&3.14f64.to_le_bytes());
+        data.extend_from_slice(&3.25f64.to_le_bytes());
 
         let attr = AttributeMessage::parse(&data, 8).unwrap();
         let vals = attr.read_as_f64().unwrap();
-        assert_eq!(vals, vec![3.14]);
+        assert_eq!(vals, vec![3.25]);
     }
 
     #[test]

crates/clawhdf5-format/src/btree_v2.rs

@@ -416,6 +416,7 @@ fn header_max_total_records(max_leaf_nrec: u64, depth: u16) -> u64 {
 mod tests {
     use super::*;
 
+    #[allow(clippy::too_many_arguments)]
     fn build_btree_v2_header(
         tree_type: u8,
         node_size: u32,

crates/clawhdf5-format/src/chunked_read.rs

@@ -132,6 +132,47 @@ fn ensure_len(data: &[u8], offset: usize, needed: usize) -> Result<(), FormatErr
     Ok(())
 }
 
+/// `elements * elem_size` for sizes that come from the file. Dataspace and
+/// chunk dimensions are untrusted 64-bit fields, so a crafted file can make
+/// the plain product wrap to a small number (or to something enormous).
+pub(crate) fn checked_byte_len(elements: u64, elem_size: usize) -> Result<usize, FormatError> {
+    usize::try_from(elements)
+        .ok()
+        .and_then(|n| n.checked_mul(elem_size))
+        .ok_or_else(|| {
+            FormatError::Overflow(format!(
+                "{elements} elements of {elem_size} bytes exceeds the addressable size"
+            ))
+        })
+}
+
+/// Product of chunk dimensions times the element size, overflow-checked.
+pub(crate) fn checked_chunk_byte_len(
+    chunk_dims: &[usize],
+    elem_size: usize,
+) -> Result<usize, FormatError> {
+    chunk_dims
+        .iter()
+        .try_fold(elem_size, |acc, &d| acc.checked_mul(d))
+        .ok_or_else(|| {
+            FormatError::Overflow(format!(
+                "chunk dimensions {chunk_dims:?} x {elem_size} bytes exceeds the addressable size"
+            ))
+        })
+}
+
+/// A zero-filled output buffer of `len` bytes. `vec![0; len]` aborts the
+/// process when the allocation fails; a size taken from the file must surface
+/// as an error instead.
+pub(crate) fn alloc_output(len: usize) -> Result<Vec<u8>, FormatError> {
+    let mut out = Vec::new();
+    out.try_reserve_exact(len).map_err(|_| {
+        FormatError::Overflow(format!("cannot allocate {len} bytes for dataset output"))
+    })?;
+    out.resize(len, 0);
+    Ok(out)
+}
+
 fn read_offset(data: &[u8], pos: usize, size: u8) -> Result<u64, FormatError> {
     let s = size as usize;
     if pos.checked_add(s).is_none_or(|end| end > data.len()) {
@@ -321,15 +362,17 @@ pub fn generate_implicit_chunks(
 }
 
 /// Read a chunked dataset, decompressing chunks as needed.
-pub fn read_chunked_data(
+/// Every allocated chunk of a chunked dataset, for any supported chunk index,
+/// plus the spatial chunk dimensions. Chunks the file never allocated (sparse
+/// datasets) are simply absent from the list.
+pub fn list_chunks(
     file_data: &[u8],
     layout: &DataLayout,
     dataspace: &Dataspace,
-    datatype: &Datatype,
+    elem_size: usize,
-    pipeline: Option<&FilterPipeline>,
     offset_size: u8,
     length_size: u8,
-) -> Result<Vec<u8>, FormatError> {
+) -> Result<(Vec<ChunkInfo>, Vec<usize>), FormatError> {
     let (
         chunk_dimensions,
         version,
@@ -363,8 +406,6 @@ pub fn read_chunked_data(
     let addr = addr_opt
         .ok_or_else(|| FormatError::ChunkedReadError("no address for chunked layout".into()))?;
 
-    let elem_size = datatype.type_size() as usize;
-
     // Both v3 and v4 include element size as last dim (rank+1)
     let ndims = chunk_dimensions.len();
     let rank = ndims
@@ -393,7 +434,7 @@ pub fn read_chunked_data(
         }
         (4, Some(1)) => {
             // Single chunk — one chunk covering the entire dataset
-            let chunk_byte_size: usize = chunk_dims.iter().product::<usize>() * elem_size;
+            let chunk_byte_size = checked_chunk_byte_len(&chunk_dims, elem_size)?;
             let (csize, fmask) = if let Some(fs) = single_filtered_size {
                 (fs as u32, single_filter_mask.unwrap_or(0))
             } else {
@@ -453,10 +494,38 @@ pub fn read_chunked_data(
         }
     };
 
+    Ok((chunks, chunk_dims))
+}
+
+pub fn read_chunked_data(
+    file_data: &[u8],
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    datatype: &Datatype,
+    pipeline: Option<&FilterPipeline>,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<u8>, FormatError> {
+    let elem_size = datatype.type_size() as usize;
+    let (chunks, chunk_dims) = list_chunks(
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        offset_size,
+        length_size,
+    )?;
+    let rank = chunk_dims.len();
+    let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
+
     // Assemble output
-    let total_elements = dataspace.num_elements() as usize;
+    let total_bytes = checked_byte_len(dataspace.checked_num_elements()?, elem_size)?;
-    let total_bytes = total_elements * elem_size;
+    if total_bytes == 0 {
-    let mut output = vec![0u8; total_bytes];
+        // Also keeps the stride products below in range: with a zero-sized
+        // dimension the total is 0 even if other dimensions are huge.
+        return Ok(Vec::new());
+    }
+    let mut output = alloc_output(total_bytes)?;
 
     let mut ds_strides = vec![1usize; rank];
     for i in (0..rank.saturating_sub(1)).rev() {
@@ -468,8 +537,7 @@ pub fn read_chunked_data(
         chunk_strides[i] = chunk_strides[i + 1] * chunk_dims[i + 1];
     }
 
-    let chunk_total_elements: usize = chunk_dims.iter().product();
+    let chunk_total_bytes = checked_chunk_byte_len(&chunk_dims, elem_size)?;
-    let chunk_total_bytes = chunk_total_elements * elem_size;
 
     // Fast path: no filters — copy directly from file_data without intermediate alloc
     if pipeline.is_none() {
@@ -623,7 +691,7 @@ pub fn read_chunked_data_cached(
         let chunks = match (version, chunk_index_type) {
             (3, _) => collect_chunk_info(file_data, addr, ndims, offset_size, length_size)?,
             (4, Some(1)) => {
-                let chunk_byte_size: usize = chunk_dims.iter().product::<usize>() * elem_size;
+                let chunk_byte_size = checked_chunk_byte_len(&chunk_dims, elem_size)?;
                 let (csize, fmask) = if let Some(fs) = single_filtered_size {
                     (fs as u32, single_filter_mask.unwrap_or(0))
                 } else {
@@ -689,9 +757,13 @@ pub fn read_chunked_data_cached(
     let chunks = cache.all_indexed_chunks().unwrap_or_default();
 
     // Assemble output
-    let total_elements = dataspace.num_elements() as usize;
+    let total_bytes = checked_byte_len(dataspace.checked_num_elements()?, elem_size)?;
-    let total_bytes = total_elements * elem_size;
+    if total_bytes == 0 {
-    let mut output = vec![0u8; total_bytes];
+        // Also keeps the stride products below in range: with a zero-sized
+        // dimension the total is 0 even if other dimensions are huge.
+        return Ok(Vec::new());
+    }
+    let mut output = alloc_output(total_bytes)?;
 
     let mut ds_strides = vec![1usize; rank];
     for i in (0..rank.saturating_sub(1)).rev() {
@@ -703,8 +775,7 @@ pub fn read_chunked_data_cached(
         chunk_strides[i] = chunk_strides[i + 1] * chunk_dims[i + 1];
     }
 
-    let chunk_total_elements: usize = chunk_dims.iter().product();
+    let chunk_total_bytes = checked_chunk_byte_len(&chunk_dims, elem_size)?;
-    let chunk_total_bytes = chunk_total_elements * elem_size;
 
     for chunk_info in &chunks {
         let coord: Vec<u64> = chunk_info.offsets.iter().take(rank).copied().collect();
@@ -976,7 +1047,7 @@ pub fn read_chunked_data_sweep(
         let chunks = match (version, chunk_index_type) {
             (3, _) => collect_chunk_info(file_data, addr, ndims, offset_size, length_size)?,
             (4, Some(1)) => {
-                let chunk_byte_size: usize = chunk_dims.iter().product::<usize>() * elem_size;
+                let chunk_byte_size = checked_chunk_byte_len(&chunk_dims, elem_size)?;
                 let (csize, fmask) = if let Some(fs) = single_filtered_size {
                     (fs as u32, single_filter_mask.unwrap_or(0))
                 } else {
@@ -1042,9 +1113,13 @@ pub fn read_chunked_data_sweep(
     let chunks = cache.all_indexed_chunks().unwrap_or_default();
 
     // Assemble output
-    let total_elements = dataspace.num_elements() as usize;
+    let total_bytes = checked_byte_len(dataspace.checked_num_elements()?, elem_size)?;
-    let total_bytes = total_elements * elem_size;
+    if total_bytes == 0 {
-    let mut output = vec![0u8; total_bytes];
+        // Also keeps the stride products below in range: with a zero-sized
+        // dimension the total is 0 even if other dimensions are huge.
+        return Ok(Vec::new());
+    }
+    let mut output = alloc_output(total_bytes)?;
 
     let mut ds_strides = vec![1usize; rank];
     for i in (0..rank.saturating_sub(1)).rev() {
@@ -1056,8 +1131,7 @@ pub fn read_chunked_data_sweep(
         chunk_strides[i] = chunk_strides[i + 1] * chunk_dims[i + 1];
     }
 
-    let chunk_total_elements: usize = chunk_dims.iter().product();
+    let chunk_total_bytes = checked_chunk_byte_len(&chunk_dims, elem_size)?;
-    let chunk_total_bytes = chunk_total_elements * elem_size;
 
     for chunk_info in &chunks {
         let coord: Vec<u64> = chunk_info.offsets.iter().take(rank).copied().collect();
@@ -1199,7 +1273,7 @@ pub fn read_chunked_data_indexed(
         let chunks = match (version, chunk_index_type) {
             (3, _) => collect_chunk_info(file_data, addr, ndims, offset_size, length_size)?,
             (4, Some(1)) => {
-                let chunk_byte_size: usize = chunk_dims.iter().product::<usize>() * elem_size;
+                let chunk_byte_size = checked_chunk_byte_len(&chunk_dims, elem_size)?;
                 let (csize, fmask) = if let Some(fs) = single_filtered_size {
                     (fs as u32, single_filter_mask.unwrap_or(0))
                 } else {
@@ -1463,6 +1537,64 @@ fn copy_chunk_to_output(
 mod tests {
     use super::*;
 
+    fn simple_space(dimensions: Vec<u64>) -> Dataspace {
+        Dataspace {
+            space_type: crate::dataspace::DataspaceType::Simple,
+            rank: dimensions.len() as u8,
+            dimensions,
+            max_dimensions: None,
+        }
+    }
+
+    #[test]
+    fn crafted_dimensions_are_errors_not_wraparound() {
+        // 2^63 * 2 wraps to 0 with a plain product; 2^40 * 2^40 wraps too.
+        for dims in [
+            vec![1u64 << 63, 2],
+            vec![1 << 40, 1 << 40],
+            vec![u64::MAX, u64::MAX],
+        ] {
+            let space = simple_space(dims.clone());
+            assert!(
+                matches!(space.checked_num_elements(), Err(FormatError::Overflow(_))),
+                "{dims:?}"
+            );
+            // The infallible accessor saturates instead of wrapping.
+            assert_eq!(space.num_elements(), u64::MAX, "{dims:?}");
+        }
+        assert_eq!(simple_space(vec![3, 4]).checked_num_elements().unwrap(), 12);
+        // A zero-sized dimension makes the whole product 0, not an overflow.
+        assert_eq!(
+            simple_space(vec![0, 1 << 40, 1 << 40])
+                .checked_num_elements()
+                .unwrap(),
+            0
+        );
+    }
+
+    #[test]
+    fn byte_length_helpers_check_overflow() {
+        assert_eq!(checked_byte_len(10, 8).unwrap(), 80);
+        assert!(matches!(
+            checked_byte_len(u64::MAX, 8),
+            Err(FormatError::Overflow(_))
+        ));
+        assert_eq!(checked_chunk_byte_len(&[10, 10], 4).unwrap(), 400);
+        assert!(matches!(
+            checked_chunk_byte_len(&[usize::MAX, 2], 4),
+            Err(FormatError::Overflow(_))
+        ));
+    }
+
+    #[test]
+    fn unallocatable_output_is_an_error_not_an_abort() {
+        assert_eq!(alloc_output(16).unwrap(), vec![0u8; 16]);
+        assert!(matches!(
+            alloc_output(usize::MAX / 2),
+            Err(FormatError::Overflow(_))
+        ));
+    }
+
     fn write_offset(buf: &mut Vec<u8>, val: u64, size: u8) {
         match size {
             4 => buf.extend_from_slice(&(val as u32).to_le_bytes()),
@@ -1657,9 +1789,9 @@ mod tests {
             let chunk_bytes = chunk_size_elems * elem_size; // full chunk allocation
 
             // Write chunk data (full chunk size, padding with zeros)
-            for i in start..end {
+            for (i, value) in values.iter().enumerate().take(end).skip(start) {
                 let byte_offset = data_offset + (i - start) * elem_size;
-                file_data[byte_offset..byte_offset + 8].copy_from_slice(&values[i].to_le_bytes());
+                file_data[byte_offset..byte_offset + 8].copy_from_slice(&value.to_le_bytes());
             }
 
             chunk_infos.push(ChunkInfo {
@@ -1837,8 +1969,8 @@ mod tests {
         for chunk_idx in 0..2 {
             let start = chunk_idx * chunk_elems;
             let mut chunk_bytes = Vec::new();
-            for i in start..start + chunk_elems {
+            for value in values.iter().skip(start).take(chunk_elems) {
-                chunk_bytes.extend_from_slice(&values[i].to_le_bytes());
+                chunk_bytes.extend_from_slice(&value.to_le_bytes());
             }
             let compressed = compress_chunk(&chunk_bytes, &pipeline, elem_size as u32).unwrap();
 

crates/clawhdf5-format/src/data_read.rs

@@ -475,8 +475,10 @@ fn read_virtual_data(
     use crate::selection::Selection;
 
     let elem_size = datatype.type_size() as usize;
-    let total_elems = dataspace.num_elements() as usize;
+    let mut out = crate::chunked_read::alloc_output(crate::chunked_read::checked_byte_len(
-    let mut out = vec![0u8; total_elems.saturating_mul(elem_size)];
+        dataspace.checked_num_elements()?,
+        elem_size,
+    )?)?;
 
     let virtual_dims = &dataspace.dimensions;
 
@@ -598,7 +600,7 @@ fn read_named_dataset_raw(
 }
 
 /// Extract selected elements from a full dataset buffer.
-fn extract_selection_from_buffer(
+pub fn extract_selection_from_buffer(
     full_data: &[u8],
     dims: &[u64],
     elem_size: usize,
@@ -616,12 +618,14 @@ fn extract_selection_from_buffer(
             block,
         } => {
             let rank = dims.len();
-            let output_elements: usize = count
+            let output_elements = count
                 .iter()
                 .zip(block.iter())
-                .map(|(&c, &b)| (c * b) as usize)
+                .try_fold(1u64, |acc, (&c, &b)| acc.checked_mul(c.checked_mul(b)?))
-                .product();
+                .ok_or_else(|| FormatError::Overflow("hyperslab count x block overflows".into()))?;
-            let mut output = vec![0u8; output_elements * elem_size];
+            let mut output = crate::chunked_read::alloc_output(
+                crate::chunked_read::checked_byte_len(output_elements, elem_size)?,
+            )?;
 
             // Compute dataset strides (row-major)
             let mut ds_strides = vec![1usize; rank];
@@ -1763,11 +1767,11 @@ mod tests {
     fn f16_bits(v: f32) -> u16 {
         // Encode a few exact values used by the test.
         match v {
-            x if x == 0.0 => 0x0000,
+            0.0 => 0x0000,
-            x if x == 1.0 => 0x3c00,
+            1.0 => 0x3c00,
-            x if x == -2.0 => 0xc000,
+            -2.0 => 0xc000,
-            x if x == 0.5 => 0x3800,
+            0.5 => 0x3800,
-            x if x == 65504.0 => 0x7bff, // f16 max
+            65504.0 => 0x7bff, // f16 max
             _ => panic!("unsupported test value {v}"),
         }
     }
@@ -2186,7 +2190,7 @@ mod tests {
             ],
         };
         let mut raw = Vec::new();
-        raw.extend_from_slice(&3.14f64.to_le_bytes());
+        raw.extend_from_slice(&3.25f64.to_le_bytes());
         raw.extend_from_slice(&42i32.to_le_bytes());
 
         let field = read_compound_field(&raw, &dt, "id").unwrap();

crates/clawhdf5-format/src/dataspace.rs

@@ -1,5 +1,7 @@
 //! HDF5 Dataspace message parsing (message type 0x0001).
 
+#[cfg(not(feature = "std"))]
+use alloc::format;
 #[cfg(not(feature = "std"))]
 use alloc::vec::Vec;
 
@@ -167,6 +169,27 @@ impl Dataspace {
         }
     }
 
+    /// [`Dataspace::num_elements`] with the product overflow-checked. The
+    /// dimensions are untrusted 64-bit fields; read paths that size a buffer
+    /// from them must use this one.
+    pub fn checked_num_elements(&self) -> Result<u64, FormatError> {
+        match self.space_type {
+            DataspaceType::Null => Ok(0),
+            DataspaceType::Scalar => Ok(1),
+            DataspaceType::Simple if self.dimensions.is_empty() => Ok(0),
+            DataspaceType::Simple => self
+                .dimensions
+                .iter()
+                .try_fold(1u64, |acc, &d| acc.checked_mul(d))
+                .ok_or_else(|| {
+                    FormatError::Overflow(format!(
+                        "dataspace dimensions {:?} overflow the element count",
+                        self.dimensions
+                    ))
+                }),
+        }
+    }
+
     /// Total number of elements. Scalar = 1, Null = 0.
     pub fn num_elements(&self) -> u64 {
         match self.space_type {
@@ -176,7 +199,12 @@ impl Dataspace {
                 if self.dimensions.is_empty() {
                     0
                 } else {
-                    self.dimensions.iter().product()
+                    // Saturate rather than wrap: a wrapped product could
+                    // under-size a buffer. Size-critical callers use
+                    // `checked_num_elements`.
+                    self.dimensions
+                        .iter()
+                        .fold(1u64, |acc, &d| acc.saturating_mul(d))
                 }
             }
         }
@@ -189,11 +217,7 @@ mod tests {
 
     fn build_v1_dataspace(rank: u8, flags: u8, dims: &[u64], max_dims: Option<&[u64]>) -> Vec<u8> {
         let length_size = 8u8;
-        let mut buf = Vec::new();
+        let mut buf = vec![1, rank, flags, 0]; // version, rank, flags, reserved
-        buf.push(1); // version
-        buf.push(rank);
-        buf.push(flags);
-        buf.push(0); // reserved
         buf.extend_from_slice(&[0u8; 4]); // reserved(4)
         for &d in dims {
             buf.extend_from_slice(&d.to_le_bytes());
@@ -214,11 +238,7 @@ mod tests {
         dims: &[u64],
         max_dims: Option<&[u64]>,
     ) -> Vec<u8> {
-        let mut buf = Vec::new();
+        let mut buf = vec![2, rank, flags, type_byte]; // version, rank, flags, type
-        buf.push(2); // version
-        buf.push(rank);
-        buf.push(flags);
-        buf.push(type_byte);
         for &d in dims {
             buf.extend_from_slice(&d.to_le_bytes());
         }
@@ -298,11 +318,7 @@ mod tests {
 
     #[test]
     fn v1_with_4byte_length() {
-        let mut buf = Vec::new();
+        let mut buf = vec![1, 1, 0, 0]; // version, rank, flags, reserved
-        buf.push(1); // version
-        buf.push(1); // rank
-        buf.push(0); // flags
-        buf.push(0); // reserved
         buf.extend_from_slice(&[0u8; 4]); // reserved(4)
         buf.extend_from_slice(&10u32.to_le_bytes()); // dim with length_size=4
         let ds = Dataspace::parse(&buf, 4).unwrap();

crates/clawhdf5-format/src/datatype.rs

@@ -372,7 +372,8 @@ impl Datatype {
                         pos += name_len;
                         let byte_offset = read_uint(data, pos, ob)?;
                         pos += ob;
-                        let (member_dt, consumed) = Self::parse_with_depth(&data[pos..], depth + 1)?;
+                        let (member_dt, consumed) =
+                            Self::parse_with_depth(&data[pos..], depth + 1)?;
                         pos += consumed;
                         members.push(CompoundMember {
                             name,
@@ -381,24 +382,29 @@ impl Datatype {
                         });
                     }
                 } else if version == 1 || version == 2 {
-                    // v1/v2: name, offset(4), dimensionality(1), reserved(3), dim_perm(4),
+                    // v1/v2: name (null-terminated, padded to a multiple of 8
-                    //         reserved_dims(up to 4*4=16), member datatype
+                    // bytes), offset(4), member datatype. v1 additionally
+                    // carries the legacy per-member array fields between the
+                    // offset and the member datatype: dimensionality(1),
+                    // reserved(3), dim_perm(4), reserved(4), 4 dim sizes(16).
+                    // v1 is what default (non-`latest`) libver bounds emit.
                     for _ in 0..num_members {
                         let (name, name_len) = read_null_terminated_string(data, pos)?;
-                        pos += name_len;
+                        let padded = name_len.checked_add(7).ok_or(FormatError::UnexpectedEof {
-                        // v1: names padded to 8-byte boundary
+                            expected: usize::MAX,
-                        if version == 1 {
+                            available: data.len(),
-                            let total_name_bytes = name_len;
+                        })? & !7;
-                            let padded = (total_name_bytes + 7) & !7;
+                        ensure_len(data, pos, padded)?;
-                            pos = pos - name_len + padded;
+                        pos += padded;
-                        }
                         ensure_len(data, pos, 4)?;
                         let byte_offset = LittleEndian::read_u32(&data[pos..pos + 4]) as u64;
                         pos += 4;
-                        // dimensionality(1) + reserved(3) + dim_perm(4) + 4 dim slots(16) = 24
+                        if version == 1 {
-                        ensure_len(data, pos, 24)?;
+                            ensure_len(data, pos, 28)?;
-                        pos += 24;
+                            pos += 28;
-                        let (member_dt, consumed) = Self::parse_with_depth(&data[pos..], depth + 1)?;
+                        }
+                        let (member_dt, consumed) =
+                            Self::parse_with_depth(&data[pos..], depth + 1)?;
                         pos += consumed;
                         members.push(CompoundMember {
                             name,
@@ -1138,6 +1144,82 @@ mod tests {
         }
     }
 
+    /// Real datatype message bytes emitted by h5py 3.16 / HDF5 2.0 with
+    /// *default* libver bounds for [('x','f8'),('y','f8'),('id','i4')]:
+    /// compound datatype version 1 (padded names + 28 bytes of legacy
+    /// per-member array fields).
+    fn compound_v1_bytes() -> Vec<u8> {
+        let f64le: [u8; 20] = [
+            0x11, 0x20, 0x3f, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x34, 0x0b,
+            0x00, 0x34, 0xff, 0x03, 0x00, 0x00,
+        ];
+        let i32le: [u8; 12] = [
+            0x10, 0x08, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x00,
+        ];
+        let mut b = vec![0x16, 0x03, 0x00, 0x00, 0x14, 0x00, 0x00, 0x00];
+        for (name, offset, dt) in [
+            (&b"x"[..], 0u32, &f64le[..]),
+            (&b"y"[..], 8, &f64le[..]),
+            (&b"id"[..], 16, &i32le[..]),
+        ] {
+            let mut padded = name.to_vec();
+            padded.resize((name.len() + 1 + 7) & !7, 0);
+            b.extend_from_slice(&padded);
+            b.extend_from_slice(&offset.to_le_bytes());
+            b.extend_from_slice(&[0u8; 28]);
+            b.extend_from_slice(dt);
+        }
+        b
+    }
+
+    fn assert_xyid_compound(dt: Datatype) {
+        match dt {
+            Datatype::Compound { size, members } => {
+                assert_eq!(size, 20);
+                let got: Vec<(&str, u64, u32)> = members
+                    .iter()
+                    .map(|m| (m.name.as_str(), m.byte_offset, m.datatype.type_size()))
+                    .collect();
+                assert_eq!(got, vec![("x", 0, 8), ("y", 8, 8), ("id", 16, 4)]);
+            }
+            other => panic!("expected Compound, got {other:?}"),
+        }
+    }
+
+    #[test]
+    fn test_compound_v1_default_libver() {
+        let bytes = compound_v1_bytes();
+        let (dt, consumed) = Datatype::parse(&bytes).unwrap();
+        assert_eq!(consumed, bytes.len());
+        assert_xyid_compound(dt);
+    }
+
+    #[test]
+    fn test_compound_v2_padded_names_no_array_fields() {
+        // v2 = v1 without the 28 bytes of per-member array fields; names are
+        // still padded to a multiple of 8 (matches libhdf5's H5O decoder).
+        let v1 = compound_v1_bytes();
+        let mut v2 = vec![0x26, 0x03, 0x00, 0x00, 0x14, 0x00, 0x00, 0x00];
+        let mut pos = 8;
+        for dt_len in [20usize, 20, 12] {
+            v2.extend_from_slice(&v1[pos..pos + 8 + 4]); // padded name + offset
+            pos += 8 + 4 + 28;
+            v2.extend_from_slice(&v1[pos..pos + dt_len]);
+            pos += dt_len;
+        }
+        let (dt, consumed) = Datatype::parse(&v2).unwrap();
+        assert_eq!(consumed, v2.len());
+        assert_xyid_compound(dt);
+    }
+
+    #[test]
+    fn test_compound_v1_truncated_is_error_not_panic() {
+        let bytes = compound_v1_bytes();
+        for cut in 8..bytes.len() {
+            assert!(Datatype::parse(&bytes[..cut]).is_err(), "cut at {cut}");
+        }
+    }
+
     /// Real datatype message bytes emitted by HDF5 2.0 for the native complex
     /// type `H5T_COMPLEX_IEEE_F64LE`: class 11, version 5, size 16, followed by
     /// the base IEEE f64 datatype message.
@@ -1172,7 +1254,9 @@ mod tests {
         // Compound { z: complex f64 @0, k: i64 @16 } as written by HDF5 2.0.
         // Regression guard: the complex member must consume exactly its own
         // bytes so the following member parses.
-        let mut bytes = vec![0x56, 0x02, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, b'z', 0x00, 0x00];
+        let mut bytes = vec![
+            0x56, 0x02, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00, b'z', 0x00, 0x00,
+        ];
         bytes.extend_from_slice(&COMPLEX_F64_HDF5_2_0);
         bytes.extend_from_slice(&[b'k', 0x00, 0x10]);
         bytes.extend_from_slice(&[
@@ -1188,7 +1272,10 @@ mod tests {
                     &members[0].datatype,
                     Datatype::Compound { size: 16, members } if members.len() == 2
                 ));
-                assert_eq!((members[1].name.as_str(), members[1].byte_offset), ("k", 16));
+                assert_eq!(
+                    (members[1].name.as_str(), members[1].byte_offset),
+                    ("k", 16)
+                );
             }
             other => panic!("expected Compound, got {other:?}"),
         }

crates/clawhdf5-format/src/error.rs

@@ -114,6 +114,20 @@ pub enum FormatError {
     InvalidAttributeInfoVersion(u8),
     /// Invalid shared message version.
     InvalidSharedMessageVersion(u8),
+    /// A message is marked shared but was parsed without access to the file,
+    /// so the reference to the real message could not be followed.
+    UnresolvedSharedMessage,
+    /// The dataset's raw data is stored in external files (External Data
+    /// Files message), which this reader does not follow.
+    ExternalDataFilesUnsupported,
+    /// The path goes through an external link (a link into another file),
+    /// which this reader does not follow.
+    ExternalLinkUnsupported {
+        /// The file the link points into.
+        filename: String,
+        /// The object path within that file.
+        object_path: String,
+    },
     /// Invalid SOHM table version.
     InvalidSohmTableVersion(u8),
     /// Invalid SOHM table signature (expected "SMTB").
@@ -307,6 +321,22 @@ impl fmt::Display for FormatError {
             FormatError::InvalidSharedMessageVersion(v) => {
                 write!(f, "invalid shared message version: {v}")
             }
+            FormatError::ExternalLinkUnsupported {
+                filename,
+                object_path,
+            } => write!(
+                f,
+                "path goes through an external link to {object_path} in {filename}, which is \
+                 not supported"
+            ),
+            FormatError::ExternalDataFilesUnsupported => write!(
+                f,
+                "dataset raw data is stored in external file(s), which is not supported"
+            ),
+            FormatError::UnresolvedSharedMessage => write!(
+                f,
+                "message is shared but no file data was available to resolve it"
+            ),
             FormatError::InvalidSohmTableVersion(v) => {
                 write!(f, "invalid SOHM table version: {v}")
             }

crates/clawhdf5-format/src/fill_value.rs

@@ -0,0 +1,407 @@
+//! Fill Value messages (0x0005, and the old 0x0004) and applying them on read.
+//!
+//! HDF5 allocates storage lazily: a chunk nobody wrote to does not exist in the
+//! file, and a contiguous dataset nobody wrote to has no data address at all.
+//! Reading such a region must yield the dataset's *fill value* (zeros unless
+//! the creator chose otherwise). The readers in [`crate::chunked_read`] leave
+//! those regions zeroed; [`apply_to_unallocated_chunks`] then overwrites exactly
+//! the chunk-grid cells that are absent from the chunk index — so it can never
+//! mistake a stored zero for a hole — and is skipped entirely in the common
+//! case of a zero fill value.
+
+#[cfg(not(feature = "std"))]
+use alloc::{format, vec, vec::Vec};
+
+use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks};
+use crate::data_layout::DataLayout;
+use crate::dataspace::Dataspace;
+use crate::error::FormatError;
+use crate::message_type::MessageType;
+use crate::object_header::HeaderMessage;
+
+/// Largest fill value accepted. A fill value is one element of the dataset's
+/// datatype; this only bounds the allocation driven by the message's size field.
+const MAX_FILL_VALUE_SIZE: usize = 1 << 20;
+
+/// Parse a Fill Value message, returning the user-defined fill value bytes, or
+/// `None` when the dataset uses the default (all zeros) or has the fill value
+/// explicitly undefined.
+pub fn parse_fill_value(msg: &HeaderMessage) -> Result<Option<Vec<u8>>, FormatError> {
+    let data = msg.data.as_slice();
+    let value_at = |pos: usize| -> Result<Option<Vec<u8>>, FormatError> {
+        let size_bytes = data.get(pos..pos + 4).ok_or(FormatError::UnexpectedEof {
+            expected: pos + 4,
+            available: data.len(),
+        })?;
+        let size = u32::from_le_bytes([size_bytes[0], size_bytes[1], size_bytes[2], size_bytes[3]])
+            as usize;
+        if size == 0 {
+            return Ok(None);
+        }
+        if size > MAX_FILL_VALUE_SIZE {
+            return Err(FormatError::Overflow(format!(
+                "fill value of {size} bytes exceeds the {MAX_FILL_VALUE_SIZE}-byte limit"
+            )));
+        }
+        let start = pos + 4;
+        let value =
+            data.get(start..start.saturating_add(size))
+                .ok_or(FormatError::UnexpectedEof {
+                    expected: start.saturating_add(size),
+                    available: data.len(),
+                })?;
+        Ok(Some(value.to_vec()))
+    };
+
+    match msg.msg_type {
+        // Old fill value message: size(4), value.
+        MessageType::FillValueOld => value_at(0),
+        MessageType::FillValue => {
+            let version = *data.first().ok_or(FormatError::UnexpectedEof {
+                expected: 1,
+                available: 0,
+            })?;
+            match version {
+                // version, alloc time, write time, defined, [size, value]
+                1 | 2 => {
+                    let defined = *data.get(3).ok_or(FormatError::UnexpectedEof {
+                        expected: 4,
+                        available: data.len(),
+                    })?;
+                    if version == 2 && defined == 0 {
+                        Ok(None)
+                    } else if data.len() < 8 && version == 1 {
+                        // v1 always carries a size, but tolerate its absence.
+                        Ok(None)
+                    } else {
+                        value_at(4)
+                    }
+                }
+                // version, flags (bit 4 = undefined, bit 5 = defined), [size, value]
+                3 => {
+                    let flags = *data.get(1).ok_or(FormatError::UnexpectedEof {
+                        expected: 2,
+                        available: data.len(),
+                    })?;
+                    if flags & 0x10 != 0 || flags & 0x20 == 0 {
+                        Ok(None)
+                    } else {
+                        value_at(2)
+                    }
+                }
+                v => Err(FormatError::UnsupportedVersion(v)),
+            }
+        }
+        _ => Ok(None),
+    }
+}
+
+/// The fill value that applies to a dataset given its header messages. The new
+/// message wins over the old one when both are present.
+pub fn dataset_fill_value(messages: &[HeaderMessage]) -> Result<Option<Vec<u8>>, FormatError> {
+    for wanted in [MessageType::FillValue, MessageType::FillValueOld] {
+        if let Some(msg) = messages.iter().find(|m| m.msg_type == wanted) {
+            if crate::shared_message::is_shared(msg.flags) {
+                // A shared fill value is legal but vanishingly rare; treat it
+                // as the default rather than misparsing the reference.
+                return Ok(None);
+            }
+            if let Some(value) = parse_fill_value(msg)? {
+                return Ok(Some(value));
+            }
+        }
+    }
+    Ok(None)
+}
+
+/// `true` when a fill value is absent or all zeros, i.e. identical to what the
+/// readers already produce for unallocated storage.
+pub fn is_default(fill: Option<&[u8]>) -> bool {
+    fill.is_none_or(|f| f.iter().all(|&b| b == 0))
+}
+
+/// A whole dataset's worth of fill value: what reading a dataset with no
+/// allocated storage at all must return.
+pub fn filled_dataset(
+    dataspace: &Dataspace,
+    elem_size: usize,
+    fill: Option<&[u8]>,
+) -> Result<Vec<u8>, FormatError> {
+    let total = checked_byte_len(dataspace.checked_num_elements()?, elem_size)?;
+    let mut out = alloc_output(total)?;
+    if let Some(fill) = fill.filter(|f| f.len() == elem_size && !is_default(Some(f))) {
+        for element in out.chunks_exact_mut(elem_size) {
+            element.copy_from_slice(fill);
+        }
+    }
+    Ok(out)
+}
+
+/// Whether the layout has any storage in the file at all. A dataset that was
+/// created but never written to has none.
+pub fn has_storage(layout: &DataLayout) -> bool {
+    !matches!(
+        layout,
+        DataLayout::Contiguous { address: None, .. }
+            | DataLayout::Chunked {
+                btree_address: None,
+                ..
+            }
+    )
+}
+
+/// Run a full-dataset `read`, giving unallocated storage its fill value: a
+/// dataset with no storage at all reads as entirely fill value (instead of
+/// failing), and a chunked dataset has the fill value written into every
+/// chunk the file never allocated.
+#[allow(clippy::too_many_arguments)]
+pub fn read_full_with_fill<E: From<FormatError>>(
+    messages: &[HeaderMessage],
+    file_data: &[u8],
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    offset_size: u8,
+    length_size: u8,
+    read: impl FnOnce() -> Result<Vec<u8>, E>,
+) -> Result<Vec<u8>, E> {
+    // A dataset with external raw data also has no data address in this
+    // file. It is NOT unallocated — its values live elsewhere — so it must
+    // never be answered with the fill value.
+    if messages
+        .iter()
+        .any(|m| m.msg_type == MessageType::ExternalDataFiles)
+    {
+        return Err(FormatError::ExternalDataFilesUnsupported.into());
+    }
+    let fill = dataset_fill_value(messages)?;
+    if !has_storage(layout) {
+        return Ok(filled_dataset(dataspace, elem_size, fill.as_deref())?);
+    }
+    let mut output = read()?;
+    apply_to_unallocated_chunks(
+        &mut output,
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        fill.as_deref(),
+        offset_size,
+        length_size,
+    )?;
+    Ok(output)
+}
+
+/// Overwrite, in a fully read chunked dataset `output`, every region whose
+/// chunk was never allocated with `fill`. No-op for non-chunked layouts, a
+/// default fill value, or a fill value whose size doesn't match the element.
+#[allow(clippy::too_many_arguments)]
+pub fn apply_to_unallocated_chunks(
+    output: &mut [u8],
+    file_data: &[u8],
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    fill: Option<&[u8]>,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<(), FormatError> {
+    let Some(fill) = fill.filter(|f| f.len() == elem_size && !is_default(Some(f))) else {
+        return Ok(());
+    };
+    if !matches!(layout, DataLayout::Chunked { .. }) || elem_size == 0 {
+        return Ok(());
+    }
+    let (chunks, chunk_dims) = list_chunks(
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        offset_size,
+        length_size,
+    )?;
+    let rank = chunk_dims.len();
+    let ds_dims: Vec<usize> = dataspace.dimensions.iter().map(|&d| d as usize).collect();
+    if rank == 0 || ds_dims.len() != rank || chunk_dims.contains(&0) {
+        return Ok(());
+    }
+
+    // Row-major strides over the dataset and over the chunk grid.
+    let mut ds_strides = vec![1usize; rank];
+    for i in (0..rank - 1).rev() {
+        ds_strides[i] = ds_strides[i + 1].saturating_mul(ds_dims[i + 1]);
+    }
+    let grid: Vec<usize> = ds_dims
+        .iter()
+        .zip(&chunk_dims)
+        .map(|(&d, &c)| d.div_ceil(c))
+        .collect();
+    let cells = grid
+        .iter()
+        .try_fold(1usize, |acc, &g| acc.checked_mul(g))
+        .ok_or_else(|| FormatError::Overflow("chunk grid size overflows".into()))?;
+    if cells == 0 {
+        return Ok(());
+    }
+
+    let mut allocated = vec![false; cells];
+    for chunk in &chunks {
+        // Undefined address: the index has a slot for the chunk but no storage.
+        if chunk.address == u64::MAX || chunk.offsets.len() < rank {
+            continue;
+        }
+        let mut cell = 0usize;
+        let mut in_range = true;
+        for d in 0..rank {
+            let coord = chunk.offsets[d] as usize / chunk_dims[d];
+            if coord >= grid[d] {
+                in_range = false;
+                break;
+            }
+            cell = cell * grid[d] + coord;
+        }
+        if in_range {
+            allocated[cell] = true;
+        }
+    }
+
+    let mut coord = vec![0usize; rank];
+    for (cell, is_allocated) in allocated.iter().enumerate() {
+        if *is_allocated {
+            continue;
+        }
+        // Decode the cell index into grid coordinates.
+        let mut rem = cell;
+        for d in (0..rank).rev() {
+            coord[d] = rem % grid[d];
+            rem /= grid[d];
+        }
+        fill_cell(
+            output,
+            &coord,
+            &chunk_dims,
+            &ds_dims,
+            &ds_strides,
+            elem_size,
+            fill,
+        );
+    }
+    Ok(())
+}
+
+/// Fill the part of chunk-grid cell `coord` that lies inside the dataset.
+fn fill_cell(
+    output: &mut [u8],
+    coord: &[usize],
+    chunk_dims: &[usize],
+    ds_dims: &[usize],
+    ds_strides: &[usize],
+    elem_size: usize,
+    fill: &[u8],
+) {
+    let rank = coord.len();
+    let start: Vec<usize> = (0..rank).map(|d| coord[d] * chunk_dims[d]).collect();
+    let end: Vec<usize> = (0..rank)
+        .map(|d| (start[d] + chunk_dims[d]).min(ds_dims[d]))
+        .collect();
+    if (0..rank).any(|d| start[d] >= end[d]) {
+        return;
+    }
+    // Walk every row (all dims but the last) and fill the run along the last.
+    let run = end[rank - 1] - start[rank - 1];
+    let mut idx = start.clone();
+    loop {
+        let first: usize = (0..rank).map(|d| idx[d] * ds_strides[d]).sum();
+        let from = first * elem_size;
+        let to = from + run * elem_size;
+        if let Some(region) = output.get_mut(from..to) {
+            for element in region.chunks_exact_mut(elem_size) {
+                element.copy_from_slice(fill);
+            }
+        }
+        // Advance the odometer over dims 0..rank-1.
+        let mut d = rank - 1;
+        loop {
+            if d == 0 {
+                return;
+            }
+            d -= 1;
+            idx[d] += 1;
+            if idx[d] < end[d] {
+                break;
+            }
+            idx[d] = start[d];
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn msg(msg_type: MessageType, data: &[u8]) -> HeaderMessage {
+        HeaderMessage {
+            msg_type,
+            size: data.len(),
+            flags: 0,
+            creation_order: None,
+            data: data.to_vec(),
+        }
+    }
+
+    #[test]
+    fn parses_v3_defined_undefined_and_default() {
+        // Real message for h5py `fillvalue=-1` on an i4 dataset (HDF5 2.0).
+        let defined = msg(
+            MessageType::FillValue,
+            &[3, 0x2b, 4, 0, 0, 0, 0xff, 0xff, 0xff, 0xff],
+        );
+        assert_eq!(parse_fill_value(&defined).unwrap(), Some(vec![0xff; 4]));
+        let default = msg(MessageType::FillValue, &[3, 0x0a]);
+        assert_eq!(parse_fill_value(&default).unwrap(), None);
+        let undefined = msg(MessageType::FillValue, &[3, 0x19]);
+        assert_eq!(parse_fill_value(&undefined).unwrap(), None);
+    }
+
+    #[test]
+    fn parses_v2_and_old_messages() {
+        let v2 = msg(MessageType::FillValue, &[2, 2, 2, 1, 2, 0, 0, 0, 7, 0]);
+        assert_eq!(parse_fill_value(&v2).unwrap(), Some(vec![7, 0]));
+        let v2_undefined = msg(MessageType::FillValue, &[2, 2, 2, 0]);
+        assert_eq!(parse_fill_value(&v2_undefined).unwrap(), None);
+        let old = msg(MessageType::FillValueOld, &[2, 0, 0, 0, 9, 9]);
+        assert_eq!(parse_fill_value(&old).unwrap(), Some(vec![9, 9]));
+    }
+
+    #[test]
+    fn truncated_or_oversized_fill_is_an_error() {
+        let short = msg(MessageType::FillValue, &[3, 0x29, 4, 0, 0, 0, 0xff]);
+        assert!(parse_fill_value(&short).is_err());
+        let huge = msg(MessageType::FillValue, &[3, 0x29, 0xff, 0xff, 0xff, 0x7f]);
+        assert!(matches!(
+            parse_fill_value(&huge),
+            Err(FormatError::Overflow(_))
+        ));
+    }
+
+    #[test]
+    fn fill_cell_clips_edge_chunks_in_2d() {
+        // 3x5 dataset, 2x2 chunks; fill grid cell (1, 2): rows 2..3, cols 4..5.
+        let mut out = vec![0u8; 15];
+        fill_cell(&mut out, &[1, 2], &[2, 2], &[3, 5], &[5, 1], 1, &[9]);
+        let mut expected = vec![0u8; 15];
+        expected[2 * 5 + 4] = 9;
+        assert_eq!(out, expected);
+
+        // Interior cell (0, 1): rows 0..2, cols 2..4.
+        let mut out = vec![0u8; 15];
+        fill_cell(&mut out, &[0, 1], &[2, 2], &[3, 5], &[5, 1], 1, &[7]);
+        let filled: Vec<usize> = out
+            .iter()
+            .enumerate()
+            .filter(|(_, b)| **b == 7)
+            .map(|(i, _)| i)
+            .collect();
+        assert_eq!(filled, [2, 3, 7, 8]);
+    }
+}

crates/clawhdf5-format/src/filters.rs

@@ -1045,24 +1045,30 @@ fn pcodec_compress(data: &[u8], element_size: usize) -> Result<Vec<u8>, FormatEr
     match element_size {
         4 => {
             let nums: Vec<f32> = data
-                .chunks_exact(4)
+                .as_chunks::<4>()
-                .map(|b| f32::from_le_bytes(b.try_into().unwrap()))
+                .0
+                .iter()
+                .map(|b| f32::from_le_bytes(*b))
                 .collect();
             simple_compress(&nums, &config)
                 .map_err(|e| FormatError::CompressionError(format!("pco: {e}")))
         }
         8 => {
             let nums: Vec<f64> = data
-                .chunks_exact(8)
+                .as_chunks::<8>()
-                .map(|b| f64::from_le_bytes(b.try_into().unwrap()))
+                .0
+                .iter()
+                .map(|b| f64::from_le_bytes(*b))
                 .collect();
             simple_compress(&nums, &config)
                 .map_err(|e| FormatError::CompressionError(format!("pco: {e}")))
         }
         _ => {
             let nums: Vec<u32> = data
-                .chunks_exact(4)
+                .as_chunks::<4>()
-                .map(|b| u32::from_le_bytes(b.try_into().unwrap()))
+                .0
+                .iter()
+                .map(|b| u32::from_le_bytes(*b))
                 .collect();
             simple_compress(&nums, &config)
                 .map_err(|e| FormatError::CompressionError(format!("pco: {e}")))
@@ -1092,11 +1098,7 @@ fn pcodec_decompress(
     } else {
         MAX_DECOMPRESS_SIZE
     };
-    let n = if element_size != 0 {
+    let n = limit_bytes.checked_div(element_size).unwrap_or(0);
-        limit_bytes / element_size
-    } else {
-        0
-    };
     match element_size {
         4 => {
             let mut buf = vec![0f32; n];
@@ -1543,8 +1545,10 @@ mod tests {
 
     fn as_f32(bytes: &[u8]) -> Vec<f32> {
         bytes
-            .chunks_exact(4)
+            .as_chunks::<4>()
-            .map(|c| f32::from_le_bytes(c.try_into().unwrap()))
+            .0
+            .iter()
+            .map(|c| f32::from_le_bytes(*c))
             .collect()
     }
 
@@ -1578,8 +1582,10 @@ mod tests {
 
     fn as_f64(bytes: &[u8]) -> Vec<f64> {
         bytes
-            .chunks_exact(8)
+            .as_chunks::<8>()
-            .map(|c| f64::from_le_bytes(c.try_into().unwrap()))
+            .0
+            .iter()
+            .map(|c| f64::from_le_bytes(*c))
             .collect()
     }
 

crates/clawhdf5-format/src/global_heap.rs

@@ -184,9 +184,7 @@ mod tests {
             buf.extend_from_slice(data);
             // Pad to 8 bytes
             let padded = pad8(data.len());
-            for _ in data.len()..padded {
+            buf.resize(buf.len() + (padded - data.len()), 0);
-                buf.push(0);
-            }
         }
 
         // Free space marker

crates/clawhdf5-format/src/group_v1.rs

@@ -60,6 +60,54 @@ pub fn resolve_v1_group_entries(
     Ok(entries)
 }
 
+/// Symbol table cache type for a soft link: the scratch pad's first four bytes
+/// are the local-heap offset of the link's target path, and the entry's object
+/// header address is undefined.
+const CACHE_TYPE_SOFT_LINK: u32 = 2;
+
+/// The target path of the soft link called `name` in a v1 group, if any.
+pub fn find_v1_soft_link(
+    file_data: &[u8],
+    sym_table_msg: &SymbolTableMessage,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Option<String>, FormatError> {
+    let heap = LocalHeap::parse(
+        file_data,
+        sym_table_msg.local_heap_address as usize,
+        offset_size,
+        length_size,
+    )?;
+    let snod_addrs = collect_symbol_table_nodes(
+        file_data,
+        sym_table_msg.btree_address,
+        offset_size,
+        length_size,
+    )?;
+    for snod_addr in snod_addrs {
+        let snod = SymbolTableNode::parse(file_data, snod_addr as usize, offset_size)?;
+        for entry in &snod.entries {
+            if entry.cache_type != CACHE_TYPE_SOFT_LINK {
+                continue;
+            }
+            if heap.read_string(file_data, entry.link_name_offset)? != name {
+                continue;
+            }
+            let value_offset = u32::from_le_bytes([
+                entry.scratch_pad[0],
+                entry.scratch_pad[1],
+                entry.scratch_pad[2],
+                entry.scratch_pad[3],
+            ]);
+            return heap
+                .read_string(file_data, u64::from(value_offset))
+                .map(Some);
+        }
+    }
+    Ok(None)
+}
+
 /// Extract the SymbolTableMessage from an object header's messages.
 fn find_symbol_table_message(
     obj_header: &ObjectHeader,

crates/clawhdf5-format/src/group_v2.rs

@@ -63,14 +63,15 @@ fn resolve_compact_entries(
     Ok(entries)
 }
 
-/// Resolve entries from dense storage (fractal heap + B-tree v2).
+/// Visit every link in dense storage (fractal heap + B-tree v2 name index).
-fn resolve_dense_entries(
+fn for_each_dense_link(
     file_data: &[u8],
     link_info: &LinkInfoMessage,
     fh_addr: u64,
     offset_size: u8,
     length_size: u8,
-) -> Result<Vec<GroupEntry>, FormatError> {
+    mut visit: impl FnMut(LinkMessage),
+) -> Result<(), FormatError> {
     // Parse fractal heap
     let fh = FractalHeapHeader::parse(file_data, fh_addr as usize, offset_size, length_size)?;
 
@@ -81,7 +82,6 @@ fn resolve_dense_entries(
     let btree_hdr = BTreeV2Header::parse(file_data, btree_addr as usize, offset_size, length_size)?;
     let records = collect_btree_v2_records(file_data, &btree_hdr, offset_size, length_size)?;
 
-    let mut entries = Vec::new();
     for record in &records {
         // For type 5 (name index): hash(4) + heap_id(heap_id_length)
         // For type 6 (creation order): creation_order(8) + heap_id(heap_id_length)
@@ -98,9 +98,27 @@ fn resolve_dense_entries(
 
         // Read managed object from fractal heap
         let link_data = fh.read_managed_object(file_data, id_bytes, offset_size)?;
+        visit(LinkMessage::parse(&link_data, offset_size)?);
+    }
+    Ok(())
+}
 
-        // Parse as Link message
+/// Resolve entries from dense storage (fractal heap + B-tree v2).
-        let link = LinkMessage::parse(&link_data, offset_size)?;
+fn resolve_dense_entries(
+    file_data: &[u8],
+    link_info: &LinkInfoMessage,
+    fh_addr: u64,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Vec<GroupEntry>, FormatError> {
+    let mut entries = Vec::new();
+    for_each_dense_link(
+        file_data,
+        link_info,
+        fh_addr,
+        offset_size,
+        length_size,
+        |link| {
             if let LinkTarget::Hard {
                 object_header_address,
             } = link.link_target
@@ -111,9 +129,63 @@ fn resolve_dense_entries(
                     cache_type: 0,
                 });
             }
+        },
+    )?;
+    Ok(entries)
 }
 
-    Ok(entries)
+/// The soft or external link called `name` in this group, if there is one.
+/// Hard links are what `resolve_group_entries` returns; this is consulted only
+/// when a path component isn't among them.
+fn find_symbolic_link(
+    file_data: &[u8],
+    object_header: &ObjectHeader,
+    name: &str,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Option<LinkTarget>, FormatError> {
+    if is_v1_group(object_header) {
+        let Some(sym_msg) = object_header
+            .messages
+            .iter()
+            .find(|m| m.msg_type == MessageType::SymbolTable)
+        else {
+            return Ok(None);
+        };
+        let stm = SymbolTableMessage::parse(&sym_msg.data, offset_size)?;
+        return group_v1::find_v1_soft_link(file_data, &stm, name, offset_size, length_size)
+            .map(|target| target.map(|target_path| LinkTarget::Soft { target_path }));
+    }
+    if !is_v2_group(object_header) {
+        return Ok(None);
+    }
+    let is_symbolic = |t: &LinkTarget| !matches!(t, LinkTarget::Hard { .. });
+    let link_info = find_link_info(object_header, offset_size)?;
+    let mut found = None;
+    if let Some(fh_addr) = link_info.fractal_heap_address {
+        for_each_dense_link(
+            file_data,
+            &link_info,
+            fh_addr,
+            offset_size,
+            length_size,
+            |link| {
+                if link.name == name && is_symbolic(&link.link_target) {
+                    found = Some(link.link_target);
+                }
+            },
+        )?;
+    } else {
+        for msg in &object_header.messages {
+            if msg.msg_type == MessageType::Link {
+                let link = LinkMessage::parse(&msg.data, offset_size)?;
+                if link.name == name && is_symbolic(&link.link_target) {
+                    found = Some(link.link_target);
+                }
+            }
+        }
+    }
+    Ok(found)
 }
 
 /// Find and parse the Link Info message from an object header.
@@ -158,6 +230,19 @@ pub fn resolve_path_any(
     file_data: &[u8],
     superblock: &Superblock,
     path: &str,
+) -> Result<u64, FormatError> {
+    resolve_path_following_links(file_data, superblock, path, 0)
+}
+
+/// Soft links followed while resolving one path. Guards against link cycles
+/// (`a -> b -> a`), which are legal to create.
+const MAX_SOFT_LINK_DEPTH: u8 = 16;
+
+fn resolve_path_following_links(
+    file_data: &[u8],
+    superblock: &Superblock,
+    path: &str,
+    depth: u8,
 ) -> Result<u64, FormatError> {
     let components: Vec<&str> = path.split('/').filter(|s| !s.is_empty()).collect();
     if components.is_empty() {
@@ -176,7 +261,9 @@ pub fn resolve_path_any(
     for (i, component) in components.iter().enumerate() {
         let entries = resolve_group_entries(file_data, &current_header, os, ls)?;
 
-        let found = entries.iter().find(|e| e.name == *component);
+        let found = entries
+            .iter()
+            .find(|e| e.name == *component && e.object_header_address != u64::MAX);
         match found {
             Some(entry) => {
                 if i == components.len() - 1 {
@@ -186,7 +273,37 @@ pub fn resolve_path_any(
                 current_header = ObjectHeader::parse(file_data, current_addr as usize, os, ls)?;
             }
             None => {
-                return Err(FormatError::PathNotFound(String::from(*component)));
+                return match find_symbolic_link(file_data, &current_header, component, os, ls)? {
+                    Some(LinkTarget::Soft { target_path }) => {
+                        if depth >= MAX_SOFT_LINK_DEPTH {
+                            return Err(FormatError::NestingDepthExceeded);
+                        }
+                        // A relative target is relative to the group holding
+                        // the link; then the rest of the original path.
+                        let mut full = String::new();
+                        if !target_path.starts_with('/') {
+                            for parent in &components[..i] {
+                                full.push('/');
+                                full.push_str(parent);
+                            }
+                        }
+                        full.push('/');
+                        full.push_str(&target_path);
+                        for rest in &components[i + 1..] {
+                            full.push('/');
+                            full.push_str(rest);
+                        }
+                        resolve_path_following_links(file_data, superblock, &full, depth + 1)
+                    }
+                    Some(LinkTarget::External {
+                        filename,
+                        object_path,
+                    }) => Err(FormatError::ExternalLinkUnsupported {
+                        filename,
+                        object_path,
+                    }),
+                    _ => Err(FormatError::PathNotFound(String::from(*component))),
+                };
             }
         }
     }

crates/clawhdf5-format/src/lib.rs

@@ -67,6 +67,7 @@ pub mod ea_writer;
 pub mod error;
 pub mod extensible_array;
 pub mod file_writer;
+pub mod fill_value;
 pub mod filter_pipeline;
 pub mod filters;
 mod filters_szip;

crates/clawhdf5-format/src/link_message.rs

@@ -413,11 +413,8 @@ mod tests {
     #[test]
     fn soft_link() {
         let target = "/group1/dataset";
-        let mut data = Vec::new();
+        // version, flags (bit 3 = link type present, name size = 1 byte), link type = soft, name length = 4
-        data.push(1); // version
+        let mut data = vec![1, 0x08, 1, 4];
-        data.push(0x08); // flags: bit 3 = link type present, name size = 1 byte (bits 0-1 = 0)
-        data.push(1); // link type = soft
-        data.push(4); // name length = 4
         data.extend_from_slice(b"link");
         data.extend_from_slice(&(target.len() as u16).to_le_bytes());
         data.extend_from_slice(target.as_bytes());
@@ -455,12 +452,8 @@ mod tests {
 
     #[test]
     fn invalid_link_type() {
-        let mut data = Vec::new();
+        // version, flags (bit 3 = link type present), invalid link type = 99, name length = 1, name = 'x'
-        data.push(1); // version
+        let data = vec![1, 0x08, 99, 1, b'x'];
-        data.push(0x08); // flags: bit 3 = link type present
-        data.push(99); // invalid link type
-        data.push(1); // name length = 1
-        data.push(b'x');
         let err = LinkMessage::parse(&data, 8).unwrap_err();
         assert_eq!(err, FormatError::InvalidLinkType(99));
     }

crates/clawhdf5-format/src/message_type.rs

@@ -9,6 +9,9 @@ pub enum MessageType {
     Datatype,
     FillValueOld,
     FillValue,
+    /// External Data Files (0x0007): the dataset's raw data lives in other
+    /// files, listed by this message.
+    ExternalDataFiles,
     Link,
     DataLayout,
     GroupInfo,
@@ -36,6 +39,7 @@ impl MessageType {
             0x0004 => MessageType::FillValueOld,
             0x0005 => MessageType::FillValue,
             0x0006 => MessageType::Link,
+            0x0007 => MessageType::ExternalDataFiles,
             0x0008 => MessageType::DataLayout,
             0x000A => MessageType::GroupInfo,
             0x000B => MessageType::FilterPipeline,
@@ -60,6 +64,7 @@ impl MessageType {
             MessageType::Datatype => 0x0003,
             MessageType::FillValueOld => 0x0004,
             MessageType::FillValue => 0x0005,
+            MessageType::ExternalDataFiles => 0x0007,
             MessageType::Link => 0x0006,
             MessageType::DataLayout => 0x0008,
             MessageType::GroupInfo => 0x000A,
@@ -90,6 +95,7 @@ mod tests {
             (0x0003, MessageType::Datatype),
             (0x0004, MessageType::FillValueOld),
             (0x0005, MessageType::FillValue),
+            (0x0007, MessageType::ExternalDataFiles),
             (0x0006, MessageType::Link),
             (0x0008, MessageType::DataLayout),
             (0x000A, MessageType::GroupInfo),
@@ -119,8 +125,13 @@ mod tests {
 
     #[test]
     fn unknown_type_zero_gap() {
-        // 0x0007 is not a defined type
+        // 0x0009 is reserved for the library's own testing; no file uses it.
-        let mt = MessageType::from_u16(0x0007);
+        let mt = MessageType::from_u16(0x0009);
-        assert_eq!(mt, MessageType::Unknown(0x0007));
+        assert_eq!(mt, MessageType::Unknown(0x0009));
+        // 0x0007 used to be treated as unknown: it is External Data Files.
+        assert_eq!(
+            MessageType::from_u16(0x0007),
+            MessageType::ExternalDataFiles
+        );
     }
 }

crates/clawhdf5-format/src/parallel_read.rs

@@ -73,9 +73,12 @@ pub fn decompress_chunks_lane_partitioned(
                 let c_addr = chunk_info.address as usize;
                 let size = chunk_info.chunk_size as usize;
 
-                if c_addr + size > file_data.len() {
+                if c_addr
+                    .checked_add(size)
+                    .is_none_or(|end| end > file_data.len())
+                {
                     return Err(FormatError::UnexpectedEof {
-                        expected: c_addr + size,
+                        expected: c_addr.saturating_add(size),
                         available: file_data.len(),
                     });
                 }
@@ -144,9 +147,12 @@ pub fn decompress_chunks_parallel(
         .map(|(index, chunk_info)| {
             let c_addr = chunk_info.address as usize;
             let size = chunk_info.chunk_size as usize;
-            if c_addr + size > file_data.len() {
+            if c_addr
+                .checked_add(size)
+                .is_none_or(|end| end > file_data.len())
+            {
                 return Err(FormatError::UnexpectedEof {
-                    expected: c_addr + size,
+                    expected: c_addr.saturating_add(size),
                     available: file_data.len(),
                 });
             }
@@ -182,9 +188,12 @@ pub fn decompress_chunks_sequential(
     for chunk_info in chunks {
         let c_addr = chunk_info.address as usize;
         let size = chunk_info.chunk_size as usize;
-        if c_addr + size > file_data.len() {
+        if c_addr
+            .checked_add(size)
+            .is_none_or(|end| end > file_data.len())
+        {
             return Err(FormatError::UnexpectedEof {
-                expected: c_addr + size,
+                expected: c_addr.saturating_add(size),
                 available: file_data.len(),
             });
         }

crates/clawhdf5-format/src/selection.rs

@@ -509,7 +509,7 @@ mod tests {
 
     #[test]
     fn selection_slice_1d() {
-        let sel = Selection::slice(&[5..15]);
+        let sel = Selection::slice(std::slice::from_ref(&(5..15)));
         assert_eq!(sel.num_elements(&[100]), 10);
         assert_eq!(sel.output_shape(&[100]), vec![10]);
     }

crates/clawhdf5-format/src/shared_message.rs

@@ -16,8 +16,12 @@
 //! - SMLI list structure: simple list of shared message entries
 //! - B-tree v2 type 7: indexed shared message entries
 
+#[cfg(not(feature = "std"))]
+use alloc::borrow::Cow;
 #[cfg(not(feature = "std"))]
 use alloc::vec::Vec;
+#[cfg(feature = "std")]
+use std::borrow::Cow;
 
 use crate::btree_v2::{BTreeV2Header, collect_btree_v2_records};
 use crate::error::FormatError;
@@ -28,6 +32,14 @@ use crate::object_header::ObjectHeader;
 /// Fractal heap ID length for SOHM entries (fixed at 8 bytes).
 const FHEAP_ID_LEN: usize = 8;
 
+/// Shared-message `type` values (version 3 encoding).
+/// The message is in the file's shared-message (SOHM) fractal heap.
+const SHARE_TYPE_SOHM: u8 = 1;
+/// The message is in another object's header (a committed/named datatype).
+const SHARE_TYPE_COMMITTED: u8 = 2;
+/// The message is stored here but is sharable.
+const SHARE_TYPE_HERE: u8 = 3;
+
 /// A resolved shared message reference.
 #[derive(Debug, Clone)]
 pub struct SharedMessageRef {
@@ -35,9 +47,10 @@ pub struct SharedMessageRef {
     pub ref_type: u8,
     /// Version of the shared message encoding.
     pub version: u8,
-    /// Address of the object header containing the shared message (type 1, 3).
+    /// Address of the object header holding the message (committed). Set for
+    /// every v1/v2 reference and for v3 types 2 and 3.
     pub object_header_address: Option<u64>,
-    /// Fractal heap ID for type 2 (SOHM) references.
+    /// Fractal heap ID for a v3 SOHM (type 1) reference.
     pub heap_id: Option<[u8; FHEAP_ID_LEN]>,
 }
 
@@ -146,35 +159,27 @@ pub fn parse_shared_ref(data: &[u8], offset_size: u8) -> Result<SharedMessageRef
     let version = data[0];
     let ref_type = data[1];
 
-    match version {
+    // Layouts (HDF5 spec IV.A.2 "Shared Message", and libhdf5's decoder):
-        1 | 2 => {
+    //   v1: version, type, reserved(6), address     — always "committed"
-            // v1/v2: reserved(6) + address(offset_size)
+    //   v2: version, type, address                  — always "committed"
-            let pos = 2 + 6; // skip reserved bytes
+    //   v3: version, type, then a fractal-heap ID if type == SOHM, otherwise
+    //       an address
+    // Verified against h5py/HDF5 2.0 output, which writes `02 02 <address>`
+    // for a dataset using a committed datatype under both default and
+    // `latest` libver bounds.
+    let address_at = |pos: usize| -> Result<SharedMessageRef, FormatError> {
         ensure_len(data, pos, offset_size as usize)?;
-            let addr = read_offset(data, pos, offset_size)?;
         Ok(SharedMessageRef {
             ref_type,
             version,
-                object_header_address: Some(addr),
+            object_header_address: Some(read_offset(data, pos, offset_size)?),
             heap_id: None,
         })
-        }
+    };
-        3 => {
+    match version {
-            match ref_type {
+        1 => address_at(2 + 6),
-                1 | 3 => {
+        2 => address_at(2),
-                    // type 1/3: message in another object header
+        3 if ref_type == SHARE_TYPE_SOHM => {
-                    // v3 layout: version(1) + type(1) + address(offset_size)
-                    ensure_len(data, 2, offset_size as usize)?;
-                    let addr = read_offset(data, 2, offset_size)?;
-                    Ok(SharedMessageRef {
-                        ref_type,
-                        version,
-                        object_header_address: Some(addr),
-                        heap_id: None,
-                    })
-                }
-                2 => {
-                    // type 2: SOHM table (fractal heap ID)
             ensure_len(data, 2, FHEAP_ID_LEN)?;
             let mut id = [0u8; FHEAP_ID_LEN];
             id.copy_from_slice(&data[2..2 + FHEAP_ID_LEN]);
@@ -185,9 +190,8 @@ pub fn parse_shared_ref(data: &[u8], offset_size: u8) -> Result<SharedMessageRef
                 heap_id: Some(id),
             })
         }
-                _ => Err(FormatError::InvalidSharedMessageVersion(ref_type)),
+        3 if ref_type == SHARE_TYPE_COMMITTED || ref_type == SHARE_TYPE_HERE => address_at(2),
-            }
+        3 => Err(FormatError::InvalidSharedMessageVersion(ref_type)),
-        }
         _ => Err(FormatError::InvalidSharedMessageVersion(version)),
     }
 }
@@ -422,6 +426,35 @@ pub fn resolve_sohm_message(
     fh_header.read_managed_object(file_data, heap_id, offset_size)
 }
 
+/// The payload of an object-header message, following the indirection if the
+/// message is *shared* (header flag bit 1).
+///
+/// A shared message's bytes are not the message itself but a reference to
+/// where it lives — e.g. a dataset created with a committed (named) datatype
+/// stores only a pointer to that datatype's object header. Every reader of a
+/// message that may be shared (datatype, dataspace, fill value, filter
+/// pipeline, attribute) must go through this; parsing the reference bytes as
+/// the message yields garbage rather than an error.
+pub fn message_data<'a>(
+    file_data: &[u8],
+    msg: &'a crate::object_header::HeaderMessage,
+    offset_size: u8,
+    length_size: u8,
+) -> Result<Cow<'a, [u8]>, FormatError> {
+    if !is_shared(msg.flags) {
+        return Ok(Cow::Borrowed(&msg.data));
+    }
+    let shared_ref = parse_shared_ref(&msg.data, offset_size)?;
+    resolve_shared_message(
+        file_data,
+        &shared_ref,
+        msg.msg_type,
+        offset_size,
+        length_size,
+    )
+    .map(Cow::Owned)
+}
+
 /// Resolve a shared message to its actual message data.
 ///
 /// For type 1/3 (shared in another object header), reads the target object header
@@ -453,14 +486,14 @@ pub fn resolve_shared_message_with_sohm(
     length_size: u8,
     sohm_table: Option<&SohmTable>,
 ) -> Result<Vec<u8>, FormatError> {
-    match shared_ref.ref_type {
+    // Dispatch on what the reference carries rather than on `ref_type`: v1/v2
-        1 | 3 => {
+    // references are always an object-header address whatever their type
-            let addr = shared_ref
+    // byte says.
-                .object_header_address
+    match (
-                .ok_or(FormatError::UnexpectedEof {
+        shared_ref.object_header_address,
-                    expected: 1,
+        shared_ref.heap_id.as_ref(),
-                    available: 0,
+    ) {
-                })?;
+        (Some(addr), _) => {
             let target_header =
                 ObjectHeader::parse(file_data, addr as usize, offset_size, length_size)?;
             for msg in &target_header.messages {
@@ -487,11 +520,7 @@ pub fn resolve_shared_message_with_sohm(
                 available: 0,
             })
         }
-        2 => {
+        (None, Some(heap_id)) => {
-            let heap_id = shared_ref
-                .heap_id
-                .as_ref()
-                .ok_or(FormatError::InvalidSharedMessageVersion(2))?;
             let table = sohm_table.ok_or(FormatError::InvalidSharedMessageVersion(2))?;
             resolve_sohm_message(
                 file_data,
@@ -502,7 +531,7 @@ pub fn resolve_shared_message_with_sohm(
                 length_size,
             )
         }
-        _ => Err(FormatError::InvalidSharedMessageVersion(
+        (None, None) => Err(FormatError::InvalidSharedMessageVersion(
             shared_ref.ref_type,
         )),
     }
@@ -522,15 +551,15 @@ mod tests {
     }
 
     #[test]
-    fn parse_v3_type1_ref() {
+    fn parse_v3_committed_ref() {
         let mut data = Vec::new();
         data.push(3); // version
-        data.push(1); // type 1 = shared in another OH
+        data.push(SHARE_TYPE_COMMITTED); // message lives in another object header
         data.extend_from_slice(&0x1234u64.to_le_bytes()); // address
 
         let shared = parse_shared_ref(&data, 8).unwrap();
         assert_eq!(shared.version, 3);
-        assert_eq!(shared.ref_type, 1);
+        assert_eq!(shared.ref_type, SHARE_TYPE_COMMITTED);
         assert_eq!(shared.object_header_address, Some(0x1234));
         assert!(shared.heap_id.is_none());
     }
@@ -539,7 +568,7 @@ mod tests {
     fn parse_v3_type3_ref() {
         let mut data = Vec::new();
         data.push(3); // version
-        data.push(3); // type 3 = shared in another OH (v3 encoding)
+        data.push(SHARE_TYPE_HERE); // stored here but sharable: an address
         data.extend_from_slice(&0xABCDu64.to_le_bytes());
 
         let shared = parse_shared_ref(&data, 8).unwrap();
@@ -563,10 +592,10 @@ mod tests {
 
     #[test]
     fn parse_v2_ref() {
+        // v2 dropped v1's six reserved bytes: the address follows the type.
         let mut data = Vec::new();
         data.push(2); // version
-        data.push(0); // type
+        data.push(SHARE_TYPE_COMMITTED);
-        data.extend_from_slice(&[0u8; 6]); // reserved
         data.extend_from_slice(&0x9000u32.to_le_bytes());
 
         let shared = parse_shared_ref(&data, 4).unwrap();
@@ -575,15 +604,26 @@ mod tests {
     }
 
     #[test]
-    fn parse_v3_type2_sohm() {
+    fn parse_v2_ref_from_hdf5_2_0() {
+        // Datatype message of a dataset created with a committed datatype,
+        // as written by h5py 3.16 / HDF5 2.0 (libver='latest'): header flags
+        // 0x03 (shared), payload `02 02 <8-byte object header address>`.
+        let data = [0x02, 0x02, 0xb3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00];
+        let shared = parse_shared_ref(&data, 8).unwrap();
+        assert_eq!(shared.object_header_address, Some(0xb3));
+        assert!(shared.heap_id.is_none());
+    }
+
+    #[test]
+    fn parse_v3_sohm_ref() {
         let mut data = Vec::new();
         data.push(3); // version
-        data.push(2); // type 2 = SOHM heap
+        data.push(SHARE_TYPE_SOHM); // message lives in the SOHM fractal heap
         data.extend_from_slice(&[0xAA, 0xBB, 0xCC, 0xDD, 0x11, 0x22, 0x33, 0x44]);
 
         let shared = parse_shared_ref(&data, 8).unwrap();
         assert_eq!(shared.version, 3);
-        assert_eq!(shared.ref_type, 2);
+        assert_eq!(shared.ref_type, SHARE_TYPE_SOHM);
         assert_eq!(shared.object_header_address, None);
         assert_eq!(
             shared.heap_id,
@@ -592,10 +632,10 @@ mod tests {
     }
 
     #[test]
-    fn parse_v3_type2_too_short() {
+    fn parse_v3_sohm_too_short() {
         let mut data = Vec::new();
         data.push(3); // version
-        data.push(2); // type 2 = SOHM heap
+        data.push(SHARE_TYPE_SOHM);
         data.extend_from_slice(&[0xAA, 0xBB]); // only 2 bytes, need 8
 
         let err = parse_shared_ref(&data, 8).unwrap_err();
@@ -620,7 +660,7 @@ mod tests {
     fn parse_four_byte_offsets() {
         let mut data = Vec::new();
         data.push(3); // version
-        data.push(1); // type 1
+        data.push(SHARE_TYPE_COMMITTED);
         data.extend_from_slice(&0x1000u32.to_le_bytes());
 
         let shared = parse_shared_ref(&data, 4).unwrap();

crates/clawhdf5-format/src/symbol_table.rs

@@ -80,7 +80,10 @@ impl SymbolTableNode {
         offset_size: u8,
     ) -> Result<SymbolTableNode, FormatError> {
         // signature(4) + version(1) + reserved(1) + number_of_symbols(2) = 8
-        if offset.checked_add(8).is_none_or(|end| end > file_data.len()) {
+        if offset
+            .checked_add(8)
+            .is_none_or(|end| end > file_data.len())
+        {
             return Err(FormatError::UnexpectedEof {
                 expected: offset.saturating_add(8),
                 available: file_data.len(),
@@ -103,12 +106,12 @@ impl SymbolTableNode {
         // Each entry: link_name_offset(os) + obj_hdr_addr(os) + cache_type(4) + reserved(4) + scratch(16)
         let entry_size = os + os + 4 + 4 + 16;
         let entries_start = offset + 8;
-        let needed = entries_start
+        let needed = entries_start.checked_add(num_symbols * entry_size).ok_or(
-            .checked_add(num_symbols * entry_size)
+            FormatError::UnexpectedEof {
-            .ok_or(FormatError::UnexpectedEof {
                 expected: usize::MAX,
                 available: file_data.len(),
-            })?;
+            },
+        )?;
         if needed > file_data.len() {
             return Err(FormatError::UnexpectedEof {
                 expected: needed,

crates/clawhdf5-format/src/type_builders.rs

@@ -279,6 +279,43 @@ pub(crate) fn build_attr_message(name: &str, value: &AttrValue) -> AttributeMess
             dataspace: scalar_ds(),
             raw_data: v.to_le_bytes().to_vec(),
         },
+        AttrValue::U64Array(arr) => {
+            let mut raw = Vec::with_capacity(arr.len() * 8);
+            for v in arr {
+                raw.extend_from_slice(&v.to_le_bytes());
+            }
+            AttributeMessage {
+                name: name.to_string(),
+                datatype: Datatype::FixedPoint {
+                    size: 8,
+                    byte_order: DatatypeByteOrder::LittleEndian,
+                    signed: false,
+                    bit_offset: 0,
+                    bit_precision: 64,
+                },
+                dataspace: simple_1d(arr.len() as u64),
+                raw_data: raw,
+            }
+        }
+        AttrValue::Raw {
+            datatype,
+            shape,
+            data,
+        } => AttributeMessage {
+            name: name.to_string(),
+            datatype: datatype.clone(),
+            dataspace: if shape.is_empty() {
+                scalar_ds()
+            } else {
+                Dataspace {
+                    space_type: DataspaceType::Simple,
+                    rank: shape.len() as u8,
+                    dimensions: shape.clone(),
+                    max_dimensions: None,
+                }
+            },
+            raw_data: data.clone(),
+        },
         AttrValue::String(s) => {
             let bytes = s.as_bytes();
             AttributeMessage {
@@ -334,7 +371,7 @@ pub(crate) fn simple_1d(n: u64) -> Dataspace {
 
 // ---- Attribute values ----
 
-/// Convenient attribute values for the write API.
+/// Attribute values, for both the write API and what reading returns.
 #[derive(Debug, Clone)]
 pub enum AttrValue {
     F64(f64),
@@ -342,8 +379,21 @@ pub enum AttrValue {
     I64(i64),
     I64Array(Vec<i64>),
     U64(u64),
+    /// Unsigned integers, kept unsigned so values above `i64::MAX` survive.
+    U64Array(Vec<u64>),
     String(String),
     StringArray(Vec<String>),
+    /// An attribute whose datatype has no dedicated variant above (compound,
+    /// general enum, complex, reference, opaque, array, ...), carried verbatim
+    /// so it is never silently lost: the datatype, the dataspace dimensions
+    /// (empty for a scalar) and the element bytes exactly as stored. Decode
+    /// `data` with `clawhdf5_format::data_read` (e.g. `read_compound_fields`)
+    /// against `datatype`. Writing a `Raw` value stores it back unchanged.
+    Raw {
+        datatype: Datatype,
+        shape: Vec<u64>,
+        data: Vec<u8>,
+    },
 }
 
 // ---- Dataset builder ----

crates/clawhdf5-format/tests/integration_test.rs

@@ -343,7 +343,11 @@ fn attrs_h5_dataset_scale() {
     let scale_attr = find_attribute(&attrs, "scale").expect("scale attr not found");
     let vals = scale_attr.read_as_f64().unwrap();
     assert_eq!(vals.len(), 1);
-    assert!((vals[0] - 3.14).abs() < 1e-10);
+    // 3.14 here is the literal value baked into the binary fixture (fixtures/attrs.h5),
+    // not an arbitrary sample value, so it cannot be swapped for another constant.
+    #[allow(clippy::approx_constant)]
+    let expected = 3.14;
+    assert!((vals[0] - expected).abs() < 1e-10);
 }
 
 #[test]
@@ -556,8 +560,8 @@ fn chunked_deflate_read_values() {
     let (raw, datatype, _) = read_chunked_dataset(file_data, "data");
     let values = read_as_f64(&raw, &datatype).unwrap();
     assert_eq!(values.len(), 100);
-    for i in 0..100 {
+    for (i, &v) in values.iter().enumerate() {
-        assert_eq!(values[i], i as f64, "mismatch at index {i}");
+        assert_eq!(v, i as f64, "mismatch at index {i}");
     }
 }
 
@@ -567,8 +571,8 @@ fn chunked_shuffle_deflate_read_values() {
     let (raw, datatype, _) = read_chunked_dataset(file_data, "data");
     let values = read_as_f64(&raw, &datatype).unwrap();
     assert_eq!(values.len(), 100);
-    for i in 0..100 {
+    for (i, &v) in values.iter().enumerate() {
-        assert_eq!(values[i], i as f64, "mismatch at index {i}");
+        assert_eq!(v, i as f64, "mismatch at index {i}");
     }
 }
 
@@ -578,8 +582,8 @@ fn chunked_fletcher32_read_values() {
     let (raw, datatype, _) = read_chunked_dataset(file_data, "data");
     let values = read_as_f64(&raw, &datatype).unwrap();
     assert_eq!(values.len(), 100);
-    for i in 0..100 {
+    for (i, &v) in values.iter().enumerate() {
-        assert_eq!(values[i], i as f64, "mismatch at index {i}");
+        assert_eq!(v, i as f64, "mismatch at index {i}");
     }
 }
 
@@ -589,11 +593,10 @@ fn chunked_2d_read_values() {
     let (raw, datatype, _) = read_chunked_dataset(file_data, "matrix");
     let values = read_as_f32(&raw, &datatype).unwrap();
     assert_eq!(values.len(), 60);
-    for i in 0..60 {
+    for (i, &v) in values.iter().enumerate() {
         assert!(
-            (values[i] - i as f32).abs() < 1e-6,
+            (v - i as f32).abs() < 1e-6,
-            "mismatch at index {i}: got {}",
+            "mismatch at index {i}: got {v}"
-            values[i]
         );
     }
 }
@@ -604,8 +607,8 @@ fn chunked_large_read_values() {
     let (raw, datatype, _) = read_chunked_dataset(file_data, "big");
     let values = read_as_i32(&raw, &datatype).unwrap();
     assert_eq!(values.len(), 1000);
-    for i in 0..1000 {
+    for (i, &v) in values.iter().enumerate() {
-        assert_eq!(values[i], i as i32, "mismatch at index {i}");
+        assert_eq!(v, i as i32, "mismatch at index {i}");
     }
 }
 
@@ -615,8 +618,8 @@ fn chunked_nofilter_read_values() {
     let (raw, datatype, _) = read_chunked_dataset(file_data, "raw");
     let values = read_as_f64(&raw, &datatype).unwrap();
     assert_eq!(values.len(), 50);
-    for i in 0..50 {
+    for (i, &v) in values.iter().enumerate() {
-        assert_eq!(values[i], i as f64, "mismatch at index {i}");
+        assert_eq!(v, i as f64, "mismatch at index {i}");
     }
 }
 
@@ -646,8 +649,8 @@ fn v4_implicit_read() {
     let (raw, datatype, _) = read_chunked_dataset(file_data, "data");
     let values = read_as_f64(&raw, &datatype).unwrap();
     assert_eq!(values.len(), 100);
-    for i in 0..100 {
+    for (i, &v) in values.iter().enumerate() {
-        assert_eq!(values[i], i as f64, "mismatch at index {i}");
+        assert_eq!(v, i as f64, "mismatch at index {i}");
     }
 }
 
@@ -657,8 +660,8 @@ fn v4_fixed_array_read() {
     let (raw, datatype, _) = read_chunked_dataset(file_data, "data");
     let values = read_as_f64(&raw, &datatype).unwrap();
     assert_eq!(values.len(), 100);
-    for i in 0..100 {
+    for (i, &v) in values.iter().enumerate() {
-        assert_eq!(values[i], i as f64, "mismatch at index {i}");
+        assert_eq!(v, i as f64, "mismatch at index {i}");
     }
 }
 
@@ -871,11 +874,10 @@ fn v4_2d_fixed_array_read() {
     let (raw, datatype, _) = read_chunked_dataset(file_data, "matrix");
     let values = read_as_f32(&raw, &datatype).unwrap();
     assert_eq!(values.len(), 60);
-    for i in 0..60 {
+    for (i, &v) in values.iter().enumerate() {
         assert!(
-            (values[i] - i as f32).abs() < 1e-6,
+            (v - i as f32).abs() < 1e-6,
-            "mismatch at index {i}: got {}",
+            "mismatch at index {i}: got {v}"
-            values[i]
         );
     }
 }
@@ -1272,7 +1274,7 @@ fn write_roundtrip_scalar_f64_attr() {
     let mut fw = FileWriter::new();
     fw.create_dataset("data")
         .with_f64_data(&[1.0])
-        .set_attr("scale", AttrValue::F64(3.14));
+        .set_attr("scale", AttrValue::F64(3.25));
     let bytes = fw.finish().unwrap();
 
     let sig = find_signature(&bytes).unwrap();
@@ -1283,7 +1285,7 @@ fn write_roundtrip_scalar_f64_attr() {
     let scale = find_attribute(&attrs, "scale").expect("scale attr not found");
     let vals = scale.read_as_f64().unwrap();
     assert_eq!(vals.len(), 1);
-    assert!((vals[0] - 3.14).abs() < 1e-10);
+    assert!((vals[0] - 3.25).abs() < 1e-10);
 }
 
 #[test]

crates/clawhdf5-format/tests/reference_tests.rs

@@ -180,15 +180,20 @@ print('ok')
     let output = match output {
         Ok(o) if o.status.success() => o,
         _ => {
+            // CI sets CLAWHDF5_REQUIRE_INTEROP=1 so this can't silently skip.
+            assert!(
+                !std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1"),
+                "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+            );
             eprintln!("skipping h5py_object_reference_roundtrip: python3+h5py not available");
             return;
         }
     };
     let stdout = String::from_utf8(output.stdout).unwrap();
-    if !stdout.trim().contains("ok") {
+    assert!(
-        eprintln!("skipping h5py_object_reference_roundtrip: h5py script failed");
+        stdout.trim().contains("ok"),
-        return;
+        "h5py reference-file generator did not report ok: {stdout}"
-    }
+    );
 
     // Read the file and parse object references
     let file_data = std::fs::read(&path).unwrap();

crates/clawhdf5-format/tests/writer_h5py_tests.rs

@@ -235,17 +235,31 @@ fn h5py_reads_our_array_dataset() {
 #[test]
 #[ignore = "requires Python h5py module"]
 fn read_h5py_generated_compound() {
-    let path = std::env::temp_dir().join("clawhdf5_h5py_compound.h5");
+    check_h5py_generated_compound("latest", ", libver='latest'");
+}
+
+/// Same file written with h5py's default format bounds. HDF5 2.0 raised the
+/// default low bound to 1.8, so "default" files exercise different on-disk
+/// structures than both `libver='latest'` and pre-2.0 defaults.
+#[test]
+#[ignore = "requires Python h5py module"]
+fn read_h5py_generated_compound_default_libver() {
+    check_h5py_generated_compound("default", "");
+}
+
+fn check_h5py_generated_compound(tag: &str, libver_kw: &str) {
+    let path = std::env::temp_dir().join(format!("clawhdf5_h5py_compound_{tag}.h5"));
     let gen_script = format!(
         r#"
 import h5py, numpy as np
 dt = np.dtype([('x', 'f8'), ('y', 'f8'), ('id', 'i4')])
 data = np.array([(1.0, 2.0, 10), (3.0, 4.0, 20)], dtype=dt)
-f = h5py.File('{}', 'w', libver='latest')
+f = h5py.File('{}', 'w'{})
 f.create_dataset('particles', data=data)
 f.close()
 "#,
-        path.display()
+        path.display(),
+        libver_kw
     );
     h5py_read(&path, &gen_script);
 
@@ -363,17 +377,31 @@ else:
 #[test]
 #[ignore = "requires Python h5py module"]
 fn read_h5py_generated_enum() {
-    let path = std::env::temp_dir().join("clawhdf5_h5py_enum.h5");
+    check_h5py_generated_enum("latest", ", libver='latest'");
+}
+
+/// Same file written with h5py's default format bounds. HDF5 2.0 raised the
+/// default low bound to 1.8, so "default" files exercise different on-disk
+/// structures than both `libver='latest'` and pre-2.0 defaults.
+#[test]
+#[ignore = "requires Python h5py module"]
+fn read_h5py_generated_enum_default_libver() {
+    check_h5py_generated_enum("default", "");
+}
+
+fn check_h5py_generated_enum(tag: &str, libver_kw: &str) {
+    let path = std::env::temp_dir().join(format!("clawhdf5_h5py_enum_{tag}.h5"));
     let gen_script = format!(
         r#"
 import h5py, numpy as np
 dt = h5py.enum_dtype({{"RED": 0, "GREEN": 1, "BLUE": 2}}, basetype=np.int32)
 data = np.array([1, 0, 2, 1], dtype=np.int32)
-f = h5py.File('{}', 'w', libver='latest')
+f = h5py.File('{}', 'w'{})
 f.create_dataset('colors', data=data, dtype=dt)
 f.close()
 "#,
-        path.display()
+        path.display(),
+        libver_kw
     );
     h5py_read(&path, &gen_script);
 

crates/clawhdf5-gpu/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-gpu"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "GPU-accelerated vector operations for rustyhdf5 using wgpu compute shaders"
 license = "MIT"

crates/clawhdf5-gpu/src/accelerator.rs

@@ -6,6 +6,9 @@ use crate::shaders;
 use bytemuck::Pod;
 use wgpu::util::DeviceExt;
 
+/// Upper bound on a single GPU→CPU readback wait.
+const READBACK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(30);
+
 /// GPU-accelerated vector search engine.
 ///
 /// Upload vectors once, then run many searches against them.
@@ -1033,10 +1036,12 @@ impl GpuAccelerator {
         slice.map_async(wgpu::MapMode::Read, move |result| {
             let _ = tx.send(result);
         });
+        // Bounded wait: a wedged driver must surface as an error, not hang
+        // the caller forever.
         self.device
             .poll(wgpu::PollType::Wait {
                 submission_index: None,
-                timeout: None,
+                timeout: Some(READBACK_TIMEOUT),
             })
             .map_err(|e| GpuError::BufferMap(format!("device poll failed: {e}")))?;
         rx.recv()

crates/clawhdf5-gpu/tests/gpu_tests.rs

@@ -6,9 +6,41 @@
 mod tests {
     use clawhdf5_gpu::{GpuAccelerator, GpuError};
 
-    fn skip_if_no_gpu() -> Option<GpuAccelerator> {
+    /// Serialises GPU access across tests. The harness runs tests on many
+    /// threads; letting each create its own wgpu instance + device (with
+    /// adapter-maximum limits) at the same time can wedge the driver and hang
+    /// the whole suite, so every test holds this lock while it owns a device.
+    static GPU_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
+
+    fn gpu_lock() -> std::sync::MutexGuard<'static, ()> {
+        // A panicking test poisons the lock; the guarded state is `()`.
+        GPU_LOCK.lock().unwrap_or_else(|e| e.into_inner())
+    }
+
+    /// A `GpuAccelerator` plus the lock that keeps other tests off the GPU.
+    /// Field order matters: the device is dropped before the lock is released.
+    struct LockedGpu {
+        gpu: GpuAccelerator,
+        _guard: std::sync::MutexGuard<'static, ()>,
+    }
+
+    impl std::ops::Deref for LockedGpu {
+        type Target = GpuAccelerator;
+        fn deref(&self) -> &GpuAccelerator {
+            &self.gpu
+        }
+    }
+
+    impl std::ops::DerefMut for LockedGpu {
+        fn deref_mut(&mut self) -> &mut GpuAccelerator {
+            &mut self.gpu
+        }
+    }
+
+    fn skip_if_no_gpu() -> Option<LockedGpu> {
+        let guard = gpu_lock();
         match GpuAccelerator::new() {
-            Ok(gpu) => Some(gpu),
+            Ok(gpu) => Some(LockedGpu { gpu, _guard: guard }),
             Err(_) => {
                 eprintln!("SKIPPED: no GPU available");
                 None
@@ -69,6 +101,7 @@ mod tests {
     #[test]
     fn test_gpu_availability_detection() {
         // Should not panic regardless of GPU presence
+        let _guard = gpu_lock();
         let available = GpuAccelerator::is_available();
         eprintln!("GPU available: {available}");
     }
@@ -425,6 +458,7 @@ mod tests {
     #[test]
     fn test_graceful_no_gpu_fallback() {
         // This test just demonstrates the pattern — it always passes
+        let _guard = gpu_lock();
         match GpuAccelerator::new() {
             Ok(gpu) => {
                 eprintln!("GPU found: {}", gpu.device_info());

crates/clawhdf5-io/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-io"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "I/O abstraction layer for rustyhdf5"
 license = "MIT"
@@ -10,7 +10,7 @@ keywords = ["hdf5", "io", "science", "data"]
 categories = ["filesystem", "science"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.2.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.3.0" }
 memmap2 = { version = "0.9", optional = true }
 libc = { version = "0.2", optional = true }
 tokio = { version = "1", features = ["fs", "io-util"], optional = true }

crates/clawhdf5-migrate/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-migrate"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "CLI to migrate SQLite agent memory databases to HDF5 format"
 license = "MIT"
@@ -14,9 +14,9 @@ name = "clawhdf5-migrate"
 path = "src/main.rs"
 
 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.2.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.3.0" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.2.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.3.0" }
-clawhdf5 = { path = "../clawhdf5", version = "2.2.0" }
+clawhdf5 = { path = "../clawhdf5", version = "2.3.0" }
 rusqlite = { version = "0.31", features = ["bundled"] }
 clap = { version = "4", features = ["derive"] }
 half = { workspace = true }

crates/clawhdf5-migrate/src/hdf5_writer.rs

@@ -57,7 +57,11 @@ fn iso8601_now() -> String {
         .as_secs();
     let days = (secs / 86_400) as i64;
     let time_of_day = secs % 86_400;
-    let (h, m, s) = (time_of_day / 3600, (time_of_day % 3600) / 60, time_of_day % 60);
+    let (h, m, s) = (
+        time_of_day / 3600,
+        (time_of_day % 3600) / 60,
+        time_of_day % 60,
+    );
     let (y, mo, d) = civil_from_days(days);
     format!("{y:04}-{mo:02}-{d:02}T{h:02}:{m:02}:{s:02}Z")
 }

crates/clawhdf5-migrate/src/sqlite_reader.rs

@@ -185,8 +185,10 @@ fn detect_embedding_dim(conn: &Connection, config: &SchemaConfig) -> SqlResult<O
 
 /// Parse a raw byte BLOB into a Vec<f32>.
 fn blob_to_f32(blob: &[u8]) -> Vec<f32> {
-    blob.chunks_exact(4)
+    blob.as_chunks::<4>()
-        .map(|b| f32::from_le_bytes([b[0], b[1], b[2], b[3]]))
+        .0
+        .iter()
+        .map(|b| f32::from_le_bytes(*b))
         .collect()
 }
 

crates/clawhdf5-napi/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-napi"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "Node.js native addon (napi-rs) exposing clawhdf5-agent to TypeScript/JavaScript"
 license = "MIT"
@@ -10,7 +10,7 @@ repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
 crate-type = ["cdylib"]
 
 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.2.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.3.0" }
 napi = { version = "2", default-features = false, features = ["napi9"] }
 napi-derive = "2"
 

crates/clawhdf5-netcdf4/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-netcdf4"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "NetCDF-4 read support built on rustyhdf5 — pure Rust, no C dependencies"
 license = "MIT"
@@ -10,8 +10,8 @@ keywords = ["netcdf", "netcdf4", "hdf5", "science", "climate"]
 categories = ["parser-implementations", "science"]
 
 [dependencies]
-clawhdf5 = { path = "../clawhdf5", version = "2.2.0" }
+clawhdf5 = { path = "../clawhdf5", version = "2.3.0" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.2.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.3.0" }
 
 [dev-dependencies]
 tempfile = { workspace = true }

crates/clawhdf5-netcdf4/src/cf.rs

@@ -142,6 +142,7 @@ fn get_fill_value(attrs: &HashMap<String, AttrValue>, key: &str) -> Option<FillV
         Some(AttrValue::String(s)) => Some(FillValue::String(s.clone())),
         Some(AttrValue::F64Array(arr)) if !arr.is_empty() => Some(FillValue::Float(arr[0])),
         Some(AttrValue::I64Array(arr)) if !arr.is_empty() => Some(FillValue::Int(arr[0])),
+        Some(AttrValue::U64Array(arr)) if !arr.is_empty() => Some(FillValue::UInt(arr[0])),
         _ => None,
     }
 }
@@ -155,6 +156,10 @@ fn get_valid_range(attrs: &HashMap<String, AttrValue>) -> Option<(f64, f64)> {
         Some(AttrValue::I64Array(arr)) if arr.len() >= 2 => {
             return Some((arr[0] as f64, arr[1] as f64));
         }
+        // Unsigned variables (NC_UBYTE..NC_UINT64) carry unsigned attributes.
+        Some(AttrValue::U64Array(arr)) if arr.len() >= 2 => {
+            return Some((arr[0] as f64, arr[1] as f64));
+        }
         _ => {}
     }
 

crates/clawhdf5-netcdf4/tests/interop_tests.rs

@@ -10,6 +10,12 @@ use clawhdf5_netcdf4::{AttrValue, NetCDF4File};
 // Helpers
 // ---------------------------------------------------------------------------
 
+/// When `CLAWHDF5_REQUIRE_INTEROP=1` (set in CI), a missing Python dependency
+/// is a test failure instead of a silent skip.
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
 fn netcdf4_python_available() -> bool {
     Command::new("python3")
         .args(["-c", "import netCDF4; print(netCDF4.__version__)"])
@@ -29,6 +35,10 @@ fn xarray_available() -> bool {
 macro_rules! skip_if_no_netcdf4 {
     () => {
         if !netcdf4_python_available() {
+            assert!(
+                !interop_required(),
+                "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with netCDF4 is not available"
+            );
             eprintln!("SKIP: python3 with netCDF4 not available");
             return;
         }
@@ -38,6 +48,10 @@ macro_rules! skip_if_no_netcdf4 {
 macro_rules! skip_if_no_xarray {
     () => {
         if !xarray_available() {
+            assert!(
+                !interop_required(),
+                "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with xarray is not available"
+            );
             eprintln!("SKIP: python3 with xarray not available");
             return;
         }

crates/clawhdf5-py/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-py"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 license = "MIT"
@@ -14,8 +14,8 @@ name = "clawhdf5"
 crate-type = ["cdylib", "rlib"]
 
 [dependencies]
-clawhdf5_rs = { path = "../clawhdf5", version = "2.2.0", package = "clawhdf5" }
+clawhdf5_rs = { path = "../clawhdf5", version = "2.3.0", package = "clawhdf5" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.2.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.3.0" }
 pyo3 = "0.29"
 numpy = "0.29"
 

crates/clawhdf5-py/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "rustyhdf5"
-version = "2.2.0"
+version = "2.3.0"
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 requires-python = ">=3.8"
 license = { text = "MIT" }

crates/clawhdf5-py/src/lib.rs

@@ -128,10 +128,28 @@ pub(crate) fn attr_value_to_py(py: Python<'_>, val: &clawhdf5_rs::AttrValue) ->
             let list = pyo3::types::PyList::new(py, a).unwrap();
             list.into_any().unbind()
         }
+        clawhdf5_rs::AttrValue::U64Array(a) => {
+            let list = pyo3::types::PyList::new(py, a).unwrap();
+            list.into_any().unbind()
+        }
         clawhdf5_rs::AttrValue::StringArray(a) => {
             let list = pyo3::types::PyList::new(py, a).unwrap();
             list.into_any().unbind()
         }
+        // No Python-side decoding for this datatype: hand back everything
+        // needed to interpret it rather than dropping the attribute.
+        clawhdf5_rs::AttrValue::Raw {
+            datatype,
+            shape,
+            data,
+        } => {
+            let dict = pyo3::types::PyDict::new(py);
+            dict.set_item("dtype", format!("{datatype:?}")).unwrap();
+            dict.set_item("shape", shape).unwrap();
+            dict.set_item("data", pyo3::types::PyBytes::new(py, data))
+                .unwrap();
+            dict.into_any().unbind()
+        }
     }
 }
 

crates/clawhdf5/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5"
-version = "2.2.0"
+version = "2.3.0"
 edition = "2024"
 description = "Pure-Rust HDF5 reader/writer — no C dependencies"
 license = "MIT"
@@ -10,16 +10,16 @@ keywords = ["hdf5", "science", "data", "binary"]
 categories = ["parser-implementations", "science", "encoding"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.2.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.3.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.2.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.3.0" }
 rayon = { version = "1", optional = true }
 
 [dev-dependencies]
 tempfile = { workspace = true }
 criterion = { workspace = true }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.2.0", features = ["mmap"] }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.3.0", features = ["mmap"] }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.2.0", features = ["parallel", "fast-checksum"] }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.3.0", features = ["parallel", "fast-checksum"] }
-clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.2.0" }
+clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.3.0" }
 
 [[bench]]
 name = "mmap_bench"

crates/clawhdf5/src/lazy.rs

@@ -416,15 +416,43 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
         ))
     }
 
+    /// A header message's payload, resolved through the shared-message
+    /// indirection when needed (e.g. a committed datatype). See
+    /// [`clawhdf5_format::shared_message::message_data`].
+    fn message_payload(
+        &self,
+        msg_type: MessageType,
+    ) -> Result<Option<std::borrow::Cow<'_, [u8]>>, Error> {
+        self.header
+            .messages
+            .iter()
+            .find(|m| m.msg_type == msg_type)
+            .map(|msg| {
+                clawhdf5_format::shared_message::message_data(
+                    self.file.as_bytes(),
+                    msg,
+                    self.file.offset_size(),
+                    self.file.length_size(),
+                )
+                .map_err(Error::Format)
+            })
+            .transpose()
+    }
+
+    fn required_payload(&self, msg_type: MessageType) -> Result<std::borrow::Cow<'_, [u8]>, Error> {
+        self.message_payload(msg_type)?
+            .ok_or(Error::MissingMessage(msg_type))
+    }
+
     fn datatype(&self) -> Result<Datatype, Error> {
-        let msg = find_message(&self.header, MessageType::Datatype)?;
+        let data = self.required_payload(MessageType::Datatype)?;
-        let (dt, _) = Datatype::parse(&msg.data)?;
+        let (dt, _) = Datatype::parse(&data)?;
         Ok(dt)
     }
 
     fn dataspace(&self) -> Result<Dataspace, Error> {
-        let msg = find_message(&self.header, MessageType::Dataspace)?;
+        let data = self.required_payload(MessageType::Dataspace)?;
-        Ok(Dataspace::parse(&msg.data, self.file.length_size())?)
+        Ok(Dataspace::parse(&data, self.file.length_size())?)
     }
 
     fn data_layout(&self) -> Result<DataLayout, Error> {
@@ -436,20 +464,32 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
         )?)
     }
 
-    fn filter_pipeline(&self) -> Option<FilterPipeline> {
+    /// `Ok(None)` means the dataset has no filter pipeline. A pipeline message
-        self.header
+    /// that is present but unparseable is an error: treating it as "no
-            .messages
+    /// filters" would hand the caller the still-compressed bytes as if they
-            .iter()
+    /// were the data.
-            .find(|m| m.msg_type == MessageType::FilterPipeline)
+    fn filter_pipeline(&self) -> Result<Option<FilterPipeline>, Error> {
-            .and_then(|msg| FilterPipeline::parse(&msg.data).ok())
+        self.message_payload(MessageType::FilterPipeline)?
+            .map(|data| FilterPipeline::parse(&data).map_err(Error::Format))
+            .transpose()
     }
 
     fn read_raw(&self) -> Result<Vec<u8>, Error> {
         let dt = self.datatype()?;
         let ds = self.dataspace()?;
         let dl = self.data_layout()?;
-        let pipeline = self.filter_pipeline();
+        let pipeline = self.filter_pipeline()?;
         let data = self.file.reader.as_bytes();
+        // Unallocated storage reads as the dataset's fill value.
+        clawhdf5_format::fill_value::read_full_with_fill(
+            &self.header.messages,
+            data,
+            &dl,
+            &ds,
+            dt.type_size() as usize,
+            self.file.offset_size(),
+            self.file.length_size(),
+            || {
                 Ok(data_read::read_raw_data_full(
                     data,
                     &dl,
@@ -459,6 +499,8 @@ impl<'f, R: HDF5Read> LazyDataset<'f, R> {
                     self.file.offset_size(),
                     self.file.length_size(),
                 )?)
+            },
+        )
     }
 }
 

crates/clawhdf5/src/lib.rs

@@ -48,11 +48,11 @@ pub use clawhdf5_format::dict_encoding::{DictEncoded, DictionaryEncoder};
 pub use clawhdf5_format::property_list::{
     DatasetCreateProps, FileAccessProps, FileCreateProps, lib_version,
 };
+#[cfg(feature = "provenance")]
+pub use clawhdf5_format::provenance;
 pub use clawhdf5_format::selection::Selection;
 pub use clawhdf5_format::superblock::swmr_flags;
 pub use clawhdf5_format::type_builders::{CompoundTypeBuilder, EnumTypeBuilder, FillTime};
-#[cfg(feature = "provenance")]
-pub use clawhdf5_format::provenance;
 
 #[cfg(test)]
 mod tests {
@@ -469,13 +469,10 @@ mod tests {
         let ds = file.dataset("data").unwrap();
 
         // Zero-copy should succeed for contiguous LE f64 on mmap
-        match ds.read_f64_zerocopy() {
+        if let Ok(slice) = ds.read_f64_zerocopy() {
-            Ok(slice) => {
             assert_eq!(slice, &original[..]);
             assert_eq!(slice, &ds.read_f64().unwrap()[..]);
-            }
+        } // else: alignment issue, acceptable
-            Err(_) => {} // alignment issue, acceptable
-        }
         assert_eq!(ds.read_f64().unwrap(), original);
 
         std::fs::remove_file(&path).ok();

crates/clawhdf5/src/mmap_file.rs

@@ -357,15 +357,43 @@ impl<'f> MmapDataset<'f> {
         ))
     }
 
+    /// A header message's payload, resolved through the shared-message
+    /// indirection when needed (e.g. a committed datatype). See
+    /// [`clawhdf5_format::shared_message::message_data`].
+    fn message_payload(
+        &self,
+        msg_type: MessageType,
+    ) -> Result<Option<std::borrow::Cow<'_, [u8]>>, Error> {
+        self.header
+            .messages
+            .iter()
+            .find(|m| m.msg_type == msg_type)
+            .map(|msg| {
+                clawhdf5_format::shared_message::message_data(
+                    self.file.as_bytes(),
+                    msg,
+                    self.file.offset_size(),
+                    self.file.length_size(),
+                )
+                .map_err(Error::Format)
+            })
+            .transpose()
+    }
+
+    fn required_payload(&self, msg_type: MessageType) -> Result<std::borrow::Cow<'_, [u8]>, Error> {
+        self.message_payload(msg_type)?
+            .ok_or(Error::MissingMessage(msg_type))
+    }
+
     fn datatype(&self) -> Result<Datatype, Error> {
-        let msg = find_message(&self.header, MessageType::Datatype)?;
+        let data = self.required_payload(MessageType::Datatype)?;
-        let (dt, _) = Datatype::parse(&msg.data)?;
+        let (dt, _) = Datatype::parse(&data)?;
         Ok(dt)
     }
 
     fn dataspace(&self) -> Result<Dataspace, Error> {
-        let msg = find_message(&self.header, MessageType::Dataspace)?;
+        let data = self.required_payload(MessageType::Dataspace)?;
-        Ok(Dataspace::parse(&msg.data, self.file.length_size())?)
+        Ok(Dataspace::parse(&data, self.file.length_size())?)
     }
 
     fn data_layout(&self) -> Result<DataLayout, Error> {
@@ -377,19 +405,31 @@ impl<'f> MmapDataset<'f> {
         )?)
     }
 
-    fn filter_pipeline(&self) -> Option<FilterPipeline> {
+    /// `Ok(None)` means the dataset has no filter pipeline. A pipeline message
-        self.header
+    /// that is present but unparseable is an error: treating it as "no
-            .messages
+    /// filters" would hand the caller the still-compressed bytes as if they
-            .iter()
+    /// were the data.
-            .find(|m| m.msg_type == MessageType::FilterPipeline)
+    fn filter_pipeline(&self) -> Result<Option<FilterPipeline>, Error> {
-            .and_then(|msg| FilterPipeline::parse(&msg.data).ok())
+        self.message_payload(MessageType::FilterPipeline)?
+            .map(|data| FilterPipeline::parse(&data).map_err(Error::Format))
+            .transpose()
     }
 
     fn read_raw(&self) -> Result<Vec<u8>, Error> {
         let dt = self.datatype()?;
         let ds = self.dataspace()?;
         let dl = self.data_layout()?;
-        let pipeline = self.filter_pipeline();
+        let pipeline = self.filter_pipeline()?;
+        // Unallocated storage reads as the dataset's fill value.
+        clawhdf5_format::fill_value::read_full_with_fill(
+            &self.header.messages,
+            self.file.reader.as_bytes(),
+            &dl,
+            &ds,
+            dt.type_size() as usize,
+            self.file.offset_size(),
+            self.file.length_size(),
+            || {
                 Ok(data_read::read_raw_data_full(
                     self.file.reader.as_bytes(),
                     &dl,
@@ -399,6 +439,8 @@ impl<'f> MmapDataset<'f> {
                     self.file.offset_size(),
                     self.file.length_size(),
                 )?)
+            },
+        )
     }
 }
 

crates/clawhdf5/src/reader.rs

@@ -426,7 +426,6 @@ impl<'f> Dataset<'f> {
         Ok(data_read::read_as_strings(&raw, &dt)?)
     }
 
-
     // ----- Selection-based read methods -----
 
     /// Read selected elements as raw bytes.
@@ -448,7 +447,25 @@ impl<'f> Dataset<'f> {
         let dt = self.datatype()?;
         let ds = self.dataspace()?;
         let dl = self.data_layout()?;
-        let pipeline = self.filter_pipeline();
+        let pipeline = self.filter_pipeline()?;
+        // The selection reader knows nothing about fill values. When they
+        // matter — no storage at all, or a non-zero fill on a chunked (possibly
+        // sparse) dataset — select from a fill-aware full read instead. (The
+        // selection reader currently decodes the full dataset too, so this
+        // costs nothing extra.)
+        let fill = clawhdf5_format::fill_value::dataset_fill_value(&self.header.messages)?;
+        let fill_matters = !clawhdf5_format::fill_value::has_storage(&dl)
+            || (matches!(dl, DataLayout::Chunked { .. })
+                && !clawhdf5_format::fill_value::is_default(fill.as_deref()));
+        if fill_matters {
+            let full = self.read_raw()?;
+            return Ok(data_read::extract_selection_from_buffer(
+                &full,
+                &ds.dimensions,
+                dt.type_size() as usize,
+                selection,
+            )?);
+        }
         Ok(data_read::read_raw_data_selection(
             self.file.data.as_bytes(),
             &dl,
@@ -724,15 +741,43 @@ impl<'f> Dataset<'f> {
         )?)
     }
 
+    /// A header message's payload, resolved through the shared-message
+    /// indirection when needed (e.g. a committed datatype). See
+    /// [`clawhdf5_format::shared_message::message_data`].
+    fn message_payload(
+        &self,
+        msg_type: MessageType,
+    ) -> Result<Option<std::borrow::Cow<'_, [u8]>>, Error> {
+        self.header
+            .messages
+            .iter()
+            .find(|m| m.msg_type == msg_type)
+            .map(|msg| {
+                clawhdf5_format::shared_message::message_data(
+                    self.file.as_bytes(),
+                    msg,
+                    self.file.offset_size(),
+                    self.file.length_size(),
+                )
+                .map_err(Error::Format)
+            })
+            .transpose()
+    }
+
+    fn required_payload(&self, msg_type: MessageType) -> Result<std::borrow::Cow<'_, [u8]>, Error> {
+        self.message_payload(msg_type)?
+            .ok_or(Error::MissingMessage(msg_type))
+    }
+
     fn datatype(&self) -> Result<Datatype, Error> {
-        let msg = find_message(&self.header, MessageType::Datatype)?;
+        let data = self.required_payload(MessageType::Datatype)?;
-        let (dt, _) = Datatype::parse(&msg.data)?;
+        let (dt, _) = Datatype::parse(&data)?;
         Ok(dt)
     }
 
     fn dataspace(&self) -> Result<Dataspace, Error> {
-        let msg = find_message(&self.header, MessageType::Dataspace)?;
+        let data = self.required_payload(MessageType::Dataspace)?;
-        Ok(Dataspace::parse(&msg.data, self.file.length_size())?)
+        Ok(Dataspace::parse(&data, self.file.length_size())?)
     }
 
     fn data_layout(&self) -> Result<DataLayout, Error> {
@@ -744,19 +789,21 @@ impl<'f> Dataset<'f> {
         )?)
     }
 
-    fn filter_pipeline(&self) -> Option<FilterPipeline> {
+    /// `Ok(None)` means the dataset has no filter pipeline. A pipeline message
-        self.header
+    /// that is present but unparseable is an error: treating it as "no
-            .messages
+    /// filters" would hand the caller the still-compressed bytes as if they
-            .iter()
+    /// were the data.
-            .find(|m| m.msg_type == MessageType::FilterPipeline)
+    fn filter_pipeline(&self) -> Result<Option<FilterPipeline>, Error> {
-            .and_then(|msg| FilterPipeline::parse(&msg.data).ok())
+        self.message_payload(MessageType::FilterPipeline)?
+            .map(|data| FilterPipeline::parse(&data).map_err(Error::Format))
+            .transpose()
     }
 
     fn read_raw(&self) -> Result<Vec<u8>, Error> {
         let dt = self.datatype()?;
         let ds = self.dataspace()?;
         let dl = self.data_layout()?;
-        let pipeline = self.filter_pipeline();
+        let pipeline = self.filter_pipeline()?;
 
         // Virtual datasets are assembled from source datasets; the per-file
         // chunk cache does not apply. Route them through the resolver path so
@@ -765,7 +812,7 @@ impl<'f> Dataset<'f> {
             let base_dir = self.file.base_dir.clone();
             let resolver = move |name: &str| -> Option<Vec<u8>> {
                 let dir = base_dir.as_ref()?;
-                std::fs::read(dir.join(name)).ok()
+                std::fs::read(dir.join(sibling_file_name(name)?)).ok()
             };
             return Ok(data_read::read_raw_data_full_with_resolver(
                 self.file.data.as_bytes(),
@@ -779,6 +826,16 @@ impl<'f> Dataset<'f> {
             )?);
         }
 
+        // Unallocated storage reads as the dataset's fill value.
+        clawhdf5_format::fill_value::read_full_with_fill(
+            &self.header.messages,
+            self.file.data.as_bytes(),
+            &dl,
+            &ds,
+            dt.type_size() as usize,
+            self.file.offset_size(),
+            self.file.length_size(),
+            || {
                 Ok(data_read::read_raw_data_cached(
                     self.file.data.as_bytes(),
                     &dl,
@@ -789,6 +846,8 @@ impl<'f> Dataset<'f> {
                     self.file.length_size(),
                     &self.file.chunk_cache,
                 )?)
+            },
+        )
     }
 }
 
@@ -829,6 +888,23 @@ fn datatype_byte_order(dt: &Datatype) -> DatatypeByteOrder {
     }
 }
 
+/// A source-file name taken from inside an HDF5 file, accepted only if it
+/// stays within the directory of the file that named it.
+///
+/// The name is untrusted input. Joining it blindly lets a crafted file make
+/// the reader open any path the process can reach — an absolute path replaces
+/// the base directory entirely, and `..` components climb out of it. Only
+/// plain relative paths made of normal components are allowed.
+fn sibling_file_name(name: &str) -> Option<&std::path::Path> {
+    use std::path::Component;
+    let path = std::path::Path::new(name);
+    let mut components = path.components().peekable();
+    components.peek()?;
+    components
+        .all(|c| matches!(c, Component::Normal(_) | Component::CurDir))
+        .then_some(path)
+}
+
 fn find_message(
     header: &ObjectHeader,
     msg_type: MessageType,
@@ -882,3 +958,24 @@ fn resolve_group_entries(
         Ok(Vec::new())
     }
 }
+
+#[cfg(test)]
+mod sibling_file_name_tests {
+    use super::sibling_file_name;
+
+    #[test]
+    fn only_paths_inside_the_base_directory_are_accepted() {
+        for ok in ["source.h5", "./source.h5", "sub/dir/source.h5"] {
+            assert!(sibling_file_name(ok).is_some(), "{ok}");
+        }
+        for bad in [
+            "",
+            "/etc/passwd",
+            "../secret.h5",
+            "sub/../../secret.h5",
+            "sub/../ok.h5",
+        ] {
+            assert!(sibling_file_name(bad).is_none(), "{bad}");
+        }
+    }
+}

crates/clawhdf5/src/types.rs

@@ -163,13 +163,53 @@ pub(crate) fn attrs_to_map(
 ) -> HashMap<String, AttrValue> {
     let mut map = HashMap::new();
     for attr in attrs {
-        if let Some(val) = decode_attr_value(attr, file_data, offset_size, length_size) {
+        // Every attribute is reported. One that has no dedicated `AttrValue`
+        // variant, or that fails to decode as its declared type, is returned
+        // verbatim as `AttrValue::Raw` rather than dropped — a partial
+        // attribute list with no indication anything is missing is worse than
+        // an undecoded value.
+        let val =
+            decode_attr_value(attr, file_data, offset_size, length_size).unwrap_or_else(|| {
+                AttrValue::Raw {
+                    datatype: attr.datatype.clone(),
+                    shape: attr.dataspace.dimensions.clone(),
+                    data: attr.raw_data.clone(),
+                }
+            });
         map.insert(attr.name.clone(), val);
     }
-    }
     map
 }
 
+/// `Some(values)` if `attr` is a numpy/h5py-style boolean: an enumeration over
+/// an integer base type whose members are exactly `FALSE` = 0 and `TRUE` = 1.
+/// This is how `attrs["flag"] = True` is stored. Reported as 0/1 integers.
+fn decode_bool_enum(attr: &clawhdf5_format::attribute::AttributeMessage) -> Option<Vec<i64>> {
+    use clawhdf5_format::datatype::Datatype;
+
+    let Datatype::Enumeration {
+        base_type, members, ..
+    } = &attr.datatype
+    else {
+        return None;
+    };
+    if members.len() != 2 {
+        return None;
+    }
+    let member_value = |name: &str| {
+        let m = members.iter().find(|m| m.name.eq_ignore_ascii_case(name))?;
+        clawhdf5_format::data_read::read_as_i64(&m.value, base_type)
+            .ok()?
+            .first()
+            .copied()
+    };
+    if member_value("FALSE")? != 0 || member_value("TRUE")? != 1 {
+        return None;
+    }
+    let values = clawhdf5_format::data_read::read_as_i64(&attr.raw_data, base_type).ok()?;
+    values.iter().all(|v| *v == 0 || *v == 1).then_some(values)
+}
+
 fn decode_attr_value(
     attr: &clawhdf5_format::attribute::AttributeMessage,
     file_data: &[u8],
@@ -200,12 +240,7 @@ fn decode_attr_value(
             if vals.len() == 1 {
                 Some(AttrValue::U64(vals[0]))
             } else {
-                // No U64Array variant, store as I64Array.
+                Some(AttrValue::U64Array(vals))
-                // NOTE: This cast is lossy for values > i64::MAX (bit 63 set).
-                // Those values will appear as negative i64. A dedicated U64Array
-                // variant would be needed to handle the full u64 range.
-                let i64_vals: Vec<i64> = vals.iter().map(|&v| v as i64).collect();
-                Some(AttrValue::I64Array(i64_vals))
             }
         }
         Datatype::String { .. } => {
@@ -228,6 +263,14 @@ fn decode_attr_value(
                 Some(AttrValue::StringArray(strings))
             }
         }
+        Datatype::Enumeration { .. } => {
+            let vals = decode_bool_enum(attr)?;
+            if vals.len() == 1 {
+                Some(AttrValue::I64(vals[0]))
+            } else {
+                Some(AttrValue::I64Array(vals))
+            }
+        }
         _ => None,
     }
 }

crates/clawhdf5/src/writer.rs

@@ -72,7 +72,7 @@ impl FileBuilder {
     /// Serialize and write the file to the given path.
     pub fn write<P: AsRef<std::path::Path>>(self, path: P) -> Result<(), Error> {
         let bytes = self.finish()?;
-        std::fs::write(path, bytes).map_err(Error::Io)
+        write_file_atomically(path.as_ref(), &bytes).map_err(Error::Io)
     }
 }
 
@@ -186,3 +186,86 @@ pub fn create_datasets_parallel(specs: Vec<DatasetSpec>) -> Result<Vec<u8>, Erro
     let bytes = clawhdf5_format::file_writer::finalize_parallel(blocks)?;
     Ok(bytes)
 }
+
+/// Write `bytes` to `path` so that a crash or power loss leaves either the old
+/// file or the complete new one — never a truncated mix. `std::fs::write`
+/// truncates the destination first, so dying mid-write used to destroy the
+/// existing file.
+fn write_file_atomically(path: &std::path::Path, bytes: &[u8]) -> std::io::Result<()> {
+    use std::io::Write;
+
+    // Same directory as the target, so the rename stays on one filesystem.
+    let mut tmp_name = path
+        .file_name()
+        .ok_or_else(|| {
+            std::io::Error::new(std::io::ErrorKind::InvalidInput, "path has no file name")
+        })?
+        .to_os_string();
+    tmp_name.push(format!(".tmp-{}", std::process::id()));
+    let tmp_path = path.with_file_name(tmp_name);
+
+    let result = (|| {
+        let mut f = std::fs::File::create(&tmp_path)?;
+        f.write_all(bytes)?;
+        f.sync_all()?;
+        std::fs::rename(&tmp_path, path)
+    })();
+    if result.is_err() {
+        let _ = std::fs::remove_file(&tmp_path);
+        return result;
+    }
+    // Make the rename itself durable. Best-effort: not every filesystem
+    // supports syncing a directory, and the new file is already in place.
+    #[cfg(unix)]
+    if let Some(dir) = path.parent() {
+        let dir = if dir.as_os_str().is_empty() {
+            std::path::Path::new(".")
+        } else {
+            dir
+        };
+        if let Ok(d) = std::fs::File::open(dir) {
+            let _ = d.sync_all();
+        }
+    }
+    Ok(())
+}
+
+#[cfg(test)]
+mod atomic_write_tests {
+    use super::write_file_atomically;
+
+    fn entries(dir: &std::path::Path) -> Vec<String> {
+        let mut names: Vec<String> = std::fs::read_dir(dir)
+            .unwrap()
+            .map(|e| e.unwrap().file_name().to_string_lossy().into_owned())
+            .collect();
+        names.sort();
+        names
+    }
+
+    #[test]
+    fn replaces_existing_file_and_leaves_no_temp_behind() {
+        let dir = tempfile::TempDir::new().unwrap();
+        let path = dir.path().join("out.h5");
+        std::fs::write(&path, b"old contents").unwrap();
+
+        write_file_atomically(&path, b"new").unwrap();
+
+        assert_eq!(std::fs::read(&path).unwrap(), b"new");
+        assert_eq!(entries(dir.path()), ["out.h5"]);
+    }
+
+    #[test]
+    fn failure_leaves_the_existing_file_untouched() {
+        let dir = tempfile::TempDir::new().unwrap();
+        // The target is a directory, so the final rename cannot succeed.
+        let path = dir.path().join("taken");
+        std::fs::create_dir(&path).unwrap();
+        std::fs::write(path.join("keep"), b"x").unwrap();
+
+        assert!(write_file_atomically(&path, b"new").is_err());
+
+        assert!(path.is_dir());
+        assert_eq!(entries(dir.path()), ["taken"], "temp file cleaned up");
+    }
+}

crates/clawhdf5/tests/h5py_interop_tests.rs

@@ -10,6 +10,12 @@ use clawhdf5::{AttrValue, CompoundTypeBuilder, DType, File, FileBuilder};
 // Helpers
 // ---------------------------------------------------------------------------
 
+/// When `CLAWHDF5_REQUIRE_INTEROP=1` (set in CI), a missing Python dependency
+/// is a test failure instead of a silent skip.
+fn interop_required() -> bool {
+    std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
+}
+
 fn python_available() -> bool {
     Command::new("python3")
         .args(["-c", "import h5py; print(h5py.__version__)"])
@@ -21,6 +27,10 @@ fn python_available() -> bool {
 macro_rules! skip_if_no_python {
     () => {
         if !python_available() {
+            assert!(
+                !interop_required(),
+                "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+            );
             eprintln!("SKIP: python3 with h5py not available");
             return;
         }
@@ -555,3 +565,351 @@ print("OK")
     );
     assert_eq!(run_python_output(&script), "OK");
 }
+
+// ---------------------------------------------------------------------------
+// h5py uses committed (named) datatypes -> clawhdf5 reads
+// ---------------------------------------------------------------------------
+
+/// A dataset or attribute created from a committed datatype stores only a
+/// *shared message* reference to it. These used to be parsed as the datatype
+/// itself (yielding `Time { size: 0 }` and unreadable data) and the attribute
+/// was silently dropped.
+#[test]
+fn h5py_committed_datatypes_clawhdf5_reads() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    for (tag, kwargs) in [("default", ""), ("latest", ", libver='latest'")] {
+        let path = dir.path().join(format!("committed_{tag}.h5"));
+        let path_str = path.display().to_string();
+        let script = format!(
+            r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w"{kwargs}) as f:
+    f["f8type"] = np.dtype("<f8")
+    f["cmpd"] = np.dtype([("a", "<i4"), ("b", "<f8")])
+    f.create_dataset("d", data=np.arange(6, dtype="<f8"), dtype=f["f8type"])
+    f.create_dataset("c", data=np.array([(1, 2.5), (3, 4.5)], dtype=f["cmpd"].dtype), dtype=f["cmpd"])
+    f["d"].attrs.create("att", 7.0, dtype=f["f8type"])
+"#
+        );
+        run_python(&script);
+
+        let file = File::open(&path).unwrap();
+        let d = file.dataset("d").unwrap();
+        assert_eq!(d.dtype().unwrap(), DType::F64, "{tag}");
+        assert_eq!(
+            d.read_f64().unwrap(),
+            vec![0.0, 1.0, 2.0, 3.0, 4.0, 5.0],
+            "{tag}"
+        );
+        assert!(
+            matches!(d.attrs().unwrap().get("att"), Some(AttrValue::F64(v)) if *v == 7.0),
+            "{tag}: attribute with a committed datatype"
+        );
+        let c = file.dataset("c").unwrap();
+        assert_eq!(
+            c.dtype().unwrap(),
+            DType::Compound(vec![("a".into(), DType::I32), ("b".into(), DType::F64)]),
+            "{tag}"
+        );
+    }
+}
+
+// ---------------------------------------------------------------------------
+// h5py writes sparse / never-written datasets -> clawhdf5 applies fill values
+// ---------------------------------------------------------------------------
+
+/// Parse h5py's `print(arr.ravel().tolist())` output for integer data.
+fn parse_int_list(s: &str) -> Vec<i32> {
+    s.trim()
+        .trim_matches(|c| c == '[' || c == ']')
+        .split(',')
+        .filter(|t| !t.trim().is_empty())
+        .map(|t| t.trim().parse().unwrap())
+        .collect()
+}
+
+/// Storage HDF5 never allocated must read as the dataset's fill value. These
+/// used to read as zeros (silently wrong for a non-zero fill value) or fail
+/// outright (`NoDataAllocated`) for a dataset that was never written.
+#[test]
+fn h5py_fill_values_clawhdf5_reads() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    for (tag, kwargs) in [("default", ""), ("latest", ", libver='latest'")] {
+        let path = dir.path().join(format!("fill_{tag}.h5"));
+        let path_str = path.display().to_string();
+        let script = format!(
+            r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w"{kwargs}) as f:
+    d = f.create_dataset("partial", shape=(20,), dtype="<i4", chunks=(5,), fillvalue=-1)
+    d[0:5] = np.arange(5)
+    f.create_dataset("never", shape=(4,), dtype="<i4", fillvalue=25)
+    f.create_dataset("never_chunked", shape=(6,), dtype="<i4", chunks=(3,), fillvalue=9)
+    f.create_dataset("default_fill", shape=(3,), dtype="<i4")
+    g = f.create_dataset("gz", shape=(20,), dtype="<i4", chunks=(5,), fillvalue=7, compression="gzip")
+    g[10:15] = 1
+    s = f.create_dataset("sparse2d", shape=(5, 7), dtype="<i4", chunks=(2, 3), fillvalue=-3)
+    s[2:4, 3:6] = 8
+    s[4, 6] = 5
+with h5py.File("{path_str}", "r") as f:
+    for name in ["partial", "never", "never_chunked", "default_fill", "gz", "sparse2d"]:
+        print(name, f[name][...].ravel().tolist())
+    print("slab", f["sparse2d"][1:5, 2:7].ravel().tolist())
+"#
+        );
+        let expected: std::collections::HashMap<String, Vec<i32>> = run_python_output(&script)
+            .lines()
+            .map(|line| {
+                let (name, list) = line.split_once(' ').unwrap();
+                (name.to_string(), parse_int_list(list))
+            })
+            .collect();
+
+        let file = File::open(&path).unwrap();
+        for name in [
+            "partial",
+            "never",
+            "never_chunked",
+            "default_fill",
+            "gz",
+            "sparse2d",
+        ] {
+            assert_eq!(
+                file.dataset(name).unwrap().read_i32().unwrap(),
+                expected[name],
+                "{tag}/{name}"
+            );
+        }
+        // A hyperslab straddling allocated and unallocated chunks.
+        let slab = clawhdf5_format::selection::Selection::Hyperslab {
+            start: vec![1, 2],
+            stride: vec![1, 1],
+            count: vec![4, 5],
+            block: vec![1, 1],
+        };
+        assert_eq!(
+            file.dataset("sparse2d")
+                .unwrap()
+                .read_i32_selection(&slab)
+                .unwrap(),
+            expected["slab"],
+            "{tag}/sparse2d hyperslab"
+        );
+    }
+}
+
+// ---------------------------------------------------------------------------
+// h5py writes soft / external links and external raw data -> clawhdf5
+// ---------------------------------------------------------------------------
+
+/// Soft links are followed (absolute, relative, through groups, with a cycle
+/// guard). Things this reader does not follow — external links, and datasets
+/// whose raw data lives in another file — are explicit errors. They used to
+/// surface as a misleading `PathNotFound`, and external raw data could read
+/// back as fill values.
+#[test]
+fn h5py_links_clawhdf5_resolves_or_refuses() {
+    use clawhdf5::Error;
+    use clawhdf5_format::error::FormatError;
+
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let dir_str = dir.path().display().to_string();
+    for (tag, kwargs) in [("default", ""), ("latest", ", libver='latest'")] {
+        let script = format!(
+            r#"
+import h5py, numpy as np, os
+os.chdir("{dir_str}")
+with h5py.File("other_{tag}.h5", "w"{kwargs}) as o:
+    o.create_dataset("remote", data=np.arange(3, dtype="<i4"))
+with h5py.File("links_{tag}.h5", "w"{kwargs}) as f:
+    f.create_dataset("real", data=np.arange(4, dtype="<i4"))
+    g = f.create_group("grp")
+    g.create_dataset("inner", data=np.arange(2, dtype="<i4"))
+    g["rel"] = h5py.SoftLink("inner")
+    f["soft"] = h5py.SoftLink("/real")
+    f["soft_grp"] = h5py.SoftLink("/grp")
+    f["dangling"] = h5py.SoftLink("/nope")
+    f["loop_a"] = h5py.SoftLink("/loop_b")
+    f["loop_b"] = h5py.SoftLink("/loop_a")
+    f["ext"] = h5py.ExternalLink("other_{tag}.h5", "/remote")
+    f.create_dataset("extdata", shape=(4,), dtype="<i4", external=[("raw_{tag}.bin", 0, 16)])
+    f["extdata"][...] = np.array([11, 22, 33, 44], dtype="<i4")
+"#
+        );
+        run_python(&script);
+
+        let file = File::open(dir.path().join(format!("links_{tag}.h5"))).unwrap();
+        let read = |path: &str| file.dataset(path).and_then(|d| d.read_i32());
+        assert_eq!(read("soft").unwrap(), vec![0, 1, 2, 3], "{tag}");
+        assert_eq!(read("soft_grp/inner").unwrap(), vec![0, 1], "{tag}");
+        assert_eq!(
+            read("grp/rel").unwrap(),
+            vec![0, 1],
+            "{tag}: relative target"
+        );
+        assert_eq!(
+            read("soft_grp/rel").unwrap(),
+            vec![0, 1],
+            "{tag}: link via link"
+        );
+
+        assert!(
+            matches!(read("dangling"), Err(Error::Format(FormatError::PathNotFound(p))) if p == "nope"),
+            "{tag}: dangling link names its missing target"
+        );
+        assert!(
+            matches!(
+                read("loop_a"),
+                Err(Error::Format(FormatError::NestingDepthExceeded))
+            ),
+            "{tag}: link cycle"
+        );
+        assert!(
+            matches!(
+                read("ext"),
+                Err(Error::Format(FormatError::ExternalLinkUnsupported { ref object_path, .. }))
+                    if object_path == "/remote"
+            ),
+            "{tag}: external link"
+        );
+        assert!(
+            matches!(
+                read("extdata"),
+                Err(Error::Format(FormatError::ExternalDataFilesUnsupported))
+            ),
+            "{tag}: external raw data must not read as fill values"
+        );
+    }
+}
+
+// ---------------------------------------------------------------------------
+// Attribute fidelity: nothing is dropped, unsigned stays unsigned
+// ---------------------------------------------------------------------------
+
+/// `attrs()` used to omit, without any error, every attribute whose datatype
+/// had no `AttrValue` variant — including every Python `bool` (stored as an
+/// enum) — and to wrap unsigned 64-bit arrays into negative `i64`s.
+#[test]
+fn h5py_attribute_kinds_are_all_reported() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("attrs_kinds.h5");
+    let path_str = path.display().to_string();
+    let script = format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w") as f:
+    d = f.create_dataset("d", data=np.arange(3))
+    a = d.attrs
+    a["float"] = 1.5; a["int"] = 7; a["str"] = "hello"; a["int_array"] = np.arange(4)
+    a["str_list"] = ["a", "bc"]
+    a["uint64_array"] = np.array([2**63, 1], dtype="u8")
+    a["bool_true"] = True
+    a["bool_false"] = False
+    a["bool_array"] = np.array([True, False, True])
+    a["complex"] = 1 + 2j
+    a["compound"] = np.array([(1, 2.5)], dtype=[("a", "<i4"), ("b", "<f8")])
+    a["object_ref"] = d.ref
+    a["color"] = np.array(2, dtype=h5py.enum_dtype({{"RED": 0, "GREEN": 1, "BLUE": 2}}, basetype="i1"))
+    print(len(a))
+"#
+    );
+    let written: usize = run_python_output(&script).trim().parse().unwrap();
+
+    let file = File::open(&path).unwrap();
+    let attrs = file.dataset("d").unwrap().attrs().unwrap();
+    assert_eq!(
+        attrs.len(),
+        written,
+        "every attribute is reported: {attrs:?}"
+    );
+
+    // Booleans decode as 0/1.
+    assert!(matches!(attrs["bool_true"], AttrValue::I64(1)));
+    assert!(matches!(attrs["bool_false"], AttrValue::I64(0)));
+    assert!(matches!(&attrs["bool_array"], AttrValue::I64Array(v) if v == &[1, 0, 1]));
+    // Unsigned stays unsigned.
+    assert!(matches!(&attrs["uint64_array"], AttrValue::U64Array(v) if v == &[1u64 << 63, 1]));
+
+    // A general enum is not a boolean: kept verbatim, member names intact.
+    match &attrs["color"] {
+        AttrValue::Raw { datatype, data, .. } => {
+            assert_eq!(data, &[2]);
+            assert!(format!("{datatype:?}").contains("BLUE"));
+        }
+        other => panic!("color: {other:?}"),
+    }
+    // A compound attribute is kept verbatim and decodes against its datatype.
+    match &attrs["compound"] {
+        AttrValue::Raw {
+            datatype,
+            shape,
+            data,
+        } => {
+            assert_eq!(shape, &[1]);
+            let fields = clawhdf5_format::data_read::read_compound_fields(data, datatype).unwrap();
+            assert_eq!(fields[0].name, "a");
+            let b =
+                clawhdf5_format::data_read::read_as_f64(&fields[1].raw_data, &fields[1].datatype)
+                    .unwrap();
+            assert_eq!(b, vec![2.5]);
+        }
+        other => panic!("compound: {other:?}"),
+    }
+    for name in ["complex", "object_ref"] {
+        assert!(
+            matches!(attrs[name], AttrValue::Raw { .. }),
+            "{name}: {:?}",
+            attrs[name]
+        );
+    }
+}
+
+/// `Raw` and `U64Array` are writable, so an attribute read from one file can
+/// be stored in another unchanged.
+#[test]
+fn clawhdf5_writes_raw_and_unsigned_attrs_h5py_reads() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let src = dir.path().join("src.h5");
+    let dst = dir.path().join("dst.h5");
+    let (src_str, dst_str) = (src.display().to_string(), dst.display().to_string());
+    run_python(&format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{src_str}", "w") as f:
+    d = f.create_dataset("d", data=np.arange(3))
+    d.attrs["compound"] = np.array([(1, 2.5), (3, 4.5)], dtype=[("a", "<i4"), ("b", "<f8")])
+"#
+    ));
+    let compound = File::open(&src)
+        .unwrap()
+        .dataset("d")
+        .unwrap()
+        .attrs()
+        .unwrap()["compound"]
+        .clone();
+
+    let mut builder = FileBuilder::new();
+    let ds = builder.create_dataset("d");
+    ds.with_f64_data(&[1.0]);
+    ds.set_attr("compound", compound);
+    ds.set_attr("big", AttrValue::U64Array(vec![u64::MAX, 0, 1 << 63]));
+    builder.write(&dst).unwrap();
+
+    let out = run_python_output(&format!(
+        r#"
+import h5py
+with h5py.File("{dst_str}", "r") as f:
+    a = f["d"].attrs
+    print(a["compound"].tolist(), a["compound"].dtype.names, a["big"].tolist(), a["big"].dtype)
+"#
+    ));
+    assert_eq!(
+        out.trim(),
+        "[(1, 2.5), (3, 4.5)] ('a', 'b') [18446744073709551615, 0, 9223372036854775808] uint64"
+    );
+}

crates/clawhdf5/tests/integration_tests.rs

@@ -955,8 +955,10 @@ fn read_selection_all_matches_read_raw_on_chunked_dataset() {
     let via_read_f64 = ds.read_f64().unwrap();
     let via_selection_bytes = ds.read_selection(&Selection::All).unwrap();
     let via_selection: Vec<f64> = via_selection_bytes
-        .chunks_exact(8)
+        .as_chunks::<8>()
-        .map(|c| f64::from_le_bytes(c.try_into().unwrap()))
+        .0
+        .iter()
+        .map(|c| f64::from_le_bytes(*c))
         .collect();
 
     assert_eq!(via_read_f64, data);

crates/clawhdf5/tests/zerocopy_tests.rs

@@ -315,7 +315,7 @@ fn zerocopy_roundtrip_write_read() {
     let dir = std::env::temp_dir();
     let path = dir.join("zc_roundtrip.h5");
 
-    let original = vec![3.14, 2.718, 1.414, 1.732, 0.577];
+    let original = vec![3.25, 2.75, 1.414, 1.732, 0.577];
     let mut b = FileBuilder::new();
     b.create_dataset("data")
         .with_f64_data(&original)

docs/known-issues.md

@@ -72,11 +72,68 @@ generated yet; one written with the C API (`H5T_STD_REF`) is needed.
 
 ## `clawhdf5-gpu` `gpu_tests` can hang under the default parallel test runner
 
-**Status:** open. Observed 2026-09-18 (RTX 5060 Ti, Linux).
+**Status:** fixed 2026-09-19.
 
-**Summary:** during `cargo test --workspace`, the `gpu_tests` binary sat idle
+**Summary:** during `cargo test --workspace` the `gpu_tests` binary sat idle for
-(~1% CPU) for 25+ minutes and had to be killed. Run single-threaded it passes
+25+ minutes. Every test created its own `wgpu::Instance` + device (requesting
-in seconds (20/20): `cargo test -p clawhdf5-gpu --test gpu_tests -- --test-threads=1`.
+adapter-maximum limits) concurrently, and readback used an unbounded
-Suspected cause: several tests creating wgpu devices concurrently (possibly
+`device.poll(Wait)`.
-compounded by the rest of the workspace's tests loading the machine). Not yet
+
-root-caused; workaround is `--test-threads=1` for that crate.
+**Fix:** tests hold a process-wide lock while they own a device, and
+`GpuAccelerator` readback waits time out after 30 s with `GpuError::BufferMap`.
+
+## Compound datatype versions 1 and 2 are mis-parsed (default libver files)
+
+**Status:** fixed 2026-09-19. Found by adding a default-libver axis to the h5py
+interop tests.
+
+**Summary:** any compound dataset written with default libver bounds (plain
+`h5py.File(path, 'w')`, datatype message version 1) failed to read, typically
+with `Overflow("compound member 'x': byte_offset(0) + field_size(4136977) ...")`.
+Only `libver='latest'` files (version 3+) and files written by clawhdf5 itself
+worked, which is why the existing tests never caught it.
+
+**Root cause:** `Datatype::parse` skipped 24 bytes of legacy per-member array
+fields for v1 where the format has 28 (dimensionality 1 + reserved 3 +
+permutation 4 + reserved 4 + 4 dimension sizes 16), and treated v2 like v1 minus
+name padding, whereas v2 keeps the 8-byte name padding and has no array fields.
+
+## Attributes with unsupported datatypes are silently dropped
+
+**Status:** fixed 2026-09-19.
+
+**Summary:** `Dataset::attrs()` / `Group::attrs()` returned only attributes
+convertible to `AttrValue` and omitted the rest without any indication — every
+Python `bool` (an HDF5 enum), complex, compound and reference attributes. Unsigned
+64-bit arrays were also cast to `I64Array`, turning values above `i64::MAX`
+negative.
+
+**Fix:** booleans decode as 0/1 integers, `AttrValue::U64Array` keeps unsigned
+arrays unsigned, and `AttrValue::Raw { datatype, shape, data }` carries any other
+attribute verbatim. Both new variants are writable. Still lossy: a
+multi-dimensional numeric attribute is returned as a flat array (its shape is
+not reported).
+
+## B-tree v2 chunk index (layout v4, index type 5) is not supported
+
+**Status:** open.
+
+**Summary:** a chunked dataset with **two or more unlimited dimensions** written
+with `libver='latest'` indexes its chunks with a version-2 B-tree. Reading it
+fails with `ChunkedReadError("unsupported chunked layout version=4,
+index_type=Some(5)")`. Single-chunk, implicit, fixed-array and
+extensible-array indexes (and the v3 B-tree v1) are supported.
+
+**Repro:** `f.create_dataset("d", shape=(5, 7), chunks=(2, 3), maxshape=(None, None))`
+with `h5py.File(..., libver='latest')`.
+
+## External links and external raw data are not followed
+
+**Status:** open (by design for now); both are explicit errors.
+
+**Summary:** a path through an external link returns
+`FormatError::ExternalLinkUnsupported { filename, object_path }`, and a dataset
+created with `external=[...]` storage returns
+`FormatError::ExternalDataFilesUnsupported`. Neither is resolved. If support is
+added, file names must be confined to the opened file's directory, as the
+virtual-dataset resolver now does.

packages/clawhdf5-node/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@redclaw/clawhdf5",
-  "version": "2.2.0",
+  "version": "2.3.0",
   "description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
   "main": "index.js",
   "types": "index.d.ts",

scripts/ci-test.sh

@@ -1,9 +1,18 @@
 #!/usr/bin/env bash
-# CI test script — runs fmt, clippy, tests, and no_std checks.
+# CI test script — runs fmt, clippy (all targets + feature matrix), tests,
+# Python interop suites, bench compilation, and no_std checks.
 #
 # Usage:
 #   ./scripts/ci-test.sh
 #
+# Environment:
+#   CLAWHDF5_REQUIRE_INTEROP=1  Fail (instead of skip) when python3 with
+#                               h5py/netCDF4/xarray is missing. CI sets this.
+#                               Unset locally, the interop steps are skipped
+#                               if python3+h5py is not importable.
+#   CLAWHDF5_FUZZ_SECONDS=N     Run each cargo-fuzz target for N seconds
+#                               (needs nightly + cargo-fuzz). Default: skip.
+#
 # Exit codes:
 #   0 — all checks passed
 #   1 — one or more checks failed
@@ -31,23 +40,74 @@ run_step() {
     fi
 }
 
+# All steps always run so one failure doesn't hide the others; the summary
+# and the exit code at the end are the verdict.
+
 # 1. Format check
 run_step "cargo fmt --check" cargo fmt --check
 
-# 2. Clippy (exclude clawhdf5-py which needs PyO3/Python)
+# 2. Clippy over every target (lib, bins, tests, benches, examples). Without
-run_step "cargo clippy" cargo clippy \
+#    --all-targets, test and bench code is never linted. clawhdf5-py is
+#    excluded because it needs PyO3/Python headers.
+run_step "cargo clippy --all-targets" cargo clippy \
     --workspace \
     --exclude clawhdf5-py \
+    --all-targets \
     -- -D warnings
 
-# 3. Tests (exclude clawhdf5-py)
+# 3. Clippy over clawhdf5-format's optional features, which the default
+#    workspace build never compiles (szip is left out: it needs libaec).
+run_step "cargo clippy (format feature matrix)" cargo clippy \
+    -p clawhdf5-format \
+    --all-targets \
+    --features parallel,lz4,zstd,pcodec,fast-checksum \
+    -- -D warnings
+
+# 4. Tests (exclude clawhdf5-py)
 run_step "cargo test" cargo test \
     --workspace \
     --exclude clawhdf5-py
 
-# 4. no_std check
+run_step "cargo test (format feature matrix)" cargo test \
+    -p clawhdf5-format \
+    --features parallel,lz4,zstd,pcodec,fast-checksum
+
+# 5. Python interop suites. The h5py writer tests are #[ignore]d so a plain
+#    `cargo test` stays hermetic; run them explicitly here.
+if python3 -c "import h5py" >/dev/null 2>&1 || [ "${CLAWHDF5_REQUIRE_INTEROP:-0}" = "1" ]; then
+    run_step "h5py interop (format, ignored tests)" cargo test \
+        -p clawhdf5-format --test writer_h5py_tests -- --include-ignored
+else
+    echo ""
+    echo "==> [h5py interop] SKIPPED: python3 with h5py not available"
+    STEPS+=("SKIP: h5py interop (format, ignored tests)")
+fi
+
+# 6. Benches must keep compiling (they are not run).
+run_step "cargo bench --no-run" cargo bench \
+    --workspace \
+    --exclude clawhdf5-py \
+    --no-run
+
+# 7. no_std check
 run_step "check-nostd.sh" "$SCRIPT_DIR/check-nostd.sh"
 
+# 8. Optional fuzz smoke run
+if [ -n "${CLAWHDF5_FUZZ_SECONDS:-}" ]; then
+    fuzz_smoke() {
+        local crate target
+        for crate in clawhdf5-format clawhdf5-agent; do
+            cd "$SCRIPT_DIR/../crates/$crate" || return 1
+            for target in $(cargo +nightly fuzz list); do
+                echo "--- fuzz: $crate/$target"
+                cargo +nightly fuzz run "$target" -- \
+                    -max_total_time="$CLAWHDF5_FUZZ_SECONDS" || return 1
+            done
+        done
+    }
+    run_step "fuzz smoke (${CLAWHDF5_FUZZ_SECONDS}s/target)" fuzz_smoke
+fi
+
 # Summary
 echo ""
 echo "========================================"