docs: ClawBrainHub is the one verified consumer

The previous commit said clawhdf5 has no integration at all. ClawBrainHub
(clawverse/clawbrainhub) does use it: cbh-core reads and writes .brain
files through the facade, cbh-scanner uses the facade, and cbh-cli uses
clawhdf5_agent::bm25::BM25Index, all via path dependencies on this repo.
Checked on 2026-09-25 against main: it builds on its pinned toolchain and
its 204 tests pass. CLAUDE.md now records that, and that path
dependencies mean API changes here reach it directly.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

.gitea/workflows/ci.yml

@@ -9,15 +9,15 @@ jobs:
     runs-on: ubuntu-latest
     container: rust:latest
     steps:
-      - uses: actions/checkout@v4
-      - name: Cache cargo registry/target
-        uses: actions/cache@v4
-        with:
-          path: |
-            ~/.cargo/registry
-            ~/.cargo/git
-            target
-          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+      # Plain git rather than actions/checkout: that is a JavaScript action,
+      # and rust:latest has no `node`, so it failed with exit 127 before any
+      # code was built — on every push. actions/cache went for the same reason.
+      - name: Check out
+        run: |
+          git init -q .
+          git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
+          for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
+          git checkout -q FETCH_HEAD
       - name: Install rustfmt & clippy components
         run: rustup component add rustfmt clippy
       - name: Install thumbv7em-none-eabihf target
@@ -28,7 +28,10 @@ jobs:
         # dependency a failure (CLAWHDF5_REQUIRE_INTEROP below).
         run: |
           apt-get update
-          apt-get install -y --no-install-recommends python3 python3-venv
+          # cmake builds libz-ng-sys for the opt-in `fast-deflate` (zlib-ng)
+          # steps in ci-test.sh; rust:latest does not ship it. The default
+          # build (pure-Rust zlib-rs) does not need it.
+          apt-get install -y --no-install-recommends python3 python3-venv cmake
           python3 -m venv /opt/interop
           /opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray
           echo "/opt/interop/bin" >> "$GITHUB_PATH"
@@ -44,3 +47,40 @@ jobs:
           CLAWHDF5_PYTHON: /opt/interop/bin/python
           CLAWHDF5_REQUIRE_INTEROP: "1"
         run: bash scripts/ci-test.sh
+
+  test-arm64:
+    # The aarch64 kernels in clawhdf5-accel — NEON `dot_i8`, including the
+    # SDOT path, and the f32 NEON kernels — are cfg'd out on x86, so the job
+    # above never compiles, lints or tests them.
+    #
+    # `linux_arm64` is served by two runners that execute differently:
+    # vision-01 runs steps on the host (Rust already installed) and vision-02
+    # runs them in docker.gitea.com/runner-images. So the steps work in both:
+    # no `container:`, no JavaScript actions (they are fetched from GitHub,
+    # which not every runner reliably reaches), and an explicit `+stable`
+    # toolchain rather than whatever a host happens to default to.
+    runs-on: linux_arm64
+    env:
+      CARGO_NET_RETRY: "10"
+      CARGO_TERM_COLOR: always
+    steps:
+      - name: Check out
+        run: |
+          git init -q .
+          git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
+          for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
+          git checkout -q FETCH_HEAD
+      - name: Rust stable
+        run: |
+          export PATH="$HOME/.cargo/bin:$PATH"
+          command -v rustup >/dev/null || curl -sSf --retry 5 https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain none
+          rustup toolchain install stable --profile minimal --component clippy
+          echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
+      - name: Confirm aarch64
+        run: |
+          test "$(uname -m)" = aarch64
+          if grep -q asimddp /proc/cpuinfo; then echo "dot-product extension present: SDOT kernel runs"; else echo "no dot-product extension: plain NEON kernel runs"; fi
+      - name: Clippy (aarch64 kernels)
+        run: cargo +stable clippy -p clawhdf5-accel --all-targets -- -D warnings
+      - name: Test
+        run: cargo +stable test -p clawhdf5-accel -p clawhdf5-ann -p clawhdf5-format

CHANGELOG.md

@@ -1,5 +1,409 @@
 # Changelog
 
+## Unreleased
+
+### Upgrade Notes
+- **ZeroClaw does not use clawhdf5.** The project described itself as
+  ZeroClaw's memory backend ("imported as a `clawhdf5` Cargo feature"). Checked
+  against ZeroClaw v0.8.5 (the latest release), the `osobh/zeroclaw` fork and
+  their full history: no such feature or backend has ever existed. And
+  `clawhdf5-migrate`'s "ZeroClaw layout" (`memory_chunks`, `sessions`,
+  `entities`, `relations`) is not ZeroClaw's schema — ZeroClaw uses a single
+  `memories` table — so the migrator cannot read a ZeroClaw database. The
+  claims are withdrawn; the migrator's layout is documented as its own.
+- **OpenClaw is not supported, and never was.** The docs described a
+  "drop-in" OpenClaw memory backend enabled with `memory.backend = "clawhdf5"`.
+  That config was never valid in any OpenClaw release (v2026.2–v2026.7
+  accepted only `builtin`/`qmd` and rejected unknown keys, so a Gateway given
+  it refuses to start; OpenClaw 2.0 removed the key), no plugin was ever built,
+  and `@redclaw/clawhdf5` was never published. The integration docs
+  (`openclaw-integration.md`, `openclaw-config.md`, `migration-guide.md`) are
+  removed; `docs/openclaw.md` explains the status and what a real plugin would
+  need against OpenClaw v2026.9.6. `ClawhdfBackend` stays as a library API.
+- **Breaking:** `MemoryError` is now `#[non_exhaustive]` and gained
+  `SigningKeyRequired`; a `match` on it needs a wildcard arm. Future variants
+  will no longer be breaking.
+- **Breaking:** `clawhdf5-agent`'s `agent` feature is removed. It enabled
+  nothing — the agent layer is always built — but the README and guides told
+  people to pass it; drop `agent` from `features = [...]`.
+- **`clawhdf5-migrate` now writes a real agent store.** Its output used to be
+  a layout of its own (`/chunks`, `/sessions`, `/entities`, `/relations`, no
+  `/meta`) that `HDF5Memory::open` rejected, so a migrated file could not be
+  used as agent memory. Files it wrote before this release are not agent
+  stores; re-run the migration. Also: embeddings default to `float16` like
+  any new store (`--f32` opts out; `--float16` is a hidden no-op); a row with
+  the wrong embedding length is an error instead of being truncated or
+  padded; `--incremental` now matches rows by content against an existing
+  store and follows the source's deleted flags; a source with no memory rows
+  needs `--embedding-dim`. The per-dataset SHA-256 provenance attributes of
+  the old layout are gone (the agent schema has no place for them).
+- **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
+  dataset we wrote — including every agent store's embeddings — was refused
+  with "sign bit position out of bounds", and every empty dataset with
+  "invalid dataset size". Both were write-side bugs present in every release;
+  clawhdf5's own reader was unaffected. An agent store is rewritten in full at
+  each checkpoint, so it becomes readable at its next checkpoint on this
+  version; other files with `f32` or empty datasets need rewriting. Details in
+  `docs/known-issues.md`.
+- **New stores store embeddings as half precision by default.**
+  `MemoryConfig::float16` was persisted and otherwise ignored; it now writes
+  `float16` embeddings (48% smaller files at 100K) and rounds each embedding
+  to half precision as it is saved — and it defaults to `true` for new
+  stores. On the full LongMemEval haystack with real MiniLM embeddings every
+  retrieval metric matched `f32`. **Existing stores are unaffected**: every
+  agent store has recorded `float16 = false`, and keeps it (a v2.5.0 fixture
+  guards this). A store that already had `float16 = true` rounds its
+  embeddings when next opened and writes them as `float16` at its next
+  checkpoint. Opt out with `float16 = false` or `create --f32`; the CLI's
+  `--float16` is still accepted and now a no-op. Values beyond ±65504 are
+  refused, so keep `f32` for unnormalised vectors.
+- **Breaking:** `MemoryError` gained `InvalidEntry`, returned when a
+  `float16` store is given an embedding value beyond ±65504. Exhaustive
+  matches need the new arm.
+- **The default build no longer compiles any C.** Deflate now defaults to the
+  pure-Rust zlib-rs instead of zlib-ng, so building the core crates needs
+  neither cmake nor a C compiler. Speed on HDF5 reads and writes is within 6%
+  of zlib-ng, and compressed output is byte-identical. To keep zlib-ng, enable
+  `fast-deflate` (on `clawhdf5`, `clawhdf5-format` or `clawhdf5-filters`); it
+  overrides zlib-rs wherever it is on.
+- **A truncated deflate chunk is now an error.** It used to read back short,
+  with no error.
+- **Minimum supported Rust is 1.92**, now declared in every crate's
+  `rust-version` and checked in CI.
+- **New stores use the int8 vector index by default.**
+  `MemoryConfig::quantized_index` now defaults to `true`: a quarter of the
+  index memory, builds 1.8x (x86-64) and 2.3x (Raspberry Pi 5) faster, and
+  searches 1.63x and 1.18x faster at equal recall, measured on every
+  configuration tested. **Existing stores are unaffected** — a store written
+  with v2.6.0 or later keeps its persisted setting, and one written before the
+  setting existed opens as `false` and keeps its f32 index. Set
+  `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
+  out. The CLI's `--quantized-index` is still accepted but is now a no-op.
+
+### Signing
+- `clawhdf5-agent`: **Ed25519-signed checkpoints** — the README's
+  "cryptographically verifiable memory", now true. With
+  `HDF5Memory::set_signing_key(key)`, every checkpoint stores a signed
+  manifest: a SHA-256 per record (text, embedding as stored, channel,
+  timestamp, session, tags, deleted flag, activation) in a Merkle tree, plus
+  hashes of the settings (and WAL mark), sessions and knowledge graph, with
+  the per-record hashes in `/integrity/record_hashes`.
+  `HDF5Memory::verify(path, &public_key)` recomputes everything from the file
+  and reports which part changed and which records (`changed_records`); a
+  forged manifest fails the signature. The key is never persisted; a signed
+  store refuses to checkpoint without it (`MemoryError::SigningKeyRequired`),
+  and `remove_signature()` is the deliberate way back to unsigned. Saves still
+  in the WAL are not covered (`wal_entries_unsigned`). Tests include every
+  kind of edit, and an edit made with h5py in place, which verify pinpoints.
+  Cost: ~20% of a checkpoint, 32 bytes per record (`BENCHMARKS.md`, "Signed
+  checkpoints"). New dependencies `ed25519-dalek`, `sha2`, `rand_core` — pure
+  Rust; the no-C check still passes.
+- `clawhdf5-cli`: `keygen --out <file>` (owner-only key file),
+  `--signing-key <file>` / `CLAWHDF5_SIGNING_KEY` on writing commands
+  (`create` signs immediately), `verify --public-key <hex|file>` (JSON report;
+  exit status 2 if not valid), and `signed` in `create`/`stats` output.
+
+### Migration
+- `clawhdf5-migrate`: writes through the agent's own API (`HDF5Memory::create`
+  / `open`, `save_batch`, the session cache and knowledge graph), so there is
+  no second copy of the schema. Sessions and entities/relations carry over;
+  deleted rows become deleted records (or are left out with
+  `--skip-deleted`). Every source row is checked before the output is created,
+  so a source that cannot be migrated leaves an existing store untouched.
+  Validation reads the result back with `HDF5Memory::open_read_only`, compares
+  every field (embeddings bit for bit — `round_to_f16` of the source for a
+  `float16` store) and checks that a migrated record is found by search. The
+  `half`-based conversion is gone; `clawhdf5_format::float16` is the only one.
+  42 tests, including h5py opening a migrated store; an adversarial review's
+  two blocker and four major findings are fixed with regression tests.
+- `clawhdf5-agent`: `HDF5Memory::sessions()` / `sessions_mut()`,
+  `HDF5Memory::delete_batch(&[usize])` (one save, all-or-nothing, never
+  auto-compacts), `SessionCache::add_at`, and `SessionCache` / `SessionEntry`
+  re-exported from the crate root.
+
+### Search
+- `clawhdf5-agent`: **`HDF5Memory::search` with `SearchOptions`** — source
+  filtering, re-ranking and confidence rejection in the store's own search
+  path. Re-ranking and confidence rejection used to be reachable only
+  through the OpenClaw backend, which now calls `search` with both on.
+  - `with_sources([..])` restricts a search to records from those source
+    channels. It applies before ranking, so a filtered search still returns up
+    to `k` results, normalised over what it can return. Measured at 100K: the
+    exact filtered top 10 for filters keeping 50%, 10% and 1% of the store and
+    for records far from the query, and never slower than an unfiltered search
+    (2.3 ms for a 1% filter vs 4.6 ms unfiltered). See `BENCHMARKS.md`,
+    "Search options".
+  - `with_rerank(ReRankConfig)` re-ranks a pool of `max(3k, 10)` candidates
+    (`rerank_pool` to change it) by relevance, recency, source authority and
+    activation; `with_confidence(ConfidenceConfig)` drops low-confidence
+    results; `at_time(now)` pins the clock for recency. About 3% on latency.
+  - `hybrid_search` and `hybrid_search_with` are unchanged (tested bit for
+    bit against `search` with default options).
+- `clawhdf5-agent`: the OpenClaw backend's search now boosts the Hebbian
+  activation of the `k` results it returns, not of the whole `3k` candidate
+  pool it re-ranks.
+
+### Documentation
+- OpenClaw claims withdrawn across the README, QUICKSTART, USE_CASES, ROADMAP
+  (Track 7 marked withdrawn) and the `openclaw` module docs; the dead
+  `github.com/redclawsystems/openclaw` link is gone. The Node package is
+  marked unpublished and broken (now `"private": true` so it cannot be
+  published by accident), with its bugs recorded in `docs/known-issues.md`.
+
+### Benchmarks
+- Every undated or pre-September section of `BENCHMARKS.md` re-run on one
+  machine on one day (tank, 2026-09-24, commit 5c8323c), with the command for
+  each and every number traced back to the raw output by a separate check.
+  Where a figure moved, the section says so. Two apparent regressions were
+  isolated rather than published: knowledge-graph traversal (a real bug,
+  fixed above) and the write path, which measures the same at v2.3.0 on this
+  machine — the old 18 µs / 6.17 ms figures came from an undated run on other
+  hardware; `float16` adds ~2 µs per save and the int8 index nothing.
+- New `multimodal_bench`: cross-modal search at 1K and 10K records, which the
+  README claimed but nothing measured.
+- `footprint_bench` reports whether it built `float16` or `f32` stores and
+  takes `--f32`; it had kept printing "f32" after the default changed.
+
+### Interop
+- `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
+  libhdf5.** The float datatype encoder hard-coded the sign bit's position to
+  63, correct only for `f64`; libhdf5 validates it and refused the dataset. It
+  is now derived from the type (15 / 31 / 63). Our reader ignores the field,
+  and the interop suites only wrote `f64`, which is how it went unnoticed.
+- `clawhdf5-format`: **every empty dataset was unreadable by h5py and
+  libhdf5.** It was written with a real address and zero bytes, which trips
+  libhdf5's `addr + size <= addr` overflow check. An empty contiguous dataset
+  now gets the undefined address, as libhdf5 writes it. This affected every
+  agent store without sessions or a knowledge graph.
+- New interop tests: `f32` and `float16` datasets in both directions (our
+  `float16` rounding matches numpy's bit for bit on 4 020 probe values,
+  including ties, subnormals and the overflow boundary), and an agent store —
+  `f32` and `float16` — opened by h5py with every dataset decoded.
+
+### Storage
+- `clawhdf5-format`: **half-precision datasets.**
+  `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy `float16`),
+  rounding to nearest-even; `make_f16_type`, and `clawhdf5_format::float16`
+  with the conversions, which are checked against the `half` crate on 16.7M
+  values and round-trip all 65 536 half values. Reading `float16` as `f32`
+  gained a little-endian fast path.
+- `clawhdf5-agent`: **`MemoryConfig::float16` stores embeddings as half
+  precision.** At 100K x 384 the file goes from 154.0 to 80.8 MiB (−48%), a
+  checkpoint from 752 to 512 ms and open from 300 to 252 ms, with the same
+  vector recall@10 against an exact scan (0.999 vs 0.994) and the same
+  `hybrid_search` latency; at 10K open is 3 ms slower. On the full
+  LongMemEval haystack with real MiniLM embeddings every retrieval metric is
+  identical to `f32` (`longmemeval_bench --float16`). The cache rounds each
+  embedding as it is saved, so memory and file agree bit for bit and a store
+  returns the same results before and after a reopen (tested). Out-of-range
+  values are refused with `MemoryError::InvalidEntry` rather than stored as
+  infinity; batches are all or nothing. CLI: `create --float16`. See
+  `BENCHMARKS.md`, "float16 embedding storage".
+
+### Build
+- **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
+  `clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
+  (zlib-ng) is opt-in. No crate in the default dependency tree of the core
+  crates compiles C, and `ci-test.sh` now fails if one appears. The facade's
+  `fast-deflate` was on by default and is now off. See `BENCHMARKS.md`,
+  "Deflate backend".
+- `zlib-rs` also enables flate2's `runtime_detection`. Without it zlib-rs has
+  no `std`, cannot detect SIMD at runtime, and inflates 3.5x slower; the
+  workspace builds flate2 with `default-features = false`, which had been
+  switching it off.
+- `rust-version = "1.92"` for the whole workspace (the floor: `wgpu` requires
+  it), and CI checks the workspace on exactly that toolchain.
+- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
+
+### Correctness
+- `clawhdf5-format`: **a truncated deflate chunk read back short, with no
+  error.** The deflate filter used flate2's streaming reader, which returns the
+  bytes it has when the input runs out before the end-of-stream marker. It now
+  decodes in one pass into a buffer sized to the chunk and reports a
+  truncated stream as `DecompressionError`. Same fix in `clawhdf5-filters`,
+  where output longer than the stated size was also silently cut off; it is
+  now an error.
+
+### Defaults
+- `clawhdf5-agent`: `MemoryConfig::float16` defaults to `true` for new stores,
+  measured rather than assumed: identical LongMemEval retrieval on real
+  embeddings, 48% smaller files and faster checkpoints and opens at 100K.
+  `clawhdf5-cli create --f32` opts out; like `--f32-index`, it only ever
+  switches the default off.
+- `clawhdf5-agent`: `MemoryConfig::quantized_index` defaults to `true` for new
+  stores. The reason it had been off — that int8 search was slower on ARM —
+  did not survive measurement (see Corrections). Stores that predate the
+  setting still load it as `false`, so reopening one never changes how its
+  index is held; a store written by the v2.5.0 CLI is now a test fixture that
+  guards exactly that, and the test fails if the load default is changed.
+- `clawhdf5-cli`: `create --f32-index` opts out. `create` used to assign
+  `--quantized-index` straight into the config, which under the new default
+  would have forced every CLI-created store back to f32 unless the caller
+  knew to ask; it now only ever switches the default off.
+
+### Performance
+- `clawhdf5-agent`: consolidation's novelty scoring (each `add_memory` against
+  the whole working tier) computes the new record's norm once, takes each
+  comparison in one vectorised pass instead of three, and splits a working
+  tier of 4 096+ records across threads — same results, tested against the
+  old formula. It had made `consolidation_efficiency` stall at 100K; the
+  complete run now takes 8 min and fills in the 100K cycle row (46.66 ms) and
+  the memory-reduction table.
+- `clawhdf5-bench`: `consolidation_efficiency` no longer prints a record-count
+  ratio as a "BM25 Speedup" (it was never measured), nor claims cycle time
+  grows sub-linearly (its own numbers grow slightly faster than linearly).
+- `clawhdf5-agent`: **knowledge-graph traversal was 6.5x slower than it
+  should be.** `bfs_neighbors` and `spreading_activation` built an adjacency
+  index over the whole graph on every call (1efd82c), so a 2-hop BFS over 1K
+  entities took 155 µs. The index is now cached on `KnowledgeCache` and
+  checked against a fingerprint of the graph on each use — one pass over
+  entity ids and relation endpoints, no allocation — so any change, including
+  direct edits of its public `Vec`s, still rebuilds it (tested). BFS over 1K
+  entities: 155.1 -> 23.1 µs; spreading activation over 100: 22.8 -> 10.1 µs.
+- `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
+  whole chunk in one call, into a buffer allocated once, instead of streaming
+  it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
+  1 MB inflate.
+- `clawhdf5-accel`: **`dot_i8` has aarch64 kernels** — `SDOT` for CPUs with
+  the ARMv8.2 dot-product extension (Cortex-A76 and later, Neoverse-N1, every
+  Apple Silicon generation) and plain NEON (`vmull_s8` + `vpadalq_s16`) for
+  the rest, selected at runtime. `SDOT` is issued through inline assembly,
+  because the `vdotq_s32` intrinsic is still behind the unstable
+  `stdarch_neon_dotprod` feature. On a Raspberry Pi 5 at N = 100 000 and
+  equal recall, the quantised index answers **1.18x the queries per second**
+  of f32 (7 267 vs 6 164) and builds **2.3x faster** (14 464 vs 33 413 ms).
+  Both kernels are tested bit-for-bit against scalar on real hardware, each
+  explicitly — dispatch only ever takes one path on a given CPU, so testing
+  through it alone would have left the plain-NEON fallback unexercised on any
+  machine with `SDOT`.
+
+### Corrections
+- The v2.7.0 entry for `dot_i8` said `quantized_index` stayed off by default
+  because "aarch64 falls back to the scalar loop", implying the ~13% search
+  penalty measured on x86 applied on ARM too. It did not. That figure came
+  from scalar int8 against hand-written AVX2 f32 kernels on x86, whose
+  portable baseline is SSE2; on aarch64 NEON is the baseline, and measured on
+  a Pi 5 the scalar int8 loop already matched f32 for search while building
+  1.76x faster. The claim was extrapolated rather than measured.
+
+## v2.7.0 (2026-09-20)
+
+### Upgrade Notes
+- **Two read-path bugs fixed, one of them silent.** Datasets indexed by an
+  Extensible Array (any dataset with one unlimited dimension) returned data
+  from the wrong chunks past their first few dozen. If you have readings taken
+  from such a dataset with an earlier release, they may be wrong; re-read them.
+- **A corrupt chunk index is now an error.** Fixed and Extensible Array
+  structures carry checksums that were previously ignored, so damage surfaced
+  as plausible data from the wrong offset. Code that read a damaged file and
+  got numbers will now get `ChecksumMismatch` instead. That is the point.
+- **Breaking:** `MemoryConfig` gained `hnsw_m`, `hnsw_ef_construction` and
+  `hnsw_ef_search`, so literal constructions need updating;
+  `..Default::default()` does not. All three default to the previous
+  behaviour.
+
+### Correctness
+- `clawhdf5-format`: **datasets indexed by an Extensible Array returned wrong
+  data beyond their first few dozen chunks.** One unlimited dimension gives a
+  dataset an Extensible Array chunk index, whose first elements (4 by default)
+  sit inline in the index block and whose rest live in data blocks sized by a
+  formula the reader got wrong. In the default layout everything through the
+  36th chunk happened to line up and the 37th onwards did not: a 400-chunk
+  dataset silently returned wrong values from chunk 37, and datasets past
+  about a thousand chunks failed outright with "invalid Extensible Array data
+  block signature". **Reads were wrong, not
+  merely refused** — the caller got plausible numbers from the wrong chunks.
+  Four separate layout errors, each checked against files written by HDF5 2.0
+  and against the library source:
+  - the number of data blocks in super block `u` is `2^(u/2)`, not `2^u`;
+  - each holds `2^((u+1)/2) * data_blk_min_elmts` elements, which doubles
+    every *other* level rather than every level;
+  - a super block carries a block-offset field before its data block
+    addresses, which was not skipped;
+  - the page-init bitmap belongs to the super block, one bit per page packed
+    across all its data blocks (MSB first), and was being read from inside the
+    data block instead; a paged data block also ends its prefix with a
+    checksum before the first page.
+  Covered now by interop tests at 4, 37, 400, 5 000 and 200 000 chunks (the
+  last large enough for paged data blocks), plus sparse, gzip-filtered and
+  2-D cases. Writing is unaffected; this is a read-path bug.
+- `clawhdf5-format`: the sibling Fixed Array index (fixed dimensions written
+  with `libver='latest'`) was checked against the same range and is correct,
+  including paged data blocks and sparse datasets — it really does keep its
+  page-init bitmap in the data block, where the Extensible Array does not.
+  It had no real-file coverage above the inline sizes either, so it now has
+  the same tests.
+
+### Security
+- `clawhdf5-format`: **a crafted file could crash any reader through B-tree v2
+  traversal.** Recursion was bounded only by the depth the file claimed (a
+  `u16`), and child addresses were never checked for sharing. A node listing
+  itself as its own child under a header claiming 65 535 levels — under 100
+  bytes — overflowed the stack and **aborted the process** (SIGABRT, not a
+  catchable error). Levels whose children all point at one shared node below
+  reached it fan-out^depth times: 29.5 million records from ~5 KB, and one
+  more level would exhaust memory. Both are now errors, returned in under a
+  millisecond: depth is capped at 64 (as the fractal heap already was), and
+  traversal stops once it has produced more records than the file has bytes
+  to hold. Every B-tree v2 user goes through this path — dense attributes,
+  v2 groups, shared messages and chunk indexes. Valid files are unaffected,
+  including a depth-2 HDF5 2.0 chunk index with 40 000 records, now covered by
+  an interop test.
+
+### Integrity
+- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
+  their checksums** (the `checksum` feature, on by default). Every structure
+  in both — header, index block, super block, data block and each data block
+  page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
+  The consequence of skipping it is not a missing warning but wrong data: a
+  single flipped bit in a chunk address still parses, still points inside the
+  file, and the reader hands back whatever bytes now sit there as the chunk's
+  contents. Verified in both directions — the checksums accept files written
+  by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
+  and an interop test corrupts an address to confirm the read now fails
+  instead of returning data (it does return data when the check is removed).
+
+### Performance
+- `clawhdf5-agent`: **opening a store is ~28% faster** (455 ms -> 327 ms at
+  100k x 384). `read_from_disk` memory-mapped the file and then copied the
+  entire mapping into a `Vec` for `File::from_bytes`, when `File::open`
+  memory-maps it directly — so every open paid a full-file memcpy for nothing.
+  Process peak memory is unchanged: the peak falls after the parse, during the
+  index build, so the transient never reached the high-water mark. The
+  footprint harness now reports that peak next to the retained figure, which
+  is how this was checked rather than assumed.
+- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
+  sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
+  elsewhere). The quantised HNSW index used a scalar loop while the `f32` path
+  it was measured against ran AVX2, so the ~13% throughput cost recorded for
+  `MemoryConfig::quantized_index` was a missing kernel rather than a property
+  of int8. With the kernel, at N = 100 000 x 384 and equal recall, the
+  quantised index answers **1.63x as many queries per second** (21 848 vs
+  13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778
+  vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three
+  alternating runs. It remains off by default only because the kernel is
+  AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so
+  the SIMD path is tested to agree with scalar bit for bit.
+
+### Tuning
+- `clawhdf5-agent`: **the HNSW parameters are configurable** —
+  `MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search`
+  (defaults 16, 64, and 0 meaning "scale with `k`", i.e. today's behaviour).
+  They were constants, so a deployment could not trade recall against memory
+  or query speed at all. All three are persisted with the store. Values are
+  clamped where the index requires it: `clawhdf5-ann` asserts a graph degree
+  of at least 2, so a configured 0 — from a file, or from a caller who took 0
+  to mean "default" — used to abort the process inside the builder. Lowering
+  `ef_search` also no longer narrows the candidate pool that fusion sees.
+  **Breaking:** `MemoryConfig` gained fields, so literal constructions need
+  updating; `..Default::default()` does not.
+
+### Documentation
+- `clawhdf5-agent`: `BM25Index::search` claimed to use Block-Max WAND for early
+  termination. It never did; it scores every match exhaustively. It now says
+  so, and why no pruning would help the store: `hybrid_search` uses `scores()`,
+  since fusion normalises over every match.
+
 ## v2.6.0 (2026-09-20)
 
 ### Upgrade Notes

CLAUDE.md

@@ -1,7 +1,7 @@
 # clawhdf5
 
 ## Purpose
-Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. Used by ZeroClaw as its persistent memory and knowledge graph backend.
+Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated I/O. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
 
 ## Architecture
 
@@ -19,7 +19,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
 | `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
 | `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
 | `clawhdf5-accel` | CPU SIMD acceleration path |
-| `clawhdf5-migrate` | Schema migration engine |
+| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
 | `clawhdf5-android` | Android JNI bindings |
 | `clawhdf5-cli` | Command-line interface |
 | `clawhdf5-napi` | Node.js native addon bindings |
@@ -27,7 +27,12 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
 | `clawhdf5-bench` | Benchmark suite |
 
 ## Key Features
-- Zero-dependency HDF5 read/write (no libhdf5 C library required)
+- Zero-C-dependency HDF5 read/write: no libhdf5, and deflate defaults to
+  pure-Rust zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake).
+  `ci-test.sh` fails if a C-building crate enters the core crates' default
+  tree. flate2 must keep `runtime_detection` with zlib-rs — without it zlib-rs
+  loses SIMD and inflates 3.5x slower. MSRV is 1.92 (`rust-version`, checked
+  in CI).
 - HNSW vector index for semantic similarity search over agent memories — the
   `clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses
   the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
@@ -39,13 +44,20 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
   (plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its
   graph is saved to `<store>.h5.ann` at each checkpoint and reloaded by `open()`
   (tied to the checkpoint by a generation id; stale/damaged sidecars are
-  ignored and the index rebuilt). `MemoryConfig::quantized_index` (off by
-  default, persisted) stores the index's own copy of the embeddings as `i8`,
+  ignored and the index rebuilt). `MemoryConfig::quantized_index` (**on by
+  default** for new stores, persisted; stores predating the setting load as
+  `false` and keep their f32 index — guarded by
+  `tests/fixtures/store_v2_5_0.h5`; CLI opt-out is `create --f32-index`)
+  stores the index's own copy of the embeddings as `i8`,
   which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors
   at 100K); because quantised distances are approximate and `ef` cannot
   compensate, the query path then re-scores the candidate pool against the
-  exact embeddings, which holds recall at the f32 index's level and costs
-  ~13% of QPS. `hybrid_search` keeps one incremental BM25
+  exact embeddings, which holds recall at the f32 index's level. It is also
+  faster at equal recall: 1.63x the QPS on x86-64 (AVX2) and 1.18x on a
+  Raspberry Pi 5 (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm since
+  the intrinsic is unstable; plain NEON on pre-dotprod cores). The aarch64
+  code is `cfg`'d out on x86, so x86 CI never compiles or lints it — test it
+  on real ARM (`rpivision02`, 10.0.2.3, is a Pi 5). `hybrid_search` keeps one incremental BM25
   index for the life of the store and never writes the store: Hebbian
   activation boosts are persisted by the next checkpoint (or on drop), not per
   query. Measure any search-path change with
@@ -75,8 +87,51 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
   `export` do). An unreadable WAL (torn header, bad magic) is quarantined to
   `<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
   unknown *newer* version still fails and is left untouched.
-- `MemoryConfig::compression` uses deflate by default; enable the agent's
-  `zstd` feature to compress embeddings with Zstd instead (links libzstd).
+- `MemoryConfig::float16` (**on by default** for new stores, persisted;
+  existing stores keep their recorded `false` — guarded by the v2.5.0
+  fixture in `tests/float16_store.rs`; CLI opt-out is `create --f32`) writes
+  `/memory/embeddings` as IEEE half precision (48% smaller file at 100K;
+  LongMemEval with real MiniLM embeddings identical to f32).
+  `MemoryCache::half_precision` rounds each embedding as it enters the cache (push, update, WAL replay, and on load of a store still
+  `f32` on disk), so memory and file agree bit for bit; the conversions live
+  in `clawhdf5_format::float16` and must stay the single implementation.
+  Values beyond ±65504 are `MemoryError::InvalidEntry`. Interop: every file
+  must open in h5py — `f32` datasets and empty datasets did not until
+  2026-09-23 (see `docs/known-issues.md`); the agent's `h5py_interop` test
+  guards a whole store.
+- `HDF5Memory::search(query_emb, text, &SearchOptions)` is the full search
+  path: optional source-channel filter (applied before ranking; exact scan of
+  the allowed records whenever cheaper than `pool × M` index distance
+  evaluations, and as the fallback when the pool comes back short), fusion,
+  activation scaling, optional re-ranking and confidence rejection.
+  `hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend`
+  (the `openclaw` module) is `search` with re-rank + confidence on.
+- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an
+  OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"`
+  config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md`
+  records what a real plugin would need.
+- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream
+  v0.8.5 and the `osobh/zeroclaw` fork, and their full history): no
+  `clawhdf5` feature or backend exists; ZeroClaw's memory backends are
+  sqlite/lucid/postgres/qdrant/markdown/none behind its own `Memory` trait.
+  `clawhdf5-migrate`'s default SQLite layout (`memory_chunks`, `sessions`,
+  `entities`, `relations`) is not ZeroClaw's schema either (ZeroClaw's is a
+  `memories` table). Don't reintroduce integration claims without an
+  integration and a test against the real consumer. Measure changes with
+  `search_harness --options-study`.
+- `MemoryConfig::compression` is off by default; when on, embeddings are
+  deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
+- Signed checkpoints (`clawhdf5-agent` `signing` module): with
+  `HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed
+  manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph
+  hashes; per-record hashes in `/integrity/record_hashes`);
+  `HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly
+  what the file persists in the form the loader returns it (strings lose
+  trailing NULs; an empty WAL mark is not written) or untouched stores stop
+  verifying — `tests/signed_store.rs` round-trips awkward strings. The key is
+  never persisted; a signed store refuses to checkpoint without it
+  (`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`).
+  WAL entries after the checkpoint are not covered.
 - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
   default) recomputes a dataset's SHA-256 and compares it against the
   `_provenance_sha256` attribute written automatically on save when
@@ -109,6 +164,24 @@ cargo build --release
 cargo test --workspace
 ```
 
+### CI
+`.gitea/workflows/ci.yml` has two jobs, both green as of 2026-09-22:
+- **`test`** (`ubuntu-latest`, in `rust:latest`) runs `scripts/ci-test.sh` with
+  the h5py/netCDF4 interop suites required (`CLAWHDF5_REQUIRE_INTEROP=1`).
+  Served by the `tank` and `architect` runners.
+- **`test-arm64`** (`linux_arm64`) lints and tests the aarch64 code — the NEON
+  kernels are `cfg`'d out on x86, so this is the only place they are built.
+  Served by `vision-01` (host mode) and `vision-02` (Docker), so steps must
+  work in both.
+
+Keep workflows free of JavaScript actions (`actions/checkout`, `actions/cache`,
+…): `rust:latest` has no `node`, and not every runner reaches GitHub, where
+they are fetched from. Check out with plain `git` instead. The `test` job
+installs `cmake` for the opt-in `fast-deflate` (zlib-ng) steps; the default
+build needs no C toolchain, so `test-arm64` does not.
+All runners are on `gitea-runner` 3.5.0, from `docker.gitea.com/act_runner`
+— `gitea/act_runner:latest` on Docker Hub is frozen at 0.6.1.
+
 ### CLI
 ```bash
 cargo run -p clawhdf5-cli -- --help
@@ -123,4 +196,12 @@ python -c "import clawhdf5; print(clawhdf5.__version__)"
 ```
 
 ## Integration
-ZeroClaw imports this as a Cargo feature (`clawhdf5` feature flag) to persist agent memory with HNSW vector search for context retrieval.
+- **ClawBrainHub** (`clawverse/clawbrainhub` on git.redclaw.dev) is the one
+  verified consumer: `cbh-core` reads and writes `.brain` files through the
+  facade (`File`, `FileBuilder`, `AttrValue`, `Selection`), `cbh-scanner`
+  uses the facade, and `cbh-cli` uses `clawhdf5_agent::bm25::BM25Index`. It
+  depends on this repo by path (`../clawhdf5`), so it builds against whatever
+  is checked out — changes to those APIs reach it directly. Verified
+  2026-09-25 against main: builds, and its 204 tests pass.
+- OpenClaw and ZeroClaw were both described as consumers; neither integrates
+  clawhdf5 (see Key Features and `docs/openclaw.md`).

Cargo.toml

@@ -21,8 +21,11 @@ members = [
 resolver = "2"
 
 [workspace.package]
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+# Oldest toolchain that builds the whole workspace; CI checks it. wgpu (in
+# clawhdf5-gpu) requires 1.92.
+rust-version = "1.92"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
 

README.md

@@ -3,24 +3,102 @@
 **The memory layer AI agents deserve. One file. Pure Rust. Zero C dependencies.**
 
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
-[![Rust](https://img.shields.io/badge/rust-1.75%2B-orange.svg)](https://www.rust-lang.org)
-[![Tests](https://img.shields.io/badge/tests-1650%2B%20passing-brightgreen.svg)](#performance)
-[![LongMemEval](https://img.shields.io/badge/LongMemEval%20oracle-Turn--Level%20Hit@5%2084%25%20BM25--only-blue.svg)](BENCHMARKS.md#longmemeval-results)
-[![Footprint](https://img.shields.io/badge/footprint-6.5%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint)
+[![Rust](https://img.shields.io/badge/rust-1.92%2B-orange.svg)](https://www.rust-lang.org)
+[![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building)
+[![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
+[![Footprint](https://img.shields.io/badge/on--disk-~820%20B%2Frecord%20float16%2C%20synthetic%20text-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
 
-ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory — all stored in a single portable file.
+ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory (Ed25519-signed checkpoints) — all stored in a single portable file.
 
 > **Two things live here:**
 > - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
 > - **An agent memory layer built on top of it** — vector search, knowledge graph, hippocampal-style consolidation, in `clawhdf5-agent`.
 
-```
-cargo add clawhdf5                          # core HDF5 read/write, no agent layer
-cargo add clawhdf5-agent --features agent   # + agent memory layer
+The crates are not on crates.io yet, so depend on them from git:
+
+```toml
+[dependencies]
+clawhdf5       = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # core HDF5 read/write
+clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # + agent memory layer
 ```
 
+> **C dependencies, precisely:** the core crates (`clawhdf5`, `clawhdf5-agent`,
+> `-format`, `-io`, `-filters`, `-ann`, `-accel`, `-netcdf4`, `-cli`) build no C
+> code by default — no libhdf5, and deflate is the pure-Rust
+> [zlib-rs](https://github.com/trifectatechfoundation/zlib-rs), which matches
+> zlib-ng on HDF5 reads and writes and produces byte-identical output
+> ([BENCHMARKS.md § Deflate backend](BENCHMARKS.md#deflate-backend-zlib-rs-vs-zlib-ng)).
+> CI fails if a C-building crate enters their default dependency tree. C comes
+> in only when you ask for it: `fast-deflate` (zlib-ng, needs cmake), `zstd`,
+> `szip`, the BLAS backends, `clawhdf5-migrate` (bundled SQLite) and the
+> Node.js bindings.
+
 > **New here?** Start with the **[Quickstart Guide](docs/QUICKSTART.md)** · See **[Use Cases](docs/USE_CASES.md)** · Read **[Benchmarks](BENCHMARKS.md)**
 
+## What's new (v2.2 → v2.7, and unreleased)
+
+Five releases in September 2026. Details, including upgrade notes and every
+breaking change, are in [CHANGELOG.md](CHANGELOG.md).
+
+**HDF5 correctness (read these if you read files with an earlier release)**
+- **Extensible Array chunk indexes returned wrong data** past the 36th chunk —
+  any dataset with one unlimited dimension. Silent: plausible numbers from the
+  wrong chunks. Fixed in v2.7.0; re-read affected data.
+- Fixed and Extensible Array checksums are now verified, so a corrupt chunk
+  index is `ChecksumMismatch` instead of wrong data (v2.7.0).
+- Compound datatypes written with default libver bounds (plain
+  `h5py.File(path, 'w')`) were mis-parsed; HDF5 2.0 compound v5 and native
+  complex (class 11) types now parse (v2.2.0–v2.3.0).
+- Committed datatypes, fill values, soft links and `H5T_STD_REF` references now
+  read correctly; external links and external raw data are explicit errors;
+  `attrs()` no longer silently drops attributes (v2.3.0–v2.5.0).
+- Datasets indexed by a version-2 B-tree now read (v2.5.0).
+
+**Security and robustness**
+- A crafted file could abort any reader via B-tree v2 recursion or explode it
+  via shared children; both are now fast errors (v2.7.0).
+- Virtual-dataset source paths are confined to the file's directory; chunked
+  reads use overflow-checked sizes and fallible allocation, and the facade
+  writes files atomically (v2.3.0).
+- Agent store: single-writer lock plus `open_read_only`; a crash between
+  checkpoint and WAL truncate no longer duplicates entries; unreadable WALs are
+  quarantined instead of blocking `open()` (v2.3.0).
+
+**Search quality and speed**
+- HNSW neighbour selection now uses the paper's diversity heuristic: recall@10
+  at 100K went from 0.31 to 0.98 (v2.4.0).
+- `hybrid_search` is 79–190× faster than v2.3.0 (p50 0.07 ms at 1K, 4.65 ms at
+  100K). It no longer rebuilds BM25 or rewrites the store per query, and the
+  HNSW graph is persisted (v2.4.0).
+- Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
+  been discarding the retrieval score, costing the Markdown backend 40.6pp of
+  Hit@1; fixed in v2.6.0.
+- Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
+  0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
+
+**Memory**
+- A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
+  int8 HNSW index, **on by default for new stores** (unreleased), brings a
+  100K × 384 store to 1.74× the raw vectors. At equal recall it is also faster
+  than `f32`: 1.63× QPS on AVX2, 1.18× on a Raspberry Pi 5 (NEON `SDOT`).
+
+**Interop and search (unreleased)**
+- **Files we write now open in h5py and libhdf5.** Every `f32` dataset —
+  including every agent store's embeddings — and every empty dataset was
+  refused by libhdf5. Both were write-side bugs in every release; agent stores
+  fix themselves at their next checkpoint. See
+  [docs/known-issues.md](docs/known-issues.md).
+- `MemoryConfig::float16` now stores half-precision embeddings (it was
+  ignored), and is on by default for new stores: 48% smaller files, and
+  identical LongMemEval retrieval on real embeddings.
+- `HDF5Memory::search` with `SearchOptions`: filter by source channel (exact
+  filtered top-k, never slower than unfiltered), and opt-in re-ranking and
+  confidence rejection, which used to be reachable only through `ClawhdfBackend`.
+
+**Tooling**
+- CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
+  silently) and runs an aarch64 job for the NEON kernels.
+
 ---
 
 ## Why ClawhDF5?
@@ -33,16 +111,16 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
 | Keyword search | Separate FTS engine | Integrated BM25 |
 | Knowledge graph | Neo4j or none | In-file graph with spreading activation |
 | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
-| Temporal queries | Custom code | Native temporal index (716ns) |
-| Multi-modal | Multiple stores | Unified cross-modal search |
-| Security | Hope for the best | Provenance tracking + anomaly detection |
+| Temporal queries | Custom code | Native temporal index (622 ns range query over 10K) |
+| Multi-modal | Multiple stores | Unified cross-modal search (exact scan: 842 µs over 1K records) |
+| Integrity | Hope for the best | Ed25519-signed checkpoints that pinpoint any edited record, chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts |
 | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
 
 ---
 
 ## Performance
 
-Vector search and agent-memory operations below are benchmarked on Intel i7-12650H (10C/16T), 384-dim embeddings, Criterion.rs. The HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note).
+The brute-force/IVF vector search, agent-memory, on-disk footprint and consolidation figures below were measured 2026-09-24 on tank (AMD Ryzen 7 7800X3D, 8C/16T), commit 5c8323c, 384-dim embeddings; the commands are in [BENCHMARKS.md](BENCHMARKS.md). Exceptions are marked where they appear: the HDF5 Core I/O table immediately below is from a separate, independently reproduced run (see its own hardware note), and the HNSW `f32`/`i8` table and the in-memory `i8` column were not re-measured on 2026-09-24.
 
 ### HDF5 Core I/O (vs libhdf5 1.14.6)
 
@@ -58,31 +136,63 @@ Figures below are from an independent reproduction run on a second machine (AMD
 | Sequential read (100K f32) | 23.3 µs | 63.6 µs | **2.7×** |
 | Sequential write (100K f32) | 210 µs | 189 µs | **≈ tie** |
 
+The chunked-write row was re-measured on the same machine on 2026-09-23, after
+the default deflate backend became pure-Rust zlib-rs: 1.46 ms against
+libhdf5's 51.4 ms (**35×**), and 1.48 ms with zlib-ng. libhdf5's own time on
+that machine moved from 65.0 to 51.4 ms between the two dates, which is most
+of the difference from 45×; compare same-day numbers only.
+
 ### Vector Search
 
-| Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
-|-------|------|-----------------|--------|----------|
-| 1K | **54 µs** | — | — | — |
-| 10K | 753 µs | **27 µs** | — | — |
-| 100K | 11.4 ms | 1.32 ms | **1.19 ms** | ~8–76× (see caveat) |
+**HNSW (the default backend for `hybrid_search`)** — `search_harness`, clustered
+384-dim data, M = 16, ef_construction = 64, recall measured against an exact scan.
+See [BENCHMARKS.md § Search harness](BENCHMARKS.md#search-harness-baseline-v230)
+and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index):
 
-> Reproduced on the same second machine (Ryzen 7 7800X3D) with a corrected,
-> apples-to-apples SIMD/scalar/parallel comparison methodology — see
-> [BENCHMARKS.md § Independent Validation: tank — LongMemEval & Vector
-> Search](BENCHMARKS.md#independent-validation-tank--longmemeval--vector-search-ryzen-7-7800x3d-2026-08-05).
+| N = 100K, ef = 64 | recall@10 | QPS | build |
+|---|---:|---:|---:|
+| `f32` index | 0.9945 | 13 399 | 3.2 s |
+| `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
+
+Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31. These
+two rows are a paired comparison (medians of alternating runs, same binary).
+A single `f32` run on 2026-09-24 measured recall 0.9945, 19 001 QPS and a
+2.7 s build; the int8 row was not re-run, so the pair has not been re-checked
+([§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index)).
+
+**Brute-force and IVF paths** (Criterion, tank, 2026-09-24):
+
+| Scale | Flat | IVF (nprobe=10) | IVF-PQ | MemX¹ (claimed, end-to-end) |
+|-------|------|-----------------|--------|----------|
+| 1K | **47.4 µs** | — | — | — |
+| 10K | 500.5 µs | **24.8 µs** | — | — |
+| 100K | 6.58 ms | 592 µs | **869 µs** | <90 ms |
+
+> These replace figures from the original i7-12650H run (flat 54 µs / 753 µs /
+> 11.4 ms); a 2026-08-05 run on tank had already matched the new ones — see
+> [BENCHMARKS.md § Vector Search Latency](BENCHMARKS.md#vector-search-latency).
 
 ### Agent Memory Operations
 
 | Operation | Latency | Scale |
 |-----------|---------|-------|
-| Hybrid search (RRF) | **222 µs** | 1K records |
-| BM25 keyword search | **67 µs** | 1K records |
-| Knowledge graph BFS | **24 µs** | 1K entities |
-| Spreading activation | **17 µs** | 100 entities |
-| Temporal range query | **716 ns** | 10K timestamps |
-| Consolidation cycle | **164 µs** | 1K records |
-| Memory write (WAL) | **18 µs** | per record (group-commit append; HDF5 batched at flush) |
-| Importance gate | **61 ns** | per record |
+| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **0.07 ms** / 0.49 ms / 4.69 ms | 1K / 10K / 100K records |
+| BM25 keyword search | **20.4 µs** | 1K records |
+| Knowledge graph BFS | **23.1 µs** | 1K entities |
+| Spreading activation | **10.1 µs** | 100 entities |
+| Temporal range query | **622 ns** | 10K timestamps |
+| Consolidation cycle | **115.2 µs** | 1K records |
+| Cross-modal search (exact scan, 2 embeddings per record) | **842.0 µs** / 8.44 ms | 1K / 10K records |
+| Memory write (WAL) | **26.1 µs** | per record (group-commit append; HDF5 batched at flush) |
+| Importance gate | **57.6 ns** | per record (trivial skip) |
+
+The old 18 µs WAL write was undated, from another machine: v2.3.0 measures
+24.3 µs on the same hardware as this table, the same as an `f32` store today.
+`float16` stores (the new default) add ~2 µs for rounding; the int8 index adds
+nothing. See [BENCHMARKS.md § Write Path](BENCHMARKS.md#write-path).
+Knowledge-graph traversal was briefly 6.5x slower (155 µs) until this re-run
+found and fixed an adjacency index rebuilt on every traversal; see
+[§ Knowledge Graph](BENCHMARKS.md#knowledge-graph).
 
 ### Chunked Write Throughput (codec comparison)
 
@@ -97,7 +207,7 @@ by default (AoS→SoA byte transpose, +157–204% throughput for float data):
 
 Use `.with_zstd(3)` or `.with_deflate(6)` for write-heavy workloads — both now perform at ~720–750 MiB/s on large matrices. Use `.with_pcodec()` for write-once/read-many workloads where compression ratio matters more than encode speed. Disable auto-shuffle with `.without_shuffle()` for byte arrays that don't benefit from AoS→SoA transposition.
 
-> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search). The ratio overstates the real advantage by an unquantified margin — order-of-magnitude indication only. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
+> ¹ MemX ([arxiv:2603.16171](https://arxiv.org/abs/2603.16171), March 2026): Rust + libSQL, claims <90ms at 100K records. **Not like-for-like:** MemX's figure is *end-to-end* (embeddings + FTS5 + four-factor re-ranking); ours is a *single component* (raw vector search), so the two columns are not comparable and no ratio is given. See [BENCHMARKS.md](BENCHMARKS.md#comparison-to-memx-arxiv260316171).
 
 ### LongMemEval Retrieval Recall
 
@@ -115,13 +225,17 @@ declaration:
 
 Hybrid is the strongest configuration, which is what running two retrieval stages
 is for. The weights matter more than the stages: a sweep of `vector_weight` from
-0.0 to 1.0 found the long-standing `0.7/0.3` default is **strictly dominated** by
-`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Use
-`0.4/0.6`, or `0.3/0.7` if rank-1 precision matters most. See
-[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#longmemeval-results).
+0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by
+`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since
+v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by
+`unified_search`, `hybrid_search_with` and `ClawhdfBackend`); callers that
+pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if
+rank-1 precision matters most. Reciprocal rank fusion is selectable
+(`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See
+[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#weight-sweep--full-haystack-n500).
 
-Vector embeddings require `--features embeddings`; without it the vector stage is
-inert and only the BM25 row is produced, which is what every previously published
+The benchmark's vector stage requires `clawhdf5-bench`'s `embeddings` feature
+(real MiniLM embeddings); without it the vector stage is inert and only the BM25 row is produced, which is what every previously published
 number here measured.
 
 On the easier `longmemeval_oracle` variant (evidence sessions only) the same
@@ -146,19 +260,53 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 
 ### Memory Footprint
 
-| Records | File Size | Bytes/Record | With Compression |
-|---------|-----------|--------------|------------------|
-| 1K | ~6.5 MB | ~6.5 KB | ~2.1 MB (3.1x) |
-| 10K | ~65 MB | ~6.5 KB | ~21 MB (3.1x) |
-| 100K | ~645 MB | ~6.5 KB | ~208 MB (3.1x) |
+**On disk** — 384-dim `float16` embeddings (the default for new stores),
+200-char text, `footprint_bench`
+([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):
+
+| Records | File Size | Bytes/Record | Gzip-6 compressed |
+|---------|-----------|--------------|-------------------|
+| 1K | 810.4 KB | 829 B | 56.4 KB |
+| 10K | 7.8 MB | 820 B | 471.3 KB |
+| 100K | 76.7 MB | 803 B | 4.5 MB |
+
+The benchmark's synthetic embeddings and text are far more repetitive than
+real data (only 40 distinct texts), so no column here is an expectation for
+real data. The compressed column is an upper bound, and the Bytes/Record
+column is optimistic too: it is not an uncompressed figure, because the store
+always deflates its text (any string dataset of 4 KiB or more) whatever
+`MemoryConfig::compression` says. The `float16` embeddings alone are 768 B per
+record, so 200 characters of real text would take a record above 820 B.
+This table used to show `f32` stores (1.7 KB per record, 169.8 MB at 100K);
+those were not re-measured. The float16 study compares the two on the same
+data: 100K × 384 records take 80.8 MiB as `float16` and 154.0 MiB as `f32`.
+
+**In memory** — a store reopened from disk, 384-dim `f32`, measured with a
+counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
+
+| Records | Raw vectors | Reopened, `f32` index | Reopened, `i8` index (default) |
+|---------|-------------|-----------------------|--------------------------------|
+| 1K | 1 MiB | 4 MiB (2.40x) | 2 MiB (1.64x) |
+| 10K | 15 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
+| 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |
+
+Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
+embedding twice. The `f32` column was re-measured on 2026-09-24 and reproduced
+exactly; the `i8` column was not re-run.
 
 ### Consolidation Efficiency
 
+1,000 records (10 signal + 990 noise), `working_capacity = 100`
+([BENCHMARKS.md § Consolidation Efficiency](BENCHMARKS.md#consolidation-efficiency)):
+
 | Metric | Before | After | Delta |
 |--------|--------|-------|-------|
-| Records in store | 1,000 | ~110 | −89% |
-| Hit@1 recall | ~60% | ~90% | +30% |
-| Search latency | ~2.8 ms | ~0.3 ms | **9x faster** |
+| Records in store | 1,000 | 100 | −90% |
+| Hit@1 recall (signal records) | 100% | 100% | no loss |
+| Search latency (avg) | 2.22 ms | 0.24 ms | **9.3x faster** |
+
+The consolidation cycle that does this took 0.13 ms; a cycle over 10K records
+takes 2.81 ms and over 100K 46.7 ms.
 
 **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**
 
@@ -166,74 +314,74 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 
 ## Agent Memory Architecture
 
-ClawhDF5's agent memory engine implements research from 15+ recent papers on agentic memory systems. It's not a toy — it's the real thing.
+ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory systems (see [Research Foundation](#research-foundation)).
 
 ```
                          ┌─────────────────┐
                          │   Agent Query   │
                          └────────┬────────┘
                                   │
-                    ┌────────────▼────────────┐
-                    │   Hybrid Retrieval      │
-                    │  Vector + BM25 + RRF    │
-                    └────────────┬────────────┘
-                                 │
-              ┌──────────────────▼──────────────────┐
-              │         Multi-Factor Re-Ranking      │
-              │  temporal · authority · activation    │
-              └──────────────────┬──────────────────┘
-                                 │
-                    ┌────────────▼────────────┐
-                    │  Confidence Rejection   │
+                ┌─────────────────▼──────────────────┐
+                │ HDF5Memory::search                 │
+                │ optional source-channel filter     │
+                │ HNSW vector + BM25 keyword         │
+                │ weighted fusion (0.4 / 0.6)        │
+                │ × √(Hebbian activation)            │
+                └─────────────────┬──────────────────┘
+                                  │  opt-in (SearchOptions);
+                                  │  ClawhdfBackend turns both on
+                ┌─────────────────▼──────────────────┐
+                │ Multi-factor re-ranking            │
+                │ relevance · recency · authority ·  │
+                │ activation                         │
+                ├────────────────────────────────────┤
+                │ Confidence rejection               │
                 │ (suppress bad matches)             │
-                    └────────────┬────────────┘
+                └─────────────────┬──────────────────┘
                                   │
-        ┌────────────────────────▼────────────────────────┐
-        │              Memory Store (HDF5)                │
-        │                                                 │
-        │  ┌───────────┐ ┌───────────┐ ┌───────────────┐  │
-        │  │ Working   │→│ Episodic  │→│  Semantic     │  │
-        │  │ (bounded) │ │ (bounded) │ │ (long-term)   │  │
-        │  └───────────┘ └───────────┘ └───────────────┘  │
-        │                                                 │
-        │  ┌──────────┐ ┌──────────┐ ┌────────────────┐   │
-        │  │Knowledge │ │Temporal  │ │  Multi-Modal   │   │
-        │  │  Graph   │ │  Index   │ │  Embeddings    │   │
-        │  └──────────┘ └──────────┘ └────────────────┘   │
-        │                                                 │
-        │  ┌──────────┐ ┌──────────┐ ┌────────────────┐   │
-        │  │Provenance│ │ Anomaly  │ │   Source       │   │
-        │  │ Tracking │ │Detection │ │  Isolation     │   │
-        │  └──────────┘ └──────────┘ └────────────────┘   │
-        └─────────────────────────────────────────────────┘
-                              │
-                     ┌────────┴────────┐
-                     │ agent_memory.h5 │
-                     │   single file   │
-                     └─────────────────┘
+     ┌────────────────────────────▼────────────────────────────┐
+     │ In memory                                               │
+     │  cache (flat f32 embeddings) · BM25 index · HNSW index  │
+     │  provenance ledger + anomaly alerts (session-scoped)    │
+     └────────────────────────────┬────────────────────────────┘
+                                  │ WAL append; checkpoint
+     ┌────────────────────────────▼────────────────────────────┐
+     │ agent_memory.h5   /meta · /memory · /sessions ·         │
+     │                   /knowledge_graph                      │
+     │ agent_memory.h5.wal   chained-CRC write-ahead log       │
+     │ agent_memory.h5.ann   HNSW graph (derived, rebuildable) │
+     │ agent_memory.h5.lock  single-writer lock                │
+     └─────────────────────────────────────────────────────────┘
 ```
 
+Consolidation tiers (Working → Episodic → Semantic), the knowledge-graph
+algorithms, temporal and multi-modal indexes are library components you drive
+directly; the store persists the records, sessions and graph they work over.
+
 ### Module Overview
 
 | Module | What It Does |
 |--------|-------------|
-| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy entity resolution |
-| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring and time-decay |
-| **`hybrid`** | Vector + BM25 fusion with Reciprocal Rank Fusion (RRF, k=60). The vector stage uses the HNSW index by default (`hnsw` feature, on by default); disable with `--no-default-features --features float16` for an exact linear scan |
-| **`reranker`** | Multi-factor re-ranking: temporal recency, source authority, activation weight |
-| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches |
+| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
+| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
+| **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
+| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Opt-in via `SearchOptions::with_rerank`; on in `ClawhdfBackend` |
+| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches. Opt-in via `SearchOptions::with_confidence`; on in `ClawhdfBackend` |
 | **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
 | **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
-| **`provenance`** | Source attribution, FNV-1a content hashing, integrity verification |
-| **`anomaly`** | Write rate limiting, 15 injection pattern detectors, source distribution analysis |
-| **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
+| **`signing`** | Ed25519-signed checkpoints: SHA-256 per record in a Merkle tree, plus hashes of settings, sessions and the knowledge graph; `HDF5Memory::verify` names any edited record |
+| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
+| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
+| **`openclaw`** | `ClawhdfBackend`: a Markdown-oriented backend (ingest by section, search, read back by path, export). Named for OpenClaw, but **not an OpenClaw plugin** — see [docs/openclaw.md](docs/openclaw.md) |
 | **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
-| **`ivf` / `pq`** | IVF-PQ approximate nearest neighbor for billion-scale search |
-| **`bm25`** | BM25 keyword index with TF-IDF scoring |
+| **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW |
+| **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming |
+| **`query_expand`** | Synonym / acronym / temporal query expansion |
 | **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph |
-| **`wal`** | Write-ahead log for crash-safe persistence; each entry is CRC32-checked on replay, so a corrupted entry stops replay there instead of loading bad data |
+| **`wal`** | Write-ahead log (v4) with a chained CRC32 per entry, so a corrupted, reordered, duplicated or spliced entry stops replay; checkpoints record a WAL mark so nothing is applied twice. Appends are not fsynced |
 | **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection |
 | **`decision_gate`** | Sub-microsecond trivial/substantive classification |
+| **`ephemeral`** | In-memory TTL/LFU working tier |
 | **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) |
 
 ---
@@ -265,7 +413,7 @@ assert_eq!(values, vec![22.5, 23.1, 21.8]);
 use clawhdf5_agent::{HDF5Memory, MemoryConfig, MemoryEntry, AgentMemory};
 
 // Create memory store
-let config = MemoryConfig::new("agent.h5", "my-agent", 384);
+let config = MemoryConfig::new("agent.h5".into(), "my-agent", 384);
 let mut memory = HDF5Memory::create(config)?;
 
 // Save a memory
@@ -278,13 +426,67 @@ memory.save(MemoryEntry {
     tags: "preference".into(),
 })?;
 
-// Search
-let results = memory.search(&query_embedding, 5)?;
+// Hybrid search: vector + BM25, weighted 0.4 / 0.6 (the measured default)
+let results = memory.hybrid_search(&query_embedding, "user preferences", 0.4, 0.6, 5);
 for result in results {
     println!("[{:.3}] {}", result.score, result.chunk);
 }
 ```
 
+### Search Options
+
+```rust
+use clawhdf5_agent::SearchOptions;
+use clawhdf5_agent::confidence::ConfidenceConfig;
+use clawhdf5_agent::reranker::ReRankConfig;
+
+// Only memories from these source channels; still a full page of k results.
+let work = memory.search(
+    &query_embedding,
+    "deadline",
+    &SearchOptions::new(5).with_sources(["slack", "email"]),
+);
+
+// Re-rank by relevance, recency, source authority and activation, then drop
+// low-confidence results — the pipeline ClawhdfBackend runs.
+let careful = memory.search(
+    &query_embedding,
+    "user preferences",
+    &SearchOptions::new(5)
+        .with_rerank(ReRankConfig::default())
+        .with_confidence(ConfidenceConfig::default()),
+);
+```
+
+### Signed Checkpoints
+
+```rust
+use clawhdf5_agent::signing;
+
+// Once, somewhere safe: keep the secret key, publish the public key.
+let key = signing::generate_key();
+let public = key.verifying_key();
+
+// Every checkpoint is signed from now on. The key is never written to disk;
+// a signed store refuses to checkpoint without it.
+memory.set_signing_key(key);
+memory.flush_wal()?;
+
+// Anyone holding the public key can check the file, e.g. after copying it.
+let report = HDF5Memory::verify(std::path::Path::new("agent.h5"), &public)?;
+assert!(report.is_valid());
+// On a tampered file: report.changed_records lists the records that differ.
+```
+
+The signature covers every record (text, embedding as stored, channel,
+timestamp, session, tags, deleted flag, activation), the store's settings,
+its sessions and its knowledge graph — a change made with any tool is caught.
+It covers checkpoints, not saves still in the WAL
+(`report.wal_entries_unsigned` counts those). CLI: `clawhdf5-cli keygen`,
+`--signing-key <file>` on writing commands, and `verify --public-key`.
+Signing adds about 20% to a checkpoint and 32 bytes per record to the file
+([BENCHMARKS.md § Signed checkpoints](BENCHMARKS.md#signed-checkpoints)).
+
 ### Knowledge Graph
 
 ```rust
@@ -309,8 +511,8 @@ let neighbors = kg.bfs_neighbors(alice, 2);  // 2-hop neighborhood
 let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5);
 
 // Entity resolution — fuzzy matching
-let resolved = kg.resolve_or_create("alice", "person", -1, 2);
-// Returns existing Alice entity (Levenshtein distance ≤ 2)
+let (id, created) = kg.resolve_or_create("alice", "person", -1, 2);
+// id == alice, created == false: matched the existing entity (Levenshtein distance ≤ 2)
 ```
 
 ### Memory Consolidation
@@ -321,15 +523,19 @@ use clawhdf5_agent::consolidation::*;
 let config = ConsolidationConfig::default();
 let mut engine = ConsolidationEngine::new(config);
 
-// Add memories — automatically scored for importance
-engine.add_memory("User prefers dark mode", vec![0.1, 0.2, ...], MemorySource::User);
-engine.add_memory("ok", vec![0.0, 0.0, ...], MemorySource::System);
+let now = 1_700_000_000.0; // seconds since the epoch
+
+// Add memories — automatically scored for importance.
+// Elevated sources (System, …) go through a separate, explicit API.
+let id = engine.add_memory("User prefers dark mode".into(), vec![0.1, 0.2, ...], UntrustedSource::User, now);
+engine.add_trusted_memory("ok".into(), vec![0.0, 0.0, ...], TrustedSource::System, now);
 
 // Access a memory (reactivates it)
-engine.access_memory(0);
+engine.access_memory(id, now);
 
 // Run consolidation cycle
-let stats = engine.consolidate();
+engine.consolidate(now);
+let stats = engine.get_stats();
 // Working memories promote to Episodic (if important enough)
 // Episodic memories promote to Semantic (if accessed enough)
 // Low-decay memories get evicted when tiers are full
@@ -351,19 +557,25 @@ let ids = index.range_query(1700000000.0, 1700010800.0);
 let recent = index.latest(10);
 ```
 
-### OpenClaw Integration
+### Markdown Backend
+
+`ClawhdfBackend` ingests Markdown by section and searches it with the full
+pipeline. It is a library API — clawhdf5 is **not** an OpenClaw memory plugin
+([docs/openclaw.md](docs/openclaw.md)). Sections stored this way carry no
+embedding, so their search is keyword-only unless you save records with
+vectors through `save_entry`.
 
 ```rust
 use clawhdf5_agent::openclaw::*;
 
 // Create backend
-let mut backend = ClawhdfBackend::create("memory.h5", "agent-1", 384)?;
+let mut backend = ClawhdfBackend::create(std::path::Path::new("memory.h5"), 384)?;
 
 // Ingest existing Markdown memory files
 let md = std::fs::read_to_string("MEMORY.md")?;
 let count = backend.ingest_markdown("MEMORY.md", &md)?;
 
-// Search (uses full pipeline: RRF → re-rank → confidence filter)
+// Search (full pipeline: weighted vector + BM25 fusion → re-rank → confidence filter)
 let results = backend.search("user preferences", &query_embedding, 5);
 
 // Export back to Markdown
@@ -375,22 +587,23 @@ let exported = backend.export_markdown("MEMORY.md")?;
 ## Crate Map
 
 ```
-clawhdf5 workspace (16 crates, ~92K lines of Rust; plus libaec-sys, an
-                     internal FFI bindings crate for the optional szip feature)
+clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
+                     and benches; plus libaec-sys, an internal FFI bindings
+                     crate for the optional szip feature)
 │
 ├── Core HDF5
-│   ├── clawhdf5-format      — Binary parser/writer (no_std), shared type definitions
-│   ├── clawhdf5-io          — I/O abstraction (buffered, mmap, async)
+│   ├── clawhdf5-format      — Binary parser/writer (no_std-capable), shared type definitions
+│   ├── clawhdf5-io          — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
 │   ├── clawhdf5-filters     — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format
 │   ├── clawhdf5-derive      — Proc macros
 │   ├── clawhdf5             — High-level API
 │   ├── clawhdf5-netcdf4     — NetCDF-4 support
-│   ├── clawhdf5-accel       — SIMD (NEON, AVX2, AVX-512)
+│   ├── clawhdf5-accel       — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
 │   └── clawhdf5-gpu         — GPU compute (wgpu, hand-written WGSL compute shaders)
 │
 ├── Agent Memory
-│   ├── clawhdf5-agent       — Memory engine (20.9K lines, 32 modules; WAL is CRC32-checked per entry)
-│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor (default backend; optional `parallel` feature)
+│   ├── clawhdf5-agent       — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
+│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor (default backend; f32 or int8 storage; `parallel` build)
 │   ├── clawhdf5-migrate     — SQLite → HDF5 migration
 │   ├── clawhdf5-android     — Android JNI bridge
 │   └── clawhdf5-cli         — CLI tool
@@ -411,10 +624,10 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
 
 | Paper | Key Insight | ClawhDF5 Module |
 |-------|-------------|-----------------|
-| **MemX** (2026) | RRF + multi-factor re-ranking | `hybrid`, `reranker` |
-| **Graph-Native Cognitive Memory** (2026) | Graph-structured belief revision | `knowledge` |
+| **MemX** (2026) | Hybrid fusion + multi-factor re-ranking | `hybrid`, `reranker` |
+| **Graph-Native Cognitive Memory** (2026) | Graph-structured memory (weighted, timestamped relations; entity timelines) | `knowledge`, `temporal` |
 | **CraniMem** (2026) | Bounded hippocampal memory | `consolidation` |
-| **D-MEM** (2026) | Reward prediction error gating | `consolidation` |
+| **D-MEM** (2026) | Surprise-gated storage (implemented as a novelty score) | `consolidation` |
 | **SYNAPSE** (2025) | Spreading activation for recall | `knowledge` |
 | **RAGdb** (2025) | Zero-dependency edge RAG | Architecture |
 | **MemoryGraft** (2025) | Memory poisoning attacks | `anomaly`, `provenance` |
@@ -429,23 +642,45 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
 
 | Flag | Default | Description |
 |------|---------|-------------|
-| `agent` | no | Full agent memory layer |
-| `float16` | **yes** | Half-precision embedding storage (2× compression) |
+| `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). Half-precision *storage* is the `MemoryConfig::float16` setting below, and needs no feature |
 | `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
-
-`MemoryConfig::quantized_index` (off by default) stores the HNSW index's own
-copy of the embeddings as `i8`, roughly halving a loaded store's memory
-(2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are
-approximate, so the query path re-scores the candidate pool against the exact
-embeddings the store already holds — recall matches the `f32` index, at about
-13% fewer queries per second. See `BENCHMARKS.md`, "Quantising the index copy".
-| `parallel` | no | Rayon parallel search |
+| `parallel` | **yes** | Parallel HNSW bulk build (same graph, ~3× faster on 16 cores) and Rayon brute-force search strategies |
+| `zstd` | no | Compress embeddings with Zstd instead of deflate when `MemoryConfig::compression` is on (links libzstd) |
 | `fast-math` | no | BLAS matrix-vector multiply |
 | `accelerate` | no | Apple Accelerate / AMX (macOS) |
 | `openblas` | no | OpenBLAS (Linux) |
 | `gpu` | no | GPU search via wgpu |
 | `async` | no | Tokio async with background flush |
 
+To opt out of the parallel build: `--no-default-features --features float16,hnsw`.
+For an exact linear cosine scan instead of HNSW: `--no-default-features --features float16`.
+
+`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search` tune the
+vector index (16 / 64 / scale-with-`k` by default) and are stored with the
+file.
+
+`MemoryConfig::quantized_index` (**on by default** for new stores) holds the
+HNSW index's own copy of the embeddings as `i8`, roughly halving a loaded
+store's memory (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised
+distances are approximate, so the query path re-scores the candidate pool
+against the exact embeddings the store already holds, which keeps recall at the
+`f32` index's level. It is also **faster**: 1.63x the queries per second at
+equal recall on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with
+index builds 1.8x and 2.3x faster respectively. Stores created before the
+setting existed keep their `f32` index; opt out for new stores with
+`quantized_index = false` or `clawhdf5-cli create --f32-index`. See
+[BENCHMARKS.md § Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index).
+
+`MemoryConfig::float16` (**on by default** for new stores) stores the
+embeddings on disk as IEEE half precision (numpy `float16`): at 100K × 384 the
+file drops from 154 to 81 MiB, checkpoints and opens get faster, and on the
+full LongMemEval haystack with real MiniLM embeddings every retrieval metric
+matches `f32`. Embeddings are rounded as they are saved, so the store searches
+the same before and after a reopen; values must lie within ±65504. Existing
+stores keep their setting. Opt out with `float16 = false` or
+`clawhdf5-cli create --f32` — e.g. for unnormalised vectors. See
+[BENCHMARKS.md § float16 embedding storage](BENCHMARKS.md#float16-embedding-storage-memoryconfigfloat16).
+
 ### `clawhdf5-format`
 
 | Flag | Default | Description |
@@ -454,26 +689,31 @@ embeddings the store already holds — recall matches the `f32` index, at about
 | `deflate` | yes | Deflate compression |
 | `checksum` | yes | Jenkins lookup3 verification |
 | `provenance` | yes | SHA-256 provenance attributes |
-| `fast-deflate` | **yes** | zlib-ng backend for faster deflate |
-| `system-zlib-decompress` | **yes** | Use the system zlib for decompression where available |
+| `zlib-rs` | **yes** | Pure-Rust deflate backend ([zlib-rs](https://github.com/trifectatechfoundation/zlib-rs)) |
+| `fast-deflate` | no | zlib-ng deflate backend instead (C; needs `cmake`). Overrides `zlib-rs` when both are on |
+| `system-zlib-decompress` | **yes** | Use Apple's system libz for decompression (macOS only; no effect elsewhere) |
 | `parallel` | no | Parallel chunk encoding + compression (rayon) |
 | `fast-checksum` | no | crc32fast-accelerated checksums |
 | `lz4` | no | LZ4 block compression filter (id 32004) |
 | `zstd` | no | Zstandard compression filter (id 32015) |
 | `pcodec` | no | Pcodec lossless numerical codec (id 32023, via `pco` crate) |
-| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
+| `system-zlib` | no | System zlib backend for deflate (C) |
 | `blake3_hash` | no | BLAKE3 content hashing for provenance |
+| `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) |
 
 ### `clawhdf5-ann`
 
 | Flag | Default | Description |
 |------|---------|-------------|
-| `parallel` | no | Rayon-parallel neighbor-distance computation during HNSW graph pruning |
+| `parallel` | no | Batched bulk build runs neighbour planning and back-link pruning on a Rayon pool; the graph is identical with or without it (enabled by `clawhdf5-agent`'s default `parallel`) |
 
 ### `clawhdf5-io`
 
 | Flag | Default | Description |
 |------|---------|-------------|
+| `mmap` | no | Memory-mapped reads (`memmap2`) |
+| `async` | no | Tokio-based async I/O |
+| `hsds` | no | HSDS (HDF REST service) client |
 | `mpi-io` | no | MPI-backed I/O via the `mpi` crate |
 
 > **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read
@@ -487,17 +727,17 @@ embeddings the store already holds — recall matches the `f32` index, at about
 ## Building
 
 ```bash
-# Default
+# Default (pure Rust: no cmake or C compiler needed)
 cargo build --workspace
 
 # Agent memory with all accelerations (Linux)
-cargo build -p clawhdf5-agent --features "agent,float16,parallel,fast-math"
+cargo build -p clawhdf5-agent --features fast-math
 
 # Agent memory with Apple Accelerate (macOS)
-cargo build -p clawhdf5-agent --features "agent,float16,accelerate,parallel,gpu"
+cargo build -p clawhdf5-agent --features "accelerate,gpu"
 
 # Tests
-cargo test --workspace            # all 1,650+ tests
+cargo test --workspace            # all 1,850+ tests
 cargo test -p clawhdf5-agent      # agent memory tests
 scripts/ci-test.sh                # what CI runs: fmt, clippy matrix, tests,
                                   # h5py/netCDF4 interop, no_std
@@ -519,25 +759,42 @@ cargo bench -p clawhdf5-bench               # h5bench-equivalent I/O suite
 
 ```
 agent_memory.h5
-├── /meta
-│   ├── schema_version: "1.0"
-│   ├── agent_id, embedder, embedding_dim
-│   └── created_at
+├── /meta                          (attributes)
+│   ├── schema_version: "1.0", edgehdf5_version
+│   ├── agent_id, embedder, embedding_dim, chunk_size, overlap, created_at
+│   ├── float16, compression, compression_level, compact_threshold,
+│   │   hebbian_boost, decay_factor, wal_enabled, wal_max_entries
+│   ├── quantized_index, hnsw_m, hnsw_ef_construction, hnsw_ef_search
+│   ├── wal_applied_len, wal_applied_crc   (WAL mark of the last checkpoint)
+│   └── ann_generation                     (ties the .ann sidecar to this checkpoint)
 ├── /memory
 │   ├── chunks:              string[N]
-│   ├── embeddings:  f32[N × D]  (or f16 with float16 flag)
+│   ├── embeddings:          f32[N × D], or f16 for a `float16` store
+│   │                        (chunked; deflate, or Zstd with the `zstd`
+│   │                         feature, when compression is on)
+│   ├── source_channel:      string[N]
+│   ├── timestamps:          f64[N]
+│   ├── session_ids:         string[N]
+│   ├── tags:                string[N]
 │   ├── tombstones:          u8[N]
-│   └── norms:       f32[N]      (pre-computed L2)
+│   ├── norms:               f32[N]      (pre-computed L2)
+│   └── activation_weights:  f32[N]      (Hebbian)
 ├── /sessions
-│   ├── ids:         string[S]
-│   └── summaries:   string[S]
+│   ├── ids, channels, summaries:  string[S]
+│   ├── start_idxs, end_idxs:      i64[S]
+│   └── timestamps:                f64[S]
 └── /knowledge_graph
-    ├── entity_names:    string[E]
-    ├── relation_srcs:   i64[R]
-    ├── relation_tgts:   i64[R]
-    └── relation_types:  string[R]
+    ├── entity_ids, entity_emb_idxs: i64[E]; entity_names, entity_types: string[E]
+    ├── relation_srcs, relation_tgts: i64[R]; relation_types: string[R]
+    ├── relation_weights: f32[R]; relation_ts: f64[R]
+    └── alias_strings: string[A]; alias_entity_ids: i64[A]   (when aliases exist)
 ```
 
+Alongside the store: `<store>.h5.wal` (write-ahead log), `<store>.h5.ann`
+(HNSW graph; derived, safe to delete) and `<store>.h5.lock` (single-writer
+lock). A second writer gets `MemoryError::Locked`; use
+`HDF5Memory::open_read_only` for a lock-free point-in-time view.
+
 ---
 
 ## Migration
@@ -556,9 +813,39 @@ Replace in `Cargo.toml` and source:
 
 ```bash
 cargo install --path crates/clawhdf5-migrate
-clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedding-dim 384
+clawhdf5-migrate --sqlite old.db --hdf5 memory.h5 --agent-id my-agent --embedder minilm
 ```
 
+The output is an ordinary `clawhdf5-agent` store, written through the agent's
+own API: open it with `HDF5Memory::open` (or `clawhdf5-cli --path memory.h5 …`)
+and search it straight away. The source must use the `memory_chunks` / `sessions` / `entities` / `relations` layout (names are
+configurable with `--*-table`); note that this is not ZeroClaw's schema, and
+ZeroClaw does not use clawhdf5. What carries over:
+
+| SQLite | Agent store |
+|--------|-------------|
+| `memory_chunks` | memory records (text, embedding, source channel, timestamp, session id, tags); rows with `deleted = 1` become deleted records, or are left out with `--skip-deleted` |
+| `sessions` | sessions (id, start/end index, channel, summary, timestamp) |
+| `entities`, `relations` | knowledge graph entities and relations; entities get new ids and relations are re-pointed at them |
+
+The chunk `id` column has no counterpart in the agent store, so records are
+written in `id` order and numbered from 0. Embeddings are stored as float16
+like any new store; `--f32` keeps full precision (and is required for values
+beyond ±65504). The embedding dimension is detected from the first row unless
+`--embedding-dim` is given, and every row must have it: a row of another length
+is an error, never truncated or padded. A source with no memory records (only
+sessions or the graph) needs `--embedding-dim`, since a store's dimension is
+fixed when it is created. Every row is checked before the output is created,
+so a source that cannot be migrated leaves an existing store at `--hdf5` as it
+was. `--incremental` adds to an existing store only the rows it does not
+already hold; the source must have the store's dimension, and records already
+in the store take the source's deleted flag (a row deleted in SQLite since the
+last run is deleted in the store; one un-deleted there is written again, as
+the agent has no un-delete). The tool reads the result back with
+`HDF5Memory::open_read_only`, compares it with the source (every row with
+`--validate-full`) and checks that a migrated record is found by search;
+`--dry-run` only counts the rows.
+
 ---
 
 ## Roadmap
@@ -572,10 +859,10 @@ See [ROADMAP.md](ROADMAP.md) for the full implementation tracker.
 - ✅ Temporal reasoning with sub-µs queries
 - ✅ Memory security + anomaly detection
 - ✅ Multi-modal memory (text/image/audio/video)
-- ✅ OpenClaw integration layer
+- ✅ Markdown ingest/export backend (`ClawhdfBackend`); an OpenClaw plugin was never built — see [docs/openclaw.md](docs/openclaw.md)
 - ✅ Comprehensive Criterion benchmarks
 
-**Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: publish the OpenClaw TypeScript bridge to npm, crates.io/PyPI publishing.
+**Phase 2** — MemoryArena and LongMemEval academic benchmarks are done (see [BENCHMARKS.md](BENCHMARKS.md), reproduced on a second machine); remaining: crates.io/PyPI publishing. The Node bindings are unpublished and known to be broken ([known issues](docs/known-issues.md)).
 
 ---
 
@@ -592,6 +879,6 @@ MIT
 ---
 
 <p align="center">
-  <em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em><br>
-  <em>~92,000 lines of Rust. Zero C dependencies. One file to remember everything.</em>
+  <em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em><br>
+  <em>~86,000 lines of Rust. Zero C dependencies. One file to remember everything.</em>
 </p>

ROADMAP.md

@@ -105,24 +105,30 @@
 
 ---
 
-## Track 7: OpenClaw Integration
-**Status:** 🟢 Complete
+## Track 7: OpenClaw Integration — withdrawn (2026-09-25)
+**Status:** ⚪ Withdrawn (the items below were library work; no OpenClaw integration shipped)
 **Priority:** Critical (for adoption)
 **Crates:** `clawhdf5-agent`, `clawhdf5-napi`
 
 - [x] **7.1** Memory backend trait — MemoryBackend with search/get/write/ingest/export/stats
 - [x] **7.2** Hybrid retrieval pipeline — ClawhdfBackend wires RRF → reranker → confidence rejection
 - [x] **7.3** Markdown import/export — MarkdownParser + MarkdownExporter with line tracking + metadata
-- [x] **7.4** memory_search tool — backed by full hybrid retrieval pipeline
-- [x] **7.5** memory_get tool — get() with path + line range support
+- [x] **7.4** `search()` — backed by the full hybrid retrieval pipeline (a Rust method; no OpenClaw tool was ever registered)
+- [x] **7.5** `get()` — read back by path, with a line slice (not an OpenClaw tool either)
 - [x] **7.6** Compaction integration — run_compaction() (decay + compact + WAL flush), run_consolidation() (hippocampal engine), tick_session(), flush_wal()
-- [x] **7.7** Config surface — `memory.backend = "clawhdf5"` schema documented in docs/openclaw-config.md
-- [x] **7.8** Documentation + migration guide — docs/migration-guide.md, docs/openclaw-integration.md (architecture, full API reference, code patterns)
+- [ ] **7.7** ~~Config surface — `memory.backend = "clawhdf5"`~~ — never valid OpenClaw config; docs removed
+- [ ] **7.8** ~~Documentation + migration guide~~ — removed: they described an integration that never worked
 
-**Node.js bridge:** `clawhdf5-napi` (napi-rs) → `@redclaw/clawhdf5` npm package with full TypeScript types.
+**Node.js bridge:** `clawhdf5-napi` (napi-rs) and a TypeScript wrapper in `packages/clawhdf5-node` exist but are unpublished, untested in CI and known to be broken (docs/known-issues.md).
 
 ---
 
+> **Withdrawn.** None of this track produced a working OpenClaw integration: no
+> plugin was built, the documented `memory.backend = "clawhdf5"` config was never
+> valid in any OpenClaw release, and the Node package was never published. The
+> Rust `ClawhdfBackend` remains as a library API. Not pursued for now; see
+> [docs/openclaw.md](docs/openclaw.md) for what a plugin would need today.
+
 ## Track 8: Benchmarking & Validation
 **Status:** 🟢 Complete
 **Priority:** High
@@ -142,7 +148,7 @@
 
 **Phase 1:** ~~Tracks 1, 2, 3 — core memory intelligence~~ 🟢 Complete
 **Phase 2:** ~~Track 4 (temporal) + Track 5 (security)~~ 🟢 Complete
-**Phase 3:** ~~Track 6 (multi-modal) + Track 7 (OpenClaw integration)~~ 🟢 Complete
+**Phase 3:** ~~Track 6 (multi-modal)~~ 🟢 Complete; Track 7 (OpenClaw integration) withdrawn
 **Phase 4:** ~~Track 8 (benchmarking + validation)~~ 🟢 Complete
 
 All 8 tracks delivered. 1,650+ tests passing, zero clippy warnings.

crates/clawhdf5-accel/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-accel"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "SIMD-accelerated operations for rustyhdf5"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"

crates/clawhdf5-accel/src/avx2.rs

@@ -25,6 +25,55 @@ unsafe fn hsum_256(v: __m256) -> f32 {
     _mm_cvtss_f32(result)
 }
 
+/// AVX2 dot product of two `i8` slices, widened to `i32`.
+///
+/// Each 16-byte half is sign-extended to sixteen `i16` lanes and multiplied
+/// pairwise with `madd_epi16`, which sums adjacent products straight into
+/// eight `i32` lanes — the widening that an autovectorised scalar loop does
+/// in several shuffles is one instruction here. A pair sum is at most
+/// `2 * 127 * 127`, far inside `i32`.
+///
+/// # Safety
+/// Caller must verify is_x86_feature_detected!("avx2").
+// SAFETY: Caller must have verified AVX2 via is_x86_feature_detected!.
+#[target_feature(enable = "avx2")]
+pub unsafe fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
+    // SAFETY: Caller guarantees AVX2 is available per the # Safety contract;
+    // every load reads 32 bytes at an index checked against `len` first.
+    unsafe {
+        assert_eq!(a.len(), b.len());
+        let len = a.len();
+        let mut i = 0;
+        let mut acc0 = _mm256_setzero_si256();
+        let mut acc1 = _mm256_setzero_si256();
+
+        while i + 32 <= len {
+            let va = _mm256_loadu_si256(a.as_ptr().add(i).cast());
+            let vb = _mm256_loadu_si256(b.as_ptr().add(i).cast());
+            let a_lo = _mm256_cvtepi8_epi16(_mm256_castsi256_si128(va));
+            let b_lo = _mm256_cvtepi8_epi16(_mm256_castsi256_si128(vb));
+            let a_hi = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(va, 1));
+            let b_hi = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(vb, 1));
+            acc0 = _mm256_add_epi32(acc0, _mm256_madd_epi16(a_lo, b_lo));
+            acc1 = _mm256_add_epi32(acc1, _mm256_madd_epi16(a_hi, b_hi));
+            i += 32;
+        }
+
+        // Horizontal sum of the eight i32 lanes.
+        let v = _mm256_add_epi32(acc0, acc1);
+        let s128 = _mm_add_epi32(_mm256_castsi256_si128(v), _mm256_extracti128_si256(v, 1));
+        let s64 = _mm_add_epi32(s128, _mm_unpackhi_epi64(s128, s128));
+        let s32 = _mm_add_epi32(s64, _mm_shuffle_epi32(s64, 0b01));
+        let mut sum = _mm_cvtsi128_si32(s32);
+
+        while i < len {
+            sum += i32::from(a[i]) * i32::from(b[i]);
+            i += 1;
+        }
+        sum
+    }
+}
+
 /// AVX2 dot product for f32 slices.
 ///
 /// # Safety

crates/clawhdf5-accel/src/lib.rs

@@ -122,6 +122,36 @@ pub fn dot_product(a: &[f32], b: &[f32]) -> f32 {
     }
 }
 
+/// Dot product of two `i8` slices, widened to `i32`.
+///
+/// The kernel behind int8-quantised vector search. On x86-64 it uses the AVX2
+/// path whenever AVX2 is present (including on AVX-512 machines, where it is
+/// what the f32 kernels use too on a default build). On aarch64 it uses the
+/// ARMv8.2 `SDOT` instruction when the CPU has the dot-product extension, and
+/// plain NEON otherwise.
+pub fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
+    match detect_backend() {
+        #[cfg(target_arch = "aarch64")]
+        Backend::Neon => {
+            if std::arch::is_aarch64_feature_detected!("dotprod") {
+                // SAFETY: the dotprod extension was just detected at runtime.
+                unsafe { neon::dot_i8_dotprod(a, b) }
+            } else {
+                // SAFETY: NEON is always available on aarch64.
+                unsafe { neon::dot_i8(a, b) }
+            }
+        }
+
+        #[cfg(target_arch = "x86_64")]
+        // SAFETY: both variants imply AVX2 was detected at runtime (the
+        // AVX-512 backend is only selected on CPUs that also have AVX2).
+        Backend::Avx2 | Backend::Avx512 if is_x86_feature_detected!("avx2") => unsafe {
+            avx2::dot_i8(a, b)
+        },
+        _ => scalar::dot_i8(a, b),
+    }
+}
+
 /// Compute the L2 norm (magnitude) of a vector.
 pub fn vector_norm(v: &[f32]) -> f32 {
     dot_product(v, v).sqrt()
@@ -713,3 +743,78 @@ mod tests {
         }
     }
 }
+
+#[cfg(test)]
+mod dot_i8_tests {
+    use super::*;
+
+    fn codes(n: usize, seed: u64) -> Vec<i8> {
+        let mut state = seed;
+        (0..n)
+            .map(|_| {
+                state = state
+                    .wrapping_mul(6_364_136_223_846_793_005)
+                    .wrapping_add(1_442_695_040_888_963_407);
+                // Full range, including the extremes.
+                ((state >> 56) as u8) as i8
+            })
+            .collect()
+    }
+
+    #[test]
+    fn dispatched_kernel_matches_scalar_exactly() {
+        // Integer arithmetic: the SIMD path must agree bit for bit, at every
+        // length — including ones that are not multiples of the 32-byte block,
+        // which exercise the tail.
+        for len in [0, 1, 7, 31, 32, 33, 63, 64, 100, 384, 385, 1536] {
+            let a = codes(len, 1 + len as u64);
+            let b = codes(len, 1000 + len as u64);
+            assert_eq!(dot_i8(&a, &b), scalar::dot_i8(&a, &b), "len {len}");
+        }
+    }
+
+    /// Dispatch only ever takes one path on a given CPU, so on a machine with
+    /// the dot-product extension the plain-NEON kernel would otherwise go
+    /// untested. Check each aarch64 kernel against scalar directly.
+    #[cfg(target_arch = "aarch64")]
+    #[test]
+    fn every_aarch64_kernel_matches_scalar_exactly() {
+        for len in [0, 1, 7, 15, 16, 17, 31, 32, 33, 63, 64, 100, 384, 385, 1536] {
+            let a = codes(len, 7 + len as u64);
+            let b = codes(len, 7000 + len as u64);
+            let want = scalar::dot_i8(&a, &b);
+            // SAFETY: NEON is always available on aarch64.
+            assert_eq!(unsafe { neon::dot_i8(&a, &b) }, want, "neon, len {len}");
+            if std::arch::is_aarch64_feature_detected!("dotprod") {
+                // SAFETY: the dotprod extension was just detected.
+                assert_eq!(
+                    unsafe { neon::dot_i8_dotprod(&a, &b) },
+                    want,
+                    "dotprod, len {len}"
+                );
+            }
+        }
+        // The extremes, through both kernels.
+        let lo = vec![-128i8; 4096];
+        let hi = vec![127i8; 4096];
+        // SAFETY: NEON is always available on aarch64.
+        assert_eq!(unsafe { neon::dot_i8(&lo, &lo) }, 4096 * 128 * 128);
+        // SAFETY: NEON is always available on aarch64.
+        assert_eq!(unsafe { neon::dot_i8(&lo, &hi) }, -4096 * 128 * 127);
+        if std::arch::is_aarch64_feature_detected!("dotprod") {
+            // SAFETY: the dotprod extension was just detected.
+            assert_eq!(unsafe { neon::dot_i8_dotprod(&lo, &lo) }, 4096 * 128 * 128);
+            // SAFETY: the dotprod extension was just detected.
+            assert_eq!(unsafe { neon::dot_i8_dotprod(&lo, &hi) }, -4096 * 128 * 127);
+        }
+    }
+
+    #[test]
+    fn extremes_do_not_overflow() {
+        // -128 * -128 is the largest product; a long run of it must still fit.
+        let a = vec![-128i8; 4096];
+        assert_eq!(dot_i8(&a, &a), 4096 * 128 * 128);
+        let b = vec![127i8; 4096];
+        assert_eq!(dot_i8(&a, &b), -4096 * 128 * 127);
+    }
+}

crates/clawhdf5-accel/src/neon.rs

@@ -180,3 +180,130 @@ pub fn checksum_fletcher32(data: &[u8]) -> u32 {
 
     (sum2 << 16) | sum1
 }
+
+/// NEON dot product of two `i8` slices, widened to `i32`, for any aarch64 CPU.
+///
+/// `vmull_s8` multiplies eight lanes into `i16` — even `-128 * -128` is 16 384,
+/// inside `i16` — and `vpadalq_s16` adds adjacent pairs of those into `i32`
+/// accumulators, so nothing can overflow before the final horizontal sum.
+///
+/// CPUs with the ARMv8.2 dot-product extension should use
+/// [`dot_i8_dotprod`], which does the multiply and the accumulate in one
+/// instruction.
+///
+/// # Safety
+/// Caller must ensure aarch64 target (NEON always available).
+// SAFETY: NEON is always available on aarch64 targets; caller guarantees aarch64.
+#[target_feature(enable = "neon")]
+pub unsafe fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
+    assert_eq!(a.len(), b.len());
+    let len = a.len();
+    let mut i = 0;
+    let mut acc0 = vdupq_n_s32(0);
+    let mut acc1 = vdupq_n_s32(0);
+
+    while i + 16 <= len {
+        // SAFETY: NEON is available per the # Safety contract, and both
+        // 16-byte loads start at an index checked against `len` above.
+        unsafe {
+            let va = vld1q_s8(a.as_ptr().add(i));
+            let vb = vld1q_s8(b.as_ptr().add(i));
+            acc0 = vpadalq_s16(acc0, vmull_s8(vget_low_s8(va), vget_low_s8(vb)));
+            acc1 = vpadalq_s16(acc1, vmull_high_s8(va, vb));
+        }
+        i += 16;
+    }
+
+    let mut sum = vaddvq_s32(vaddq_s32(acc0, acc1));
+    while i < len {
+        sum += i32::from(a[i]) * i32::from(b[i]);
+        i += 1;
+    }
+    sum
+}
+
+/// One `SDOT`: for each of the four `i32` lanes of `acc`, add the dot
+/// product of the corresponding four `i8` pairs from `a` and `b`.
+///
+/// Written as inline assembly because the `vdotq_s32` intrinsic is still
+/// behind the unstable `stdarch_neon_dotprod` feature; inline assembly is
+/// stable on aarch64.
+///
+/// # Safety
+/// Caller must ensure the CPU supports the `dotprod` extension.
+#[inline]
+#[target_feature(enable = "neon,dotprod")]
+unsafe fn sdot(acc: int32x4_t, a: int8x16_t, b: int8x16_t) -> int32x4_t {
+    let mut acc = acc;
+    // SAFETY: `dotprod` is enabled for this function and the caller
+    // guarantees the CPU supports it. The instruction reads only its three
+    // vector registers and touches no memory.
+    unsafe {
+        std::arch::asm!(
+            "sdot {acc:v}.4s, {a:v}.16b, {b:v}.16b",
+            acc = inout(vreg) acc,
+            a = in(vreg) a,
+            b = in(vreg) b,
+            options(pure, nomem, nostack),
+        );
+    }
+    acc
+}
+
+/// NEON dot product of two `i8` slices using the ARMv8.2 dot-product
+/// extension (`SDOT`): sixteen multiply-accumulates per instruction, straight
+/// into `i32` lanes.
+///
+/// Present on the cores this crate actually runs on — Cortex-A76 and later
+/// (Raspberry Pi 5, current Android phones), Neoverse-N1 (Graviton2, Ampere
+/// Altra), and every Apple Silicon generation.
+///
+/// # Safety
+/// Caller must verify `is_aarch64_feature_detected!("dotprod")`.
+// SAFETY: caller has verified the dotprod extension at runtime.
+#[target_feature(enable = "neon,dotprod")]
+pub unsafe fn dot_i8_dotprod(a: &[i8], b: &[i8]) -> i32 {
+    assert_eq!(a.len(), b.len());
+    let len = a.len();
+    let mut i = 0;
+    let mut acc0 = vdupq_n_s32(0);
+    let mut acc1 = vdupq_n_s32(0);
+
+    // Two independent accumulators so consecutive SDOTs are not serialised on
+    // one register.
+    while i + 32 <= len {
+        // SAFETY: dotprod is available per the # Safety contract, and every
+        // 16-byte load starts at an index checked against `len` above.
+        unsafe {
+            acc0 = sdot(
+                acc0,
+                vld1q_s8(a.as_ptr().add(i)),
+                vld1q_s8(b.as_ptr().add(i)),
+            );
+            acc1 = sdot(
+                acc1,
+                vld1q_s8(a.as_ptr().add(i + 16)),
+                vld1q_s8(b.as_ptr().add(i + 16)),
+            );
+        }
+        i += 32;
+    }
+    if i + 16 <= len {
+        // SAFETY: as above; the load is bounds-checked by this condition.
+        unsafe {
+            acc0 = sdot(
+                acc0,
+                vld1q_s8(a.as_ptr().add(i)),
+                vld1q_s8(b.as_ptr().add(i)),
+            );
+        }
+        i += 16;
+    }
+
+    let mut sum = vaddvq_s32(vaddq_s32(acc0, acc1));
+    while i < len {
+        sum += i32::from(a[i]) * i32::from(b[i]);
+        i += 1;
+    }
+    sum
+}

crates/clawhdf5-accel/src/scalar.rs

@@ -140,3 +140,33 @@ fn f16_to_f32_soft(h: u16) -> f32 {
 
     f32::from_bits(f32_bits)
 }
+
+/// Dot product of two `i8` slices, widened to `i32`.
+///
+/// `dim` terms of at most `127 * 127` fit an `i32` for any realistic
+/// dimension (over 130 000 terms before overflow is possible).
+pub fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
+    assert_eq!(a.len(), b.len());
+    // Four independent accumulators over 32-lane blocks: the widening product
+    // has to sit in a fixed-length chunk for the vectoriser to see it, and the
+    // separate accumulators keep it off one dependency chain.
+    const LANE: usize = 8;
+    let (a_blocks, a_tail) = a.as_chunks::<{ LANE * 4 }>();
+    let (b_blocks, b_tail) = b.as_chunks::<{ LANE * 4 }>();
+    let mut acc = [0i32; 4];
+    for (x, y) in a_blocks.iter().zip(b_blocks) {
+        for (lane, slot) in acc.iter_mut().enumerate() {
+            let mut sum = 0i32;
+            for k in 0..LANE {
+                sum += i32::from(x[lane * LANE + k]) * i32::from(y[lane * LANE + k]);
+            }
+            *slot += sum;
+        }
+    }
+    let tail: i32 = a_tail
+        .iter()
+        .zip(b_tail)
+        .map(|(&x, &y)| i32::from(x) * i32::from(y))
+        .sum();
+    acc[0] + acc[1] + acc[2] + acc[3] + tail
+}

crates/clawhdf5-agent/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-agent"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "HDF5-backed persistent memory store for on-device AI agents"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -10,14 +11,18 @@ keywords = ["agent", "memory", "hdf5", "vector-search", "embedding"]
 categories = ["database", "science", "algorithms"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0", features = ["parallel", "fast-checksum"] }
-clawhdf5 = { path = "../clawhdf5", version = "2.6.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.6.0", features = ["mmap"] }
-clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.6.0" }
-clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.6.0", optional = true }
-clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.6.0", optional = true, default-features = false }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0", features = ["parallel", "fast-checksum"] }
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0", features = ["mmap"] }
+clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.7.0" }
+clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true }
+clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
 serde = { workspace = true }
 byteorder = "1"
+# Signed checkpoints (MemoryConfig-independent; see `signing`). Pure Rust.
+ed25519-dalek = { version = "2", features = ["rand_core"] }
+sha2 = "0.10"
+rand_core = { version = "0.6", features = ["getrandom"] }
 half = { workspace = true, optional = true }
 rayon = { version = "1", optional = true }
 matrixmultiply = { version = "0.3", optional = true }
@@ -44,6 +49,10 @@ harness = false
 name = "memory_bench"
 harness = false
 
+[[bench]]
+name = "multimodal_bench"
+harness = false
+
 [features]
 default = ["float16", "hnsw", "parallel"]
 float16 = ["half"]
@@ -59,7 +68,6 @@ zstd = ["clawhdf5/zstd"]
 # `--no-default-features` (plus re-enabling other defaults) to force the exact
 # linear cosine scan.
 hnsw = ["clawhdf5-ann"]
-agent = []
 gpu = ["clawhdf5-gpu/gpu-wgpu"]
 fast-math = ["matrixmultiply"]
 accelerate = ["accelerate-src", "cblas-sys"]

crates/clawhdf5-agent/benches/multimodal_bench.rs

@@ -0,0 +1,107 @@
+//! Multi-modal memory search benchmarks (`clawhdf5_agent::multimodal`).
+//!
+//! Covers `MultiModalStore::search_cross_modal` (every embedding of every
+//! record, whatever its modality) and, for comparison,
+//! `MultiModalStore::search_by_modality` restricted to one modality.
+//!
+//! Corpus: N records (1K and 10K), each carrying two 384-dim embeddings —
+//! a text embedding of its caption plus one embedding of its primary modality,
+//! cycling Image / Audio / Video — so a cross-modal query scores 2N vectors.
+//! All data comes from a fixed-seed LCG, so every run sees the same corpus.
+//!
+//! Run: `cargo bench -p clawhdf5-agent --bench multimodal_bench`
+
+use std::collections::HashMap;
+
+use clawhdf5_agent::multimodal::{
+    MediaRef, ModalEmbedding, Modality, MultiModalRecord, MultiModalStore,
+};
+use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
+
+// ---------------------------------------------------------------------------
+// Simple deterministic PRNG (LCG), same as the other agent benches
+// ---------------------------------------------------------------------------
+
+struct Rng(u32);
+
+impl Rng {
+    fn new(seed: u32) -> Self {
+        Self(seed)
+    }
+    fn next_u32(&mut self) -> u32 {
+        self.0 = self.0.wrapping_mul(1103515245).wrapping_add(12345);
+        self.0 >> 16
+    }
+    fn next_f32(&mut self) -> f32 {
+        self.next_u32() as f32 / 65536.0 - 0.5
+    }
+}
+
+fn make_vec(rng: &mut Rng, dim: usize) -> Vec<f32> {
+    (0..dim).map(|_| rng.next_f32()).collect()
+}
+
+// ---------------------------------------------------------------------------
+// Corpus
+// ---------------------------------------------------------------------------
+
+const DIM: usize = 384;
+const K: usize = 10;
+
+const MEDIA: [(Modality, &str, &str); 3] = [
+    (Modality::Image, "image/png", "clip-vit-base"),
+    (Modality::Audio, "audio/wav", "clap-base"),
+    (Modality::Video, "video/mp4", "xclip-base"),
+];
+
+fn build_store(n: usize, seed: u32) -> MultiModalStore {
+    let mut rng = Rng::new(seed);
+    let mut store = MultiModalStore::new();
+    for i in 0..n {
+        let (modality, mime, model) = &MEDIA[i % MEDIA.len()];
+        let embeddings = vec![
+            ModalEmbedding::new(Modality::Text, make_vec(&mut rng, DIM), "minilm-l6"),
+            ModalEmbedding::new(modality.clone(), make_vec(&mut rng, DIM), *model),
+        ];
+        store.add_record(MultiModalRecord {
+            id: 0,
+            primary_modality: modality.clone(),
+            text_content: Some(format!("{modality} memory {i}")),
+            media_ref: Some(MediaRef::path(format!("/media/{i}"), *mime)),
+            embeddings,
+            observation: None,
+            timestamp: 1_700_000_000.0 + i as f64,
+            metadata: HashMap::new(),
+        });
+    }
+    store
+}
+
+// ---------------------------------------------------------------------------
+// Benchmarks
+// ---------------------------------------------------------------------------
+
+fn multimodal_search_benches(c: &mut Criterion) {
+    let query = make_vec(&mut Rng::new(99), DIM);
+
+    let mut group = c.benchmark_group("multimodal_search");
+    group.sample_size(50);
+
+    for (label, n) in [("1k", 1_000usize), ("10k", 10_000)] {
+        let store = build_store(n, 42);
+        assert_eq!(store.count(), n);
+
+        group.bench_with_input(BenchmarkId::new("cross_modal", label), &n, |b, _| {
+            b.iter(|| store.search_cross_modal(&query, K));
+        });
+
+        group.bench_with_input(BenchmarkId::new("by_modality_image", label), &n, |b, _| {
+            b.iter(|| store.search_by_modality(&Modality::Image, &query, K));
+        });
+    }
+
+    group.finish();
+}
+
+criterion_group!(multimodal_benches, multimodal_search_benches);
+criterion_main!(multimodal_benches);

crates/clawhdf5-agent/src/agents_md.rs

@@ -119,6 +119,9 @@ mod tests {
             wal_enabled: false,
             wal_max_entries: 500,
             quantized_index: false,
+            hnsw_m: 16,
+            hnsw_ef_construction: 64,
+            hnsw_ef_search: 0,
         }
     }
 

crates/clawhdf5-agent/src/bm25.rs

@@ -88,8 +88,11 @@ impl BM25Index {
     /// Search the index for a query, returning the top `k` results
     /// as `(doc_id, score)` pairs sorted by score descending.
     ///
-    /// Uses Block-Max WAND for early termination when remaining documents
-    /// cannot beat the current top-k threshold.
+    /// Scores every matching document exhaustively, then keeps the top `k`.
+    /// There is no early termination (WAND, MaxScore): the store's hot path
+    /// is [`scores`](Self::scores), because score fusion normalises over the
+    /// whole matching set and so needs every score, which no pruning scheme
+    /// can skip. This method is for BM25-only callers.
     pub fn search(&self, query: &str, k: usize) -> Vec<(usize, f32)> {
         if k == 0 {
             return Vec::new();
@@ -561,8 +564,9 @@ mod tests {
     }
 
     #[test]
-    fn wand_returns_same_results_as_exhaustive() {
-        // WAND-style search should produce same scores as exhaustive
+    fn top_k_search_matches_ranking_every_score() {
+        // `search` must agree with ranking the full `scores` set — the
+        // bounded heap is an optimisation over sorting, not an approximation.
         let docs: Vec<String> = (0..100)
             .map(|i| {
                 if i % 3 == 0 {

crates/clawhdf5-agent/src/cache.rs

@@ -1,6 +1,7 @@
 //! In-memory cache for memory entries, sessions, and knowledge graph.
 
 use crate::vector_search;
+use clawhdf5_format::float16::round_to_f16;
 
 /// Every entry's embedding, in one contiguous `[N x dim]` buffer.
 ///
@@ -149,6 +150,11 @@ pub struct MemoryCache {
     pub norms: Vec<f32>,
     /// Hebbian activation weights (default 1.0 per entry).
     pub activation_weights: Vec<f32>,
+    /// Round every embedding to IEEE half precision as it enters the cache,
+    /// so the cache holds exactly what a `float16` store writes to disk. Set
+    /// it with [`MemoryCache::set_half_precision`], which also rounds the
+    /// rows already held.
+    pub half_precision: bool,
 }
 
 impl MemoryCache {
@@ -164,9 +170,44 @@ impl MemoryCache {
             embedding_dim,
             norms: Vec::new(),
             activation_weights: Vec::new(),
+            half_precision: false,
         }
     }
 
+    /// Switch half-precision rounding on or off. Turning it on rounds every
+    /// embedding already held (and recomputes norms where one changed) —
+    /// e.g. a `float16` store whose last checkpoint predates half-precision
+    /// storage and so is still `f32` on disk.
+    pub fn set_half_precision(&mut self, on: bool) {
+        self.half_precision = on;
+        if !on {
+            return;
+        }
+        for i in 0..self.embeddings.len() {
+            let row = &self.embeddings[i];
+            if row
+                .iter()
+                .all(|&v| round_to_f16(v).to_bits() == v.to_bits())
+            {
+                continue;
+            }
+            let rounded: Vec<f32> = row.iter().map(|&v| round_to_f16(v)).collect();
+            self.norms[i] = vector_search::compute_norm(&rounded);
+            self.embeddings.set(i, &rounded);
+        }
+    }
+
+    /// The embedding as the cache will hold it: rounded to half precision
+    /// when [`Self::half_precision`] is on, otherwise unchanged.
+    fn stored_form(&self, mut embedding: Vec<f32>) -> Vec<f32> {
+        if self.half_precision {
+            for v in &mut embedding {
+                *v = round_to_f16(*v);
+            }
+        }
+        embedding
+    }
+
     /// Kept for callers that used to have to re-flatten after a bulk load.
     /// The buffer is always flat now, so there is nothing to rebuild.
     #[deprecated(note = "embeddings are stored flat; this is a no-op")]
@@ -202,6 +243,7 @@ impl MemoryCache {
         tags: String,
     ) -> usize {
         let idx = self.chunks.len();
+        let embedding = self.stored_form(embedding);
         let norm = vector_search::compute_norm(&embedding);
         self.chunks.push(chunk);
         self.embeddings.push(&embedding);
@@ -240,6 +282,7 @@ impl MemoryCache {
         session_id: String,
     ) {
         if idx < self.chunks.len() {
+            let embedding = self.stored_form(embedding);
             let norm = vector_search::compute_norm(&embedding);
             self.chunks[idx] = chunk;
             self.embeddings.set(idx, &embedding);
@@ -439,4 +482,60 @@ mod tests {
             .reset_from(2, vec![vec![1.0, 2.0], vec![3.0, 4.0]]);
         assert_eq!(cache.embeddings.as_flat(), vec![1.0, 2.0, 3.0, 4.0]);
     }
+
+    #[test]
+    fn set_half_precision_rounds_existing_rows_and_their_norms() {
+        // A store with float16 set whose checkpoint is still f32 on disk
+        // loads full-precision rows; switching rounding on must bring them to
+        // exactly what the next checkpoint will write.
+        let mut cache = MemoryCache::new(3);
+        cache.push(
+            "a".into(),
+            vec![0.1, 0.2, 0.3],
+            "c".into(),
+            0.0,
+            "s".into(),
+            "".into(),
+        );
+        cache.push(
+            "b".into(),
+            vec![0.5, 0.25, 1.0],
+            "c".into(),
+            0.0,
+            "s".into(),
+            "".into(),
+        );
+        let exact_norm = cache.norms[0];
+
+        cache.set_half_precision(true);
+        let row0: Vec<f32> = [0.1f32, 0.2, 0.3]
+            .iter()
+            .map(|&v| round_to_f16(v))
+            .collect();
+        assert_eq!(&cache.embeddings[0], row0.as_slice());
+        assert_eq!(cache.norms[0], vector_search::compute_norm(&row0));
+        assert_ne!(cache.norms[0], exact_norm);
+        // Already representable: untouched.
+        assert_eq!(&cache.embeddings[1], &[0.5, 0.25, 1.0]);
+
+        // New rows are rounded as they arrive, and updates too.
+        cache.push(
+            "c".into(),
+            vec![0.1, 0.0, 0.0],
+            "c".into(),
+            0.0,
+            "s".into(),
+            "".into(),
+        );
+        assert_eq!(cache.embeddings[2][0], round_to_f16(0.1));
+        cache.update(
+            2,
+            "c".into(),
+            vec![0.3, 0.0, 0.0],
+            "c".into(),
+            0.0,
+            "s".into(),
+        );
+        assert_eq!(cache.embeddings[2][0], round_to_f16(0.3));
+    }
 }

crates/clawhdf5-agent/src/consolidation.rs

@@ -144,9 +144,44 @@ pub struct ConsolidationStats {
 
 pub struct ImportanceScorer;
 
+/// Sum of squares, in 8-wide lanes so it vectorises.
+fn sum_of_squares(a: &[f32]) -> f32 {
+    let (blocks, tail) = a.as_chunks::<8>();
+    let mut acc = [0.0f32; 8];
+    for b in blocks {
+        for i in 0..8 {
+            acc[i] += b[i] * b[i];
+        }
+    }
+    acc.iter().sum::<f32>() + tail.iter().map(|x| x * x).sum::<f32>()
+}
+
+/// `(a · b, |b|²)` in one pass over equal-length slices, in 8-wide lanes.
+fn dot_and_norm2(a: &[f32], b: &[f32]) -> (f32, f32) {
+    let (a_blocks, a_tail) = a.as_chunks::<8>();
+    let (b_blocks, b_tail) = b.as_chunks::<8>();
+    let mut dot = [0.0f32; 8];
+    let mut nb = [0.0f32; 8];
+    for (x, y) in a_blocks.iter().zip(b_blocks) {
+        for i in 0..8 {
+            dot[i] += x[i] * y[i];
+            nb[i] += y[i] * y[i];
+        }
+    }
+    let mut d = dot.iter().sum::<f32>();
+    let mut n = nb.iter().sum::<f32>();
+    for (x, y) in a_tail.iter().zip(b_tail) {
+        d += x * y;
+        n += y * y;
+    }
+    (d, n)
+}
+
 impl ImportanceScorer {
     /// Cosine similarity between two embedding slices.
-    /// Returns 0.0 if either norm is zero.
+    /// Returns 0.0 if either norm is zero. The reference that
+    /// [`Self::score_surprise`] is tested against.
+    #[cfg(test)]
     fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 {
         let len = a.len().min(b.len());
         if len == 0 {
@@ -167,13 +202,54 @@ impl ImportanceScorer {
 
     /// Novelty score: 1.0 − max cosine similarity against all existing records.
     /// Returns 1.0 when there are no existing memories.
+    ///
+    /// Same result as the reference cosine similarity against each record, but the
+    /// new embedding's norm is computed once rather than per record, each
+    /// record costs one fused pass (dot product and its norm together) rather
+    /// than three, and a large working set is scored in parallel. Every insert
+    /// scores against the whole working tier, so this is what an unbounded
+    /// working tier pays for: at 100K records it was the difference between a
+    /// benchmark finishing and not (`BENCHMARKS.md`, "Consolidation Efficiency").
     pub fn score_surprise(embedding: &[f32], existing_memories: &[&MemoryRecord]) -> f32 {
         if existing_memories.is_empty() {
             return 1.0;
         }
+        let query_norm2 = sum_of_squares(embedding);
+        let similarity = |r: &&MemoryRecord| -> f32 {
+            let other = &r.embedding;
+            let len = embedding.len().min(other.len());
+            if len == 0 {
+                return 0.0;
+            }
+            let (dot, other_norm2) = dot_and_norm2(&embedding[..len], &other[..len]);
+            // A shorter record compares against the query's matching prefix.
+            let q2 = if len == embedding.len() {
+                query_norm2
+            } else {
+                sum_of_squares(&embedding[..len])
+            };
+            if q2 == 0.0 || other_norm2 == 0.0 {
+                return 0.0;
+            }
+            dot / (q2.sqrt() * other_norm2.sqrt())
+        };
+        #[cfg(feature = "parallel")]
+        let max_sim = if existing_memories.len() >= 4096 {
+            use rayon::prelude::*;
+            existing_memories
+                .par_iter()
+                .map(similarity)
+                .reduce(|| f32::NEG_INFINITY, f32::max)
+        } else {
+            existing_memories
+                .iter()
+                .map(similarity)
+                .fold(f32::NEG_INFINITY, f32::max)
+        };
+        #[cfg(not(feature = "parallel"))]
         let max_sim = existing_memories
             .iter()
-            .map(|r| Self::cosine_similarity(embedding, &r.embedding))
+            .map(similarity)
             .fold(f32::NEG_INFINITY, f32::max);
         (1.0 - max_sim).clamp(0.0, 1.0)
     }
@@ -471,6 +547,54 @@ impl ConsolidationEngine {
 mod tests {
     use super::*;
 
+    #[test]
+    fn score_surprise_matches_the_reference_cosine() {
+        let mut x = 0x2545_F491_4F6C_DD1Du64;
+        let mut next = || {
+            x ^= x << 13;
+            x ^= x >> 7;
+            x ^= x << 17;
+            (x >> 40) as f32 / (1u64 << 24) as f32 - 0.5
+        };
+        let make = |id: u64, v: Vec<f32>| MemoryRecord {
+            id,
+            chunk: String::new(),
+            embedding: v,
+            tier: MemoryTier::Working,
+            importance: 0.0,
+            access_count: 0,
+            last_accessed: 0.0,
+            created_at: 0.0,
+            source: MemorySource::User,
+        };
+        // Ordinary rows, a shorter one, an empty one and a zero vector; and
+        // enough rows to take the parallel path too.
+        for n in [5usize, 5000] {
+            let mut recs: Vec<MemoryRecord> = (0..n as u64)
+                .map(|i| make(i, (0..37).map(|_| next()).collect()))
+                .collect();
+            recs.push(make(9_000, (0..20).map(|_| next()).collect()));
+            recs.push(make(9_001, Vec::new()));
+            recs.push(make(9_002, vec![0.0; 37]));
+            let refs: Vec<&MemoryRecord> = recs.iter().collect();
+            for _ in 0..5 {
+                let q: Vec<f32> = (0..37).map(|_| next()).collect();
+                let expected = (1.0
+                    - refs
+                        .iter()
+                        .map(|r| ImportanceScorer::cosine_similarity(&q, &r.embedding))
+                        .fold(f32::NEG_INFINITY, f32::max))
+                .clamp(0.0, 1.0);
+                let got = ImportanceScorer::score_surprise(&q, &refs);
+                assert!((got - expected).abs() < 1e-5, "n={n}: {got} vs {expected}");
+            }
+        }
+        assert_eq!(
+            ImportanceScorer::score_surprise(&[0.0; 4], &[&make(1, vec![1.0; 4])]),
+            1.0
+        );
+    }
+
     // Helper: build a simple normalised embedding of given dimension.
     fn unit_vec(dim: usize, hot: usize) -> Vec<f32> {
         let mut v = vec![0.0f32; dim];

crates/clawhdf5-agent/src/hybrid.rs

@@ -65,31 +65,34 @@ pub fn hybrid_search_fused(
 ) -> Vec<(usize, f32)> {
     // Get raw scores from both systems. Request all results so normalization
     // covers the full distribution.
-    // Use parallel search when rayon feature is enabled and vector count > 10K.
-    let vec_scores = {
-        #[cfg(feature = "parallel")]
-        {
-            if vectors.count() > 10_000 {
-                vector_search::parallel_cosine_batch(
-                    query_embedding,
-                    vectors,
-                    tombstones,
-                    vectors.count(),
-                )
-            } else {
-                vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
-            }
-        }
-        #[cfg(not(feature = "parallel"))]
-        {
-            vector_search::cosine_similarity_batch(query_embedding, vectors, tombstones)
-        }
-    };
+    let vec_scores = exact_vector_scores(query_embedding, vectors, tombstones);
     let kw_scores = bm25_index.scores(query_text);
 
     fuse(vec_scores, kw_scores, fusion, k)
 }
 
+/// Cosine similarity of `query_embedding` to every vector whose `skip` byte is
+/// 0 (a tombstone, or any other exclusion mask). Parallel above 10K vectors
+/// when the `parallel` feature is on.
+pub fn exact_vector_scores(
+    query_embedding: &[f32],
+    vectors: &(impl crate::vector_search::VectorSet + Sync + ?Sized),
+    skip: &[u8],
+) -> Vec<(usize, f32)> {
+    #[cfg(feature = "parallel")]
+    {
+        if vectors.count() > 10_000 {
+            return vector_search::parallel_cosine_batch(
+                query_embedding,
+                vectors,
+                skip,
+                vectors.count(),
+            );
+        }
+    }
+    vector_search::cosine_similarity_batch(query_embedding, vectors, skip)
+}
+
 /// Merge pre-computed vector-similarity and keyword scores into a single ranking.
 ///
 /// Both score sets are independently min-max normalized to [0, 1] and combined

crates/clawhdf5-agent/src/knowledge.rs

@@ -163,12 +163,13 @@ fn levenshtein(a: &str, b: &str) -> usize {
 /// entities-slice-index map, and an entity-id -> relation-indices map (edges
 /// touching that entity as either source or target).
 ///
-/// Built fresh per traversal call rather than cached on `KnowledgeCache`:
-/// entities/relations are plain `pub` `Vec`s that get pushed to directly
-/// (e.g. `schema.rs`'s load path bypasses `add_entity`/`add_relation`), so a
-/// persistent index would need extra bookkeeping to avoid drifting stale. A
-/// one-off O(V+E) build per call is still a large win over the O(V·E) (BFS)
-/// / O(steps·active·E) (spreading activation) scans it replaces.
+/// Cached on `KnowledgeCache` and checked against a fingerprint of the graph
+/// on every use ([`graph_fingerprint`]). entities/relations are plain `pub`
+/// `Vec`s that get changed directly (e.g. `schema.rs`'s load path bypasses
+/// `add_entity`/`add_relation`), so the cache cannot rely on being told about
+/// changes; the fingerprint notices any of them. Rebuilding it on every
+/// traversal instead made a 2-hop BFS over 1K entities 6.5x slower than the
+/// scan it replaced (24 -> 155 µs; `BENCHMARKS.md`, "Knowledge Graph").
 struct AdjacencyIndex {
     entity_index: HashMap<u64, usize>,
     by_entity: HashMap<u64, Vec<usize>>,
@@ -204,6 +205,45 @@ impl AdjacencyIndex {
     }
 }
 
+/// A hash of everything [`AdjacencyIndex`] depends on — each entity's id and
+/// position, each relation's endpoints and position. One linear pass, no
+/// allocation: far cheaper than building the index, which hashes the same
+/// values into two maps.
+fn graph_fingerprint(entities: &[Entity], relations: &[Relation]) -> u64 {
+    // splitmix64-style mixing; order matters, so positions are covered.
+    fn mix(h: u64, v: u64) -> u64 {
+        let mut z = (h ^ v).wrapping_add(0x9E37_79B9_7F4A_7C15);
+        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+        z ^ (z >> 31)
+    }
+    let mut h = mix(entities.len() as u64, relations.len() as u64);
+    for e in entities {
+        h = mix(h, e.id);
+    }
+    for r in relations {
+        h = mix(mix(h, r.src), r.tgt);
+    }
+    h
+}
+
+/// The cached [`AdjacencyIndex`] and the fingerprint it was built for.
+/// Cloning a `KnowledgeCache` starts the clone with an empty cache.
+#[derive(Default)]
+struct AdjacencyCache(std::sync::Mutex<Option<(u64, std::sync::Arc<AdjacencyIndex>)>>);
+
+impl Clone for AdjacencyCache {
+    fn clone(&self) -> Self {
+        Self::default()
+    }
+}
+
+impl std::fmt::Debug for AdjacencyCache {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        f.write_str("AdjacencyCache")
+    }
+}
+
 // ---------------------------------------------------------------------------
 // KnowledgeCache
 // ---------------------------------------------------------------------------
@@ -216,6 +256,7 @@ pub struct KnowledgeCache {
     pub alias_strings: Vec<String>,
     pub alias_entity_ids: Vec<i64>,
     next_entity_id: u64,
+    adjacency: AdjacencyCache,
 }
 
 impl KnowledgeCache {
@@ -226,6 +267,7 @@ impl KnowledgeCache {
             alias_strings: Vec::new(),
             alias_entity_ids: Vec::new(),
             next_entity_id: 0,
+            adjacency: AdjacencyCache::default(),
         }
     }
 
@@ -236,9 +278,29 @@ impl KnowledgeCache {
             alias_strings: Vec::new(),
             alias_entity_ids: Vec::new(),
             next_entity_id: next_id,
+            adjacency: AdjacencyCache::default(),
         }
     }
 
+    /// The adjacency index for the graph as it is now: the cached one if the
+    /// graph's fingerprint still matches, otherwise rebuilt and cached.
+    fn adjacency_index(&self) -> std::sync::Arc<AdjacencyIndex> {
+        let fp = graph_fingerprint(&self.entities, &self.relations);
+        let mut slot = self
+            .adjacency
+            .0
+            .lock()
+            .unwrap_or_else(std::sync::PoisonError::into_inner);
+        if let Some((cached_fp, idx)) = slot.as_ref()
+            && *cached_fp == fp
+        {
+            return idx.clone();
+        }
+        let idx = std::sync::Arc::new(AdjacencyIndex::build(&self.entities, &self.relations));
+        *slot = Some((fp, idx.clone()));
+        idx
+    }
+
     // -----------------------------------------------------------------------
     // Entity management
     // -----------------------------------------------------------------------
@@ -397,7 +459,7 @@ impl KnowledgeCache {
     /// together with their discovered depth.  The seed entity itself is NOT
     /// included.  Traversal follows both outgoing and incoming relation edges.
     pub fn bfs_neighbors(&self, entity_id: u64, max_depth: usize) -> Vec<(Entity, usize)> {
-        let idx = AdjacencyIndex::build(&self.entities, &self.relations);
+        let idx = self.adjacency_index();
         let mut visited: HashSet<u64> = HashSet::new();
         let mut queue: VecDeque<(u64, usize)> = VecDeque::new();
         let mut results: Vec<(Entity, usize)> = Vec::new();
@@ -502,7 +564,7 @@ impl KnowledgeCache {
         min_activation: f32,
         max_steps: usize,
     ) -> Vec<(u64, f32)> {
-        let idx = AdjacencyIndex::build(&self.entities, &self.relations);
+        let idx = self.adjacency_index();
         let mut activation: HashMap<u64, f32> = HashMap::new();
 
         // Initialise seeds with activation 1.0.
@@ -631,6 +693,51 @@ impl Default for KnowledgeCache {
 mod tests {
     use super::*;
 
+    #[test]
+    fn cached_adjacency_sees_direct_changes_to_the_graph() {
+        // The index is cached across traversals, but entities/relations are
+        // pub Vecs anyone can edit; every kind of edit must be seen.
+        let mut kg = KnowledgeCache::new();
+        let a = kg.add_entity("a", "t", -1);
+        let b = kg.add_entity("b", "t", -1);
+        let c = kg.add_entity("c", "t", -1);
+        kg.add_relation(a, b, "r", 1.0);
+        let ids = |kg: &KnowledgeCache| -> Vec<u64> {
+            let mut v: Vec<u64> = kg.bfs_neighbors(a, 3).iter().map(|(e, _)| e.id).collect();
+            v.sort();
+            v
+        };
+        assert_eq!(ids(&kg), vec![b]);
+        assert_eq!(ids(&kg), vec![b], "cached index reused");
+
+        // Pushed directly, bypassing add_relation.
+        kg.relations.push(Relation {
+            src: b,
+            tgt: c,
+            ..Relation::default()
+        });
+        assert_eq!(ids(&kg), vec![b, c]);
+
+        // Rewired in place: same lengths, different edge.
+        kg.relations[1].tgt = a;
+        assert_eq!(ids(&kg), vec![b]);
+
+        // Removed and replaced: same lengths again.
+        kg.relations.pop();
+        kg.relations.push(Relation {
+            src: a,
+            tgt: c,
+            ..Relation::default()
+        });
+        assert_eq!(ids(&kg), vec![b, c]);
+        let act: Vec<u64> = kg
+            .spreading_activation(&[a], 0.5, 0.0, 2)
+            .iter()
+            .map(|(id, _)| *id)
+            .collect();
+        assert!(act.contains(&c));
+    }
+
     // -----------------------------------------------------------------------
     // Original tests — must remain passing
     // -----------------------------------------------------------------------

crates/clawhdf5-agent/src/lib.rs

@@ -1,4 +1,4 @@
-//! ZeroClaw agent memory HDF5 backend.
+//! Agent memory stored in a single HDF5 file.
 //!
 //! Provides persistent memory storage for AI agents using HDF5 files.
 //! All data is cached in-memory for fast access and flushed to disk
@@ -36,6 +36,7 @@ pub mod reranker;
 pub mod schema;
 pub mod search;
 pub mod session;
+pub mod signing;
 pub mod storage;
 mod store_lock;
 pub mod temporal;
@@ -63,25 +64,22 @@ use std::path::{Path, PathBuf};
 use cache::MemoryCache;
 #[cfg(feature = "hnsw")]
 use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};
+use clawhdf5_format::float16::round_to_f16;
 use ephemeral::{EphemeralConfig, EphemeralStore};
 
-/// HNSW construction parameters used for the agent's vector index. Cosine is the
-/// agent's similarity metric, so the index is built with cosine distance.
-#[cfg(feature = "hnsw")]
-const HNSW_M: usize = 16;
-#[cfg(feature = "hnsw")]
-const HNSW_EF_CONSTRUCTION: usize = 64;
 // EphemeralEntry and EphemeralStats are part of the crate public API via
 // the `ephemeral` module; they are not needed directly in lib.rs internals.
 #[allow(unused_imports)]
 pub use ephemeral::{EphemeralEntry, EphemeralStats};
 use knowledge::KnowledgeCache;
 use memory_strategy::{Exchange, MemoryStrategy, StrategyOutput};
-use session::SessionCache;
+pub use search::SearchOptions;
+pub use session::{SessionCache, SessionEntry};
 
 // --- Error type ---
 
 #[derive(Debug)]
+#[non_exhaustive]
 pub enum MemoryError {
     Io(std::io::Error),
     Hdf5(String),
@@ -89,6 +87,14 @@ pub enum MemoryError {
     NotFound(String),
     /// Another `HDF5Memory` (in this or another process) has the store open.
     Locked(String),
+    /// A record the store cannot hold as given, e.g. an embedding value
+    /// outside the half-precision range of a `float16` store.
+    InvalidEntry(String),
+    /// The store's checkpoints are signed and no signing key is set, so a
+    /// checkpoint would leave it unsigned. Set the key with
+    /// [`HDF5Memory::set_signing_key`], or drop the signature on purpose with
+    /// [`HDF5Memory::remove_signature`].
+    SigningKeyRequired(String),
 }
 
 impl std::fmt::Display for MemoryError {
@@ -99,6 +105,8 @@ impl std::fmt::Display for MemoryError {
             MemoryError::Schema(e) => write!(f, "schema error: {e}"),
             MemoryError::NotFound(e) => write!(f, "not found: {e}"),
             MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
+            MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
+            MemoryError::SigningKeyRequired(e) => write!(f, "signing key required: {e}"),
         }
     }
 }
@@ -130,6 +138,19 @@ pub struct MemoryConfig {
     pub embedding_dim: usize,
     pub chunk_size: usize,
     pub overlap: usize,
+    /// Store embeddings as IEEE half precision (numpy `float16`): half the
+    /// bytes of the embeddings dataset on disk. Every embedding is rounded to
+    /// the nearest half as it enters the store, in memory as well as on disk,
+    /// so search results are the same before and after a reopen. Values must
+    /// lie within ±65504; a save outside that is `MemoryError::InvalidEntry`.
+    /// Fixed when the store is created (persisted in `/meta`).
+    ///
+    /// **On by default for new stores**: on the full LongMemEval haystack with
+    /// real MiniLM embeddings every retrieval metric matched `f32`, and at
+    /// 100K records the file is 48% smaller (`BENCHMARKS.md`). Existing
+    /// stores keep the setting they were created with. Set it to `false` for
+    /// full-precision embeddings, e.g. for unnormalised vectors that may
+    /// exceed the half-precision range.
     pub float16: bool,
     pub compression: bool,
     pub compression_level: u32,
@@ -140,16 +161,39 @@ pub struct MemoryConfig {
     pub wal_enabled: bool,
     pub wal_max_entries: usize,
     /// Store the vector index's own copy of the embeddings as int8 rather than
-    /// f32, a quarter of the memory.
+    /// f32, a quarter of the memory. **On by default** for new stores.
     ///
     /// The index's copy is the single largest part of a loaded store's
     /// footprint. Quantised distances are approximate, so the candidate pool
     /// is re-scored against the cache's exact embeddings before fusion, which
-    /// restores recall; what it costs is throughput — roughly 13% of queries
-    /// per second and 16% of build time at 100K x 384. See `BENCHMARKS.md`.
+    /// holds recall at the f32 index's level. It is also faster, not slower:
+    /// at equal recall, 1.63x the queries per second on x86-64 (AVX2) and
+    /// 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with builds 1.8x and 2.3x
+    /// faster. See `BENCHMARKS.md`.
+    ///
+    /// Persisted with the store. Stores written before this setting existed
+    /// have no stored value and open as `false`, so reopening an old store
+    /// never changes how its index is held.
     ///
     /// Has no effect without the `hnsw` feature.
     pub quantized_index: bool,
+    /// HNSW graph degree. Higher means a denser graph: better recall, more
+    /// memory and slower builds. Clamped to at least 2 when the index is
+    /// built, since a graph with fewer connections is not one.
+    ///
+    /// Has no effect without the `hnsw` feature.
+    pub hnsw_m: usize,
+    /// Candidate list size while building the HNSW graph. Higher means a
+    /// better graph and a slower build; it does not affect query cost.
+    ///
+    /// Has no effect without the `hnsw` feature.
+    pub hnsw_ef_construction: usize,
+    /// Candidate list size for a query, trading throughput for recall. `0`
+    /// keeps the default, which scales with the requested `k`
+    /// (`max(k * 8, 64)`) so that fusion still sees a useful pool.
+    ///
+    /// Has no effect without the `hnsw` feature.
+    pub hnsw_ef_search: usize,
 }
 
 impl MemoryConfig {
@@ -162,7 +206,7 @@ impl MemoryConfig {
             embedding_dim,
             chunk_size: 512,
             overlap: 50,
-            float16: false,
+            float16: true,
             compression: false,
             compression_level: 0,
             compact_threshold: 0.3,
@@ -171,7 +215,10 @@ impl MemoryConfig {
             created_at,
             wal_enabled: true,
             wal_max_entries: 500,
-            quantized_index: false,
+            quantized_index: true,
+            hnsw_m: 16,
+            hnsw_ef_construction: 64,
+            hnsw_ef_search: 0,
         }
     }
 }
@@ -278,6 +325,12 @@ pub struct HDF5Memory {
     activations_dirty: bool,
     /// Opened with [`HDF5Memory::open_read_only`]: nothing may reach the disk.
     read_only: bool,
+    /// Key that signs every checkpoint; never persisted. See
+    /// [`HDF5Memory::set_signing_key`].
+    signing_key: Option<signing::SigningKey>,
+    /// Checkpoints of this store are signed: the file on disk is, or a key
+    /// has been set. A checkpoint without a key is then refused.
+    signed: bool,
     /// A WAL that `open()` could not read and moved aside; see
     /// [`HDF5Memory::quarantined_wal`].
     quarantined_wal: Option<PathBuf>,
@@ -297,7 +350,8 @@ impl HDF5Memory {
     /// Create a new HDF5 memory file with the given configuration.
     pub fn create(config: MemoryConfig) -> Result<Self> {
         let lock = store_lock::StoreLock::acquire(&config.path)?;
-        let cache = MemoryCache::new(config.embedding_dim);
+        let mut cache = MemoryCache::new(config.embedding_dim);
+        cache.set_half_precision(config.float16);
         let sessions = SessionCache::new();
         let knowledge = KnowledgeCache::new();
 
@@ -332,6 +386,8 @@ impl HDF5Memory {
             bm25_filter: bm25::TokenFilter::default(),
             activations_dirty: false,
             read_only: false,
+            signing_key: None,
+            signed: false,
             quarantined_wal: None,
             _lock: Some(lock),
         })
@@ -510,6 +566,8 @@ impl HDF5Memory {
             bm25_filter: bm25::TokenFilter::default(),
             activations_dirty: false,
             read_only,
+            signing_key: None,
+            signed: checkpoint.signed,
             quarantined_wal,
             _lock: lock,
         })
@@ -670,6 +728,39 @@ impl HDF5Memory {
         }
     }
 
+    /// Sign every checkpoint from now on with `key` (Ed25519). The key is
+    /// never written anywhere; set it again after every `open`. Once a store
+    /// is signed, a checkpoint without the key is refused
+    /// ([`MemoryError::SigningKeyRequired`]) rather than silently leaving it
+    /// unsigned. Setting a different key re-signs the store under that key
+    /// from the next checkpoint; a verifier trusting the old key will then
+    /// reject it, which is the point. Call [`AgentMemory::flush_wal`] to sign
+    /// right away.
+    pub fn set_signing_key(&mut self, key: signing::SigningKey) {
+        self.signing_key = Some(key);
+        self.signed = true;
+    }
+
+    /// Stop signing: the next checkpoint writes the store unsigned. The
+    /// deliberate way out of [`MemoryError::SigningKeyRequired`].
+    pub fn remove_signature(&mut self) {
+        self.signing_key = None;
+        self.signed = false;
+    }
+
+    /// Checkpoints of this store are signed (on disk, or from the next
+    /// checkpoint because a key has been set).
+    pub fn is_signed(&self) -> bool {
+        self.signed
+    }
+
+    /// Check the checkpoint at `path` against the public key the caller
+    /// trusts; see [`signing::verify_store`]. Reads the file only: it works
+    /// on a store another process has open.
+    pub fn verify(path: &Path, trusted: &signing::VerifyingKey) -> Result<signing::VerifyReport> {
+        signing::verify_store(path, trusted)
+    }
+
     /// Flush current state to disk and truncate the WAL.
     ///
     /// Every code path that persists the full cache to the .h5 file must
@@ -685,10 +776,28 @@ impl HDF5Memory {
         // Record which WAL prefix this checkpoint contains, so a crash before
         // the truncate below can't replay those entries a second time.
         let wal_applied = self.wal.as_ref().map(|w| w.mark());
+        let signature = match &self.signing_key {
+            Some(key) => Some(signing::sign(
+                key,
+                &self.config,
+                &self.cache,
+                &self.sessions,
+                &self.knowledge,
+                wal_applied,
+            )),
+            None if self.signed => {
+                return Err(MemoryError::SigningKeyRequired(format!(
+                    "{} is signed; set its signing key before a checkpoint \
+                     (saves so far are held in the WAL or in memory)",
+                    self.config.path.display()
+                )));
+            }
+            None => None,
+        };
         // Written before the .h5 so a crash in between leaves a sidecar whose
         // generation matches no checkpoint (ignored), never the reverse.
         let ann_generation = self.persist_vector_index();
-        storage::write_to_disk_with_meta(
+        storage::write_to_disk_signed(
             &self.config.path,
             &self.config,
             &self.cache,
@@ -697,7 +806,9 @@ impl HDF5Memory {
             &schema::CheckpointMeta {
                 wal_applied,
                 ann_generation,
+                signed: signature.is_some(),
             },
+            signature.as_ref(),
         )?;
         if let Some(ref mut w) = self.wal {
             w.truncate()?;
@@ -826,6 +937,32 @@ impl HDF5Memory {
     // the index length drifts from the cache length (covering any mutation path
     // that doesn't call a hook, e.g. consolidation pushes).
 
+    /// Graph degree for the index, never below the 2 the builder requires:
+    /// a config value of 0 or 1 would otherwise panic inside `clawhdf5-ann`.
+    #[cfg(feature = "hnsw")]
+    fn hnsw_m(&self) -> usize {
+        self.config.hnsw_m.max(2)
+    }
+
+    /// Build-time candidate list size, never below the graph degree — a
+    /// smaller one cannot fill a node's connections.
+    #[cfg(feature = "hnsw")]
+    fn hnsw_ef_construction(&self) -> usize {
+        self.config.hnsw_ef_construction.max(self.hnsw_m())
+    }
+
+    /// Query-time candidate list size for a `k`-result search. `0` means the
+    /// default, which scales with `k`.
+    #[cfg(feature = "hnsw")]
+    pub(crate) fn hnsw_ef_search(&self, k: usize) -> usize {
+        let default = (k * 8).max(64);
+        if self.config.hnsw_ef_search == 0 {
+            default
+        } else {
+            self.config.hnsw_ef_search.max(k)
+        }
+    }
+
     /// How the index should store its copy of the vectors, per the config.
     #[cfg(feature = "hnsw")]
     fn index_storage(&self) -> Storage {
@@ -856,8 +993,8 @@ impl HDF5Memory {
         let rows: Vec<Vec<f32>> = self.cache.embeddings.iter().map(<[f32]>::to_vec).collect();
         let mut index = HnswIndex::build_with(
             &rows,
-            HNSW_M,
-            HNSW_EF_CONSTRUCTION,
+            self.hnsw_m(),
+            self.hnsw_ef_construction(),
             DistanceMetric::Cosine,
             self.index_storage(),
         );
@@ -962,6 +1099,18 @@ impl HDF5Memory {
         &self.config
     }
 
+    /// The sessions recorded in this store.
+    pub fn sessions(&self) -> &SessionCache {
+        &self.sessions
+    }
+
+    /// Mutable access to the sessions, e.g. to add many at once. Changes
+    /// reach the disk at the next checkpoint (any flushing call, such as
+    /// [`HDF5Memory::flush_wal`] or `save_batch`), not immediately.
+    pub fn sessions_mut(&mut self) -> &mut SessionCache {
+        &mut self.sessions
+    }
+
     /// Get a reference to the knowledge cache.
     pub fn knowledge(&self) -> &KnowledgeCache {
         &self.knowledge
@@ -1024,7 +1173,29 @@ impl HDF5Memory {
     /// Upsert: if an active entry with the same tags (key) exists, update it in-place.
     /// Otherwise append a new entry. Use this for key-based memory stores where
     /// the same key should not create duplicates.
+    /// A `float16` store holds embeddings as IEEE half precision, which has no
+    /// finite value beyond ±65504. Refuse such an embedding rather than
+    /// silently store infinity. (Values that are already infinite or NaN are
+    /// stored as they are, as in an `f32` store.)
+    fn check_embedding(&self, embedding: &[f32]) -> Result<()> {
+        if !self.config.float16 {
+            return Ok(());
+        }
+        let overflow = embedding
+            .iter()
+            .enumerate()
+            .find(|&(_, &v)| v.is_finite() && round_to_f16(v).is_infinite());
+        match overflow {
+            None => Ok(()),
+            Some((i, v)) => Err(MemoryError::InvalidEntry(format!(
+                "embedding[{i}] = {v} is outside the half-precision range (±65504) \
+                 of this float16 store"
+            ))),
+        }
+    }
+
     pub fn save_or_update(&mut self, entry: MemoryEntry) -> Result<usize> {
+        self.check_embedding(&entry.embedding)?;
         if let Some(existing_idx) = self.cache.find_by_tags(&entry.tags) {
             if let Some(ref mut w) = self.wal {
                 let wal_entry = wal::WalEntry {
@@ -1080,6 +1251,7 @@ impl HDF5Memory {
 
 impl AgentMemory for HDF5Memory {
     fn save(&mut self, entry: MemoryEntry) -> Result<usize> {
+        self.check_embedding(&entry.embedding)?;
         if let Some(ref mut w) = self.wal {
             let wal_entry = wal::WalEntry {
                 entry_type: wal::WalEntryType::Save,
@@ -1121,6 +1293,10 @@ impl AgentMemory for HDF5Memory {
     }
 
     fn save_batch(&mut self, entries: Vec<MemoryEntry>) -> Result<Vec<usize>> {
+        // All or nothing: check every entry before storing any.
+        for entry in &entries {
+            self.check_embedding(&entry.embedding)?;
+        }
         let mut indices = Vec::with_capacity(entries.len());
         for entry in entries {
             let idx = self.cache.push(
@@ -1281,6 +1457,9 @@ impl HDF5Memory {
         })?;
         let view = memory_strategy::CacheStoreView::new(&self.cache, &self.knowledge);
         let output = strat.evaluate(&exchange, &view);
+        for e in &output.entries {
+            self.check_embedding(&e.embedding)?;
+        }
         for e in &output.entries {
             self.cache.push(
                 e.chunk.clone(),
@@ -1305,6 +1484,35 @@ impl HDF5Memory {
 }
 
 impl HDF5Memory {
+    /// Delete many records with a single checkpoint, where
+    /// [`AgentMemory::delete`] checkpoints once per record.
+    ///
+    /// All or nothing: if any id is out of range or already deleted (or
+    /// repeated), nothing is deleted and `MemoryError::NotFound` is returned.
+    /// Unlike `delete`, this never auto-compacts, so the records stay in the
+    /// store as tombstones (their indices unchanged) until [`AgentMemory::compact`]
+    /// is called — importers use it to carry over records that were already
+    /// deleted in the source.
+    pub fn delete_batch(&mut self, ids: &[usize]) -> Result<()> {
+        let mut seen = std::collections::HashSet::with_capacity(ids.len());
+        for &id in ids {
+            if self.cache.tombstones.get(id).copied() != Some(0) || !seen.insert(id) {
+                return Err(MemoryError::NotFound(format!(
+                    "entry {id} not found or already deleted"
+                )));
+            }
+        }
+        if ids.is_empty() {
+            return Ok(());
+        }
+        for &id in ids {
+            self.cache.mark_deleted(id);
+            self.hnsw_on_delete(id);
+            self.bm25_on_delete(id);
+        }
+        self.flush()
+    }
+
     pub fn tick_session(&mut self) -> Result<()> {
         let d = self.config.decay_factor;
         for w in self.cache.activation_weights.iter_mut() {
@@ -1365,6 +1573,16 @@ impl HDF5Memory {
         let mut promoted = 0;
 
         for key in candidates {
+            // Check before taking, so a rejected entry stays in the ephemeral
+            // tier rather than being lost.
+            if let Some(emb) = self
+                .ephemeral
+                .as_ref()
+                .and_then(|s| s.get_entry(&key))
+                .and_then(|e| e.embedding.as_deref())
+            {
+                self.check_embedding(emb)?;
+            }
             let entry = match self
                 .ephemeral
                 .as_mut()
@@ -1493,6 +1711,79 @@ mod tests {
         }
     }
 
+    #[test]
+    fn delete_batch_tombstones_without_compacting() {
+        let dir = TempDir::new().unwrap();
+        let path = dir.path().join("test.h5");
+        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
+        mem.save_batch(
+            (0..4)
+                .map(|i| make_entry(&format!("record {i}"), &[i as f32, 1.0, 0.0, 0.0]))
+                .collect(),
+        )
+        .unwrap();
+        // 3 of 4 is far past compact_threshold (0.3): delete() would compact.
+        mem.delete_batch(&[0, 1, 3]).unwrap();
+        assert_eq!(mem.count(), 4);
+        assert_eq!(mem.count_active(), 1);
+        drop(mem);
+
+        let mut mem = HDF5Memory::open(&path).unwrap();
+        assert_eq!(mem.cache.tombstones, vec![1, 1, 0, 1]);
+        let hits = mem.hybrid_search(&[0.0, 1.0, 0.0, 0.0], "record", 0.5, 0.5, 10);
+        assert!(
+            hits.iter().all(|r| r.index == 2),
+            "tombstoned record returned"
+        );
+    }
+
+    #[test]
+    fn delete_batch_is_all_or_nothing() {
+        let dir = TempDir::new().unwrap();
+        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
+        mem.save_batch(vec![
+            make_entry("a", &[1.0, 0.0, 0.0, 0.0]),
+            make_entry("b", &[0.0, 1.0, 0.0, 0.0]),
+        ])
+        .unwrap();
+        for bad in [&[0, 5][..], &[1, 1][..]] {
+            assert!(matches!(
+                mem.delete_batch(bad),
+                Err(MemoryError::NotFound(_))
+            ));
+            assert_eq!(mem.count_active(), 2, "{bad:?} deleted something");
+        }
+        mem.delete_batch(&[]).unwrap();
+        assert_eq!(mem.count_active(), 2);
+    }
+
+    #[test]
+    fn sessions_mut_add_at_keeps_timestamp_across_reopen() {
+        let dir = TempDir::new().unwrap();
+        let path = dir.path().join("test.h5");
+        let mut mem = HDF5Memory::create(make_config(&dir)).unwrap();
+        mem.sessions_mut()
+            .add_at("s-old", 2, 7, "discord", "old summary", 1.7e15);
+        mem.flush_wal().unwrap();
+        drop(mem);
+
+        let mem = HDF5Memory::open_read_only(&path).unwrap();
+        let s = mem.sessions();
+        assert_eq!(s.len(), 1);
+        let e = &s.entries[0];
+        assert_eq!(
+            (
+                e.id.as_str(),
+                e.start_idx,
+                e.end_idx,
+                e.channel.as_str(),
+                e.ts
+            ),
+            ("s-old", 2, 7, "discord", 1.7e15)
+        );
+        assert_eq!(s.summaries[0], "old summary");
+    }
+
     #[test]
     fn create_new_file() {
         let dir = TempDir::new().unwrap();

crates/clawhdf5-agent/src/openclaw.rs

@@ -1,10 +1,12 @@
-//! OpenClaw Integration Layer.
+//! A Markdown-oriented memory backend over [`crate::HDF5Memory`].
 //!
-//! Bridge between OpenClaw agent gateway (Markdown + sqlite-vec) and the
-//! clawhdf5 HDF5-backed memory backend.  Provides:
+//! Named for OpenClaw, whose workspace memory is Markdown, but **not an
+//! OpenClaw plugin**: nothing here registers with OpenClaw, and the
+//! integration it was written for never worked (see `docs/openclaw.md`).
+//! Provides:
 //!
-//! - [`MemoryBackend`] — the trait OpenClaw implements against.
-//! - [`ClawhdfBackend`] — concrete HDF5-backed implementation.
+//! - [`MemoryBackend`] — search / read back / write / ingest / export.
+//! - [`ClawhdfBackend`] — the HDF5-backed implementation.
 //! - [`MarkdownParser`] — splits Markdown into [`MarkdownSection`] records.
 //! - [`MarkdownExporter`] — renders sections back to Markdown text.
 
@@ -13,9 +15,8 @@ use std::path::{Path, PathBuf};
 use std::time::{SystemTime, UNIX_EPOCH};
 
 use crate::{
-    AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry,
-    confidence::{ConfidenceConfig, ScoredResult, reject_low_confidence},
-    reranker::{ReRankConfig, RerankInput, rerank},
+    AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions,
+    confidence::ConfidenceConfig, reranker::ReRankConfig,
 };
 
 // ─────────────────────────────────────────────────────────────────────────────
@@ -62,7 +63,8 @@ pub struct BackendStats {
 // MemoryBackend trait
 // ─────────────────────────────────────────────────────────────────────────────
 
-/// Interface that OpenClaw uses to interact with a memory backend.
+/// A Markdown-oriented memory backend: search, read back by path, write,
+/// ingest and export.
 ///
 /// Implementors provide persistent storage, full-text + vector search,
 /// Markdown ingestion / export, and statistics.
@@ -319,7 +321,7 @@ impl MarkdownExporter {
 ///
 /// # Path mapping
 ///
-/// OpenClaw addresses memories by file path (e.g. `"memory/user.md"`).
+/// Memories are addressed by file path (e.g. `"memory/user.md"`).
 /// Internally every [`MemoryEntry`] stores the originating path as its
 /// `source_channel`.  Section sub-paths are stored as
 /// `"<path>::<heading>"`.
@@ -422,7 +424,7 @@ impl ClawhdfBackend {
 
     // ── Compaction & Consolidation hooks (7.6) ────────────────────────────
 
-    /// Run a compaction cycle — called by OpenClaw during session compaction.
+    /// Run a compaction cycle (decay, compaction, WAL flush).
     ///
     /// Sequence:
     /// 1. `tick_session()` — apply Hebbian decay to all activation weights.
@@ -524,71 +526,27 @@ impl ClawhdfBackend {
 
 impl MemoryBackend for ClawhdfBackend {
     /// Search using hybrid vector + BM25 retrieval, then re-rank and
-    /// confidence-filter.
+    /// confidence-filter — [`HDF5Memory::search`] with both stages on.
     fn search(
         &mut self,
         query_text: &str,
         query_embedding: &[f32],
         k: usize,
     ) -> Vec<MemorySearchResult> {
-        // 1. Hybrid retrieval (vector + BM25, fused by score).
-        let candidates = k.saturating_mul(3).max(10);
-        let raw = self.memory.hybrid_search_with(
-            query_embedding,
-            query_text,
-            crate::hybrid::DEFAULT_FUSION,
-            candidates,
-        );
-
-        if raw.is_empty() {
-            return Vec::new();
-        }
-
-        let now = Self::now_secs();
-
-        // 2. Re-rank using temporal recency, source authority, Hebbian weight.
-        let rerank_inputs: Vec<RerankInput> = raw
-            .iter()
-            .map(|r| RerankInput {
-                index: r.index,
-                timestamp: r.timestamp,
-                source_channel: r.source_channel.clone(),
-                raw_activation: r.activation,
-                relevance: r.score,
-            })
-            .collect();
-
-        let reranked = rerank(&rerank_inputs, &self.rerank_config, now);
-
-        // 3. Confidence rejection.
-        let scored: Vec<ScoredResult> = reranked
-            .iter()
-            .map(|r| ScoredResult {
-                index: r.index,
-                score: r.combined_score,
-            })
-            .collect();
-
-        let confident = reject_low_confidence(&scored, &self.confidence_config);
-
-        // 4. Map back to MemorySearchResult; preserve raw text via index lookup.
-        let raw_by_idx: HashMap<usize, &crate::SearchResult> =
-            raw.iter().map(|r| (r.index, r)).collect();
-
-        confident
+        let options = SearchOptions::new(k)
+            .with_rerank(self.rerank_config)
+            .with_confidence(self.confidence_config.clone())
+            .at_time(Self::now_secs());
+        self.memory
+            .search(query_embedding, query_text, &options)
             .into_iter()
-            .take(k)
-            .filter_map(|sr| {
-                let r = raw_by_idx.get(&sr.index)?;
-                let path = r.source_channel.clone();
-                Some(MemorySearchResult {
-                    text: r.chunk.clone(),
-                    score: sr.score,
-                    path: path.clone(),
+            .map(|r| MemorySearchResult {
+                text: r.chunk,
+                score: r.score,
+                path: r.source_channel.clone(),
                 line_range: None,
                 timestamp: Some(r.timestamp),
-                    source: path,
-                })
+                source: r.source_channel,
             })
             .collect()
     }

crates/clawhdf5-agent/src/provenance.rs

@@ -2,7 +2,7 @@
 //!
 //! Records the origin, authorship, and a content hash of every memory chunk
 //! so the system can detect *accidental* corruption and trace data lineage.
-//! The hash is unkeyed (see [`fnv1a_64`]) — this is not a tamper-evidence or
+//! The hash is unkeyed (FNV-1a) — this is not a tamper-evidence or
 //! authenticity guarantee.
 
 use std::collections::HashMap;

crates/clawhdf5-agent/src/schema.rs

@@ -15,6 +15,9 @@ use crate::session::SessionCache;
 use crate::wal::WalMark;
 
 pub const SCHEMA_VERSION: &str = "1.0";
+/// Writer-version tag stored in `/meta` as `edgehdf5_version`. Kept for file
+/// compatibility; despite the name it has nothing to do with ZeroClaw, which
+/// does not use clawhdf5.
 pub const ZEROCLAW_VERSION: &str = "0.8.0";
 
 /// `/meta` attributes holding the [`WalMark`] of the WAL prefix already folded
@@ -23,6 +26,7 @@ pub const ZEROCLAW_VERSION: &str = "0.8.0";
 const WAL_APPLIED_LEN_ATTR: &str = "wal_applied_len";
 const WAL_APPLIED_CRC_ATTR: &str = "wal_applied_crc";
 const ANN_GENERATION_ATTR: &str = "ann_generation";
+const SIG_VERSION_ATTR: &str = "sig_version";
 
 /// Build a complete HDF5 file from the in-memory state.
 pub fn build_hdf5_file(
@@ -46,7 +50,7 @@ pub fn build_hdf5_file_with_mark(
 ) -> Result<Vec<u8>, MemoryError> {
     let meta = CheckpointMeta {
         wal_applied,
-        ann_generation: None,
+        ..CheckpointMeta::default()
     };
     build_hdf5_file_with_meta(config, cache, sessions, knowledge, &meta)
 }
@@ -61,6 +65,10 @@ pub struct CheckpointMeta {
     /// one left over from another checkpoint can never be attached to records
     /// it wasn't built from.
     pub ann_generation: Option<u64>,
+    /// The checkpoint carries an Ed25519 signature (see [`crate::signing`]).
+    /// Read-only: whether a checkpoint is *written* signed is decided by the
+    /// signature passed to [`build_hdf5_file_signed`].
+    pub signed: bool,
 }
 
 /// [`build_hdf5_file`] with checkpoint bookkeeping.
@@ -70,6 +78,19 @@ pub fn build_hdf5_file_with_meta(
     sessions: &SessionCache,
     knowledge: &KnowledgeCache,
     checkpoint: &CheckpointMeta,
+) -> Result<Vec<u8>, MemoryError> {
+    build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, None)
+}
+
+/// [`build_hdf5_file_with_meta`], plus a signed manifest of the contents
+/// (see [`crate::signing`]).
+pub fn build_hdf5_file_signed(
+    config: &MemoryConfig,
+    cache: &MemoryCache,
+    sessions: &SessionCache,
+    knowledge: &KnowledgeCache,
+    checkpoint: &CheckpointMeta,
+    signature: Option<&crate::signing::StoredSignature>,
 ) -> Result<Vec<u8>, MemoryError> {
     let wal_applied = checkpoint.wal_applied;
     let mut builder = clawhdf5::FileBuilder::new();
@@ -108,6 +129,15 @@ pub fn build_hdf5_file_with_meta(
         "quantized_index",
         AttrValue::I64(config.quantized_index.into()),
     );
+    meta.set_attr("hnsw_m", AttrValue::I64(config.hnsw_m as i64));
+    meta.set_attr(
+        "hnsw_ef_construction",
+        AttrValue::I64(config.hnsw_ef_construction as i64),
+    );
+    meta.set_attr(
+        "hnsw_ef_search",
+        AttrValue::I64(config.hnsw_ef_search as i64),
+    );
     meta.set_attr(
         "edgehdf5_version",
         AttrValue::String(ZEROCLAW_VERSION.into()),
@@ -121,11 +151,42 @@ pub fn build_hdf5_file_with_meta(
         // round trip through every reader.
         meta.set_attr(ANN_GENERATION_ATTR, AttrValue::I64(generation as i64));
     }
+    if let Some(sig) = signature {
+        use crate::signing::to_hex;
+        let m = &sig.manifest;
+        meta.set_attr(
+            SIG_VERSION_ATTR,
+            AttrValue::I64(crate::signing::MANIFEST_VERSION),
+        );
+        meta.set_attr("sig_algorithm", AttrValue::String("ed25519".into()));
+        meta.set_attr("sig_public_key", AttrValue::String(to_hex(&sig.public_key)));
+        meta.set_attr("sig_signature", AttrValue::String(to_hex(&sig.signature)));
+        meta.set_attr("sig_record_count", AttrValue::I64(m.record_count as i64));
+        meta.set_attr(
+            "sig_records_root",
+            AttrValue::String(to_hex(&m.records_root)),
+        );
+        meta.set_attr("sig_settings", AttrValue::String(to_hex(&m.settings)));
+        meta.set_attr("sig_sessions", AttrValue::String(to_hex(&m.sessions)));
+        meta.set_attr("sig_graph", AttrValue::String(to_hex(&m.graph)));
+    }
     // Need at least one dataset in the group for it to be a proper group
     meta.create_dataset("_marker").with_u8_data(&[1]).compact();
     let finished_meta = meta.finish();
     builder.add_group(finished_meta);
 
+    // /integrity: the signed per-record hashes, so verification can say
+    // which records changed.
+    if let Some(sig) = signature {
+        let mut group = builder.create_group("integrity");
+        let flat: Vec<u8> = sig.record_hashes.iter().flatten().copied().collect();
+        group
+            .create_dataset("record_hashes")
+            .with_u8_data(&flat)
+            .with_shape(&[sig.record_hashes.len() as u64, 32]);
+        builder.add_group(group.finish());
+    }
+
     // /memory group
     build_memory_group(&mut builder, config, cache)?;
 
@@ -150,20 +211,27 @@ fn build_memory_group(
     // chunks: fixed-length string array
     write_string_dataset(&mut group, "chunks", &cache.chunks);
 
-    // embeddings: f32 [N x D]
+    // embeddings: [N x D], f32 — or IEEE half precision for a `float16`
+    // store. The cache already holds half-rounded values then, so this
+    // conversion is exact and a reopened store sees the same numbers.
     let n = cache.embeddings.len() as u64;
     let d = cache.embedding_dim as u64;
     let flat = cache.flat_embeddings();
     {
-        let ds = group
-            .create_dataset("embeddings")
-            .with_f32_data(flat)
-            .with_shape(&[n, d]);
+        let ds = group.create_dataset("embeddings");
+        let elem_bytes: u64 = if config.float16 {
+            ds.with_f16_data(flat);
+            2
+        } else {
+            ds.with_f32_data(flat);
+            4
+        };
+        ds.with_shape(&[n, d]);
 
         // Chunk size tuning: target ~256KB per chunk for optimal I/O
         if n > 0 && d > 0 {
             let target_chunk_bytes: u64 = 256 * 1024;
-            let rows_per_chunk = (target_chunk_bytes / (d * 4)).max(1).min(n);
+            let rows_per_chunk = (target_chunk_bytes / (d * elem_bytes)).max(1).min(n);
             ds.with_chunks(&[rows_per_chunk, d]);
 
             // Compression. Shuffle is applied automatically (auto-shuffle
@@ -424,6 +492,64 @@ pub fn read_wal_mark(file: &clawhdf5::File) -> Option<WalMark> {
     Some(WalMark { len, crc })
 }
 
+/// Read a checkpoint's signature, if it has one. A signature whose
+/// attributes are present but malformed is an error, not "unsigned".
+pub fn read_signature(
+    file: &clawhdf5::File,
+) -> Result<Option<crate::signing::StoredSignature>, MemoryError> {
+    use crate::signing::{Manifest, StoredSignature, from_hex};
+    let attrs = file
+        .group("meta")
+        .and_then(|g| g.attrs())
+        .map_err(|e| MemoryError::Schema(format!("cannot read /meta attrs: {e}")))?;
+    let version = match attrs.get(SIG_VERSION_ATTR) {
+        None => return Ok(None),
+        Some(AttrValue::I64(v)) => *v,
+        Some(_) => return Err(MemoryError::Schema("malformed sig_version".into())),
+    };
+    if version != crate::signing::MANIFEST_VERSION {
+        return Err(MemoryError::Schema(format!(
+            "unsupported signature version {version}"
+        )));
+    }
+    fn hex<const N: usize>(
+        attrs: &std::collections::HashMap<String, AttrValue>,
+        name: &str,
+    ) -> Result<[u8; N], MemoryError> {
+        match attrs.get(name) {
+            Some(AttrValue::String(s)) => from_hex::<N>(s),
+            _ => None,
+        }
+        .ok_or_else(|| MemoryError::Schema(format!("malformed or missing {name}")))
+    }
+    let record_count = match attrs.get("sig_record_count") {
+        Some(AttrValue::I64(v)) if *v >= 0 => *v as u64,
+        _ => return Err(MemoryError::Schema("malformed sig_record_count".into())),
+    };
+    let group = file
+        .group("integrity")
+        .map_err(|e| MemoryError::Schema(format!("signed checkpoint without /integrity: {e}")))?;
+    let flat = read_u8_dataset(&group, "record_hashes")?;
+    if flat.len() % 32 != 0 {
+        return Err(MemoryError::Schema(
+            "/integrity/record_hashes is not a whole number of hashes".into(),
+        ));
+    }
+    let record_hashes = flat.as_chunks::<32>().0.to_vec();
+    Ok(Some(StoredSignature {
+        manifest: Manifest {
+            record_count,
+            records_root: hex::<32>(&attrs, "sig_records_root")?,
+            settings: hex::<32>(&attrs, "sig_settings")?,
+            sessions: hex::<32>(&attrs, "sig_sessions")?,
+            graph: hex::<32>(&attrs, "sig_graph")?,
+        },
+        record_hashes,
+        public_key: hex::<32>(&attrs, "sig_public_key")?,
+        signature: hex::<64>(&attrs, "sig_signature")?,
+    }))
+}
+
 /// Read the checkpoint bookkeeping from `/meta`.
 pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
     let ann_generation = file
@@ -434,9 +560,14 @@ pub fn read_checkpoint_meta(file: &clawhdf5::File) -> CheckpointMeta {
             Some(AttrValue::I64(v)) => Some(*v as u64),
             _ => None,
         });
+    let signed = file
+        .group("meta")
+        .and_then(|g| g.attrs())
+        .is_ok_and(|attrs| attrs.contains_key(SIG_VERSION_ATTR));
     CheckpointMeta {
         wal_applied: read_wal_mark(file),
         ann_generation,
+        signed,
     }
 }
 
@@ -488,11 +619,32 @@ pub fn validate_and_load(
         wal_max_entries: optional_i64_attr(&attrs, "wal_max_entries")
             .and_then(|v| usize::try_from(v).ok())
             .unwrap_or(500),
+        // `false`, not the new-store default: a store written before this
+        // setting existed was built with an f32 index, and reopening it must
+        // not silently change that.
         quantized_index: optional_bool_attr(&attrs, "quantized_index", false),
+        hnsw_m: optional_i64_attr(&attrs, "hnsw_m")
+            .and_then(|v| usize::try_from(v).ok())
+            .unwrap_or(16),
+        hnsw_ef_construction: optional_i64_attr(&attrs, "hnsw_ef_construction")
+            .and_then(|v| usize::try_from(v).ok())
+            .unwrap_or(64),
+        hnsw_ef_search: optional_i64_attr(&attrs, "hnsw_ef_search")
+            .and_then(|v| usize::try_from(v).ok())
+            .unwrap_or(0),
     };
 
     // Load /memory group
-    let memory_cache = load_memory_group(file, embedding_dim)?;
+    let mut memory_cache = load_memory_group(file, embedding_dim)?;
+    // A float16 store's cache holds half-rounded embeddings. Embeddings read
+    // from an f16 dataset already are; a float16 store whose last checkpoint
+    // predates half-precision storage is still f32 on disk and is rounded
+    // here.
+    if config.float16 && embeddings_are_f16(file) {
+        memory_cache.half_precision = true;
+    } else {
+        memory_cache.set_half_precision(config.float16);
+    }
 
     // Load /sessions group
     let session_cache = load_sessions_group(file)?;
@@ -735,6 +887,13 @@ fn read_string_dataset_from_group(
         .map_err(|e| MemoryError::Hdf5(format!("cannot read strings from {name}: {e}")))
 }
 
+/// Whether `/memory/embeddings` is stored as IEEE half precision.
+fn embeddings_are_f16(file: &clawhdf5::File) -> bool {
+    file.dataset("memory/embeddings")
+        .and_then(|ds| ds.dtype())
+        .is_ok_and(|dt| matches!(dt, clawhdf5::DType::Other(ref s) if s == "float16"))
+}
+
 fn read_f32_dataset(group: &clawhdf5::Group<'_>, name: &str) -> Result<Vec<f32>, MemoryError> {
     let ds = group
         .dataset(name)

crates/clawhdf5-agent/src/search.rs

@@ -2,18 +2,107 @@
 
 use std::path::Path;
 
+use std::collections::HashSet;
+
 use crate::bm25;
+use crate::confidence::{ConfidenceConfig, ScoredResult, reject_low_confidence};
 use crate::hybrid;
+use crate::reranker::{ReRankConfig, RerankInput, rerank};
 use crate::{HDF5Memory, MAX_ACTIVATION_WEIGHT, MemoryError, Result, SearchResult};
 
+/// Options for [`HDF5Memory::search`].
+///
+/// [`SearchOptions::new`] is plain hybrid search with the tuned default
+/// fusion — the same as `hybrid_search_with(.., hybrid::DEFAULT_FUSION, k)`.
+/// Every stage beyond that is opt-in.
+#[derive(Debug, Clone)]
+pub struct SearchOptions {
+    /// Number of results to return.
+    pub k: usize,
+    /// How the vector and keyword stages are combined.
+    pub fusion: hybrid::Fusion,
+    /// Only consider records whose `source_channel` is one of these. The
+    /// filter applies *before* ranking, so a filtered search still returns up
+    /// to `k` results and scores are normalised over the records it can
+    /// return. `None` searches everything; an empty list matches nothing.
+    pub source_channels: Option<Vec<String>>,
+    /// Re-rank a candidate pool by retrieval relevance, recency, source
+    /// authority and activation — the pipeline the OpenClaw backend runs.
+    pub rerank: Option<ReRankConfig>,
+    /// Candidates retrieved for re-ranking; 0 means `max(3k, 10)`.
+    pub rerank_pool: usize,
+    /// Drop low-confidence results (after re-ranking, when that is on).
+    pub confidence: Option<ConfidenceConfig>,
+    /// The time recency is measured from, in seconds since the epoch.
+    /// `None` uses the system clock.
+    pub now: Option<f64>,
+}
+
+impl SearchOptions {
+    pub fn new(k: usize) -> Self {
+        Self {
+            k,
+            fusion: hybrid::DEFAULT_FUSION,
+            source_channels: None,
+            rerank: None,
+            rerank_pool: 0,
+            confidence: None,
+            now: None,
+        }
+    }
+
+    pub fn with_fusion(mut self, fusion: hybrid::Fusion) -> Self {
+        self.fusion = fusion;
+        self
+    }
+
+    /// Search only records from these source channels.
+    pub fn with_sources<S: Into<String>>(mut self, channels: impl IntoIterator<Item = S>) -> Self {
+        self.source_channels = Some(channels.into_iter().map(Into::into).collect());
+        self
+    }
+
+    pub fn with_rerank(mut self, config: ReRankConfig) -> Self {
+        self.rerank = Some(config);
+        self
+    }
+
+    pub fn with_confidence(mut self, config: ConfidenceConfig) -> Self {
+        self.confidence = Some(config);
+        self
+    }
+
+    /// Measure recency from `now` (seconds since the epoch) instead of the
+    /// system clock — for reproducible results and tests.
+    pub fn at_time(mut self, now: f64) -> Self {
+        self.now = Some(now);
+        self
+    }
+}
+
+impl Default for SearchOptions {
+    fn default() -> Self {
+        Self::new(10)
+    }
+}
+
 impl HDF5Memory {
-    /// Vector + keyword scoring stage of [`HDF5Memory::hybrid_search`].
+    /// Vector + keyword scoring stage of [`HDF5Memory::search`].
     ///
     /// Without the `hnsw` feature this is a full linear cosine scan (the exact
     /// previous behaviour, also used as the correctness oracle in tests). With
     /// `hnsw` enabled and an index available, the vector candidates come from an
     /// approximate-nearest-neighbour search over an over-fetched pool, then merge
     /// with BM25 via the shared [`hybrid::merge_vector_keyword`].
+    ///
+    /// `exclude`, when given, marks records that must not be returned (1 =
+    /// excluded; it covers tombstones too). The index is over-fetched in
+    /// proportion to how much the mask removes. Surfacing `pool` candidates
+    /// costs the index roughly `pool × M` distance evaluations, while an exact
+    /// scan of the allowed records costs one each — so whenever that scan is
+    /// the cheaper of the two it is used instead, and it is also the fallback
+    /// if the pool comes back with too few allowed hits (the allowed records
+    /// sit away from the query). A filtered search never comes back short.
     #[cfg(feature = "hnsw")]
     fn vector_keyword_search(
         &mut self,
@@ -22,14 +111,32 @@ impl HDF5Memory {
         bm25: &bm25::BM25Index,
         fusion: hybrid::Fusion,
         k: usize,
+        exclude: Option<&[u8]>,
     ) -> Vec<(usize, f32)> {
         self.ensure_hnsw_fresh();
+        let n = self.cache.len();
+        // Over-fetch so the merge sees a useful vector pool. `ef` is
+        // configurable, but the pool the fusion stage sees is not tied to it:
+        // a caller lowering `ef` for speed should not silently narrow what
+        // fusion has to work with.
+        let mut pool = (k * 8).max(64);
+        let mut allowed = n;
+        if let Some(ex) = exclude {
+            allowed = ex.iter().filter(|&&e| e == 0).count();
+            if allowed == 0 {
+                return Vec::new();
+            }
+            // Expect `pool` allowed hits if the filter is independent of the
+            // query's neighbourhood.
+            pool = pool.saturating_mul(n).div_ceil(allowed);
+            if allowed <= pool.saturating_mul(self.hnsw_m()) {
+                return self.exact_masked_search(query_embedding, query_text, bm25, fusion, k, ex);
+            }
+        }
         match self.hnsw.as_ref() {
             Some(index) if !index.is_empty() && index.dimension() == query_embedding.len() => {
-                // Over-fetch so the merge sees a useful vector pool; cosine
-                // distance from the index converts back to similarity (1 - d).
-                let pool = (k * 8).max(64);
-                let candidates = index.search(query_embedding, pool, pool);
+                let ef = self.hnsw_ef_search(k).max(pool);
+                let candidates = index.search(query_embedding, pool, ef);
                 // A quantised index returns approximate distances, and no
                 // amount of `ef` fixes that — the loss is in the distances,
                 // not the graph. Re-score the pool against the cache's exact
@@ -38,6 +145,7 @@ impl HDF5Memory {
                 let exact = index.storage() == clawhdf5_ann::Storage::Int8;
                 let vec_scores: Vec<(usize, f32)> = candidates
                     .into_iter()
+                    .filter(|(id, _)| exclude.is_none_or(|ex| ex[*id] == 0))
                     .map(|(id, dist)| {
                         let score = if exact {
                             crate::vector_search::cosine_similarity(
@@ -52,10 +160,54 @@ impl HDF5Memory {
                     .collect();
                 // Fusion normalises over every keyword match, so it needs all
                 // the scores — but not ranked.
-                let kw_scores = bm25.scores(query_text);
+                let mut kw_scores = bm25.scores(query_text);
+                if let Some(ex) = exclude {
+                    if vec_scores.len() < k.min(allowed) {
+                        // The allowed records are not where the index looked.
+                        return self.exact_masked_search(
+                            query_embedding,
+                            query_text,
+                            bm25,
+                            fusion,
+                            k,
+                            ex,
+                        );
+                    }
+                    kw_scores.retain(|(id, _)| ex[*id] == 0);
+                }
                 hybrid::fuse(vec_scores, kw_scores, fusion, k)
             }
-            _ => hybrid::hybrid_search_fused(
+            _ => match exclude {
+                Some(ex) => {
+                    self.exact_masked_search(query_embedding, query_text, bm25, fusion, k, ex)
+                }
+                None => hybrid::hybrid_search_fused(
+                    query_embedding,
+                    query_text,
+                    &self.cache.embeddings,
+                    &self.cache.chunks,
+                    &self.cache.tombstones,
+                    bm25,
+                    fusion,
+                    k,
+                ),
+            },
+        }
+    }
+
+    #[cfg(not(feature = "hnsw"))]
+    fn vector_keyword_search(
+        &mut self,
+        query_embedding: &[f32],
+        query_text: &str,
+        bm25: &bm25::BM25Index,
+        fusion: hybrid::Fusion,
+        k: usize,
+        exclude: Option<&[u8]>,
+    ) -> Vec<(usize, f32)> {
+        match exclude {
+            Some(ex) => self.exact_masked_search(query_embedding, query_text, bm25, fusion, k, ex),
+            None => hybrid::hybrid_search_fused(
                 query_embedding,
                 query_text,
                 &self.cache.embeddings,
@@ -68,25 +220,33 @@ impl HDF5Memory {
         }
     }
 
-    #[cfg(not(feature = "hnsw"))]
-    fn vector_keyword_search(
-        &mut self,
+    /// Exact hybrid search over the records `exclude` leaves (0 = allowed).
+    fn exact_masked_search(
+        &self,
         query_embedding: &[f32],
         query_text: &str,
         bm25: &bm25::BM25Index,
         fusion: hybrid::Fusion,
         k: usize,
+        exclude: &[u8],
     ) -> Vec<(usize, f32)> {
-        hybrid::hybrid_search_fused(
-            query_embedding,
-            query_text,
-            &self.cache.embeddings,
-            &self.cache.chunks,
-            &self.cache.tombstones,
-            bm25,
-            fusion,
-            k,
-        )
+        let vec_scores =
+            hybrid::exact_vector_scores(query_embedding, &self.cache.embeddings, exclude);
+        let mut kw_scores = bm25.scores(query_text);
+        kw_scores.retain(|(id, _)| exclude.get(*id) == Some(&0));
+        hybrid::fuse(vec_scores, kw_scores, fusion, k)
+    }
+
+    /// The exclusion mask for a source-channel filter: 1 for a tombstoned
+    /// record or one from a channel not in `channels`.
+    fn source_mask(&self, channels: &[String]) -> Vec<u8> {
+        let allowed: HashSet<&str> = channels.iter().map(String::as_str).collect();
+        self.cache
+            .source_channels
+            .iter()
+            .zip(&self.cache.tombstones)
+            .map(|(ch, &t)| u8::from(t != 0 || !allowed.contains(ch.as_str())))
+            .collect()
     }
 
     /// Perform hybrid search combining cosine vector similarity and BM25 keyword search.
@@ -121,12 +281,53 @@ impl HDF5Memory {
         fusion: hybrid::Fusion,
         k: usize,
     ) -> Vec<SearchResult> {
+        self.search(
+            query_embedding,
+            query_text,
+            &SearchOptions::new(k).with_fusion(fusion),
+        )
+    }
+
+    /// Hybrid search with optional source filtering, re-ranking and
+    /// confidence rejection — see [`SearchOptions`].
+    ///
+    /// Stages, in order: vector + keyword retrieval over the records the
+    /// source filter allows; fusion; scaling by Hebbian activation; re-ranking
+    /// (if on) of a `rerank_pool` of candidates; confidence rejection (if on);
+    /// the top `k`. The records returned with a positive score get their
+    /// Hebbian boost.
+    pub fn search(
+        &mut self,
+        query_embedding: &[f32],
+        query_text: &str,
+        options: &SearchOptions,
+    ) -> Vec<SearchResult> {
+        let k = options.k;
+        let fetch = match options.rerank {
+            Some(_) if options.rerank_pool > 0 => options.rerank_pool.max(k),
+            Some(_) => k.saturating_mul(3).max(10),
+            None => k,
+        };
+        let exclude = options
+            .source_channels
+            .as_deref()
+            .map(|channels| self.source_mask(channels));
+
         // The keyword index lives for the life of the store and is updated
         // incrementally. Take it out for the duration of the call so the
         // vector stage can borrow `self` mutably, then put it back.
         self.ensure_bm25_fresh();
         let bm25 = self.bm25.take().expect("ensure_bm25_fresh leaves an index");
-        let scored = self.vector_keyword_search(query_embedding, query_text, &bm25, fusion, k);
+        let scored = self.vector_keyword_search(
+            query_embedding,
+            query_text,
+            &bm25,
+            options.fusion,
+            fetch,
+            exclude.as_deref(),
+        );
+        self.bm25 = Some(bm25);
+
         let mut results: Vec<SearchResult> = scored
             .into_iter()
             .map(|(idx, score)| {
@@ -150,6 +351,25 @@ impl HDF5Memory {
                 .then(a.index.cmp(&b.index))
         });
 
+        if let Some(config) = &options.rerank {
+            results = Self::rerank_results(results, config, options.now);
+        }
+        if let Some(config) = &options.confidence {
+            let scored: Vec<ScoredResult> = results
+                .iter()
+                .map(|r| ScoredResult {
+                    index: r.index,
+                    score: r.score,
+                })
+                .collect();
+            let keep: HashSet<usize> = reject_low_confidence(&scored, config)
+                .into_iter()
+                .map(|r| r.index)
+                .collect();
+            results.retain(|r| keep.contains(&r.index));
+        }
+        results.truncate(k);
+
         // Only reinforce records that actually matched. When fewer than `k`
         // records are relevant, the rest of the list is zero-score filler;
         // boosting it would teach the store that arbitrary records are
@@ -160,11 +380,45 @@ impl HDF5Memory {
             .map(|r| r.index)
             .collect();
         self.apply_hebbian_boost(&hit_indices);
-        self.bm25 = Some(bm25);
 
         results
     }
 
+    /// Reorder by the re-ranker's combined score, which also becomes each
+    /// result's `score`.
+    fn rerank_results(
+        results: Vec<SearchResult>,
+        config: &ReRankConfig,
+        now: Option<f64>,
+    ) -> Vec<SearchResult> {
+        let now = now.unwrap_or_else(|| {
+            std::time::SystemTime::now()
+                .duration_since(std::time::UNIX_EPOCH)
+                .map(|d| d.as_secs_f64())
+                .unwrap_or(0.0)
+        });
+        let inputs: Vec<RerankInput> = results
+            .iter()
+            .map(|r| RerankInput {
+                index: r.index,
+                timestamp: r.timestamp,
+                source_channel: r.source_channel.clone(),
+                raw_activation: r.activation,
+                relevance: r.score,
+            })
+            .collect();
+        let mut by_index: std::collections::HashMap<usize, SearchResult> =
+            results.into_iter().map(|r| (r.index, r)).collect();
+        rerank(&inputs, config, now)
+            .into_iter()
+            .filter_map(|rr| {
+                let mut r = by_index.remove(&rr.index)?;
+                r.score = rr.combined_score;
+                Some(r)
+            })
+            .collect()
+    }
+
     /// Reinforce the records a query returned. The new weights are persisted by
     /// the next checkpoint (any write that flushes, `flush_wal`, or drop) — not
     /// by rewriting the whole store inside the query, which is what made

crates/clawhdf5-agent/src/session.rs

@@ -33,7 +33,7 @@ impl SessionCache {
         self.entries.is_empty()
     }
 
-    /// Add a new session with its summary.
+    /// Add a new session with its summary, timestamped now.
     pub fn add(
         &mut self,
         id: &str,
@@ -47,6 +47,21 @@ impl SessionCache {
             .unwrap_or_default()
             .as_secs_f64()
             * 1_000_000.0; // microseconds
+        self.add_at(id, start_idx, end_idx, channel, summary, ts);
+    }
+
+    /// Add a session with an explicit timestamp (Unix **microseconds**, the
+    /// unit [`SessionEntry::ts`] uses) — for importers carrying sessions over
+    /// from another store, whose original time should be kept.
+    pub fn add_at(
+        &mut self,
+        id: &str,
+        start_idx: usize,
+        end_idx: usize,
+        channel: &str,
+        summary: &str,
+        ts: f64,
+    ) {
         self.entries.push(SessionEntry {
             id: id.to_string(),
             start_idx: start_idx as u64,

crates/clawhdf5-agent/src/signing.rs

@@ -0,0 +1,419 @@
+//! Ed25519-signed checkpoints.
+//!
+//! When a signing key is set ([`crate::HDF5Memory::set_signing_key`]), every
+//! checkpoint writes a signed manifest of the store: a SHA-256 per memory
+//! record rolled into a Merkle root, plus hashes of the store's settings, its
+//! sessions and its knowledge graph. [`verify_store`] recomputes all of it from
+//! the file and checks the signature against a public key the caller trusts,
+//! so any change to the checkpointed file — a record's text or embedding, a
+//! setting, a session, a graph edge, made through this crate or any other HDF5
+//! tool — is detected, and the per-record hashes say which records changed.
+//!
+//! What it does not cover: saves still only in the WAL (made since the last
+//! checkpoint). [`VerifyReport::wal_entries_unsigned`] counts them.
+//!
+//! The hashes cover exactly what the file persists, in the form the loader
+//! returns it, so a store verifies after any number of reopen/checkpoint
+//! cycles. Derived data (L2 norms, the vector index) is not covered; it is
+//! recomputed from covered data.
+
+use ed25519_dalek::{Signature, Signer, Verifier};
+pub use ed25519_dalek::{SigningKey, VerifyingKey};
+use sha2::{Digest, Sha256};
+
+use crate::MemoryConfig;
+use crate::cache::MemoryCache;
+use crate::knowledge::KnowledgeCache;
+use crate::session::SessionCache;
+use crate::wal::WalMark;
+
+/// Version of the manifest encoding; part of what is signed.
+pub const MANIFEST_VERSION: i64 = 1;
+
+type Hash = [u8; 32];
+
+/// The hashes a signature covers.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Manifest {
+    pub record_count: u64,
+    /// Merkle root over the per-record hashes.
+    pub records_root: Hash,
+    /// Settings persisted in `/meta`, plus the checkpoint's WAL mark.
+    pub settings: Hash,
+    pub sessions: Hash,
+    pub graph: Hash,
+}
+
+impl Manifest {
+    /// The exact bytes that are signed.
+    pub fn signed_bytes(&self) -> Vec<u8> {
+        let mut m = Vec::with_capacity(160);
+        m.extend_from_slice(b"clawhdf5-agent signed checkpoint\0");
+        m.extend_from_slice(&MANIFEST_VERSION.to_le_bytes());
+        m.extend_from_slice(&self.record_count.to_le_bytes());
+        m.extend_from_slice(&self.records_root);
+        m.extend_from_slice(&self.settings);
+        m.extend_from_slice(&self.sessions);
+        m.extend_from_slice(&self.graph);
+        m
+    }
+}
+
+/// A signature as stored in a checkpoint.
+#[derive(Debug, Clone)]
+pub struct StoredSignature {
+    pub manifest: Manifest,
+    pub record_hashes: Vec<Hash>,
+    pub public_key: [u8; 32],
+    pub signature: [u8; 64],
+}
+
+/// Build the manifest (and per-record hashes) for the state about to be
+/// checkpointed, and sign it.
+pub fn sign(
+    key: &SigningKey,
+    config: &MemoryConfig,
+    cache: &MemoryCache,
+    sessions: &SessionCache,
+    knowledge: &KnowledgeCache,
+    wal_applied: Option<WalMark>,
+) -> StoredSignature {
+    let (manifest, record_hashes) = manifest(config, cache, sessions, knowledge, wal_applied);
+    let signature = key.sign(&manifest.signed_bytes()).to_bytes();
+    StoredSignature {
+        manifest,
+        record_hashes,
+        public_key: key.verifying_key().to_bytes(),
+        signature,
+    }
+}
+
+/// Compute the manifest of a store's state.
+pub fn manifest(
+    config: &MemoryConfig,
+    cache: &MemoryCache,
+    sessions: &SessionCache,
+    knowledge: &KnowledgeCache,
+    wal_applied: Option<WalMark>,
+) -> (Manifest, Vec<Hash>) {
+    let record_hashes: Vec<Hash> = (0..cache.len()).map(|i| record_hash(cache, i)).collect();
+    let manifest = Manifest {
+        record_count: cache.len() as u64,
+        records_root: merkle_root(&record_hashes),
+        settings: settings_hash(config, wal_applied),
+        sessions: sessions_hash(sessions),
+        graph: graph_hash(knowledge),
+    };
+    (manifest, record_hashes)
+}
+
+// ---------------------------------------------------------------------------
+// Canonical encoding
+// ---------------------------------------------------------------------------
+
+/// A SHA-256 over length-prefixed fields, so no two different field lists
+/// hash the same bytes.
+struct Fields(Sha256);
+
+impl Fields {
+    fn new(domain: &str) -> Self {
+        let mut h = Sha256::new();
+        h.update((domain.len() as u64).to_le_bytes());
+        h.update(domain.as_bytes());
+        Self(h)
+    }
+    fn bytes(&mut self, b: &[u8]) -> &mut Self {
+        self.0.update((b.len() as u64).to_le_bytes());
+        self.0.update(b);
+        self
+    }
+    /// Strings as the loader returns them: stored null-padded, so a trailing
+    /// NUL cannot survive a round trip and must not be part of the hash.
+    fn str(&mut self, s: &str) -> &mut Self {
+        self.bytes(s.trim_end_matches('\0').as_bytes())
+    }
+    fn u64(&mut self, v: u64) -> &mut Self {
+        self.0.update(v.to_le_bytes());
+        self
+    }
+    fn f64(&mut self, v: f64) -> &mut Self {
+        self.0.update(v.to_bits().to_le_bytes());
+        self
+    }
+    fn f32(&mut self, v: f32) -> &mut Self {
+        self.0.update(v.to_bits().to_le_bytes());
+        self
+    }
+    fn finish(self) -> Hash {
+        self.0.finalize().into()
+    }
+}
+
+/// Everything persisted about record `i`, including its position. The
+/// embedding is hashed as the cache holds it — for a `float16` store that is
+/// the half-rounded value the file holds.
+fn record_hash(cache: &MemoryCache, i: usize) -> Hash {
+    let mut f = Fields::new("clawhdf5-agent/record");
+    f.u64(i as u64).str(&cache.chunks[i]);
+    let emb: Vec<u8> = cache.embeddings[i]
+        .iter()
+        .flat_map(|v| v.to_bits().to_le_bytes())
+        .collect();
+    f.bytes(&emb)
+        .str(&cache.source_channels[i])
+        .f64(cache.timestamps[i])
+        .str(&cache.session_ids[i])
+        .str(&cache.tags[i])
+        .u64(u64::from(cache.tombstones[i]))
+        .f32(cache.activation_weights[i]);
+    f.finish()
+}
+
+/// Binary Merkle tree: leaves are the record hashes; a parent hashes its two
+/// children with a node prefix; an odd node is carried up unchanged.
+fn merkle_root(leaves: &[Hash]) -> Hash {
+    if leaves.is_empty() {
+        return Fields::new("clawhdf5-agent/merkle-empty").finish();
+    }
+    let mut level: Vec<Hash> = leaves.to_vec();
+    while level.len() > 1 {
+        level = level
+            .chunks(2)
+            .map(|pair| match pair {
+                [l, r] => {
+                    let mut h = Sha256::new();
+                    h.update([1u8]);
+                    h.update(l);
+                    h.update(r);
+                    h.finalize().into()
+                }
+                [only] => *only,
+                _ => unreachable!(),
+            })
+            .collect();
+    }
+    level[0]
+}
+
+fn settings_hash(c: &MemoryConfig, wal_applied: Option<WalMark>) -> Hash {
+    let mut f = Fields::new("clawhdf5-agent/settings");
+    f.str(crate::schema::SCHEMA_VERSION)
+        .str(&c.created_at)
+        .str(&c.agent_id)
+        .str(&c.embedder)
+        .u64(c.embedding_dim as u64)
+        .u64(c.chunk_size as u64)
+        .u64(c.overlap as u64)
+        .u64(u64::from(c.float16))
+        .u64(u64::from(c.compression))
+        .u64(u64::from(c.compression_level))
+        .f32(c.compact_threshold)
+        .f32(c.hebbian_boost)
+        .f32(c.decay_factor)
+        .u64(u64::from(c.wal_enabled))
+        .u64(c.wal_max_entries as u64)
+        .u64(u64::from(c.quantized_index))
+        .u64(c.hnsw_m as u64)
+        .u64(c.hnsw_ef_construction as u64)
+        .u64(c.hnsw_ef_search as u64);
+    // An empty mark is not written to the file, so it must hash as none.
+    match wal_applied.filter(|m| m.len > 0) {
+        Some(m) => f.u64(1).u64(m.len).u64(u64::from(m.crc)),
+        None => f.u64(0),
+    };
+    f.finish()
+}
+
+fn sessions_hash(s: &SessionCache) -> Hash {
+    let mut f = Fields::new("clawhdf5-agent/sessions");
+    f.u64(s.entries.len() as u64);
+    for (i, e) in s.entries.iter().enumerate() {
+        f.str(&e.id)
+            .u64(e.start_idx)
+            .u64(e.end_idx)
+            .str(&e.channel)
+            .f64(e.ts)
+            .str(s.summaries.get(i).map(String::as_str).unwrap_or(""));
+    }
+    f.finish()
+}
+
+fn graph_hash(k: &KnowledgeCache) -> Hash {
+    let mut f = Fields::new("clawhdf5-agent/graph");
+    f.u64(k.entities.len() as u64);
+    for e in &k.entities {
+        f.u64(e.id)
+            .str(&e.name)
+            .str(&e.entity_type)
+            .u64(e.embedding_idx as u64);
+    }
+    f.u64(k.relations.len() as u64);
+    for r in &k.relations {
+        f.u64(r.src)
+            .u64(r.tgt)
+            .str(&r.relation)
+            .f32(r.weight)
+            .f64(r.ts);
+    }
+    f.u64(k.alias_strings.len() as u64);
+    for (s, id) in k.alias_strings.iter().zip(&k.alias_entity_ids) {
+        f.str(s).u64(*id as u64);
+    }
+    f.finish()
+}
+
+// ---------------------------------------------------------------------------
+// Verification
+// ---------------------------------------------------------------------------
+
+/// The outcome of [`verify_store`].
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct VerifyReport {
+    /// The checkpoint carries a signature.
+    pub signed: bool,
+    /// The signature was made by the key the caller trusts.
+    pub key_matches: bool,
+    /// The signature over the stored manifest is valid.
+    pub signature_valid: bool,
+    /// The file's current contents match the signed manifest.
+    pub records_match: bool,
+    pub settings_match: bool,
+    pub sessions_match: bool,
+    pub graph_match: bool,
+    /// Records whose contents differ from what was signed (by position),
+    /// when the stored per-record hashes are themselves authentic.
+    pub changed_records: Vec<usize>,
+    /// Records in the file versus in the signed manifest.
+    pub record_count: u64,
+    pub signed_record_count: u64,
+    /// The public key the checkpoint claims to be signed by.
+    pub public_key: Option<[u8; 32]>,
+    /// Saves in the WAL after the checkpoint: not covered by the signature.
+    pub wal_entries_unsigned: usize,
+}
+
+impl VerifyReport {
+    /// Signed by the trusted key, signature valid, and every part of the
+    /// file unchanged since it was signed.
+    pub fn is_valid(&self) -> bool {
+        self.signed
+            && self.key_matches
+            && self.signature_valid
+            && self.records_match
+            && self.settings_match
+            && self.sessions_match
+            && self.graph_match
+    }
+}
+
+/// Check a store file against the public key the caller trusts.
+///
+/// Reads the checkpoint (not the WAL), recomputes every hash from its
+/// contents and checks the signature. Never writes.
+pub fn verify_store(
+    path: &std::path::Path,
+    trusted: &VerifyingKey,
+) -> Result<VerifyReport, crate::MemoryError> {
+    let file = clawhdf5::File::open(path)
+        .map_err(|e| crate::MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
+    let (config, cache, sessions, knowledge) = crate::schema::validate_and_load(&file)?;
+    let checkpoint = crate::schema::read_checkpoint_meta(&file);
+    let stored = crate::schema::read_signature(&file)?;
+    let wal_entries_unsigned = count_wal_entries_after(path, checkpoint.wal_applied);
+
+    let (current, current_hashes) = manifest(
+        &config,
+        &cache,
+        &sessions,
+        &knowledge,
+        checkpoint.wal_applied,
+    );
+
+    let Some(stored) = stored else {
+        return Ok(VerifyReport {
+            signed: false,
+            key_matches: false,
+            signature_valid: false,
+            records_match: false,
+            settings_match: false,
+            sessions_match: false,
+            graph_match: false,
+            changed_records: Vec::new(),
+            record_count: current.record_count,
+            signed_record_count: 0,
+            public_key: None,
+            wal_entries_unsigned,
+        });
+    };
+
+    let key_matches = stored.public_key == trusted.to_bytes();
+    let signature_valid = trusted
+        .verify(
+            &stored.manifest.signed_bytes(),
+            &Signature::from_bytes(&stored.signature),
+        )
+        .is_ok();
+    // The stored per-record hashes can localise a change only if they are
+    // the ones that were signed.
+    let hashes_authentic = signature_valid
+        && stored.record_hashes.len() as u64 == stored.manifest.record_count
+        && merkle_root(&stored.record_hashes) == stored.manifest.records_root;
+    let changed_records = if hashes_authentic {
+        let n = current_hashes.len().max(stored.record_hashes.len());
+        (0..n)
+            .filter(|&i| current_hashes.get(i) != stored.record_hashes.get(i))
+            .collect()
+    } else {
+        Vec::new()
+    };
+
+    Ok(VerifyReport {
+        signed: true,
+        key_matches,
+        signature_valid,
+        records_match: signature_valid
+            && current.record_count == stored.manifest.record_count
+            && current.records_root == stored.manifest.records_root,
+        settings_match: signature_valid && current.settings == stored.manifest.settings,
+        sessions_match: signature_valid && current.sessions == stored.manifest.sessions,
+        graph_match: signature_valid && current.graph == stored.manifest.graph,
+        changed_records,
+        record_count: current.record_count,
+        signed_record_count: stored.manifest.record_count,
+        public_key: Some(stored.public_key),
+        wal_entries_unsigned,
+    })
+}
+
+fn count_wal_entries_after(store: &std::path::Path, mark: Option<WalMark>) -> usize {
+    let wal = store.with_extension("h5.wal");
+    if !wal.exists() {
+        return 0;
+    }
+    crate::wal::WalFile::read_entries_for_migration(&wal, mark)
+        .map(|e| e.len())
+        .unwrap_or(0)
+}
+
+/// A new random signing key from the operating system's RNG.
+pub fn generate_key() -> SigningKey {
+    SigningKey::generate(&mut rand_core::OsRng)
+}
+
+/// Hex encoding for keys and signatures in attributes and the CLI.
+pub fn to_hex(bytes: &[u8]) -> String {
+    bytes.iter().map(|b| format!("{b:02x}")).collect()
+}
+
+/// Parse hex into exactly `N` bytes.
+pub fn from_hex<const N: usize>(s: &str) -> Option<[u8; N]> {
+    let s = s.trim();
+    if s.len() != 2 * N {
+        return None;
+    }
+    let mut out = [0u8; N];
+    for (i, byte) in out.iter_mut().enumerate() {
+        *byte = u8::from_str_radix(&s[2 * i..2 * i + 2], 16).ok()?;
+    }
+    Some(out)
+}

crates/clawhdf5-agent/src/storage.rs

@@ -36,7 +36,7 @@ pub fn write_to_disk_with_mark(
 ) -> Result<(), MemoryError> {
     let meta = schema::CheckpointMeta {
         wal_applied,
-        ann_generation: None,
+        ..schema::CheckpointMeta::default()
     };
     write_to_disk_with_meta(path, config, cache, sessions, knowledge, &meta)
 }
@@ -50,7 +50,21 @@ pub fn write_to_disk_with_meta(
     knowledge: &KnowledgeCache,
     checkpoint: &schema::CheckpointMeta,
 ) -> Result<(), MemoryError> {
-    let bytes = schema::build_hdf5_file_with_meta(config, cache, sessions, knowledge, checkpoint)?;
+    write_to_disk_signed(path, config, cache, sessions, knowledge, checkpoint, None)
+}
+
+/// [`write_to_disk_with_meta`] with a signed manifest of the contents.
+pub fn write_to_disk_signed(
+    path: &Path,
+    config: &MemoryConfig,
+    cache: &MemoryCache,
+    sessions: &SessionCache,
+    knowledge: &KnowledgeCache,
+    checkpoint: &schema::CheckpointMeta,
+    signature: Option<&crate::signing::StoredSignature>,
+) -> Result<(), MemoryError> {
+    let bytes =
+        schema::build_hdf5_file_signed(config, cache, sessions, knowledge, checkpoint, signature)?;
 
     if bytes.is_empty() {
         return Err(MemoryError::Hdf5("build_hdf5_file produced 0 bytes".into()));
@@ -113,13 +127,11 @@ pub type StoreState = (MemoryConfig, MemoryCache, SessionCache, KnowledgeCache);
 /// [`read_from_disk`], plus the checkpoint's [`WalMark`] (if any) so the
 /// caller can skip WAL entries this file already contains.
 pub fn read_from_disk_with_mark(path: &Path) -> Result<(StoreState, Option<WalMark>), MemoryError> {
-    let mmap = clawhdf5_io::MmapReader::open(path).map_err(MemoryError::Io)?;
-
-    // Advise the OS we'll need the whole file for parsing
-    mmap.advise_willneed(0, mmap.len());
-
-    // Parse the HDF5 file from the mmap'd bytes
-    let file = clawhdf5::File::from_bytes(mmap.as_bytes().to_vec())
+    // `File::open` memory-maps the file itself (the facade's `mmap` feature is
+    // on by default). Mapping it here and handing over `as_bytes().to_vec()`
+    // did the same work and then copied the whole store — a second full copy
+    // of the file, live for the whole parse, on top of the mapping.
+    let file = clawhdf5::File::open(path)
         .map_err(|e| MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
 
     let (mut config, cache, sessions, knowledge) = schema::validate_and_load(&file)?;
@@ -133,9 +145,7 @@ pub fn read_from_disk_with_mark(path: &Path) -> Result<(StoreState, Option<WalMa
 pub fn read_from_disk_with_meta(
     path: &Path,
 ) -> Result<(StoreState, schema::CheckpointMeta), MemoryError> {
-    let mmap = clawhdf5_io::MmapReader::open(path).map_err(MemoryError::Io)?;
-    mmap.advise_willneed(0, mmap.len());
-    let file = clawhdf5::File::from_bytes(mmap.as_bytes().to_vec())
+    let file = clawhdf5::File::open(path)
         .map_err(|e| MemoryError::Hdf5(format!("cannot open {}: {e}", path.display())))?;
     let (mut config, cache, sessions, knowledge) = schema::validate_and_load(&file)?;
     config.path = path.to_path_buf();

crates/clawhdf5-agent/tests/float16_store.rs

@@ -0,0 +1,260 @@
+//! `MemoryConfig::float16`: embeddings stored as IEEE half precision.
+//!
+//! The setting used to be recorded in `/meta` and otherwise ignored — the
+//! embeddings dataset was always `f32`. These tests pin what it now does: the
+//! dataset is `float16`, the in-memory cache holds exactly the values the file
+//! holds (so search results survive a reopen bit for bit), and a value half
+//! precision cannot represent is refused rather than stored as infinity.
+
+use std::path::{Path, PathBuf};
+
+use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError};
+use clawhdf5_format::float16::round_to_f16;
+use tempfile::TempDir;
+
+const DIM: usize = 64;
+
+/// Deterministic, embedding-like unit vectors.
+fn embedding(seed: u64) -> Vec<f32> {
+    let mut x = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
+    let v: Vec<f32> = (0..DIM)
+        .map(|_| {
+            x ^= x << 13;
+            x ^= x >> 7;
+            x ^= x << 17;
+            (x >> 40) as f32 / (1u64 << 24) as f32 - 0.5
+        })
+        .collect();
+    let norm = v.iter().map(|a| a * a).sum::<f32>().sqrt();
+    v.iter().map(|a| a / norm).collect()
+}
+
+fn entry(i: u64) -> MemoryEntry {
+    MemoryEntry {
+        chunk: format!("memory number {i} about topic {}", i % 7),
+        embedding: embedding(i),
+        source_channel: "test".into(),
+        timestamp: i as f64,
+        session_id: "s".into(),
+        tags: format!("t{i}"),
+    }
+}
+
+fn config(dir: &TempDir, name: &str, float16: bool) -> MemoryConfig {
+    let mut c = MemoryConfig::new(dir.path().join(name), "agent", DIM);
+    c.float16 = float16;
+    c
+}
+
+fn embeddings_dtype_and_values(path: &Path) -> (String, Vec<f32>) {
+    let file = clawhdf5::File::open(path).unwrap();
+    let ds = file.dataset("memory/embeddings").unwrap();
+    (format!("{:?}", ds.dtype().unwrap()), ds.read_f32().unwrap())
+}
+
+fn search_bits(m: &mut HDF5Memory, q: u64) -> Vec<(usize, u32)> {
+    m.hybrid_search(&embedding(q), "memory topic 3", 0.4, 0.6, 10)
+        .iter()
+        .map(|r| (r.index, r.score.to_bits()))
+        .collect()
+}
+
+#[test]
+fn float16_store_writes_half_precision_and_reopens_identically() {
+    let dir = TempDir::new().unwrap();
+    // Two identical stores. Search is not read-only (it boosts the Hebbian
+    // activation of what it returns, and checkpoints persist that), so each
+    // is queried exactly once: one live, one after a checkpoint and reopen.
+    let live_cfg = config(&dir, "live.h5", true);
+    let cfg = config(&dir, "f16.h5", true);
+    let path: PathBuf = cfg.path.clone();
+
+    let mut live = HDF5Memory::create(live_cfg).unwrap();
+    live.save_batch((0..200).map(entry).collect()).unwrap();
+    let mut m = HDF5Memory::create(cfg).unwrap();
+    m.save_batch((0..200).map(entry).collect()).unwrap();
+    drop(m);
+
+    // On disk: a genuine float16 dataset holding the rounded inputs.
+    let (dtype, values) = embeddings_dtype_and_values(&path);
+    assert_eq!(dtype, "Other(\"float16\")");
+    let expected: Vec<u32> = (0..200)
+        .flat_map(|i| embedding(i).into_iter().map(|v| round_to_f16(v).to_bits()))
+        .collect();
+    let got: Vec<u32> = values.iter().map(|v| v.to_bits()).collect();
+    assert_eq!(got, expected);
+
+    // Reopened, the store answers exactly as the live one does: the cache
+    // held the half-rounded values before the checkpoint.
+    let mut reopened = HDF5Memory::open(&path).unwrap();
+    for q in 0..5 {
+        assert_eq!(
+            search_bits(&mut live, 1000 + q),
+            search_bits(&mut reopened, 1000 + q),
+            "query {q}"
+        );
+    }
+}
+
+#[test]
+fn float16_halves_the_embeddings_on_disk() {
+    let dir = TempDir::new().unwrap();
+    let mut sizes = Vec::new();
+    for float16 in [false, true] {
+        let cfg = config(&dir, &format!("s{float16}.h5"), float16);
+        let path = cfg.path.clone();
+        let mut m = HDF5Memory::create(cfg).unwrap();
+        m.save_batch((0..2000).map(entry).collect()).unwrap();
+        drop(m);
+        sizes.push(std::fs::metadata(&path).unwrap().len());
+    }
+    let embedding_bytes_f32 = (2000 * DIM * 4) as u64;
+    let saved = sizes[0] - sizes[1];
+    // Half of the f32 embeddings, give or take metadata and alignment.
+    assert!(
+        saved.abs_diff(embedding_bytes_f32 / 2) < 16 * 1024,
+        "f32 {} B, f16 {} B, saved {saved} B, expected ~{} B",
+        sizes[0],
+        sizes[1],
+        embedding_bytes_f32 / 2
+    );
+}
+
+#[test]
+fn f32_store_is_unchanged() {
+    let dir = TempDir::new().unwrap();
+    let cfg = config(&dir, "f32.h5", false);
+    let path = cfg.path.clone();
+    let mut m = HDF5Memory::create(cfg).unwrap();
+    m.save_batch((0..50).map(entry).collect()).unwrap();
+    drop(m);
+    let (dtype, values) = embeddings_dtype_and_values(&path);
+    assert_eq!(dtype, "F32");
+    let expected: Vec<f32> = (0..50).flat_map(embedding).collect();
+    assert_eq!(values, expected);
+}
+
+#[test]
+fn out_of_range_values_are_refused_not_stored_as_infinity() {
+    let dir = TempDir::new().unwrap();
+    let mut cfg = config(&dir, "range.h5", true);
+    cfg.wal_enabled = true;
+    let path = cfg.path.clone();
+    let mut m = HDF5Memory::create(cfg).unwrap();
+    m.save(entry(1)).unwrap();
+
+    let mut bad = entry(2);
+    bad.embedding[5] = 70_000.0;
+    match m.save(bad.clone()) {
+        Err(MemoryError::InvalidEntry(msg)) => assert!(msg.contains("embedding[5]"), "{msg}"),
+        other => panic!("expected InvalidEntry, got {other:?}"),
+    }
+    assert!(matches!(
+        m.save_or_update(bad.clone()),
+        Err(MemoryError::InvalidEntry(_))
+    ));
+    // A batch is all or nothing.
+    assert!(matches!(
+        m.save_batch(vec![entry(3), bad.clone(), entry(4)]),
+        Err(MemoryError::InvalidEntry(_))
+    ));
+    assert_eq!(m.count(), 1);
+
+    // The largest finite half, and values that round down to it, are fine.
+    let mut edge = entry(5);
+    edge.embedding[0] = 65504.0;
+    edge.embedding[1] = -65519.0;
+    m.save(edge).unwrap();
+    assert_eq!(m.count(), 2);
+    drop(m);
+
+    // Nothing rejected reached the WAL or the file.
+    let m = HDF5Memory::open(&path).unwrap();
+    assert_eq!(m.count(), 2);
+
+    // An f32 store takes the same value as it always did.
+    let mut m32 = HDF5Memory::create(config(&dir, "range32.h5", false)).unwrap();
+    m32.save(bad).unwrap();
+}
+
+#[test]
+fn wal_replay_rounds_like_a_live_save() {
+    let dir = TempDir::new().unwrap();
+    let mut cfg = config(&dir, "wal.h5", true);
+    cfg.wal_enabled = true;
+    cfg.wal_max_entries = 10_000; // keep everything in the WAL
+    let path = cfg.path.clone();
+    let mut m = HDF5Memory::create(cfg).unwrap();
+    for i in 0..30 {
+        m.save(entry(i)).unwrap();
+    }
+    let live = search_bits(&mut m, 77);
+
+    // Crash image: the .h5 is still the empty checkpoint; everything is in
+    // the WAL, which holds the caller's f32 values.
+    let crash = TempDir::new().unwrap();
+    let image = crash.path().join("image.h5");
+    std::fs::copy(&path, &image).unwrap();
+    std::fs::copy(
+        path.with_extension("h5.wal"),
+        image.with_extension("h5.wal"),
+    )
+    .unwrap();
+    drop(m);
+
+    let mut recovered = HDF5Memory::open(&image).unwrap();
+    assert_eq!(recovered.count(), 30);
+    assert_eq!(search_bits(&mut recovered, 77), live);
+}
+
+#[test]
+fn new_stores_default_to_float16() {
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("default.h5");
+    let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", DIM)).unwrap();
+    assert!(m.config().float16);
+    m.save_batch((0..10).map(entry).collect()).unwrap();
+    drop(m);
+    assert_eq!(embeddings_dtype_and_values(&path).0, "Other(\"float16\")");
+    assert!(HDF5Memory::open(&path).unwrap().config().float16);
+}
+
+#[test]
+fn an_existing_f32_store_stays_f32() {
+    // Written by the v2.5.0 CLI, with `float16 = 0` in /meta (every agent
+    // store has recorded it). Flipping the default for new stores must not
+    // reach back and round an existing store's embeddings.
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("legacy.h5");
+    std::fs::copy(
+        concat!(
+            env!("CARGO_MANIFEST_DIR"),
+            "/tests/fixtures/store_v2_5_0.h5"
+        ),
+        &path,
+    )
+    .unwrap();
+    let before = embeddings_dtype_and_values(&path);
+    assert_eq!(before.0, "F32");
+
+    let mut m = HDF5Memory::open(&path).unwrap();
+    assert!(!m.config().float16, "an old store must reopen as f32");
+    let dim = m.config().embedding_dim;
+    let odd: Vec<f32> = (0..dim).map(|i| 0.1 + i as f32 * 1e-4).collect();
+    m.save_batch(vec![MemoryEntry {
+        chunk: "added after the upgrade".into(),
+        embedding: odd.clone(),
+        source_channel: "test".into(),
+        timestamp: 1.0,
+        session_id: "s".into(),
+        tags: String::new(),
+    }])
+    .unwrap();
+    drop(m);
+
+    // Checkpointed: still f32, the old rows untouched and the new one exact.
+    let (dtype, values) = embeddings_dtype_and_values(&path);
+    assert_eq!(dtype, "F32");
+    assert_eq!(&values[..before.1.len()], before.1.as_slice());
+    assert_eq!(&values[before.1.len()..], odd.as_slice());
+}

crates/clawhdf5-agent/tests/h5py_interop.rs

@@ -0,0 +1,155 @@
+//! An agent store is a standard HDF5 file: h5py can open it and read every
+//! dataset.
+//!
+//! It could not: the float datatype's sign-bit position was hard-coded for
+//! f64, so every f32 dataset (embeddings, norms, activation weights) made
+//! libhdf5 refuse the file with "sign bit position out of bounds".
+
+use std::process::Command;
+
+use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
+
+fn python() -> String {
+    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
+}
+
+fn h5py_available() -> bool {
+    Command::new(python())
+        .args(["-c", "import h5py"])
+        .output()
+        .map(|o| o.status.success())
+        .unwrap_or(false)
+}
+
+#[test]
+fn h5py_reads_every_dataset_of_an_agent_store() {
+    if !h5py_available() {
+        assert!(
+            std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
+            "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+        );
+        eprintln!("SKIP: python3 with h5py not available");
+        return;
+    }
+    let dir = tempfile::tempdir().unwrap();
+    for float16 in [false, true] {
+        let path = dir.path().join(format!("store_{float16}.h5"));
+        let mut cfg = MemoryConfig::new(path.clone(), "agent", 8);
+        cfg.float16 = float16;
+        let mut m = HDF5Memory::create(cfg).unwrap();
+        // save_batch checkpoints, so the records are in the .h5, not the WAL.
+        m.save_batch(
+            (0..20)
+                .map(|i| MemoryEntry {
+                    chunk: format!("memory {i}"),
+                    embedding: (0..8).map(|j| ((i * 8 + j) as f32).sin()).collect(),
+                    source_channel: "test".into(),
+                    timestamp: i as f64,
+                    session_id: "s".into(),
+                    tags: String::new(),
+                })
+                .collect(),
+        )
+        .unwrap();
+        drop(m);
+
+        // Exact expected values, as bits: numpy's sin need not match Rust's
+        // to the last place.
+        let bits = (0..160)
+            .map(|k| (k as f32).sin().to_bits().to_string())
+            .collect::<Vec<_>>()
+            .join(",");
+        let script = format!(
+            r#"
+import h5py, numpy as np
+want = np.float16 if {py_bool} else np.float32
+with h5py.File("{path}", "r") as f:
+    names = []
+    f.visititems(lambda n, o: names.append(n) if isinstance(o, h5py.Dataset) else None)
+    for n in names:
+        f[n][()]  # every dataset must decode
+    e = f["memory/embeddings"]
+    assert e.dtype == want, e.dtype
+    assert e.shape == (20, 8), e.shape
+    ref = np.array([{bits}], dtype=np.uint32).view(np.float32).astype(want).reshape(20, 8)
+    assert (e[()] == ref).all()
+    assert f["memory/norms"].dtype == np.float32
+print(len(names))
+"#,
+            py_bool = if float16 { "True" } else { "False" },
+            path = path.display()
+        );
+        let out = Command::new(python())
+            .args(["-c", &script])
+            .output()
+            .unwrap();
+        assert!(
+            out.status.success(),
+            "float16={float16}: {}",
+            String::from_utf8_lossy(&out.stderr)
+        );
+        let n: usize = String::from_utf8_lossy(&out.stdout).trim().parse().unwrap();
+        assert!(n >= 10, "only {n} datasets");
+    }
+}
+
+#[test]
+fn an_edit_made_with_h5py_breaks_the_signature_and_names_the_record() {
+    if !h5py_available() {
+        assert!(
+            std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
+            "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
+        );
+        eprintln!("SKIP: python3 with h5py not available");
+        return;
+    }
+    use clawhdf5_agent::signing::SigningKey;
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("signed.h5");
+    let key = SigningKey::from_bytes(&[42; 32]);
+    let mut m = HDF5Memory::create(MemoryConfig::new(path.clone(), "agent", 8)).unwrap();
+    m.set_signing_key(key.clone());
+    m.save_batch(
+        (0..10)
+            .map(|i| MemoryEntry {
+                chunk: format!("memory {i}"),
+                embedding: (0..8).map(|j| ((i * 8 + j) as f32).cos()).collect(),
+                source_channel: "test".into(),
+                timestamp: i as f64,
+                session_id: "s".into(),
+                tags: String::new(),
+            })
+            .collect(),
+    )
+    .unwrap();
+    drop(m);
+    assert!(
+        HDF5Memory::verify(&path, &key.verifying_key())
+            .unwrap()
+            .is_valid()
+    );
+
+    // Someone edits one timestamp in place with h5py.
+    let script = format!(
+        r#"
+import h5py
+with h5py.File("{}", "r+") as f:
+    ts = f["memory/timestamps"]
+    ts[3] = 12345.0
+"#,
+        path.display()
+    );
+    let out = Command::new(python())
+        .args(["-c", &script])
+        .output()
+        .unwrap();
+    assert!(
+        out.status.success(),
+        "{}",
+        String::from_utf8_lossy(&out.stderr)
+    );
+
+    let r = HDF5Memory::verify(&path, &key.verifying_key()).unwrap();
+    assert!(r.signature_valid && !r.is_valid(), "{r:?}");
+    assert_eq!(r.changed_records, vec![3]);
+}

crates/clawhdf5-agent/tests/hnsw_integration.rs

@@ -234,3 +234,113 @@ fn quantized_index_setting_survives_a_reopen() {
     let reopened = HDF5Memory::open(&path).unwrap();
     assert!(reopened.config().quantized_index);
 }
+
+#[test]
+fn hnsw_parameters_are_configurable_and_persisted() {
+    // The graph degree and both candidate-list sizes used to be constants, so
+    // a deployment could not trade recall against memory or speed at all.
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("mem.h5");
+    let mut config = MemoryConfig::new(path.clone(), "agent", 16);
+    config.hnsw_m = 8;
+    config.hnsw_ef_construction = 32;
+    config.hnsw_ef_search = 128;
+    let mut mem = HDF5Memory::create(config).unwrap();
+
+    let mut seed = 99;
+    let vectors: Vec<Vec<f32>> = (0..300).map(|_| make_vector(&mut seed, 16)).collect();
+    for (i, v) in vectors.iter().enumerate() {
+        mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap();
+    }
+    // Still correct with a smaller graph: an exact match must rank first.
+    let top = mem.hybrid_search(&vectors[42], "", 1.0, 0.0, 1);
+    assert_eq!(top[0].index, 42);
+
+    mem.flush_wal().unwrap();
+    drop(mem);
+    let reopened = HDF5Memory::open(&path).unwrap();
+    assert_eq!(reopened.config().hnsw_m, 8);
+    assert_eq!(reopened.config().hnsw_ef_construction, 32);
+    assert_eq!(reopened.config().hnsw_ef_search, 128);
+}
+
+#[test]
+fn degenerate_hnsw_parameters_do_not_panic() {
+    // `clawhdf5-ann` asserts m >= 2, so a zero from a config file — or from a
+    // caller who assumed 0 meant "default" — would abort the process inside
+    // the index builder. The store clamps instead.
+    let dir = TempDir::new().unwrap();
+    let mut config = MemoryConfig::new(dir.path().join("mem.h5"), "agent", 8);
+    config.hnsw_m = 0;
+    config.hnsw_ef_construction = 0;
+    config.hnsw_ef_search = 1;
+    let mut mem = HDF5Memory::create(config).unwrap();
+
+    let mut seed = 5;
+    let vectors: Vec<Vec<f32>> = (0..50).map(|_| make_vector(&mut seed, 8)).collect();
+    for (i, v) in vectors.iter().enumerate() {
+        mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap();
+    }
+    let results = mem.hybrid_search(&vectors[7], "", 1.0, 0.0, 5);
+    assert_eq!(results[0].index, 7, "exact match should still rank first");
+}
+
+#[test]
+fn new_stores_default_to_the_quantized_index() {
+    // int8 is the default because it is smaller and, with an exact re-score,
+    // faster at equal recall on every platform measured (see BENCHMARKS.md).
+    let dir = TempDir::new().unwrap();
+    let config = MemoryConfig::new(dir.path().join("mem.h5"), "agent", 8);
+    assert!(config.quantized_index);
+
+    let path = config.path.clone();
+    let mut mem = HDF5Memory::create(config).unwrap();
+    let mut seed = 3;
+    let vectors: Vec<Vec<f32>> = (0..40).map(|_| make_vector(&mut seed, 8)).collect();
+    for (i, v) in vectors.iter().enumerate() {
+        mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap();
+    }
+    assert_eq!(
+        mem.hybrid_search(&vectors[11], "", 1.0, 0.0, 1)[0].index,
+        11
+    );
+    mem.flush_wal().unwrap();
+    drop(mem);
+    assert!(HDF5Memory::open(&path).unwrap().config().quantized_index);
+}
+
+#[test]
+fn a_store_written_before_the_setting_existed_stays_f32() {
+    // `store_v2_5_0.h5` was written by the v2.5.0 CLI, before
+    // `quantized_index` or the HNSW parameters were persisted, so it carries
+    // none of them. Flipping the default for new stores must not reach back
+    // and change how an existing store's index is held.
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("legacy.h5");
+    std::fs::copy(
+        concat!(
+            env!("CARGO_MANIFEST_DIR"),
+            "/tests/fixtures/store_v2_5_0.h5"
+        ),
+        &path,
+    )
+    .unwrap();
+
+    let bytes = std::fs::read(&path).unwrap();
+    assert!(
+        !bytes.windows(15).any(|w| w == b"quantized_index"),
+        "the fixture must predate the setting, or it tests nothing"
+    );
+
+    let mut mem = HDF5Memory::open(&path).unwrap();
+    assert!(
+        !mem.config().quantized_index,
+        "an old store must reopen with an f32 index"
+    );
+    assert_eq!(mem.config().hnsw_m, 16);
+    assert_eq!(mem.config().hnsw_ef_construction, 64);
+    assert_eq!(mem.count(), 6);
+    // And it still searches: entry 3's own embedding finds it first.
+    let hit = mem.hybrid_search(&[3.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "", 1.0, 0.0, 1);
+    assert_eq!(hit[0].index, 3);
+}

crates/clawhdf5-agent/tests/search_options.rs

@@ -0,0 +1,344 @@
+//! `HDF5Memory::search` with `SearchOptions`: source filtering, re-ranking and
+//! confidence rejection in the store's own search path.
+
+use std::collections::HashSet;
+
+use clawhdf5_agent::confidence::ConfidenceConfig;
+use clawhdf5_agent::reranker::ReRankConfig;
+use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions, hybrid};
+use tempfile::TempDir;
+
+const DIM: usize = 32;
+const N: usize = 3000;
+const CLUSTERS: usize = 20;
+
+struct Rng(u64);
+impl Rng {
+    fn next(&mut self) -> u64 {
+        self.0 = self.0.wrapping_add(0x9E37_79B9_7F4A_7C15);
+        let mut z = self.0;
+        z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
+        z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
+        z ^ (z >> 31)
+    }
+    fn unit(&mut self) -> f32 {
+        (self.next() >> 40) as f32 / (1u64 << 24) as f32 - 0.5
+    }
+}
+
+fn normalize(v: &mut [f32]) {
+    let n = v.iter().map(|x| x * x).sum::<f32>().sqrt();
+    v.iter_mut().for_each(|x| *x /= n);
+}
+
+struct Data {
+    vectors: Vec<Vec<f32>>,
+    cluster: Vec<usize>,
+    centres: Vec<Vec<f32>>,
+}
+
+fn data() -> Data {
+    let mut rng = Rng(42);
+    let centres: Vec<Vec<f32>> = (0..CLUSTERS)
+        .map(|_| {
+            let mut c: Vec<f32> = (0..DIM).map(|_| rng.unit()).collect();
+            normalize(&mut c);
+            c
+        })
+        .collect();
+    let mut vectors = Vec::new();
+    let mut cluster = Vec::new();
+    for i in 0..N {
+        let c = i % CLUSTERS;
+        let mut v: Vec<f32> = centres[c].iter().map(|x| x + rng.unit() * 0.3).collect();
+        normalize(&mut v);
+        vectors.push(v);
+        cluster.push(c);
+    }
+    Data {
+        vectors,
+        cluster,
+        centres,
+    }
+}
+
+/// Channel of record `i` for a filter keeping `percent`% of the store at
+/// random (independent of the vectors).
+fn random_channel(i: usize, rng_seed: u64, percent: u64) -> String {
+    let mut r = Rng(rng_seed ^ (i as u64 * 7919));
+    if r.next() % 100 < percent {
+        "keep".into()
+    } else {
+        "other".into()
+    }
+}
+
+fn build(data: &Data, channel: impl Fn(usize) -> String) -> (TempDir, HDF5Memory) {
+    let dir = TempDir::new().unwrap();
+    let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
+    cfg.hebbian_boost = 0.0; // every query sees the same store
+    let mut m = HDF5Memory::create(cfg).unwrap();
+    let entries = data
+        .vectors
+        .iter()
+        .enumerate()
+        .map(|(i, v)| MemoryEntry {
+            chunk: format!("record {i} cluster {}", data.cluster[i]),
+            embedding: v.clone(),
+            source_channel: channel(i),
+            timestamp: i as f64,
+            session_id: "s".into(),
+            tags: format!("t{i}"),
+        })
+        .collect();
+    m.save_batch(entries).unwrap();
+    (dir, m)
+}
+
+/// Exact top-k by cosine among the records `allowed` keeps.
+fn exact_top(data: &Data, q: &[f32], k: usize, allowed: impl Fn(usize) -> bool) -> Vec<usize> {
+    let mut s: Vec<(usize, f32)> = (0..N)
+        .filter(|&i| allowed(i))
+        .map(|i| (i, data.vectors[i].iter().zip(q).map(|(a, b)| a * b).sum()))
+        .collect();
+    s.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
+    s.into_iter().take(k).map(|(i, _)| i).collect()
+}
+
+fn query(data: &Data, i: usize) -> Vec<f32> {
+    let mut rng = Rng(1000 + i as u64);
+    let mut q: Vec<f32> = data.centres[i % CLUSTERS]
+        .iter()
+        .map(|x| x + rng.unit() * 0.3)
+        .collect();
+    normalize(&mut q);
+    q
+}
+
+fn vector_only(k: usize) -> SearchOptions {
+    SearchOptions::new(k).with_fusion(hybrid::Fusion::Weighted {
+        vector: 1.0,
+        keyword: 0.0,
+    })
+}
+
+#[test]
+fn source_filter_returns_only_allowed_records_and_a_full_page() {
+    let d = data();
+    // At N = 3000 and k = 10 the index serves a filter only when that is
+    // cheaper than scanning the allowed records: pool = 80 * N / allowed
+    // candidates at ~M = 16 distances each, against `allowed` distances. So
+    // 90% goes through the index, 50% and 1% to the exact scan.
+    for percent in [90, 50, 1] {
+        let (_dir, mut m) = build(&d, |i| random_channel(i, 5, percent));
+        let allowed = |i: usize| random_channel(i, 5, percent) == "keep";
+        let mut hits = 0;
+        for qi in 0..40 {
+            let q = query(&d, qi);
+            let got = m.search(&q, "", &vector_only(10).with_sources(["keep"]));
+            assert_eq!(got.len(), 10, "{percent}%: short page");
+            assert!(got.iter().all(|r| r.source_channel == "keep"));
+            let want: HashSet<usize> = exact_top(&d, &q, 10, allowed).into_iter().collect();
+            hits += got.iter().filter(|r| want.contains(&r.index)).count();
+        }
+        let recall = hits as f64 / 400.0;
+        let floor = if percent == 90 { 0.95 } else { 1.0 };
+        assert!(recall >= floor, "{percent}%: recall@10 {recall}");
+    }
+}
+
+#[test]
+fn filter_away_from_the_query_falls_back_to_an_exact_scan() {
+    // Channel = cluster, and the filter keeps two clusters (10% of the
+    // store) that are not the query's: the index's neighbourhood of the
+    // query holds none of them. The search must still return the exact
+    // top 10 among the allowed records, not a short or empty page.
+    let d = data();
+    let (_dir, mut m) = build(&d, |i| format!("c{}", d.cluster[i]));
+    for qi in 0..20 {
+        let q = query(&d, qi);
+        let a = format!("c{}", (qi + 7) % CLUSTERS);
+        let b = format!("c{}", (qi + 13) % CLUSTERS);
+        let got: Vec<usize> = m
+            .search(
+                &q,
+                "",
+                &vector_only(10).with_sources([a.clone(), b.clone()]),
+            )
+            .iter()
+            .map(|r| r.index)
+            .collect();
+        let want = exact_top(&d, &q, 10, |i| {
+            let c = format!("c{}", d.cluster[i]);
+            c == a || c == b
+        });
+        assert_eq!(got, want, "query {qi}");
+    }
+}
+
+#[test]
+fn filter_edge_cases() {
+    let d = data();
+    let (_dir, mut m) = build(&d, |i| random_channel(i, 9, 50));
+    let q = query(&d, 0);
+    assert!(
+        m.search(
+            &q,
+            "cluster",
+            &SearchOptions::new(10).with_sources(Vec::<String>::new())
+        )
+        .is_empty()
+    );
+    assert!(
+        m.search(
+            &q,
+            "cluster",
+            &SearchOptions::new(10).with_sources(["nope"])
+        )
+        .is_empty()
+    );
+    // Keyword matches from other channels are filtered too.
+    let got = m.search(
+        &q,
+        "record cluster",
+        &SearchOptions::new(50).with_sources(["keep"]),
+    );
+    assert_eq!(got.len(), 50);
+    assert!(got.iter().all(|r| r.source_channel == "keep"));
+    // Deleted records never come back, filtered or not.
+    let first = got[0].index;
+    m.delete(first).unwrap();
+    let again = m.search(
+        &q,
+        "record cluster",
+        &SearchOptions::new(50).with_sources(["keep"]),
+    );
+    assert!(again.iter().all(|r| r.index != first));
+}
+
+#[test]
+fn plain_options_equal_hybrid_search_with() {
+    // Two identical stores, so neither query sees the other's boosts.
+    let d = data();
+    let (_a, mut a) = build(&d, |i| random_channel(i, 3, 50));
+    let (_b, mut b) = build(&d, |i| random_channel(i, 3, 50));
+    for qi in 0..10 {
+        let q = query(&d, qi);
+        let x: Vec<(usize, u32)> = a
+            .search(&q, "record cluster 3", &SearchOptions::new(10))
+            .iter()
+            .map(|r| (r.index, r.score.to_bits()))
+            .collect();
+        let y: Vec<(usize, u32)> = b
+            .hybrid_search_with(&q, "record cluster 3", hybrid::DEFAULT_FUSION, 10)
+            .iter()
+            .map(|r| (r.index, r.score.to_bits()))
+            .collect();
+        assert_eq!(x, y);
+    }
+}
+
+fn small_store(entries: &[(&str, &str, f64)]) -> (TempDir, HDF5Memory) {
+    let dir = TempDir::new().unwrap();
+    let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("r.h5"), "a", 4)).unwrap();
+    m.save_batch(
+        entries
+            .iter()
+            .map(|(chunk, channel, ts)| MemoryEntry {
+                chunk: chunk.to_string(),
+                embedding: vec![1.0, 0.0, 0.0, 0.0],
+                source_channel: channel.to_string(),
+                timestamp: *ts,
+                session_id: "s".into(),
+                tags: String::new(),
+            })
+            .collect(),
+    )
+    .unwrap();
+    (dir, m)
+}
+
+#[test]
+fn rerank_breaks_relevance_ties_by_recency() {
+    // Identical text and vectors, so retrieval ties; re-ranking must put the
+    // newer record first and report the combined score.
+    let now = 1_000_000.0;
+    let (_d, mut m) = small_store(&[
+        ("user prefers dark mode", "chat", now - 30.0 * 86_400.0),
+        ("user prefers dark mode", "chat", now - 60.0),
+    ]);
+    let q = [1.0, 0.0, 0.0, 0.0];
+    let plain = m.search(&q, "dark mode", &SearchOptions::new(2));
+    assert_eq!(plain[0].index, 0, "ties break by index without re-ranking");
+    let reranked = m.search(
+        &q,
+        "dark mode",
+        &SearchOptions::new(2)
+            .with_rerank(ReRankConfig::default())
+            .at_time(now),
+    );
+    assert_eq!(reranked[0].index, 1);
+    assert!(reranked[0].score > reranked[1].score);
+    assert_ne!(reranked[0].score.to_bits(), plain[0].score.to_bits());
+}
+
+#[test]
+fn confidence_rejects_when_nothing_is_good_enough() {
+    let (_d, mut m) = small_store(&[("alpha", "chat", 0.0), ("beta", "chat", 0.0)]);
+    let q = [1.0, 0.0, 0.0, 0.0];
+    let strict = ConfidenceConfig {
+        min_score: 10.0,
+        ..ConfidenceConfig::default()
+    };
+    assert!(
+        m.search(&q, "alpha", &SearchOptions::new(2).with_confidence(strict))
+            .is_empty()
+    );
+    let lenient = ConfidenceConfig {
+        min_score: 0.0,
+        min_gap: f32::INFINITY,
+        max_results: 1,
+    };
+    assert_eq!(
+        m.search(&q, "alpha", &SearchOptions::new(2).with_confidence(lenient))
+            .len(),
+        1
+    );
+}
+
+#[test]
+fn only_returned_results_are_reinforced() {
+    // With re-ranking, a pool of max(3k, 10) candidates is retrieved; only
+    // the k returned should gain activation.
+    let d = data();
+    let dir = TempDir::new().unwrap();
+    let path = dir.path().join("h.h5");
+    let mut m = HDF5Memory::create(MemoryConfig::new(path, "a", DIM)).unwrap();
+    m.save_batch(
+        (0..200)
+            .map(|i| MemoryEntry {
+                chunk: format!("record {i}"),
+                embedding: d.vectors[i].clone(),
+                source_channel: "chat".into(),
+                timestamp: i as f64,
+                session_id: "s".into(),
+                tags: String::new(),
+            })
+            .collect(),
+    )
+    .unwrap();
+    let q = query(&d, 0);
+    let got = m.search(
+        &q,
+        "record",
+        &SearchOptions::new(3).with_rerank(ReRankConfig::default()),
+    );
+    assert_eq!(got.len(), 3);
+    let returned: HashSet<usize> = got.iter().map(|r| r.index).collect();
+    // A second plain search reports each record's current activation.
+    let all = m.search(&q, "record", &SearchOptions::new(200));
+    for r in &all {
+        let boosted = r.activation > 1.0;
+        assert_eq!(boosted, returned.contains(&r.index), "record {}", r.index);
+    }
+}

crates/clawhdf5-agent/tests/signed_store.rs

@@ -0,0 +1,330 @@
+//! Ed25519-signed checkpoints: `HDF5Memory::set_signing_key` and
+//! `HDF5Memory::verify`.
+
+use std::path::Path;
+
+use clawhdf5_agent::signing::{SigningKey, VerifyReport, VerifyingKey};
+use clawhdf5_agent::storage;
+use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError, schema};
+use tempfile::TempDir;
+
+const DIM: usize = 16;
+
+fn key(seed: u8) -> SigningKey {
+    SigningKey::from_bytes(&[seed; 32])
+}
+
+fn entry(i: usize, chunk: &str) -> MemoryEntry {
+    MemoryEntry {
+        chunk: chunk.to_string(),
+        embedding: (0..DIM)
+            .map(|j| ((i * DIM + j) as f32 * 0.37).sin())
+            .collect(),
+        source_channel: "chat".into(),
+        timestamp: 1_700_000_000.0 + i as f64,
+        session_id: format!("s{}", i % 3),
+        tags: format!("t{i}"),
+    }
+}
+
+/// Awkward strings on purpose: they must hash the same after a round trip.
+const TEXTS: [&str; 6] = [
+    "plain text",
+    "ünïcödé — 日本語 🙂",
+    "",
+    "trailing spaces   ",
+    "tab\tand\nnewline",
+    "x",
+];
+
+fn signed_store(dir: &TempDir, float16: bool, k: &SigningKey) -> std::path::PathBuf {
+    let mut cfg = MemoryConfig::new(dir.path().join("s.h5"), "agent", DIM);
+    cfg.float16 = float16;
+    let path = cfg.path.clone();
+    let mut m = HDF5Memory::create(cfg).unwrap();
+    m.set_signing_key(k.clone());
+    let entries = (0..30).map(|i| entry(i, TEXTS[i % TEXTS.len()])).collect();
+    m.save_batch(entries).unwrap();
+    // Some graph and a deleted record, so every part of the manifest is used.
+    let a = m.knowledge_mut().add_entity("Alice", "person", 0);
+    let b = m.knowledge_mut().add_entity("Acme", "org", -1);
+    m.knowledge_mut().add_relation(a, b, "works_at", 0.75);
+    m.sessions_mut()
+        .add_at("s0", 0, 9, "chat", "first session", 1_700_000_000.0);
+    m.delete(4).unwrap();
+    m.flush_wal().unwrap();
+    path
+}
+
+fn verify(path: &Path, k: &SigningKey) -> VerifyReport {
+    HDF5Memory::verify(path, &k.verifying_key()).unwrap()
+}
+
+#[test]
+fn a_signed_store_verifies_through_reopen_and_checkpoint_cycles() {
+    for float16 in [true, false] {
+        let dir = TempDir::new().unwrap();
+        let k = key(7);
+        let path = signed_store(&dir, float16, &k);
+        let r = verify(&path, &k);
+        assert!(r.is_valid(), "float16={float16}: {r:?}");
+        assert_eq!(r.public_key, Some(k.verifying_key().to_bytes()));
+        assert_eq!(r.record_count, 30);
+        assert!(r.changed_records.is_empty());
+
+        // Reopen, change nothing, checkpoint again (with the key): still valid.
+        for _ in 0..3 {
+            let mut m = HDF5Memory::open(&path).unwrap();
+            assert!(m.is_signed());
+            m.set_signing_key(k.clone());
+            m.flush_wal().unwrap();
+            drop(m);
+            assert!(verify(&path, &k).is_valid());
+        }
+        // And after real changes, re-signed.
+        let mut m = HDF5Memory::open(&path).unwrap();
+        m.set_signing_key(k.clone());
+        m.save(entry(99, "added later")).unwrap();
+        m.hybrid_search(&entry(1, "").embedding, "text", 0.4, 0.6, 5);
+        m.flush_wal().unwrap();
+        drop(m);
+        let r = verify(&path, &k);
+        assert!(r.is_valid(), "{r:?}");
+        assert_eq!(r.record_count, 31);
+    }
+}
+
+#[test]
+fn a_signed_store_refuses_to_checkpoint_without_its_key() {
+    let dir = TempDir::new().unwrap();
+    let k = key(1);
+    let path = signed_store(&dir, true, &k);
+
+    let mut m = HDF5Memory::open(&path).unwrap();
+    m.save(entry(50, "pending")).unwrap();
+    match m.flush_wal() {
+        Err(MemoryError::SigningKeyRequired(msg)) => assert!(msg.contains("signed"), "{msg}"),
+        other => panic!("expected SigningKeyRequired, got {other:?}"),
+    }
+    // The file is untouched and still valid; the save is still in the WAL.
+    let r = verify(&path, &k);
+    assert!(r.is_valid());
+    assert_eq!(r.wal_entries_unsigned, 1);
+
+    // Supplying the key lets the checkpoint through, signed.
+    m.set_signing_key(k.clone());
+    m.flush_wal().unwrap();
+    drop(m);
+    let r = verify(&path, &k);
+    assert!(r.is_valid());
+    assert_eq!((r.record_count, r.wal_entries_unsigned), (31, 0));
+
+    // Removing the signature on purpose writes it unsigned.
+    let mut m = HDF5Memory::open(&path).unwrap();
+    m.remove_signature();
+    m.flush_wal().unwrap();
+    drop(m);
+    let r = verify(&path, &k);
+    assert!(!r.signed && !r.is_valid());
+    assert!(!HDF5Memory::open(&path).unwrap().is_signed());
+}
+
+#[test]
+fn the_wrong_key_does_not_verify_and_a_new_key_re_signs() {
+    let dir = TempDir::new().unwrap();
+    let (a, b) = (key(1), key(2));
+    let path = signed_store(&dir, true, &a);
+    let r = verify(&path, &b);
+    assert!(r.signed && !r.key_matches && !r.signature_valid && !r.is_valid());
+
+    let mut m = HDF5Memory::open(&path).unwrap();
+    m.set_signing_key(b.clone());
+    m.flush_wal().unwrap();
+    drop(m);
+    assert!(verify(&path, &b).is_valid());
+    assert!(!verify(&path, &a).is_valid());
+}
+
+/// Rewrite the store with changed contents but the *old* signature — what
+/// someone with write access to the file, but not the key, can do.
+fn tamper(path: &Path, change: impl FnOnce(&mut Tampered)) {
+    let file = clawhdf5::File::open(path).unwrap();
+    let (config, cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
+    let checkpoint = schema::read_checkpoint_meta(&file);
+    let signature = schema::read_signature(&file).unwrap().unwrap();
+    drop(file);
+    let mut t = Tampered {
+        config,
+        cache,
+        sessions,
+        knowledge,
+    };
+    change(&mut t);
+    storage::write_to_disk_signed(
+        path,
+        &t.config,
+        &t.cache,
+        &t.sessions,
+        &t.knowledge,
+        &checkpoint,
+        Some(&signature),
+    )
+    .unwrap();
+}
+
+struct Tampered {
+    config: MemoryConfig,
+    cache: clawhdf5_agent::cache::MemoryCache,
+    sessions: clawhdf5_agent::SessionCache,
+    knowledge: clawhdf5_agent::knowledge::KnowledgeCache,
+}
+
+#[test]
+fn every_kind_of_edit_is_detected_and_located() {
+    let k = key(3);
+    type Edit = Box<dyn FnOnce(&mut Tampered)>;
+    type Case = (&'static str, Edit, fn(&VerifyReport) -> bool);
+    let cases: Vec<Case> = vec![
+        (
+            "record text",
+            Box::new(|t: &mut Tampered| t.cache.chunks[7] = "rewritten".into()),
+            |r| !r.records_match && r.changed_records == vec![7],
+        ),
+        (
+            "one embedding value",
+            Box::new(|t: &mut Tampered| {
+                let mut e = t.cache.embeddings[12].to_vec();
+                e[3] = 0.5;
+                t.cache.embeddings.set(12, &e);
+            }),
+            |r| r.changed_records == vec![12],
+        ),
+        (
+            "undelete",
+            Box::new(|t: &mut Tampered| t.cache.tombstones[4] = 0),
+            |r| r.changed_records == vec![4],
+        ),
+        (
+            "timestamp",
+            Box::new(|t: &mut Tampered| t.cache.timestamps[20] += 1.0),
+            |r| r.changed_records == vec![20],
+        ),
+        (
+            "record appended",
+            Box::new(|t: &mut Tampered| {
+                t.cache.push(
+                    "new".into(),
+                    vec![0.1; DIM],
+                    "x".into(),
+                    1.0,
+                    "s".into(),
+                    "".into(),
+                );
+            }),
+            |r| !r.records_match && r.changed_records == vec![30] && r.record_count == 31,
+        ),
+        (
+            "setting",
+            Box::new(|t: &mut Tampered| t.config.agent_id = "someone-else".into()),
+            |r| !r.settings_match && r.records_match,
+        ),
+        (
+            "session summary",
+            Box::new(|t: &mut Tampered| t.sessions.summaries[0] = "edited".into()),
+            |r| !r.sessions_match && r.records_match,
+        ),
+        (
+            "graph edge",
+            Box::new(|t: &mut Tampered| t.knowledge.relations[0].weight = 1.0),
+            |r| !r.graph_match && r.records_match,
+        ),
+    ];
+    for (name, edit, check) in cases {
+        let dir = TempDir::new().unwrap();
+        let path = signed_store(&dir, true, &k);
+        tamper(&path, edit);
+        let r = verify(&path, &k);
+        assert!(
+            r.signed && r.key_matches && r.signature_valid,
+            "{name}: {r:?}"
+        );
+        assert!(!r.is_valid(), "{name}: edit not detected: {r:?}");
+        assert!(check(&r), "{name}: {r:?}");
+    }
+}
+
+#[test]
+fn a_forged_manifest_fails_the_signature() {
+    // Recomputing the hashes for tampered contents does not help without the
+    // key: the signature no longer matches the manifest.
+    let dir = TempDir::new().unwrap();
+    let k = key(5);
+    let path = signed_store(&dir, true, &k);
+    let file = clawhdf5::File::open(&path).unwrap();
+    let (config, mut cache, sessions, knowledge) = schema::validate_and_load(&file).unwrap();
+    let checkpoint = schema::read_checkpoint_meta(&file);
+    let mut sig = schema::read_signature(&file).unwrap().unwrap();
+    drop(file);
+    cache.chunks[0] = "forged".into();
+    // Re-sign with an attacker key, then splice the victim's public key back.
+    let forged = clawhdf5_agent::signing::sign(
+        &key(66),
+        &config,
+        &cache,
+        &sessions,
+        &knowledge,
+        checkpoint.wal_applied,
+    );
+    sig.manifest = forged.manifest;
+    sig.record_hashes = forged.record_hashes;
+    storage::write_to_disk_signed(
+        &path,
+        &config,
+        &cache,
+        &sessions,
+        &knowledge,
+        &checkpoint,
+        Some(&sig),
+    )
+    .unwrap();
+    let r = verify(&path, &k);
+    assert!(
+        r.key_matches && !r.signature_valid && !r.is_valid(),
+        "{r:?}"
+    );
+}
+
+#[test]
+fn an_unsigned_store_reports_unsigned() {
+    let dir = TempDir::new().unwrap();
+    let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("u.h5"), "a", DIM)).unwrap();
+    m.save_batch(vec![entry(0, "hello")]).unwrap();
+    drop(m);
+    let r = HDF5Memory::verify(&dir.path().join("u.h5"), &VerifyingKey::from(&key(1))).unwrap();
+    assert!(!r.signed && !r.is_valid());
+    assert_eq!(r.record_count, 1);
+}
+
+#[test]
+fn nul_bytes_in_text_still_verify() {
+    // Strings are stored null-padded; the hash must follow what a reopened
+    // store actually holds, or an untouched store would fail to verify.
+    let dir = TempDir::new().unwrap();
+    let k = key(9);
+    let mut m = HDF5Memory::create(MemoryConfig::new(dir.path().join("n.h5"), "a", DIM)).unwrap();
+    m.set_signing_key(k.clone());
+    m.save_batch(vec![
+        entry(0, "inner\0nul"),
+        entry(1, "trailing nul\0"),
+        entry(2, "\0leading"),
+    ])
+    .unwrap();
+    drop(m);
+    let r = verify(&dir.path().join("n.h5"), &k);
+    assert!(r.is_valid(), "{r:?}");
+    let m = HDF5Memory::open(&dir.path().join("n.h5")).unwrap();
+    eprintln!(
+        "reloaded: {:?}",
+        (0..3).map(|i| m.get_chunk(i)).collect::<Vec<_>>()
+    );
+}

crates/clawhdf5-android/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-android"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "Android JNI bridge for edgehdf5-memory HDF5 backend"
 license = "MIT"
 

crates/clawhdf5-ann/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-ann"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "HNSW approximate nearest neighbor index stored as HDF5"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -10,9 +11,9 @@ keywords = ["hdf5", "ann", "hnsw", "nearest-neighbor"]
 categories = ["algorithms", "science"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.6.0" }
-clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.6.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0" }
+clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.7.0" }
 rayon = { version = "1", optional = true }
 
 [features]

crates/clawhdf5-ann/src/hnsw.rs

@@ -358,31 +358,12 @@ enum Query {
     Int8(Vec<i8>, f32),
 }
 
-/// Sum of products, widened so it cannot overflow: `dim` terms of at most
-/// `127 * 127`, so `i32` suffices for any realistic dimension.
+/// Sum of products, widened so it cannot overflow. Runtime-dispatched to the
+/// same SIMD backend as the f32 kernels, so the two storages are compared on
+/// equal terms.
+#[inline]
 fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
-    // Four independent accumulators over 32-lane blocks: the widening product
-    // has to sit in a fixed-length chunk for the vectoriser to see it, and the
-    // separate accumulators keep it off one dependency chain.
-    const LANE: usize = 8;
-    let (a_blocks, a_tail) = a.as_chunks::<{ LANE * 4 }>();
-    let (b_blocks, b_tail) = b.as_chunks::<{ LANE * 4 }>();
-    let mut acc = [0i32; 4];
-    for (x, y) in a_blocks.iter().zip(b_blocks) {
-        for (lane, slot) in acc.iter_mut().enumerate() {
-            let mut sum = 0i32;
-            for k in 0..LANE {
-                sum += i32::from(x[lane * LANE + k]) * i32::from(y[lane * LANE + k]);
-            }
-            *slot += sum;
-        }
-    }
-    let tail: i32 = a_tail
-        .iter()
-        .zip(b_tail)
-        .map(|(&x, &y)| i32::from(x) * i32::from(y))
-        .sum();
-    acc[0] + acc[1] + acc[2] + acc[3] + tail
+    clawhdf5_accel::dot_i8(a, b)
 }
 
 /// Magic for [`HnswIndex::graph_to_bytes`].

crates/clawhdf5-bench/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-bench"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "Benchmark harnesses for clawhdf5-agent (Track 8)"
 license = "MIT"
 

crates/clawhdf5-bench/src/bin/consolidation_efficiency.rs

@@ -396,7 +396,7 @@ fn run_memory_reduction_benchmark() {
     println!();
     println!(
         "{:>8}  {:>10}  {:>10}  {:>10}  {:>12}",
-        "Initial", "Remaining", "Eviction%", "Signal OK?", "BM25 Speedup"
+        "Initial", "Remaining", "Eviction%", "Signal OK?", "Records ÷"
     );
     println!("{}", "-".repeat(58));
 
@@ -440,7 +440,8 @@ fn run_memory_reduction_benchmark() {
         // Check all signal records survived
         let signal_survived = signal_ids.iter().all(|&id| engine.get_by_id(id).is_some());
 
-        // Rough speedup: BM25 scales roughly linearly with record count
+        // How many times fewer records there are. Not a measured speedup —
+        // Part 1 measures search latency before and after.
         let speedup = before_count as f64 / after_count.max(1) as f64;
 
         println!(
@@ -480,7 +481,7 @@ fn main() {
     println!("  3. Reducing search latency proportional to record reduction");
     println!();
     println!(
-        "Cycle time scales sub-linearly: 100 records ~microseconds, 100K records ~tens of ms."
+        "Cycle time grows a little faster than linearly: 100 records ~microseconds, 100K records ~tens of ms."
     );
     println!("Signal records with Correction source + high access_count survive eviction.");
 }

crates/clawhdf5-bench/src/bin/footprint_bench.rs

@@ -11,12 +11,14 @@
 //!
 //! Configuration matrix:
 //!   - Text lengths: short (50 chars), medium (200 chars), long (1000 chars)
-//!   - Embedding: 384-dim f32 (1536 bytes raw per record)
+//!   - Embedding: 384-dim, stored as float16 (the default for new stores) or
+//!     f32 with `--f32`; "raw" bytes are counted as f32 input either way
 //!   - WAL: enabled and disabled
 //!
 //! # Usage
 //! ```
-//! cargo run --release --bin footprint_bench
+//! cargo run --release --bin footprint_bench           # float16 stores
+//! cargo run --release --bin footprint_bench -- --f32  # f32 stores
 //! ```
 
 use std::time::Instant;
@@ -24,6 +26,9 @@ use std::time::Instant;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
 use tempfile::TempDir;
 
+/// `--f32`: build f32 stores instead of the library's float16 default.
+static F32: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
+
 const EMBEDDING_DIM: usize = 384;
 
 // Raw bytes per record: 384 f32 embeddings + median text + overhead
@@ -152,6 +157,9 @@ fn measure_footprint(
     config.compression = compression;
     config.compression_level = if compression { 6 } else { 0 };
     config.compact_threshold = 0.0;
+    if F32.load(std::sync::atomic::Ordering::Relaxed) {
+        config.float16 = false;
+    }
 
     let mut memory = HDF5Memory::create(config).expect("HDF5Memory::create failed");
 
@@ -241,11 +249,19 @@ fn fmt_n(n: usize) -> String {
 // ---------------------------------------------------------------------------
 
 fn main() {
+    if std::env::args().skip(1).any(|a| a == "--f32") {
+        F32.store(true, std::sync::atomic::Ordering::Relaxed);
+    }
+    let stored = if F32.load(std::sync::atomic::Ordering::Relaxed) {
+        "f32 (1,536 bytes per record)"
+    } else {
+        "float16 (768 bytes per record; the default for new stores)"
+    };
     println!("=================================================================");
     println!("  ClawhDF5 Memory Footprint Benchmark");
     println!("=================================================================");
     println!();
-    println!("Embedding: 384-dim f32 = 1,536 bytes raw per record");
+    println!("Embedding: 384-dim, stored as {stored}; raw input counted as f32");
     println!("Text lengths: short=50 chars, medium=200 chars, long=1000 chars");
     println!();
 

crates/clawhdf5-bench/src/bin/longmemeval_bench.rs

@@ -64,6 +64,11 @@ use tempfile::TempDir;
 
 const EMBEDDING_DIM: usize = 384;
 
+/// `--float16`: build every per-question store with `MemoryConfig::float16`,
+/// so embeddings are rounded to half precision as they are saved — exactly
+/// what such a store searches over.
+static FLOAT16: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
+
 /// A mode's fusion, as one short string for the reports.
 fn describe(mode: Mode) -> String {
     let fusion = match mode.fusion {
@@ -431,6 +436,7 @@ fn evaluate_question(
     let mut config = MemoryConfig::new(dir.path().join("lme.h5"), "lme-bench", EMBEDDING_DIM);
     config.wal_enabled = false;
     config.compact_threshold = 0.0;
+    config.float16 = FLOAT16.load(std::sync::atomic::Ordering::Relaxed);
 
     let mut memory = HDF5Memory::create(config).expect("failed to create HDF5Memory");
     memory.set_token_filter(mode.tokens);
@@ -940,6 +946,10 @@ fn main() {
                 limit = Some(v.parse().expect("--limit must be a positive integer"));
             }
             "--sweep" => sweep = true,
+            "--float16" => {
+                FLOAT16.store(true, std::sync::atomic::Ordering::Relaxed);
+                eprintln!("Stores use MemoryConfig::float16 (half-precision embeddings)");
+            }
             "--rerank-sweep" => {
                 // Re-ranking needs the vector stage to have candidates worth
                 // reordering, so this is an embeddings-only comparison.
@@ -971,6 +981,9 @@ fn main() {
                      --rerank-sweep\n\
                               compare re-ranking off, metadata-only (the old\n\
                               behaviour) and blended at several half-lives.\n\
+                     --float16\n\
+                              build each store with MemoryConfig::float16, to\n\
+                              compare retrieval on half-precision embeddings.\n\
                      --sweep  instead of the three named modes, sweep vector_weight\n\
                               from 0.0 to 1.0 in 0.1 steps. The 0.7/0.3 default was\n\
                               never searched; this is what searches it."

crates/clawhdf5-bench/src/bin/search_harness.rs

@@ -19,6 +19,9 @@
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --full  # + 100K
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --json out.json
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
+//! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
+//! cargo run --release -p clawhdf5-bench --bin search_harness -- --options-study --full
+//! cargo run --release -p clawhdf5-bench --bin search_harness -- --signing-study --full
 //! ```
 
 use std::time::{Duration, Instant};
@@ -88,6 +91,9 @@ static UNIFORM: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::n
 /// the memory) instead of f32, to price the recall it costs.
 static INT8: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
 
+/// `--f16-first`: in `--float16-study`, run the float16 store first.
+static F16_FIRST: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
+
 /// `--rerank`: re-score the candidate pool against the exact vectors before
 /// taking the top K.
 static RERANK: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
@@ -230,13 +236,27 @@ struct CountingAllocator;
 
 static LIVE_BYTES: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0);
 
+/// High-water mark of [`LIVE_BYTES`] since it was last reset.
+///
+/// Live bytes at a checkpoint cannot see a buffer that was allocated and
+/// freed in between, and that is exactly the shape of a transient copy —
+/// which still has to fit in memory while it exists.
+static PEAK_BYTES: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0);
+
+fn note_peak(live: i64) {
+    PEAK_BYTES.fetch_max(live, std::sync::atomic::Ordering::Relaxed);
+}
+
 // SAFETY: every method forwards to the system allocator with the same layout
 // it was given, and only adds bookkeeping around it.
 unsafe impl std::alloc::GlobalAlloc for CountingAllocator {
     unsafe fn alloc(&self, layout: std::alloc::Layout) -> *mut u8 {
         let ptr = unsafe { std::alloc::System.alloc(layout) };
         if !ptr.is_null() {
-            LIVE_BYTES.fetch_add(layout.size() as i64, std::sync::atomic::Ordering::Relaxed);
+            let live = LIVE_BYTES
+                .fetch_add(layout.size() as i64, std::sync::atomic::Ordering::Relaxed)
+                + layout.size() as i64;
+            note_peak(live);
         }
         ptr
     }
@@ -249,10 +269,9 @@ unsafe impl std::alloc::GlobalAlloc for CountingAllocator {
     unsafe fn realloc(&self, ptr: *mut u8, layout: std::alloc::Layout, new_size: usize) -> *mut u8 {
         let new_ptr = unsafe { std::alloc::System.realloc(ptr, layout, new_size) };
         if !new_ptr.is_null() {
-            LIVE_BYTES.fetch_add(
-                new_size as i64 - layout.size() as i64,
-                std::sync::atomic::Ordering::Relaxed,
-            );
+            let delta = new_size as i64 - layout.size() as i64;
+            let live = LIVE_BYTES.fetch_add(delta, std::sync::atomic::Ordering::Relaxed) + delta;
+            note_peak(live);
         }
         new_ptr
     }
@@ -266,6 +285,19 @@ fn heap_bytes() -> u64 {
     LIVE_BYTES.load(std::sync::atomic::Ordering::Relaxed).max(0) as u64
 }
 
+/// Start watching for a new high-water mark from the current live total.
+fn reset_peak() {
+    PEAK_BYTES.store(
+        LIVE_BYTES.load(std::sync::atomic::Ordering::Relaxed),
+        std::sync::atomic::Ordering::Relaxed,
+    );
+}
+
+/// The highest live total seen since [`reset_peak`].
+fn peak_bytes() -> u64 {
+    PEAK_BYTES.load(std::sync::atomic::Ordering::Relaxed).max(0) as u64
+}
+
 fn mib(bytes: u64) -> f64 {
     bytes as f64 / (1 << 20) as f64
 }
@@ -457,6 +489,394 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
     }));
 }
 
+// ---------------------------------------------------------------------------
+// Signing study: what does an Ed25519-signed checkpoint cost?
+// ---------------------------------------------------------------------------
+
+/// `--signing-study`: checkpoint time unsigned vs signed, `verify` time, and
+/// the file-size cost of the stored per-record hashes. Default store
+/// settings (float16, int8 index). Medians of five checkpoints / three
+/// verifies.
+fn signing_study(n: usize) {
+    use clawhdf5_agent::signing::SigningKey;
+    let data = make_dataset(n, 0x516 ^ n as u64);
+    let mut rng = Rng(9);
+    let entries: Vec<MemoryEntry> = data
+        .vectors
+        .iter()
+        .enumerate()
+        .map(|(i, v)| MemoryEntry {
+            chunk: text_for(data.cluster_of[i], i, &mut rng),
+            embedding: v.clone(),
+            source_channel: "bench".into(),
+            timestamp: i as f64,
+            session_id: format!("s{}", i % 50),
+            tags: format!("t{i}"),
+        })
+        .collect();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("sign.h5");
+    let mut mem = HDF5Memory::create(MemoryConfig::new(path.clone(), "bench", DIM)).unwrap();
+    mem.save_batch(entries).unwrap();
+    std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
+
+    let median = |mut v: Vec<Duration>| {
+        v.sort();
+        v[v.len() / 2]
+    };
+    let checkpoint = |mem: &mut HDF5Memory| {
+        median(
+            (0..5)
+                .map(|_| {
+                    let t = Instant::now();
+                    mem.flush_wal().unwrap();
+                    t.elapsed()
+                })
+                .collect(),
+        )
+    };
+    let unsigned = checkpoint(&mut mem);
+    let unsigned_bytes = std::fs::metadata(&path).unwrap().len();
+    let key = SigningKey::from_bytes(&[7; 32]);
+    mem.set_signing_key(key.clone());
+    let signed = checkpoint(&mut mem);
+    let signed_bytes = std::fs::metadata(&path).unwrap().len();
+    drop(mem);
+    let vk = key.verifying_key();
+    let verify = median(
+        (0..3)
+            .map(|_| {
+                let t = Instant::now();
+                let r = HDF5Memory::verify(&path, &vk).unwrap();
+                let d = t.elapsed();
+                assert!(r.is_valid());
+                d
+            })
+            .collect(),
+    );
+    println!(
+        "| {n} | {:.1} | {:.1} | {:+.1} | {:.1} | {:+.2} |",
+        millis(unsigned),
+        millis(signed),
+        millis(signed) - millis(unsigned),
+        millis(verify),
+        (signed_bytes as f64 - unsigned_bytes as f64) / (1024.0 * 1024.0),
+    );
+}
+
+// ---------------------------------------------------------------------------
+// Search options study: source filters, re-ranking, confidence rejection
+// ---------------------------------------------------------------------------
+
+/// `--options-study`: what `HDF5Memory::search`'s options cost and whether a
+/// filtered search finds the right records. Filters keep 50%, 10% or 1% of
+/// the store at random, or two whole clusters away from the query (the case
+/// the index cannot serve, which falls back to an exact scan). Recall is
+/// vector-only against an exact scan of the allowed records; latency is full
+/// hybrid search. Hebbian boosting is off.
+fn options_study(n: usize) {
+    use clawhdf5_agent::SearchOptions;
+    use clawhdf5_agent::confidence::ConfidenceConfig;
+    use clawhdf5_agent::hybrid::Fusion;
+    use clawhdf5_agent::reranker::ReRankConfig;
+
+    let data = make_dataset(n, 0x0B7 ^ n as u64);
+    let n_clusters = data.cluster_of.iter().max().map_or(1, |m| m + 1);
+    let mut rng = Rng(5);
+    let bucket_of: Vec<usize> = (0..n).map(|_| rng.below(100)).collect();
+    let bucket = &bucket_of;
+    let query_texts: Vec<String> = data
+        .query_cluster
+        .iter()
+        .enumerate()
+        .map(|(i, c)| text_for(*c, i, &mut rng))
+        .collect();
+    let exact_top = |q: &[f32], allowed: &dyn Fn(usize) -> bool| -> Vec<usize> {
+        let mut s: Vec<(usize, f32)> = (0..n)
+            .filter(|&i| allowed(i))
+            .map(|i| (i, data.vectors[i].iter().zip(q).map(|(a, b)| a * b).sum()))
+            .collect();
+        s.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
+        s.into_iter().take(K).map(|(i, _)| i).collect()
+    };
+
+    // Two stores: channel = random bucket, and channel = cluster.
+    let dir = tempfile::tempdir().unwrap();
+    let mut stores = Vec::new();
+    for by_cluster in [false, true] {
+        let mut rng = Rng(3);
+        let entries: Vec<MemoryEntry> = data
+            .vectors
+            .iter()
+            .enumerate()
+            .map(|(i, v)| MemoryEntry {
+                chunk: text_for(data.cluster_of[i], i, &mut rng),
+                embedding: v.clone(),
+                source_channel: if by_cluster {
+                    format!("c{}", data.cluster_of[i])
+                } else {
+                    format!("b{}", bucket[i])
+                },
+                timestamp: i as f64,
+                session_id: format!("s{}", i % 50),
+                tags: format!("t{i}"),
+            })
+            .collect();
+        let mut config = MemoryConfig::new(
+            dir.path().join(format!("opt_{by_cluster}.h5")),
+            "bench",
+            DIM,
+        );
+        config.hebbian_boost = 0.0;
+        let mut mem = HDF5Memory::create(config).unwrap();
+        mem.save_batch(entries).unwrap();
+        std::hint::black_box(mem.search(&data.queries[0], "", &SearchOptions::new(K)));
+        stores.push(mem);
+    }
+
+    let vector_only = SearchOptions::new(K).with_fusion(Fusion::Weighted {
+        vector: 1.0,
+        keyword: 0.0,
+    });
+    // (label, store, channels for query i, allowed(i, record))
+    type Case<'a> = (
+        String,
+        usize,
+        Box<dyn Fn(usize) -> Option<Vec<String>> + 'a>,
+        Box<dyn Fn(usize, usize) -> bool + 'a>,
+    );
+    let mut cases: Vec<Case> = vec![(
+        "no filter".into(),
+        0,
+        Box::new(|_| None),
+        Box::new(|_, _| true),
+    )];
+    for pct in [50usize, 10, 1] {
+        cases.push((
+            format!("random {pct}%"),
+            0,
+            Box::new(move |_| Some((0..pct).map(|b| format!("b{b}")).collect())),
+            Box::new(move |_, i| bucket[i] < pct),
+        ));
+    }
+    let d = &data;
+    let away = move |qi: usize| {
+        let qc = d.query_cluster[qi];
+        [
+            (qc + n_clusters / 3) % n_clusters,
+            (qc + 2 * n_clusters / 3) % n_clusters,
+        ]
+    };
+    cases.push((
+        "2 clusters away from the query".into(),
+        1,
+        Box::new(move |qi| Some(away(qi).iter().map(|c| format!("c{c}")).collect())),
+        Box::new(move |qi, i| away(qi).contains(&d.cluster_of[i])),
+    ));
+
+    for (label, store, channels, allowed) in &cases {
+        let mem = &mut stores[*store];
+        let mut hits = 0;
+        let mut kept = 0;
+        for (qi, q) in data.queries.iter().enumerate() {
+            let mut opts = vector_only.clone();
+            opts.source_channels = channels(qi);
+            let got = mem.search(q, "", &opts);
+            let want = exact_top(q, &|i| allowed(qi, i));
+            kept += want.len();
+            hits += got.iter().filter(|r| want.contains(&r.index)).count();
+        }
+        let latency = summarize(
+            (0..N_QUERIES)
+                .map(|qi| {
+                    let mut opts = SearchOptions::new(K);
+                    opts.source_channels = channels(qi);
+                    let t = Instant::now();
+                    std::hint::black_box(mem.search(&data.queries[qi], &query_texts[qi], &opts));
+                    t.elapsed()
+                })
+                .collect(),
+        );
+        println!(
+            "| {n} | {label} | {:.4} | {:.3} | {:.3} |",
+            hits as f64 / kept.max(1) as f64,
+            millis(latency.p50),
+            millis(latency.p99),
+        );
+    }
+
+    let mem = &mut stores[0];
+    for (label, opts) in [
+        (
+            "re-rank",
+            SearchOptions::new(K).with_rerank(ReRankConfig::default()),
+        ),
+        (
+            "re-rank + confidence",
+            SearchOptions::new(K)
+                .with_rerank(ReRankConfig::default())
+                .with_confidence(ConfidenceConfig::default()),
+        ),
+    ] {
+        let latency = summarize(
+            (0..N_QUERIES)
+                .map(|qi| {
+                    let t = Instant::now();
+                    std::hint::black_box(mem.search(&data.queries[qi], &query_texts[qi], &opts));
+                    t.elapsed()
+                })
+                .collect(),
+        );
+        println!(
+            "| {n} | {label} | — | {:.3} | {:.3} |",
+            millis(latency.p50),
+            millis(latency.p99)
+        );
+    }
+}
+
+// ---------------------------------------------------------------------------
+// float16 study: what does half-precision embedding storage cost?
+// ---------------------------------------------------------------------------
+
+/// `--float16-study`: the same data in an `f32` store and a `float16` store.
+/// Reports file size, checkpoint and open time, vector-search recall@10
+/// against an exact scan of the *original* f32 vectors, how often the two
+/// stores return the same top 10, and `hybrid_search` latency. Hebbian
+/// boosting is off, so every query sees the same store.
+fn float16_study(n: usize) {
+    let data = make_dataset(n, 0xF16 ^ n as u64);
+    let mut rng = Rng(11);
+    let query_texts: Vec<String> = data
+        .query_cluster
+        .iter()
+        .enumerate()
+        .map(|(i, c)| text_for(*c, i, &mut rng))
+        .collect();
+
+    // Exact top K by cosine (the vectors are unit length) on the f32 inputs.
+    let exact: Vec<Vec<usize>> = data
+        .queries
+        .iter()
+        .map(|q| {
+            let mut scored: Vec<(usize, f32)> = data
+                .vectors
+                .iter()
+                .enumerate()
+                .map(|(i, v)| (i, v.iter().zip(q).map(|(a, b)| a * b).sum()))
+                .collect();
+            scored.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
+            scored.into_iter().take(K).map(|(i, _)| i).collect()
+        })
+        .collect();
+
+    let dir = tempfile::tempdir().unwrap();
+    let mut per_variant: Vec<(bool, Vec<Vec<usize>>)> = Vec::new();
+    // `--f16-first` swaps the order, to check the numbers do not depend on
+    // which store runs first (page cache, allocator, CPU frequency).
+    let order = if F16_FIRST.load(std::sync::atomic::Ordering::Relaxed) {
+        [true, false]
+    } else {
+        [false, true]
+    };
+    for float16 in order {
+        let path = dir.path().join(format!("f16study_{float16}.h5"));
+        let mut rng = Rng(3);
+        let entries: Vec<MemoryEntry> = data
+            .vectors
+            .iter()
+            .enumerate()
+            .map(|(i, v)| MemoryEntry {
+                chunk: text_for(data.cluster_of[i], i, &mut rng),
+                embedding: v.clone(),
+                source_channel: "bench".into(),
+                timestamp: i as f64,
+                session_id: format!("s{}", i % 50),
+                tags: format!("t{i}"),
+            })
+            .collect();
+        let mut config = MemoryConfig::new(path.clone(), "bench", DIM);
+        config.float16 = float16;
+        config.hebbian_boost = 0.0;
+        let mut mem = HDF5Memory::create(config).unwrap();
+        mem.save_batch(entries).unwrap();
+        // Build the indexes, then time a checkpoint that writes everything.
+        std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
+        let t = Instant::now();
+        mem.flush_wal().unwrap();
+        let checkpoint = t.elapsed();
+        drop(mem);
+        let file_bytes = std::fs::metadata(&path).unwrap().len();
+
+        // Median of three opens.
+        let mut opens: Vec<Duration> = (0..3)
+            .map(|_| {
+                let t = Instant::now();
+                let m = HDF5Memory::open(&path).unwrap();
+                let d = t.elapsed();
+                drop(m);
+                d
+            })
+            .collect();
+        opens.sort();
+        let mut mem = HDF5Memory::open(&path).unwrap();
+
+        // Vector-only search: empty text, all weight on the vector stage.
+        let results: Vec<Vec<usize>> = data
+            .queries
+            .iter()
+            .map(|q| {
+                mem.hybrid_search(q, "", 1.0, 0.0, K)
+                    .iter()
+                    .map(|r| r.index)
+                    .collect()
+            })
+            .collect();
+        let hits: usize = results
+            .iter()
+            .zip(&exact)
+            .map(|(got, want)| got.iter().filter(|i| want.contains(i)).count())
+            .sum();
+        let recall = hits as f64 / (K * data.queries.len()) as f64;
+
+        let latency = summarize(
+            (0..N_QUERIES)
+                .map(|i| {
+                    let t = Instant::now();
+                    std::hint::black_box(mem.hybrid_search(
+                        &data.queries[i],
+                        &query_texts[i],
+                        0.4,
+                        0.6,
+                        K,
+                    ));
+                    t.elapsed()
+                })
+                .collect(),
+        );
+        let overlap = match per_variant.first() {
+            Some((_, other)) => {
+                let same: usize = results
+                    .iter()
+                    .zip(other)
+                    .map(|(a, b)| a.iter().filter(|i| b.contains(i)).count())
+                    .sum();
+                format!("{:.4}", same as f64 / (K * data.queries.len()) as f64)
+            }
+            None => "—".into(),
+        };
+        println!(
+            "| {n} | {} | {:.1} | {:.0} | {:.1} | {recall:.4} | {overlap} | {:.3} |",
+            if float16 { "float16" } else { "f32" },
+            mib(file_bytes),
+            millis(checkpoint),
+            millis(opens[1]),
+            millis(latency.p50),
+        );
+        per_variant.push((float16, results));
+    }
+}
+
 // ---------------------------------------------------------------------------
 // Fusion study: does capping the keyword candidate pool change the ranking?
 // ---------------------------------------------------------------------------
@@ -580,19 +1000,24 @@ fn bench_footprint(n: usize) {
     let path = mem.config().path.clone();
     drop(mem);
     let before_open = heap_bytes();
+    reset_peak();
     let reopened = HDF5Memory::open(&path).unwrap();
     let after_open = heap_bytes();
     let loaded = after_open.saturating_sub(before_open);
+    // Peak over the open, not just what it leaves behind: a buffer allocated
+    // and freed during the parse never shows up in the live total.
+    let peak = peak_bytes().saturating_sub(before_open);
     drop(reopened);
 
     let raw = (n * DIM * 4) as u64;
     println!(
-        "| {n} | {:.0} | {:.0} | {:.0} | {:.0} | {:.0} | {:.2}x |",
+        "| {n} | {:.0} | {:.0} | {:.0} | {:.0} | {:.0} | {:.0} | {:.2}x |",
         mib(raw),
         mib(after_entries.saturating_sub(base)),
         mib(after_store.saturating_sub(after_entries)),
         mib(after_indexes.saturating_sub(after_store)),
         mib(loaded),
+        mib(peak),
         loaded as f64 / raw as f64,
     );
 }
@@ -611,6 +1036,52 @@ fn main() {
         }
         return;
     }
+    if args.iter().any(|a| a == "--signing-study") {
+        println!("## Signed checkpoints ({DIM}-dim, float16, int8 index)\n");
+        println!(
+            "| N | checkpoint ms, unsigned | checkpoint ms, signed | signing adds ms | verify ms | file MiB added |"
+        );
+        println!("|---:|---:|---:|---:|---:|---:|");
+        for &n in if full {
+            &[1_000, 10_000, 100_000][..]
+        } else {
+            &[1_000, 10_000][..]
+        } {
+            signing_study(n);
+        }
+        return;
+    }
+    if args.iter().any(|a| a == "--options-study") {
+        println!("## Search options ({DIM}-dim, k = {K}, Hebbian boost off)\n");
+        println!("| N | options | filtered recall@10 | p50 ms | p99 ms |");
+        println!("|---:|---|---:|---:|---:|");
+        for &n in if full {
+            &[10_000, 100_000][..]
+        } else {
+            &[10_000][..]
+        } {
+            options_study(n);
+        }
+        return;
+    }
+    if args.iter().any(|a| a == "--f16-first") {
+        F16_FIRST.store(true, std::sync::atomic::Ordering::Relaxed);
+    }
+    if args.iter().any(|a| a == "--float16-study") {
+        println!("## float16 embedding storage ({DIM}-dim, int8 index, Hebbian boost off)\n");
+        println!(
+            "| N | embeddings | file MiB | checkpoint ms | open ms | recall@10 | top-10 overlap with the other | hybrid p50 ms |"
+        );
+        println!("|---:|---|---:|---:|---:|---:|---:|---:|");
+        for &n in if full {
+            &[1_000, 10_000, 100_000][..]
+        } else {
+            &[1_000, 10_000][..]
+        } {
+            float16_study(n);
+        }
+        return;
+    }
     if args.iter().any(|a| a == "--int8") {
         INT8.store(true, std::sync::atomic::Ordering::Relaxed);
         println!("(int8-quantised index vectors)");
@@ -644,9 +1115,9 @@ fn main() {
     if args.iter().any(|a| a == "--footprint") {
         println!("\n### Resident memory, {DIM}-dim f32\n");
         println!(
-            "| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | reopened / raw |"
+            "| N | vectors (raw) | entries MiB | store MiB | indexes MiB | reopened MiB | peak during open MiB | reopened / raw |"
         );
-        println!("|---:|---:|---:|---:|---:|---:|---:|");
+        println!("|---:|---:|---:|---:|---:|---:|---:|---:|");
         for &n in sizes {
             bench_footprint(n);
         }

crates/clawhdf5-cli/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-cli"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 license = "MIT"
 description = "CLI for clawhdf5 agent memory — create, save, search, recall, stats"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -14,7 +15,7 @@ name = "clawhdf5"
 path = "src/main.rs"
 
 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.6.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
 clap = { version = "4", features = ["derive", "env"] }
 serde_json = "1"
 serde = { workspace = true }

crates/clawhdf5-cli/src/main.rs

@@ -1,15 +1,22 @@
-use std::path::PathBuf;
+use std::path::{Path, PathBuf};
 
 use clap::{Parser, Subcommand};
+use clawhdf5_agent::signing::{self, SigningKey, VerifyingKey};
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
 
 /// ClawhDF5 — HDF5-backed cognitive memory for AI agents
 #[derive(Parser)]
 #[command(name = "clawhdf5", version, about)]
 struct Cli {
-    /// Path to the .h5 memory file
+    /// Path to the .h5 memory file (not needed for `keygen`)
     #[arg(short, long, env = "CLAWHDF5_PATH")]
-    path: PathBuf,
+    path: Option<PathBuf>,
+
+    /// File holding an Ed25519 signing key (64 hex characters, from
+    /// `keygen`). Every checkpoint this command makes is then signed; a
+    /// signed store refuses to checkpoint without it.
+    #[arg(long, env = "CLAWHDF5_SIGNING_KEY", global = true)]
+    signing_key: Option<PathBuf>,
 
     #[command(subcommand)]
     command: Commands,
@@ -28,10 +35,22 @@ enum Commands {
         /// Enable write-ahead log
         #[arg(long)]
         wal: bool,
-        /// Store the vector index's copy of the embeddings as int8, roughly
-        /// halving a loaded store's memory at about 13% fewer queries/second
+        /// Hold the vector index's copy of the embeddings as f32 instead of
+        /// the default int8 (which uses a quarter of the memory and is faster
+        /// at equal recall)
         #[arg(long)]
+        f32_index: bool,
+        /// Accepted for compatibility; int8 is now the default
+        #[arg(long, hide = true, conflicts_with = "f32_index")]
         quantized_index: bool,
+        /// Store embeddings as full-precision f32 instead of the default
+        /// half precision (float16: half the bytes, about three significant
+        /// digits, values within ±65504)
+        #[arg(long)]
+        f32: bool,
+        /// Accepted for compatibility; float16 is now the default
+        #[arg(long, hide = true, conflicts_with = "f32")]
+        float16: bool,
     },
     /// Save a memory entry (reads JSON from stdin or --json)
     Save {
@@ -79,6 +98,38 @@ enum Commands {
         /// Destination path
         dest: PathBuf,
     },
+    /// Generate an Ed25519 signing key for signed checkpoints
+    Keygen {
+        /// Where to write the secret key (created new, owner-only on Unix)
+        #[arg(long)]
+        out: PathBuf,
+    },
+    /// Verify a signed store against a public key; exit status 2 if not valid
+    Verify {
+        /// The trusted public key: 64 hex characters, or a file holding them
+        #[arg(long)]
+        public_key: String,
+    },
+}
+
+fn read_signing_key(path: &Path) -> Result<SigningKey, Box<dyn std::error::Error>> {
+    let text = std::fs::read_to_string(path)
+        .map_err(|e| format!("cannot read signing key {}: {e}", path.display()))?;
+    let bytes = signing::from_hex::<32>(&text)
+        .ok_or_else(|| format!("{} is not a 64-hex-character key", path.display()))?;
+    Ok(SigningKey::from_bytes(&bytes))
+}
+
+/// Open for writing, with the signing key applied if one was given.
+fn open_writable(
+    path: &Path,
+    key: &Option<SigningKey>,
+) -> Result<HDF5Memory, Box<dyn std::error::Error>> {
+    let mut mem = HDF5Memory::open(path)?;
+    if let Some(k) = key {
+        mem.set_signing_key(k.clone());
+    }
+    Ok(mem)
 }
 
 fn main() {
@@ -91,24 +142,76 @@ fn main() {
 }
 
 fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
+    if let Commands::Keygen { out } = &cli.command {
+        let key = signing::generate_key();
+        let mut opts = std::fs::OpenOptions::new();
+        opts.write(true).create_new(true);
+        #[cfg(unix)]
+        {
+            use std::os::unix::fs::OpenOptionsExt;
+            opts.mode(0o600);
+        }
+        use std::io::Write;
+        let mut f = opts
+            .open(out)
+            .map_err(|e| format!("cannot create {}: {e}", out.display()))?;
+        writeln!(f, "{}", signing::to_hex(&key.to_bytes()))?;
+        let j = serde_json::json!({
+            "status": "generated",
+            "secret_key_file": out.display().to_string(),
+            "public_key": signing::to_hex(&key.verifying_key().to_bytes()),
+        });
+        println!("{}", serde_json::to_string_pretty(&j)?);
+        return Ok(());
+    }
+    let path = cli
+        .path
+        .clone()
+        .ok_or("--path (or CLAWHDF5_PATH) is required")?;
+    let key = cli
+        .signing_key
+        .as_deref()
+        .map(read_signing_key)
+        .transpose()?;
     match cli.command {
         Commands::Create {
             agent_id,
             dim,
             wal,
-            quantized_index,
+            f32_index,
+            quantized_index: _,
+            f32,
+            float16: _,
         } => {
-            let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim);
+            let mut config = MemoryConfig::new(path.clone(), &agent_id, dim);
             config.wal_enabled = wal;
-            config.quantized_index = quantized_index;
-            let mem = HDF5Memory::create(config)?;
+            // As with --f32-index: only ever switch the library default off.
+            if f32 {
+                config.float16 = false;
+            }
+            let config_float16 = config.float16;
+            // Only ever switch *off* the library default: assigning the flag
+            // outright would force every CLI-created store back to f32 unless
+            // the caller knew to ask for int8.
+            if f32_index {
+                config.quantized_index = false;
+            }
+            let config_quantized = config.quantized_index;
+            let mut mem = HDF5Memory::create(config)?;
+            // Sign straight away, so the store is never on disk unsigned.
+            if let Some(k) = &key {
+                mem.set_signing_key(k.clone());
+                mem.flush_wal()?;
+            }
             let j = serde_json::json!({
                 "status": "created",
-                "path": cli.path.display().to_string(),
+                "path": path.display().to_string(),
                 "agent_id": agent_id,
                 "embedding_dim": dim,
                 "wal_enabled": wal,
-                "quantized_index": quantized_index,
+                "quantized_index": config_quantized,
+                "float16": config_float16,
+                "signed": mem.is_signed(),
                 "count": mem.count(),
             });
             println!("{}", serde_json::to_string_pretty(&j)?);
@@ -125,7 +228,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
                 }
             };
             let entry: MemoryEntry = serde_json::from_str(&input)?;
-            let mut mem = HDF5Memory::open(&cli.path)?;
+            let mut mem = open_writable(&path, &key)?;
             let idx = mem.save(entry)?;
             let j = serde_json::json!({ "status": "saved", "index": idx, "count": mem.count() });
             println!("{}", serde_json::to_string(&j)?);
@@ -139,7 +242,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
             keyword_weight,
         } => {
             let emb: Vec<f32> = serde_json::from_str(&embedding)?;
-            let mut mem = HDF5Memory::open(&cli.path)?;
+            let mut mem = open_writable(&path, &key)?;
             let results = mem.hybrid_search(&emb, &query, vector_weight, keyword_weight, top_k);
             let j: Vec<serde_json::Value> = results
                 .iter()
@@ -157,7 +260,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
         }
 
         Commands::Recall { index } => {
-            let mem = HDF5Memory::open_read_only(&cli.path)?;
+            let mem = HDF5Memory::open_read_only(&path)?;
             match mem.get_chunk(index) {
                 Some(content) => {
                     let j = serde_json::json!({ "index": index, "chunk": content });
@@ -171,22 +274,23 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
         }
 
         Commands::Stats => {
-            let mem = HDF5Memory::open_read_only(&cli.path)?;
+            let mem = HDF5Memory::open_read_only(&path)?;
             let cfg = mem.config();
             let j = serde_json::json!({
-                "path": cli.path.display().to_string(),
+                "path": path.display().to_string(),
                 "agent_id": cfg.agent_id,
                 "embedding_dim": cfg.embedding_dim,
                 "count": mem.count(),
                 "active": mem.count_active(),
                 "wal_enabled": cfg.wal_enabled,
                 "wal_pending": mem.wal_pending_count(),
+                "signed": mem.is_signed(),
             });
             println!("{}", serde_json::to_string_pretty(&j)?);
         }
 
         Commands::FlushWal => {
-            let mut mem = HDF5Memory::open(&cli.path)?;
+            let mut mem = open_writable(&path, &key)?;
             let before = mem.wal_pending_count();
             mem.flush_wal()?;
             let j = serde_json::json!({
@@ -198,7 +302,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
         }
 
         Commands::AgentsMd { output } => {
-            let mem = HDF5Memory::open_read_only(&cli.path)?;
+            let mem = HDF5Memory::open_read_only(&path)?;
             let md = mem.generate_agents_md();
             match output {
                 Some(p) => {
@@ -210,7 +314,7 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
         }
 
         Commands::Export => {
-            let mem = HDF5Memory::open_read_only(&cli.path)?;
+            let mem = HDF5Memory::open_read_only(&path)?;
             for i in 0..mem.count() {
                 if let Some(chunk) = mem.get_chunk(i) {
                     let j = serde_json::json!({ "index": i, "chunk": chunk });
@@ -219,11 +323,44 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
             }
         }
 
+        Commands::Keygen { .. } => unreachable!("handled before opening a store"),
+
+        Commands::Verify { public_key } => {
+            let text = if Path::new(&public_key).is_file() {
+                std::fs::read_to_string(&public_key)?
+            } else {
+                public_key
+            };
+            let bytes = signing::from_hex::<32>(&text)
+                .ok_or("--public-key must be 64 hex characters or a file holding them")?;
+            let trusted = VerifyingKey::from_bytes(&bytes)?;
+            let r = HDF5Memory::verify(&path, &trusted)?;
+            let j = serde_json::json!({
+                "valid": r.is_valid(),
+                "signed": r.signed,
+                "key_matches": r.key_matches,
+                "signature_valid": r.signature_valid,
+                "records_match": r.records_match,
+                "settings_match": r.settings_match,
+                "sessions_match": r.sessions_match,
+                "graph_match": r.graph_match,
+                "changed_records": r.changed_records,
+                "record_count": r.record_count,
+                "signed_record_count": r.signed_record_count,
+                "signed_by": r.public_key.map(|k| signing::to_hex(&k)),
+                "wal_entries_unsigned": r.wal_entries_unsigned,
+            });
+            println!("{}", serde_json::to_string_pretty(&j)?);
+            if !r.is_valid() {
+                std::process::exit(2);
+            }
+        }
+
         Commands::Snapshot { dest } => {
-            let _result = clawhdf5_agent::storage::snapshot_file(&cli.path, &dest)?;
+            let _result = clawhdf5_agent::storage::snapshot_file(&path, &dest)?;
             let j = serde_json::json!({
                 "status": "snapshot_created",
-                "source": cli.path.display().to_string(),
+                "source": path.display().to_string(),
                 "dest": dest.display().to_string(),
             });
             println!("{}", serde_json::to_string(&j)?);

crates/clawhdf5-derive/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-derive"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "Derive macros for rustyhdf5 HDF5 traits"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"

crates/clawhdf5-filters/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-filters"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "Filter and compression pipeline for clawhdf5"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -25,8 +26,12 @@ name = "compression_bench"
 harness = false
 
 [features]
-default = ["fast-deflate"]
+# Pure-Rust zlib-rs by default; `fast-deflate` (zlib-ng, C) overrides it.
+default = ["zlib-rs"]
 fast-deflate = ["flate2/zlib-ng"]
 system-zlib = ["flate2/zlib-default"]
-zlib-rs = ["flate2/zlib-rs"]
+# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
+# SIMD at runtime. flate2 enables it by default, but we build flate2 with
+# default-features = false, and without it zlib-rs inflates 3.5x slower.
+zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
 apple-compression = []

crates/clawhdf5-filters/README.md

@@ -8,16 +8,18 @@ Filter and compression pipeline for clawhdf5.
 ## Features
 
 - DEFLATE compression/decompression
-- Fast deflate via zlib-ng (`fast-deflate` feature)
+- Pure-Rust deflate via zlib-rs (default, `zlib-rs` feature)
+- zlib-ng instead, if you want it (`fast-deflate` feature; C, needs cmake)
 - Apple Compression framework support (`apple-compression` feature)
 
 ## Usage
 
 ```rust
-use clawhdf5_filters::{deflate_decode, deflate_encode};
+use clawhdf5_filters::{deflate_compress, deflate_decompress};
 
-let compressed = deflate_encode(&data, 6).unwrap();
-let decompressed = deflate_decode(&compressed).unwrap();
+let compressed = deflate_compress(&data, 6).unwrap();
+// The second argument bounds the output: the expected decompressed size.
+let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
 ```
 
 ## License

crates/clawhdf5-filters/src/fast_deflate.rs

@@ -1,12 +1,13 @@
-//! Fast deflate backends: Apple Compression Framework and zlib-ng.
+//! Deflate backends: Apple Compression Framework, zlib-ng and zlib-rs.
 //!
 //! Backend selection priority (decompression & compression):
 //! 1. Apple Compression Framework (macOS only, `apple-compression` feature)
-//! 2. flate2 with zlib-ng backend (`fast-deflate` feature) or miniz_oxide (default)
+//! 2. flate2 with zlib-ng (`fast-deflate`), else zlib-rs (`zlib-rs`, the
+//!    default), else miniz_oxide
 //!
 //! The Apple Compression Framework uses hardware-accelerated zlib on Apple Silicon
-//! and is typically the fastest option on macOS. zlib-ng is the fastest portable
-//! option and what C HDF5 uses internally.
+//! and is typically the fastest option on macOS. zlib-rs is a pure-Rust port of
+//! zlib-ng; see `BENCHMARKS.md` for how the two compare.
 
 // ---------------------------------------------------------------------------
 // Apple Compression Framework FFI (macOS only)
@@ -243,65 +244,117 @@ mod apple {
 }
 
 // ---------------------------------------------------------------------------
-// Streaming decompression via flate2 (uses zlib-ng when fast-deflate enabled)
+// One-shot (de)compression via flate2 (whichever backend flate2 was built with)
+//
+// The whole input goes to the codec in one call, into an output buffer sized
+// up front. `flate2::read::ZlibDecoder` / `write::ZlibEncoder` stream through a
+// 32 KiB buffer instead, which cost zlib-rs up to 3.7x against zlib-ng on a
+// 1 MB chunk. clawhdf5-format's deflate filter does the same; see
+// `BENCHMARKS.md`, "Deflate backend".
 // ---------------------------------------------------------------------------
 
-/// Streaming decompress with pre-allocated output buffer.
-///
-/// When the output size is known (typical for HDF5 chunks), this avoids
-/// dynamic reallocation by writing directly into a pre-sized buffer.
+/// Decompress into a buffer pre-sized to `output_size`, the expected
+/// decompressed length (known for HDF5 chunks). Output longer than that is an
+/// error, as is a stream that ends early.
 pub(crate) fn flate2_decompress_preallocated(
     data: &[u8],
     output_size: usize,
 ) -> Result<Vec<u8>, String> {
-    use std::io::Read;
-    let mut decoder = flate2::read::ZlibDecoder::new(data);
-    let mut output = vec![0u8; output_size];
-    let mut total_read = 0;
-
-    loop {
-        match decoder.read(&mut output[total_read..]) {
-            Ok(0) => break,
-            Ok(n) => total_read += n,
-            Err(e) => return Err(e.to_string()),
-        }
-    }
-    output.truncate(total_read);
-    Ok(output)
+    inflate_bounded(data, output_size, output_size)
 }
 
 /// Absolute ceiling on decompressed output when the caller has no size hint,
 /// preventing unbounded allocation from a hostile/corrupted zlib stream.
 const MAX_DECOMPRESS_SIZE: usize = 256 * 1024 * 1024;
 
-/// Streaming decompress with dynamic sizing (when output size is unknown).
-///
-/// Bounded by [`MAX_DECOMPRESS_SIZE`] since there is no chunk-size hint to
-/// validate against here — an unbounded `read_to_end` would let a hostile
-/// zlib stream force arbitrarily large allocation (a "zlib bomb").
+/// Decompress with no size hint, bounded by [`MAX_DECOMPRESS_SIZE`] so a
+/// hostile zlib stream cannot force arbitrarily large allocation (a "zlib
+/// bomb").
 pub(crate) fn flate2_decompress_streaming(data: &[u8]) -> Result<Vec<u8>, String> {
-    use std::io::Read;
-    let decoder = flate2::read::ZlibDecoder::new(data);
-    let mut result = Vec::new();
-    decoder
-        .take(MAX_DECOMPRESS_SIZE as u64 + 1)
-        .read_to_end(&mut result)
-        .map_err(|e| e.to_string())?;
-    if result.len() > MAX_DECOMPRESS_SIZE {
-        return Err(format!(
+    let hint = data.len().saturating_mul(4).min(1 << 20);
+    inflate_bounded(data, hint, MAX_DECOMPRESS_SIZE).map_err(|e| {
+        if e.ends_with("exceeds size limit") {
+            format!(
                 "decompressed output exceeds {} MiB limit",
                 MAX_DECOMPRESS_SIZE / 1024 / 1024
-        ));
+            )
+        } else {
+            e
         }
-    Ok(result)
+    })
 }
 
-/// Compress data using flate2 (zlib-ng when fast-deflate enabled, else miniz_oxide).
+/// Inflate a zlib stream, starting from `size_hint` bytes of output and
+/// failing past `limit`.
+fn inflate_bounded(data: &[u8], size_hint: usize, limit: usize) -> Result<Vec<u8>, String> {
+    use flate2::{Decompress, FlushDecompress, Status};
+
+    // One byte of headroom past the limit distinguishes an over-size stream
+    // from one that legitimately ends exactly at the limit.
+    let max_capacity = limit.saturating_add(1);
+    let mut out = Vec::new();
+    out.try_reserve_exact(size_hint.clamp(1, max_capacity))
+        .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
+
+    let mut inflater = Decompress::new(true);
+    loop {
+        let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
+        let status = inflater
+            .decompress_vec(
+                &data[in_before as usize..],
+                &mut out,
+                FlushDecompress::Finish,
+            )
+            .map_err(|e| format!("deflate: {e}"))?;
+        if out.len() > limit {
+            return Err("deflate: output exceeds size limit".into());
+        }
+        match status {
+            Status::StreamEnd => return Ok(out),
+            Status::Ok | Status::BufError if out.len() == out.capacity() => {
+                let grow = out.capacity().min(max_capacity - out.capacity()).max(1);
+                out.try_reserve_exact(grow)
+                    .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
+            }
+            Status::Ok | Status::BufError => {
+                if inflater.total_in() as usize >= data.len()
+                    || (inflater.total_in(), inflater.total_out()) == (in_before, out_before)
+                {
+                    return Err("deflate: truncated stream".into());
+                }
+            }
+        }
+    }
+}
+
+/// Compress data using flate2 (zlib-ng, zlib-rs or miniz_oxide; see module docs).
 pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
-    use std::io::Write;
-    let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(level));
-    encoder.write_all(data).map_err(|e| e.to_string())?;
-    encoder.finish().map_err(|e| e.to_string())
+    use flate2::{Compress, Compression, FlushCompress, Status};
+
+    // zlib's compressBound, plus the zlib header and trailer.
+    let bound = data.len() + (data.len() >> 12) + (data.len() >> 14) + (data.len() >> 25) + 13 + 6;
+    let mut out = Vec::new();
+    out.try_reserve_exact(bound)
+        .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
+
+    let mut deflater = Compress::new(Compression::new(level), true);
+    loop {
+        let (in_before, out_before) = (deflater.total_in(), deflater.total_out());
+        let status = deflater
+            .compress_vec(&data[in_before as usize..], &mut out, FlushCompress::Finish)
+            .map_err(|e| format!("deflate: {e}"))?;
+        match status {
+            Status::StreamEnd => return Ok(out),
+            Status::Ok | Status::BufError if out.len() == out.capacity() => out
+                .try_reserve(out.capacity().max(4096))
+                .map_err(|e| format!("deflate: cannot allocate output: {e}"))?,
+            Status::Ok | Status::BufError => {
+                if (deflater.total_in(), deflater.total_out()) == (in_before, out_before) {
+                    return Err("deflate: encoder made no progress".into());
+                }
+            }
+        }
+    }
 }
 
 // ---------------------------------------------------------------------------
@@ -312,7 +365,7 @@ pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String
 ///
 /// Selection order:
 /// 1. Apple Compression Framework (macOS + `apple-compression` feature)
-/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
+/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
 ///
 /// When `output_hint` > 0, pre-allocates the output buffer for zero-copy
 /// decompression (avoids reallocation).
@@ -344,7 +397,7 @@ pub fn decompress(data: &[u8], output_hint: usize) -> Result<Vec<u8>, String> {
 ///
 /// Selection order:
 /// 1. Apple Compression Framework (macOS + `apple-compression` feature)
-/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
+/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
 pub fn compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
     #[cfg(all(target_os = "macos", feature = "apple-compression"))]
     {
@@ -377,9 +430,19 @@ pub fn active_backend() -> &'static str {
     {
         "zlib-ng"
     }
+    // flate2 prefers a C zlib over zlib-rs when both are enabled.
+    #[cfg(all(
+        not(all(target_os = "macos", feature = "apple-compression")),
+        not(feature = "fast-deflate"),
+        feature = "zlib-rs"
+    ))]
+    {
+        "zlib-rs"
+    }
     #[cfg(not(any(
         all(target_os = "macos", feature = "apple-compression"),
-        feature = "fast-deflate"
+        feature = "fast-deflate",
+        feature = "zlib-rs"
     )))]
     {
         "miniz_oxide"
@@ -436,7 +499,7 @@ mod tests {
     fn backend_name_is_set() {
         let name = active_backend();
         assert!(
-            ["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
+            ["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
             "unexpected backend: {name}"
         );
     }

crates/clawhdf5-filters/src/lib.rs

@@ -2,12 +2,14 @@
 //!
 //! Provides deflate (zlib) decompression/compression with multiple backend options:
 //!
-//! - **Default**: `miniz_oxide` (pure Rust, no C dependencies)
-//! - **`fast-deflate` feature**: `zlib-ng` via flate2 (~2-3x faster, matches C HDF5)
+//! - **Default (`zlib-rs` feature)**: `zlib-rs` via flate2 (pure Rust, no C
+//!   dependencies)
+//! - **`fast-deflate` feature**: `zlib-ng` via flate2 (C, built with cmake)
 //! - **`apple-compression` feature**: Apple Compression Framework on macOS
 //!   (hardware-accelerated on Apple Silicon)
+//! - With none of the above: `miniz_oxide` (pure Rust, slower)
 //!
-//! Backend priority: apple-compression > zlib-ng > miniz_oxide.
+//! Backend priority: apple-compression > zlib-ng > zlib-rs > miniz_oxide.
 
 pub mod fast_deflate;
 
@@ -115,7 +117,7 @@ mod tests {
     fn backend_reports_name() {
         let name = deflate_backend();
         assert!(
-            ["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
+            ["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
             "unexpected backend: {name}"
         );
     }

crates/clawhdf5-format/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-format"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "Pure-Rust HDF5 binary format parsing and writing — no C dependencies"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -23,16 +24,20 @@ libaec-sys = { path = "../libaec-sys", version = "0.1", optional = true }
 pco = { version = "1.0", optional = true }
 
 [dev-dependencies]
+half = { workspace = true }
 serde_json = "1"
 criterion = { workspace = true }
-clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.6.0" }
+clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.7.0" }
 
 [[bench]]
 name = "bench"
 harness = false
 
 [features]
-default = ["std", "checksum", "deflate", "provenance", "fast-deflate", "system-zlib-decompress"]
+# Deflate backend: `zlib-rs` (pure Rust) by default. `fast-deflate` selects
+# zlib-ng instead (C, built with cmake); flate2 prefers a C zlib whenever one
+# is enabled, so turning it on anywhere in the build overrides the default.
+default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress"]
 std = []
 checksum = []
 deflate = ["flate2"]
@@ -42,7 +47,10 @@ fast-checksum = ["crc32fast"]
 fast-deflate = ["flate2/zlib-ng"]
 system-zlib = ["flate2/zlib-default"]
 system-zlib-decompress = []
-zlib-rs = ["flate2/zlib-rs"]
+# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
+# SIMD at runtime. flate2 enables it by default, but we build flate2 with
+# default-features = false, and without it zlib-rs inflates 3.5x slower.
+zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
 lz4 = ["lz4_flex"]
 zstd = ["dep:zstd"]
 blake3_hash = ["blake3"]

crates/clawhdf5-format/fuzz/.gitignore

@@ -1 +1,4 @@
 target/
+corpus/
+artifacts/
+coverage/

crates/clawhdf5-format/fuzz/fuzz_targets/fuzz_btree_v2.rs

@@ -1,15 +1,36 @@
 #![no_main]
+use clawhdf5_format::btree_v2::{BTreeV2Header, collect_btree_v2_records};
 use libfuzzer_sys::fuzz_target;
 
 fuzz_target!(|data: &[u8]| {
     for &offset_size in &[4u8, 8] {
         for &length_size in &[4u8, 8] {
-            let _ = clawhdf5_format::btree_v2::BTreeV2Header::parse(
-                data,
-                0,
-                offset_size,
-                length_size,
-            );
+            if let Ok(header) = BTreeV2Header::parse(data, 0, offset_size, length_size) {
+                let _ = collect_btree_v2_records(data, &header, offset_size, length_size);
             }
         }
+    }
+
+    // Parsing a header requires a valid checksum, which random input almost
+    // never has, so the traversal behind it went unfuzzed — and that is where
+    // a node listing itself as its own child overflowed the stack. Take the
+    // header fields straight from the input instead and walk the rest.
+    let Some((fields, file)) = data.split_first_chunk::<20>() else {
+        return;
+    };
+    let header = BTreeV2Header {
+        tree_type: fields[0],
+        node_size: u32::from_le_bytes([fields[1], fields[2], fields[3], fields[4]]),
+        record_size: u16::from_le_bytes([fields[5], fields[6]]),
+        depth: u16::from_le_bytes([fields[7], fields[8]]),
+        root_node_address: u64::from(u32::from_le_bytes([
+            fields[9], fields[10], fields[11], fields[12],
+        ])),
+        num_records_in_root: u16::from_le_bytes([fields[13], fields[14]]),
+        total_records: u64::from(u32::from_le_bytes([
+            fields[15], fields[16], fields[17], fields[18],
+        ])),
+    };
+    let offset_size = if fields[19] & 1 == 0 { 4 } else { 8 };
+    let _ = collect_btree_v2_records(file, &header, offset_size, 8);
 });

crates/clawhdf5-format/src/btree_v2.rs

@@ -172,6 +172,17 @@ fn max_records_leaf(node_size: u32, record_size: u16) -> u64 {
     ((node_size - overhead) / record_size as u32) as u64
 }
 
+/// Deepest B-tree v2 accepted. See [`collect_btree_v2_records`].
+const MAX_DEPTH: u16 = 64;
+
+/// Take `n` records from the traversal's budget, or refuse the tree.
+fn spend(budget: &mut usize, n: usize) -> Result<(), FormatError> {
+    *budget = budget
+        .checked_sub(n)
+        .ok_or(FormatError::NestingDepthExceeded)?;
+    Ok(())
+}
+
 /// Collect all records from a B-tree v2 by traversing from the root.
 pub fn collect_btree_v2_records(
     file_data: &[u8],
@@ -182,6 +193,22 @@ pub fn collect_btree_v2_records(
     if header.total_records == 0 || header.num_records_in_root == 0 {
         return Ok(Vec::new());
     }
+    // Recursion is one frame per level, and the depth is read from the file:
+    // a crafted header claiming 65 535 levels over a node that is its own
+    // child overflowed the stack. 64 matches the fractal heap's guard, and no
+    // real tree comes close — even at the minimum fan-out of two it would
+    // hold more than 2^64 records.
+    if header.depth > MAX_DEPTH {
+        return Err(FormatError::NestingDepthExceeded);
+    }
+    // A valid tree stores each record once, in its own bytes, so it cannot
+    // hold more records than the file has room for. Children are addresses,
+    // though, and nothing makes them distinct: levels whose children all
+    // point at one shared node below reach it fan-out^depth times, which is
+    // millions of records from a few kilobytes. Counting against what the
+    // file could physically contain bounds that without trusting the
+    // header's own `total_records`.
+    let mut budget = file_data.len() / usize::from(header.record_size.max(1));
 
     let max_leaf_nrec = max_records_leaf(header.node_size, header.record_size);
 
@@ -206,6 +233,7 @@ pub fn collect_btree_v2_records(
             offset_size,
             length_size,
             max_leaf_nrec,
+            &mut budget,
             &mut records,
         )?;
         Ok(records)
@@ -273,6 +301,7 @@ fn collect_internal_records(
     offset_size: u8,
     length_size: u8,
     max_leaf_nrec: u64,
+    budget: &mut usize,
     out: &mut Vec<BTreeV2Record>,
 ) -> Result<(), FormatError> {
     // signature(4) + version(1) + type(1) = 6
@@ -350,6 +379,8 @@ fn collect_internal_records(
     // We collect child[0] records, then record[0], then child[1], etc.
     for (i, &(child_addr, child_nrec)) in children.iter().enumerate() {
         if child_depth == 0 {
+            // Before parsing, so a refused tree is not also a large allocation.
+            spend(budget, usize::from(child_nrec))?;
             let leaf_recs =
                 parse_leaf_records(file_data, child_addr as usize, child_nrec, record_size)?;
             out.extend(leaf_recs);
@@ -364,6 +395,7 @@ fn collect_internal_records(
                 offset_size,
                 length_size,
                 max_leaf_nrec,
+                budget,
                 out,
             )?;
         }
@@ -393,6 +425,7 @@ fn collect_internal_records(
                     available: file_data.len(),
                 });
             }
+            spend(budget, 1)?;
             out.push(BTreeV2Record {
                 data: file_data[rec_start..rec_end].to_vec(),
             });
@@ -466,6 +499,124 @@ mod tests {
         buf
     }
 
+    /// An internal node laid out exactly as `collect_internal_records` will
+    /// read it at `depth`: `records` zeroed records, then `children` pointers,
+    /// all to `child_addr` claiming `child_nrec` records.
+    fn internal_node(
+        depth: u16,
+        node_size: u32,
+        record_size: u16,
+        records: usize,
+        children: usize,
+        child_addr: u64,
+        child_nrec: u64,
+    ) -> Vec<u8> {
+        let max_leaf = max_records_leaf(node_size, record_size);
+        let nrec_width = bytes_for_max_records(if depth == 1 { max_leaf } else { max_leaf * 2 });
+        let total_width = if depth > 1 {
+            bytes_for_max_records(header_max_total_records(max_leaf, depth - 1))
+        } else {
+            0
+        };
+        let mut buf = b"BTIN".to_vec();
+        buf.extend_from_slice(&[0, 5]);
+        buf.resize(buf.len() + records * record_size as usize, 0);
+        for _ in 0..children {
+            buf.extend_from_slice(&child_addr.to_le_bytes());
+            buf.extend_from_slice(&child_nrec.to_le_bytes()[..nrec_width]);
+            buf.resize(buf.len() + total_width, 0);
+        }
+        buf
+    }
+
+    fn header(depth: u16, root: u64, root_nrec: u16, total: u64) -> BTreeV2Header {
+        BTreeV2Header {
+            tree_type: 5,
+            node_size: 512,
+            record_size: 8,
+            depth,
+            root_node_address: root,
+            num_records_in_root: root_nrec,
+            total_records: total,
+        }
+    }
+
+    #[test]
+    fn a_node_that_is_its_own_child_is_rejected_not_recursed() {
+        // One internal node whose two children are itself, under a header
+        // claiming the deepest tree a u16 allows. The layout stops depending
+        // on depth once the subtree-total width saturates, so every level
+        // parses cleanly and recursion runs ~65 000 frames deep: before the
+        // cap this overflowed the stack and aborted the process, from a file
+        // of under 100 bytes.
+        let mut data = internal_node(u16::MAX, 512, 8, 1, 2, 0, 1);
+        data.resize(4096, 0);
+        let result = collect_btree_v2_records(&data, &header(u16::MAX, 0, 1, 1), 8, 8);
+        assert!(result.is_err(), "{result:?}");
+    }
+
+    #[test]
+    fn a_shared_subtree_cannot_multiply_the_work() {
+        // A chain of distinct levels, each node's children all pointing at the
+        // single node below, ending in a real leaf. Every node parses and
+        // nothing is cyclic, yet the leaf is reached fan-out^depth times: 62
+        // children over 4 levels is ~15 million leaf visits from a few
+        // kilobytes. A valid tree cannot hold more records than the file has
+        // room for, so that bounds the traversal instead.
+        let (node_size, record_size) = (512u32, 8u16);
+        let fanout = 62usize;
+        let depth = 4u16;
+        let leaf = build_leaf_node(5, &[&[0u8; 8][..]]);
+
+        // Lay out root first, then each lower level, then the leaf.
+        let mut nodes: Vec<Vec<u8>> = Vec::new();
+        let mut addrs = Vec::new();
+        let mut at = 0u64;
+        let mut sizes = Vec::new();
+        for d in (1..=depth).rev() {
+            let n = internal_node(d, node_size, record_size, fanout - 1, fanout, 0, 0);
+            sizes.push(n.len());
+        }
+        for size in &sizes {
+            addrs.push(at);
+            at += *size as u64;
+        }
+        let leaf_addr = at;
+        for (i, d) in (1..=depth).rev().enumerate() {
+            let (child, child_nrec) = if d == 1 {
+                (leaf_addr, 1)
+            } else {
+                (addrs[i + 1], fanout as u64 - 1)
+            };
+            nodes.push(internal_node(
+                d,
+                node_size,
+                record_size,
+                fanout - 1,
+                fanout,
+                child,
+                child_nrec,
+            ));
+        }
+        let mut data: Vec<u8> = nodes.concat();
+        data.extend_from_slice(&leaf);
+        data.resize(data.len() + 64, 0);
+
+        let started = std::time::Instant::now();
+        let result =
+            collect_btree_v2_records(&data, &header(depth, 0, fanout as u16 - 1, u64::MAX), 8, 8);
+        assert!(
+            result.is_err(),
+            "expected a refusal, got {} records",
+            result.map_or(0, |r| r.len())
+        );
+        assert!(
+            started.elapsed() < std::time::Duration::from_secs(2),
+            "took {:?}",
+            started.elapsed()
+        );
+    }
+
     #[test]
     fn parse_header() {
         let data = build_btree_v2_header(5, 512, 11, 0, 0x1000, 3, 3, 8, 8);

crates/clawhdf5-format/src/data_read.rs

@@ -1076,6 +1076,21 @@ pub fn read_as_f32(raw: &[u8], datatype: &Datatype) -> Result<Vec<f32>, FormatEr
     ) {
         return Ok(native_le_to_vec::<f32>(raw, count));
     }
+    // Little-endian half precision (numpy float16): widen directly.
+    if matches!(
+        datatype,
+        Datatype::FloatingPoint {
+            size: 2,
+            byte_order: DatatypeByteOrder::LittleEndian,
+            ..
+        }
+    ) {
+        let (halves, _) = raw[..count * 2].as_chunks::<2>();
+        return Ok(halves
+            .iter()
+            .map(|&b| f16_bits_to_f32(u16::from_le_bytes(b)))
+            .collect());
+    }
 
     let order = get_byte_order(datatype);
     let mut result = Vec::with_capacity(count);
@@ -1622,36 +1637,7 @@ fn read_f16_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f32 {
     f16_bits_to_f32(u16::from_le_bytes(buf))
 }
 
-/// Convert the bit pattern of an IEEE-754 half (binary16) to an `f32`.
-fn f16_bits_to_f32(h: u16) -> f32 {
-    let h = h as u32;
-    let sign = (h & 0x8000) << 16;
-    let exp = (h >> 10) & 0x1f;
-    let mant = h & 0x3ff;
-    let bits = if exp == 0 {
-        if mant == 0 {
-            sign // signed zero
-        } else {
-            // Subnormal: normalize into an f32 normal.
-            let mut e: i32 = -1;
-            let mut m = mant;
-            loop {
-                e += 1;
-                m <<= 1;
-                if m & 0x400 != 0 {
-                    break;
-                }
-            }
-            let m = m & 0x3ff;
-            sign | (((127 - 15 - e) as u32) << 23) | (m << 13)
-        }
-    } else if exp == 0x1f {
-        sign | 0x7f80_0000 | (mant << 13) // inf / NaN
-    } else {
-        sign | ((exp + (127 - 15)) << 23) | (mant << 13)
-    };
-    f32::from_bits(bits)
-}
+use crate::float16::f16_bits_to_f32;
 
 fn read_f32_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f32 {
     let mut buf = [0u8; 4];

crates/clawhdf5-format/src/datatype.rs

@@ -640,7 +640,8 @@ impl Datatype {
                 mantissa_size,
                 exponent_bias,
             } => {
-                let mut bf0 = 0x20u8; // bit 5: sign location bit (standard IEEE 754)
+                // Bits 4-5: mantissa normalization = 2 (implied leading 1, IEEE 754).
+                let mut bf0 = 0x20u8;
                 match byte_order {
                     DatatypeByteOrder::BigEndian => {
                         bf0 |= 0x01;
@@ -650,9 +651,14 @@ impl Datatype {
                     }
                     _ => {}
                 }
-                // bf[1] bits 0-1: mantissa normalization = 2 (MSB not stored, IEEE 754)
-                let bf1 = 0x3fu8; // matching what h5py generates
-                let mut buf = Self::build_header(1, 1, [bf0, bf1, 0], *size);
+                // Bits 8-15: the sign bit's position, the top bit of the value.
+                // This was hard-coded to 63, which is right only for f64: the
+                // HDF5 library rejects any other float with "sign bit position
+                // out of bounds", so every f32 dataset and attribute we wrote
+                // was unreadable by h5py and libhdf5.
+                let sign_location =
+                    (u32::from(*bit_offset) + u32::from(*bit_precision)).saturating_sub(1) as u8;
+                let mut buf = Self::build_header(1, 1, [bf0, sign_location, 0], *size);
                 buf.extend_from_slice(&bit_offset.to_le_bytes());
                 buf.extend_from_slice(&bit_precision.to_le_bytes());
                 buf.push(*exponent_location);
@@ -818,6 +824,24 @@ fn build_dt_header(class: u8, version: u8, bf: [u8; 3], size: u32) -> Vec<u8> {
 mod tests {
     use super::*;
 
+    #[test]
+    fn float_sign_location_is_the_top_bit_of_the_value() {
+        // The HDF5 library rejects a float whose sign position is not inside
+        // its precision; this was hard-coded to 63, so every f32 we wrote was
+        // unreadable by h5py. Byte 2 of the message is the sign position.
+        use crate::type_builders::{make_f16_type, make_f32_type, make_f64_type};
+        for (dt, sign) in [
+            (make_f16_type(), 15),
+            (make_f32_type(), 31),
+            (make_f64_type(), 63),
+        ] {
+            let bytes = dt.serialize();
+            assert_eq!(bytes[2], sign, "{dt:?}");
+            let (parsed, _) = Datatype::parse(&bytes).unwrap();
+            assert_eq!(parsed, dt);
+        }
+    }
+
     // Helper to build a fixed-point datatype message
     fn build_fixed_point(
         size: u32,

crates/clawhdf5-format/src/extensible_array.rs

@@ -12,6 +12,31 @@ use alloc::{format, vec, vec::Vec};
 use crate::chunked_read::ChunkInfo;
 use crate::error::FormatError;
 
+/// Verify the Jenkins lookup3 checksum stored immediately after
+/// `data[start..end]`, as every Extensible Array structure carries one.
+///
+/// A corrupt chunk index yields addresses pointing at the wrong bytes, so a
+/// mismatch is an error: otherwise the damage surfaces as plausible data read
+/// from the wrong chunk.
+#[cfg(feature = "checksum")]
+fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatError> {
+    ensure_len(data, end, 4)?;
+    let stored = u32::from_le_bytes([data[end], data[end + 1], data[end + 2], data[end + 3]]);
+    let computed = crate::checksum::jenkins_lookup3(&data[start..end]);
+    if computed != stored {
+        return Err(FormatError::ChecksumMismatch {
+            expected: stored,
+            computed,
+        });
+    }
+    Ok(())
+}
+
+#[cfg(not(feature = "checksum"))]
+fn verify_checksum(_data: &[u8], _start: usize, _end: usize) -> Result<(), FormatError> {
+    Ok(())
+}
+
 /// Parsed Extensible Array header (AEHD).
 #[derive(Debug, Clone)]
 pub struct ExtensibleArrayHeader {
@@ -145,6 +170,8 @@ impl ExtensibleArrayHeader {
         pos += ls; // skip nelmts
         pos += ls; // skip max_idx_set (6th stats field)
         let index_block_address = read_offset(d, pos, offset_size)?;
+        pos += offset_size as usize;
+        verify_checksum(file_data, offset, offset + pos)?;
 
         Ok(ExtensibleArrayHeader {
             client_id,
@@ -270,6 +297,40 @@ fn index_to_chunk_offsets(
 
 /// Collect elements from a data block at the given offset.
 #[allow(clippy::too_many_arguments)]
+/// Layout of super block `u`, per the HDF5 spec: the number of data blocks it
+/// owns and how many elements each of them holds.
+///
+/// `ndblks` and `dblk_nelmts` each double every *other* level, a half-step
+/// apart, so the blocks grow as 1x16, 1x32, 2x32, 2x64, 4x64 ... for a
+/// 16-element minimum. Treating either as doubling every level (the previous
+/// implementation) puts every element after the first data block at the wrong
+/// index.
+fn sblk_info(u: usize, data_blk_min_elmts: usize) -> Option<(usize, usize)> {
+    let ndblks = 1usize.checked_shl((u / 2) as u32)?;
+    let dblk_nelmts = 1usize
+        .checked_shl(u.div_ceil(2) as u32)?
+        .checked_mul(data_blk_min_elmts)?;
+    Some((ndblks, dblk_nelmts))
+}
+
+/// Width of the "offset of the block in the array" field carried by super and
+/// data blocks (`hdr->arr_off_size`).
+fn arr_off_size(header: &ExtensibleArrayHeader) -> usize {
+    (header.max_nelmts_bits as usize).div_ceil(8)
+}
+
+/// Elements per data block page, once a data block is large enough to be paged.
+fn page_nelmts(header: &ExtensibleArrayHeader) -> Option<usize> {
+    1usize.checked_shl(u32::from(header.max_dblk_nelmts_bits))
+}
+
+/// Read the elements of one data block (EADB).
+///
+/// `page_init` is the owning super block's page-init bitmap and `first_page`
+/// this block's first bit in it; both are only consulted when the block is
+/// paged. The bitmap lives in the super block, not here — a paged data block
+/// stores only its prefix, then one slot per page.
+#[allow(clippy::too_many_arguments)]
 fn read_data_block_elements(
     file_data: &[u8],
     db_offset: usize,
@@ -280,117 +341,101 @@ fn read_data_block_elements(
     start_index: usize,
     num_chunks_per_dim: &[u64],
     chunk_dimensions: &[u32],
+    page_init: &[u8],
+    first_page: usize,
 ) -> Result<Vec<ChunkInfo>, FormatError> {
-    // AEDB: signature(4) + version(1) + client_id(1) + header_address(offset_size)
-    let db_header_size = 4 + 1 + 1 + offset_size as usize;
+    // EADB: signature(4) + version(1) + client_id(1) + header_address(offset_size)
+    //       + block offset(arr_off_size)
+    let db_header_size = 4 + 1 + 1 + offset_size as usize + arr_off_size(header);
     ensure_len(file_data, db_offset, db_header_size)?;
 
-    let d = &file_data[db_offset..];
-    if &d[0..4] != b"EADB" {
+    if &file_data[db_offset..db_offset + 4] != b"EADB" {
         return Err(FormatError::ChunkedReadError(
             "invalid Extensible Array data block signature".into(),
         ));
     }
-    // Skip version(1) + client_id(1) + header_address(offset_size) + block_offset
-    // Block offset is encoded in ceil(max_nelmts_bits/8) bytes
-    let blk_off_size = (header.max_nelmts_bits as usize).div_ceil(8);
-    let mut pos = db_offset + db_header_size + blk_off_size;
 
-    // Check if paged
-    if header.max_nelmts_bits >= usize::BITS as u8 {
-        return Err(FormatError::Overflow(
-            "max_nelmts_bits exceeds usize bit width".into(),
-        ));
-    }
-    let page_nelmts = 1usize << header.max_nelmts_bits;
-    let is_paged = nelmts > page_nelmts;
+    let mut pos = db_offset + db_header_size;
+    let page = page_nelmts(header).ok_or_else(|| {
+        FormatError::Overflow("Extensible Array page element count overflows usize".into())
+    })?;
 
     let mut chunks = Vec::new();
-
-    if !is_paged {
-        for i in 0..nelmts {
+    let read_run = |from: usize,
+                    count: usize,
+                    first_index: usize,
+                    chunks: &mut Vec<ChunkInfo>|
+     -> Result<usize, FormatError> {
+        let mut p = from;
+        for i in 0..count {
             let (info, consumed) = read_element(
                 file_data,
-                pos,
+                p,
                 header.client_id,
                 header.element_size,
                 offset_size,
                 chunk_byte_size,
-                start_index + i,
+                first_index + i,
                 num_chunks_per_dim,
                 chunk_dimensions,
             )?;
             if let Some(ci) = info {
                 chunks.push(ci);
             }
-            pos += consumed;
+            p += consumed;
         }
-    } else {
-        // Paged: elements are split into pages of page_nelmts.
-        // After the data block header comes a page bitmap, then each page
-        // has page_nelmts elements followed by a 4-byte checksum.
-        let npages = nelmts.div_ceil(page_nelmts);
-        // Page bitmap: ceil(npages / 8) bytes
-        let bitmap_size = npages.div_ceil(8);
-        // Read bitmap
-        if pos + bitmap_size > file_data.len() {
-            return Err(FormatError::UnexpectedEof {
-                expected: pos + bitmap_size,
-                available: file_data.len(),
-            });
-        }
-        let bitmap = &file_data[pos..pos + bitmap_size];
-        pos += bitmap_size;
+        Ok(p)
+    };
 
+    if nelmts <= page {
+        // Prefix and elements are covered by one checksum.
         let elem_bytes = if header.client_id == 0 {
             offset_size as usize
         } else {
             header.element_size as usize
         };
-
-        let mut global_idx = start_index;
-        for page_idx in 0..npages {
-            let byte_idx = page_idx / 8;
-            let bit_idx = page_idx % 8;
-            let page_has_data = (bitmap[byte_idx] >> bit_idx) & 1 != 0;
-
-            let elems_this_page = if page_idx == npages - 1 {
-                let remainder = nelmts % page_nelmts;
-                if remainder == 0 {
-                    page_nelmts
-                } else {
-                    remainder
+        let end = nelmts
+            .checked_mul(elem_bytes)
+            .and_then(|b| pos.checked_add(b))
+            .ok_or_else(|| FormatError::Overflow("Extensible Array data block span".into()))?;
+        verify_checksum(file_data, db_offset, end)?;
+        read_run(pos, nelmts, start_index, &mut chunks)?;
+        return Ok(chunks);
     }
-            } else {
-                page_nelmts
-            };
 
-            if page_has_data {
-                for i in 0..elems_this_page {
-                    let (info, consumed) = read_element(
-                        file_data,
-                        pos,
-                        header.client_id,
-                        header.element_size,
-                        offset_size,
-                        chunk_byte_size,
-                        global_idx + i,
-                        num_chunks_per_dim,
-                        chunk_dimensions,
-                    )?;
-                    if let Some(ci) = info {
-                        chunks.push(ci);
-                    }
-                    pos += consumed;
-                }
-                // Skip page checksum (4 bytes)
+    // Paged: the prefix ends with its own checksum, then one slot per page,
+    // each holding `page` elements followed by a checksum. Pages whose bit is
+    // clear were never written; their slot still occupies the file, so stride
+    // over it rather than reading zeros as addresses.
+    verify_checksum(file_data, db_offset, pos)?;
     pos += 4;
+    let elem_bytes = if header.client_id == 0 {
+        offset_size as usize
     } else {
-                // Empty page: skip all elements + checksum
-                pos += elems_this_page * elem_bytes + 4;
-            }
-            global_idx += elems_this_page;
+        header.element_size as usize
+    };
+    let page_stride = page
+        .checked_mul(elem_bytes)
+        .and_then(|b| b.checked_add(4))
+        .ok_or_else(|| FormatError::Overflow("Extensible Array page stride".into()))?;
+    let npages = nelmts.div_ceil(page);
+    for p in 0..npages {
+        // One bit per page across the whole super block, packed contiguously
+        // and MSB-first within each byte, as H5VM_bit_get reads it.
+        let bit = first_page + p;
+        let initialised = page_init
+            .get(bit / 8)
+            .is_some_and(|byte| byte & (0x80 >> (bit % 8)) != 0);
+        if initialised {
+            let count = core::cmp::min(page, nelmts - p * page);
+            // Each page carries its own checksum, over a full page's worth of
+            // slots even when the last one holds fewer live elements.
+            verify_checksum(file_data, pos, pos + page * elem_bytes)?;
+            read_run(pos, count, start_index + p * page, &mut chunks)?;
         }
+        pos = pos
+            .checked_add(page_stride)
+            .ok_or_else(|| FormatError::Overflow("Extensible Array page offset".into()))?;
     }
 
     Ok(chunks)
@@ -427,30 +472,83 @@ pub fn read_extensible_array_chunks(
     let chunk_byte_size: u64 =
         chunk_dimensions.iter().map(|&d| d as u64).product::<u64>() * element_size as u64;
 
-    // Parse index block (AEIB)
+    // Parse index block (EAIB): signature(4) + version(1) + client_id(1)
+    // + header address(offset_size), then the inline elements, then the
+    // direct data block addresses, then the super block addresses.
     let ib_offset = header.index_block_address as usize;
-    let ib_header_size = 4 + 1 + 1 + offset_size as usize; // sig + ver + client + hdr_addr
+    let ib_header_size = 4 + 1 + 1 + os;
     ensure_len(file_data, ib_offset, ib_header_size)?;
 
-    let ib = &file_data[ib_offset..];
-    if &ib[0..4] != b"EAIB" {
+    if &file_data[ib_offset..ib_offset + 4] != b"EAIB" {
         return Err(FormatError::ChunkedReadError(
             "invalid Extensible Array index block signature".into(),
         ));
     }
-    // Skip version(1) + client_id(1) + header_address(offset_size)
     let mut pos = ib_offset + ib_header_size;
 
     let mut chunks = Vec::new();
-    let mut global_index = 0usize;
     let total_elements = header.num_elements as usize;
 
-    // 1. Read inline elements in index block
-    let n_inline = header.idx_blk_elmts as usize;
-    for i in 0..n_inline {
-        if global_index + i >= total_elements {
-            break;
+    let dmin = header.min_dblk_nelmts as usize;
+    if dmin == 0 || !dmin.is_power_of_two() {
+        return Err(FormatError::ChunkedReadError(
+            "Extensible Array data block minimum is not a power of two".into(),
+        ));
     }
+    // nsblks = 1 + (max_nelmts_bits - log2(data_blk_min_elmts)), and the index
+    // block holds 2 * (sup_blk_min_data_ptrs - 1) data block addresses.
+    let log2_dmin = dmin.trailing_zeros() as usize;
+    let nsblks = 1 + (header.max_nelmts_bits as usize).saturating_sub(log2_dmin);
+    let ndblk_addrs = 2 * (header.super_blk_min_nelmts as usize).saturating_sub(1);
+
+    // The data blocks listed directly in the index block are the first
+    // `ndblk_addrs` in super-block order, each sized by the level it belongs
+    // to; the super block addresses that follow resume at the next level.
+    let mut direct: Vec<usize> = Vec::with_capacity(ndblk_addrs);
+    let mut level = 0usize;
+    while direct.len() < ndblk_addrs {
+        if level >= nsblks {
+            return Err(FormatError::ChunkedReadError(
+                "Extensible Array index block claims more data blocks than the array has".into(),
+            ));
+        }
+        let (ndblks, dblk_nelmts) = sblk_info(level, dmin).ok_or_else(|| {
+            FormatError::Overflow("Extensible Array super block layout overflows usize".into())
+        })?;
+        for _ in 0..ndblks {
+            direct.push(dblk_nelmts);
+        }
+        level += 1;
+    }
+    if direct.len() != ndblk_addrs {
+        // A partial level in the index block is not a layout HDF5 produces,
+        // and guessing where the super blocks resume would misplace elements.
+        return Err(FormatError::ChunkedReadError(
+            "Extensible Array index block ends mid super block".into(),
+        ));
+    }
+
+    // One checksum covers the prefix, every inline element slot, and every
+    // data block and super block address.
+    let elem_bytes = if header.client_id == 0 {
+        os
+    } else {
+        header.element_size as usize
+    };
+    let ib_end = (header.idx_blk_elmts as usize)
+        .checked_mul(elem_bytes)
+        .and_then(|b| pos.checked_add(b))
+        .and_then(|p| {
+            ndblk_addrs
+                .checked_add(nsblks - level)
+                .and_then(|n| n.checked_mul(os).and_then(|b| p.checked_add(b)))
+        })
+        .ok_or_else(|| FormatError::Overflow("Extensible Array index block span".into()))?;
+    verify_checksum(file_data, ib_offset, ib_end)?;
+
+    // 1. Elements stored inline in the index block.
+    let n_inline = (header.idx_blk_elmts as usize).min(total_elements);
+    for i in 0..n_inline {
         let (info, consumed) = read_element(
             file_data,
             pos,
@@ -458,7 +556,7 @@ pub fn read_extensible_array_chunks(
             header.element_size,
             offset_size,
             chunk_byte_size,
-            global_index + i,
+            i,
             &num_chunks_per_dim,
             chunk_dimensions,
         )?;
@@ -467,154 +565,90 @@ pub fn read_extensible_array_chunks(
         }
         pos += consumed;
     }
-    global_index += n_inline.min(total_elements);
-
-    // If all elements were inline, we're done
+    let mut global_index = n_inline;
     if global_index >= total_elements {
         return Ok(chunks);
     }
 
-    // Compute data block and super block counts
-    let min_dblk = header.min_dblk_nelmts as usize;
-    let sblk_min = header.super_blk_min_nelmts as usize;
-
-    // The first sblk_min super block levels have their data blocks listed directly
-    // in the index block. Compute their sizes.
-    let mut n_direct_dblks = 0usize;
-    let mut dblk_sizes: Vec<usize> = Vec::new();
-    {
-        let mut nelmts = min_dblk;
-        for sb_level in 0..sblk_min {
-            if sb_level >= usize::BITS as usize {
-                return Err(FormatError::Overflow(
-                    "sb_level exceeds usize bit width".into(),
+    // 2. Data blocks listed directly in the index block.
+    for &dblk_nelmts in &direct {
+        if global_index >= total_elements {
+            return Ok(chunks);
+        }
+        ensure_len(file_data, pos, os)?;
+        let addr = read_offset(file_data, pos, offset_size)?;
+        pos += os;
+        if !is_undefined_addr(addr, offset_size) {
+            if dblk_nelmts > page_nelmts(header).unwrap_or(usize::MAX) {
+                // Would need a page-init bitmap, which only a super block
+                // carries. HDF5 never pages these small early blocks.
+                return Err(FormatError::ChunkedReadError(
+                    "Extensible Array index block references a paged data block".into(),
                 ));
             }
-            let ndblks = 1usize << sb_level;
-            for _ in 0..ndblks {
-                dblk_sizes.push(nelmts);
-                n_direct_dblks += 1;
-            }
-            if sb_level > 0 {
-                nelmts *= 2;
-            }
-        }
-    }
-
-    // Read direct data block addresses from index block
-    let mut dblk_addrs: Vec<u64> = Vec::with_capacity(n_direct_dblks);
-    for _ in 0..n_direct_dblks {
-        if pos + os > file_data.len() {
-            break;
-        }
-        let addr = read_offset(file_data, pos, offset_size)?;
-        dblk_addrs.push(addr);
-        pos += os;
-    }
-
-    // Read elements from direct data blocks
-    for (i, &addr) in dblk_addrs.iter().enumerate() {
-        if i >= dblk_sizes.len() {
-            break;
-        }
-        let nelmts = dblk_sizes[i];
-        if is_undefined_addr(addr, offset_size) {
-            global_index += nelmts;
-            continue;
-        }
-        let block_chunks = read_data_block_elements(
+            chunks.extend(read_data_block_elements(
                 file_data,
                 addr as usize,
-            nelmts,
+                dblk_nelmts,
                 header,
                 offset_size,
                 chunk_byte_size,
                 global_index,
                 &num_chunks_per_dim,
                 chunk_dimensions,
-        )?;
-        chunks.extend(block_chunks);
-        global_index += nelmts;
+                &[],
+                0,
+            )?);
+        }
+        global_index += dblk_nelmts;
     }
 
-    // Remaining elements are in super blocks
-    let total_in_ib_and_direct: usize = n_inline + dblk_sizes.iter().sum::<usize>();
-    if total_elements <= total_in_ib_and_direct {
-        return Ok(chunks);
-    }
-    let remaining_elements = total_elements - total_in_ib_and_direct;
-
-    // Compute super block layout
-    let mut sb_addrs: Vec<u64> = Vec::new();
-    let mut sb_infos: Vec<(usize, usize)> = Vec::new();
-    {
-        let mut covered = 0usize;
-        let mut sb_level = sblk_min;
-        let mut nelmts_per_dblk = min_dblk;
-        for lev in 0..sblk_min {
-            if lev > 0 {
-                nelmts_per_dblk *= 2;
-            }
-        }
-
-        while covered < remaining_elements {
-            if sb_level >= usize::BITS as usize {
-                return Err(FormatError::Overflow(
-                    "sb_level exceeds usize bit width".into(),
-                ));
-            }
-            let ndblks = 1usize << sb_level;
-            nelmts_per_dblk *= 2;
-            let total_in_sb = ndblks * nelmts_per_dblk;
-            sb_infos.push((ndblks, nelmts_per_dblk));
-            covered += total_in_sb;
-            sb_level += 1;
-        }
-    }
-
-    // Read super block addresses from index block
-    for _ in 0..sb_infos.len() {
-        if pos + os > file_data.len() {
+    // 3. Everything else lives in super blocks, one address per remaining
+    //    level, starting at the level after the direct data blocks.
+    for u in level..nsblks {
+        if global_index >= total_elements {
             break;
         }
-        let addr = read_offset(file_data, pos, offset_size)?;
-        sb_addrs.push(addr);
+        ensure_len(file_data, pos, os)?;
+        let sb_addr = read_offset(file_data, pos, offset_size)?;
         pos += os;
-    }
-
-    // Process each super block
-    for (sb_idx, &sb_addr) in sb_addrs.iter().enumerate() {
-        let (ndblks, nelmts_per_dblk) = sb_infos[sb_idx];
-        if is_undefined_addr(sb_addr, offset_size) {
-            global_index += ndblks * nelmts_per_dblk;
-            continue;
-        }
-        let sb_chunks = read_super_block(
+        let (ndblks, dblk_nelmts) = sblk_info(u, dmin).ok_or_else(|| {
+            FormatError::Overflow("Extensible Array super block layout overflows usize".into())
+        })?;
+        if !is_undefined_addr(sb_addr, offset_size) {
+            chunks.extend(read_super_block(
                 file_data,
                 sb_addr as usize,
                 ndblks,
-            nelmts_per_dblk,
+                dblk_nelmts,
                 header,
                 offset_size,
                 chunk_byte_size,
                 global_index,
                 &num_chunks_per_dim,
                 chunk_dimensions,
-        )?;
-        chunks.extend(sb_chunks);
-        global_index += ndblks * nelmts_per_dblk;
+            )?);
+        }
+        global_index =
+            global_index.saturating_add(ndblks.checked_mul(dblk_nelmts).ok_or_else(|| {
+                FormatError::Overflow("Extensible Array super block span".into())
+            })?);
     }
 
     Ok(chunks)
 }
 
-/// Read a super block (AESB) and its data blocks.
+/// Read a super block (EASB) and the data blocks it owns.
+///
+/// On disk: signature(4) + version(1) + client_id(1) + header address
+/// + block offset + the page-init bitmap for every data block it owns
+/// + one address per data block + checksum.
 #[allow(clippy::too_many_arguments)]
 fn read_super_block(
     file_data: &[u8],
     sb_offset: usize,
     ndblks: usize,
-    nelmts_per_dblk: usize,
+    dblk_nelmts: usize,
     header: &ExtensibleArrayHeader,
     offset_size: u8,
     chunk_byte_size: u64,
@@ -623,9 +657,7 @@ fn read_super_block(
     chunk_dimensions: &[u32],
 ) -> Result<Vec<ChunkInfo>, FormatError> {
     let os = offset_size as usize;
-
-    // AESB: signature(4) + version(1) + client_id(1) + header_address(offset_size)
-    let sb_header_size = 4 + 1 + 1 + os;
+    let sb_header_size = 4 + 1 + 1 + os + arr_off_size(header);
     ensure_len(file_data, sb_offset, sb_header_size)?;
 
     if &file_data[sb_offset..sb_offset + 4] != b"EASB" {
@@ -634,43 +666,57 @@ fn read_super_block(
         ));
     }
 
-    let mut pos = sb_offset + sb_header_size;
-
-    // Read data block addresses
-    let mut dblk_addrs: Vec<u64> = Vec::with_capacity(ndblks);
-    for _ in 0..ndblks {
-        if pos + os > file_data.len() {
-            return Err(FormatError::UnexpectedEof {
-                expected: pos + os,
-                available: file_data.len(),
-            });
-        }
-        let addr = read_offset(file_data, pos, offset_size)?;
-        dblk_addrs.push(addr);
-        pos += os;
-    }
+    // Page-init bitmap: one bit per page, `npages` bits per data block, packed
+    // contiguously. HDF5 sizes the buffer `ndblks * ceil(npages / 8)`, which
+    // is bigger than the bits need when `npages` is not a multiple of eight.
+    // Zero-sized unless this level's data blocks are paged.
+    let page = page_nelmts(header).ok_or_else(|| {
+        FormatError::Overflow("Extensible Array page element count overflows usize".into())
+    })?;
+    let npages = if dblk_nelmts > page {
+        dblk_nelmts / page
+    } else {
+        0
+    };
+    let per_dblk_bitmap = npages.div_ceil(8);
+    let bitmap_bytes = per_dblk_bitmap
+        .checked_mul(ndblks)
+        .ok_or_else(|| FormatError::Overflow("Extensible Array page bitmap size".into()))?;
+    let bitmap_start = sb_offset + sb_header_size;
+    ensure_len(file_data, bitmap_start, bitmap_bytes)?;
+    let bitmap = &file_data[bitmap_start..bitmap_start + bitmap_bytes];
 
+    let mut pos = bitmap_start + bitmap_bytes;
     let mut chunks = Vec::new();
     let mut global_idx = start_index;
 
-    for &addr in &dblk_addrs {
-        if is_undefined_addr(addr, offset_size) {
-            global_idx += nelmts_per_dblk;
-            continue;
-        }
-        let block_chunks = read_data_block_elements(
+    // One checksum covers the prefix, the bitmap and every data block address.
+    let sb_end = ndblks
+        .checked_mul(os)
+        .and_then(|b| pos.checked_add(b))
+        .ok_or_else(|| FormatError::Overflow("Extensible Array super block span".into()))?;
+    verify_checksum(file_data, sb_offset, sb_end)?;
+
+    for i in 0..ndblks {
+        ensure_len(file_data, pos, os)?;
+        let addr = read_offset(file_data, pos, offset_size)?;
+        pos += os;
+        if !is_undefined_addr(addr, offset_size) {
+            chunks.extend(read_data_block_elements(
                 file_data,
                 addr as usize,
-            nelmts_per_dblk,
+                dblk_nelmts,
                 header,
                 offset_size,
                 chunk_byte_size,
                 global_idx,
                 num_chunks_per_dim,
                 chunk_dimensions,
-        )?;
-        chunks.extend(block_chunks);
-        global_idx += nelmts_per_dblk;
+                bitmap,
+                i * npages,
+            )?);
+        }
+        global_idx += dblk_nelmts;
     }
 
     Ok(chunks)
@@ -679,6 +725,14 @@ fn read_super_block(
 #[cfg(test)]
 mod tests {
     use super::*;
+
+    /// Stamp the Jenkins checksum a real file would carry over
+    /// `data[start..end]`, writing it at `end`. Hand-built fixtures need this
+    /// now that the reader validates it, exactly as HDF5 writes it.
+    fn stamp_checksum(data: &mut [u8], start: usize, end: usize) {
+        let sum = crate::checksum::jenkins_lookup3(&data[start..end]);
+        data[end..end + 4].copy_from_slice(&sum.to_le_bytes());
+    }
     #[test]
     fn index_to_offsets_1d() {
         let num_chunks = vec![5u64];
@@ -734,6 +788,7 @@ mod tests {
         buf[44..52].copy_from_slice(&5u64.to_le_bytes()); // stat[4] = num_elements
         buf[52..60].copy_from_slice(&0u64.to_le_bytes()); // stat[5]
         buf[60..68].copy_from_slice(&0x1000u64.to_le_bytes()); // index_block_address
+        stamp_checksum(&mut buf, 0, 68);
 
         let hdr = ExtensibleArrayHeader::parse(&buf, 0, os, ls).unwrap();
         assert_eq!(hdr.client_id, 0);
@@ -819,6 +874,7 @@ mod tests {
             .copy_from_slice(&(num_chunks as u64).to_le_bytes());
         file_data[aehd_offset + 60..aehd_offset + 68]
             .copy_from_slice(&(aeib_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, aehd_offset, aehd_offset + 68);
         // checksum (4 bytes at +68) — not validated
 
         // Build AEIB at aeib_offset
@@ -836,6 +892,23 @@ mod tests {
             let p = elem_start + i * osv;
             file_data[p..p + osv].copy_from_slice(&addr.to_le_bytes());
         }
+        // The index block's checksum covers its prefix, every inline element
+        // slot, and every data block and super block address slot:
+        // ndblk_addrs = 2 * (sup_blk_min_data_ptrs - 1), and the super block
+        // pointers make up the rest of nsblks levels.
+        let sup_ptrs = file_data[aehd_offset + 10] as usize;
+        let dmin = file_data[aehd_offset + 9] as usize;
+        let nsblks = 1 + 10 - dmin.trailing_zeros() as usize;
+        let ndblk_addrs = 2 * (sup_ptrs - 1);
+        // Levels consumed by those direct data blocks (1, 1, 2, 2, ... per level).
+        let mut consumed = 0usize;
+        let mut levels = 0usize;
+        while consumed < ndblk_addrs {
+            consumed += 1 << (levels / 2);
+            levels += 1;
+        }
+        let ib_end = elem_start + num_chunks * osv + (ndblk_addrs + nsblks - levels) * osv;
+        stamp_checksum(&mut file_data, aeib_offset, ib_end);
 
         let header = ExtensibleArrayHeader::parse(&file_data, aehd_offset, os, ls).unwrap();
         let ds_dims = vec![40u64]; // 2 chunks × 20 elements
@@ -885,6 +958,7 @@ mod tests {
         // idx_blk_addr at offset 12 + 6*8 = 60
         file_data[aehd_offset + 60..aehd_offset + 68]
             .copy_from_slice(&(aeib_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, aehd_offset, aehd_offset + 68);
 
         // AEIB
         file_data[aeib_offset..aeib_offset + 4].copy_from_slice(b"EAIB");
@@ -903,42 +977,48 @@ mod tests {
             pos += osv;
         }
 
-        // Direct data block addresses: first sb_level=0 has 1 dblk, sb_level=1 has 1 dblk
-        // Total direct dblks for sblk_min=2: 2^0 + 2^1 = 1 + 2 = 3 (oops)
-        // Actually: sblk_min levels. level 0: 2^0=1 dblk, level 1: 2^1=2 dblks => 3 dblks
-        // But we only have 2 remaining elements.
-        // dblk sizes: level 0: 1 dblk of min_dblk=2; level 1: 2 dblks of 2 each (nelmts doubles at level > 0)
-        // Wait, re-reading the code: at level 0, nelmts=min_dblk=2, 1 dblk.
-        // At level 1, 1 dblk, nelmts still 2 (doubles only at level > 0... but the code says
-        // `if sb_level > 0 { nelmts *= 2 }` after pushing). Let me re-check.
-        // After push at level 0: nelmts=2. Then if 0>0 false, no double. Push 1 dblk of 2.
-        // Level 1: ndblks=2. Push 2 dblks of 2. Then 1>0 true, nelmts=4.
-        // Total: 3 dblks with sizes [2, 2, 2]. Total = 6.
-        // We only need 2 more elements. So only the first dblk has data.
-        let n_direct_dblks = 3;
+        // Direct data block addresses. With sup_blk_min_data_ptrs = 2 the index
+        // block holds 2 * (2 - 1) = 2 of them, which are the data blocks of
+        // super block levels 0 and 1: one of `min_dblk_nelmts` elements, then
+        // one of twice that (ndblks = 2^(u/2), dblk_nelmts = 2^((u+1)/2) * min).
+        // Only the first is allocated here; the rest of the array is empty.
+        let ndblk_addrs = 2 * (sblk_min as usize - 1);
         file_data[pos..pos + osv].copy_from_slice(&(aedb_offset as u64).to_le_bytes());
         pos += osv;
-        // 2 more dblk addresses - undefined
-        for _ in 1..n_direct_dblks {
+        for _ in 1..ndblk_addrs {
             file_data[pos..pos + osv].copy_from_slice(&u64::MAX.to_le_bytes());
             pos += osv;
         }
+        // Super block addresses fill the remaining levels; all unallocated.
+        let nsblks = 1 + 10 - (min_dblk_nelmts as usize).trailing_zeros() as usize;
+        let mut consumed = 0usize;
+        let mut levels = 0usize;
+        while consumed < ndblk_addrs {
+            consumed += 1 << (levels / 2);
+            levels += 1;
+        }
+        for _ in 0..(nsblks - levels) {
+            file_data[pos..pos + osv].copy_from_slice(&u64::MAX.to_le_bytes());
+            pos += osv;
+        }
+        stamp_checksum(&mut file_data, aeib_offset, pos);
 
-        // EADB at aedb_offset (min_dblk_nelmts elements)
+        // EADB holding the first data block's `min_dblk_nelmts` elements.
         file_data[aedb_offset..aedb_offset + 4].copy_from_slice(b"EADB");
         file_data[aedb_offset + 4] = 0;
         file_data[aedb_offset + 5] = 0;
         file_data[aedb_offset + 6..aedb_offset + 14]
             .copy_from_slice(&(aehd_offset as u64).to_le_bytes());
-        // block_offset: ceil(max_nelmts_bits/8) = ceil(10/8) = 2 bytes
-        // block_offset = 0 for first data block
-        let blk_off_size = (10usize).div_ceil(8); // max_nelmts_bits=10
-        let mut dbpos = aedb_offset + 6 + osv + blk_off_size;
+        // Block offset field: ceil(max_nelmts_bits / 8) bytes, zero here.
+        let blk_off_size = (10usize).div_ceil(8);
+        let db_elems = aedb_offset + 6 + osv + blk_off_size;
+        let mut dbpos = db_elems;
         for i in 0..min_dblk_nelmts as usize {
             let addr = base_addr + (idx_blk_elmts as u64 + i as u64) * chunk_byte_size;
             file_data[dbpos..dbpos + osv].copy_from_slice(&addr.to_le_bytes());
             dbpos += osv;
         }
+        stamp_checksum(&mut file_data, aedb_offset, dbpos);
 
         let header = ExtensibleArrayHeader::parse(&file_data, aehd_offset, os, ls).unwrap();
         let ds_dims = vec![40u64];

crates/clawhdf5-format/src/file_writer.rs

@@ -86,6 +86,12 @@ pub(crate) fn build_dataset_oh(
     let mut dl = Vec::new();
     dl.push(4); // version
     dl.push(1); // class = contiguous
+    // An empty dataset has no storage: its address must be the undefined
+    // address, as libhdf5 writes it. A real address with size 0 trips
+    // libhdf5's `addr + size <= addr` overflow check, and it refuses the
+    // dataset as "invalid dataset size, likely file corruption" — which made
+    // every store with no sessions or knowledge graph unreadable by h5py.
+    let data_addr = if data_size == 0 { u64::MAX } else { data_addr };
     dl.extend_from_slice(&data_addr.to_le_bytes());
     dl.extend_from_slice(&data_size.to_le_bytes());
     w.add_message(MessageType::DataLayout, dl);

crates/clawhdf5-format/src/filters.rs

@@ -629,21 +629,70 @@ fn deflate_decompress(data: &[u8], expected_bytes: usize) -> Result<Vec<u8>, For
         // Fall through to flate2 on error
     }
 
-    use std::io::Read;
-    let decoder = flate2::read::ZlibDecoder::new(data);
-    let mut result = Vec::with_capacity(limit.min(1 << 20));
-    // Read one byte past the limit so an over-size stream is distinguishable
+    // A chunk's decompressed size is known, so allocate it once; without one,
+    // start from a multiple of the input and grow.
+    let size_hint = if expected_bytes != 0 {
+        expected_bytes
+    } else {
+        data.len().saturating_mul(4).min(1 << 20)
+    };
+    inflate_bounded(data, size_hint, limit).map_err(FormatError::DecompressionError)
+}
+
+/// Inflate a zlib stream into a buffer sized up front, handing the decoder the
+/// whole input at once.
+///
+/// `flate2::read::ZlibDecoder` feeds its input through a 32 KiB buffer and
+/// grows the output as it goes; on single chunks that cost zlib-rs up to 3.7x
+/// against zlib-ng (`BENCHMARKS.md`, "Deflate backend"). Output beyond `limit`
+/// is an error, as is a stream that ends before its end-of-stream marker (the
+/// streaming reader returned the bytes it had and no error).
+#[cfg(feature = "deflate")]
+pub(crate) fn inflate_bounded(
+    data: &[u8],
+    size_hint: usize,
+    limit: usize,
+) -> Result<Vec<u8>, String> {
+    use flate2::{Decompress, FlushDecompress, Status};
+
+    // One byte of headroom past the limit distinguishes an over-size stream
     // from one that legitimately ends exactly at the limit.
-    decoder
-        .take(limit as u64 + 1)
-        .read_to_end(&mut result)
-        .map_err(|e| FormatError::DecompressionError(e.to_string()))?;
-    if result.len() > limit {
-        return Err(FormatError::DecompressionError(
-            "deflate: output exceeds size limit".into(),
-        ));
+    let max_capacity = limit.saturating_add(1);
+    let mut out = Vec::new();
+    out.try_reserve_exact(size_hint.clamp(1, max_capacity))
+        .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
+
+    let mut inflater = Decompress::new(true);
+    loop {
+        let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
+        let status = inflater
+            .decompress_vec(
+                &data[in_before as usize..],
+                &mut out,
+                FlushDecompress::Finish,
+            )
+            .map_err(|e| format!("deflate: {e}"))?;
+        if out.len() > limit {
+            return Err("deflate: output exceeds size limit".into());
+        }
+        match status {
+            Status::StreamEnd => return Ok(out),
+            Status::Ok | Status::BufError if out.len() == out.capacity() => {
+                // Out of room: double, up to the limit.
+                let grow = out.capacity().min(max_capacity - out.capacity()).max(1);
+                out.try_reserve_exact(grow)
+                    .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
+            }
+            Status::Ok | Status::BufError => {
+                // Room left, so the decoder stopped for want of input.
+                if inflater.total_in() as usize >= data.len()
+                    || (inflater.total_in(), inflater.total_out()) == (in_before, out_before)
+                {
+                    return Err("deflate: truncated stream".into());
+                }
+            }
+        }
     }
-    Ok(result)
 }
 
 /// Direct FFI to Apple's system libz for fast decompression.
@@ -722,14 +771,41 @@ fn deflate_decompress(_data: &[u8], _expected_bytes: usize) -> Result<Vec<u8>, F
 /// Compress data with zlib.
 #[cfg(feature = "deflate")]
 fn deflate_compress(data: &[u8], level: u32) -> Result<Vec<u8>, FormatError> {
-    use std::io::Write;
-    let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(level));
-    encoder
-        .write_all(data)
-        .map_err(|e| FormatError::CompressionError(e.to_string()))?;
-    encoder
-        .finish()
-        .map_err(|e| FormatError::CompressionError(e.to_string()))
+    deflate_bounded(data, level).map_err(FormatError::CompressionError)
+}
+
+/// Deflate `data` into a zlib stream in one pass, into a buffer sized for the
+/// worst case up front (the same reasoning as [`inflate_bounded`]).
+#[cfg(feature = "deflate")]
+pub(crate) fn deflate_bounded(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
+    use flate2::{Compress, Compression, FlushCompress, Status};
+
+    // zlib's compressBound, plus the zlib header and trailer.
+    let bound = data.len() + (data.len() >> 12) + (data.len() >> 14) + (data.len() >> 25) + 13 + 6;
+    let mut out = Vec::new();
+    out.try_reserve_exact(bound)
+        .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
+
+    let mut deflater = Compress::new(Compression::new(level), true);
+    loop {
+        let (in_before, out_before) = (deflater.total_in(), deflater.total_out());
+        let status = deflater
+            .compress_vec(&data[in_before as usize..], &mut out, FlushCompress::Finish)
+            .map_err(|e| format!("deflate: {e}"))?;
+        match status {
+            Status::StreamEnd => return Ok(out),
+            // The bound should make running out of room unreachable; grow
+            // rather than fail if it happens.
+            Status::Ok | Status::BufError if out.len() == out.capacity() => out
+                .try_reserve(out.capacity().max(4096))
+                .map_err(|e| format!("deflate: cannot allocate output: {e}"))?,
+            Status::Ok | Status::BufError => {
+                if (deflater.total_in(), deflater.total_out()) == (in_before, out_before) {
+                    return Err("deflate: encoder made no progress".into());
+                }
+            }
+        }
+    }
 }
 
 #[cfg(not(feature = "deflate"))]
@@ -1833,6 +1909,74 @@ mod tests {
         assert!(deflate_decompress(&compressed, 64).is_err());
     }
 
+    #[cfg(feature = "deflate")]
+    fn noisy_bytes(n: usize) -> Vec<u8> {
+        // Compressible but not trivially so.
+        (0..n)
+            .map(|i| ((i as f64 * 0.01).sin() * 127.0 + 128.0) as u8 ^ (i as u8 & 3))
+            .collect()
+    }
+
+    #[test]
+    #[cfg(feature = "deflate")]
+    fn deflate_decompress_accepts_output_exactly_at_chunk_size() {
+        let data = noisy_bytes(100_000);
+        let compressed = deflate_compress(&data, 6).unwrap();
+        assert_eq!(deflate_decompress(&compressed, data.len()).unwrap(), data);
+        // One byte short of the real size is over the limit.
+        assert!(deflate_decompress(&compressed, data.len() - 1).is_err());
+    }
+
+    #[test]
+    #[cfg(feature = "deflate")]
+    fn deflate_decompress_without_size_grows_the_buffer() {
+        // No chunk size: the output starts at 4x the input and has to grow.
+        let data = vec![7u8; 3 * 1024 * 1024];
+        let compressed = deflate_compress(&data, 6).unwrap();
+        assert!(compressed.len() * 4 < data.len());
+        assert_eq!(deflate_decompress(&compressed, 0).unwrap(), data);
+    }
+
+    #[test]
+    #[cfg(feature = "deflate")]
+    fn deflate_decompress_rejects_truncated_stream() {
+        // The streaming reader this replaced returned the bytes it had and no
+        // error, so a truncated chunk read back short.
+        let data = noisy_bytes(100_000);
+        let compressed = deflate_compress(&data, 6).unwrap();
+        for cut in [compressed.len() - 1, compressed.len() / 2, 3] {
+            assert!(
+                deflate_decompress(&compressed[..cut], data.len()).is_err(),
+                "truncated to {cut} of {} bytes",
+                compressed.len()
+            );
+        }
+    }
+
+    #[test]
+    #[cfg(feature = "deflate")]
+    fn deflate_compress_roundtrips_incompressible_data() {
+        // Random-looking input compresses to slightly more than it started
+        // as; the output must still fit the pre-sized buffer (or grow).
+        let mut x = 0x9E37_79B9_7F4A_7C15u64;
+        let data: Vec<u8> = (0..200_000)
+            .map(|_| {
+                x ^= x << 13;
+                x ^= x >> 7;
+                x ^= x << 17;
+                x as u8
+            })
+            .collect();
+        for level in [0, 1, 6, 9] {
+            let compressed = deflate_compress(&data, level).unwrap();
+            assert_eq!(deflate_decompress(&compressed, data.len()).unwrap(), data);
+        }
+        assert_eq!(
+            deflate_decompress(&deflate_compress(&[], 6).unwrap(), 0).unwrap(),
+            Vec::<u8>::new()
+        );
+    }
+
     #[test]
     #[cfg(feature = "zstd")]
     fn zstd_decompress_rejects_output_exceeding_chunk_size() {

crates/clawhdf5-format/src/fixed_array.rs

@@ -9,6 +9,31 @@ use alloc::{format, vec, vec::Vec};
 use crate::chunked_read::ChunkInfo;
 use crate::error::FormatError;
 
+/// Verify the Jenkins lookup3 checksum stored immediately after
+/// `data[start..end]`, as every Fixed Array structure carries one.
+///
+/// A corrupt chunk index silently yields addresses pointing at the wrong
+/// bytes, so a mismatch has to be an error rather than a shrug: without this
+/// the damage surfaces as plausible-looking data from the wrong chunk.
+#[cfg(feature = "checksum")]
+fn verify_checksum(data: &[u8], start: usize, end: usize) -> Result<(), FormatError> {
+    ensure_len(data, end, 4)?;
+    let stored = u32::from_le_bytes([data[end], data[end + 1], data[end + 2], data[end + 3]]);
+    let computed = crate::checksum::jenkins_lookup3(&data[start..end]);
+    if computed != stored {
+        return Err(FormatError::ChecksumMismatch {
+            expected: stored,
+            computed,
+        });
+    }
+    Ok(())
+}
+
+#[cfg(not(feature = "checksum"))]
+fn verify_checksum(_data: &[u8], _start: usize, _end: usize) -> Result<(), FormatError> {
+    Ok(())
+}
+
 /// Parsed Fixed Array header (FAHD).
 #[derive(Debug, Clone)]
 pub struct FixedArrayHeader {
@@ -103,6 +128,8 @@ impl FixedArrayHeader {
         let num_elements = read_length(d, pos, length_size)?;
         pos += length_size as usize;
         let data_block_address = read_offset(d, pos, offset_size)?;
+        pos += offset_size as usize;
+        verify_checksum(file_data, offset, offset + pos)?;
 
         Ok(FixedArrayHeader {
             client_id,
@@ -223,7 +250,8 @@ pub fn read_fixed_array_chunks(
 
     if !is_paged {
         // Non-paged: prefix, then `num_elements` elements packed directly,
-        // then a trailing checksum (which we don't validate).
+        // then a checksum over both.
+        verify_checksum(file_data, db_offset, elem_at(elements_start, num_elements)?)?;
         for i in 0..num_elements {
             push_element(i, elem_at(elements_start, i)?, &mut chunks)?;
         }
@@ -254,6 +282,9 @@ pub fn read_fixed_array_chunks(
             available: file_data.len(),
         });
     }
+    // The prefix and page bitmap are covered by their own checksum, and each
+    // initialised page by one of its own.
+    verify_checksum(file_data, db_offset, bitmap_start + bitmap_size)?;
 
     for p in 0..npages {
         let page_first = p * page_nelmts; // < num_elements, cannot overflow
@@ -270,6 +301,7 @@ pub fn read_fixed_array_chunks(
             .checked_mul(page_stride)
             .and_then(|o| pages_start.checked_add(o))
             .ok_or_else(stride_overflow)?;
+        verify_checksum(file_data, page_off, elem_at(page_off, page_count)?)?;
         for e in 0..page_count {
             push_element(page_first + e, elem_at(page_off, e)?, &mut chunks)?;
         }
@@ -374,6 +406,14 @@ fn read_variable_length(data: &[u8], size: usize) -> Result<u64, FormatError> {
 mod tests {
     use super::*;
 
+    /// Stamp the Jenkins checksum a real file would carry over
+    /// `data[start..end]`, writing it at `end`. Fixtures built by hand need
+    /// this now that the reader validates it — as every HDF5 writer does.
+    fn stamp_checksum(data: &mut [u8], start: usize, end: usize) {
+        let sum = crate::checksum::jenkins_lookup3(&data[start..end]);
+        data[end..end + 4].copy_from_slice(&sum.to_le_bytes());
+    }
+
     #[test]
     fn index_to_offsets_1d() {
         let num_chunks = vec![5u64];
@@ -439,7 +479,7 @@ mod tests {
         buf[8..16].copy_from_slice(&5u64.to_le_bytes());
         // data_block_address (offset_size=8)
         buf[16..24].copy_from_slice(&0x1000u64.to_le_bytes());
-        // checksum (4 bytes, we don't validate in parse)
+        stamp_checksum(&mut buf, 0, 24);
 
         let header = FixedArrayHeader::parse(&buf, 0, 8, 8).unwrap();
         assert_eq!(header.client_id, 1);
@@ -449,6 +489,54 @@ mod tests {
         assert_eq!(header.data_block_address, 0x1000);
     }
 
+    /// Corruption anywhere in the index must be an error, not a wrong
+    /// address. Every structure carries a checksum; flipping a bit in each in
+    /// turn must be caught, because the alternative is reading a chunk from
+    /// the wrong offset and returning it as data.
+    #[test]
+    fn corrupting_any_fixed_array_structure_is_detected() {
+        let build = || -> (Vec<u8>, usize) {
+            let (os, fahd, db) = (8usize, 0x100usize, 0x200usize);
+            let mut f = vec![0u8; 0x3000];
+            f[fahd..fahd + 4].copy_from_slice(b"FAHD");
+            f[fahd + 6] = os as u8;
+            f[fahd + 7] = 10;
+            f[fahd + 8..fahd + 16].copy_from_slice(&3u64.to_le_bytes());
+            f[fahd + 16..fahd + 24].copy_from_slice(&(db as u64).to_le_bytes());
+            stamp_checksum(&mut f, fahd, fahd + 24);
+            f[db..db + 4].copy_from_slice(b"FADB");
+            f[db + 6..db + 14].copy_from_slice(&(fahd as u64).to_le_bytes());
+            let elems = db + 6 + os;
+            for i in 0..3usize {
+                let addr = 0x1000u64 + i as u64 * 0x100;
+                f[elems + i * os..elems + (i + 1) * os].copy_from_slice(&addr.to_le_bytes());
+            }
+            stamp_checksum(&mut f, db, elems + 3 * os);
+            (f, fahd)
+        };
+
+        let read = |f: &[u8], fahd: usize| -> Result<Vec<ChunkInfo>, FormatError> {
+            let h = FixedArrayHeader::parse(f, fahd, 8, 8)?;
+            read_fixed_array_chunks(f, &h, &[60], &[20], 8, 8, 8)
+        };
+
+        let (clean, fahd) = build();
+        assert!(read(&clean, fahd).is_ok(), "the intact fixture must read");
+
+        // A byte inside the header, and one inside a data block element.
+        for &at in &[0x108usize, 0x210usize] {
+            let (mut damaged, fahd) = build();
+            damaged[at] ^= 0x01;
+            assert!(
+                matches!(
+                    read(&damaged, fahd),
+                    Err(FormatError::ChecksumMismatch { .. })
+                ),
+                "corruption at {at:#x} went undetected"
+            );
+        }
+    }
+
     #[test]
     fn parse_fixed_array_header_invalid_signature() {
         let mut buf = vec![0u8; 256];
@@ -469,6 +557,7 @@ mod tests {
         buf[fahd + 7] = 200; // max_nelmts_bits — absurd, would overflow a shift
         buf[fahd + 8..fahd + 16].copy_from_slice(&3u64.to_le_bytes()); // num_elements
         buf[fahd + 16..fahd + 24].copy_from_slice(&0x100u64.to_le_bytes());
+        stamp_checksum(&mut buf, fahd, fahd + 24);
         // FADB so parsing reaches the paged check
         let db = 0x100usize;
         buf[db..db + 4].copy_from_slice(b"FADB");
@@ -486,6 +575,8 @@ mod tests {
         buf[fahd + 7] = 10;
         buf[fahd + 8..fahd + 16].copy_from_slice(&u64::MAX.to_le_bytes()); // absurd count
         buf[fahd + 16..fahd + 24].copy_from_slice(&0x80u64.to_le_bytes());
+        // Valid checksum, so it is the element count that must be rejected.
+        stamp_checksum(&mut buf, fahd, fahd + 24);
         buf[0x80..0x84].copy_from_slice(b"FADB");
         let header = FixedArrayHeader::parse(&buf, fahd, 8, 8).unwrap();
         let r = read_fixed_array_chunks(&buf, &header, &[100], &[20], 8, 8, 8);
@@ -545,6 +636,7 @@ mod tests {
         file_data[fahd_offset + 8..fahd_offset + 16].copy_from_slice(&num_chunks.to_le_bytes());
         file_data[fahd_offset + 16..fahd_offset + 24]
             .copy_from_slice(&(db_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, fahd_offset, fahd_offset + 24);
 
         // Build FADB at db_offset
         file_data[db_offset..db_offset + 4].copy_from_slice(b"FADB");
@@ -562,6 +654,7 @@ mod tests {
             let pos = elem_start + i * os;
             file_data[pos..pos + os].copy_from_slice(&addr.to_le_bytes());
         }
+        stamp_checksum(&mut file_data, db_offset, elem_start + 5 * os);
 
         let header =
             FixedArrayHeader::parse(&file_data, fahd_offset, offset_size, length_size).unwrap();
@@ -611,6 +704,7 @@ mod tests {
         file_data[fahd_offset + 8..fahd_offset + 16].copy_from_slice(&num_chunks.to_le_bytes());
         file_data[fahd_offset + 16..fahd_offset + 24]
             .copy_from_slice(&(db_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, fahd_offset, fahd_offset + 24);
 
         file_data[db_offset..db_offset + 4].copy_from_slice(b"FADB");
         file_data[db_offset + 4] = 0;
@@ -632,6 +726,11 @@ mod tests {
             file_data[pos + os..pos + os + 4].copy_from_slice(&csize.to_le_bytes());
             file_data[pos + os + 4..pos + os + 8].copy_from_slice(&fmask.to_le_bytes());
         }
+        stamp_checksum(
+            &mut file_data,
+            db_offset,
+            elem_start + test_chunks.len() * elem_size,
+        );
 
         let header =
             FixedArrayHeader::parse(&file_data, fahd_offset, offset_size, length_size).unwrap();
@@ -696,6 +795,7 @@ mod tests {
         file_data[fahd_offset + 8..fahd_offset + 16].copy_from_slice(&num_elements.to_le_bytes());
         file_data[fahd_offset + 16..fahd_offset + 24]
             .copy_from_slice(&(db_offset as u64).to_le_bytes());
+        stamp_checksum(&mut file_data, fahd_offset, fahd_offset + 24);
 
         // FADB prefix
         file_data[db_offset..db_offset + 4].copy_from_slice(b"FADB");
@@ -715,6 +815,9 @@ mod tests {
         let base_addr = 0x1000u64;
         // Page 0 (elements 0..4) and page 2 (elements 8..11) carry addresses;
         // page 1's slot is left zero-filled and must be skipped.
+        // The prefix and bitmap carry one checksum, each initialised page
+        // another — as a real file does.
+        stamp_checksum(&mut file_data, db_offset, bitmap_off + bitmap_size);
         for &p in &[0usize, 2usize] {
             let page_off = pages_start + p * page_total;
             let count = core::cmp::min(page_nelmts, num_elements as usize - p * page_nelmts);
@@ -724,6 +827,7 @@ mod tests {
                 let pos = page_off + e * os;
                 file_data[pos..pos + os].copy_from_slice(&addr.to_le_bytes());
             }
+            stamp_checksum(&mut file_data, page_off, page_off + count * os);
         }
 
         let header =

crates/clawhdf5-format/src/float16.rs

@@ -0,0 +1,155 @@
+//! IEEE-754 half precision (binary16) conversions.
+//!
+//! Pure integer bit manipulation, so it works under `no_std` and needs no
+//! `libm`. The writer ([`crate::type_builders::DatasetBuilder::with_f16_data`]),
+//! the reader and `clawhdf5-agent`'s half-precision embedding store all use
+//! these two functions, so a value rounded in memory is bit-for-bit the value
+//! that reads back from the file.
+
+/// Largest finite half-precision value. Anything larger in magnitude rounds
+/// to infinity.
+pub const F16_MAX: f32 = 65504.0;
+
+/// Convert an `f32` to the bit pattern of the nearest half-precision value,
+/// rounding ties to even (the IEEE default, and what numpy and the `half`
+/// crate do).
+///
+/// Values beyond ±[`F16_MAX`] become ±infinity, values too small for a
+/// subnormal become signed zero, and NaN stays NaN (quiet, payload
+/// truncated).
+pub fn f32_to_f16_bits(value: f32) -> u16 {
+    let x = value.to_bits();
+    let sign = (x >> 16) & 0x8000;
+    let exp = x & 0x7F80_0000;
+    let man = x & 0x007F_FFFF;
+
+    // Infinity and NaN.
+    if exp == 0x7F80_0000 {
+        let quiet_nan = if man == 0 { 0 } else { 0x0200 };
+        return (sign | 0x7C00 | quiet_nan | (man >> 13)) as u16;
+    }
+
+    let half_exp = ((exp >> 23) as i32) - 127 + 15;
+
+    // Too large: infinity.
+    if half_exp >= 0x1F {
+        return (sign | 0x7C00) as u16;
+    }
+
+    // Subnormal half, or zero.
+    if half_exp <= 0 {
+        if 14 - half_exp > 24 {
+            return sign as u16;
+        }
+        let man = man | 0x0080_0000; // implicit leading bit
+        let shift = (14 - half_exp) as u32;
+        let mut half_man = man >> shift;
+        let round_bit = 1u32 << (shift - 1);
+        // Round half to even: up if above half, or exactly half and odd.
+        if (man & round_bit) != 0 && (man & (3 * round_bit - 1)) != 0 {
+            half_man += 1;
+        }
+        return (sign | half_man) as u16;
+    }
+
+    // Normal half. A mantissa carry correctly rolls into the exponent (and
+    // from the largest finite value into infinity).
+    let half = sign | ((half_exp as u32) << 10) | (man >> 13);
+    let round_bit = 0x0000_1000;
+    if (man & round_bit) != 0 && (man & (3 * round_bit - 1)) != 0 {
+        (half + 1) as u16
+    } else {
+        half as u16
+    }
+}
+
+/// Convert the bit pattern of a half-precision value to `f32` (exact: every
+/// half value is representable as an `f32`).
+pub fn f16_bits_to_f32(h: u16) -> f32 {
+    let h = h as u32;
+    let sign = (h & 0x8000) << 16;
+    let exp = (h >> 10) & 0x1f;
+    let mant = h & 0x3ff;
+    let bits = if exp == 0 {
+        if mant == 0 {
+            sign // signed zero
+        } else {
+            // Subnormal: normalize into an f32 normal.
+            let mut e: i32 = -1;
+            let mut m = mant;
+            loop {
+                e += 1;
+                m <<= 1;
+                if m & 0x400 != 0 {
+                    break;
+                }
+            }
+            let m = m & 0x3ff;
+            sign | (((127 - 15 - e) as u32) << 23) | (m << 13)
+        }
+    } else if exp == 0x1f {
+        sign | 0x7f80_0000 | (mant << 13) // inf / NaN
+    } else {
+        sign | ((exp + 127 - 15) << 23) | (mant << 13)
+    };
+    f32::from_bits(bits)
+}
+
+/// Round an `f32` to the nearest half-precision value, returned as `f32`.
+pub fn round_to_f16(value: f32) -> f32 {
+    f16_bits_to_f32(f32_to_f16_bits(value))
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn every_half_value_round_trips() {
+        for bits in 0..=u16::MAX {
+            let v = f16_bits_to_f32(bits);
+            if v.is_nan() {
+                assert!(f16_bits_to_f32(f32_to_f16_bits(v)).is_nan(), "{bits:#06x}");
+            } else {
+                assert_eq!(f32_to_f16_bits(v), bits, "{bits:#06x} -> {v}");
+            }
+        }
+    }
+
+    #[test]
+    fn matches_the_half_crate() {
+        // Every 257th f32 bit pattern (~16.7M values) covers every exponent,
+        // the subnormal range, both signs, ties and the overflow boundary.
+        let mut bits: u32 = 0;
+        loop {
+            let v = f32::from_bits(bits);
+            let ours = f32_to_f16_bits(v);
+            let theirs = half::f16::from_f32(v);
+            if v.is_nan() {
+                assert!(theirs.is_nan() && f16_bits_to_f32(ours).is_nan());
+            } else {
+                assert_eq!(ours, theirs.to_bits(), "{bits:#010x} ({v:e})");
+                assert_eq!(f16_bits_to_f32(ours).to_bits(), theirs.to_f32().to_bits());
+            }
+            match bits.checked_add(257) {
+                Some(b) => bits = b,
+                None => break,
+            }
+        }
+    }
+
+    #[test]
+    fn rounds_ties_to_even_and_saturates_to_infinity() {
+        // 1 + 2^-11 is exactly halfway between 1.0 and the next half (1 + 2^-10).
+        assert_eq!(round_to_f16(1.0 + 2f32.powi(-11)), 1.0);
+        assert_eq!(
+            round_to_f16(1.0 + 3.0 * 2f32.powi(-11)),
+            1.0 + 2.0 * 2f32.powi(-10)
+        );
+        assert_eq!(round_to_f16(F16_MAX), F16_MAX);
+        assert_eq!(round_to_f16(65520.0), f32::INFINITY); // halfway to 2^16 rounds up
+        assert_eq!(round_to_f16(-1e9), f32::NEG_INFINITY);
+        assert_eq!(round_to_f16(1e-9).to_bits(), 0);
+        assert_eq!(round_to_f16(-1e-9).to_bits(), (-0.0f32).to_bits());
+    }
+}

crates/clawhdf5-format/src/lib.rs

@@ -72,6 +72,7 @@ pub mod filter_pipeline;
 pub mod filters;
 mod filters_szip;
 pub mod fixed_array;
+pub mod float16;
 pub mod fractal_heap;
 pub mod global_heap;
 pub mod group_info;

crates/clawhdf5-format/src/type_builders.rs

@@ -56,6 +56,21 @@ pub fn make_f64_type() -> Datatype {
     }
 }
 
+/// IEEE-754 half precision (binary16), little-endian — numpy's `float16`.
+pub fn make_f16_type() -> Datatype {
+    Datatype::FloatingPoint {
+        size: 2,
+        byte_order: DatatypeByteOrder::LittleEndian,
+        bit_offset: 0,
+        bit_precision: 16,
+        exponent_location: 10,
+        exponent_size: 5,
+        mantissa_location: 0,
+        mantissa_size: 10,
+        exponent_bias: 15,
+    }
+}
+
 pub fn make_f32_type() -> Datatype {
     Datatype::FloatingPoint {
         size: 4,
@@ -478,6 +493,24 @@ impl DatasetBuilder {
         self
     }
 
+    /// Store `data` as IEEE half precision (numpy `float16`), rounding each
+    /// value to the nearest half ([`crate::float16::f32_to_f16_bits`]).
+    /// Half the bytes of [`Self::with_f32_data`], at about three significant
+    /// decimal digits; values beyond ±65504 become ±infinity. Reading it back
+    /// with `read_f32` yields the rounded values exactly.
+    pub fn with_f16_data(&mut self, data: &[f32]) -> &mut Self {
+        self.datatype = Some(make_f16_type());
+        let mut b = Vec::with_capacity(data.len() * 2);
+        for &v in data {
+            b.extend_from_slice(&crate::float16::f32_to_f16_bits(v).to_le_bytes());
+        }
+        self.data = Some(b);
+        if self.shape.is_none() {
+            self.shape = Some(vec![data.len() as u64]);
+        }
+        self
+    }
+
     pub fn with_i32_data(&mut self, data: &[i32]) -> &mut Self {
         self.datatype = Some(make_i32_type());
         let mut b = Vec::with_capacity(data.len() * 4);

crates/clawhdf5-gpu/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-gpu"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "GPU-accelerated vector operations for rustyhdf5 using wgpu compute shaders"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"

crates/clawhdf5-io/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-io"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "I/O abstraction layer for rustyhdf5"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -10,7 +11,7 @@ keywords = ["hdf5", "io", "science", "data"]
 categories = ["filesystem", "science"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 memmap2 = { version = "0.9", optional = true }
 libc = { version = "0.2", optional = true }
 tokio = { version = "1", features = ["fs", "io-util"], optional = true }

crates/clawhdf5-migrate/Cargo.toml

@@ -1,8 +1,9 @@
 [package]
 name = "clawhdf5-migrate"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
-description = "CLI to migrate SQLite agent memory databases to HDF5 format"
+rust-version.workspace = true
+description = "CLI to migrate SQLite agent memory databases to clawhdf5-agent stores"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
 readme = "README.md"
@@ -14,12 +15,10 @@ name = "clawhdf5-migrate"
 path = "src/main.rs"
 
 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.6.0" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
-clawhdf5 = { path = "../clawhdf5", version = "2.6.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 rusqlite = { version = "0.31", features = ["bundled"] }
 clap = { version = "4", features = ["derive"] }
-half = { workspace = true }
 
 [dev-dependencies]
 tempfile = { workspace = true }

crates/clawhdf5-migrate/README.md

@@ -3,9 +3,15 @@
 [![crates.io](https://img.shields.io/crates/v/clawhdf5-migrate.svg)](https://crates.io/crates/clawhdf5-migrate)
 [![docs.rs](https://img.shields.io/docsrs/clawhdf5-migrate)](https://docs.rs/clawhdf5-migrate)
 
-CLI tool to migrate SQLite agent memory databases to HDF5 format.
+CLI tool to migrate a SQLite agent-memory database in the `memory_chunks` / `sessions` / `entities` / `relations` layout (table and
+column names are configurable) to a
+[clawhdf5-agent](https://crates.io/crates/clawhdf5-agent) store. This is **not**
+ZeroClaw's schema — ZeroClaw keeps memories in a single `memories` table and
+does not use clawhdf5.
 
-Converts existing SQLite-based agent memory stores (embeddings, text chunks, metadata) into the HDF5 format used by [clawhdf5-agent](https://crates.io/crates/clawhdf5-agent).
+The output is written through `clawhdf5-agent`'s own API, so it opens with
+`HDF5Memory::open` and is searchable immediately: memory records, sessions and
+the knowledge graph (entities and relations) are carried over.
 
 ## Installation
 
@@ -16,9 +22,19 @@ cargo install clawhdf5-migrate
 ## Usage
 
 ```bash
-clawhdf5-migrate --input agent.db --output agent.h5
+clawhdf5-migrate --sqlite agent.db --hdf5 agent.h5 --agent-id my-agent
 ```
 
+Embeddings are stored as float16 (the library default for new stores); pass
+`--f32` for full precision. Every embedding must have the same dimension
+(the first row's, or `--embedding-dim`, which a source with no memory records
+requires); rows are never truncated, and the whole source is checked before an
+existing output store is replaced. `--incremental` adds only new rows to an
+existing store of the same dimension and carries over changes to rows'
+deleted flags, `--skip-deleted` leaves out tombstoned rows, and `--dry-run`
+only counts.
+See `clawhdf5-migrate --help` for every option.
+
 ## License
 
 MIT

crates/clawhdf5-migrate/src/hdf5_reader.rs

@@ -1,163 +0,0 @@
-//! Read a migration HDF5 file back into the in-memory data model.
-//!
-//! Used to verify migrated content (real validation) and to merge new rows into
-//! an existing output (incremental migration). Mirrors the layout produced by
-//! [`crate::hdf5_writer`].
-
-use clawhdf5::reader::{File, Group};
-use clawhdf5_format::type_builders::AttrValue;
-
-use crate::sqlite_reader::{Entity, MemoryChunk, Relation, Session, SqliteData};
-
-type BoxErr = Box<dyn std::error::Error>;
-
-fn read_strings(group: &Group<'_>, name: &str) -> Result<Vec<String>, BoxErr> {
-    Ok(group.dataset(name)?.read_string()?)
-}
-
-fn read_i64s(group: &Group<'_>, name: &str) -> Result<Vec<i64>, BoxErr> {
-    Ok(group.dataset(name)?.read_i64()?)
-}
-
-fn read_f64s(group: &Group<'_>, name: &str) -> Result<Vec<f64>, BoxErr> {
-    Ok(group.dataset(name)?.read_f64()?)
-}
-
-/// Read the embeddings dataset as a flat `Vec<f32>` of `n * dim` values,
-/// handling both f32 and (lossy) f16 storage.
-fn read_embeddings_flat(group: &Group<'_>) -> Result<Vec<f32>, BoxErr> {
-    Ok(group.dataset("embeddings")?.read_f32()?)
-}
-
-/// Read a migration HDF5 file into a [`SqliteData`].
-pub fn read_hdf5(path: &str) -> Result<SqliteData, BoxErr> {
-    let file = File::open(path)?;
-
-    let embedding_dim = match file.root().attrs()?.get("embedding_dim") {
-        Some(AttrValue::I64(d)) => *d as usize,
-        _ => 0,
-    };
-
-    let chunks = read_chunks(&file, embedding_dim)?;
-    let sessions = read_sessions(&file)?;
-    let entities = read_entities(&file)?;
-    let relations = read_relations(&file)?;
-
-    Ok(SqliteData {
-        chunks,
-        sessions,
-        entities,
-        relations,
-        embedding_dim,
-        // Not a SQLite read — the caller (incremental migration) carries
-        // forward the current run's actual `source_path` from the fresh
-        // SQLite read instead of using this placeholder.
-        source_path: String::new(),
-    })
-}
-
-fn read_chunks(file: &File, dim: usize) -> Result<Vec<MemoryChunk>, BoxErr> {
-    let g = file.group("chunks")?;
-    let count = group_count(&g)?;
-    if count == 0 {
-        return Ok(Vec::new());
-    }
-    let ids = read_i64s(&g, "id")?;
-    let texts = read_strings(&g, "text")?;
-    let channels = read_strings(&g, "source_channel")?;
-    let timestamps = read_f64s(&g, "timestamp")?;
-    let session_ids = read_strings(&g, "session_id")?;
-    let tags = read_strings(&g, "tags")?;
-    let deleted = g.dataset("deleted")?.read_i32()?;
-    let emb_flat = read_embeddings_flat(&g)?;
-    let dim = dim.max(1);
-
-    let mut chunks = Vec::with_capacity(ids.len());
-    for (i, &id) in ids.iter().enumerate() {
-        let embedding = emb_flat
-            .get(i * dim..(i + 1) * dim)
-            .map(|s| s.to_vec())
-            .unwrap_or_default();
-        chunks.push(MemoryChunk {
-            id,
-            chunk: texts.get(i).cloned().unwrap_or_default(),
-            embedding,
-            source_channel: channels.get(i).cloned().unwrap_or_default(),
-            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
-            session_id: session_ids.get(i).cloned().unwrap_or_default(),
-            tags: tags.get(i).cloned().unwrap_or_default(),
-            deleted: deleted.get(i).copied().unwrap_or(0),
-        });
-    }
-    Ok(chunks)
-}
-
-fn read_sessions(file: &File) -> Result<Vec<Session>, BoxErr> {
-    let g = file.group("sessions")?;
-    if group_count(&g)? == 0 {
-        return Ok(Vec::new());
-    }
-    let ids = read_strings(&g, "id")?;
-    let starts = read_i64s(&g, "start_idx")?;
-    let ends = read_i64s(&g, "end_idx")?;
-    let channels = read_strings(&g, "channel")?;
-    let timestamps = read_f64s(&g, "timestamp")?;
-    let summaries = read_strings(&g, "summary")?;
-    Ok((0..ids.len())
-        .map(|i| Session {
-            id: ids[i].clone(),
-            start_idx: starts.get(i).copied().unwrap_or(0),
-            end_idx: ends.get(i).copied().unwrap_or(0),
-            channel: channels.get(i).cloned().unwrap_or_default(),
-            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
-            summary: summaries.get(i).cloned().unwrap_or_default(),
-        })
-        .collect())
-}
-
-fn read_entities(file: &File) -> Result<Vec<Entity>, BoxErr> {
-    let g = file.group("entities")?;
-    if group_count(&g)? == 0 {
-        return Ok(Vec::new());
-    }
-    let ids = read_i64s(&g, "id")?;
-    let names = read_strings(&g, "name")?;
-    let types = read_strings(&g, "type")?;
-    let emb_idxs = read_i64s(&g, "embedding_idx")?;
-    Ok((0..ids.len())
-        .map(|i| Entity {
-            id: ids[i],
-            name: names.get(i).cloned().unwrap_or_default(),
-            entity_type: types.get(i).cloned().unwrap_or_default(),
-            embedding_idx: emb_idxs.get(i).copied().unwrap_or(-1),
-        })
-        .collect())
-}
-
-fn read_relations(file: &File) -> Result<Vec<Relation>, BoxErr> {
-    let g = file.group("relations")?;
-    if group_count(&g)? == 0 {
-        return Ok(Vec::new());
-    }
-    let srcs = read_i64s(&g, "src")?;
-    let tgts = read_i64s(&g, "tgt")?;
-    let rels = read_strings(&g, "relation")?;
-    let weights = read_f64s(&g, "weight")?;
-    let timestamps = read_f64s(&g, "timestamp")?;
-    Ok((0..srcs.len())
-        .map(|i| Relation {
-            src: srcs[i],
-            tgt: tgts.get(i).copied().unwrap_or(0),
-            relation: rels.get(i).cloned().unwrap_or_default(),
-            weight: weights.get(i).copied().unwrap_or(1.0),
-            timestamp: timestamps.get(i).copied().unwrap_or(0.0),
-        })
-        .collect())
-}
-
-fn group_count(group: &Group<'_>) -> Result<u64, BoxErr> {
-    match group.attrs()?.get("count") {
-        Some(AttrValue::I64(n)) => Ok(*n as u64),
-        _ => Ok(0),
-    }
-}

crates/clawhdf5-migrate/src/hdf5_writer.rs

@@ -1,366 +0,0 @@
-use clawhdf5::writer::FileBuilder;
-use clawhdf5_format::datatype::{CharacterSet, Datatype, StringPadding};
-use clawhdf5_format::type_builders::AttrValue;
-
-use crate::sqlite_reader::SqliteData;
-
-/// Options controlling HDF5 output.
-pub struct WriteOptions {
-    pub agent_id: String,
-    pub embedder: String,
-    pub compression: bool,
-    pub compression_level: u32,
-    pub float16: bool,
-}
-
-/// Write SQLite data to an HDF5 file.
-pub fn write_hdf5(
-    path: &str,
-    data: &SqliteData,
-    opts: &WriteOptions,
-) -> Result<(), Box<dyn std::error::Error>> {
-    let mut builder = FileBuilder::new();
-    let timestamp = iso8601_now();
-
-    // Root-level metadata attributes
-    builder.set_attr("agent_id", AttrValue::String(opts.agent_id.clone()));
-    builder.set_attr("embedder", AttrValue::String(opts.embedder.clone()));
-    builder.set_attr("embedding_dim", AttrValue::I64(data.embedding_dim as i64));
-    builder.set_attr("source", AttrValue::String("sqlite-migration".into()));
-    builder.set_attr("version", AttrValue::I64(1));
-    // Lineage: which SQLite database this output was migrated from and when,
-    // plus the migrator tool version — so a chain of `--incremental` runs
-    // still has an audit trail instead of every run overwriting the same
-    // static attributes (see research/03_provenance.md, INT-03).
-    builder.set_attr("source_path", AttrValue::String(data.source_path.clone()));
-    builder.set_attr("migrated_at", AttrValue::String(timestamp.clone()));
-    builder.set_attr(
-        "migrator_version",
-        AttrValue::String(env!("CARGO_PKG_VERSION").to_owned()),
-    );
-
-    write_chunks_group(&mut builder, data, opts, &timestamp);
-    write_sessions_group(&mut builder, data);
-    write_entities_group(&mut builder, data);
-    write_relations_group(&mut builder, data);
-
-    builder.write(path)?;
-    Ok(())
-}
-
-/// Current UTC time formatted as an ISO-8601 / RFC-3339 timestamp
-/// (`YYYY-MM-DDTHH:MM:SSZ`), with no external date/time dependency.
-fn iso8601_now() -> String {
-    let secs = std::time::SystemTime::now()
-        .duration_since(std::time::UNIX_EPOCH)
-        .unwrap_or_default()
-        .as_secs();
-    let days = (secs / 86_400) as i64;
-    let time_of_day = secs % 86_400;
-    let (h, m, s) = (
-        time_of_day / 3600,
-        (time_of_day % 3600) / 60,
-        time_of_day % 60,
-    );
-    let (y, mo, d) = civil_from_days(days);
-    format!("{y:04}-{mo:02}-{d:02}T{h:02}:{m:02}:{s:02}Z")
-}
-
-/// Days-since-epoch to (year, month, day), Howard Hinnant's `civil_from_days`
-/// algorithm (proleptic Gregorian calendar, valid for the full `i64` range).
-fn civil_from_days(z: i64) -> (i64, u32, u32) {
-    let z = z + 719_468;
-    let era = if z >= 0 { z } else { z - 146_096 } / 146_097;
-    let doe = (z - era * 146_097) as u64; // [0, 146096]
-    let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365; // [0, 399]
-    let y = yoe as i64 + era * 400;
-    let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); // [0, 365]
-    let mp = (5 * doy + 2) / 153; // [0, 11]
-    let d = (doy - (153 * mp + 2) / 5 + 1) as u32; // [1, 31]
-    let m = (if mp < 10 { mp + 3 } else { mp - 9 }) as u32; // [1, 12]
-    let y = if m <= 2 { y + 1 } else { y };
-    (y, m, d)
-}
-
-/// Build a fixed-length string Datatype from the max byte length of the items.
-fn string_dtype(max_len: usize) -> Datatype {
-    Datatype::String {
-        size: max_len.max(1) as u32,
-        padding: StringPadding::NullPad,
-        charset: CharacterSet::Utf8,
-    }
-}
-
-/// Pack a slice of strings into null-padded raw bytes of uniform width.
-fn pack_strings(strings: &[String]) -> (Vec<u8>, usize) {
-    let max_len = strings.iter().map(|s| s.len()).max().unwrap_or(0).max(1);
-    let mut buf = vec![0u8; strings.len() * max_len];
-    for (i, s) in strings.iter().enumerate() {
-        let start = i * max_len;
-        let bytes = s.as_bytes();
-        let copy_len = bytes.len().min(max_len);
-        buf[start..start + copy_len].copy_from_slice(&bytes[..copy_len]);
-    }
-    (buf, max_len)
-}
-
-fn apply_compression(ds: &mut clawhdf5_format::type_builders::DatasetBuilder, opts: &WriteOptions) {
-    if opts.compression {
-        ds.with_deflate(opts.compression_level);
-        ds.with_shuffle();
-    }
-}
-
-fn write_chunks_group(
-    builder: &mut FileBuilder,
-    data: &SqliteData,
-    opts: &WriteOptions,
-    timestamp: &str,
-) {
-    let mut group = builder.create_group("chunks");
-    let n = data.chunks.len() as u64;
-
-    if n == 0 {
-        group.set_attr("count", AttrValue::I64(0));
-        builder.add_group(group.finish());
-        return;
-    }
-
-    group.set_attr("count", AttrValue::I64(n as i64));
-
-    // Source attribution attached directly to the content-bearing datasets
-    // (SHA-256 of the raw bytes + creator/timestamp/source), so the chunk
-    // text and embeddings each carry their own verifiable provenance
-    // (see clawhdf5_format::provenance / `Dataset::verify_provenance`).
-    let source_opt = if data.source_path.is_empty() {
-        None
-    } else {
-        Some(data.source_path.as_str())
-    };
-
-    // ids
-    let ids: Vec<i64> = data.chunks.iter().map(|c| c.id).collect();
-    group.create_dataset("id").with_i64_data(&ids);
-
-    // text
-    let texts: Vec<String> = data.chunks.iter().map(|c| c.chunk.clone()).collect();
-    let (text_raw, text_len) = pack_strings(&texts);
-    group
-        .create_dataset("text")
-        .with_compound_data(string_dtype(text_len), text_raw, n)
-        .with_provenance("clawhdf5-migrate", timestamp, source_opt);
-
-    // embeddings - flatten to [N, dim]
-    let dim = data.embedding_dim;
-    if opts.float16 {
-        let f16_data: Vec<u16> = data
-            .chunks
-            .iter()
-            .flat_map(|c| {
-                c.embedding
-                    .iter()
-                    .map(|&v| half::f16::from_f32(v).to_bits())
-            })
-            .collect();
-        let raw: Vec<u8> = f16_data.iter().flat_map(|v| v.to_le_bytes()).collect();
-        let f16_dtype = Datatype::FloatingPoint {
-            size: 2,
-            byte_order: clawhdf5_format::datatype::DatatypeByteOrder::LittleEndian,
-            bit_offset: 0,
-            bit_precision: 16,
-            exponent_location: 10,
-            exponent_size: 5,
-            mantissa_location: 0,
-            mantissa_size: 10,
-            exponent_bias: 15,
-        };
-        let ds = group
-            .create_dataset("embeddings")
-            .with_compound_data(f16_dtype, raw, n)
-            .with_shape(&[n, dim as u64])
-            .with_provenance("clawhdf5-migrate", timestamp, source_opt);
-        apply_compression(ds, opts);
-    } else {
-        let flat: Vec<f32> = data
-            .chunks
-            .iter()
-            .flat_map(|c| c.embedding.iter().copied())
-            .collect();
-        let ds = group
-            .create_dataset("embeddings")
-            .with_f32_data(&flat)
-            .with_shape(&[n, dim as u64])
-            .with_provenance("clawhdf5-migrate", timestamp, source_opt);
-        apply_compression(ds, opts);
-    }
-
-    // source_channel
-    let channels: Vec<String> = data
-        .chunks
-        .iter()
-        .map(|c| c.source_channel.clone())
-        .collect();
-    let (ch_raw, ch_len) = pack_strings(&channels);
-    group
-        .create_dataset("source_channel")
-        .with_compound_data(string_dtype(ch_len), ch_raw, n);
-
-    // timestamp
-    let timestamps: Vec<f64> = data.chunks.iter().map(|c| c.timestamp).collect();
-    group.create_dataset("timestamp").with_f64_data(&timestamps);
-
-    // session_id
-    let sess_ids: Vec<String> = data.chunks.iter().map(|c| c.session_id.clone()).collect();
-    let (sid_raw, sid_len) = pack_strings(&sess_ids);
-    group
-        .create_dataset("session_id")
-        .with_compound_data(string_dtype(sid_len), sid_raw, n);
-
-    // tags
-    let tags: Vec<String> = data.chunks.iter().map(|c| c.tags.clone()).collect();
-    let (tag_raw, tag_len) = pack_strings(&tags);
-    group
-        .create_dataset("tags")
-        .with_compound_data(string_dtype(tag_len), tag_raw, n);
-
-    // deleted
-    let deleted: Vec<i32> = data.chunks.iter().map(|c| c.deleted).collect();
-    group.create_dataset("deleted").with_i32_data(&deleted);
-
-    builder.add_group(group.finish());
-}
-
-fn write_sessions_group(builder: &mut FileBuilder, data: &SqliteData) {
-    let mut group = builder.create_group("sessions");
-    let n = data.sessions.len() as u64;
-    group.set_attr("count", AttrValue::I64(n as i64));
-
-    if n == 0 {
-        builder.add_group(group.finish());
-        return;
-    }
-
-    let ids: Vec<String> = data.sessions.iter().map(|s| s.id.clone()).collect();
-    let (id_raw, id_len) = pack_strings(&ids);
-    group
-        .create_dataset("id")
-        .with_compound_data(string_dtype(id_len), id_raw, n);
-
-    let start_idxs: Vec<i64> = data.sessions.iter().map(|s| s.start_idx).collect();
-    group.create_dataset("start_idx").with_i64_data(&start_idxs);
-
-    let end_idxs: Vec<i64> = data.sessions.iter().map(|s| s.end_idx).collect();
-    group.create_dataset("end_idx").with_i64_data(&end_idxs);
-
-    let channels: Vec<String> = data.sessions.iter().map(|s| s.channel.clone()).collect();
-    let (ch_raw, ch_len) = pack_strings(&channels);
-    group
-        .create_dataset("channel")
-        .with_compound_data(string_dtype(ch_len), ch_raw, n);
-
-    let timestamps: Vec<f64> = data.sessions.iter().map(|s| s.timestamp).collect();
-    group.create_dataset("timestamp").with_f64_data(&timestamps);
-
-    let summaries: Vec<String> = data.sessions.iter().map(|s| s.summary.clone()).collect();
-    let (sum_raw, sum_len) = pack_strings(&summaries);
-    group
-        .create_dataset("summary")
-        .with_compound_data(string_dtype(sum_len), sum_raw, n);
-
-    builder.add_group(group.finish());
-}
-
-fn write_entities_group(builder: &mut FileBuilder, data: &SqliteData) {
-    let mut group = builder.create_group("entities");
-    let n = data.entities.len() as u64;
-    group.set_attr("count", AttrValue::I64(n as i64));
-
-    if n == 0 {
-        builder.add_group(group.finish());
-        return;
-    }
-
-    let ids: Vec<i64> = data.entities.iter().map(|e| e.id).collect();
-    group.create_dataset("id").with_i64_data(&ids);
-
-    let names: Vec<String> = data.entities.iter().map(|e| e.name.clone()).collect();
-    let (name_raw, name_len) = pack_strings(&names);
-    group
-        .create_dataset("name")
-        .with_compound_data(string_dtype(name_len), name_raw, n);
-
-    let types: Vec<String> = data
-        .entities
-        .iter()
-        .map(|e| e.entity_type.clone())
-        .collect();
-    let (type_raw, type_len) = pack_strings(&types);
-    group
-        .create_dataset("type")
-        .with_compound_data(string_dtype(type_len), type_raw, n);
-
-    let emb_idxs: Vec<i64> = data.entities.iter().map(|e| e.embedding_idx).collect();
-    group
-        .create_dataset("embedding_idx")
-        .with_i64_data(&emb_idxs);
-
-    builder.add_group(group.finish());
-}
-
-fn write_relations_group(builder: &mut FileBuilder, data: &SqliteData) {
-    let mut group = builder.create_group("relations");
-    let n = data.relations.len() as u64;
-    group.set_attr("count", AttrValue::I64(n as i64));
-
-    if n == 0 {
-        builder.add_group(group.finish());
-        return;
-    }
-
-    let srcs: Vec<i64> = data.relations.iter().map(|r| r.src).collect();
-    group.create_dataset("src").with_i64_data(&srcs);
-
-    let tgts: Vec<i64> = data.relations.iter().map(|r| r.tgt).collect();
-    group.create_dataset("tgt").with_i64_data(&tgts);
-
-    let rels: Vec<String> = data.relations.iter().map(|r| r.relation.clone()).collect();
-    let (rel_raw, rel_len) = pack_strings(&rels);
-    group
-        .create_dataset("relation")
-        .with_compound_data(string_dtype(rel_len), rel_raw, n);
-
-    let weights: Vec<f64> = data.relations.iter().map(|r| r.weight).collect();
-    group.create_dataset("weight").with_f64_data(&weights);
-
-    let timestamps: Vec<f64> = data.relations.iter().map(|r| r.timestamp).collect();
-    group.create_dataset("timestamp").with_f64_data(&timestamps);
-
-    builder.add_group(group.finish());
-}
-
-#[cfg(test)]
-mod time_tests {
-    use super::civil_from_days;
-
-    #[test]
-    fn epoch_day_zero_is_1970_01_01() {
-        assert_eq!(civil_from_days(0), (1970, 1, 1));
-    }
-
-    #[test]
-    fn known_dates_roundtrip() {
-        // 2026-08-16 is 20,681 days after 1970-01-01.
-        assert_eq!(civil_from_days(20_681), (2026, 8, 16));
-        // 2000-02-29 (leap day itself) and 2000-03-01 (the day after).
-        assert_eq!(civil_from_days(11_016), (2000, 2, 29));
-        assert_eq!(civil_from_days(11_017), (2000, 3, 1));
-    }
-
-    #[test]
-    fn iso8601_now_has_expected_shape() {
-        let ts = super::iso8601_now();
-        assert_eq!(ts.len(), "2026-08-16T00:00:00Z".len());
-        assert!(ts.starts_with("20")); // sanity: 21st-century year
-        assert!(ts.ends_with('Z'));
-    }
-}

crates/clawhdf5-migrate/src/sqlite_reader.rs

@@ -43,19 +43,15 @@ pub struct Relation {
     pub timestamp: f64,
 }
 
-/// All data read from a ZeroClaw SQLite database.
+/// All data read from a source SQLite database.
 #[derive(Debug)]
 pub struct SqliteData {
     pub chunks: Vec<MemoryChunk>,
     pub sessions: Vec<Session>,
     pub entities: Vec<Entity>,
     pub relations: Vec<Relation>,
+    /// `--embedding-dim`, or the first row's; 0 when neither exists.
     pub embedding_dim: usize,
-    /// Filesystem path of the SQLite database this data was read from, for
-    /// provenance attribution on the HDF5 output. Empty when the data did
-    /// not come directly from a SQLite read (e.g. re-read of a prior HDF5
-    /// migration output for an incremental merge).
-    pub source_path: String,
 }
 
 /// A table name plus the ordered column names the reader maps by position.
@@ -67,7 +63,9 @@ pub struct TableSchema {
 
 /// Configurable mapping from a SQLite layout to the migration's data model.
 ///
-/// Defaults to the ZeroClaw schema; the CLI can override the table names so the
+/// Defaults to the `memory_chunks` / `sessions` / `entities` / `relations`
+/// layout (not ZeroClaw's schema, despite what earlier docs said); the CLI can
+/// override the table names so the
 /// tool can migrate databases whose tables are named differently. Column names
 /// (and order) are part of the config too, so a library caller can remap them.
 #[derive(Debug, Clone)]
@@ -167,11 +165,13 @@ pub fn read_counts(
     })
 }
 
-/// Auto-detect embedding dimension from the first chunk's BLOB size.
+/// Auto-detect embedding dimension from the BLOB size of the first chunk (in
+/// id order, deleted or not).
 fn detect_embedding_dim(conn: &Connection, config: &SchemaConfig) -> SqlResult<Option<usize>> {
     let emb_col = config.chunks.columns.get(2).copied().unwrap_or("embedding");
+    let id_col = config.chunks.columns.first().copied().unwrap_or("id");
     let mut stmt = conn.prepare(&format!(
-        "SELECT {emb_col} FROM {} LIMIT 1",
+        "SELECT {emb_col} FROM {} ORDER BY {id_col} LIMIT 1",
         config.chunks.table
     ))?;
     let mut rows = stmt.query([])?;
@@ -192,27 +192,19 @@ fn blob_to_f32(blob: &[u8]) -> Vec<f32> {
         .collect()
 }
 
-/// Read all data from a ZeroClaw SQLite database.
+/// Read all data from a source SQLite database.
 ///
 /// If `skip_deleted` is true, rows with `deleted=1` are excluded from chunks.
-/// If `embedding_dim` is `None`, auto-detect from the first row.
+/// If `embedding_dim` is `None`, auto-detect from the first row (0 when there
+/// are no rows). Embeddings are returned at their full stored length whatever
+/// the dimension: checking that every row matches it is the writer's job
+/// (`store_writer::write_store`), so a mismatch is an error, not silent
+/// truncation.
 pub fn read_sqlite(
     path: &str,
     skip_deleted: bool,
     embedding_dim: Option<usize>,
     config: &SchemaConfig,
-) -> Result<SqliteData, Box<dyn std::error::Error>> {
-    read_sqlite_filtered(path, skip_deleted, embedding_dim, config, 0)
-}
-
-/// Like [`read_sqlite`] but only reads chunks whose id is greater than
-/// `min_chunk_id` (0 = all). Used for incremental migration.
-pub fn read_sqlite_filtered(
-    path: &str,
-    skip_deleted: bool,
-    embedding_dim: Option<usize>,
-    config: &SchemaConfig,
-    min_chunk_id: i64,
 ) -> Result<SqliteData, Box<dyn std::error::Error>> {
     let conn = Connection::open(path)?;
 
@@ -221,7 +213,7 @@ pub fn read_sqlite_filtered(
         None => detect_embedding_dim(&conn, config)?.unwrap_or(0),
     };
 
-    let chunks = read_chunks(&conn, skip_deleted, dim, config, min_chunk_id)?;
+    let chunks = read_chunks(&conn, skip_deleted, config)?;
     let sessions = read_sessions(&conn, config)?;
     let entities = read_entities(&conn, config)?;
     let relations = read_relations(&conn, config)?;
@@ -232,42 +224,43 @@ pub fn read_sqlite_filtered(
         entities,
         relations,
         embedding_dim: dim,
-        source_path: path.to_owned(),
     })
 }
 
 fn read_chunks(
     conn: &Connection,
     skip_deleted: bool,
-    expected_dim: usize,
     config: &SchemaConfig,
-    min_chunk_id: i64,
 ) -> SqlResult<Vec<MemoryChunk>> {
     let id_col = config.chunks.columns.first().copied().unwrap_or("id");
     let deleted_col = config.chunks.columns.get(7).copied().unwrap_or("deleted");
-    let mut conds = Vec::new();
+    let mut where_clause = String::new();
     if skip_deleted {
-        conds.push(format!("{deleted_col} = 0"));
+        where_clause = format!(" WHERE {deleted_col} = 0");
     }
-    if min_chunk_id > 0 {
-        conds.push(format!("{id_col} > {min_chunk_id}"));
-    }
-    let where_clause = if conds.is_empty() {
-        String::new()
-    } else {
-        format!(" WHERE {}", conds.join(" AND "))
-    };
+    // In id order, so the store's records follow the source's order.
+    where_clause.push_str(&format!(" ORDER BY {id_col}"));
     let sql = config.chunks.select(&where_clause);
 
     let mut stmt = conn.prepare(&sql)?;
     let rows = stmt.query_map([], |row| {
         let blob: Vec<u8> = row.get(2)?;
-        let mut embedding = blob_to_f32(&blob);
-
-        // Validate/truncate to expected dimension
-        if expected_dim > 0 {
-            embedding.truncate(expected_dim);
+        if !blob.len().is_multiple_of(4) {
+            let id: i64 = row.get(0)?;
+            return Err(rusqlite::Error::FromSqlConversionFailure(
+                2,
+                rusqlite::types::Type::Blob,
+                format!(
+                    "chunk id {id}: embedding BLOB is {} bytes, not a whole number of \
+                     little-endian f32 values",
+                    blob.len()
+                )
+                .into(),
+            ));
         }
+        // Read at full length: rows of the wrong dimension are rejected by
+        // the writer, never truncated to fit.
+        let embedding = blob_to_f32(&blob);
 
         Ok(MemoryChunk {
             id: row.get(0)?,

crates/clawhdf5-migrate/src/store_writer.rs

@@ -0,0 +1,407 @@
+//! Write migrated SQLite data into a clawhdf5-agent store.
+//!
+//! Everything goes through `clawhdf5-agent`'s own API — `HDF5Memory::create`
+//! (or `open` for `--incremental`), `save_batch`, `delete_batch`, the session
+//! cache and the knowledge graph — so the result is an ordinary agent store
+//! that `HDF5Memory::open` accepts, not a second hand-built copy of its schema.
+
+use std::collections::{HashMap, HashSet};
+use std::path::Path;
+
+use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
+use clawhdf5_format::float16::round_to_f16;
+
+use crate::sqlite_reader::{MemoryChunk, SqliteData};
+
+type BoxErr = Box<dyn std::error::Error>;
+
+/// SQLite timestamps are Unix seconds; the agent's session and relation
+/// timestamps are Unix microseconds (memory records stay in seconds).
+pub const US_PER_SEC: f64 = 1_000_000.0;
+
+/// Options controlling the output store.
+#[derive(Debug, Clone)]
+pub struct WriteOptions {
+    pub agent_id: String,
+    pub embedder: String,
+    pub compression: bool,
+    pub compression_level: u32,
+    /// Store full-precision `f32` embeddings instead of the library default
+    /// (half precision). Only applies to a newly created store: an existing
+    /// store keeps the precision it was created with.
+    pub f32: bool,
+    /// Add to the store at the output path if there is one, instead of
+    /// replacing it.
+    pub incremental: bool,
+    /// Leave out deleted source rows that are not in the store. (A deleted
+    /// row that matches an active store record still tombstones it, so pass
+    /// deleted rows in `data` for an incremental run.)
+    pub skip_deleted: bool,
+}
+
+/// What the migration wrote, and where each source row went, so validation
+/// can compare the store with the source row by row.
+#[derive(Debug, Default)]
+pub struct Migration {
+    /// Whether the output store existed and was added to (`--incremental`).
+    pub appended_to_existing: bool,
+    /// The store's embedding precision.
+    pub float16: bool,
+    pub embedding_dim: usize,
+    /// Records in the store after the migration (including tombstones).
+    pub store_count: usize,
+    /// `(store index, source chunk index)` of every record written.
+    pub records: Vec<(usize, usize)>,
+    /// Source chunks already in the store (incremental), not written again.
+    pub chunks_present: usize,
+    /// `(store index, source chunk index)` of records that were active in
+    /// the store but whose source row is now deleted (incremental): they were
+    /// tombstoned by this run.
+    pub deleted_in_store: Vec<(usize, usize)>,
+    /// Source rows that were deleted in the store but are active in the
+    /// source (incremental): the agent has no un-delete, so each was written
+    /// again as a new record (counted in `records` too).
+    pub restored: usize,
+    /// Deleted source rows left out because of `skip_deleted`.
+    pub deleted_skipped: usize,
+    /// `(store session index, source session index)` of each session written.
+    pub sessions: Vec<(usize, usize)>,
+    pub sessions_present: usize,
+    /// `(store entity id, source entity index)` of each entity written.
+    pub entities: Vec<(u64, usize)>,
+    pub entities_present: usize,
+    /// SQLite entity id -> store entity id, for every source entity.
+    pub entity_ids: HashMap<i64, u64>,
+    /// `(store relation index, source relation index)` of each relation written.
+    pub relations: Vec<(usize, usize)>,
+    pub relations_present: usize,
+    /// Source relations naming an entity id that is not in the entities
+    /// table; the knowledge graph cannot hold them, so they are skipped.
+    pub dangling_relations: Vec<usize>,
+    /// Messages of the write-anomaly alerts the agent raised while importing
+    /// (informational; they never block a save — a bulk import typically
+    /// trips the write-rate check).
+    pub anomaly_alerts: Vec<String>,
+}
+
+/// Identity of a memory record for incremental de-duplication: every field
+/// the agent stores except the embedding (whose stored form depends on the
+/// store's precision).
+type RecordKey = (String, String, String, String, u64);
+
+fn record_key(
+    chunk: &str,
+    source_channel: &str,
+    session_id: &str,
+    tags: &str,
+    ts: f64,
+) -> RecordKey {
+    (
+        chunk.to_owned(),
+        source_channel.to_owned(),
+        session_id.to_owned(),
+        tags.to_owned(),
+        ts.to_bits(),
+    )
+}
+
+/// Reject rows the agent would otherwise store differently from the source,
+/// or not at all: an embedding of a different length from the store's
+/// dimension (the agent pads/truncates silently), an empty embedding, or, in
+/// a float16 store, a value beyond the half-precision range.
+///
+/// Every source row is checked, including ones that end up not being written
+/// (already in the store, or deleted and skipped): the source must be
+/// consistent as a whole, and the check runs before the store is touched.
+fn check_chunks(chunks: &[MemoryChunk], dim: usize, float16: bool) -> Result<(), BoxErr> {
+    for c in chunks {
+        if c.embedding.is_empty() {
+            return Err(format!(
+                "chunk id {}: the embedding is empty; an agent store needs an embedding \
+                 for every record",
+                c.id
+            )
+            .into());
+        }
+        if c.embedding.len() != dim {
+            return Err(format!(
+                "chunk id {}: embedding has {} values, expected {dim}; every row must have \
+                 the store's dimension (detected from the first row unless --embedding-dim \
+                 is given), and rows are never truncated or padded to fit",
+                c.id,
+                c.embedding.len()
+            )
+            .into());
+        }
+        if float16
+            && let Some((k, v)) = c
+                .embedding
+                .iter()
+                .enumerate()
+                .find(|&(_, &v)| v.is_finite() && round_to_f16(v).is_infinite())
+        {
+            return Err(format!(
+                "chunk id {}: embedding[{k}] = {v} is outside the half-precision range \
+                 (±65504) of a float16 store; migrate with --f32",
+                c.id
+            )
+            .into());
+        }
+    }
+    Ok(())
+}
+
+/// Migrate `data` into the agent store at `path`.
+///
+/// Without `opts.incremental` (or when nothing exists at `path`) a new store
+/// is created, replacing any file there — but only once every source row has
+/// passed [`check_chunks`], so a source that cannot be migrated leaves an
+/// existing store untouched. With it, the existing store is opened and only
+/// source rows it does not already hold are added: memory records are
+/// matched on their content, sessions on their id, entities on name and
+/// type, relations on (source, target, relation). A matched record then
+/// takes the source row's deleted flag: see [`Migration::deleted_in_store`]
+/// and [`Migration::restored`].
+pub fn write_store(
+    path: &Path,
+    data: &SqliteData,
+    opts: &WriteOptions,
+) -> Result<Migration, BoxErr> {
+    let existing = opts.incremental && path.exists();
+    let mut mem = if existing {
+        // `open` does not modify the store beyond what the agent itself does
+        // on open; the checks below run before anything is written.
+        let mem = HDF5Memory::open(path)?;
+        let dim = mem.config().embedding_dim;
+        // `data.embedding_dim` is 0 only for a source with no records and no
+        // --embedding-dim, which has no dimension to disagree with.
+        if data.embedding_dim != 0 && dim != data.embedding_dim {
+            let hint = if dim == 0 {
+                " (a store created from a source with no memory records; re-create it \
+                 with --embedding-dim)"
+            } else {
+                ""
+            };
+            return Err(format!(
+                "the store at {} has embedding_dim {dim}{hint}, the source {}; \
+                 embeddings of a different dimension cannot be added to it",
+                path.display(),
+                data.embedding_dim
+            )
+            .into());
+        }
+        check_chunks(&data.chunks, dim, mem.config().float16)?;
+        mem
+    } else {
+        // (With records, a dimension of 0 means an empty first embedding,
+        // which `check_chunks` reports more precisely.)
+        if data.embedding_dim == 0 && data.chunks.is_empty() {
+            return Err(
+                "the source has no memory records to detect the embedding dimension \
+                        from; pass --embedding-dim (the dimension of the agent's embedder), \
+                        or the store could never hold a record"
+                    .into(),
+            );
+        }
+        let mut config = MemoryConfig::new(path.to_path_buf(), &opts.agent_id, data.embedding_dim);
+        config.embedder = opts.embedder.clone();
+        config.compression = opts.compression;
+        config.compression_level = opts.compression_level;
+        // Only ever switch the library default off (as `clawhdf5-cli create`).
+        if opts.f32 {
+            config.float16 = false;
+        }
+        // Before `create`, which replaces whatever is at `path`.
+        check_chunks(&data.chunks, config.embedding_dim, config.float16)?;
+        HDF5Memory::create(config)?
+    };
+    let float16 = mem.config().float16;
+    let dim = mem.config().embedding_dim;
+
+    let mut m = Migration {
+        appended_to_existing: existing,
+        float16,
+        embedding_dim: dim,
+        ..Migration::default()
+    };
+
+    // ---- Memory records --------------------------------------------------
+    // Store indices of every record the store already holds, by content, so
+    // a source row that appears twice is only treated as present as often
+    // as the store has it.
+    let mut present: HashMap<RecordKey, Vec<usize>> = HashMap::new();
+    if existing {
+        let c = &mem.cache;
+        for i in 0..c.len() {
+            let key = record_key(
+                &c.chunks[i],
+                &c.source_channels[i],
+                &c.session_ids[i],
+                &c.tags[i],
+                c.timestamps[i],
+            );
+            present.entry(key).or_default().push(i);
+        }
+    }
+    let key_of = |c: &MemoryChunk| {
+        record_key(
+            &c.chunk,
+            &c.source_channel,
+            &c.session_id,
+            &c.tags,
+            c.timestamp,
+        )
+    };
+    let tombstoned = |idx: usize| mem.cache.tombstones[idx] != 0;
+    // Pass 1: a store record in the same deleted state as the source row.
+    let mut unmatched: Vec<usize> = Vec::new();
+    for (i, c) in data.chunks.iter().enumerate() {
+        let src_deleted = c.deleted != 0;
+        let hit = present.get_mut(&key_of(c)).and_then(|idxs| {
+            let at = idxs.iter().position(|&x| tombstoned(x) == src_deleted)?;
+            Some(idxs.remove(at))
+        });
+        match hit {
+            Some(_) => m.chunks_present += 1,
+            None => unmatched.push(i),
+        }
+    }
+    // Pass 2: a store record whose deleted state differs — the source row
+    // was deleted or restored since the last migration. The source wins.
+    let mut new_chunks: Vec<usize> = Vec::with_capacity(unmatched.len());
+    let mut delete_in_store: Vec<usize> = Vec::new();
+    for i in unmatched {
+        let c = &data.chunks[i];
+        let hit = present
+            .get_mut(&key_of(c))
+            .and_then(|idxs| (!idxs.is_empty()).then(|| idxs.remove(0)));
+        match hit {
+            // Active in the store, deleted in the source: tombstone it.
+            Some(idx) if c.deleted != 0 => {
+                m.deleted_in_store.push((idx, i));
+                delete_in_store.push(idx);
+            }
+            // Deleted in the store, active in the source. The agent has no
+            // un-delete, so the row is written again as a new active record
+            // (the tombstone stays until the store is compacted).
+            Some(_) => {
+                m.restored += 1;
+                new_chunks.push(i);
+            }
+            None if c.deleted != 0 && opts.skip_deleted => m.deleted_skipped += 1,
+            None => new_chunks.push(i),
+        }
+    }
+    new_chunks.sort_unstable();
+    let to_write: Vec<&MemoryChunk> = new_chunks.iter().map(|&i| &data.chunks[i]).collect();
+
+    // ---- Sessions (in the cache; persisted by the save_batch checkpoint) ---
+    let known_sessions: HashSet<String> = mem
+        .sessions()
+        .entries
+        .iter()
+        .map(|e| e.id.clone())
+        .collect();
+    for (i, s) in data.sessions.iter().enumerate() {
+        if known_sessions.contains(&s.id) {
+            m.sessions_present += 1;
+            continue;
+        }
+        let sessions = mem.sessions_mut();
+        let at = sessions.len();
+        sessions.add_at(
+            &s.id,
+            s.start_idx.max(0) as usize,
+            s.end_idx.max(0) as usize,
+            &s.channel,
+            &s.summary,
+            s.timestamp * US_PER_SEC,
+        );
+        m.sessions.push((at, i));
+    }
+
+    // ---- Knowledge graph -------------------------------------------------
+    let kg = mem.knowledge_mut();
+    // Matched only against what the store held before this run: the source
+    // itself is copied as it is, duplicates included.
+    let by_name_type: HashMap<(String, String), u64> = kg
+        .entities
+        .iter()
+        .map(|e| ((e.name.clone(), e.entity_type.clone()), e.id))
+        .collect();
+    for (i, e) in data.entities.iter().enumerate() {
+        let key = (e.name.clone(), e.entity_type.clone());
+        let id = match by_name_type.get(&key) {
+            Some(&id) => {
+                m.entities_present += 1;
+                id
+            }
+            None => {
+                let id = kg.add_entity(&e.name, &e.entity_type, e.embedding_idx);
+                m.entities.push((id, i));
+                id
+            }
+        };
+        m.entity_ids.insert(e.id, id);
+    }
+    let known_relations: HashSet<(u64, u64, String)> = kg
+        .relations
+        .iter()
+        .map(|r| (r.src, r.tgt, r.relation.clone()))
+        .collect();
+    for (i, r) in data.relations.iter().enumerate() {
+        let (Some(&src), Some(&tgt)) = (m.entity_ids.get(&r.src), m.entity_ids.get(&r.tgt)) else {
+            m.dangling_relations.push(i);
+            continue;
+        };
+        if known_relations.contains(&(src, tgt, r.relation.clone())) {
+            m.relations_present += 1;
+            continue;
+        }
+        let at = kg.relations.len();
+        kg.add_relation(src, tgt, &r.relation, r.weight as f32);
+        kg.relations[at].ts = r.timestamp * US_PER_SEC;
+        m.relations.push((at, i));
+    }
+
+    // ---- Write: one checkpoint for records, sessions and graph -----------
+    let entries: Vec<MemoryEntry> = to_write
+        .iter()
+        .map(|c| MemoryEntry {
+            chunk: c.chunk.clone(),
+            embedding: c.embedding.clone(),
+            source_channel: c.source_channel.clone(),
+            timestamp: c.timestamp,
+            session_id: c.session_id.clone(),
+            tags: c.tags.clone(),
+        })
+        .collect();
+    let indices = mem.save_batch(entries)?;
+    m.records = indices
+        .iter()
+        .copied()
+        .zip(new_chunks.iter().copied())
+        .collect();
+
+    // Rows deleted in the source stay deleted: tombstones, as the agent's own
+    // `delete` leaves them (not compacted away).
+    // Records matched in the store whose source row has since been deleted
+    // are tombstoned too.
+    let tombstones: Vec<usize> = m
+        .records
+        .iter()
+        .filter(|&&(_, src)| data.chunks[src].deleted != 0)
+        .map(|&(idx, _)| idx)
+        .chain(delete_in_store)
+        .collect();
+    mem.delete_batch(&tombstones)?;
+
+    m.anomaly_alerts = mem
+        .take_anomaly_alerts()
+        .into_iter()
+        .map(|a| a.message)
+        .collect();
+    m.store_count = mem.count();
+    drop(mem); // release the single-writer lock before anyone re-opens it
+    Ok(m)
+}

crates/clawhdf5-migrate/src/validate.rs

@@ -1,192 +1,266 @@
-use clawhdf5::reader::File as Hdf5File;
-use clawhdf5_format::provenance::VerifyResult;
+//! Validate a migration by reading the store back the way an agent would:
+//! through `HDF5Memory::open_read_only`, comparing what it loads with the
+//! SQLite source, and running a search for a migrated record.
+
+use std::path::Path;
+
+use clawhdf5_agent::{AgentMemory, HDF5Memory, SearchOptions};
+use clawhdf5_format::float16::round_to_f16;
 
-use crate::hdf5_reader::read_hdf5;
 use crate::sqlite_reader::SqliteData;
+use crate::store_writer::{Migration, US_PER_SEC};
 
 type BoxErr = Box<dyn std::error::Error>;
 
 /// Summary of a migration validation.
 #[derive(Debug)]
 pub struct ValidationSummary {
-    pub chunks: u64,
-    pub sessions: u64,
-    pub entities: u64,
-    pub relations: u64,
-    pub embedding_dim: u64,
-    /// Number of rows whose full content was compared against the source.
+    /// Records in the store (including tombstones).
+    pub count: usize,
+    /// Records in the store that are not deleted.
+    pub active: usize,
+    pub sessions: usize,
+    pub entities: usize,
+    pub relations: usize,
+    pub embedding_dim: usize,
+    pub float16: bool,
+    /// Rows whose full content was compared against the source.
     pub rows_checked: u64,
-    /// Whether the `chunks/text` and `chunks/embeddings` SHINES provenance
-    /// hashes (written via [`crate::hdf5_writer`]) were both present and
-    /// matched their recomputed SHA-256 on read-back. `false` when either
-    /// dataset has no provenance metadata (e.g. an older output file) or
-    /// there are zero chunks to check.
-    pub provenance_verified: bool,
+    /// Whether a search for a migrated record found it (`false` when there
+    /// was no active migrated record with an embedding to search for).
+    pub search_checked: bool,
 }
 
-/// Validate a migrated HDF5 file against the source data.
+/// Validate the store at `path` against the source rows `migration` wrote.
 ///
-/// Reads the written file back and compares actual content — chunk text,
-/// embeddings, and every session/entity/relation field — to the source, not
-/// just the row counts. When `full` is false a representative sample of chunk
-/// rows is content-checked (counts and all other groups are always checked in
-/// full); when `full` is true every chunk row is compared too. `float16` widens
-/// the embedding tolerance to allow for half-precision quantization.
-pub fn validate_hdf5(
-    path: &str,
+/// Counts and the session / entity / relation rows are always checked in
+/// full. Memory records are content-checked on a representative sample, or
+/// all of them with `full`. Embeddings must match exactly: the source values
+/// themselves in an `f32` store, their [`round_to_f16`] in a `float16` one.
+pub fn validate_store(
+    path: &Path,
     source: &SqliteData,
+    migration: &Migration,
     full: bool,
-    float16: bool,
 ) -> Result<ValidationSummary, BoxErr> {
-    let got = read_hdf5(path)?;
-    let provenance_verified = verify_chunk_provenance(path)?;
+    let mut mem = HDF5Memory::open_read_only(path)?;
+    let float16 = mem.config().float16;
+    let dim = mem.config().embedding_dim;
 
     // ---- Counts ----
-    check_count("chunk", got.chunks.len(), source.chunks.len())?;
-    check_count("session", got.sessions.len(), source.sessions.len())?;
-    check_count("entity", got.entities.len(), source.entities.len())?;
-    check_count("relation", got.relations.len(), source.relations.len())?;
-    if got.embedding_dim != source.embedding_dim {
+    check_count("record", mem.count(), migration.store_count)?;
+    if float16 != migration.float16 {
         return Err(format!(
-            "embedding_dim mismatch: HDF5 has {}, source has {}",
-            got.embedding_dim, source.embedding_dim
+            "float16 mismatch: store {float16}, expected {}",
+            migration.float16
         )
         .into());
     }
+    if dim != migration.embedding_dim {
+        return Err(format!(
+            "embedding_dim mismatch: store has {dim}, expected {}",
+            migration.embedding_dim
+        )
+        .into());
+    }
+    if !migration.appended_to_existing {
+        check_count("record", mem.count(), migration.records.len())?;
+        check_count("session", mem.sessions().len(), migration.sessions.len())?;
+        check_count(
+            "entity",
+            mem.knowledge().entities.len(),
+            migration.entities.len(),
+        )?;
+        check_count(
+            "relation",
+            mem.knowledge().relations.len(),
+            migration.relations.len(),
+        )?;
+    }
 
-    // ---- Chunk content (sampled or full) ----
-    let (emb_abs, emb_rel) = if float16 { (1e-2, 1e-2) } else { (1e-4, 0.0) };
+    // ---- Memory records (sampled or full) ----
     let mut rows_checked = 0u64;
-    for i in sample_indices(source.chunks.len(), full) {
-        let (s, g) = (&source.chunks[i], &got.chunks[i]);
-        if s.id != g.id {
-            return Err(field_err("chunk", i, "id", s.id, g.id));
-        }
-        if s.chunk != g.chunk {
-            return Err(format!(
-                "chunk[{i}].text mismatch: source {:?}, HDF5 {:?}",
-                truncate(&s.chunk),
-                truncate(&g.chunk)
-            )
-            .into());
-        }
-        if s.session_id != g.session_id || s.source_channel != g.source_channel || s.tags != g.tags
-        {
-            return Err(format!("chunk[{i}] string field mismatch").into());
-        }
-        if s.deleted != g.deleted {
-            return Err(field_err("chunk", i, "deleted", s.deleted, g.deleted));
-        }
-        if s.embedding.len() != g.embedding.len() {
-            return Err(format!(
-                "chunk[{i}] embedding length mismatch: {} vs {}",
-                s.embedding.len(),
-                g.embedding.len()
-            )
-            .into());
-        }
-        for (k, (&a, &b)) in s.embedding.iter().zip(g.embedding.iter()).enumerate() {
-            if (a - b).abs() > emb_abs + emb_rel * a.abs() {
+    let expected_value = |v: f32| if float16 { round_to_f16(v) } else { v };
+    for k in sample_indices(migration.records.len(), full) {
+        let (idx, src) = migration.records[k];
+        let s = &source.chunks[src];
+        let c = &mem.cache;
+        if idx >= c.len() {
             return Err(
-                    format!("chunk[{i}].embedding[{k}] mismatch: source {a}, HDF5 {b}").into(),
+                format!("record {idx} (chunk id {}) is missing from the store", s.id).into(),
             );
         }
+        let id = s.id;
+        if c.chunks[idx] != s.chunk {
+            return Err(format!(
+                "record {idx} (chunk id {id}) text mismatch: source {:?}, store {:?}",
+                truncate(&s.chunk),
+                truncate(&c.chunks[idx])
+            )
+            .into());
+        }
+        if c.source_channels[idx] != s.source_channel
+            || c.session_ids[idx] != s.session_id
+            || c.tags[idx] != s.tags
+        {
+            return Err(format!("record {idx} (chunk id {id}) string field mismatch").into());
+        }
+        if c.timestamps[idx].to_bits() != s.timestamp.to_bits() {
+            return Err(format!(
+                "record {idx} (chunk id {id}) timestamp mismatch: source {}, store {}",
+                s.timestamp, c.timestamps[idx]
+            )
+            .into());
+        }
+        let deleted = c.tombstones[idx] != 0;
+        if deleted != (s.deleted != 0) {
+            return Err(format!(
+                "record {idx} (chunk id {id}) deleted mismatch: source {}, store {deleted}",
+                s.deleted != 0
+            )
+            .into());
+        }
+        let got = c.embeddings.get(idx).unwrap_or(&[]);
+        if got.len() != s.embedding.len() {
+            return Err(format!(
+                "record {idx} (chunk id {id}) embedding length mismatch: source {}, store {}",
+                s.embedding.len(),
+                got.len()
+            )
+            .into());
+        }
+        for (j, (&a, &b)) in s.embedding.iter().zip(got).enumerate() {
+            let want = expected_value(a);
+            if want.to_bits() != b.to_bits() && !(want.is_nan() && b.is_nan()) {
+                return Err(format!(
+                    "record {idx} (chunk id {id}) embedding[{j}] mismatch: source {a}, \
+                     expected {want}, store {b}"
+                )
+                .into());
+            }
         }
         rows_checked += 1;
     }
 
-    // ---- Other groups (always full — they are small) ----
-    for (i, (s, g)) in source.sessions.iter().zip(got.sessions.iter()).enumerate() {
-        if s.id != g.id
-            || s.start_idx != g.start_idx
-            || s.end_idx != g.end_idx
-            || s.channel != g.channel
-            || s.summary != g.summary
-        {
-            return Err(format!("session[{i}] mismatch").into());
+    // ---- Records tombstoned because their source row was deleted ----
+    for &(idx, src) in &migration.deleted_in_store {
+        let s = &source.chunks[src];
+        let c = &mem.cache;
+        if idx >= c.len() || c.chunks[idx] != s.chunk || c.timestamps[idx] != s.timestamp {
+            return Err(format!("record {idx} (chunk id {}) mismatch or missing", s.id).into());
+        }
+        if c.tombstones[idx] == 0 {
+            return Err(format!(
+                "record {idx} (chunk id {}) is deleted in the source but active in the store",
+                s.id
+            )
+            .into());
         }
         rows_checked += 1;
     }
-    for (i, (s, g)) in source.entities.iter().zip(got.entities.iter()).enumerate() {
-        if s.id != g.id
-            || s.name != g.name
-            || s.entity_type != g.entity_type
-            || s.embedding_idx != g.embedding_idx
+
+    // ---- Sessions ----
+    let sessions = mem.sessions();
+    for &(at, src) in &migration.sessions {
+        let s = &source.sessions[src];
+        let (Some(e), Some(summary)) = (sessions.entries.get(at), sessions.summaries.get(at))
+        else {
+            return Err(format!("session {:?} is missing from the store", s.id).into());
+        };
+        if e.id != s.id
+            || e.start_idx != s.start_idx.max(0) as u64
+            || e.end_idx != s.end_idx.max(0) as u64
+            || e.channel != s.channel
+            || *summary != s.summary
+            || e.ts != s.timestamp * US_PER_SEC
         {
-            return Err(format!("entity[{i}] mismatch").into());
+            return Err(format!("session {:?} mismatch", s.id).into());
         }
         rows_checked += 1;
     }
-    for (i, (s, g)) in source
-        .relations
+
+    // ---- Knowledge graph ----
+    let kg = mem.knowledge();
+    for &(id, src) in &migration.entities {
+        let s = &source.entities[src];
+        let Some(e) = kg.get_entity(id) else {
+            return Err(format!(
+                "entity {:?} (id {}) is missing from the store",
+                s.name, s.id
+            )
+            .into());
+        };
+        if e.name != s.name || e.entity_type != s.entity_type || e.embedding_idx != s.embedding_idx
+        {
+            return Err(format!("entity {:?} (id {}) mismatch", s.name, s.id).into());
+        }
+        rows_checked += 1;
+    }
+    for &(at, src) in &migration.relations {
+        let s = &source.relations[src];
+        let r = kg.relations.get(at);
+        let ok = r.is_some_and(|r| {
+            Some(&r.src) == migration.entity_ids.get(&s.src)
+                && Some(&r.tgt) == migration.entity_ids.get(&s.tgt)
+                && r.relation == s.relation
+                && r.weight == s.weight as f32
+                && r.ts == s.timestamp * US_PER_SEC
+        });
+        if !ok {
+            return Err(format!(
+                "relation {} -[{}]-> {} mismatch or missing",
+                s.src, s.relation, s.tgt
+            )
+            .into());
+        }
+        rows_checked += 1;
+    }
+
+    // ---- A migrated record must be findable by search ----
+    let probe = migration
+        .records
         .iter()
-        .zip(got.relations.iter())
-        .enumerate()
-    {
-        if s.src != g.src || s.tgt != g.tgt || s.relation != g.relation {
-            return Err(format!("relation[{i}] mismatch").into());
+        .copied()
+        .find(|&(idx, _)| dim > 0 && mem.cache.tombstones[idx] == 0);
+    let search_checked = match probe {
+        None => false,
+        Some((idx, _)) => {
+            let query = mem.cache.embeddings[idx].to_vec();
+            let text = mem.cache.chunks[idx].clone();
+            let hits = mem.search(&query, &text, &SearchOptions::new(10));
+            // A record with the same text is as good a hit: the source may
+            // hold duplicates, and they tie.
+            if !hits.iter().any(|h| h.index == idx || h.chunk == text) {
+                return Err(format!(
+                    "search for migrated record {idx} ({:?}) did not return it",
+                    truncate(&text)
+                )
+                .into());
             }
-        rows_checked += 1;
+            true
         }
+    };
 
     Ok(ValidationSummary {
-        chunks: got.chunks.len() as u64,
-        sessions: got.sessions.len() as u64,
-        entities: got.entities.len() as u64,
-        relations: got.relations.len() as u64,
-        embedding_dim: got.embedding_dim as u64,
+        count: mem.count(),
+        active: mem.count_active(),
+        sessions: mem.sessions().len(),
+        entities: mem.knowledge().entities.len(),
+        relations: mem.knowledge().relations.len(),
+        embedding_dim: dim,
+        float16,
         rows_checked,
-        provenance_verified,
+        search_checked,
     })
 }
 
 fn check_count(kind: &str, got: usize, expected: usize) -> Result<(), BoxErr> {
     if got != expected {
-        return Err(format!("{kind} count mismatch: HDF5 has {got}, source has {expected}").into());
+        return Err(format!("{kind} count mismatch: store has {got}, expected {expected}").into());
     }
     Ok(())
 }
 
-/// Re-verify the SHA-256 provenance hash of `chunks/text` and
-/// `chunks/embeddings` against their actual stored bytes, catching
-/// post-write corruption that a plain content comparison against the
-/// in-memory source wouldn't (the source is compared against what
-/// `read_hdf5` decoded, not against the raw bytes on disk).
-///
-/// Returns `Ok(true)` only if both datasets exist and both hashes match.
-/// Returns `Ok(false)` (not an error) if a dataset has no provenance
-/// attributes at all (e.g. a file written before this check existed) or
-/// there are zero chunks. Returns an error only on an actual hash mismatch —
-/// that indicates real corruption.
-fn verify_chunk_provenance(path: &str) -> Result<bool, BoxErr> {
-    let file = Hdf5File::open(path)?;
-    let Ok(chunks) = file.group("chunks") else {
-        return Ok(false);
-    };
-    let mut all_present = true;
-    for name in ["text", "embeddings"] {
-        let Ok(ds) = chunks.dataset(name) else {
-            all_present = false;
-            continue;
-        };
-        match ds.verify_provenance()? {
-            VerifyResult::Ok => {}
-            VerifyResult::NoHash => all_present = false,
-            VerifyResult::Mismatch { stored, computed } => {
-                return Err(format!(
-                    "provenance hash mismatch on chunks/{name}: stored {stored}, recomputed {computed} — data may be corrupted"
-                )
-                .into());
-            }
-        }
-    }
-    Ok(all_present)
-}
-
-fn field_err<T: std::fmt::Display>(kind: &str, i: usize, field: &str, s: T, g: T) -> BoxErr {
-    format!("{kind}[{i}].{field} mismatch: source {s}, HDF5 {g}").into()
-}
-
 fn truncate(s: &str) -> String {
     if s.len() <= 40 {
         s.to_string()
@@ -196,7 +270,7 @@ fn truncate(s: &str) -> String {
     }
 }
 
-/// Indices of chunk rows to content-check. Full = all; otherwise a spread of
+/// Indices of records to content-check. Full = all; otherwise a spread of
 /// representative rows (first/last and evenly-spaced interior samples).
 fn sample_indices(n: usize, full: bool) -> Vec<usize> {
     if n == 0 {

crates/clawhdf5-napi/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-napi"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "Node.js native addon (napi-rs) exposing clawhdf5-agent to TypeScript/JavaScript"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -10,7 +11,7 @@ repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
 crate-type = ["cdylib"]
 
 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.6.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
 napi = { version = "2", default-features = false, features = ["napi9"] }
 napi-derive = "2"
 

crates/clawhdf5-netcdf4/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-netcdf4"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "NetCDF-4 read support built on rustyhdf5 — pure Rust, no C dependencies"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -10,8 +11,8 @@ keywords = ["netcdf", "netcdf4", "hdf5", "science", "climate"]
 categories = ["parser-implementations", "science"]
 
 [dependencies]
-clawhdf5 = { path = "../clawhdf5", version = "2.6.0" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 
 [dev-dependencies]
 tempfile = { workspace = true }

crates/clawhdf5-py/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5-py"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -14,8 +15,8 @@ name = "clawhdf5"
 crate-type = ["cdylib", "rlib"]
 
 [dependencies]
-clawhdf5_rs = { path = "../clawhdf5", version = "2.6.0", package = "clawhdf5" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
+clawhdf5_rs = { path = "../clawhdf5", version = "2.7.0", package = "clawhdf5" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 pyo3 = "0.29"
 numpy = "0.29"
 

crates/clawhdf5-py/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "maturin"
 
 [project]
 name = "rustyhdf5"
-version = "2.6.0"
+version = "2.7.0"
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 requires-python = ">=3.8"
 license = { text = "MIT" }

crates/clawhdf5/Cargo.toml

@@ -1,7 +1,8 @@
 [package]
 name = "clawhdf5"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
+rust-version.workspace = true
 description = "Pure-Rust HDF5 reader/writer — no C dependencies"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -10,16 +11,16 @@ keywords = ["hdf5", "science", "data", "binary"]
 categories = ["parser-implementations", "science", "encoding"]
 
 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.6.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0" }
 rayon = { version = "1", optional = true }
 
 [dev-dependencies]
 tempfile = { workspace = true }
 criterion = { workspace = true }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.6.0", features = ["mmap"] }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0", features = ["parallel", "fast-checksum"] }
-clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.6.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0", features = ["mmap"] }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0", features = ["parallel", "fast-checksum"] }
+clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.7.0" }
 
 [[bench]]
 name = "mmap_bench"
@@ -30,9 +31,10 @@ name = "parallel_bench"
 harness = false
 
 [features]
-default = ["mmap", "fast-deflate", "provenance"]
+default = ["mmap", "provenance"]
 mmap = ["clawhdf5-io/mmap"]
 parallel = ["clawhdf5-format/parallel", "rayon"]
+# zlib-ng (C, needs cmake) instead of the default pure-Rust zlib-rs.
 fast-deflate = ["clawhdf5-format/fast-deflate"]
 apple-compression = []
 zstd = ["clawhdf5-format/zstd"]

crates/clawhdf5/tests/h5py_interop_tests.rs

@@ -1047,3 +1047,506 @@ with h5py.File("{path_str}", "r") as f:
         data[start..start + cols as usize]
     );
 }
+
+#[test]
+fn h5py_deep_btree_v2_chunk_index_clawhdf5_reads() {
+    // Two unlimited dimensions give a B-tree v2 chunk index, and 2x2 chunks
+    // over 400x400 give 40 000 index records — enough for HDF5 to build a
+    // tree of depth 2. Small h5py files only ever produce depth-0 trees, so
+    // this is the one fixture that walks internal nodes: the path where the
+    // traversal's record budget (the guard against crafted shared-subtree
+    // trees) is spent, which must never refuse a real file.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("deep_btree.h5");
+    let path_str = path.display().to_string();
+
+    let script = format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=(400, 400), maxshape=(None, None),
+                         chunks=(2, 2), dtype="i4")
+    d[...] = np.arange(160000, dtype="i4").reshape(400, 400)
+"#
+    );
+    run_python(&script);
+
+    // The fixture is only meaningful if HDF5 really built internal nodes.
+    let bytes = std::fs::read(&path).unwrap();
+    let at = bytes
+        .windows(4)
+        .position(|w| w == b"BTHD")
+        .expect("expected a B-tree v2 chunk index");
+    let depth = u16::from_le_bytes([bytes[at + 12], bytes[at + 13]]);
+    assert!(
+        depth >= 1,
+        "fixture tree has depth {depth}; it tests nothing"
+    );
+
+    let file = File::open(&path).unwrap();
+    let values = file.dataset("x").unwrap().read_i32().unwrap();
+    assert_eq!(values.len(), 160_000);
+    for (i, &v) in values.iter().enumerate() {
+        assert_eq!(v, i as i32, "element {i}");
+    }
+}
+
+#[test]
+fn h5py_extensible_array_chunk_index_clawhdf5_reads() {
+    // One unlimited dimension means an Extensible Array chunk index. Only its
+    // first few elements live inline in the index block (4 by default), and
+    // every other fixture here is small enough to stop there — which is how
+    // the data block and super block layouts came to be wrong without a test
+    // noticing. The counts below step over each boundary in turn:
+    //   4      inline elements only
+    //   37     past the first direct data block
+    //   400    into the first super block
+    //   5000   several super block levels
+    //   200000 data blocks large enough to be paged
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+
+    for n in [4usize, 37, 400, 5_000, 200_000] {
+        let path = dir.path().join(format!("ea_{n}.h5"));
+        let path_str = path.display().to_string();
+        run_python(&format!(
+            r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=({n},), maxshape=(None,), chunks=(1,), dtype="i4")
+    d[...] = np.arange({n}, dtype="i4")
+"#
+        ));
+
+        let bytes = std::fs::read(&path).unwrap();
+        assert!(
+            bytes.windows(4).any(|w| w == b"EAHD"),
+            "n={n}: fixture is not indexed by an Extensible Array"
+        );
+
+        let file = File::open(&path).unwrap();
+        let values = file.dataset("x").unwrap().read_i32().unwrap();
+        assert_eq!(values.len(), n, "n={n}");
+        let wrong = values
+            .iter()
+            .enumerate()
+            .filter(|&(i, &v)| v != i as i32)
+            .count();
+        assert_eq!(wrong, 0, "n={n}: {wrong} of {n} elements read back wrong");
+    }
+}
+
+#[test]
+fn h5py_sparse_extensible_array_leaves_pages_uninitialised() {
+    // Writing a scattered subset leaves whole pages of a paged data block
+    // never initialised. Those pages still occupy their slot on disk, so the
+    // reader has to skip them by stride and take the fill value instead —
+    // driven by the page-init bitmap, which is packed one bit per page across
+    // the whole super block, MSB first.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("ea_sparse.h5");
+    let path_str = path.display().to_string();
+    let n = 200_000usize;
+    let step = 997usize;
+
+    run_python(&format!(
+        r#"
+import h5py
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=({n},), maxshape=(None,), chunks=(1,),
+                         dtype="i4", fillvalue=-1)
+    for i in list(range(0, {n}, {step})) + list(range(0, 40)):
+        d[i] = i
+"#
+    ));
+
+    let file = File::open(&path).unwrap();
+    let values = file.dataset("x").unwrap().read_i32().unwrap();
+    assert_eq!(values.len(), n);
+    let wrong = values
+        .iter()
+        .enumerate()
+        .filter(|&(i, &v)| {
+            let expected = if i % step == 0 || i < 40 {
+                i as i32
+            } else {
+                -1
+            };
+            v != expected
+        })
+        .count();
+    assert_eq!(wrong, 0, "{wrong} of {n} elements read back wrong");
+}
+
+#[test]
+fn h5py_filtered_and_2d_extensible_array_clawhdf5_reads() {
+    // Filtered elements carry a size and filter mask beside the address, and
+    // a second (fixed) dimension changes how a linear index maps back to
+    // chunk offsets. Both run through the same traversal.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+
+    let gz = dir.path().join("ea_gzip.h5");
+    let gz_str = gz.display().to_string();
+    run_python(&format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{gz_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=(5000,), maxshape=(None,), chunks=(1,),
+                         dtype="i4", compression="gzip", compression_opts=4)
+    d[...] = np.arange(5000, dtype="i4")
+"#
+    ));
+    let values = File::open(&gz)
+        .unwrap()
+        .dataset("x")
+        .unwrap()
+        .read_i32()
+        .unwrap();
+    assert_eq!(values.len(), 5000);
+    assert_eq!(
+        values
+            .iter()
+            .enumerate()
+            .filter(|&(i, &v)| v != i as i32)
+            .count(),
+        0
+    );
+
+    let two_d = dir.path().join("ea_2d.h5");
+    let two_d_str = two_d.display().to_string();
+    run_python(&format!(
+        r#"
+import h5py, numpy as np
+with h5py.File("{two_d_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=(3000, 4), maxshape=(None, 4), chunks=(1, 4), dtype="i4")
+    d[...] = np.arange(12000, dtype="i4").reshape(3000, 4)
+"#
+    ));
+    let values = File::open(&two_d)
+        .unwrap()
+        .dataset("x")
+        .unwrap()
+        .read_i32()
+        .unwrap();
+    assert_eq!(values.len(), 12_000);
+    assert_eq!(
+        values
+            .iter()
+            .enumerate()
+            .filter(|&(i, &v)| v != i as i32)
+            .count(),
+        0
+    );
+}
+
+#[test]
+fn h5py_fixed_array_chunk_index_clawhdf5_reads() {
+    // Fixed dimensions plus libver='latest' give a Fixed Array chunk index.
+    // Its data blocks are paged above 2^page_bits elements (1024 by default),
+    // and unlike the Extensible Array it keeps the page-init bitmap in the
+    // data block itself — a difference worth pinning down, since assuming
+    // otherwise is exactly what made the Extensible Array reader wrong. The
+    // sparse case leaves whole pages uninitialised so the bitmap is actually
+    // consulted rather than being all ones.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+
+    for n in [100usize, 5_000, 200_000] {
+        let path = dir.path().join(format!("fa_{n}.h5"));
+        let path_str = path.display().to_string();
+        run_python(&format!(
+            r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=({n},), chunks=(1,), dtype="i4")
+    d[...] = np.arange({n}, dtype="i4")
+"#
+        ));
+        let bytes = std::fs::read(&path).unwrap();
+        assert!(
+            bytes.windows(4).any(|w| w == b"FAHD"),
+            "n={n}: fixture is not indexed by a Fixed Array"
+        );
+        let values = File::open(&path)
+            .unwrap()
+            .dataset("x")
+            .unwrap()
+            .read_i32()
+            .unwrap();
+        assert_eq!(values.len(), n, "n={n}");
+        let wrong = values
+            .iter()
+            .enumerate()
+            .filter(|&(i, &v)| v != i as i32)
+            .count();
+        assert_eq!(wrong, 0, "n={n}: {wrong} elements read back wrong");
+    }
+
+    let sparse = dir.path().join("fa_sparse.h5");
+    let sparse_str = sparse.display().to_string();
+    let (n, step) = (200_000usize, 997usize);
+    run_python(&format!(
+        r#"
+import h5py
+with h5py.File("{sparse_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=({n},), chunks=(1,), dtype="i4", fillvalue=-1)
+    for i in list(range(0, {n}, {step})) + list(range(0, 40)):
+        d[i] = i
+"#
+    ));
+    let values = File::open(&sparse)
+        .unwrap()
+        .dataset("x")
+        .unwrap()
+        .read_i32()
+        .unwrap();
+    assert_eq!(values.len(), n);
+    let wrong = values
+        .iter()
+        .enumerate()
+        .filter(|&(i, &v)| {
+            let expected = if i % step == 0 || i < 40 {
+                i as i32
+            } else {
+                -1
+            };
+            v != expected
+        })
+        .count();
+    assert_eq!(wrong, 0, "sparse: {wrong} of {n} elements read back wrong");
+}
+
+#[test]
+fn corrupting_a_chunk_index_is_an_error_not_wrong_data() {
+    // Every Fixed/Extensible Array structure carries a Jenkins checksum, and
+    // the reader now verifies it. The point is not the checksum itself but
+    // what it prevents: a damaged index otherwise yields addresses pointing
+    // at the wrong bytes, and the caller receives another chunk's data as if
+    // it were the one asked for.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+
+    for (name, maxshape) in [("fixed", "None"), ("extensible", "(None,)")] {
+        let path = dir.path().join(format!("{name}.h5"));
+        let path_str = path.display().to_string();
+        let shape_arg = if maxshape == "None" {
+            String::new()
+        } else {
+            format!(", maxshape={maxshape}")
+        };
+        run_python(&format!(
+            r#"
+import h5py, numpy as np
+with h5py.File("{path_str}", "w", libver="latest") as f:
+    d = f.create_dataset("x", shape=(400,), chunks=(1,), dtype="i4"{shape_arg})
+    d[...] = np.arange(400, dtype="i4")
+"#
+        ));
+
+        let clean = std::fs::read(&path).unwrap();
+        assert_eq!(
+            File::open(&path)
+                .unwrap()
+                .dataset("x")
+                .unwrap()
+                .read_i32()
+                .unwrap()
+                .len(),
+            400,
+            "{name}: the intact file must read"
+        );
+
+        // Flip a low bit of a chunk address inside a data block. Structurally
+        // everything still parses — the index still has the right shape and
+        // the address still lands inside the file — so nothing but the
+        // checksum can notice. Without it the read succeeds and hands back
+        // whatever bytes now sit at that address.
+        let sig: &[u8] = if name == "fixed" { b"FADB" } else { b"EADB" };
+        let block = clean
+            .windows(4)
+            .position(|w| w == sig)
+            .unwrap_or_else(|| panic!("{name}: no data block in the fixture"));
+        // Past the prefix (signature, version, client id, header address, and
+        // for the Extensible Array a block offset), into the first address.
+        let at = block + 4 + 1 + 1 + 8 + if name == "fixed" { 0 } else { 4 } + 1;
+        let mut damaged = clean.clone();
+        damaged[at] ^= 0x10;
+        let damaged_path = dir.path().join(format!("{name}_damaged.h5"));
+        std::fs::write(&damaged_path, &damaged).unwrap();
+
+        let result = File::open(&damaged_path)
+            .unwrap()
+            .dataset("x")
+            .and_then(|d| d.read_i32());
+        assert!(
+            result.is_err(),
+            "{name}: corruption produced data instead of an error"
+        );
+    }
+}
+
+// ---------------------------------------------------------------------------
+// Half precision (float16) in both directions
+// ---------------------------------------------------------------------------
+
+/// Values that exercise rounding: ties, subnormals, the overflow boundary and
+/// ordinary embedding-sized components.
+fn f16_probe_values() -> Vec<f32> {
+    let mut v = vec![
+        0.0,
+        -0.0,
+        1.0,
+        -1.0,
+        0.5,
+        1.0 + 2f32.powi(-11),
+        1.0 + 3.0 * 2f32.powi(-11),
+        65504.0,
+        65519.0,
+        65520.0,
+        -70000.0,
+        6.0e-8,
+        3.0e-8,
+        1.0e-9,
+        1.0e-5,
+        0.1,
+        0.333_333,
+        1234.567,
+        f32::INFINITY,
+        f32::NEG_INFINITY,
+    ];
+    // A deterministic spread of embedding-like values.
+    let mut x = 0x2545_F491u32;
+    for _ in 0..4000 {
+        x ^= x << 13;
+        x ^= x >> 17;
+        x ^= x << 5;
+        v.push((x as f32 / u32::MAX as f32 - 0.5) * 0.4);
+    }
+    v
+}
+
+#[test]
+fn clawhdf5_writes_f16_h5py_reads() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("ours_f16.h5");
+    let path_str = path.display().to_string();
+    let values = f16_probe_values();
+
+    let mut fb = FileBuilder::new();
+    fb.create_dataset("plain").with_f16_data(&values);
+    fb.create_dataset("chunked")
+        .with_f16_data(&values)
+        .with_shape(&[values.len() as u64])
+        .with_chunks(&[512])
+        .with_deflate(6);
+    fb.write(&path).unwrap();
+
+    // h5py must see a genuine float16 dataset, and our rounding must agree
+    // with numpy's own float32 -> float16 conversion bit for bit.
+    let input = values
+        .iter()
+        .map(|v| format!("{:?}", v.to_bits()))
+        .collect::<Vec<_>>()
+        .join(",");
+    let script = format!(
+        r#"
+import h5py, numpy as np
+src = np.array([{input}], dtype=np.uint32).view(np.float32)
+expected = src.astype(np.float16).view(np.uint16)
+with h5py.File("{path_str}", "r") as f:
+    for name in ("plain", "chunked"):
+        d = f[name]
+        assert d.dtype == np.float16, (name, d.dtype)
+        got = d[:].view(np.uint16)
+        bad = np.nonzero(got != expected)[0]
+        assert bad.size == 0, (name, bad[:5], got[bad[:5]], expected[bad[:5]])
+print("ok")
+"#
+    );
+    assert_eq!(run_python_output(&script), "ok");
+}
+
+#[test]
+fn h5py_writes_f16_clawhdf5_reads() {
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("h5py_f16.h5");
+    let path_str = path.display().to_string();
+    let values = f16_probe_values();
+    let input = values
+        .iter()
+        .map(|v| format!("{:?}", v.to_bits()))
+        .collect::<Vec<_>>()
+        .join(",");
+
+    let script = format!(
+        r#"
+import h5py, numpy as np
+src = np.array([{input}], dtype=np.uint32).view(np.float32).astype(np.float16)
+with h5py.File("{path_str}", "w") as f:
+    f.create_dataset("plain", data=src)
+    f.create_dataset("chunked", data=src, chunks=(512,), compression="gzip", shuffle=True)
+    f.create_dataset("big_endian", data=src.astype(">f2"))
+"#
+    );
+    run_python(&script);
+
+    let expected: Vec<u32> = values
+        .iter()
+        .map(|&v| clawhdf5_format::float16::round_to_f16(v).to_bits())
+        .collect();
+    let file = File::open(&path).unwrap();
+    for name in ["plain", "chunked", "big_endian"] {
+        let ds = file.dataset(name).unwrap();
+        assert_eq!(
+            ds.dtype().unwrap(),
+            DType::Other("float16".into()),
+            "{name}"
+        );
+        let got: Vec<u32> = ds.read_f32().unwrap().iter().map(|v| v.to_bits()).collect();
+        assert_eq!(got, expected, "{name}");
+    }
+}
+
+#[test]
+fn clawhdf5_writes_f32_h5py_reads() {
+    // Every f32 dataset used to be unreadable by h5py ("sign bit position out
+    // of bounds"): the float datatype's sign position was hard-coded for f64.
+    skip_if_no_python!();
+    let dir = tempfile::tempdir().unwrap();
+    let path = dir.path().join("ours_f32.h5");
+    let path_str = path.display().to_string();
+    let values: Vec<f32> = vec![1.5, -2.25, 3.0e-7, 65536.5, f32::MAX, -0.0];
+
+    let mut fb = FileBuilder::new();
+    fb.create_dataset("plain").with_f32_data(&values);
+    fb.create_dataset("chunked")
+        .with_f32_data(&values)
+        .with_shape(&[values.len() as u64])
+        .with_chunks(&[4])
+        .with_deflate(6);
+    fb.write(&path).unwrap();
+
+    let bits = values
+        .iter()
+        .map(|v| v.to_bits().to_string())
+        .collect::<Vec<_>>()
+        .join(",");
+    let script = format!(
+        r#"
+import h5py, numpy as np
+expected = np.array([{bits}], dtype=np.uint32)
+with h5py.File("{path_str}", "r") as f:
+    for name in ("plain", "chunked"):
+        d = f[name]
+        assert d.dtype == np.float32, (name, d.dtype)
+        assert (d[:].view(np.uint32) == expected).all(), (name, d[:])
+print("ok")
+"#
+    );
+    assert_eq!(run_python_output(&script), "ok");
+}

crates/libaec-sys/Cargo.toml

@@ -2,6 +2,7 @@
 name = "libaec-sys"
 version = "0.1.0"
 edition = "2024"
+rust-version.workspace = true
 links = "aec"
 
 [build-dependencies]

docs/QUICKSTART.md

@@ -11,7 +11,7 @@ ClawhDF5 serves three audiences with different entry points:
 | You Are | You Want | Start Here |
 |---------|----------|------------|
 | **AI agent developer** | Persistent memory for your agent | [Agent Memory (Rust)](#1-agent-memory-rust-library) |
-| **OpenClaw user** | Better memory for your OpenClaw agent | [OpenClaw Integration](#2-openclaw-integration) |
+| **OpenClaw user** | clawhdf5 is not an OpenClaw memory plugin | [Status](openclaw.md) |
 | **Data scientist** | Read/write HDF5 files in Rust | [HDF5 File I/O](#3-hdf5-file-io) |
 | **CLI user** | Inspect and manage agent memories | [CLI Tool](#4-cli-tool) |
 | **Python user** | Use clawhdf5 from Python | [Python Bindings](#5-python-bindings) |
@@ -27,7 +27,7 @@ The core use case. Give your AI agent persistent, searchable memory in a single
 ```toml
 # Cargo.toml
 [dependencies]
-clawhdf5-agent = { version = "2.0", features = ["agent"] }
+clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # not on crates.io yet
 ```
 
 ### Create a Memory Store
@@ -197,80 +197,38 @@ if let Some(alert) = detector.check_rate_anomaly() {
 
 ---
 
-## 2. OpenClaw Integration
+## 2. Markdown Memory (and OpenClaw)
 
-ClawhDF5 can serve as the memory backend for [OpenClaw](https://docs.openclaw.ai) agents, replacing the default Markdown + sqlite-vec approach.
+**clawhdf5 is not an OpenClaw memory backend.** Earlier versions of this guide
+described one; it never worked — see [openclaw.md](openclaw.md) for what
+happened and what a real plugin would need.
 
-### How It Works
-
-```
-OpenClaw Agent
-    │
-    ├── memory_search("user preferences")
-    │       │
-    │       └── ClawhdfBackend
-    │               ├── Vector search (cosine)
-    │               ├── BM25 keyword search
-    │               ├── Reciprocal Rank Fusion
-    │               ├── Multi-factor re-ranking
-    │               └── Low-confidence rejection
-    │
-    └── agent_memory.h5 (single file, portable)
-```
-
-### Migration from Markdown
+What does exist is `ClawhdfBackend`, a library API that ingests Markdown files
+by section and searches them with the full pipeline (hybrid retrieval,
+re-ranking, confidence rejection):
 
 ```rust
 use clawhdf5_agent::openclaw::*;
+use std::path::Path;
 
-// Create a new HDF5 backend
-let mut backend = ClawhdfBackend::create("memory.h5", "my-agent", 384)?;
+let mut backend = ClawhdfBackend::create(Path::new("memory.h5"), 384)?;
 
-// Import your existing MEMORY.md
-let md = std::fs::read_to_string("~/.openclaw/workspace/MEMORY.md")?;
+// Each heading becomes a record, stored under "MEMORY.md::<heading>".
+let md = std::fs::read_to_string("MEMORY.md")?;
 let count = backend.ingest_markdown("MEMORY.md", &md)?;
-println!("Imported {} sections", count);
+println!("Imported {count} sections");
 
-// Import daily logs
-for entry in std::fs::read_dir("~/.openclaw/workspace/memory/")? {
-    let path = entry?.path();
-    if path.extension().map(|e| e == "md").unwrap_or(false) {
-        let content = std::fs::read_to_string(&path)?;
-        let name = path.file_name().unwrap().to_string_lossy();
-        backend.ingest_markdown(&name, &content)?;
-    }
-}
-
-// Search using the full pipeline
 let results = backend.search("what are user preferences", &query_embedding, 5);
 for r in &results {
     println!("[{:.3}] {} (from {})", r.score, r.text, r.path);
 }
-
-// Export back to Markdown (lossless roundtrip)
-let exported = backend.export_markdown("MEMORY.md")?;
 ```
 
-### What You Get Over sqlite-vec
-
-| Feature | sqlite-vec | ClawhDF5 |
-|---------|-----------|----------|
-| Vector search | ✅ | ✅ (8× faster at 100K) |
-| Keyword search | ❌ | ✅ BM25 |
-| Hybrid fusion | ❌ | ✅ RRF |
-| Re-ranking | ❌ | ✅ Multi-factor |
-| Confidence rejection | ❌ | ✅ |
-| Knowledge graph | ❌ | ✅ |
-| Memory consolidation | ❌ | ✅ |
-| Temporal queries | ❌ | ✅ (716ns) |
-| Anomaly detection | ❌ | ✅ |
-| Provenance tracking | ❌ | ✅ |
-| Multi-modal | ❌ | ✅ |
-| Single portable file | ❌ (SQLite + MD files) | ✅ |
-
-### Future: Native OpenClaw Plugin
-
-The Phase 2 roadmap includes a native OpenClaw plugin (`memory.backend = "clawhdf5"`) that transparently replaces sqlite-vec. Until then, the Rust library can be wrapped via NAPI or used from the CLI.
+Limits to know: sections ingested this way carry no embedding (search over them
+is keyword-only unless you save records with vectors via `save_entry`);
+ingesting the same file again adds the sections again rather than replacing
+them; and `export_markdown` rewrites every heading as `##`, so it is not a
+lossless round trip.
 
 ---
 
@@ -364,10 +322,11 @@ cargo install --path crates/clawhdf5-cli
 clawhdf5 --path agent.h5 create --agent-id my-agent --dim 384 --wal
 ```
 
-Add `--quantized-index` to store the vector index's copy of the embeddings as
-int8. That roughly halves a loaded store's memory at about 13% fewer queries
-per second, with recall unchanged — the query path re-scores candidates
-against the exact embeddings. The setting is recorded in the file.
+New stores hold the vector index's copy of the embeddings as int8, which
+roughly halves a loaded store's memory and is faster at equal recall — the
+query path re-scores candidates against the exact embeddings. Pass
+`--f32-index` to keep an f32 index instead. The setting is recorded in the
+file, and stores created before it existed keep their f32 index.
 
 Output:
 ```json
@@ -546,7 +505,7 @@ let final_results = confidence::reject_low_confidence(
 
 **Why not a vector database?** Pinecone, Qdrant, Weaviate — they're cloud services or heavy servers. Agent memory should be local, portable, and zero-dependency. An agent's memories should travel with it.
 
-**Why not Markdown?** OpenClaw uses Markdown today and it works for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't.
+**Why not Markdown?** Plain Markdown files work for simple cases. But it doesn't scale: no vector search, no knowledge graph, no structured retrieval. ClawhDF5 can import/export Markdown while providing everything Markdown can't.
 
 **Why HDF5 specifically?**
 - Native N-dimensional array storage (perfect for embeddings)
@@ -566,4 +525,4 @@ let final_results = confidence::reject_low_confidence(
 
 ---
 
-<p align="center"><em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em></p>
+<p align="center"><em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em></p>

docs/USE_CASES.md

@@ -42,37 +42,10 @@ conversation → embedding → save to agent.h5
 
 ---
 
-## 2. OpenClaw Memory Upgrade
+## 2. OpenClaw
 
-**Scenario:** You run OpenClaw and the default Markdown + sqlite-vec memory works OK for simple recall but falls short on complex queries like "what did we decide about the deployment architecture last Tuesday?" or "who's responsible for the billing system?"
-
-**Problem:** Markdown files have no semantic structure. sqlite-vec does flat vector search — no keyword fusion, no re-ranking, no temporal reasoning, no knowledge graph.
-
-**ClawhDF5 solution:**
-
-```bash
-# Migrate existing memories
-clawhdf5 --path memory.h5 create --agent-id openclaw --dim 384
-
-# Import your MEMORY.md and daily logs
-# (programmatically via ClawhdfBackend::ingest_markdown)
-```
-
-Then in your OpenClaw config (future):
-```json
-{
-  "memory": {
-    "backend": "clawhdf5",
-    "path": "~/.openclaw/agents/main/memory.h5"
-  }
-}
-```
-
-**What changes:**
-- "What did we discuss last Tuesday?" → temporal index finds the session, returns memories from that time range
-- "Who owns the billing system?" → knowledge graph traversal: billing_system → owned_by → Alice
-- "Preferences about deployment" → hybrid search (vector + BM25) finds relevant memories even with different wording
-- Bad search results get filtered out by confidence rejection instead of confusing the agent
+Not supported: clawhdf5 is not an OpenClaw memory plugin, and the config this
+section used to show was never valid. See [openclaw.md](openclaw.md).
 
 ---
 
@@ -212,7 +185,7 @@ This is the container image for intelligence.
 | Your Situation | Features to Enable | Why |
 |----------------|-------------------|-----|
 | **Quick prototype** | Default | Vector search works out of the box |
-| **Production agent** | `agent`, `float16`, `parallel` | Half-precision saves 50% storage, parallel search for scale |
+| **Production agent** | defaults (`float16`, `hnsw`, `parallel`) | HNSW search and a parallel index build; half-precision *storage* is `MemoryConfig::float16`, on by default for new stores |
 | **macOS** | + `accelerate` | Apple AMX coprocessor for matrix ops |
 | **Linux server** | + `openblas` or `fast-math` | BLAS acceleration |
 | **GPU available** | + `gpu` | wgpu-based search, wins at 100K+ scale |
@@ -220,16 +193,17 @@ This is the container image for intelligence.
 | **Edge device** | Default only | Minimal dependencies, smallest binary |
 
 ```toml
+# Not on crates.io yet: depend on the repository.
 # Production agent on Linux
-clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "parallel", "fast-math"] }
+clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["fast-math"] }
 
 # Edge device
-clawhdf5-agent = { version = "2.0", features = ["agent"] }
+clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 
 # macOS with GPU
-clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "accelerate", "gpu", "async"] }
+clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["accelerate", "gpu", "async"] }
 ```
 
 ---
 
-<p align="center"><em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em></p>
+<p align="center"><em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em></p>

docs/known-issues.md

@@ -169,3 +169,136 @@ python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4
 
 Set `CLAWHDF5_REQUIRE_INTEROP=1` in any automated runner so a missing
 interpreter is a failure rather than a skip.
+
+
+---
+
+## Crafted B-tree v2 structures crash or exhaust the reader
+
+**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
+and including v2.6.0 is affected.**
+
+B-tree v2 traversal (`clawhdf5-format`, `btree_v2::collect_btree_v2_records`)
+recursed one frame per level with the depth taken from the file, and followed
+child addresses without checking whether they were shared. Two consequences
+for anyone reading untrusted files:
+
+- A node that is its own child, under a header claiming 65 535 levels, overflows
+  the stack and aborts the process. The file is under 100 bytes.
+- Levels whose children all point at one node below make the traversal visit it
+  fan-out^depth times: ~30 million records from ~5 KB, and memory exhaustion one
+  level deeper.
+
+B-tree v2 backs dense attribute storage, v2 groups, shared object header
+messages and chunk indexes, so opening an object that uses any of them is
+enough. Both are now errors: depth is capped at 64, and traversal stops once it
+has produced more records than the file could physically hold.
+
+---
+
+## Extensible Array chunk indexes read back wrong data past the inline elements
+
+**Status:** fixed on `main` (2026-09-20), after v2.6.0. **Every release up to
+and including v2.6.0 is affected.**
+
+A dataset created with exactly one unlimited dimension (`maxshape=(None, ...)`,
+the usual append-only/resizable case) is indexed by an Extensible Array. Its
+index block holds the first `idx_blk_elmts` chunk entries inline — 4 by
+default — and everything after that lives in data blocks and super blocks whose
+layout `clawhdf5-format` computed incorrectly.
+
+Consequences, by dataset size (1 chunk per element):
+
+| chunks | result before the fix |
+|---|---|
+| <= 36 | correct (inline, plus two data blocks that happened to line up) |
+| 37 | 1 element wrong |
+| 400 | 364 elements wrong |
+| >= ~1000 | `invalid Extensible Array data block signature` |
+
+The dangerous case is the middle one: values were returned from the wrong
+chunks rather than an error being raised. Any reader that accepted the data at
+face value saw plausible but incorrect numbers.
+
+The root causes were the super block sizing formulas (`ndblks` and
+`dblk_nelmts` each double every *other* level, a half-step apart), a missing
+block-offset field in the super block, and a page-init bitmap read from the
+wrong structure. All four are fixed and covered by interop tests against
+HDF5 2.0 at sizes that cross each boundary, including paged data blocks.
+
+Files written by this crate are unaffected — this was purely a read-path bug.
+
+## Every `f32` dataset we wrote was unreadable by h5py / libhdf5
+
+**Status:** fixed 2026-09-23, after v2.7.0. **Every
+release up to and including v2.7.0 is affected** — the encoder was already
+wrong in v2.1.0.
+
+The floating-point datatype message carries the position of the sign bit
+(bits 8–15 of its class bit field). `clawhdf5-format` wrote 63 for every
+float, which is correct only for `f64`. libhdf5 validates the field, so opening
+any `f32` dataset written by this crate failed:
+
+```
+KeyError: 'Unable to synchronously open object (sign bit position out of bounds)'
+```
+
+That covers every agent store (`/memory/embeddings`, `norms` and
+`activation_weights` are `f32`). `clawhdf5` itself ignores the field on read,
+and the interop suites only ever wrote `f64` from our side, so nothing here
+noticed.
+
+**Fix:** the sign position is computed from the type (`bit_offset +
+bit_precision - 1`: 15, 31, 63 for half, single, double). Regression tests:
+`float_sign_location_is_the_top_bit_of_the_value` (byte level),
+`clawhdf5_writes_f32_h5py_reads` and the agent's
+`h5py_reads_every_dataset_of_an_agent_store`.
+
+**Existing files:** an agent store is rewritten in full at every checkpoint, so
+it becomes readable by h5py at its next checkpoint with a fixed build. Other
+files with `f32` datasets need to be rewritten.
+
+## Empty datasets we wrote were unreadable by h5py / libhdf5
+
+**Status:** fixed 2026-09-23, after v2.7.0. Every
+release up to and including v2.7.0 is affected.
+
+A dataset with no elements was written with a real file address and a storage
+size of 0. libhdf5 guards contiguous storage with an overflow check
+(`addr + size <= addr`) that is always true when the size is 0, so it rejected
+the dataset:
+
+```
+KeyError: 'Unable to synchronously open object (invalid dataset size, likely file corruption)'
+```
+
+In practice: every agent store without sessions or a knowledge graph — the
+`/sessions` and `/knowledge_graph` datasets are empty until something is added
+— could not be read by h5py even once the `f32` bug above was fixed. Found by
+the same agent-store interop test.
+
+**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
+which is what libhdf5 itself writes.
+
+## The Node.js package (`packages/clawhdf5-node`) does not work
+
+**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
+
+The TypeScript wrapper over `crates/clawhdf5-napi` has never run successfully:
+
+- napi-rs converts `#[napi(object)]` fields to camelCase, but the wrapper reads
+  snake_case (`r.line_range`, `s.total_records`, `s.working_count`, …), so
+  every stats and consolidation field comes back `undefined`
+  (`src/index.ts:76-120`).
+- It loads `../clawhdf5.node`, but `napi build --platform` produces
+  `clawhdf5.<triple>.node`; `main` points at `index.js` while `tsc` writes to
+  `dist/`; `napi prepublish` expects per-platform packages that are not
+  defined.
+- `save`/`saveBatch` exist in the napi layer but not in the wrapper, so a
+  TypeScript caller cannot store an embedding at all.
+- The WAL for `agent.brain` is `agent.h5.wal` (the store uses
+  `with_extension("h5.wal")`), not `agent.brain.wal` as the old docs and the
+  test cleanup assume.
+
+It was written for an OpenClaw integration that is not being pursued (see
+`docs/openclaw.md`). Fix and add CI, or remove it, before anyone depends on it.

docs/migration-guide.md

@@ -1,213 +0,0 @@
-# Migration Guide: OpenClaw sqlite-vec → clawhdf5
-
-This guide walks through migrating an OpenClaw agent from its default
-sqlite-vec + Markdown file memory to the `clawhdf5` HDF5 backend.
-
----
-
-## Why migrate?
-
-| Feature | sqlite-vec + Markdown | clawhdf5 |
-|---------|----------------------|----------|
-| Storage format | SQLite WAL + flat .md files | Single HDF5 binary file |
-| Vector search | sqlite-vec (SQLite extension) | Pure-Rust SIMD (clawhdf5-accel) |
-| Full-text search | External (FTS5 or plain string match) | Built-in BM25 |
-| Hybrid search | Manual combination | Automatic RRF blend |
-| Memory tiers | Flat | Working → Episodic → Semantic |
-| Hebbian decay | Not built-in | Automatic activation weighting |
-| Portability | SQLite binary required | Zero native deps (all Rust) |
-| Crash recovery | SQLite WAL | clawhdf5 WAL |
-| Compaction | Manual | Auto-threshold + session-end |
-| Embedding dim change | New DB required | New file required (same) |
-
----
-
-## Step 1: Install `@redclaw/clawhdf5`
-
-```bash
-npm install @redclaw/clawhdf5
-```
-
-Or, if building from the monorepo source:
-
-```bash
-npm install -g @napi-rs/cli
-cd packages/clawhdf5-node
-npm install
-npm run build
-```
-
----
-
-## Step 2: Update your OpenClaw config
-
-Change `backend` from `"sqlite-vec"` (or `"markdown"`) to `"clawhdf5"`:
-
-```json
-{
-  "memory": {
-    "backend": "clawhdf5",
-    "clawhdf5": {
-      "path": "./agent.brain",
-      "embeddingDim": 768
-    }
-  }
-}
-```
-
-See [openclaw-config.md](openclaw-config.md) for the full schema.
-
----
-
-## Step 3: Run the one-time migration
-
-clawhdf5 ships a migration helper that reads your existing Markdown memory
-files and ingests them via `ingestMarkdown()`.
-
-### Automated migration script
-
-```typescript
-import { ClawhdfMemory } from '@redclaw/clawhdf5';
-import { readFileSync, readdirSync, statSync } from 'fs';
-import { join, relative } from 'path';
-
-async function migrate(
-  memoryDir: string,
-  brainPath: string,
-  embeddingDim: number = 768,
-): Promise<void> {
-  const mem = ClawhdfMemory.create(brainPath, embeddingDim);
-
-  // Walk all .md files under memoryDir
-  function walk(dir: string): string[] {
-    return readdirSync(dir).flatMap((entry) => {
-      const full = join(dir, entry);
-      return statSync(full).isDirectory() ? walk(full) : [full];
-    });
-  }
-
-  const files = walk(memoryDir).filter((f) => f.endsWith('.md'));
-  let totalSections = 0;
-
-  for (const file of files) {
-    const content = readFileSync(file, 'utf8');
-    const relPath = relative(process.cwd(), file);
-    const count = mem.ingestMarkdown(relPath, content);
-    console.log(`  ${relPath}: ${count} sections`);
-    totalSections += count;
-  }
-
-  // Force WAL merge after bulk import
-  mem.flushWal();
-
-  console.log(`\nMigration complete: ${files.length} files, ${totalSections} sections`);
-  const s = mem.stats();
-  console.log(`  Total records:    ${s.totalRecords}`);
-  console.log(`  File size:        ${(s.fileSizeBytes / 1024).toFixed(1)} KB`);
-}
-
-// Usage
-migrate('./memory', './agent.brain', 768).catch(console.error);
-```
-
-### What the migration does
-
-1. Walks all `.md` files under your memory directory.
-2. Parses each file into sections using the same `MarkdownParser` used by
-   OpenClaw (splits on ATX headings `#`, `##`, `###`, …).
-3. Stores each section as a separate record in the HDF5 file with the file
-   path as the `source_channel` (e.g. `memory/user.md::Goals`).
-4. Flushes the WAL to merge everything into the `.brain` file.
-
-After migration, **the original `.md` files are not modified or deleted**.
-You can keep them as a backup or remove them once you have verified the
-migrated data.
-
----
-
-## Step 4: Verify
-
-```typescript
-import { ClawhdfMemory } from '@redclaw/clawhdf5';
-
-const mem = ClawhdfMemory.open('./agent.brain');
-const s = mem.stats();
-console.log('Records after migration:', s.totalRecords);
-
-// Spot-check: retrieve a known path
-const userMd = mem.get('memory/MEMORY.md');
-console.log(userMd?.slice(0, 200));
-
-// Round-trip a file back to Markdown
-const exported = mem.exportMarkdown('memory/MEMORY.md');
-console.log(exported.slice(0, 500));
-```
-
----
-
-## Step 5: Update agent code
-
-If your agent code reads memory files directly from disk, update it to use
-the clawhdf5 API instead:
-
-**Before (sqlite-vec + file reads):**
-```typescript
-const content = readFileSync('memory/user.md', 'utf8');
-const sections = parseMarkdown(content);
-const results = await vectorSearch(query, sections, k);
-```
-
-**After (clawhdf5):**
-```typescript
-import { ClawhdfMemory } from '@redclaw/clawhdf5';
-
-const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
-const embedding = await embed(query); // your embedding function
-const results = mem.search(query, new Float32Array(embedding), k);
-```
-
----
-
-## Step 6: Session lifecycle hooks
-
-Add compaction at session end for best long-term memory health:
-
-```typescript
-// At the start of your agent process
-const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
-
-// ... agent runs ...
-
-// At the end of each session
-mem.tickSession();                                 // decay activation weights
-const stats = mem.runConsolidation(Date.now() / 1000);  // promote memories
-console.log('[memory] consolidation:', stats);
-```
-
----
-
-## Rollback
-
-If you need to roll back to sqlite-vec:
-
-1. Change `memory.backend` back to `"sqlite-vec"` in your config.
-2. The original `.md` files are unchanged (if you kept them).
-3. Delete `agent.brain` (and `agent.brain.wal` if present).
-
----
-
-## Troubleshooting
-
-### `Error: no records found for path: memory/user.md`
-The path passed to `get()` or `exportMarkdown()` must exactly match the
-relative path used during `ingestMarkdown()`. Check for leading `./`
-differences.
-
-### Memory is empty after reopening
-Make sure `flushWal()` was called after bulk writes. Without it, entries
-remain in the WAL and may be lost if the process exits abnormally.
-
-### Embedding dimension mismatch
-The `embeddingDim` passed to `create()` cannot be changed after the file is
-created. If you switch embedding models, create a new `.brain` file and
-re-run the migration script.

docs/openclaw-config.md

@@ -1,144 +0,0 @@
-# OpenClaw × clawhdf5 Configuration Reference
-
-This document describes the full configuration schema for integrating
-`clawhdf5` as the memory backend in an OpenClaw agent gateway.
-
----
-
-## Minimal example
-
-```json
-{
-  "memory": {
-    "backend": "clawhdf5",
-    "clawhdf5": {
-      "path": "./agent.brain",
-      "embeddingDim": 768
-    }
-  }
-}
-```
-
----
-
-## Full schema
-
-```json
-{
-  "memory": {
-    "backend": "clawhdf5",
-    "clawhdf5": {
-      "path": "./agent.brain",
-      "embeddingDim": 768,
-      "walEnabled": true,
-      "walMaxEntries": 500,
-      "consolidation": {
-        "workingCapacity": 100,
-        "episodicCapacity": 10000,
-        "episodicHalfLifeDays": 7,
-        "semanticHalfLifeDays": 30,
-        "promotionThreshold": 0.6,
-        "semanticAccessThreshold": 10
-      },
-      "compaction": {
-        "autoCompactThreshold": 0.3,
-        "tickOnSessionEnd": true,
-        "consolidateOnCompaction": true
-      }
-    }
-  }
-}
-```
-
----
-
-## Field reference
-
-### Top level
-
-| Field | Type | Default | Description |
-|-------|------|---------|-------------|
-| `memory.backend` | `string` | `"clawhdf5"` | Must be `"clawhdf5"` to activate this backend |
-
-### `clawhdf5`
-
-| Field | Type | Default | Description |
-|-------|------|---------|-------------|
-| `path` | `string` | `"./agent.brain"` | Filesystem path for the `.brain` (HDF5) file. Relative to the OpenClaw working directory. |
-| `embeddingDim` | `number` | `768` | Dimension of the embedding vectors. Must match the embedder model. Common values: `384` (MiniLM), `768` (nomic-embed-text, BGE-base), `1536` (OpenAI text-embedding-3-small). |
-| `walEnabled` | `boolean` | `true` | Enable the Write-Ahead Log for crash recovery. Disable only on read-only stores or when crash safety is not required. |
-| `walMaxEntries` | `number` | `500` | Number of WAL entries to accumulate before an automatic merge to the .h5 file. Lower values = more frequent flushes (safer, slightly slower). |
-
-### `clawhdf5.consolidation`
-
-Controls the hippocampal three-tier memory engine (Working → Episodic →
-Semantic).
-
-| Field | Type | Default | Description |
-|-------|------|---------|-------------|
-| `workingCapacity` | `number` | `100` | Maximum records in the Working tier before lowest-decay entries are evicted. |
-| `episodicCapacity` | `number` | `10000` | Maximum records in the Episodic tier. |
-| `episodicHalfLifeDays` | `number` | `7` | Half-life (in days) for exponential decay of Episodic records. Records not accessed within roughly one half-life drop in importance. |
-| `semanticHalfLifeDays` | `number` | `30` | Half-life for Semantic records. Longer than Episodic — semantic knowledge decays slowly. |
-| `promotionThreshold` | `number` | `0.6` | Importance score (0–1) above which a Working record is promoted to the Episodic tier. Higher = more selective. |
-| `semanticAccessThreshold` | `number` | `10` | Minimum access count for an Episodic record to be promoted to Semantic. |
-
-### `clawhdf5.compaction`
-
-| Field | Type | Default | Description |
-|-------|------|---------|-------------|
-| `autoCompactThreshold` | `number` | `0.3` | Fraction of tombstoned records (0–1) that triggers automatic compaction. `0.3` = compact when 30% of records are deleted. Set to `0` to disable auto-compact. |
-| `tickOnSessionEnd` | `boolean` | `true` | Run `tickSession()` (Hebbian decay) automatically when the agent session closes. |
-| `consolidateOnCompaction` | `boolean` | `true` | Run the hippocampal consolidation engine after each compaction cycle. |
-
----
-
-## Embedder compatibility
-
-The `embeddingDim` must remain constant for the lifetime of a `.brain` file.
-Mixing embedding models in the same file is not supported.
-
-| Embedder | `embeddingDim` |
-|----------|---------------|
-| `all-MiniLM-L6-v2` | `384` |
-| `nomic-embed-text` | `768` |
-| `BGE-base-en-v1.5` | `768` |
-| `OpenAI text-embedding-3-small` | `1536` |
-| `OpenAI text-embedding-3-large` | `3072` |
-
----
-
-## OpenClaw integration code
-
-```typescript
-import { ClawhdfMemory } from '@redclaw/clawhdf5';
-
-// Load config from your OpenClaw config file
-const cfg = loadConfig(); // your config loading logic
-
-const mem = ClawhdfMemory.openOrCreate(
-  cfg.memory.clawhdf5.path,
-  cfg.memory.clawhdf5.embeddingDim ?? 768,
-);
-
-// On session end
-if (cfg.memory.clawhdf5.compaction?.tickOnSessionEnd) {
-  mem.tickSession();
-}
-if (cfg.memory.clawhdf5.compaction?.consolidateOnCompaction) {
-  const stats = mem.runConsolidation(Date.now() / 1000);
-  console.log('[clawhdf5] consolidation:', stats);
-}
-```
-
----
-
-## Environment variables
-
-The following environment variables override config file values when set:
-
-| Variable | Overrides |
-|----------|-----------|
-| `CLAWHDF5_PATH` | `clawhdf5.path` |
-| `CLAWHDF5_EMBEDDING_DIM` | `clawhdf5.embeddingDim` |
-| `CLAWHDF5_WAL_ENABLED` | `clawhdf5.walEnabled` (`"true"` / `"false"`) |

docs/openclaw-integration.md

@@ -1,337 +0,0 @@
-# OpenClaw × clawhdf5 Integration
-
-clawhdf5 provides a drop-in HDF5-backed memory backend for the
-[OpenClaw](https://github.com/redclawsystems/openclaw) agent gateway.
-This document covers architecture, the full Node.js API reference, and code
-examples for common operations.
-
----
-
-## Architecture
-
-```
-┌─────────────────────────────────────────────────────────────┐
-│  OpenClaw (Node.js/TypeScript)                              │
-│                                                             │
-│  ┌─────────────────┐    ┌──────────────────────────────┐   │
-│  │  Agent runtime  │───▶│  @redclaw/clawhdf5 (Node.js) │   │
-│  └─────────────────┘    │  TypeScript wrapper          │   │
-│                         └──────────────┬─────────────┘    │
-│                                        │ napi-rs FFI        │
-└────────────────────────────────────────┼────────────────────┘
-                                         │
-┌────────────────────────────────────────▼────────────────────┐
-│  clawhdf5-napi  (Rust, cdylib)                              │
-│                                                             │
-│  ClawhdfMemory  ──▶  ClawhdfBackend  ──▶  HDF5Memory       │
-│                       MemoryBackend        ├─ MemoryCache   │
-│                       trait impl          ├─ WalFile        │
-│                                           ├─ SessionCache   │
-│                                           └─ KnowledgeCache │
-│                                                             │
-│  ConsolidationEngine (hippocampal tiers)                    │
-│   Working (100) ──▶ Episodic (10k) ──▶ Semantic (∞)        │
-└─────────────────────────────────────────────────────────────┘
-                                         │
-                        ┌────────────────▼────────────┐
-                        │  agent.brain  (HDF5 file)    │
-                        │  agent.brain.wal  (WAL log)  │
-                        └─────────────────────────────┘
-```
-
-### Key design decisions
-
-- **Single file**: everything lives in one `.brain` HDF5 file (+ WAL sidecar).
-- **In-memory cache**: the full embedding matrix and chunk list are loaded into RAM for fast search.
-- **Hybrid search**: vector similarity (70%) and BM25 full-text (30%) are blended with Reciprocal Rank Fusion (RRF), then re-ranked by Hebbian activation weight and temporal recency.
-- **Hippocampal tiers**: records are classified as Working, Episodic, or Semantic based on importance and access frequency. Tier promotion and eviction happen during `runConsolidation()`.
-- **WAL**: writes are journaled before hitting the .h5 file. On crash, the WAL is replayed at next open.
-
----
-
-## Installation
-
-```bash
-npm install @redclaw/clawhdf5
-```
-
-See [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md)
-for build-from-source instructions.
-
----
-
-## Node.js API reference
-
-### `ClawhdfMemory` (class)
-
-All instance methods are synchronous.  The native Rust code is single-threaded
-on the Node.js side; do **not** share a `ClawhdfMemory` instance across Worker
-threads without external locking.
-
----
-
-#### Static factory methods
-
-##### `ClawhdfMemory.create(path: string, embeddingDim: number): ClawhdfMemory`
-
-Create a new `.brain` file. Throws if the file already exists.
-
-```typescript
-const mem = ClawhdfMemory.create('./agent.brain', 768);
-```
-
-##### `ClawhdfMemory.open(path: string): ClawhdfMemory`
-
-Open an existing file. Replays the WAL automatically.
-
-```typescript
-const mem = ClawhdfMemory.open('./agent.brain');
-```
-
-##### `ClawhdfMemory.openOrCreate(path: string, embeddingDim: number): ClawhdfMemory`
-
-**Recommended entry point.** Opens if the file exists, otherwise creates it.
-
-```typescript
-const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
-```
-
----
-
-#### `search(queryText, queryEmbedding, k): MemorySearchResult[]`
-
-Hybrid BM25 + vector search.
-
-```typescript
-const embedding = new Float32Array(await embed(query));
-const results = mem.search(query, embedding, 10);
-for (const r of results) {
-  console.log(r.score.toFixed(3), r.path, r.text.slice(0, 80));
-}
-```
-
-Pass an empty `Float32Array` to use BM25 only (no vector similarity).
-
-**Parameters:**
-- `queryText: string` — used for BM25 term matching
-- `queryEmbedding: Float32Array` — dense vector of length `embeddingDim`
-- `k: number` — maximum results to return
-
-**Returns:** `MemorySearchResult[]`
-
----
-
-#### `get(path, fromLine?, numLines?): string | null`
-
-Retrieve stored content by path.
-
-```typescript
-const md = mem.get('memory/user.md');        // all content
-const lines = mem.get('memory/user.md', 5, 10);  // lines 5–14
-const section = mem.get('memory/user.md::Goals'); // specific section
-```
-
-Section sub-paths use the `::heading` suffix produced by `ingestMarkdown`.
-
----
-
-#### `write(path, content): void`
-
-Store raw content at `path`.
-
-```typescript
-mem.write('memory/session.md', '# Session\n\nWorking on task X.');
-```
-
----
-
-#### `ingestMarkdown(path, content): number`
-
-Parse `content` as Markdown, split on ATX headings, and store each section
-separately.  Returns the number of sections ingested.
-
-```typescript
-import { readFileSync } from 'fs';
-const md = readFileSync('./memory/MEMORY.md', 'utf8');
-const count = mem.ingestMarkdown('memory/MEMORY.md', md);
-console.log(`Ingested ${count} sections`);
-```
-
-Sections are addressable as `memory/MEMORY.md::HeadingName`.
-
----
-
-#### `exportMarkdown(path): string`
-
-Reconstruct stored sections for `path` back into a Markdown string.
-
-```typescript
-const md = mem.exportMarkdown('memory/MEMORY.md');
-writeFileSync('./memory/MEMORY.md', md);
-```
-
----
-
-#### `stats(): BackendStats`
-
-Return aggregate statistics.
-
-```typescript
-const s = mem.stats();
-console.log(`Records: ${s.totalRecords}, Size: ${s.fileSizeBytes} bytes`);
-```
-
----
-
-#### `compact(): number`
-
-Remove tombstoned records from the store. Returns count removed.
-
----
-
-#### `tickSession(): void`
-
-Apply Hebbian decay to all activation weights. Call at session end.
-
----
-
-#### `flushWal(): void`
-
-Force a WAL merge: flush `.h5` and truncate the WAL log.
-
----
-
-#### `runConsolidation(nowSecs: number): ConsolidationStats`
-
-Run one full hippocampal consolidation cycle.
-
-```typescript
-const stats = mem.runConsolidation(Date.now() / 1000);
-console.log(stats);
-// { workingCount: 42, episodicCount: 310, semanticCount: 5,
-//   totalEvictions: 0, totalPromotions: 7 }
-```
-
----
-
-#### `walPendingCount(): number`
-
-Number of pending WAL entries (0 if WAL is disabled).
-
----
-
-### Type reference
-
-```typescript
-interface MemorySearchResult {
-  text: string;
-  score: number;            // 0–1, higher = more relevant
-  path: string;             // source file path
-  lineRange?: [number, number];
-  timestamp?: number;       // Unix epoch seconds
-  source: string;
-}
-
-interface BackendStats {
-  totalRecords: number;
-  totalEmbeddings: number;
-  fileSizeBytes: number;
-  modalities: string[];     // e.g. ["text"]
-  lastUpdated?: number;     // Unix epoch seconds
-}
-
-interface ConsolidationStats {
-  workingCount: number;
-  episodicCount: number;
-  semanticCount: number;
-  totalEvictions: number;
-  totalPromotions: number;
-}
-```
-
----
-
-## Common patterns
-
-### Session lifecycle
-
-```typescript
-import { ClawhdfMemory } from '@redclaw/clawhdf5';
-
-const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
-
-// --- agent session runs ---
-
-// On session end: decay + consolidate
-mem.tickSession();
-const consolidationStats = mem.runConsolidation(Date.now() / 1000);
-console.log('[memory] consolidation:', consolidationStats);
-```
-
-### Ingest all memory files at startup
-
-```typescript
-import { readdirSync, readFileSync, statSync } from 'fs';
-import { join, relative } from 'path';
-
-function ingestDirectory(mem: ClawhdfMemory, dir: string): void {
-  for (const entry of readdirSync(dir)) {
-    const full = join(dir, entry);
-    if (statSync(full).isDirectory()) {
-      ingestDirectory(mem, full);
-    } else if (entry.endsWith('.md')) {
-      const content = readFileSync(full, 'utf8');
-      const path = relative(process.cwd(), full);
-      mem.ingestMarkdown(path, content);
-    }
-  }
-  mem.flushWal();
-}
-
-const mem = ClawhdfMemory.openOrCreate('./agent.brain', 768);
-ingestDirectory(mem, './memory');
-```
-
-### Search with real embeddings
-
-```typescript
-import OpenAI from 'openai';
-import { ClawhdfMemory } from '@redclaw/clawhdf5';
-
-const ai = new OpenAI();
-const mem = ClawhdfMemory.openOrCreate('./agent.brain', 1536);
-
-async function searchMemory(query: string, k = 5) {
-  const resp = await ai.embeddings.create({
-    model: 'text-embedding-3-small',
-    input: query,
-  });
-  const embedding = new Float32Array(resp.data[0].embedding);
-  return mem.search(query, embedding, k);
-}
-```
-
----
-
-## Error handling
-
-All methods that can fail throw a `NapiError` (a standard JS `Error` subclass)
-with the Rust error message as `message`.
-
-```typescript
-try {
-  const md = mem.exportMarkdown('nonexistent.md');
-} catch (e) {
-  console.error('Export failed:', (e as Error).message);
-  // "no records found for path: nonexistent.md"
-}
-```
-
----
-
-## See also
-
-- [openclaw-config.md](openclaw-config.md) — Full configuration schema
-- [migration-guide.md](migration-guide.md) — Migrating from sqlite-vec
-- [packages/clawhdf5-node/README.md](../packages/clawhdf5-node/README.md) — Build instructions
-- [BENCHMARKS.md](../BENCHMARKS.md) — Performance results

docs/openclaw.md

@@ -0,0 +1,74 @@
+# OpenClaw: not supported
+
+**clawhdf5 does not currently work as an [OpenClaw](https://docs.openclaw.ai)
+memory backend, and never has.** Earlier versions of these docs described a
+"drop-in" backend enabled with `memory.backend = "clawhdf5"`. That
+configuration was never valid: from v2026.2 through v2026.7 OpenClaw's
+`memory.backend` accepted only `"builtin"` or `"qmd"` and rejected unknown
+keys, and since v2026.8.1 ("OpenClaw 2.0") the key no longer exists. A Gateway
+given that config refuses to start. No plugin was ever built or tested against
+OpenClaw, and the `@redclaw/clawhdf5` npm package was never published.
+
+As of 2026-09-25 we are not pursuing an OpenClaw plugin, and clawhdf5 has no
+framework integration at all (ZeroClaw, also named as a consumer in older
+docs, does not use it either). This page records what an OpenClaw plugin would
+need, for when that changes.
+
+## What OpenClaw expects today (v2026.9.6)
+
+Checked against the OpenClaw source at tag `v2026.9.6` and its docs on
+2026-09-25. OpenClaw marks every plugin API as experimental, so re-check before
+building anything.
+
+- **Memory lives in Markdown files**, which are the source of truth: `MEMORY.md`,
+  `USER.md`, daily notes in `memory/YYYY-MM-DD.md` in the agent workspace. The
+  memory engine is an index over them
+  ([concepts/memory](https://docs.openclaw.ai/concepts/memory)).
+- **A memory plugin is selected with `plugins.slots.memory: "<plugin-id>"`**
+  (default `memory-core`), its settings under
+  `plugins.entries.<plugin-id>.config`, validated against the plugin's own
+  schema ([gateway/config-extensions](https://docs.openclaw.ai/gateway/config-extensions)).
+  Memory search settings are under `memory.search`
+  ([reference/memory-config](https://docs.openclaw.ai/reference/memory-config)).
+- **A plugin needs** an `openclaw.plugin.json` manifest with `id`,
+  `configSchema`, `"kind": "memory"` and every tool listed in `contracts.tools`
+  ([plugins/manifest](https://docs.openclaw.ai/plugins/manifest)); a
+  `package.json` with `openclaw.extensions`, `openclaw.compat.pluginApi` and an
+  `openclaw` peer dependency; and an entry built with `definePluginEntry`.
+- **Two ways to integrate** (both exist upstream): tools only, as
+  `memory-lancedb` does (`api.registerTool`), or a full memory engine, as
+  `memory-core` does, through `api.registerMemoryCapability({ runtime, ... })`,
+  whose runtime returns a `MemorySearchManager` implementing `search`,
+  `readFile` (returning `status: "ok" | "not_found"`), `status`,
+  `probeEmbeddingAvailability` and `probeVectorAvailability`. Active Memory
+  expects `memory_search` and `memory_get` tools
+  ([plugins/sdk-overview/memory-and-context](https://docs.openclaw.ai/plugins/sdk-overview/memory-and-context)).
+- **Embeddings come from OpenClaw's providers** (`memory.search.provider`), or a
+  plugin registers one with `api.registerEmbeddingProvider`.
+- **Native code**: plugin installs run with `--ignore-scripts`, so a napi addon
+  has to ship as prebuilt per-platform packages (the pattern `memory-lancedb`
+  uses for LanceDB), loaded lazily
+  ([plugins/dependency-resolution](https://docs.openclaw.ai/plugins/dependency-resolution)).
+- **Distribution**: `openclaw plugins install` from npm or ClawHub; a first
+  install from an arbitrary source needs explicit review, and community ClawHub
+  packages go through a security audit.
+- **Churn to plan for**: the memory SDK was reshaped in 2026 (separate
+  registration functions merged into `registerMemoryCapability`;
+  `registerMemoryEmbeddingProvider` removed on 2026-08-21), and further SDK
+  surfaces become eligible for removal on 2026-10-01
+  ([plugins/sdk-migration/removal-timeline](https://docs.openclaw.ai/plugins/sdk-migration/removal-timeline)).
+
+## What this repository has
+
+Building blocks, usable as a library today, but not an OpenClaw plugin:
+
+- `clawhdf5_agent::openclaw::ClawhdfBackend` — a Markdown-oriented backend over
+  `HDF5Memory`: ingest Markdown by section, hybrid search with re-ranking and
+  confidence rejection, read back by path, export. Gaps a plugin would have to
+  close: `write`/`ingest_markdown` store no embeddings (search is keyword-only
+  for that content unless records are saved with `save_entry`), re-ingesting
+  appends rather than replaces, there is no delete, `line_range` is never set,
+  and export rewrites every heading as `##`.
+- `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a
+  TypeScript wrapper. **Not published, not built or tested in CI, and known to
+  be broken**; see `docs/known-issues.md`.

packages/clawhdf5-node/README.md

@@ -5,20 +5,20 @@ HDF5-backed agent memory system with hippocampal consolidation.
 
 Built with [napi-rs](https://napi.rs).
 
+> **Status: unpublished and known to be broken.** This package is not on npm,
+> no binaries are built, nothing in CI builds or tests it, and the wrapper
+> reads field names the native layer does not produce. It is not an OpenClaw
+> plugin. See [known issues](../../docs/known-issues.md) and
+> [docs/openclaw.md](../../docs/openclaw.md) before using it.
+
 ## Installation
 
+Not published. Building from source needs `@napi-rs/cli`:
+
 ```bash
-npm install @redclaw/clawhdf5
+npm install && npm run build
 ```
 
-Pre-built binaries are published for:
-
-| Platform | Architecture |
-|----------|-------------|
-| Linux (glibc) | x64, aarch64 |
-| macOS | x64, aarch64 (Apple Silicon) |
-| Windows | x64 |
-
 ## Quick start
 
 ```ts

packages/clawhdf5-node/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@redclaw/clawhdf5",
-  "version": "2.6.0",
-  "description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
+  "version": "2.7.0",
+  "description": "Node.js bindings for clawhdf5 \u2014 HDF5-backed agent memory with hippocampal consolidation",
   "main": "index.js",
   "types": "index.d.ts",
   "license": "MIT",
@@ -14,8 +14,7 @@
     "memory",
     "hdf5",
     "vector-search",
-    "embedding",
-    "openclaw"
+    "embedding"
   ],
   "napi": {
     "name": "clawhdf5",
@@ -51,5 +50,6 @@
   },
   "engines": {
     "node": ">= 16"
-  }
+  },
+  "private": true
 }

scripts/ci-test.sh

@@ -77,6 +77,50 @@ run_step "cargo clippy (ann parallel)" cargo clippy \
     --features parallel \
     -- -D warnings
 
+# zlib-ng is opt-in (`fast-deflate`; the default is pure-Rust zlib-rs), so
+# nothing above builds it. Keep it compiling and passing.
+run_step "cargo clippy (fast-deflate / zlib-ng)" cargo clippy \
+    -p clawhdf5-format -p clawhdf5-filters -p clawhdf5 \
+    --all-targets \
+    --features clawhdf5-format/fast-deflate,clawhdf5-filters/fast-deflate \
+    -- -D warnings
+
+# The README promises that the core crates build no C by default. Hold it to
+# that: fail if a crate that compiles C (a *-sys crate, cc or cmake) enters the
+# default dependency tree of any of them. clawhdf5-migrate (bundled SQLite),
+# clawhdf5-napi (Node) and clawhdf5-gpu (graphics drivers) are exempt.
+no_c_in_default_build() {
+    local crate found=0
+    for crate in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \
+        clawhdf5-agent clawhdf5-ann clawhdf5-accel clawhdf5-netcdf4 clawhdf5-cli; do
+        local c_deps
+        c_deps=$(cargo tree -q -p "$crate" -e normal,build --prefix none \
+            | grep -E '^([a-z0-9_-]+-sys|cc|cmake) v' | sort -u)
+        if [ -n "$c_deps" ]; then
+            echo "$crate pulls in C by default:"
+            echo "$c_deps" | sed 's/^/    /'
+            found=1
+        fi
+    done
+    return $found
+}
+run_step "no C in the default build (core crates)" no_c_in_default_build
+
+# The workspace declares a minimum Rust version (rust-version in Cargo.toml);
+# check that it really builds there, so the README badge and the manifests
+# cannot drift from the truth. Separate target dir: a different toolchain
+# would otherwise invalidate the main build.
+msrv_check() {
+    local msrv
+    msrv=$(sed -n 's/^rust-version = "\(.*\)"/\1/p' "$SCRIPT_DIR/../Cargo.toml")
+    [ -n "$msrv" ] || { echo "no rust-version in Cargo.toml"; return 1; }
+    rustup toolchain install "$msrv" --profile minimal >/dev/null || return 1
+    echo "checking with Rust $msrv"
+    CARGO_TARGET_DIR="$SCRIPT_DIR/../target/msrv" cargo "+$msrv" check \
+        --workspace --exclude clawhdf5-py --all-targets
+}
+run_step "MSRV check" msrv_check
+
 # 4. Tests (exclude clawhdf5-py)
 run_step "cargo test" cargo test \
     --workspace \
@@ -90,6 +134,10 @@ run_step "cargo test (ann parallel)" cargo test \
     -p clawhdf5-ann \
     --features parallel
 
+run_step "cargo test (fast-deflate / zlib-ng)" cargo test \
+    -p clawhdf5-format -p clawhdf5-filters -p clawhdf5 \
+    --features clawhdf5-format/fast-deflate,clawhdf5-filters/fast-deflate
+
 # 5. Python interop suites. The h5py writer tests are #[ignore]d so a plain
 #    `cargo test` stays hermetic; run them explicitly here.
 #    On a PEP 668 "externally managed" system h5py can only live in a