feat(agent): MemoryConfig::float16 stores half-precision embeddings

The setting was persisted in /meta and otherwise ignored: embeddings were always written as f32. It now does what it says. clawhdf5-format: - `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16), rounding to nearest-even, and `make_f16_type`. - `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one implementation the writer, the reader and the agent all use. Checked against the `half` crate on 16.7M f32 values and round-trips all 65536 half values; the h5py interop tests confirm the rounding matches numpy's bit for bit (4020 values incl. ties, subnormals, overflow). - Reading little-endian float16 as f32 has a fast path. clawhdf5-agent: - A float16 store writes /memory/embeddings as half precision, and `MemoryCache::half_precision` rounds each embedding as it enters the cache (save, update, WAL replay, and on load of a store still f32 on disk), so memory and file agree bit for bit and a store searches the same before and after a reopen (tested). - Values beyond +-65504 are refused with the new `MemoryError::InvalidEntry` rather than stored as infinity, on every save path; batches are all or nothing, and a rejected ephemeral entry stays in the ephemeral tier. Breaking for exhaustive matches. - CLI: `create --float16`. Off by default. Measured on tank, 384-dim, six runs alternating order, medians (search_harness --float16-study --full): at 100K the file goes from 154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms; vector recall@10 against an exact scan and hybrid_search latency do not change. At 10K open is 3 ms slower. Also a test that h5py opens a whole agent store, f32 and float16, and decodes every dataset. Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG (including the h5py interop fixes in the previous commit), CLAUDE.md. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
fix(format): files we write now open in h5py and libhdf5
2026-09-24 12:00:38 -05:00 · 2026-09-24 12:00:26 -05:00 · 2026-09-23 16:29:33 +00:00 · 2026-09-23 11:05:26 -05:00 · 2026-09-23 11:05:22 -05:00 · 2026-09-23 11:05:10 -05:00
49 changed files with 2421 additions and 310 deletions
@@ -9,15 +9,15 @@ jobs:
    runs-on: ubuntu-latest
    container: rust:latest
    steps:
-      - uses: actions/checkout@v4
+      # Plain git rather than actions/checkout: that is a JavaScript action,
-      - name: Cache cargo registry/target
+      # and rust:latest has no `node`, so it failed with exit 127 before any
-        uses: actions/cache@v4
+      # code was built — on every push. actions/cache went for the same reason.
-        with:
+      - name: Check out
-          path: |
+        run: |
-            ~/.cargo/registry
+          git init -q .
-            ~/.cargo/git
+          git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
-            target
+          for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
-          key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+          git checkout -q FETCH_HEAD
      - name: Install rustfmt & clippy components
        run: rustup component add rustfmt clippy
      - name: Install thumbv7em-none-eabihf target
@@ -28,7 +28,10 @@ jobs:
        # dependency a failure (CLAWHDF5_REQUIRE_INTEROP below).
        run: |
          apt-get update
-          apt-get install -y --no-install-recommends python3 python3-venv
+          # cmake builds libz-ng-sys for the opt-in `fast-deflate` (zlib-ng)
          # steps in ci-test.sh; rust:latest does not ship it. The default
          # build (pure-Rust zlib-rs) does not need it.
          apt-get install -y --no-install-recommends python3 python3-venv cmake
          python3 -m venv /opt/interop
          /opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray
          echo "/opt/interop/bin" >> "$GITHUB_PATH"
@@ -44,3 +47,40 @@ jobs:
          CLAWHDF5_PYTHON: /opt/interop/bin/python
          CLAWHDF5_REQUIRE_INTEROP: "1"
        run: bash scripts/ci-test.sh
  test-arm64:
    # The aarch64 kernels in clawhdf5-accel — NEON `dot_i8`, including the
    # SDOT path, and the f32 NEON kernels — are cfg'd out on x86, so the job
    # above never compiles, lints or tests them.
    #
    # `linux_arm64` is served by two runners that execute differently:
    # vision-01 runs steps on the host (Rust already installed) and vision-02
    # runs them in docker.gitea.com/runner-images. So the steps work in both:
    # no `container:`, no JavaScript actions (they are fetched from GitHub,
    # which not every runner reliably reaches), and an explicit `+stable`
    # toolchain rather than whatever a host happens to default to.
    runs-on: linux_arm64
    env:
      CARGO_NET_RETRY: "10"
      CARGO_TERM_COLOR: always
    steps:
      - name: Check out
        run: |
          git init -q .
          git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
          for i in 1 2 3; do git fetch -q --depth 1 origin "${GITHUB_SHA}" && break; sleep 5; done
          git checkout -q FETCH_HEAD
      - name: Rust stable
        run: |
          export PATH="$HOME/.cargo/bin:$PATH"
          command -v rustup >/dev/null || curl -sSf --retry 5 https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain none
          rustup toolchain install stable --profile minimal --component clippy
          echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
      - name: Confirm aarch64
        run: |
          test "$(uname -m)" = aarch64
          if grep -q asimddp /proc/cpuinfo; then echo "dot-product extension present: SDOT kernel runs"; else echo "no dot-product extension: plain NEON kernel runs"; fi
      - name: Clippy (aarch64 kernels)
        run: cargo +stable clippy -p clawhdf5-accel --all-targets -- -D warnings
      - name: Test
        run: cargo +stable test -p clawhdf5-accel -p clawhdf5-ann -p clawhdf5-format
@@ -108,11 +108,37 @@ second**, builds **1.8x faster**, and holds a quarter of the vectors. (Compare
 only at equal `ef`: with re-scoring the harness raises `ef` to at least the
 candidate pool, so the `ef = 16` and `ef = 32` rows are not like-for-like.)
-It is still **off by default**, for portability rather than performance: the
+#### On ARM (Raspberry Pi 5, Cortex-A76)
-int8 kernel is AVX2-only, and on aarch64 — including `clawhdf5-android` — it
+
-falls back to the scalar loop, where the original trade still applies. A NEON
+`dot_i8` has two aarch64 kernels: `SDOT` for CPUs with the ARMv8.2
-kernel would remove that caveat. On an x86-64 deployment, turning it on is a
+dot-product extension (Cortex-A76 and later, Neoverse-N1, all Apple Silicon)
-win on every axis measured.
+and plain NEON (`vmull_s8` + `vpadalq_s16`) otherwise. Medians of three runs
 at N = 100 000, ef = 64, recall@10 0.9940 in every int8 row against f32's
 0.9945:
 | int8 kernel | build | QPS | vs f32 |
 |---|---:|---:|---:|
 | *(f32 baseline)* | 33 413 ms | 6 164 | 1.00x |
 | scalar (what v2.7.0 shipped) | 18 950 ms | ~6 190 | 1.00x |
 | plain NEON | ~17 000 ms | 6 640 | 1.08x |
 | **SDOT** | **14 464 ms** | **7 267** | **1.18x** |
 These are Pi 5 numbers, not "ARM" numbers: a Pi has far less memory bandwidth
 and cache than an Apple M-series or a flagship phone, so the ratios will move
 on other hardware. The plain-NEON row is that code on an A76 with `SDOT`
 disabled, not a measurement of a pre-A76 core.
 **A correction.** Until this was measured, this section said aarch64 "falls
 back to the scalar loop, where the original trade still applies" — that is,
 that quantised search was ~13% slower than f32 on ARM. That was extrapolated
 from x86 and it was wrong. On x86-64 the portable baseline is SSE2 while the
 f32 kernels are hand-written AVX2, so scalar int8 lost; on aarch64 NEON *is*
 the baseline, the compiler vectorises the scalar loop well, and scalar int8
 already matched f32 for search while building 1.76x faster.
 So on every configuration measured — x86-64 AVX2, and Pi 5 with each of the
 three int8 kernels — the quantised index is at least as fast as f32 at equal
 recall, builds faster, and holds a quarter of the vectors.
 A measurement trap worth recording: the synthetic `clustered` generator in the
 `clawhdf5-ann` tests draws clusters far tighter than any real embedding, so
@@ -123,6 +149,40 @@ vectors, and recall is measured against brute-force ground truth rather than
 against the f32 index, whose own approximation errors a re-scored search is
 entitled to get right.
 ### float16 embedding storage (`MemoryConfig::float16`)
 Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D). The same clustered
 384-dim data in an `f32` store and a `float16` store, both with the default
 int8 index and Hebbian boosting off (so every query sees the same store).
 Recall is vector-only `hybrid_search` against an exact scan of the original
 `f32` vectors, 200 queries. Six runs, three with each store going first;
 medians. Nothing depended on the order.
 ```bash
 cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
 cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full --f16-first
 ```
 | N | embeddings | file MiB | checkpoint ms | open ms | recall@10 | top-10 overlap | hybrid p50 ms |
 |---:|---|---:|---:|---:|---:|---:|---:|
 | 1 000 | f32 | 1.6 | 8 | 1.6 | 1.0000 | | 0.072 |
 | 1 000 | float16 | 0.8 | 6 | 1.9 | 0.9980 | 0.9980 | 0.072 |
 | 10 000 | f32 | 15.4 | 66 | 15.2 | 1.0000 | | 0.495 |
 | 10 000 | float16 | 8.2 | 45 | 18.1 | 1.0000 | 1.0000 | 0.494 |
 | 100 000 | f32 | 154.0 | 752 | 299.8 | 0.9940 | | 4.676 |
 | 100 000 | float16 | **80.8** | **512** | **252.1** | 0.9990 | 0.9940 | 4.654 |
 The file is 48% smaller, checkpoints write less and open reads less. At
 small N opening is slightly slower (widening halves costs more than the I/O it
 saves: +3 ms at 10K). Recall does not move: half precision keeps about three
 significant digits, far finer than the gaps between neighbours on unit-length
 embeddings. Recall was identical in every run; the 0.999 against 0.994 at
 100K is two slightly different HNSW graphs, not an improvement to claim.
 The in-memory cache holds the half-rounded values, so the store searches the
 same before and after a reopen; RAM use is unchanged (the cache is still
 `f32`). What `float16` saves is disk, and the I/O that goes with it.
 ### Opening a store (`read_from_disk`)
 `HDF5Memory::open` memory-mapped the file, copied the whole mapping into a
@@ -1159,6 +1219,69 @@ cargo run --release --bin ephemeral_perf
 ---
 ## Deflate backend: zlib-rs vs zlib-ng
 Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D, 8C/16T). The default
 deflate backend is now **zlib-rs**, a pure-Rust port of zlib-ng; zlib-ng (C,
 built with cmake) was the default before and is still available as
 `fast-deflate`. Both builds were compiled once into separate target
 directories and run **alternately, three rounds each**; figures are medians.
 ```bash
 # zlib-rs (default)
 cargo bench -p clawhdf5-filters --bench deflate_bench
 cargo bench -p clawhdf5-bench --bench h5bench_write --features libhdf5-compare -- '^write_2d_chunked/'
 cargo run --release -p clawhdf5-bench --bin read_harness
 # zlib-ng: add --features fast-deflate (filters) or clawhdf5-format/fast-deflate (bench)
 ```
 | Workload | zlib-rs | zlib-ng | rs / ng |
 |---|---:|---:|---:|
 | HDF5 chunked write, deflate-6, 512×512 f32 | 1.458 ms | 1.484 ms | 0.98 |
 | HDF5 chunked write, deflate-6, 128×128 f32 | 157.6 µs | 152.8 µs | 1.03 |
 | HDF5 chunked write, deflate-6, 32×32 f32 | 62.9 µs | 60.2 µs | 1.05 |
 | HDF5 read, 64 MB chunked + deflate, full | 64.4 ms | 65.2 ms | 0.99 |
 | HDF5 read, 64×64 window (1 chunk) | 0.18 ms | 0.17 ms | 1.06 |
 | HDF5 read, 512×512 window (4–9 chunks) | 4.10 ms | 4.10 ms | 1.00 |
 | HDF5 read, one row / one column | 1.00 / 2.01 ms | 0.95 / 1.95 ms | 1.05 / 1.03 |
 | Raw inflate, 8 MB f64 | 5.92 ms | 6.06 ms | 0.98 |
 | Raw inflate, 1 MB sine | 82.8 µs | 68.6 µs | 1.21 |
 | Raw deflate-6, 8 MB f64 / 1 MB sine | 92.2 / 2.01 ms | 83.1 / 1.84 ms | 1.11 / 1.09 |
 Compressed output is **byte-identical** between the two at levels 1, 6 and 9
 on all three inputs, so files do not change size. libhdf5 1.14.6 took 51.4 ms
 for the 512×512 write in the same session (35× the zlib-rs figure).
 On the HDF5 paths zlib-rs is within 6% of zlib-ng everywhere, and ahead on the
 largest write. The raw codec loops show zlib-ng still slightly faster at
 compression (~10%), which chunked writes do not expose because encoding runs
 in parallel across chunks.
 **Two findings along the way.** The first measurement had zlib-rs 1.2–1.9×
 slower on single-chunk reads and 3.7× slower on a 1 MB inflate — slower even
 than miniz_oxide. Neither was zlib-rs's fault:
 1. **Runtime CPU detection was off.** zlib-rs needs its `std` feature to
   detect and use SIMD at runtime; flate2 turns it on through its default
   `runtime_detection` feature, which our `default-features = false` flate2
   dependency was disabling. With it, a 1 MB inflate goes 282 → 83 µs.
   `clawhdf5-format/zlib-rs` and `clawhdf5-filters/zlib-rs` now enable it.
 2. **The codec was fed through a 32 KiB buffer.** Both deflate paths used
   flate2's streaming `read::ZlibDecoder` / `write::ZlibEncoder`. A chunk's
   decompressed size is known, so they now hand the codec the whole input in
   one call, into an output buffer sized up front. Worth ~5% on chunked
   writes and ~10% on zlib-ng's 1 MB inflate. It also closed a hole: the
   streaming reader returned a truncated stream's bytes without an error, so
   a truncated chunk read back short; it is now an error.
 | 1 MB inflate, same build otherwise | zlib-rs | zlib-ng |
 |---|---:|---:|
 | streaming reader, no runtime detection | 284.1 µs | 76.7 µs |
 | one-shot, no runtime detection | 282.3 µs | 68.5 µs |
 | one-shot + runtime detection (shipped) | **82.8 µs** | **68.6 µs** |
 ---
 ## h5bench-Equivalent I/O Benchmarks
 Criterion harness mirroring h5bench serial workloads. clawhdf5 benchmarks dated 2026-07-01;
@@ -1,5 +1,142 @@
 # Changelog
 ## Unreleased
 ### Upgrade Notes
 - **Files written by clawhdf5 now open in h5py and libhdf5.** Every `f32`
  dataset we wrote — including every agent store's embeddings — was refused
  with "sign bit position out of bounds", and every empty dataset with
  "invalid dataset size". Both were write-side bugs present in every release;
  clawhdf5's own reader was unaffected. An agent store is rewritten in full at
  each checkpoint, so it becomes readable at its next checkpoint on this
  version; other files with `f32` or empty datasets need rewriting. Details in
  `docs/known-issues.md`.
 - **`MemoryConfig::float16` now does what it says.** It was persisted and
  otherwise ignored; embeddings were always stored as `f32`. A store created
  with it on now writes half-precision embeddings (48% smaller files) and
  rounds embeddings to half precision as they are saved. A store that already
  had `float16 = true` rounds its embeddings when next opened and writes them
  as `float16` at its next checkpoint. Off by default.
 - **Breaking:** `MemoryError` gained `InvalidEntry`, returned when a
  `float16` store is given an embedding value beyond ±65504. Exhaustive
  matches need the new arm.
 - **The default build no longer compiles any C.** Deflate now defaults to the
  pure-Rust zlib-rs instead of zlib-ng, so building the core crates needs
  neither cmake nor a C compiler. Speed on HDF5 reads and writes is within 6%
  of zlib-ng, and compressed output is byte-identical. To keep zlib-ng, enable
  `fast-deflate` (on `clawhdf5`, `clawhdf5-format` or `clawhdf5-filters`); it
  overrides zlib-rs wherever it is on.
 - **A truncated deflate chunk is now an error.** It used to read back short,
  with no error.
 - **Minimum supported Rust is 1.92**, now declared in every crate's
  `rust-version` and checked in CI.
 - **New stores use the int8 vector index by default.**
  `MemoryConfig::quantized_index` now defaults to `true`: a quarter of the
  index memory, builds 1.8x (x86-64) and 2.3x (Raspberry Pi 5) faster, and
  searches 1.63x and 1.18x faster at equal recall, measured on every
  configuration tested. **Existing stores are unaffected** — a store written
  with v2.6.0 or later keeps its persisted setting, and one written before the
  setting existed opens as `false` and keeps its f32 index. Set
  `quantized_index = false`, or pass `create --f32-index` to the CLI, to opt
  out. The CLI's `--quantized-index` is still accepted but is now a no-op.
 ### Interop
 - `clawhdf5-format`: **every `f32` dataset was unreadable by h5py and
  libhdf5.** The float datatype encoder hard-coded the sign bit's position to
  63, correct only for `f64`; libhdf5 validates it and refused the dataset. It
  is now derived from the type (15 / 31 / 63). Our reader ignores the field,
  and the interop suites only wrote `f64`, which is how it went unnoticed.
 - `clawhdf5-format`: **every empty dataset was unreadable by h5py and
  libhdf5.** It was written with a real address and zero bytes, which trips
  libhdf5's `addr + size <= addr` overflow check. An empty contiguous dataset
  now gets the undefined address, as libhdf5 writes it. This affected every
  agent store without sessions or a knowledge graph.
 - New interop tests: `f32` and `float16` datasets in both directions (our
  `float16` rounding matches numpy's bit for bit on 4 020 probe values,
  including ties, subnormals and the overflow boundary), and an agent store —
  `f32` and `float16` — opened by h5py with every dataset decoded.
 ### Storage
 - `clawhdf5-format`: **half-precision datasets.**
  `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy `float16`),
  rounding to nearest-even; `make_f16_type`, and `clawhdf5_format::float16`
  with the conversions, which are checked against the `half` crate on 16.7M
  values and round-trip all 65 536 half values. Reading `float16` as `f32`
  gained a little-endian fast path.
 - `clawhdf5-agent`: **`MemoryConfig::float16` stores embeddings as half
  precision.** At 100K x 384 the file goes from 154.0 to 80.8 MiB (−48%), a
  checkpoint from 752 to 512 ms and open from 300 to 252 ms, with the same
  vector recall@10 against an exact scan (0.999 vs 0.994) and the same
  `hybrid_search` latency; at 10K open is 3 ms slower. The cache rounds each
  embedding as it is saved, so memory and file agree bit for bit and a store
  returns the same results before and after a reopen (tested). Out-of-range
  values are refused with `MemoryError::InvalidEntry` rather than stored as
  infinity; batches are all or nothing. CLI: `create --float16`. See
  `BENCHMARKS.md`, "float16 embedding storage".
 ### Build
 - **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
  `clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
  (zlib-ng) is opt-in. No crate in the default dependency tree of the core
  crates compiles C, and `ci-test.sh` now fails if one appears. The facade's
  `fast-deflate` was on by default and is now off. See `BENCHMARKS.md`,
  "Deflate backend".
 - `zlib-rs` also enables flate2's `runtime_detection`. Without it zlib-rs has
  no `std`, cannot detect SIMD at runtime, and inflates 3.5x slower; the
  workspace builds flate2 with `default-features = false`, which had been
  switching it off.
 - `rust-version = "1.92"` for the whole workspace (the floor: `wgpu` requires
  it), and CI checks the workspace on exactly that toolchain.
 - CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
 ### Correctness
 - `clawhdf5-format`: **a truncated deflate chunk read back short, with no
  error.** The deflate filter used flate2's streaming reader, which returns the
  bytes it has when the input runs out before the end-of-stream marker. It now
  decodes in one pass into a buffer sized to the chunk and reports a
  truncated stream as `DecompressionError`. Same fix in `clawhdf5-filters`,
  where output longer than the stated size was also silently cut off; it is
  now an error.
 ### Defaults
 - `clawhdf5-agent`: `MemoryConfig::quantized_index` defaults to `true` for new
  stores. The reason it had been off — that int8 search was slower on ARM —
  did not survive measurement (see Corrections). Stores that predate the
  setting still load it as `false`, so reopening one never changes how its
  index is held; a store written by the v2.5.0 CLI is now a test fixture that
  guards exactly that, and the test fails if the load default is changed.
 - `clawhdf5-cli`: `create --f32-index` opts out. `create` used to assign
  `--quantized-index` straight into the config, which under the new default
  would have forced every CLI-created store back to f32 unless the caller
  knew to ask; it now only ever switches the default off.
 ### Performance
 - `clawhdf5-format`, `clawhdf5-filters`: both deflate paths hand the codec the
  whole chunk in one call, into a buffer allocated once, instead of streaming
  it through a 32 KiB buffer: about 5% on chunked writes and 10% on zlib-ng's
  1 MB inflate.
 - `clawhdf5-accel`: **`dot_i8` has aarch64 kernels** — `SDOT` for CPUs with
  the ARMv8.2 dot-product extension (Cortex-A76 and later, Neoverse-N1, every
  Apple Silicon generation) and plain NEON (`vmull_s8` + `vpadalq_s16`) for
  the rest, selected at runtime. `SDOT` is issued through inline assembly,
  because the `vdotq_s32` intrinsic is still behind the unstable
  `stdarch_neon_dotprod` feature. On a Raspberry Pi 5 at N = 100 000 and
  equal recall, the quantised index answers **1.18x the queries per second**
  of f32 (7 267 vs 6 164) and builds **2.3x faster** (14 464 vs 33 413 ms).
  Both kernels are tested bit-for-bit against scalar on real hardware, each
  explicitly — dispatch only ever takes one path on a given CPU, so testing
  through it alone would have left the plain-NEON fallback unexercised on any
  machine with `SDOT`.
 ### Corrections
 - The v2.7.0 entry for `dot_i8` said `quantized_index` stayed off by default
  because "aarch64 falls back to the scalar loop", implying the ~13% search
  penalty measured on x86 applied on ARM too. It did not. That figure came
  from scalar int8 against hand-written AVX2 f32 kernels on x86, whose
  portable baseline is SSE2; on aarch64 NEON is the baseline, and measured on
  a Pi 5 the scalar int8 loop already matched f32 for search while building
  1.76x faster. The claim was extrapolated rather than measured.
 ## v2.7.0 (2026-09-20)
 ### Upgrade Notes
@@ -19,7 +19,7 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
 | `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
 | `clawhdf5-gpu` | GPU-accelerated I/O via wgpu (hand-written WGSL compute shaders) |
 | `clawhdf5-accel` | CPU SIMD acceleration path |
-| `clawhdf5-migrate` | Schema migration engine |
+| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
 | `clawhdf5-android` | Android JNI bindings |
 | `clawhdf5-cli` | Command-line interface |
 | `clawhdf5-napi` | Node.js native addon bindings |
@@ -27,7 +27,12 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
 | `clawhdf5-bench` | Benchmark suite |
 ## Key Features
- Zero-dependency HDF5 read/write (no libhdf5 C library required)
+- Zero-C-dependency HDF5 read/write: no libhdf5, and deflate defaults to
  pure-Rust zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake).
  `ci-test.sh` fails if a C-building crate enters the core crates' default
  tree. flate2 must keep `runtime_detection` with zlib-rs — without it zlib-rs
  loses SIMD and inflates 3.5x slower. MSRV is 1.92 (`rust-version`, checked
  in CI).
 - HNSW vector index for semantic similarity search over agent memories — the
  `clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses
  the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
@@ -39,15 +44,20 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
  (plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its
  graph is saved to `<store>.h5.ann` at each checkpoint and reloaded by `open()`
  (tied to the checkpoint by a generation id; stale/damaged sidecars are
-  ignored and the index rebuilt). `MemoryConfig::quantized_index` (off by
+  ignored and the index rebuilt). `MemoryConfig::quantized_index` (**on by
-  default, persisted) stores the index's own copy of the embeddings as `i8`,
+  default** for new stores, persisted; stores predating the setting load as
  `false` and keep their f32 index — guarded by
  `tests/fixtures/store_v2_5_0.h5`; CLI opt-out is `create --f32-index`)
  stores the index's own copy of the embeddings as `i8`,
  which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors
  at 100K); because quantised distances are approximate and `ef` cannot
  compensate, the query path then re-scores the candidate pool against the
-  exact embeddings, which holds recall at the f32 index's level. On AVX2 it is
+  exact embeddings, which holds recall at the f32 index's level. It is also
-  also 1.63x the QPS and 1.8x the build speed (`clawhdf5_accel::dot_i8`); it
+  faster at equal recall: 1.63x the QPS on x86-64 (AVX2) and 1.18x on a
-  stays off by default only because that kernel is AVX2-only and aarch64 falls
+  Raspberry Pi 5 (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm since
-  back to scalar. `hybrid_search` keeps one incremental BM25
+  the intrinsic is unstable; plain NEON on pre-dotprod cores). The aarch64
  code is `cfg`'d out on x86, so x86 CI never compiles or lints it — test it
  on real ARM (`rpivision02`, 10.0.2.3, is a Pi 5). `hybrid_search` keeps one incremental BM25
  index for the life of the store and never writes the store: Hebbian
  activation boosts are persisted by the next checkpoint (or on drop), not per
  query. Measure any search-path change with
@@ -77,8 +87,18 @@ Cargo workspace with 16 crates under `crates/` (plus `libaec-sys`, an internal F
  `export` do). An unreadable WAL (torn header, bad magic) is quarantined to
  `<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
  unknown *newer* version still fails and is left untouched.
- `MemoryConfig::compression` uses deflate by default; enable the agent's
+- `MemoryConfig::float16` (off by default, persisted; CLI `create --float16`)
-  `zstd` feature to compress embeddings with Zstd instead (links libzstd).
+  writes `/memory/embeddings` as IEEE half precision (48% smaller file at
  100K, same recall). `MemoryCache::half_precision` rounds each embedding as
  it enters the cache (push, update, WAL replay, and on load of a store still
  `f32` on disk), so memory and file agree bit for bit; the conversions live
  in `clawhdf5_format::float16` and must stay the single implementation.
  Values beyond ±65504 are `MemoryError::InvalidEntry`. Interop: every file
  must open in h5py — `f32` datasets and empty datasets did not until
  2026-09-23 (see `docs/known-issues.md`); the agent's `h5py_interop` test
  guards a whole store.
 - `MemoryConfig::compression` is off by default; when on, embeddings are
  deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
 - `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
  default) recomputes a dataset's SHA-256 and compares it against the
  `_provenance_sha256` attribute written automatically on save when
@@ -111,6 +131,24 @@ cargo build --release
 cargo test --workspace
 ```
 ### CI
 `.gitea/workflows/ci.yml` has two jobs, both green as of 2026-09-22:
 - **`test`** (`ubuntu-latest`, in `rust:latest`) runs `scripts/ci-test.sh` with
  the h5py/netCDF4 interop suites required (`CLAWHDF5_REQUIRE_INTEROP=1`).
  Served by the `tank` and `architect` runners.
 - **`test-arm64`** (`linux_arm64`) lints and tests the aarch64 code — the NEON
  kernels are `cfg`'d out on x86, so this is the only place they are built.
  Served by `vision-01` (host mode) and `vision-02` (Docker), so steps must
  work in both.
 Keep workflows free of JavaScript actions (`actions/checkout`, `actions/cache`,
 …): `rust:latest` has no `node`, and not every runner reaches GitHub, where
 they are fetched from. Check out with plain `git` instead. The `test` job
 installs `cmake` for the opt-in `fast-deflate` (zlib-ng) steps; the default
 build needs no C toolchain, so `test-arm64` does not.
 All runners are on `gitea-runner` 3.5.0, from `docker.gitea.com/act_runner`
 — `gitea/act_runner:latest` on Docker Hub is frozen at 0.6.1.
 ### CLI
 ```bash
 cargo run -p clawhdf5-cli -- --help
@@ -23,6 +23,9 @@ resolver = "2"
 [workspace.package]
 version = "2.7.0"
 edition = "2024"
 # Oldest toolchain that builds the whole workspace; CI checks it. wgpu (in
 # clawhdf5-gpu) requires 1.92.
 rust-version = "1.92"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -3,24 +3,98 @@
 **The memory layer AI agents deserve. One file. Pure Rust. Zero C dependencies.**
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
-[![Rust](https://img.shields.io/badge/rust-1.75%2B-orange.svg)](https://www.rust-lang.org)
+[![Rust](https://img.shields.io/badge/rust-1.92%2B-orange.svg)](https://www.rust-lang.org)
-[![Tests](https://img.shields.io/badge/tests-1650%2B%20passing-brightgreen.svg)](#performance)
+[![Tests](https://img.shields.io/badge/tests-1850%2B-brightgreen.svg)](#building)
-[![LongMemEval](https://img.shields.io/badge/LongMemEval%20oracle-Turn--Level%20Hit@5%2084%25%20BM25--only-blue.svg)](BENCHMARKS.md#longmemeval-results)
+[![LongMemEval](https://img.shields.io/badge/LongMemEval__s-Turn--Level%20Hit@5%2081.4%25%20hybrid-blue.svg)](BENCHMARKS.md#longmemeval-results)
-[![Footprint](https://img.shields.io/badge/footprint-6.5%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint)
+[![Footprint](https://img.shields.io/badge/on--disk-1.7%20KB%2Frecord-lightgrey.svg)](BENCHMARKS.md#memory-footprint-1)
-ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, cryptographically verifiable memory — all stored in a single portable file.
+ClawHDF5 is a pure-Rust HDF5 implementation combined with a research-grade agent memory engine. It gives AI agents persistent, searchable, integrity-checked memory — all stored in a single portable file.
 > **Two things live here:**
 > - **A general-purpose, pure-Rust HDF5 library** — zero C dependencies, NetCDF-4 support, SIMD/GPU acceleration. See the **[Crate Map](#crate-map)** and **[BENCHMARKS.md](BENCHMARKS.md)** for the libhdf5 head-to-head numbers.
 > - **An agent memory layer built on top of it** — vector search, knowledge graph, hippocampal-style consolidation, in `clawhdf5-agent`.
-```
+The crates are not on crates.io yet, so depend on them from git:
-cargo add clawhdf5                          # core HDF5 read/write, no agent layer
+
-cargo add clawhdf5-agent --features agent   # + agent memory layer
+```toml
 [dependencies]
 clawhdf5       = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # core HDF5 read/write
 clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }  # + agent memory layer
 ```
 > **C dependencies, precisely:** the core crates (`clawhdf5`, `clawhdf5-agent`,
 > `-format`, `-io`, `-filters`, `-ann`, `-accel`, `-netcdf4`, `-cli`) build no C
 > code by default — no libhdf5, and deflate is the pure-Rust
 > [zlib-rs](https://github.com/trifectatechfoundation/zlib-rs), which matches
 > zlib-ng on HDF5 reads and writes and produces byte-identical output
 > ([BENCHMARKS.md § Deflate backend](BENCHMARKS.md#deflate-backend-zlib-rs-vs-zlib-ng)).
 > CI fails if a C-building crate enters their default dependency tree. C comes
 > in only when you ask for it: `fast-deflate` (zlib-ng, needs cmake), `zstd`,
 > `szip`, the BLAS backends, `clawhdf5-migrate` (bundled SQLite) and the
 > Node.js bindings.
 > **New here?** Start with the **[Quickstart Guide](docs/QUICKSTART.md)** · See **[Use Cases](docs/USE_CASES.md)** · Read **[Benchmarks](BENCHMARKS.md)**
 ## What's new (v2.2 → v2.7, and unreleased)
 Five releases in September 2026. Details, including upgrade notes and every
 breaking change, are in [CHANGELOG.md](CHANGELOG.md).
 **HDF5 correctness (read these if you read files with an earlier release)**
 - **Extensible Array chunk indexes returned wrong data** past the 36th chunk —
  any dataset with one unlimited dimension. Silent: plausible numbers from the
  wrong chunks. Fixed in v2.7.0; re-read affected data.
 - Fixed and Extensible Array checksums are now verified, so a corrupt chunk
  index is `ChecksumMismatch` instead of wrong data (v2.7.0).
 - Compound datatypes written with default libver bounds (plain
  `h5py.File(path, 'w')`) were mis-parsed; HDF5 2.0 compound v5 and native
  complex (class 11) types now parse (v2.2.0–v2.3.0).
 - Committed datatypes, fill values, soft links and `H5T_STD_REF` references now
  read correctly; external links and external raw data are explicit errors;
  `attrs()` no longer silently drops attributes (v2.3.0–v2.5.0).
 - Datasets indexed by a version-2 B-tree now read (v2.5.0).
 **Security and robustness**
 - A crafted file could abort any reader via B-tree v2 recursion or explode it
  via shared children; both are now fast errors (v2.7.0).
 - Virtual-dataset source paths are confined to the file's directory; chunked
  reads use overflow-checked sizes and fallible allocation, and the facade
  writes files atomically (v2.3.0).
 - Agent store: single-writer lock plus `open_read_only`; a crash between
  checkpoint and WAL truncate no longer duplicates entries; unreadable WALs are
  quarantined instead of blocking `open()` (v2.3.0).
 **Search quality and speed**
 - HNSW neighbour selection now uses the paper's diversity heuristic: recall@10
  at 100K went from 0.31 to 0.98 (v2.4.0).
 - `hybrid_search` is 79–190× faster than v2.3.0 (p50 0.07 ms at 1K, 4.65 ms at
  100K). It no longer rebuilds BM25 or rewrites the store per query, and the
  HNSW graph is persisted (v2.4.0).
 - Default fusion weights are now the measured 0.4 / 0.6 (v2.5.0). Re-ranking had
  been discarding the retrieval score, costing the OpenClaw backend 40.6pp of
  Hit@1; fixed in v2.6.0.
 - Selection reads decode only the chunks they touch (a 64×64 window: 105 ms to
  0.39 ms), and full reads are 1.2–1.9× faster (v2.5.0).
 **Memory**
 - A loaded store holds ~30% less (embeddings stored once, v2.6.0), and the
  int8 HNSW index, **on by default for new stores** (unreleased), brings a
  100K × 384 store to 1.74× the raw vectors. At equal recall it is also faster
  than `f32`: 1.63× QPS on AVX2, 1.18× on a Raspberry Pi 5 (NEON `SDOT`).
 **Interop (unreleased)**
 - **Files we write now open in h5py and libhdf5.** Every `f32` dataset —
  including every agent store's embeddings — and every empty dataset was
  refused by libhdf5. Both were write-side bugs in every release; agent stores
  fix themselves at their next checkpoint. See
  [docs/known-issues.md](docs/known-issues.md).
 - `MemoryConfig::float16` now stores half-precision embeddings (it was
  ignored): 48% smaller files at the same recall.
 **Tooling**
 - CI now runs the h5py/netCDF4 interop suites for real (they had been skipping
  silently) and runs an aarch64 job for the NEON kernels.
 ---
 ## Why ClawhDF5?
@@ -35,7 +109,7 @@ Every AI agent needs memory. Today that means scattered Markdown files, SQLite d
 | Memory consolidation | Manual pruning | Hippocampal-inspired automatic tiers |
 | Temporal queries | Custom code | Native temporal index (716ns) |
 | Multi-modal | Multiple stores | Unified cross-modal search |
-| Security | Hope for the best | Provenance tracking + anomaly detection |
+| Integrity | Hope for the best | Chained-CRC WAL, checksummed chunk indexes, write-anomaly alerts, opt-in SHA-256 dataset provenance |
 | Portability | Config + DB + files | **One `.h5` file. Copy it anywhere.** |
 ---
@@ -58,8 +132,28 @@ Figures below are from an independent reproduction run on a second machine (AMD
 | Sequential read (100K f32) | 23.3 µs | 63.6 µs | **2.7×** |
 | Sequential write (100K f32) | 210 µs | 189 µs | **≈ tie** |
 The chunked-write row was re-measured on the same machine on 2026-09-23, after
 the default deflate backend became pure-Rust zlib-rs: 1.46 ms against
 libhdf5's 51.4 ms (**35×**), and 1.48 ms with zlib-ng. libhdf5's own time on
 that machine moved from 65.0 to 51.4 ms between the two dates, which is most
 of the difference from 45×; compare same-day numbers only.
 ### Vector Search
 **HNSW (the default backend for `hybrid_search`)** — `search_harness`, clustered
 384-dim data, M = 16, ef_construction = 64, recall measured against an exact scan.
 See [BENCHMARKS.md § Search harness](BENCHMARKS.md#search-harness-baseline-v230)
 and [§ Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index):
 | N = 100K, ef = 64 | recall@10 | QPS | build |
 |---|---:|---:|---:|
 | `f32` index | 0.9945 | 13 399 | 3.2 s |
 | `i8` index + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
 Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was 0.31.
 **Brute-force and IVF paths** (Criterion, i7-12650H):
 | Scale | Flat | IVF (nprobe=10) | IVF-PQ | vs MemX¹ |
 |-------|------|-----------------|--------|----------|
 | 1K | **54 µs** | — | — | — |
@@ -75,7 +169,7 @@ Figures below are from an independent reproduction run on a second machine (AMD
 | Operation | Latency | Scale |
 |-----------|---------|-------|
-| Hybrid search (RRF) | **222 µs** | 1K records |
+| Hybrid search (`HDF5Memory::hybrid_search`, p50) | **70 µs** / 0.49 ms / 4.65 ms | 1K / 10K / 100K records |
 | BM25 keyword search | **67 µs** | 1K records |
 | Knowledge graph BFS | **24 µs** | 1K entities |
 | Spreading activation | **17 µs** | 100 entities |
@@ -115,13 +209,17 @@ declaration:
 Hybrid is the strongest configuration, which is what running two retrieval stages
 is for. The weights matter more than the stages: a sweep of `vector_weight` from
-0.0 to 1.0 found the long-standing `0.7/0.3` default is **strictly dominated** by
+0.0 to 1.0 found the old `0.7/0.3` default is **strictly dominated** by
-`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Use
+`0.4/0.6` — better on Hit@1, Hit@5, Hit@10 and MRR at both granularities. Since
-`0.4/0.6`, or `0.3/0.7` if rank-1 precision matters most. See
+v2.5.0 `0.4/0.6` is the default (`hybrid::DEFAULT_FUSION`, used by
-[BENCHMARKS.md § Weight sweep](BENCHMARKS.md#longmemeval-results).
+`unified_search`, `hybrid_search_with` and the OpenClaw backend); callers that
 pass weights to `hybrid_search` explicitly choose their own. Use `0.3/0.7` if
 rank-1 precision matters most. Reciprocal rank fusion is selectable
 (`hybrid::Fusion::Rrf`) but measured worse than the weighted sum. See
 [BENCHMARKS.md § Weight sweep](BENCHMARKS.md#weight-sweep--full-haystack-n500).
-Vector embeddings require `--features embeddings`; without it the vector stage is
+The benchmark's vector stage requires `clawhdf5-bench`'s `embeddings` feature
-inert and only the BM25 row is produced, which is what every previously published
+(real MiniLM embeddings); without it the vector stage is inert and only the BM25 row is produced, which is what every previously published
 number here measured.
 On the easier `longmemeval_oracle` variant (evidence sessions only) the same
@@ -146,19 +244,40 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 ### Memory Footprint
-| Records | File Size | Bytes/Record | With Compression |
+**On disk** — 384-dim embeddings, 200-char text
-|---------|-----------|--------------|------------------|
+([BENCHMARKS.md § Memory Footprint](BENCHMARKS.md#memory-footprint-1)):
-| 1K | ~6.5 MB | ~6.5 KB | ~2.1 MB (3.1x) |
+
-| 10K | ~65 MB | ~6.5 KB | ~21 MB (3.1x) |
+| Records | File Size | Bytes/Record | Gzip-6 compressed |
-| 100K | ~645 MB | ~6.5 KB | ~208 MB (3.1x) |
+|---------|-----------|--------------|-------------------|
 | 1K | 1.7 MB | 1.8 KB | 277 KB (6.1x) |
 | 10K | 17.0 MB | 1.7 KB | 2.7 MB (6.2x) |
 | 100K | 169.8 MB | 1.7 KB | 26.9 MB (6.2x) |
 With `MemoryConfig::float16` the embeddings take half the space: an agent
 store of 100K × 384 records is 80.8 MiB instead of 154.0.
 **In memory** — a store reopened from disk, 384-dim `f32`, measured with a
 counting allocator ([BENCHMARKS.md § Memory footprint](BENCHMARKS.md#memory-footprint)):
 | Records | Raw vectors | Reopened, `f32` index | Reopened, `i8` index (default) |
 |---------|-------------|-----------------------|--------------------------------|
 | 1K | 1 MiB | 4 MiB (2.40x) | 2 MiB (1.64x) |
 | 10K | 15 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
 | 100K | 146 MiB | 399 MiB (2.72x) | **256 MiB (1.74x)** |
 Down from 505 MiB (3.44x) at 100K before v2.6.0, when the cache held every
 embedding twice.
 ### Consolidation Efficiency
 1,000 records (10 signal + 990 noise), `working_capacity = 100`
 ([BENCHMARKS.md § Consolidation Efficiency](BENCHMARKS.md#consolidation-efficiency)):
 | Metric | Before | After | Delta |
 |--------|--------|-------|-------|
-| Records in store | 1,000 | ~110 | −89% |
+| Records in store | 1,000 | 100 | −90% |
-| Hit@1 recall | ~60% | ~90% | +30% |
+| Hit@1 recall (signal records) | 100% | 100% | no loss |
-| Search latency | ~2.8 ms | ~0.3 ms | **9x faster** |
+| Search latency | 2.75 ms | 0.31 ms | **8.8x faster** |
 **Full benchmark details: [BENCHMARKS.md](BENCHMARKS.md)**
@@ -166,74 +285,71 @@ retrieval recall reported as QA accuracy typically overstates by 20–30 points.
 ## Agent Memory Architecture
-ClawhDF5's agent memory engine implements research from 15+ recent papers on agentic memory systems. It's not a toy — it's the real thing.
+ClawhDF5's agent memory engine draws on 15+ recent papers on agentic memory systems (see [Research Foundation](#research-foundation)).
 ```
-                        ┌─────────────────┐
+                         ┌─────────────────┐
-                        │   Agent Query    │
+                         │   Agent Query   │
-                        └────────┬────────┘
+                         └────────┬────────┘
-                                 │
+                                  │
-                    ┌────────────▼────────────┐
+                ┌─────────────────▼──────────────────┐
-                    │   Hybrid Retrieval      │
+                │ HDF5Memory::hybrid_search          │
-                    │  Vector + BM25 + RRF    │
+                │ HNSW vector + BM25 keyword         │
-                    └────────────┬────────────┘
+                │ weighted fusion (0.4 / 0.6)        │
-                                 │
+                │ × √(Hebbian activation)            │
-              ┌──────────────────▼──────────────────┐
+                └─────────────────┬──────────────────┘
-              │         Multi-Factor Re-Ranking      │
+                                  │  OpenClaw backend adds:
-              │  temporal · authority · activation    │
+                ┌─────────────────▼──────────────────┐
-              └──────────────────┬──────────────────┘
+                │ Multi-factor re-ranking            │
-                                 │
+                │ relevance · recency · authority ·  │
-                    ┌────────────▼────────────┐
+                │ activation                         │
-                    │  Confidence Rejection   │
+                ├────────────────────────────────────┤
-                    │  (suppress bad matches) │
+                │ Confidence rejection               │
-                    └────────────┬────────────┘
+                │ (suppress bad matches)             │
-                                 │
+                └─────────────────┬──────────────────┘
-        ┌────────────────────────▼────────────────────────┐
+                                  │
-        │              Memory Store (HDF5)                │
+     ┌────────────────────────────▼────────────────────────────┐
-        │                                                 │
+     │ In memory                                               │
-        │  ┌───────────┐ ┌───────────┐ ┌───────────────┐  │
+     │  cache (flat f32 embeddings) · BM25 index · HNSW index  │
-        │  │ Working   │→│ Episodic  │→│  Semantic     │  │
+     │  provenance ledger + anomaly alerts (session-scoped)    │
-        │  │ (bounded) │ │ (bounded) │ │ (long-term)   │  │
+     └────────────────────────────┬────────────────────────────┘
-        │  └───────────┘ └───────────┘ └───────────────┘  │
+                                  │ WAL append; checkpoint
-        │                                                 │
+     ┌────────────────────────────▼────────────────────────────┐
-        │  ┌──────────┐ ┌──────────┐ ┌────────────────┐   │
+     │ agent_memory.h5   /meta · /memory · /sessions ·         │
-        │  │Knowledge │ │Temporal  │ │  Multi-Modal   │   │
+     │                   /knowledge_graph                      │
-        │  │  Graph   │ │  Index   │ │  Embeddings    │   │
+     │ agent_memory.h5.wal   chained-CRC write-ahead log       │
-        │  └──────────┘ └──────────┘ └────────────────┘   │
+     │ agent_memory.h5.ann   HNSW graph (derived, rebuildable) │
-        │                                                 │
+     │ agent_memory.h5.lock  single-writer lock                │
-        │  ┌──────────┐ ┌──────────┐ ┌────────────────┐   │
+     └─────────────────────────────────────────────────────────┘
        │  │Provenance│ │ Anomaly  │ │   Source       │   │
        │  │ Tracking │ │Detection │ │  Isolation     │   │
        │  └──────────┘ └──────────┘ └────────────────┘   │
        └─────────────────────────────────────────────────┘
                              │
                     ┌────────┴────────┐
                     │ agent_memory.h5 │
                     │   single file   │
                     └─────────────────┘
 ```
 Consolidation tiers (Working → Episodic → Semantic), the knowledge-graph
 algorithms, temporal and multi-modal indexes are library components you drive
 directly; the store persists the records, sessions and graph they work over.
 ### Module Overview
 | Module | What It Does |
 |--------|-------------|
-| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy entity resolution |
+| **`knowledge`** | Entity/relation graph with BFS traversal, spreading activation, fuzzy (Levenshtein) entity resolution |
-| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring and time-decay |
+| **`consolidation`** | Three-tier memory (Working → Episodic → Semantic) with importance scoring, novelty, and time-decay |
-| **`hybrid`** | Vector + BM25 fusion with Reciprocal Rank Fusion (RRF, k=60). The vector stage uses the HNSW index by default (`hnsw` feature, on by default); disable with `--no-default-features --features float16` for an exact linear scan |
+| **`hybrid`** | Vector + BM25 fusion. Default is a min-max-normalised weighted sum, vector 0.4 / keyword 0.6 (`hybrid::DEFAULT_FUSION`, tuned on LongMemEval); RRF is available via `Fusion::Rrf` / `hybrid_search_with`. The vector stage uses the HNSW index by default (`hnsw` feature); disable with `--no-default-features --features float16` for an exact linear scan |
-| **`reranker`** | Multi-factor re-ranking: temporal recency, source authority, activation weight |
+| **`reranker`** | Multi-factor re-ranking: retrieval relevance (leads, weight 1.0), temporal recency, source authority, activation weight. Used by the OpenClaw backend |
-| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches |
+| **`confidence`** | Low-confidence rejection — suppresses spurious recalls when nothing matches (OpenClaw backend) |
 | **`temporal`** | Sorted timestamp index, session DAG, entity timeline, temporal query hints |
 | **`multimodal`** | Cross-modal search across text/image/audio/video embeddings |
-| **`provenance`** | Source attribution, FNV-1a content hashing, integrity verification |
+| **`provenance`** | Source attribution and an unkeyed FNV-1a content hash per record, held in memory for the session, for detecting accidental corruption (not tamper-proof) |
-| **`anomaly`** | Write rate limiting, 15 injection pattern detectors, source distribution analysis |
+| **`anomaly`** | Write rate limiting, 15 injection-pattern detectors, source-distribution analysis. Alerts never block a save; drain them with `take_anomaly_alerts` |
 | **`openclaw`** | OpenClaw integration: MemoryBackend trait, Markdown ↔ HDF5 conversion |
 | **`vector_search`** | Flat cosine, pre-normed, SIMD, BLAS, GPU, parallel search paths |
-| **`ivf` / `pq`** | IVF-PQ approximate nearest neighbor for billion-scale search |
+| **`ivf` / `pq`** | Standalone IVF and IVF-PQ indexes (benchmarked to 100K vectors); not used by `HDF5Memory`, whose ANN index is HNSW |
-| **`bm25`** | BM25 keyword index with TF-IDF scoring |
+| **`bm25`** | Incremental Okapi BM25 inverted index, kept for the life of the store; optional stemming |
 | **`query_expand`** | Synonym / acronym / temporal query expansion |
 | **`entity_extract`** | Rule-based entity extraction from text chunks into the knowledge graph |
-| **`wal`** | Write-ahead log for crash-safe persistence; each entry is CRC32-checked on replay, so a corrupted entry stops replay there instead of loading bad data |
+| **`wal`** | Write-ahead log (v4) with a chained CRC32 per entry, so a corrupted, reordered, duplicated or spliced entry stops replay; checkpoints record a WAL mark so nothing is applied twice. Appends are not fsynced |
 | **`memory_strategy`** | Pluggable strategies: save-every, semantic-shift, user-correction detection |
 | **`decision_gate`** | Sub-microsecond trivial/substantive classification |
 | **`ephemeral`** | In-memory TTL/LFU working tier |
 | **`async_memory`** | Tokio-based async wrapper over the memory store (`async` feature) |
 ---
@@ -265,7 +381,7 @@ assert_eq!(values, vec![22.5, 23.1, 21.8]);
 use clawhdf5_agent::{HDF5Memory, MemoryConfig, MemoryEntry, AgentMemory};
 // Create memory store
-let config = MemoryConfig::new("agent.h5", "my-agent", 384);
+let config = MemoryConfig::new("agent.h5".into(), "my-agent", 384);
 let mut memory = HDF5Memory::create(config)?;
 // Save a memory
@@ -278,8 +394,8 @@ memory.save(MemoryEntry {
    tags: "preference".into(),
 })?;
-// Search
+// Hybrid search: vector + BM25, weighted 0.4 / 0.6 (the measured default)
-let results = memory.search(&query_embedding, 5)?;
+let results = memory.hybrid_search(&query_embedding, "user preferences", 0.4, 0.6, 5);
 for result in results {
    println!("[{:.3}] {}", result.score, result.chunk);
 }
@@ -309,8 +425,8 @@ let neighbors = kg.bfs_neighbors(alice, 2);  // 2-hop neighborhood
 let activated = kg.spreading_activation(&[alice], 0.5, 0.01, 5);
 // Entity resolution — fuzzy matching
-let resolved = kg.resolve_or_create("alice", "person", -1, 2);
+let (id, created) = kg.resolve_or_create("alice", "person", -1, 2);
-// Returns existing Alice entity (Levenshtein distance ≤ 2)
+// id == alice, created == false: matched the existing entity (Levenshtein distance ≤ 2)
 ```
 ### Memory Consolidation
@@ -321,15 +437,19 @@ use clawhdf5_agent::consolidation::*;
 let config = ConsolidationConfig::default();
 let mut engine = ConsolidationEngine::new(config);
-// Add memories — automatically scored for importance
+let now = 1_700_000_000.0; // seconds since the epoch
-engine.add_memory("User prefers dark mode", vec![0.1, 0.2, ...], MemorySource::User);
+
-engine.add_memory("ok", vec![0.0, 0.0, ...], MemorySource::System);
+// Add memories — automatically scored for importance.
 // Elevated sources (System, …) go through a separate, explicit API.
 let id = engine.add_memory("User prefers dark mode".into(), vec![0.1, 0.2, ...], UntrustedSource::User, now);
 engine.add_trusted_memory("ok".into(), vec![0.0, 0.0, ...], TrustedSource::System, now);
 // Access a memory (reactivates it)
-engine.access_memory(0);
+engine.access_memory(id, now);
 // Run consolidation cycle
-let stats = engine.consolidate();
+engine.consolidate(now);
 let stats = engine.get_stats();
 // Working memories promote to Episodic (if important enough)
 // Episodic memories promote to Semantic (if accessed enough)
 // Low-decay memories get evicted when tiers are full
@@ -357,13 +477,13 @@ let recent = index.latest(10);
 use clawhdf5_agent::openclaw::*;
 // Create backend
-let mut backend = ClawhdfBackend::create("memory.h5", "agent-1", 384)?;
+let mut backend = ClawhdfBackend::create(std::path::Path::new("memory.h5"), 384)?;
 // Ingest existing Markdown memory files
 let md = std::fs::read_to_string("MEMORY.md")?;
 let count = backend.ingest_markdown("MEMORY.md", &md)?;
-// Search (uses full pipeline: RRF → re-rank → confidence filter)
+// Search (full pipeline: weighted vector + BM25 fusion → re-rank → confidence filter)
 let results = backend.search("user preferences", &query_embedding, 5);
 // Export back to Markdown
@@ -375,22 +495,23 @@ let exported = backend.export_markdown("MEMORY.md")?;
 ## Crate Map
 ```
-clawhdf5 workspace (16 crates, ~92K lines of Rust; plus libaec-sys, an
+clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
-                     internal FFI bindings crate for the optional szip feature)
+                     and benches; plus libaec-sys, an internal FFI bindings
                     crate for the optional szip feature)
 │
 ├── Core HDF5
-│   ├── clawhdf5-format      — Binary parser/writer (no_std), shared type definitions
+│   ├── clawhdf5-format      — Binary parser/writer (no_std-capable), shared type definitions
-│   ├── clawhdf5-io          — I/O abstraction (buffered, mmap, async)
+│   ├── clawhdf5-io          — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
 │   ├── clawhdf5-filters     — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format
 │   ├── clawhdf5-derive      — Proc macros
 │   ├── clawhdf5             — High-level API
 │   ├── clawhdf5-netcdf4     — NetCDF-4 support
-│   ├── clawhdf5-accel       — SIMD (NEON, AVX2, AVX-512)
+│   ├── clawhdf5-accel       — SIMD (AVX2, NEON incl. SDOT int8; AVX-512 behind `avx512`)
 │   └── clawhdf5-gpu         — GPU compute (wgpu, hand-written WGSL compute shaders)
 │
 ├── Agent Memory
-│   ├── clawhdf5-agent       — Memory engine (20.9K lines, 32 modules; WAL is CRC32-checked per entry)
+│   ├── clawhdf5-agent       — Memory engine (24.7K lines, 32 modules; chained-CRC WAL)
-│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor (default backend; optional `parallel` feature)
+│   ├── clawhdf5-ann         — HNSW approximate nearest neighbor (default backend; f32 or int8 storage; `parallel` build)
 │   ├── clawhdf5-migrate     — SQLite → HDF5 migration
 │   ├── clawhdf5-android     — Android JNI bridge
 │   └── clawhdf5-cli         — CLI tool
@@ -411,10 +532,10 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
 | Paper | Key Insight | ClawhDF5 Module |
 |-------|-------------|-----------------|
-| **MemX** (2026) | RRF + multi-factor re-ranking | `hybrid`, `reranker` |
+| **MemX** (2026) | Hybrid fusion + multi-factor re-ranking | `hybrid`, `reranker` |
-| **Graph-Native Cognitive Memory** (2026) | Graph-structured belief revision | `knowledge` |
+| **Graph-Native Cognitive Memory** (2026) | Graph-structured memory (weighted, timestamped relations; entity timelines) | `knowledge`, `temporal` |
 | **CraniMem** (2026) | Bounded hippocampal memory | `consolidation` |
-| **D-MEM** (2026) | Reward prediction error gating | `consolidation` |
+| **D-MEM** (2026) | Surprise-gated storage (implemented as a novelty score) | `consolidation` |
 | **SYNAPSE** (2025) | Spreading activation for recall | `knowledge` |
 | **RAGdb** (2025) | Zero-dependency edge RAG | Architecture |
 | **MemoryGraft** (2025) | Memory poisoning attacks | `anomaly`, `provenance` |
@@ -429,29 +550,43 @@ ClawhDF5's agent memory design draws from 15+ recent papers:
 | Flag | Default | Description |
 |------|---------|-------------|
-| `agent` | no | Full agent memory layer |
+| `float16` | **yes** | Half-precision cosine kernel (`cosine_similarity_f16`). Half-precision *storage* is the `MemoryConfig::float16` setting below, and needs no feature |
 | `float16` | **yes** | Half-precision embedding storage (2× compression) |
 | `hnsw` | **yes** | HNSW approximate vector index for `hybrid_search` (via `clawhdf5-ann`); disable for an exact linear scan |
-
+| `parallel` | **yes** | Parallel HNSW bulk build (same graph, ~3× faster on 16 cores) and Rayon brute-force search strategies |
-`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search` tune the
+| `zstd` | no | Compress embeddings with Zstd instead of deflate when `MemoryConfig::compression` is on (links libzstd) |
 vector index (16 / 64 / scale-with-`k` by default) and are stored with the
 file.
 `MemoryConfig::quantized_index` (off by default) stores the HNSW index's own
 copy of the embeddings as `i8`, roughly halving a loaded store's memory
 (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised distances are
 approximate, so the query path re-scores the candidate pool against the exact
 embeddings the store already holds, which keeps recall at the `f32` index's
 level. On AVX2 it is also **faster** — 1.63x the queries per second and 1.8x
 the build speed at equal recall — because the int8 kernel is SIMD too. It
 stays off by default only because that kernel is AVX2-only and aarch64 falls
 back to a scalar loop. See `BENCHMARKS.md`, "Quantising the index copy".
 | `parallel` | no | Rayon parallel search |
 | `fast-math` | no | BLAS matrix-vector multiply |
 | `accelerate` | no | Apple Accelerate / AMX (macOS) |
 | `openblas` | no | OpenBLAS (Linux) |
 | `gpu` | no | GPU search via wgpu |
 | `async` | no | Tokio async with background flush |
 | `agent` | no | Reserved; currently enables nothing (the agent layer is always built) |
 To opt out of the parallel build: `--no-default-features --features float16,hnsw`.
 For an exact linear cosine scan instead of HNSW: `--no-default-features --features float16`.
 `MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search` tune the
 vector index (16 / 64 / scale-with-`k` by default) and are stored with the
 file.
 `MemoryConfig::quantized_index` (**on by default** for new stores) holds the
 HNSW index's own copy of the embeddings as `i8`, roughly halving a loaded
 store's memory (2.72x -> 1.74x the raw vectors at 100k x 384). Quantised
 distances are approximate, so the query path re-scores the candidate pool
 against the exact embeddings the store already holds, which keeps recall at the
 `f32` index's level. It is also **faster**: 1.63x the queries per second at
 equal recall on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with
 index builds 1.8x and 2.3x faster respectively. Stores created before the
 setting existed keep their `f32` index; opt out for new stores with
 `quantized_index = false` or `clawhdf5-cli create --f32-index`. See
 [BENCHMARKS.md § Quantising the index copy](BENCHMARKS.md#quantising-the-index-copy-quantized_index).
 `MemoryConfig::float16` (off by default; CLI `create --float16`) stores the
 embeddings on disk as IEEE half precision (numpy `float16`): at 100K × 384 the
 file drops from 154 to 81 MiB, checkpoints and opens get faster, and vector
 recall and search latency do not change. Embeddings are rounded as they are
 saved, so the store searches the same before and after a reopen; values must
 lie within ±65504. See
 [BENCHMARKS.md § float16 embedding storage](BENCHMARKS.md#float16-embedding-storage-memoryconfigfloat16).
 ### `clawhdf5-format`
@@ -461,26 +596,31 @@ back to a scalar loop. See `BENCHMARKS.md`, "Quantising the index copy".
 | `deflate` | yes | Deflate compression |
 | `checksum` | yes | Jenkins lookup3 verification |
 | `provenance` | yes | SHA-256 provenance attributes |
-| `fast-deflate` | **yes** | zlib-ng backend for faster deflate |
+| `zlib-rs` | **yes** | Pure-Rust deflate backend ([zlib-rs](https://github.com/trifectatechfoundation/zlib-rs)) |
-| `system-zlib-decompress` | **yes** | Use the system zlib for decompression where available |
+| `fast-deflate` | no | zlib-ng deflate backend instead (C; needs `cmake`). Overrides `zlib-rs` when both are on |
 | `system-zlib-decompress` | **yes** | Use Apple's system libz for decompression (macOS only; no effect elsewhere) |
 | `parallel` | no | Parallel chunk encoding + compression (rayon) |
 | `fast-checksum` | no | crc32fast-accelerated checksums |
 | `lz4` | no | LZ4 block compression filter (id 32004) |
 | `zstd` | no | Zstandard compression filter (id 32015) |
 | `pcodec` | no | Pcodec lossless numerical codec (id 32023, via `pco` crate) |
-| `system-zlib` / `zlib-rs` | no | Alternative zlib backends for deflate |
+| `system-zlib` | no | System zlib backend for deflate (C) |
 | `blake3_hash` | no | BLAKE3 content hashing for provenance |
 | `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) |
 ### `clawhdf5-ann`
 | Flag | Default | Description |
 |------|---------|-------------|
-| `parallel` | no | Rayon-parallel neighbor-distance computation during HNSW graph pruning |
+| `parallel` | no | Batched bulk build runs neighbour planning and back-link pruning on a Rayon pool; the graph is identical with or without it (enabled by `clawhdf5-agent`'s default `parallel`) |
 ### `clawhdf5-io`
 | Flag | Default | Description |
 |------|---------|-------------|
 | `mmap` | no | Memory-mapped reads (`memmap2`) |
 | `async` | no | Tokio-based async I/O |
 | `hsds` | no | HSDS (HDF REST service) client |
 | `mpi-io` | no | MPI-backed I/O via the `mpi` crate |
 > **Parallel I/O (MPI) limitation:** `mpi-io`'s read path is a root-rank read
@@ -494,17 +634,17 @@ back to a scalar loop. See `BENCHMARKS.md`, "Quantising the index copy".
 ## Building
 ```bash
-# Default
+# Default (pure Rust: no cmake or C compiler needed)
 cargo build --workspace
 # Agent memory with all accelerations (Linux)
-cargo build -p clawhdf5-agent --features "agent,float16,parallel,fast-math"
+cargo build -p clawhdf5-agent --features fast-math
 # Agent memory with Apple Accelerate (macOS)
-cargo build -p clawhdf5-agent --features "agent,float16,accelerate,parallel,gpu"
+cargo build -p clawhdf5-agent --features "accelerate,gpu"
 # Tests
-cargo test --workspace            # all 1,650+ tests
+cargo test --workspace            # all 1,850+ tests
 cargo test -p clawhdf5-agent      # agent memory tests
 scripts/ci-test.sh                # what CI runs: fmt, clippy matrix, tests,
                                  # h5py/netCDF4 interop, no_std
@@ -526,25 +666,42 @@ cargo bench -p clawhdf5-bench               # h5bench-equivalent I/O suite
 ```
 agent_memory.h5
-├── /meta
+├── /meta                          (attributes)
-│   ├── schema_version: "1.0"
+│   ├── schema_version: "1.0", edgehdf5_version
-│   ├── agent_id, embedder, embedding_dim
+│   ├── agent_id, embedder, embedding_dim, chunk_size, overlap, created_at
-│   └── created_at
+│   ├── float16, compression, compression_level, compact_threshold,
 │   │   hebbian_boost, decay_factor, wal_enabled, wal_max_entries
 │   ├── quantized_index, hnsw_m, hnsw_ef_construction, hnsw_ef_search
 │   ├── wal_applied_len, wal_applied_crc   (WAL mark of the last checkpoint)
 │   └── ann_generation                     (ties the .ann sidecar to this checkpoint)
 ├── /memory
-│   ├── chunks:      string[N]
+│   ├── chunks:              string[N]
-│   ├── embeddings:  f32[N × D]  (or f16 with float16 flag)
+│   ├── embeddings:          f32[N × D], or f16 for a `float16` store
-│   ├── tombstones:  u8[N]
+│   │                        (chunked; deflate, or Zstd with the `zstd`
-│   └── norms:       f32[N]      (pre-computed L2)
+│   │                         feature, when compression is on)
 │   ├── source_channel:      string[N]
 │   ├── timestamps:          f64[N]
 │   ├── session_ids:         string[N]
 │   ├── tags:                string[N]
 │   ├── tombstones:          u8[N]
 │   ├── norms:               f32[N]      (pre-computed L2)
 │   └── activation_weights:  f32[N]      (Hebbian)
 ├── /sessions
-│   ├── ids:         string[S]
+│   ├── ids, channels, summaries:  string[S]
-│   └── summaries:   string[S]
+│   ├── start_idxs, end_idxs:      i64[S]
 │   └── timestamps:                f64[S]
 └── /knowledge_graph
-    ├── entity_names:    string[E]
+    ├── entity_ids, entity_emb_idxs: i64[E]; entity_names, entity_types: string[E]
-    ├── relation_srcs:   i64[R]
+    ├── relation_srcs, relation_tgts: i64[R]; relation_types: string[R]
-    ├── relation_tgts:   i64[R]
+    ├── relation_weights: f32[R]; relation_ts: f64[R]
-    └── relation_types:  string[R]
+    └── alias_strings: string[A]; alias_entity_ids: i64[A]   (when aliases exist)
 ```
 Alongside the store: `<store>.h5.wal` (write-ahead log), `<store>.h5.ann`
 (HNSW graph; derived, safe to delete) and `<store>.h5.lock` (single-writer
 lock). A second writer gets `MemoryError::Locked`; use
 `HDF5Memory::open_read_only` for a lock-free point-in-time view.
 ---
 ## Migration
@@ -599,6 +756,6 @@ MIT
 ---
 <p align="center">
-  <em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em><br>
+  <em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em><br>
-  <em>~92,000 lines of Rust. Zero C dependencies. One file to remember everything.</em>
+  <em>~86,000 lines of Rust. Zero C dependencies. One file to remember everything.</em>
 </p>
@@ -2,6 +2,7 @@
 name = "clawhdf5-accel"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "SIMD-accelerated operations for rustyhdf5"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -124,11 +124,24 @@ pub fn dot_product(a: &[f32], b: &[f32]) -> f32 {
 /// Dot product of two `i8` slices, widened to `i32`.
 ///
-/// The kernel behind int8-quantised vector search. Uses the AVX2 path
+/// The kernel behind int8-quantised vector search. On x86-64 it uses the AVX2
-/// whenever AVX2 is present — including on AVX-512 machines, where it is
+/// path whenever AVX2 is present (including on AVX-512 machines, where it is
-/// what the f32 kernels use too on a default build.
+/// what the f32 kernels use too on a default build). On aarch64 it uses the
 /// ARMv8.2 `SDOT` instruction when the CPU has the dot-product extension, and
 /// plain NEON otherwise.
 pub fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
    match detect_backend() {
        #[cfg(target_arch = "aarch64")]
        Backend::Neon => {
            if std::arch::is_aarch64_feature_detected!("dotprod") {
                // SAFETY: the dotprod extension was just detected at runtime.
                unsafe { neon::dot_i8_dotprod(a, b) }
            } else {
                // SAFETY: NEON is always available on aarch64.
                unsafe { neon::dot_i8(a, b) }
            }
        }
        #[cfg(target_arch = "x86_64")]
        // SAFETY: both variants imply AVX2 was detected at runtime (the
        // AVX-512 backend is only selected on CPUs that also have AVX2).
@@ -760,6 +773,42 @@ mod dot_i8_tests {
        }
    }
    /// Dispatch only ever takes one path on a given CPU, so on a machine with
    /// the dot-product extension the plain-NEON kernel would otherwise go
    /// untested. Check each aarch64 kernel against scalar directly.
    #[cfg(target_arch = "aarch64")]
    #[test]
    fn every_aarch64_kernel_matches_scalar_exactly() {
        for len in [0, 1, 7, 15, 16, 17, 31, 32, 33, 63, 64, 100, 384, 385, 1536] {
            let a = codes(len, 7 + len as u64);
            let b = codes(len, 7000 + len as u64);
            let want = scalar::dot_i8(&a, &b);
            // SAFETY: NEON is always available on aarch64.
            assert_eq!(unsafe { neon::dot_i8(&a, &b) }, want, "neon, len {len}");
            if std::arch::is_aarch64_feature_detected!("dotprod") {
                // SAFETY: the dotprod extension was just detected.
                assert_eq!(
                    unsafe { neon::dot_i8_dotprod(&a, &b) },
                    want,
                    "dotprod, len {len}"
                );
            }
        }
        // The extremes, through both kernels.
        let lo = vec![-128i8; 4096];
        let hi = vec![127i8; 4096];
        // SAFETY: NEON is always available on aarch64.
        assert_eq!(unsafe { neon::dot_i8(&lo, &lo) }, 4096 * 128 * 128);
        // SAFETY: NEON is always available on aarch64.
        assert_eq!(unsafe { neon::dot_i8(&lo, &hi) }, -4096 * 128 * 127);
        if std::arch::is_aarch64_feature_detected!("dotprod") {
            // SAFETY: the dotprod extension was just detected.
            assert_eq!(unsafe { neon::dot_i8_dotprod(&lo, &lo) }, 4096 * 128 * 128);
            // SAFETY: the dotprod extension was just detected.
            assert_eq!(unsafe { neon::dot_i8_dotprod(&lo, &hi) }, -4096 * 128 * 127);
        }
    }
    #[test]
    fn extremes_do_not_overflow() {
        // -128 * -128 is the largest product; a long run of it must still fit.
@@ -180,3 +180,130 @@ pub fn checksum_fletcher32(data: &[u8]) -> u32 {
    (sum2 << 16) | sum1
 }
 /// NEON dot product of two `i8` slices, widened to `i32`, for any aarch64 CPU.
 ///
 /// `vmull_s8` multiplies eight lanes into `i16` — even `-128 * -128` is 16 384,
 /// inside `i16` — and `vpadalq_s16` adds adjacent pairs of those into `i32`
 /// accumulators, so nothing can overflow before the final horizontal sum.
 ///
 /// CPUs with the ARMv8.2 dot-product extension should use
 /// [`dot_i8_dotprod`], which does the multiply and the accumulate in one
 /// instruction.
 ///
 /// # Safety
 /// Caller must ensure aarch64 target (NEON always available).
 // SAFETY: NEON is always available on aarch64 targets; caller guarantees aarch64.
 #[target_feature(enable = "neon")]
 pub unsafe fn dot_i8(a: &[i8], b: &[i8]) -> i32 {
    assert_eq!(a.len(), b.len());
    let len = a.len();
    let mut i = 0;
    let mut acc0 = vdupq_n_s32(0);
    let mut acc1 = vdupq_n_s32(0);
    while i + 16 <= len {
        // SAFETY: NEON is available per the # Safety contract, and both
        // 16-byte loads start at an index checked against `len` above.
        unsafe {
            let va = vld1q_s8(a.as_ptr().add(i));
            let vb = vld1q_s8(b.as_ptr().add(i));
            acc0 = vpadalq_s16(acc0, vmull_s8(vget_low_s8(va), vget_low_s8(vb)));
            acc1 = vpadalq_s16(acc1, vmull_high_s8(va, vb));
        }
        i += 16;
    }
    let mut sum = vaddvq_s32(vaddq_s32(acc0, acc1));
    while i < len {
        sum += i32::from(a[i]) * i32::from(b[i]);
        i += 1;
    }
    sum
 }
 /// One `SDOT`: for each of the four `i32` lanes of `acc`, add the dot
 /// product of the corresponding four `i8` pairs from `a` and `b`.
 ///
 /// Written as inline assembly because the `vdotq_s32` intrinsic is still
 /// behind the unstable `stdarch_neon_dotprod` feature; inline assembly is
 /// stable on aarch64.
 ///
 /// # Safety
 /// Caller must ensure the CPU supports the `dotprod` extension.
 #[inline]
 #[target_feature(enable = "neon,dotprod")]
 unsafe fn sdot(acc: int32x4_t, a: int8x16_t, b: int8x16_t) -> int32x4_t {
    let mut acc = acc;
    // SAFETY: `dotprod` is enabled for this function and the caller
    // guarantees the CPU supports it. The instruction reads only its three
    // vector registers and touches no memory.
    unsafe {
        std::arch::asm!(
            "sdot {acc:v}.4s, {a:v}.16b, {b:v}.16b",
            acc = inout(vreg) acc,
            a = in(vreg) a,
            b = in(vreg) b,
            options(pure, nomem, nostack),
        );
    }
    acc
 }
 /// NEON dot product of two `i8` slices using the ARMv8.2 dot-product
 /// extension (`SDOT`): sixteen multiply-accumulates per instruction, straight
 /// into `i32` lanes.
 ///
 /// Present on the cores this crate actually runs on — Cortex-A76 and later
 /// (Raspberry Pi 5, current Android phones), Neoverse-N1 (Graviton2, Ampere
 /// Altra), and every Apple Silicon generation.
 ///
 /// # Safety
 /// Caller must verify `is_aarch64_feature_detected!("dotprod")`.
 // SAFETY: caller has verified the dotprod extension at runtime.
 #[target_feature(enable = "neon,dotprod")]
 pub unsafe fn dot_i8_dotprod(a: &[i8], b: &[i8]) -> i32 {
    assert_eq!(a.len(), b.len());
    let len = a.len();
    let mut i = 0;
    let mut acc0 = vdupq_n_s32(0);
    let mut acc1 = vdupq_n_s32(0);
    // Two independent accumulators so consecutive SDOTs are not serialised on
    // one register.
    while i + 32 <= len {
        // SAFETY: dotprod is available per the # Safety contract, and every
        // 16-byte load starts at an index checked against `len` above.
        unsafe {
            acc0 = sdot(
                acc0,
                vld1q_s8(a.as_ptr().add(i)),
                vld1q_s8(b.as_ptr().add(i)),
            );
            acc1 = sdot(
                acc1,
                vld1q_s8(a.as_ptr().add(i + 16)),
                vld1q_s8(b.as_ptr().add(i + 16)),
            );
        }
        i += 32;
    }
    if i + 16 <= len {
        // SAFETY: as above; the load is bounds-checked by this condition.
        unsafe {
            acc0 = sdot(
                acc0,
                vld1q_s8(a.as_ptr().add(i)),
                vld1q_s8(b.as_ptr().add(i)),
            );
        }
        i += 16;
    }
    let mut sum = vaddvq_s32(vaddq_s32(acc0, acc1));
    while i < len {
        sum += i32::from(a[i]) * i32::from(b[i]);
        i += 1;
    }
    sum
 }
@@ -2,6 +2,7 @@
 name = "clawhdf5-agent"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "HDF5-backed persistent memory store for on-device AI agents"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -1,6 +1,7 @@
 //! In-memory cache for memory entries, sessions, and knowledge graph.
 use crate::vector_search;
 use clawhdf5_format::float16::round_to_f16;
 /// Every entry's embedding, in one contiguous `[N x dim]` buffer.
 ///
@@ -149,6 +150,11 @@ pub struct MemoryCache {
    pub norms: Vec<f32>,
    /// Hebbian activation weights (default 1.0 per entry).
    pub activation_weights: Vec<f32>,
    /// Round every embedding to IEEE half precision as it enters the cache,
    /// so the cache holds exactly what a `float16` store writes to disk. Set
    /// it with [`MemoryCache::set_half_precision`], which also rounds the
    /// rows already held.
    pub half_precision: bool,
 }
 impl MemoryCache {
@@ -164,9 +170,44 @@ impl MemoryCache {
            embedding_dim,
            norms: Vec::new(),
            activation_weights: Vec::new(),
            half_precision: false,
        }
    }
    /// Switch half-precision rounding on or off. Turning it on rounds every
    /// embedding already held (and recomputes norms where one changed) —
    /// e.g. a `float16` store whose last checkpoint predates half-precision
    /// storage and so is still `f32` on disk.
    pub fn set_half_precision(&mut self, on: bool) {
        self.half_precision = on;
        if !on {
            return;
        }
        for i in 0..self.embeddings.len() {
            let row = &self.embeddings[i];
            if row
                .iter()
                .all(|&v| round_to_f16(v).to_bits() == v.to_bits())
            {
                continue;
            }
            let rounded: Vec<f32> = row.iter().map(|&v| round_to_f16(v)).collect();
            self.norms[i] = vector_search::compute_norm(&rounded);
            self.embeddings.set(i, &rounded);
        }
    }
    /// The embedding as the cache will hold it: rounded to half precision
    /// when [`Self::half_precision`] is on, otherwise unchanged.
    fn stored_form(&self, mut embedding: Vec<f32>) -> Vec<f32> {
        if self.half_precision {
            for v in &mut embedding {
                *v = round_to_f16(*v);
            }
        }
        embedding
    }
    /// Kept for callers that used to have to re-flatten after a bulk load.
    /// The buffer is always flat now, so there is nothing to rebuild.
    #[deprecated(note = "embeddings are stored flat; this is a no-op")]
@@ -202,6 +243,7 @@ impl MemoryCache {
        tags: String,
    ) -> usize {
        let idx = self.chunks.len();
        let embedding = self.stored_form(embedding);
        let norm = vector_search::compute_norm(&embedding);
        self.chunks.push(chunk);
        self.embeddings.push(&embedding);
@@ -240,6 +282,7 @@ impl MemoryCache {
        session_id: String,
    ) {
        if idx < self.chunks.len() {
            let embedding = self.stored_form(embedding);
            let norm = vector_search::compute_norm(&embedding);
            self.chunks[idx] = chunk;
            self.embeddings.set(idx, &embedding);
@@ -439,4 +482,60 @@ mod tests {
            .reset_from(2, vec![vec![1.0, 2.0], vec![3.0, 4.0]]);
        assert_eq!(cache.embeddings.as_flat(), vec![1.0, 2.0, 3.0, 4.0]);
    }
    #[test]
    fn set_half_precision_rounds_existing_rows_and_their_norms() {
        // A store with float16 set whose checkpoint is still f32 on disk
        // loads full-precision rows; switching rounding on must bring them to
        // exactly what the next checkpoint will write.
        let mut cache = MemoryCache::new(3);
        cache.push(
            "a".into(),
            vec![0.1, 0.2, 0.3],
            "c".into(),
            0.0,
            "s".into(),
            "".into(),
        );
        cache.push(
            "b".into(),
            vec![0.5, 0.25, 1.0],
            "c".into(),
            0.0,
            "s".into(),
            "".into(),
        );
        let exact_norm = cache.norms[0];
        cache.set_half_precision(true);
        let row0: Vec<f32> = [0.1f32, 0.2, 0.3]
            .iter()
            .map(|&v| round_to_f16(v))
            .collect();
        assert_eq!(&cache.embeddings[0], row0.as_slice());
        assert_eq!(cache.norms[0], vector_search::compute_norm(&row0));
        assert_ne!(cache.norms[0], exact_norm);
        // Already representable: untouched.
        assert_eq!(&cache.embeddings[1], &[0.5, 0.25, 1.0]);
        // New rows are rounded as they arrive, and updates too.
        cache.push(
            "c".into(),
            vec![0.1, 0.0, 0.0],
            "c".into(),
            0.0,
            "s".into(),
            "".into(),
        );
        assert_eq!(cache.embeddings[2][0], round_to_f16(0.1));
        cache.update(
            2,
            "c".into(),
            vec![0.3, 0.0, 0.0],
            "c".into(),
            0.0,
            "s".into(),
        );
        assert_eq!(cache.embeddings[2][0], round_to_f16(0.3));
    }
 }
@@ -63,6 +63,7 @@ use std::path::{Path, PathBuf};
 use cache::MemoryCache;
 #[cfg(feature = "hnsw")]
 use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};
 use clawhdf5_format::float16::round_to_f16;
 use ephemeral::{EphemeralConfig, EphemeralStore};
 // EphemeralEntry and EphemeralStats are part of the crate public API via
@@ -83,6 +84,9 @@ pub enum MemoryError {
    NotFound(String),
    /// Another `HDF5Memory` (in this or another process) has the store open.
    Locked(String),
    /// A record the store cannot hold as given, e.g. an embedding value
    /// outside the half-precision range of a `float16` store.
    InvalidEntry(String),
 }
 impl std::fmt::Display for MemoryError {
@@ -93,6 +97,7 @@ impl std::fmt::Display for MemoryError {
            MemoryError::Schema(e) => write!(f, "schema error: {e}"),
            MemoryError::NotFound(e) => write!(f, "not found: {e}"),
            MemoryError::Locked(e) => write!(f, "store is locked: {e}"),
            MemoryError::InvalidEntry(e) => write!(f, "invalid entry: {e}"),
        }
    }
 }
@@ -124,6 +129,12 @@ pub struct MemoryConfig {
    pub embedding_dim: usize,
    pub chunk_size: usize,
    pub overlap: usize,
    /// Store embeddings as IEEE half precision (numpy `float16`): half the
    /// bytes of the embeddings dataset on disk. Every embedding is rounded to
    /// the nearest half as it enters the store, in memory as well as on disk,
    /// so search results are the same before and after a reopen. Values must
    /// lie within ±65504; a save outside that is `MemoryError::InvalidEntry`.
    /// Fixed when the store is created (persisted in `/meta`).
    pub float16: bool,
    pub compression: bool,
    pub compression_level: u32,
@@ -134,13 +145,19 @@ pub struct MemoryConfig {
    pub wal_enabled: bool,
    pub wal_max_entries: usize,
    /// Store the vector index's own copy of the embeddings as int8 rather than
-    /// f32, a quarter of the memory.
+    /// f32, a quarter of the memory. **On by default** for new stores.
    ///
    /// The index's copy is the single largest part of a loaded store's
    /// footprint. Quantised distances are approximate, so the candidate pool
    /// is re-scored against the cache's exact embeddings before fusion, which
-    /// restores recall; what it costs is throughput — roughly 13% of queries
+    /// holds recall at the f32 index's level. It is also faster, not slower:
-    /// per second and 16% of build time at 100K x 384. See `BENCHMARKS.md`.
+    /// at equal recall, 1.63x the queries per second on x86-64 (AVX2) and
    /// 1.18x on a Raspberry Pi 5 (NEON `SDOT`), with builds 1.8x and 2.3x
    /// faster. See `BENCHMARKS.md`.
    ///
    /// Persisted with the store. Stores written before this setting existed
    /// have no stored value and open as `false`, so reopening an old store
    /// never changes how its index is held.
    ///
    /// Has no effect without the `hnsw` feature.
    pub quantized_index: bool,
@@ -182,7 +199,7 @@ impl MemoryConfig {
            created_at,
            wal_enabled: true,
            wal_max_entries: 500,
-            quantized_index: false,
+            quantized_index: true,
            hnsw_m: 16,
            hnsw_ef_construction: 64,
            hnsw_ef_search: 0,
@@ -311,7 +328,8 @@ impl HDF5Memory {
    /// Create a new HDF5 memory file with the given configuration.
    pub fn create(config: MemoryConfig) -> Result<Self> {
        let lock = store_lock::StoreLock::acquire(&config.path)?;
-        let cache = MemoryCache::new(config.embedding_dim);
+        let mut cache = MemoryCache::new(config.embedding_dim);
        cache.set_half_precision(config.float16);
        let sessions = SessionCache::new();
        let knowledge = KnowledgeCache::new();
@@ -1064,7 +1082,29 @@ impl HDF5Memory {
    /// Upsert: if an active entry with the same tags (key) exists, update it in-place.
    /// Otherwise append a new entry. Use this for key-based memory stores where
    /// the same key should not create duplicates.
    /// A `float16` store holds embeddings as IEEE half precision, which has no
    /// finite value beyond ±65504. Refuse such an embedding rather than
    /// silently store infinity. (Values that are already infinite or NaN are
    /// stored as they are, as in an `f32` store.)
    fn check_embedding(&self, embedding: &[f32]) -> Result<()> {
        if !self.config.float16 {
            return Ok(());
        }
        let overflow = embedding
            .iter()
            .enumerate()
            .find(|&(_, &v)| v.is_finite() && round_to_f16(v).is_infinite());
        match overflow {
            None => Ok(()),
            Some((i, v)) => Err(MemoryError::InvalidEntry(format!(
                "embedding[{i}] = {v} is outside the half-precision range (±65504) \
                 of this float16 store"
            ))),
        }
    }
    pub fn save_or_update(&mut self, entry: MemoryEntry) -> Result<usize> {
        self.check_embedding(&entry.embedding)?;
        if let Some(existing_idx) = self.cache.find_by_tags(&entry.tags) {
            if let Some(ref mut w) = self.wal {
                let wal_entry = wal::WalEntry {
@@ -1120,6 +1160,7 @@ impl HDF5Memory {
 impl AgentMemory for HDF5Memory {
    fn save(&mut self, entry: MemoryEntry) -> Result<usize> {
        self.check_embedding(&entry.embedding)?;
        if let Some(ref mut w) = self.wal {
            let wal_entry = wal::WalEntry {
                entry_type: wal::WalEntryType::Save,
@@ -1161,6 +1202,10 @@ impl AgentMemory for HDF5Memory {
    }
    fn save_batch(&mut self, entries: Vec<MemoryEntry>) -> Result<Vec<usize>> {
        // All or nothing: check every entry before storing any.
        for entry in &entries {
            self.check_embedding(&entry.embedding)?;
        }
        let mut indices = Vec::with_capacity(entries.len());
        for entry in entries {
            let idx = self.cache.push(
@@ -1321,6 +1366,9 @@ impl HDF5Memory {
        })?;
        let view = memory_strategy::CacheStoreView::new(&self.cache, &self.knowledge);
        let output = strat.evaluate(&exchange, &view);
        for e in &output.entries {
            self.check_embedding(&e.embedding)?;
        }
        for e in &output.entries {
            self.cache.push(
                e.chunk.clone(),
@@ -1405,6 +1453,16 @@ impl HDF5Memory {
        let mut promoted = 0;
        for key in candidates {
            // Check before taking, so a rejected entry stays in the ephemeral
            // tier rather than being lost.
            if let Some(emb) = self
                .ephemeral
                .as_ref()
                .and_then(|s| s.get_entry(&key))
                .and_then(|e| e.embedding.as_deref())
            {
                self.check_embedding(emb)?;
            }
            let entry = match self
                .ephemeral
                .as_mut()
@@ -159,20 +159,27 @@ fn build_memory_group(
    // chunks: fixed-length string array
    write_string_dataset(&mut group, "chunks", &cache.chunks);
-    // embeddings: f32 [N x D]
+    // embeddings: [N x D], f32 — or IEEE half precision for a `float16`
    // store. The cache already holds half-rounded values then, so this
    // conversion is exact and a reopened store sees the same numbers.
    let n = cache.embeddings.len() as u64;
    let d = cache.embedding_dim as u64;
    let flat = cache.flat_embeddings();
    {
-        let ds = group
+        let ds = group.create_dataset("embeddings");
-            .create_dataset("embeddings")
+        let elem_bytes: u64 = if config.float16 {
-            .with_f32_data(flat)
+            ds.with_f16_data(flat);
-            .with_shape(&[n, d]);
+            2
        } else {
            ds.with_f32_data(flat);
            4
        };
        ds.with_shape(&[n, d]);
        // Chunk size tuning: target ~256KB per chunk for optimal I/O
        if n > 0 && d > 0 {
            let target_chunk_bytes: u64 = 256 * 1024;
-            let rows_per_chunk = (target_chunk_bytes / (d * 4)).max(1).min(n);
+            let rows_per_chunk = (target_chunk_bytes / (d * elem_bytes)).max(1).min(n);
            ds.with_chunks(&[rows_per_chunk, d]);
            // Compression. Shuffle is applied automatically (auto-shuffle
@@ -497,6 +504,9 @@ pub fn validate_and_load(
        wal_max_entries: optional_i64_attr(&attrs, "wal_max_entries")
            .and_then(|v| usize::try_from(v).ok())
            .unwrap_or(500),
        // `false`, not the new-store default: a store written before this
        // setting existed was built with an f32 index, and reopening it must
        // not silently change that.
        quantized_index: optional_bool_attr(&attrs, "quantized_index", false),
        hnsw_m: optional_i64_attr(&attrs, "hnsw_m")
            .and_then(|v| usize::try_from(v).ok())
@@ -510,7 +520,16 @@ pub fn validate_and_load(
    };
    // Load /memory group
-    let memory_cache = load_memory_group(file, embedding_dim)?;
+    let mut memory_cache = load_memory_group(file, embedding_dim)?;
    // A float16 store's cache holds half-rounded embeddings. Embeddings read
    // from an f16 dataset already are; a float16 store whose last checkpoint
    // predates half-precision storage is still f32 on disk and is rounded
    // here.
    if config.float16 && embeddings_are_f16(file) {
        memory_cache.half_precision = true;
    } else {
        memory_cache.set_half_precision(config.float16);
    }
    // Load /sessions group
    let session_cache = load_sessions_group(file)?;
@@ -753,6 +772,13 @@ fn read_string_dataset_from_group(
        .map_err(|e| MemoryError::Hdf5(format!("cannot read strings from {name}: {e}")))
 }
 /// Whether `/memory/embeddings` is stored as IEEE half precision.
 fn embeddings_are_f16(file: &clawhdf5::File) -> bool {
    file.dataset("memory/embeddings")
        .and_then(|ds| ds.dtype())
        .is_ok_and(|dt| matches!(dt, clawhdf5::DType::Other(ref s) if s == "float16"))
 }
 fn read_f32_dataset(group: &clawhdf5::Group<'_>, name: &str) -> Result<Vec<f32>, MemoryError> {
    let ds = group
        .dataset(name)
@@ -0,0 +1,208 @@
 //! `MemoryConfig::float16`: embeddings stored as IEEE half precision.
 //!
 //! The setting used to be recorded in `/meta` and otherwise ignored — the
 //! embeddings dataset was always `f32`. These tests pin what it now does: the
 //! dataset is `float16`, the in-memory cache holds exactly the values the file
 //! holds (so search results survive a reopen bit for bit), and a value half
 //! precision cannot represent is refused rather than stored as infinity.
 use std::path::{Path, PathBuf};
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, MemoryError};
 use clawhdf5_format::float16::round_to_f16;
 use tempfile::TempDir;
 const DIM: usize = 64;
 /// Deterministic, embedding-like unit vectors.
 fn embedding(seed: u64) -> Vec<f32> {
    let mut x = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
    let v: Vec<f32> = (0..DIM)
        .map(|_| {
            x ^= x << 13;
            x ^= x >> 7;
            x ^= x << 17;
            (x >> 40) as f32 / (1u64 << 24) as f32 - 0.5
        })
        .collect();
    let norm = v.iter().map(|a| a * a).sum::<f32>().sqrt();
    v.iter().map(|a| a / norm).collect()
 }
 fn entry(i: u64) -> MemoryEntry {
    MemoryEntry {
        chunk: format!("memory number {i} about topic {}", i % 7),
        embedding: embedding(i),
        source_channel: "test".into(),
        timestamp: i as f64,
        session_id: "s".into(),
        tags: format!("t{i}"),
    }
 }
 fn config(dir: &TempDir, name: &str, float16: bool) -> MemoryConfig {
    let mut c = MemoryConfig::new(dir.path().join(name), "agent", DIM);
    c.float16 = float16;
    c
 }
 fn embeddings_dtype_and_values(path: &Path) -> (String, Vec<f32>) {
    let file = clawhdf5::File::open(path).unwrap();
    let ds = file.dataset("memory/embeddings").unwrap();
    (format!("{:?}", ds.dtype().unwrap()), ds.read_f32().unwrap())
 }
 fn search_bits(m: &mut HDF5Memory, q: u64) -> Vec<(usize, u32)> {
    m.hybrid_search(&embedding(q), "memory topic 3", 0.4, 0.6, 10)
        .iter()
        .map(|r| (r.index, r.score.to_bits()))
        .collect()
 }
 #[test]
 fn float16_store_writes_half_precision_and_reopens_identically() {
    let dir = TempDir::new().unwrap();
    // Two identical stores. Search is not read-only (it boosts the Hebbian
    // activation of what it returns, and checkpoints persist that), so each
    // is queried exactly once: one live, one after a checkpoint and reopen.
    let live_cfg = config(&dir, "live.h5", true);
    let cfg = config(&dir, "f16.h5", true);
    let path: PathBuf = cfg.path.clone();
    let mut live = HDF5Memory::create(live_cfg).unwrap();
    live.save_batch((0..200).map(entry).collect()).unwrap();
    let mut m = HDF5Memory::create(cfg).unwrap();
    m.save_batch((0..200).map(entry).collect()).unwrap();
    drop(m);
    // On disk: a genuine float16 dataset holding the rounded inputs.
    let (dtype, values) = embeddings_dtype_and_values(&path);
    assert_eq!(dtype, "Other(\"float16\")");
    let expected: Vec<u32> = (0..200)
        .flat_map(|i| embedding(i).into_iter().map(|v| round_to_f16(v).to_bits()))
        .collect();
    let got: Vec<u32> = values.iter().map(|v| v.to_bits()).collect();
    assert_eq!(got, expected);
    // Reopened, the store answers exactly as the live one does: the cache
    // held the half-rounded values before the checkpoint.
    let mut reopened = HDF5Memory::open(&path).unwrap();
    for q in 0..5 {
        assert_eq!(
            search_bits(&mut live, 1000 + q),
            search_bits(&mut reopened, 1000 + q),
            "query {q}"
        );
    }
 }
 #[test]
 fn float16_halves_the_embeddings_on_disk() {
    let dir = TempDir::new().unwrap();
    let mut sizes = Vec::new();
    for float16 in [false, true] {
        let cfg = config(&dir, &format!("s{float16}.h5"), float16);
        let path = cfg.path.clone();
        let mut m = HDF5Memory::create(cfg).unwrap();
        m.save_batch((0..2000).map(entry).collect()).unwrap();
        drop(m);
        sizes.push(std::fs::metadata(&path).unwrap().len());
    }
    let embedding_bytes_f32 = (2000 * DIM * 4) as u64;
    let saved = sizes[0] - sizes[1];
    // Half of the f32 embeddings, give or take metadata and alignment.
    assert!(
        saved.abs_diff(embedding_bytes_f32 / 2) < 16 * 1024,
        "f32 {} B, f16 {} B, saved {saved} B, expected ~{} B",
        sizes[0],
        sizes[1],
        embedding_bytes_f32 / 2
    );
 }
 #[test]
 fn f32_store_is_unchanged() {
    let dir = TempDir::new().unwrap();
    let cfg = config(&dir, "f32.h5", false);
    let path = cfg.path.clone();
    let mut m = HDF5Memory::create(cfg).unwrap();
    m.save_batch((0..50).map(entry).collect()).unwrap();
    drop(m);
    let (dtype, values) = embeddings_dtype_and_values(&path);
    assert_eq!(dtype, "F32");
    let expected: Vec<f32> = (0..50).flat_map(embedding).collect();
    assert_eq!(values, expected);
 }
 #[test]
 fn out_of_range_values_are_refused_not_stored_as_infinity() {
    let dir = TempDir::new().unwrap();
    let mut cfg = config(&dir, "range.h5", true);
    cfg.wal_enabled = true;
    let path = cfg.path.clone();
    let mut m = HDF5Memory::create(cfg).unwrap();
    m.save(entry(1)).unwrap();
    let mut bad = entry(2);
    bad.embedding[5] = 70_000.0;
    match m.save(bad.clone()) {
        Err(MemoryError::InvalidEntry(msg)) => assert!(msg.contains("embedding[5]"), "{msg}"),
        other => panic!("expected InvalidEntry, got {other:?}"),
    }
    assert!(matches!(
        m.save_or_update(bad.clone()),
        Err(MemoryError::InvalidEntry(_))
    ));
    // A batch is all or nothing.
    assert!(matches!(
        m.save_batch(vec![entry(3), bad.clone(), entry(4)]),
        Err(MemoryError::InvalidEntry(_))
    ));
    assert_eq!(m.count(), 1);
    // The largest finite half, and values that round down to it, are fine.
    let mut edge = entry(5);
    edge.embedding[0] = 65504.0;
    edge.embedding[1] = -65519.0;
    m.save(edge).unwrap();
    assert_eq!(m.count(), 2);
    drop(m);
    // Nothing rejected reached the WAL or the file.
    let m = HDF5Memory::open(&path).unwrap();
    assert_eq!(m.count(), 2);
    // An f32 store takes the same value as it always did.
    let mut m32 = HDF5Memory::create(config(&dir, "range32.h5", false)).unwrap();
    m32.save(bad).unwrap();
 }
 #[test]
 fn wal_replay_rounds_like_a_live_save() {
    let dir = TempDir::new().unwrap();
    let mut cfg = config(&dir, "wal.h5", true);
    cfg.wal_enabled = true;
    cfg.wal_max_entries = 10_000; // keep everything in the WAL
    let path = cfg.path.clone();
    let mut m = HDF5Memory::create(cfg).unwrap();
    for i in 0..30 {
        m.save(entry(i)).unwrap();
    }
    let live = search_bits(&mut m, 77);
    // Crash image: the .h5 is still the empty checkpoint; everything is in
    // the WAL, which holds the caller's f32 values.
    let crash = TempDir::new().unwrap();
    let image = crash.path().join("image.h5");
    std::fs::copy(&path, &image).unwrap();
    std::fs::copy(
        path.with_extension("h5.wal"),
        image.with_extension("h5.wal"),
    )
    .unwrap();
    drop(m);
    let mut recovered = HDF5Memory::open(&image).unwrap();
    assert_eq!(recovered.count(), 30);
    assert_eq!(search_bits(&mut recovered, 77), live);
 }
@@ -0,0 +1,94 @@
 //! An agent store is a standard HDF5 file: h5py can open it and read every
 //! dataset.
 //!
 //! It could not: the float datatype's sign-bit position was hard-coded for
 //! f64, so every f32 dataset (embeddings, norms, activation weights) made
 //! libhdf5 refuse the file with "sign bit position out of bounds".
 use std::process::Command;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry};
 fn python() -> String {
    std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
 }
 fn h5py_available() -> bool {
    Command::new(python())
        .args(["-c", "import h5py"])
        .output()
        .map(|o| o.status.success())
        .unwrap_or(false)
 }
 #[test]
 fn h5py_reads_every_dataset_of_an_agent_store() {
    if !h5py_available() {
        assert!(
            std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
            "CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
        );
        eprintln!("SKIP: python3 with h5py not available");
        return;
    }
    let dir = tempfile::tempdir().unwrap();
    for float16 in [false, true] {
        let path = dir.path().join(format!("store_{float16}.h5"));
        let mut cfg = MemoryConfig::new(path.clone(), "agent", 8);
        cfg.float16 = float16;
        let mut m = HDF5Memory::create(cfg).unwrap();
        // save_batch checkpoints, so the records are in the .h5, not the WAL.
        m.save_batch(
            (0..20)
                .map(|i| MemoryEntry {
                    chunk: format!("memory {i}"),
                    embedding: (0..8).map(|j| ((i * 8 + j) as f32).sin()).collect(),
                    source_channel: "test".into(),
                    timestamp: i as f64,
                    session_id: "s".into(),
                    tags: String::new(),
                })
                .collect(),
        )
        .unwrap();
        drop(m);
        // Exact expected values, as bits: numpy's sin need not match Rust's
        // to the last place.
        let bits = (0..160)
            .map(|k| (k as f32).sin().to_bits().to_string())
            .collect::<Vec<_>>()
            .join(",");
        let script = format!(
            r#"
 import h5py, numpy as np
 want = np.float16 if {py_bool} else np.float32
 with h5py.File("{path}", "r") as f:
    names = []
    f.visititems(lambda n, o: names.append(n) if isinstance(o, h5py.Dataset) else None)
    for n in names:
        f[n][()]  # every dataset must decode
    e = f["memory/embeddings"]
    assert e.dtype == want, e.dtype
    assert e.shape == (20, 8), e.shape
    ref = np.array([{bits}], dtype=np.uint32).view(np.float32).astype(want).reshape(20, 8)
    assert (e[()] == ref).all()
    assert f["memory/norms"].dtype == np.float32
 print(len(names))
 "#,
            py_bool = if float16 { "True" } else { "False" },
            path = path.display()
        );
        let out = Command::new(python())
            .args(["-c", &script])
            .output()
            .unwrap();
        assert!(
            out.status.success(),
            "float16={float16}: {}",
            String::from_utf8_lossy(&out.stderr)
        );
        let n: usize = String::from_utf8_lossy(&out.stdout).trim().parse().unwrap();
        assert!(n >= 10, "only {n} datasets");
    }
 }
@@ -284,3 +284,63 @@ fn degenerate_hnsw_parameters_do_not_panic() {
    let results = mem.hybrid_search(&vectors[7], "", 1.0, 0.0, 5);
    assert_eq!(results[0].index, 7, "exact match should still rank first");
 }
 #[test]
 fn new_stores_default_to_the_quantized_index() {
    // int8 is the default because it is smaller and, with an exact re-score,
    // faster at equal recall on every platform measured (see BENCHMARKS.md).
    let dir = TempDir::new().unwrap();
    let config = MemoryConfig::new(dir.path().join("mem.h5"), "agent", 8);
    assert!(config.quantized_index);
    let path = config.path.clone();
    let mut mem = HDF5Memory::create(config).unwrap();
    let mut seed = 3;
    let vectors: Vec<Vec<f32>> = (0..40).map(|_| make_vector(&mut seed, 8)).collect();
    for (i, v) in vectors.iter().enumerate() {
        mem.save(entry(&format!("c{i}"), v.clone(), "t")).unwrap();
    }
    assert_eq!(
        mem.hybrid_search(&vectors[11], "", 1.0, 0.0, 1)[0].index,
        11
    );
    mem.flush_wal().unwrap();
    drop(mem);
    assert!(HDF5Memory::open(&path).unwrap().config().quantized_index);
 }
 #[test]
 fn a_store_written_before_the_setting_existed_stays_f32() {
    // `store_v2_5_0.h5` was written by the v2.5.0 CLI, before
    // `quantized_index` or the HNSW parameters were persisted, so it carries
    // none of them. Flipping the default for new stores must not reach back
    // and change how an existing store's index is held.
    let dir = TempDir::new().unwrap();
    let path = dir.path().join("legacy.h5");
    std::fs::copy(
        concat!(
            env!("CARGO_MANIFEST_DIR"),
            "/tests/fixtures/store_v2_5_0.h5"
        ),
        &path,
    )
    .unwrap();
    let bytes = std::fs::read(&path).unwrap();
    assert!(
        !bytes.windows(15).any(|w| w == b"quantized_index"),
        "the fixture must predate the setting, or it tests nothing"
    );
    let mut mem = HDF5Memory::open(&path).unwrap();
    assert!(
        !mem.config().quantized_index,
        "an old store must reopen with an f32 index"
    );
    assert_eq!(mem.config().hnsw_m, 16);
    assert_eq!(mem.config().hnsw_ef_construction, 64);
    assert_eq!(mem.count(), 6);
    // And it still searches: entry 3's own embedding finds it first.
    let hit = mem.hybrid_search(&[3.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "", 1.0, 0.0, 1);
    assert_eq!(hit[0].index, 3);
 }
@@ -2,6 +2,7 @@
 name = "clawhdf5-android"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "Android JNI bridge for edgehdf5-memory HDF5 backend"
 license = "MIT"
@@ -2,6 +2,7 @@
 name = "clawhdf5-ann"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "HNSW approximate nearest neighbor index stored as HDF5"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -2,6 +2,7 @@
 name = "clawhdf5-bench"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "Benchmark harnesses for clawhdf5-agent (Track 8)"
 license = "MIT"
@@ -19,6 +19,7 @@
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --full  # + 100K
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --json out.json
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --ann-only --uniform
 //! cargo run --release -p clawhdf5-bench --bin search_harness -- --float16-study --full
 //! ```
 use std::time::{Duration, Instant};
@@ -88,6 +89,9 @@ static UNIFORM: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::n
 /// the memory) instead of f32, to price the recall it costs.
 static INT8: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
 /// `--f16-first`: in `--float16-study`, run the float16 store first.
 static F16_FIRST: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
 /// `--rerank`: re-score the candidate pool against the exact vectors before
 /// taking the top K.
 static RERANK: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
@@ -483,6 +487,148 @@ fn bench_end_to_end(n: usize, json: &mut Vec<serde_json::Value>) {
    }));
 }
 // ---------------------------------------------------------------------------
 // float16 study: what does half-precision embedding storage cost?
 // ---------------------------------------------------------------------------
 /// `--float16-study`: the same data in an `f32` store and a `float16` store.
 /// Reports file size, checkpoint and open time, vector-search recall@10
 /// against an exact scan of the *original* f32 vectors, how often the two
 /// stores return the same top 10, and `hybrid_search` latency. Hebbian
 /// boosting is off, so every query sees the same store.
 fn float16_study(n: usize) {
    let data = make_dataset(n, 0xF16 ^ n as u64);
    let mut rng = Rng(11);
    let query_texts: Vec<String> = data
        .query_cluster
        .iter()
        .enumerate()
        .map(|(i, c)| text_for(*c, i, &mut rng))
        .collect();
    // Exact top K by cosine (the vectors are unit length) on the f32 inputs.
    let exact: Vec<Vec<usize>> = data
        .queries
        .iter()
        .map(|q| {
            let mut scored: Vec<(usize, f32)> = data
                .vectors
                .iter()
                .enumerate()
                .map(|(i, v)| (i, v.iter().zip(q).map(|(a, b)| a * b).sum()))
                .collect();
            scored.sort_by(|a, b| b.1.total_cmp(&a.1).then(a.0.cmp(&b.0)));
            scored.into_iter().take(K).map(|(i, _)| i).collect()
        })
        .collect();
    let dir = tempfile::tempdir().unwrap();
    let mut per_variant: Vec<(bool, Vec<Vec<usize>>)> = Vec::new();
    // `--f16-first` swaps the order, to check the numbers do not depend on
    // which store runs first (page cache, allocator, CPU frequency).
    let order = if F16_FIRST.load(std::sync::atomic::Ordering::Relaxed) {
        [true, false]
    } else {
        [false, true]
    };
    for float16 in order {
        let path = dir.path().join(format!("f16study_{float16}.h5"));
        let mut rng = Rng(3);
        let entries: Vec<MemoryEntry> = data
            .vectors
            .iter()
            .enumerate()
            .map(|(i, v)| MemoryEntry {
                chunk: text_for(data.cluster_of[i], i, &mut rng),
                embedding: v.clone(),
                source_channel: "bench".into(),
                timestamp: i as f64,
                session_id: format!("s{}", i % 50),
                tags: format!("t{i}"),
            })
            .collect();
        let mut config = MemoryConfig::new(path.clone(), "bench", DIM);
        config.float16 = float16;
        config.hebbian_boost = 0.0;
        let mut mem = HDF5Memory::create(config).unwrap();
        mem.save_batch(entries).unwrap();
        // Build the indexes, then time a checkpoint that writes everything.
        std::hint::black_box(mem.hybrid_search(&data.queries[0], "", 1.0, 0.0, K));
        let t = Instant::now();
        mem.flush_wal().unwrap();
        let checkpoint = t.elapsed();
        drop(mem);
        let file_bytes = std::fs::metadata(&path).unwrap().len();
        // Median of three opens.
        let mut opens: Vec<Duration> = (0..3)
            .map(|_| {
                let t = Instant::now();
                let m = HDF5Memory::open(&path).unwrap();
                let d = t.elapsed();
                drop(m);
                d
            })
            .collect();
        opens.sort();
        let mut mem = HDF5Memory::open(&path).unwrap();
        // Vector-only search: empty text, all weight on the vector stage.
        let results: Vec<Vec<usize>> = data
            .queries
            .iter()
            .map(|q| {
                mem.hybrid_search(q, "", 1.0, 0.0, K)
                    .iter()
                    .map(|r| r.index)
                    .collect()
            })
            .collect();
        let hits: usize = results
            .iter()
            .zip(&exact)
            .map(|(got, want)| got.iter().filter(|i| want.contains(i)).count())
            .sum();
        let recall = hits as f64 / (K * data.queries.len()) as f64;
        let latency = summarize(
            (0..N_QUERIES)
                .map(|i| {
                    let t = Instant::now();
                    std::hint::black_box(mem.hybrid_search(
                        &data.queries[i],
                        &query_texts[i],
                        0.4,
                        0.6,
                        K,
                    ));
                    t.elapsed()
                })
                .collect(),
        );
        let overlap = match per_variant.first() {
            Some((_, other)) => {
                let same: usize = results
                    .iter()
                    .zip(other)
                    .map(|(a, b)| a.iter().filter(|i| b.contains(i)).count())
                    .sum();
                format!("{:.4}", same as f64 / (K * data.queries.len()) as f64)
            }
            None => "—".into(),
        };
        println!(
            "| {n} | {} | {:.1} | {:.0} | {:.1} | {recall:.4} | {overlap} | {:.3} |",
            if float16 { "float16" } else { "f32" },
            mib(file_bytes),
            millis(checkpoint),
            millis(opens[1]),
            millis(latency.p50),
        );
        per_variant.push((float16, results));
    }
 }
 // ---------------------------------------------------------------------------
 // Fusion study: does capping the keyword candidate pool change the ranking?
 // ---------------------------------------------------------------------------
@@ -642,6 +788,24 @@ fn main() {
        }
        return;
    }
    if args.iter().any(|a| a == "--f16-first") {
        F16_FIRST.store(true, std::sync::atomic::Ordering::Relaxed);
    }
    if args.iter().any(|a| a == "--float16-study") {
        println!("## float16 embedding storage ({DIM}-dim, int8 index, Hebbian boost off)\n");
        println!(
            "| N | embeddings | file MiB | checkpoint ms | open ms | recall@10 | top-10 overlap with the other | hybrid p50 ms |"
        );
        println!("|---:|---|---:|---:|---:|---:|---:|---:|");
        for &n in if full {
            &[1_000, 10_000, 100_000][..]
        } else {
            &[1_000, 10_000][..]
        } {
            float16_study(n);
        }
        return;
    }
    if args.iter().any(|a| a == "--int8") {
        INT8.store(true, std::sync::atomic::Ordering::Relaxed);
        println!("(int8-quantised index vectors)");
@@ -2,6 +2,7 @@
 name = "clawhdf5-cli"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 license = "MIT"
 description = "CLI for clawhdf5 agent memory — create, save, search, recall, stats"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -28,10 +28,19 @@ enum Commands {
        /// Enable write-ahead log
        #[arg(long)]
        wal: bool,
-        /// Store the vector index's copy of the embeddings as int8, roughly
+        /// Hold the vector index's copy of the embeddings as f32 instead of
-        /// halving a loaded store's memory at about 13% fewer queries/second
+        /// the default int8 (which uses a quarter of the memory and is faster
        /// at equal recall)
        #[arg(long)]
        f32_index: bool,
        /// Accepted for compatibility; int8 is now the default
        #[arg(long, hide = true, conflicts_with = "f32_index")]
        quantized_index: bool,
        /// Store embeddings on disk as IEEE half precision (float16): half
        /// the bytes, about three significant digits; values must lie within
        /// ±65504
        #[arg(long)]
        float16: bool,
    },
    /// Save a memory entry (reads JSON from stdin or --json)
    Save {
@@ -96,11 +105,20 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
            agent_id,
            dim,
            wal,
-            quantized_index,
+            f32_index,
            quantized_index: _,
            float16,
        } => {
            let mut config = MemoryConfig::new(cli.path.clone(), &agent_id, dim);
            config.wal_enabled = wal;
-            config.quantized_index = quantized_index;
+            config.float16 = float16;
            // Only ever switch *off* the library default: assigning the flag
            // outright would force every CLI-created store back to f32 unless
            // the caller knew to ask for int8.
            if f32_index {
                config.quantized_index = false;
            }
            let config_quantized = config.quantized_index;
            let mem = HDF5Memory::create(config)?;
            let j = serde_json::json!({
                "status": "created",
@@ -108,7 +126,8 @@ fn run(cli: Cli) -> Result<(), Box<dyn std::error::Error>> {
                "agent_id": agent_id,
                "embedding_dim": dim,
                "wal_enabled": wal,
-                "quantized_index": quantized_index,
+                "quantized_index": config_quantized,
                "float16": float16,
                "count": mem.count(),
            });
            println!("{}", serde_json::to_string_pretty(&j)?);
@@ -2,6 +2,7 @@
 name = "clawhdf5-derive"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "Derive macros for rustyhdf5 HDF5 traits"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -2,6 +2,7 @@
 name = "clawhdf5-filters"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "Filter and compression pipeline for clawhdf5"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -25,8 +26,12 @@ name = "compression_bench"
 harness = false
 [features]
-default = ["fast-deflate"]
+# Pure-Rust zlib-rs by default; `fast-deflate` (zlib-ng, C) overrides it.
 default = ["zlib-rs"]
 fast-deflate = ["flate2/zlib-ng"]
 system-zlib = ["flate2/zlib-default"]
-zlib-rs = ["flate2/zlib-rs"]
+# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
 # SIMD at runtime. flate2 enables it by default, but we build flate2 with
 # default-features = false, and without it zlib-rs inflates 3.5x slower.
 zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
 apple-compression = []
@@ -8,16 +8,18 @@ Filter and compression pipeline for clawhdf5.
 ## Features
 - DEFLATE compression/decompression
- Fast deflate via zlib-ng (`fast-deflate` feature)
+- Pure-Rust deflate via zlib-rs (default, `zlib-rs` feature)
 - zlib-ng instead, if you want it (`fast-deflate` feature; C, needs cmake)
 - Apple Compression framework support (`apple-compression` feature)
 ## Usage
 ```rust
-use clawhdf5_filters::{deflate_decode, deflate_encode};
+use clawhdf5_filters::{deflate_compress, deflate_decompress};
-let compressed = deflate_encode(&data, 6).unwrap();
+let compressed = deflate_compress(&data, 6).unwrap();
-let decompressed = deflate_decode(&compressed).unwrap();
+// The second argument bounds the output: the expected decompressed size.
 let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
 ```
 ## License
@@ -1,12 +1,13 @@
-//! Fast deflate backends: Apple Compression Framework and zlib-ng.
+//! Deflate backends: Apple Compression Framework, zlib-ng and zlib-rs.
 //!
 //! Backend selection priority (decompression & compression):
 //! 1. Apple Compression Framework (macOS only, `apple-compression` feature)
-//! 2. flate2 with zlib-ng backend (`fast-deflate` feature) or miniz_oxide (default)
+//! 2. flate2 with zlib-ng (`fast-deflate`), else zlib-rs (`zlib-rs`, the
 //!    default), else miniz_oxide
 //!
 //! The Apple Compression Framework uses hardware-accelerated zlib on Apple Silicon
-//! and is typically the fastest option on macOS. zlib-ng is the fastest portable
+//! and is typically the fastest option on macOS. zlib-rs is a pure-Rust port of
-//! option and what C HDF5 uses internally.
+//! zlib-ng; see `BENCHMARKS.md` for how the two compare.
 // ---------------------------------------------------------------------------
 // Apple Compression Framework FFI (macOS only)
@@ -243,65 +244,117 @@ mod apple {
 }
 // ---------------------------------------------------------------------------
-// Streaming decompression via flate2 (uses zlib-ng when fast-deflate enabled)
+// One-shot (de)compression via flate2 (whichever backend flate2 was built with)
 //
 // The whole input goes to the codec in one call, into an output buffer sized
 // up front. `flate2::read::ZlibDecoder` / `write::ZlibEncoder` stream through a
 // 32 KiB buffer instead, which cost zlib-rs up to 3.7x against zlib-ng on a
 // 1 MB chunk. clawhdf5-format's deflate filter does the same; see
 // `BENCHMARKS.md`, "Deflate backend".
 // ---------------------------------------------------------------------------
-/// Streaming decompress with pre-allocated output buffer.
+/// Decompress into a buffer pre-sized to `output_size`, the expected
-///
+/// decompressed length (known for HDF5 chunks). Output longer than that is an
-/// When the output size is known (typical for HDF5 chunks), this avoids
+/// error, as is a stream that ends early.
 /// dynamic reallocation by writing directly into a pre-sized buffer.
 pub(crate) fn flate2_decompress_preallocated(
    data: &[u8],
    output_size: usize,
 ) -> Result<Vec<u8>, String> {
-    use std::io::Read;
+    inflate_bounded(data, output_size, output_size)
    let mut decoder = flate2::read::ZlibDecoder::new(data);
    let mut output = vec![0u8; output_size];
    let mut total_read = 0;
    loop {
        match decoder.read(&mut output[total_read..]) {
            Ok(0) => break,
            Ok(n) => total_read += n,
            Err(e) => return Err(e.to_string()),
        }
    }
    output.truncate(total_read);
    Ok(output)
 }
 /// Absolute ceiling on decompressed output when the caller has no size hint,
 /// preventing unbounded allocation from a hostile/corrupted zlib stream.
 const MAX_DECOMPRESS_SIZE: usize = 256 * 1024 * 1024;
-/// Streaming decompress with dynamic sizing (when output size is unknown).
+/// Decompress with no size hint, bounded by [`MAX_DECOMPRESS_SIZE`] so a
-///
+/// hostile zlib stream cannot force arbitrarily large allocation (a "zlib
-/// Bounded by [`MAX_DECOMPRESS_SIZE`] since there is no chunk-size hint to
+/// bomb").
 /// validate against here — an unbounded `read_to_end` would let a hostile
 /// zlib stream force arbitrarily large allocation (a "zlib bomb").
 pub(crate) fn flate2_decompress_streaming(data: &[u8]) -> Result<Vec<u8>, String> {
-    use std::io::Read;
+    let hint = data.len().saturating_mul(4).min(1 << 20);
-    let decoder = flate2::read::ZlibDecoder::new(data);
+    inflate_bounded(data, hint, MAX_DECOMPRESS_SIZE).map_err(|e| {
-    let mut result = Vec::new();
+        if e.ends_with("exceeds size limit") {
-    decoder
+            format!(
-        .take(MAX_DECOMPRESS_SIZE as u64 + 1)
+                "decompressed output exceeds {} MiB limit",
-        .read_to_end(&mut result)
+                MAX_DECOMPRESS_SIZE / 1024 / 1024
-        .map_err(|e| e.to_string())?;
+            )
-    if result.len() > MAX_DECOMPRESS_SIZE {
+        } else {
-        return Err(format!(
+            e
-            "decompressed output exceeds {} MiB limit",
+        }
-            MAX_DECOMPRESS_SIZE / 1024 / 1024
+    })
        ));
    }
    Ok(result)
 }
-/// Compress data using flate2 (zlib-ng when fast-deflate enabled, else miniz_oxide).
+/// Inflate a zlib stream, starting from `size_hint` bytes of output and
 /// failing past `limit`.
 fn inflate_bounded(data: &[u8], size_hint: usize, limit: usize) -> Result<Vec<u8>, String> {
    use flate2::{Decompress, FlushDecompress, Status};
    // One byte of headroom past the limit distinguishes an over-size stream
    // from one that legitimately ends exactly at the limit.
    let max_capacity = limit.saturating_add(1);
    let mut out = Vec::new();
    out.try_reserve_exact(size_hint.clamp(1, max_capacity))
        .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
    let mut inflater = Decompress::new(true);
    loop {
        let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
        let status = inflater
            .decompress_vec(
                &data[in_before as usize..],
                &mut out,
                FlushDecompress::Finish,
            )
            .map_err(|e| format!("deflate: {e}"))?;
        if out.len() > limit {
            return Err("deflate: output exceeds size limit".into());
        }
        match status {
            Status::StreamEnd => return Ok(out),
            Status::Ok | Status::BufError if out.len() == out.capacity() => {
                let grow = out.capacity().min(max_capacity - out.capacity()).max(1);
                out.try_reserve_exact(grow)
                    .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
            }
            Status::Ok | Status::BufError => {
                if inflater.total_in() as usize >= data.len()
                    || (inflater.total_in(), inflater.total_out()) == (in_before, out_before)
                {
                    return Err("deflate: truncated stream".into());
                }
            }
        }
    }
 }
 /// Compress data using flate2 (zlib-ng, zlib-rs or miniz_oxide; see module docs).
 pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
-    use std::io::Write;
+    use flate2::{Compress, Compression, FlushCompress, Status};
-    let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(level));
+
-    encoder.write_all(data).map_err(|e| e.to_string())?;
+    // zlib's compressBound, plus the zlib header and trailer.
-    encoder.finish().map_err(|e| e.to_string())
+    let bound = data.len() + (data.len() >> 12) + (data.len() >> 14) + (data.len() >> 25) + 13 + 6;
    let mut out = Vec::new();
    out.try_reserve_exact(bound)
        .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
    let mut deflater = Compress::new(Compression::new(level), true);
    loop {
        let (in_before, out_before) = (deflater.total_in(), deflater.total_out());
        let status = deflater
            .compress_vec(&data[in_before as usize..], &mut out, FlushCompress::Finish)
            .map_err(|e| format!("deflate: {e}"))?;
        match status {
            Status::StreamEnd => return Ok(out),
            Status::Ok | Status::BufError if out.len() == out.capacity() => out
                .try_reserve(out.capacity().max(4096))
                .map_err(|e| format!("deflate: cannot allocate output: {e}"))?,
            Status::Ok | Status::BufError => {
                if (deflater.total_in(), deflater.total_out()) == (in_before, out_before) {
                    return Err("deflate: encoder made no progress".into());
                }
            }
        }
    }
 }
 // ---------------------------------------------------------------------------
@@ -312,7 +365,7 @@ pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String
 ///
 /// Selection order:
 /// 1. Apple Compression Framework (macOS + `apple-compression` feature)
-/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
+/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
 ///
 /// When `output_hint` > 0, pre-allocates the output buffer for zero-copy
 /// decompression (avoids reallocation).
@@ -344,7 +397,7 @@ pub fn decompress(data: &[u8], output_hint: usize) -> Result<Vec<u8>, String> {
 ///
 /// Selection order:
 /// 1. Apple Compression Framework (macOS + `apple-compression` feature)
-/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
+/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
 pub fn compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
    #[cfg(all(target_os = "macos", feature = "apple-compression"))]
    {
@@ -377,9 +430,19 @@ pub fn active_backend() -> &'static str {
    {
        "zlib-ng"
    }
    // flate2 prefers a C zlib over zlib-rs when both are enabled.
    #[cfg(all(
        not(all(target_os = "macos", feature = "apple-compression")),
        not(feature = "fast-deflate"),
        feature = "zlib-rs"
    ))]
    {
        "zlib-rs"
    }
    #[cfg(not(any(
        all(target_os = "macos", feature = "apple-compression"),
-        feature = "fast-deflate"
+        feature = "fast-deflate",
        feature = "zlib-rs"
    )))]
    {
        "miniz_oxide"
@@ -436,7 +499,7 @@ mod tests {
    fn backend_name_is_set() {
        let name = active_backend();
        assert!(
-            ["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
+            ["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
            "unexpected backend: {name}"
        );
    }
@@ -2,12 +2,14 @@
 //!
 //! Provides deflate (zlib) decompression/compression with multiple backend options:
 //!
-//! - **Default**: `miniz_oxide` (pure Rust, no C dependencies)
+//! - **Default (`zlib-rs` feature)**: `zlib-rs` via flate2 (pure Rust, no C
-//! - **`fast-deflate` feature**: `zlib-ng` via flate2 (~2-3x faster, matches C HDF5)
+//!   dependencies)
 //! - **`fast-deflate` feature**: `zlib-ng` via flate2 (C, built with cmake)
 //! - **`apple-compression` feature**: Apple Compression Framework on macOS
 //!   (hardware-accelerated on Apple Silicon)
 //! - With none of the above: `miniz_oxide` (pure Rust, slower)
 //!
-//! Backend priority: apple-compression > zlib-ng > miniz_oxide.
+//! Backend priority: apple-compression > zlib-ng > zlib-rs > miniz_oxide.
 pub mod fast_deflate;
@@ -115,7 +117,7 @@ mod tests {
    fn backend_reports_name() {
        let name = deflate_backend();
        assert!(
-            ["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
+            ["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
            "unexpected backend: {name}"
        );
    }
@@ -2,6 +2,7 @@
 name = "clawhdf5-format"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "Pure-Rust HDF5 binary format parsing and writing — no C dependencies"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -23,6 +24,7 @@ libaec-sys = { path = "../libaec-sys", version = "0.1", optional = true }
 pco = { version = "1.0", optional = true }
 [dev-dependencies]
 half = { workspace = true }
 serde_json = "1"
 criterion = { workspace = true }
 clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.7.0" }
@@ -32,7 +34,10 @@ name = "bench"
 harness = false
 [features]
-default = ["std", "checksum", "deflate", "provenance", "fast-deflate", "system-zlib-decompress"]
+# Deflate backend: `zlib-rs` (pure Rust) by default. `fast-deflate` selects
 # zlib-ng instead (C, built with cmake); flate2 prefers a C zlib whenever one
 # is enabled, so turning it on anywhere in the build overrides the default.
 default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress"]
 std = []
 checksum = []
 deflate = ["flate2"]
@@ -42,7 +47,10 @@ fast-checksum = ["crc32fast"]
 fast-deflate = ["flate2/zlib-ng"]
 system-zlib = ["flate2/zlib-default"]
 system-zlib-decompress = []
-zlib-rs = ["flate2/zlib-rs"]
+# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
 # SIMD at runtime. flate2 enables it by default, but we build flate2 with
 # default-features = false, and without it zlib-rs inflates 3.5x slower.
 zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
 lz4 = ["lz4_flex"]
 zstd = ["dep:zstd"]
 blake3_hash = ["blake3"]
@@ -1076,6 +1076,21 @@ pub fn read_as_f32(raw: &[u8], datatype: &Datatype) -> Result<Vec<f32>, FormatEr
    ) {
        return Ok(native_le_to_vec::<f32>(raw, count));
    }
    // Little-endian half precision (numpy float16): widen directly.
    if matches!(
        datatype,
        Datatype::FloatingPoint {
            size: 2,
            byte_order: DatatypeByteOrder::LittleEndian,
            ..
        }
    ) {
        let (halves, _) = raw[..count * 2].as_chunks::<2>();
        return Ok(halves
            .iter()
            .map(|&b| f16_bits_to_f32(u16::from_le_bytes(b)))
            .collect());
    }
    let order = get_byte_order(datatype);
    let mut result = Vec::with_capacity(count);
@@ -1622,36 +1637,7 @@ fn read_f16_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f32 {
    f16_bits_to_f32(u16::from_le_bytes(buf))
 }
-/// Convert the bit pattern of an IEEE-754 half (binary16) to an `f32`.
+use crate::float16::f16_bits_to_f32;
 fn f16_bits_to_f32(h: u16) -> f32 {
    let h = h as u32;
    let sign = (h & 0x8000) << 16;
    let exp = (h >> 10) & 0x1f;
    let mant = h & 0x3ff;
    let bits = if exp == 0 {
        if mant == 0 {
            sign // signed zero
        } else {
            // Subnormal: normalize into an f32 normal.
            let mut e: i32 = -1;
            let mut m = mant;
            loop {
                e += 1;
                m <<= 1;
                if m & 0x400 != 0 {
                    break;
                }
            }
            let m = m & 0x3ff;
            sign | (((127 - 15 - e) as u32) << 23) | (m << 13)
        }
    } else if exp == 0x1f {
        sign | 0x7f80_0000 | (mant << 13) // inf / NaN
    } else {
        sign | ((exp + (127 - 15)) << 23) | (mant << 13)
    };
    f32::from_bits(bits)
 }
 fn read_f32_bytes(bytes: &[u8], order: &DatatypeByteOrder) -> f32 {
    let mut buf = [0u8; 4];
@@ -640,7 +640,8 @@ impl Datatype {
                mantissa_size,
                exponent_bias,
            } => {
-                let mut bf0 = 0x20u8; // bit 5: sign location bit (standard IEEE 754)
+                // Bits 4-5: mantissa normalization = 2 (implied leading 1, IEEE 754).
                let mut bf0 = 0x20u8;
                match byte_order {
                    DatatypeByteOrder::BigEndian => {
                        bf0 |= 0x01;
@@ -650,9 +651,14 @@ impl Datatype {
                    }
                    _ => {}
                }
-                // bf[1] bits 0-1: mantissa normalization = 2 (MSB not stored, IEEE 754)
+                // Bits 8-15: the sign bit's position, the top bit of the value.
-                let bf1 = 0x3fu8; // matching what h5py generates
+                // This was hard-coded to 63, which is right only for f64: the
-                let mut buf = Self::build_header(1, 1, [bf0, bf1, 0], *size);
+                // HDF5 library rejects any other float with "sign bit position
                // out of bounds", so every f32 dataset and attribute we wrote
                // was unreadable by h5py and libhdf5.
                let sign_location =
                    (u32::from(*bit_offset) + u32::from(*bit_precision)).saturating_sub(1) as u8;
                let mut buf = Self::build_header(1, 1, [bf0, sign_location, 0], *size);
                buf.extend_from_slice(&bit_offset.to_le_bytes());
                buf.extend_from_slice(&bit_precision.to_le_bytes());
                buf.push(*exponent_location);
@@ -818,6 +824,24 @@ fn build_dt_header(class: u8, version: u8, bf: [u8; 3], size: u32) -> Vec<u8> {
 mod tests {
    use super::*;
    #[test]
    fn float_sign_location_is_the_top_bit_of_the_value() {
        // The HDF5 library rejects a float whose sign position is not inside
        // its precision; this was hard-coded to 63, so every f32 we wrote was
        // unreadable by h5py. Byte 2 of the message is the sign position.
        use crate::type_builders::{make_f16_type, make_f32_type, make_f64_type};
        for (dt, sign) in [
            (make_f16_type(), 15),
            (make_f32_type(), 31),
            (make_f64_type(), 63),
        ] {
            let bytes = dt.serialize();
            assert_eq!(bytes[2], sign, "{dt:?}");
            let (parsed, _) = Datatype::parse(&bytes).unwrap();
            assert_eq!(parsed, dt);
        }
    }
    // Helper to build a fixed-point datatype message
    fn build_fixed_point(
        size: u32,
@@ -86,6 +86,12 @@ pub(crate) fn build_dataset_oh(
    let mut dl = Vec::new();
    dl.push(4); // version
    dl.push(1); // class = contiguous
    // An empty dataset has no storage: its address must be the undefined
    // address, as libhdf5 writes it. A real address with size 0 trips
    // libhdf5's `addr + size <= addr` overflow check, and it refuses the
    // dataset as "invalid dataset size, likely file corruption" — which made
    // every store with no sessions or knowledge graph unreadable by h5py.
    let data_addr = if data_size == 0 { u64::MAX } else { data_addr };
    dl.extend_from_slice(&data_addr.to_le_bytes());
    dl.extend_from_slice(&data_size.to_le_bytes());
    w.add_message(MessageType::DataLayout, dl);
@@ -629,21 +629,70 @@ fn deflate_decompress(data: &[u8], expected_bytes: usize) -> Result<Vec<u8>, For
        // Fall through to flate2 on error
    }
-    use std::io::Read;
+    // A chunk's decompressed size is known, so allocate it once; without one,
-    let decoder = flate2::read::ZlibDecoder::new(data);
+    // start from a multiple of the input and grow.
-    let mut result = Vec::with_capacity(limit.min(1 << 20));
+    let size_hint = if expected_bytes != 0 {
-    // Read one byte past the limit so an over-size stream is distinguishable
+        expected_bytes
    } else {
        data.len().saturating_mul(4).min(1 << 20)
    };
    inflate_bounded(data, size_hint, limit).map_err(FormatError::DecompressionError)
 }
 /// Inflate a zlib stream into a buffer sized up front, handing the decoder the
 /// whole input at once.
 ///
 /// `flate2::read::ZlibDecoder` feeds its input through a 32 KiB buffer and
 /// grows the output as it goes; on single chunks that cost zlib-rs up to 3.7x
 /// against zlib-ng (`BENCHMARKS.md`, "Deflate backend"). Output beyond `limit`
 /// is an error, as is a stream that ends before its end-of-stream marker (the
 /// streaming reader returned the bytes it had and no error).
 #[cfg(feature = "deflate")]
 pub(crate) fn inflate_bounded(
    data: &[u8],
    size_hint: usize,
    limit: usize,
 ) -> Result<Vec<u8>, String> {
    use flate2::{Decompress, FlushDecompress, Status};
    // One byte of headroom past the limit distinguishes an over-size stream
    // from one that legitimately ends exactly at the limit.
-    decoder
+    let max_capacity = limit.saturating_add(1);
-        .take(limit as u64 + 1)
+    let mut out = Vec::new();
-        .read_to_end(&mut result)
+    out.try_reserve_exact(size_hint.clamp(1, max_capacity))
-        .map_err(|e| FormatError::DecompressionError(e.to_string()))?;
+        .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
-    if result.len() > limit {
+
-        return Err(FormatError::DecompressionError(
+    let mut inflater = Decompress::new(true);
-            "deflate: output exceeds size limit".into(),
+    loop {
-        ));
+        let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
        let status = inflater
            .decompress_vec(
                &data[in_before as usize..],
                &mut out,
                FlushDecompress::Finish,
            )
            .map_err(|e| format!("deflate: {e}"))?;
        if out.len() > limit {
            return Err("deflate: output exceeds size limit".into());
        }
        match status {
            Status::StreamEnd => return Ok(out),
            Status::Ok | Status::BufError if out.len() == out.capacity() => {
                // Out of room: double, up to the limit.
                let grow = out.capacity().min(max_capacity - out.capacity()).max(1);
                out.try_reserve_exact(grow)
                    .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
            }
            Status::Ok | Status::BufError => {
                // Room left, so the decoder stopped for want of input.
                if inflater.total_in() as usize >= data.len()
                    || (inflater.total_in(), inflater.total_out()) == (in_before, out_before)
                {
                    return Err("deflate: truncated stream".into());
                }
            }
        }
    }
    Ok(result)
 }
 /// Direct FFI to Apple's system libz for fast decompression.
@@ -722,14 +771,41 @@ fn deflate_decompress(_data: &[u8], _expected_bytes: usize) -> Result<Vec<u8>, F
 /// Compress data with zlib.
 #[cfg(feature = "deflate")]
 fn deflate_compress(data: &[u8], level: u32) -> Result<Vec<u8>, FormatError> {
-    use std::io::Write;
+    deflate_bounded(data, level).map_err(FormatError::CompressionError)
-    let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(level));
+}
-    encoder
+
-        .write_all(data)
+/// Deflate `data` into a zlib stream in one pass, into a buffer sized for the
-        .map_err(|e| FormatError::CompressionError(e.to_string()))?;
+/// worst case up front (the same reasoning as [`inflate_bounded`]).
-    encoder
+#[cfg(feature = "deflate")]
-        .finish()
+pub(crate) fn deflate_bounded(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
-        .map_err(|e| FormatError::CompressionError(e.to_string()))
+    use flate2::{Compress, Compression, FlushCompress, Status};
    // zlib's compressBound, plus the zlib header and trailer.
    let bound = data.len() + (data.len() >> 12) + (data.len() >> 14) + (data.len() >> 25) + 13 + 6;
    let mut out = Vec::new();
    out.try_reserve_exact(bound)
        .map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
    let mut deflater = Compress::new(Compression::new(level), true);
    loop {
        let (in_before, out_before) = (deflater.total_in(), deflater.total_out());
        let status = deflater
            .compress_vec(&data[in_before as usize..], &mut out, FlushCompress::Finish)
            .map_err(|e| format!("deflate: {e}"))?;
        match status {
            Status::StreamEnd => return Ok(out),
            // The bound should make running out of room unreachable; grow
            // rather than fail if it happens.
            Status::Ok | Status::BufError if out.len() == out.capacity() => out
                .try_reserve(out.capacity().max(4096))
                .map_err(|e| format!("deflate: cannot allocate output: {e}"))?,
            Status::Ok | Status::BufError => {
                if (deflater.total_in(), deflater.total_out()) == (in_before, out_before) {
                    return Err("deflate: encoder made no progress".into());
                }
            }
        }
    }
 }
 #[cfg(not(feature = "deflate"))]
@@ -1833,6 +1909,74 @@ mod tests {
        assert!(deflate_decompress(&compressed, 64).is_err());
    }
    #[cfg(feature = "deflate")]
    fn noisy_bytes(n: usize) -> Vec<u8> {
        // Compressible but not trivially so.
        (0..n)
            .map(|i| ((i as f64 * 0.01).sin() * 127.0 + 128.0) as u8 ^ (i as u8 & 3))
            .collect()
    }
    #[test]
    #[cfg(feature = "deflate")]
    fn deflate_decompress_accepts_output_exactly_at_chunk_size() {
        let data = noisy_bytes(100_000);
        let compressed = deflate_compress(&data, 6).unwrap();
        assert_eq!(deflate_decompress(&compressed, data.len()).unwrap(), data);
        // One byte short of the real size is over the limit.
        assert!(deflate_decompress(&compressed, data.len() - 1).is_err());
    }
    #[test]
    #[cfg(feature = "deflate")]
    fn deflate_decompress_without_size_grows_the_buffer() {
        // No chunk size: the output starts at 4x the input and has to grow.
        let data = vec![7u8; 3 * 1024 * 1024];
        let compressed = deflate_compress(&data, 6).unwrap();
        assert!(compressed.len() * 4 < data.len());
        assert_eq!(deflate_decompress(&compressed, 0).unwrap(), data);
    }
    #[test]
    #[cfg(feature = "deflate")]
    fn deflate_decompress_rejects_truncated_stream() {
        // The streaming reader this replaced returned the bytes it had and no
        // error, so a truncated chunk read back short.
        let data = noisy_bytes(100_000);
        let compressed = deflate_compress(&data, 6).unwrap();
        for cut in [compressed.len() - 1, compressed.len() / 2, 3] {
            assert!(
                deflate_decompress(&compressed[..cut], data.len()).is_err(),
                "truncated to {cut} of {} bytes",
                compressed.len()
            );
        }
    }
    #[test]
    #[cfg(feature = "deflate")]
    fn deflate_compress_roundtrips_incompressible_data() {
        // Random-looking input compresses to slightly more than it started
        // as; the output must still fit the pre-sized buffer (or grow).
        let mut x = 0x9E37_79B9_7F4A_7C15u64;
        let data: Vec<u8> = (0..200_000)
            .map(|_| {
                x ^= x << 13;
                x ^= x >> 7;
                x ^= x << 17;
                x as u8
            })
            .collect();
        for level in [0, 1, 6, 9] {
            let compressed = deflate_compress(&data, level).unwrap();
            assert_eq!(deflate_decompress(&compressed, data.len()).unwrap(), data);
        }
        assert_eq!(
            deflate_decompress(&deflate_compress(&[], 6).unwrap(), 0).unwrap(),
            Vec::<u8>::new()
        );
    }
    #[test]
    #[cfg(feature = "zstd")]
    fn zstd_decompress_rejects_output_exceeding_chunk_size() {
@@ -0,0 +1,155 @@
 //! IEEE-754 half precision (binary16) conversions.
 //!
 //! Pure integer bit manipulation, so it works under `no_std` and needs no
 //! `libm`. The writer ([`crate::type_builders::DatasetBuilder::with_f16_data`]),
 //! the reader and `clawhdf5-agent`'s half-precision embedding store all use
 //! these two functions, so a value rounded in memory is bit-for-bit the value
 //! that reads back from the file.
 /// Largest finite half-precision value. Anything larger in magnitude rounds
 /// to infinity.
 pub const F16_MAX: f32 = 65504.0;
 /// Convert an `f32` to the bit pattern of the nearest half-precision value,
 /// rounding ties to even (the IEEE default, and what numpy and the `half`
 /// crate do).
 ///
 /// Values beyond ±[`F16_MAX`] become ±infinity, values too small for a
 /// subnormal become signed zero, and NaN stays NaN (quiet, payload
 /// truncated).
 pub fn f32_to_f16_bits(value: f32) -> u16 {
    let x = value.to_bits();
    let sign = (x >> 16) & 0x8000;
    let exp = x & 0x7F80_0000;
    let man = x & 0x007F_FFFF;
    // Infinity and NaN.
    if exp == 0x7F80_0000 {
        let quiet_nan = if man == 0 { 0 } else { 0x0200 };
        return (sign | 0x7C00 | quiet_nan | (man >> 13)) as u16;
    }
    let half_exp = ((exp >> 23) as i32) - 127 + 15;
    // Too large: infinity.
    if half_exp >= 0x1F {
        return (sign | 0x7C00) as u16;
    }
    // Subnormal half, or zero.
    if half_exp <= 0 {
        if 14 - half_exp > 24 {
            return sign as u16;
        }
        let man = man | 0x0080_0000; // implicit leading bit
        let shift = (14 - half_exp) as u32;
        let mut half_man = man >> shift;
        let round_bit = 1u32 << (shift - 1);
        // Round half to even: up if above half, or exactly half and odd.
        if (man & round_bit) != 0 && (man & (3 * round_bit - 1)) != 0 {
            half_man += 1;
        }
        return (sign | half_man) as u16;
    }
    // Normal half. A mantissa carry correctly rolls into the exponent (and
    // from the largest finite value into infinity).
    let half = sign | ((half_exp as u32) << 10) | (man >> 13);
    let round_bit = 0x0000_1000;
    if (man & round_bit) != 0 && (man & (3 * round_bit - 1)) != 0 {
        (half + 1) as u16
    } else {
        half as u16
    }
 }
 /// Convert the bit pattern of a half-precision value to `f32` (exact: every
 /// half value is representable as an `f32`).
 pub fn f16_bits_to_f32(h: u16) -> f32 {
    let h = h as u32;
    let sign = (h & 0x8000) << 16;
    let exp = (h >> 10) & 0x1f;
    let mant = h & 0x3ff;
    let bits = if exp == 0 {
        if mant == 0 {
            sign // signed zero
        } else {
            // Subnormal: normalize into an f32 normal.
            let mut e: i32 = -1;
            let mut m = mant;
            loop {
                e += 1;
                m <<= 1;
                if m & 0x400 != 0 {
                    break;
                }
            }
            let m = m & 0x3ff;
            sign | (((127 - 15 - e) as u32) << 23) | (m << 13)
        }
    } else if exp == 0x1f {
        sign | 0x7f80_0000 | (mant << 13) // inf / NaN
    } else {
        sign | ((exp + 127 - 15) << 23) | (mant << 13)
    };
    f32::from_bits(bits)
 }
 /// Round an `f32` to the nearest half-precision value, returned as `f32`.
 pub fn round_to_f16(value: f32) -> f32 {
    f16_bits_to_f32(f32_to_f16_bits(value))
 }
 #[cfg(test)]
 mod tests {
    use super::*;
    #[test]
    fn every_half_value_round_trips() {
        for bits in 0..=u16::MAX {
            let v = f16_bits_to_f32(bits);
            if v.is_nan() {
                assert!(f16_bits_to_f32(f32_to_f16_bits(v)).is_nan(), "{bits:#06x}");
            } else {
                assert_eq!(f32_to_f16_bits(v), bits, "{bits:#06x} -> {v}");
            }
        }
    }
    #[test]
    fn matches_the_half_crate() {
        // Every 257th f32 bit pattern (~16.7M values) covers every exponent,
        // the subnormal range, both signs, ties and the overflow boundary.
        let mut bits: u32 = 0;
        loop {
            let v = f32::from_bits(bits);
            let ours = f32_to_f16_bits(v);
            let theirs = half::f16::from_f32(v);
            if v.is_nan() {
                assert!(theirs.is_nan() && f16_bits_to_f32(ours).is_nan());
            } else {
                assert_eq!(ours, theirs.to_bits(), "{bits:#010x} ({v:e})");
                assert_eq!(f16_bits_to_f32(ours).to_bits(), theirs.to_f32().to_bits());
            }
            match bits.checked_add(257) {
                Some(b) => bits = b,
                None => break,
            }
        }
    }
    #[test]
    fn rounds_ties_to_even_and_saturates_to_infinity() {
        // 1 + 2^-11 is exactly halfway between 1.0 and the next half (1 + 2^-10).
        assert_eq!(round_to_f16(1.0 + 2f32.powi(-11)), 1.0);
        assert_eq!(
            round_to_f16(1.0 + 3.0 * 2f32.powi(-11)),
            1.0 + 2.0 * 2f32.powi(-10)
        );
        assert_eq!(round_to_f16(F16_MAX), F16_MAX);
        assert_eq!(round_to_f16(65520.0), f32::INFINITY); // halfway to 2^16 rounds up
        assert_eq!(round_to_f16(-1e9), f32::NEG_INFINITY);
        assert_eq!(round_to_f16(1e-9).to_bits(), 0);
        assert_eq!(round_to_f16(-1e-9).to_bits(), (-0.0f32).to_bits());
    }
 }
@@ -72,6 +72,7 @@ pub mod filter_pipeline;
 pub mod filters;
 mod filters_szip;
 pub mod fixed_array;
 pub mod float16;
 pub mod fractal_heap;
 pub mod global_heap;
 pub mod group_info;
@@ -56,6 +56,21 @@ pub fn make_f64_type() -> Datatype {
    }
 }
 /// IEEE-754 half precision (binary16), little-endian — numpy's `float16`.
 pub fn make_f16_type() -> Datatype {
    Datatype::FloatingPoint {
        size: 2,
        byte_order: DatatypeByteOrder::LittleEndian,
        bit_offset: 0,
        bit_precision: 16,
        exponent_location: 10,
        exponent_size: 5,
        mantissa_location: 0,
        mantissa_size: 10,
        exponent_bias: 15,
    }
 }
 pub fn make_f32_type() -> Datatype {
    Datatype::FloatingPoint {
        size: 4,
@@ -478,6 +493,24 @@ impl DatasetBuilder {
        self
    }
    /// Store `data` as IEEE half precision (numpy `float16`), rounding each
    /// value to the nearest half ([`crate::float16::f32_to_f16_bits`]).
    /// Half the bytes of [`Self::with_f32_data`], at about three significant
    /// decimal digits; values beyond ±65504 become ±infinity. Reading it back
    /// with `read_f32` yields the rounded values exactly.
    pub fn with_f16_data(&mut self, data: &[f32]) -> &mut Self {
        self.datatype = Some(make_f16_type());
        let mut b = Vec::with_capacity(data.len() * 2);
        for &v in data {
            b.extend_from_slice(&crate::float16::f32_to_f16_bits(v).to_le_bytes());
        }
        self.data = Some(b);
        if self.shape.is_none() {
            self.shape = Some(vec![data.len() as u64]);
        }
        self
    }
    pub fn with_i32_data(&mut self, data: &[i32]) -> &mut Self {
        self.datatype = Some(make_i32_type());
        let mut b = Vec::with_capacity(data.len() * 4);
@@ -2,6 +2,7 @@
 name = "clawhdf5-gpu"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "GPU-accelerated vector operations for rustyhdf5 using wgpu compute shaders"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -2,6 +2,7 @@
 name = "clawhdf5-io"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "I/O abstraction layer for rustyhdf5"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -2,6 +2,7 @@
 name = "clawhdf5-migrate"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "CLI to migrate SQLite agent memory databases to HDF5 format"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -2,6 +2,7 @@
 name = "clawhdf5-napi"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "Node.js native addon (napi-rs) exposing clawhdf5-agent to TypeScript/JavaScript"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -2,6 +2,7 @@
 name = "clawhdf5-netcdf4"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "NetCDF-4 read support built on rustyhdf5 — pure Rust, no C dependencies"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -2,6 +2,7 @@
 name = "clawhdf5-py"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -2,6 +2,7 @@
 name = "clawhdf5"
 version = "2.7.0"
 edition = "2024"
 rust-version.workspace = true
 description = "Pure-Rust HDF5 reader/writer — no C dependencies"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -30,9 +31,10 @@ name = "parallel_bench"
 harness = false
 [features]
-default = ["mmap", "fast-deflate", "provenance"]
+default = ["mmap", "provenance"]
 mmap = ["clawhdf5-io/mmap"]
 parallel = ["clawhdf5-format/parallel", "rayon"]
 # zlib-ng (C, needs cmake) instead of the default pure-Rust zlib-rs.
 fast-deflate = ["clawhdf5-format/fast-deflate"]
 apple-compression = []
 zstd = ["clawhdf5-format/zstd"]
@@ -1387,3 +1387,166 @@ with h5py.File("{path_str}", "w", libver="latest") as f:
        );
    }
 }
 // ---------------------------------------------------------------------------
 // Half precision (float16) in both directions
 // ---------------------------------------------------------------------------
 /// Values that exercise rounding: ties, subnormals, the overflow boundary and
 /// ordinary embedding-sized components.
 fn f16_probe_values() -> Vec<f32> {
    let mut v = vec![
        0.0,
        -0.0,
        1.0,
        -1.0,
        0.5,
        1.0 + 2f32.powi(-11),
        1.0 + 3.0 * 2f32.powi(-11),
        65504.0,
        65519.0,
        65520.0,
        -70000.0,
        6.0e-8,
        3.0e-8,
        1.0e-9,
        1.0e-5,
        0.1,
        0.333_333,
        1234.567,
        f32::INFINITY,
        f32::NEG_INFINITY,
    ];
    // A deterministic spread of embedding-like values.
    let mut x = 0x2545_F491u32;
    for _ in 0..4000 {
        x ^= x << 13;
        x ^= x >> 17;
        x ^= x << 5;
        v.push((x as f32 / u32::MAX as f32 - 0.5) * 0.4);
    }
    v
 }
 #[test]
 fn clawhdf5_writes_f16_h5py_reads() {
    skip_if_no_python!();
    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("ours_f16.h5");
    let path_str = path.display().to_string();
    let values = f16_probe_values();
    let mut fb = FileBuilder::new();
    fb.create_dataset("plain").with_f16_data(&values);
    fb.create_dataset("chunked")
        .with_f16_data(&values)
        .with_shape(&[values.len() as u64])
        .with_chunks(&[512])
        .with_deflate(6);
    fb.write(&path).unwrap();
    // h5py must see a genuine float16 dataset, and our rounding must agree
    // with numpy's own float32 -> float16 conversion bit for bit.
    let input = values
        .iter()
        .map(|v| format!("{:?}", v.to_bits()))
        .collect::<Vec<_>>()
        .join(",");
    let script = format!(
        r#"
 import h5py, numpy as np
 src = np.array([{input}], dtype=np.uint32).view(np.float32)
 expected = src.astype(np.float16).view(np.uint16)
 with h5py.File("{path_str}", "r") as f:
    for name in ("plain", "chunked"):
        d = f[name]
        assert d.dtype == np.float16, (name, d.dtype)
        got = d[:].view(np.uint16)
        bad = np.nonzero(got != expected)[0]
        assert bad.size == 0, (name, bad[:5], got[bad[:5]], expected[bad[:5]])
 print("ok")
 "#
    );
    assert_eq!(run_python_output(&script), "ok");
 }
 #[test]
 fn h5py_writes_f16_clawhdf5_reads() {
    skip_if_no_python!();
    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("h5py_f16.h5");
    let path_str = path.display().to_string();
    let values = f16_probe_values();
    let input = values
        .iter()
        .map(|v| format!("{:?}", v.to_bits()))
        .collect::<Vec<_>>()
        .join(",");
    let script = format!(
        r#"
 import h5py, numpy as np
 src = np.array([{input}], dtype=np.uint32).view(np.float32).astype(np.float16)
 with h5py.File("{path_str}", "w") as f:
    f.create_dataset("plain", data=src)
    f.create_dataset("chunked", data=src, chunks=(512,), compression="gzip", shuffle=True)
    f.create_dataset("big_endian", data=src.astype(">f2"))
 "#
    );
    run_python(&script);
    let expected: Vec<u32> = values
        .iter()
        .map(|&v| clawhdf5_format::float16::round_to_f16(v).to_bits())
        .collect();
    let file = File::open(&path).unwrap();
    for name in ["plain", "chunked", "big_endian"] {
        let ds = file.dataset(name).unwrap();
        assert_eq!(
            ds.dtype().unwrap(),
            DType::Other("float16".into()),
            "{name}"
        );
        let got: Vec<u32> = ds.read_f32().unwrap().iter().map(|v| v.to_bits()).collect();
        assert_eq!(got, expected, "{name}");
    }
 }
 #[test]
 fn clawhdf5_writes_f32_h5py_reads() {
    // Every f32 dataset used to be unreadable by h5py ("sign bit position out
    // of bounds"): the float datatype's sign position was hard-coded for f64.
    skip_if_no_python!();
    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("ours_f32.h5");
    let path_str = path.display().to_string();
    let values: Vec<f32> = vec![1.5, -2.25, 3.0e-7, 65536.5, f32::MAX, -0.0];
    let mut fb = FileBuilder::new();
    fb.create_dataset("plain").with_f32_data(&values);
    fb.create_dataset("chunked")
        .with_f32_data(&values)
        .with_shape(&[values.len() as u64])
        .with_chunks(&[4])
        .with_deflate(6);
    fb.write(&path).unwrap();
    let bits = values
        .iter()
        .map(|v| v.to_bits().to_string())
        .collect::<Vec<_>>()
        .join(",");
    let script = format!(
        r#"
 import h5py, numpy as np
 expected = np.array([{bits}], dtype=np.uint32)
 with h5py.File("{path_str}", "r") as f:
    for name in ("plain", "chunked"):
        d = f[name]
        assert d.dtype == np.float32, (name, d.dtype)
        assert (d[:].view(np.uint32) == expected).all(), (name, d[:])
 print("ok")
 "#
    );
    assert_eq!(run_python_output(&script), "ok");
 }
@@ -2,6 +2,7 @@
 name = "libaec-sys"
 version = "0.1.0"
 edition = "2024"
 rust-version.workspace = true
 links = "aec"
 [build-dependencies]
@@ -364,10 +364,11 @@ cargo install --path crates/clawhdf5-cli
 clawhdf5 --path agent.h5 create --agent-id my-agent --dim 384 --wal
 ```
-Add `--quantized-index` to store the vector index's copy of the embeddings as
+New stores hold the vector index's copy of the embeddings as int8, which
-int8. That roughly halves a loaded store's memory at about 13% fewer queries
+roughly halves a loaded store's memory and is faster at equal recall — the
-per second, with recall unchanged — the query path re-scores candidates
+query path re-scores candidates against the exact embeddings. Pass
-against the exact embeddings. The setting is recorded in the file.
+`--f32-index` to keep an f32 index instead. The setting is recorded in the
 file, and stores created before it existed keep their f32 index.
 Output:
 ```json
@@ -566,4 +567,4 @@ let final_results = confidence::reject_low_confidence(
 ---
-<p align="center"><em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em></p>
+<p align="center"><em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em></p>
@@ -232,4 +232,4 @@ clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "accelerate"
 ---
-<p align="center"><em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em></p>
+<p align="center"><em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em></p>
@@ -227,3 +227,55 @@ wrong structure. All four are fixed and covered by interop tests against
 HDF5 2.0 at sizes that cross each boundary, including paged data blocks.
 Files written by this crate are unaffected — this was purely a read-path bug.
 ## Every `f32` dataset we wrote was unreadable by h5py / libhdf5
 **Status:** fixed 2026-09-23, after v2.7.0. **Every
 release up to and including v2.7.0 is affected** — the encoder was already
 wrong in v2.1.0.
 The floating-point datatype message carries the position of the sign bit
 (bits 8–15 of its class bit field). `clawhdf5-format` wrote 63 for every
 float, which is correct only for `f64`. libhdf5 validates the field, so opening
 any `f32` dataset written by this crate failed:
 ```
 KeyError: 'Unable to synchronously open object (sign bit position out of bounds)'
 ```
 That covers every agent store (`/memory/embeddings`, `norms` and
 `activation_weights` are `f32`). `clawhdf5` itself ignores the field on read,
 and the interop suites only ever wrote `f64` from our side, so nothing here
 noticed.
 **Fix:** the sign position is computed from the type (`bit_offset +
 bit_precision - 1`: 15, 31, 63 for half, single, double). Regression tests:
 `float_sign_location_is_the_top_bit_of_the_value` (byte level),
 `clawhdf5_writes_f32_h5py_reads` and the agent's
 `h5py_reads_every_dataset_of_an_agent_store`.
 **Existing files:** an agent store is rewritten in full at every checkpoint, so
 it becomes readable by h5py at its next checkpoint with a fixed build. Other
 files with `f32` datasets need to be rewritten.
 ## Empty datasets we wrote were unreadable by h5py / libhdf5
 **Status:** fixed 2026-09-23, after v2.7.0. Every
 release up to and including v2.7.0 is affected.
 A dataset with no elements was written with a real file address and a storage
 size of 0. libhdf5 guards contiguous storage with an overflow check
 (`addr + size <= addr`) that is always true when the size is 0, so it rejected
 the dataset:
 ```
 KeyError: 'Unable to synchronously open object (invalid dataset size, likely file corruption)'
 ```
 In practice: every agent store without sessions or a knowledge graph — the
 `/sessions` and `/knowledge_graph` datasets are empty until something is added
 — could not be read by h5py even once the `f32` bug above was fixed. Found by
 the same agent-store interop test.
 **Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
 which is what libhdf5 itself writes.
@@ -77,6 +77,50 @@ run_step "cargo clippy (ann parallel)" cargo clippy \
    --features parallel \
    -- -D warnings
 # zlib-ng is opt-in (`fast-deflate`; the default is pure-Rust zlib-rs), so
 # nothing above builds it. Keep it compiling and passing.
 run_step "cargo clippy (fast-deflate / zlib-ng)" cargo clippy \
    -p clawhdf5-format -p clawhdf5-filters -p clawhdf5 \
    --all-targets \
    --features clawhdf5-format/fast-deflate,clawhdf5-filters/fast-deflate \
    -- -D warnings
 # The README promises that the core crates build no C by default. Hold it to
 # that: fail if a crate that compiles C (a *-sys crate, cc or cmake) enters the
 # default dependency tree of any of them. clawhdf5-migrate (bundled SQLite),
 # clawhdf5-napi (Node) and clawhdf5-gpu (graphics drivers) are exempt.
 no_c_in_default_build() {
    local crate found=0
    for crate in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \
        clawhdf5-agent clawhdf5-ann clawhdf5-accel clawhdf5-netcdf4 clawhdf5-cli; do
        local c_deps
        c_deps=$(cargo tree -q -p "$crate" -e normal,build --prefix none \
            | grep -E '^([a-z0-9_-]+-sys|cc|cmake) v' | sort -u)
        if [ -n "$c_deps" ]; then
            echo "$crate pulls in C by default:"
            echo "$c_deps" | sed 's/^/    /'
            found=1
        fi
    done
    return $found
 }
 run_step "no C in the default build (core crates)" no_c_in_default_build
 # The workspace declares a minimum Rust version (rust-version in Cargo.toml);
 # check that it really builds there, so the README badge and the manifests
 # cannot drift from the truth. Separate target dir: a different toolchain
 # would otherwise invalidate the main build.
 msrv_check() {
    local msrv
    msrv=$(sed -n 's/^rust-version = "\(.*\)"/\1/p' "$SCRIPT_DIR/../Cargo.toml")
    [ -n "$msrv" ] || { echo "no rust-version in Cargo.toml"; return 1; }
    rustup toolchain install "$msrv" --profile minimal >/dev/null || return 1
    echo "checking with Rust $msrv"
    CARGO_TARGET_DIR="$SCRIPT_DIR/../target/msrv" cargo "+$msrv" check \
        --workspace --exclude clawhdf5-py --all-targets
 }
 run_step "MSRV check" msrv_check
 # 4. Tests (exclude clawhdf5-py)
 run_step "cargo test" cargo test \
    --workspace \
@@ -90,6 +134,10 @@ run_step "cargo test (ann parallel)" cargo test \
    -p clawhdf5-ann \
    --features parallel
 run_step "cargo test (fast-deflate / zlib-ng)" cargo test \
    -p clawhdf5-format -p clawhdf5-filters -p clawhdf5 \
    --features clawhdf5-format/fast-deflate,clawhdf5-filters/fast-deflate
 # 5. Python interop suites. The h5py writer tests are #[ignore]d so a plain
 #    `cargo test` stays hermetic; run them explicitly here.
 #    On a PEP 668 "externally managed" system h5py can only live in a
Author	SHA1	Message	Date
osobhandClaude Opus 5.5	d0db83812b	feat(agent): MemoryConfig::float16 stores half-precision embeddings CI / test-arm64 (pull_request) Successful in 1m19s Details CI / test (pull_request) Successful in 4m58s Details The setting was persisted in /meta and otherwise ignored: embeddings were always written as f32. It now does what it says. clawhdf5-format: - `DatasetBuilder::with_f16_data` writes IEEE binary16 (numpy float16), rounding to nearest-even, and `make_f16_type`. - `clawhdf5_format::float16` holds the f32 <-> f16 conversions, the one implementation the writer, the reader and the agent all use. Checked against the `half` crate on 16.7M f32 values and round-trips all 65536 half values; the h5py interop tests confirm the rounding matches numpy's bit for bit (4020 values incl. ties, subnormals, overflow). - Reading little-endian float16 as f32 has a fast path. clawhdf5-agent: - A float16 store writes /memory/embeddings as half precision, and `MemoryCache::half_precision` rounds each embedding as it enters the cache (save, update, WAL replay, and on load of a store still f32 on disk), so memory and file agree bit for bit and a store searches the same before and after a reopen (tested). - Values beyond +-65504 are refused with the new `MemoryError::InvalidEntry` rather than stored as infinity, on every save path; batches are all or nothing, and a rejected ephemeral entry stays in the ephemeral tier. Breaking for exhaustive matches. - CLI: `create --float16`. Off by default. Measured on tank, 384-dim, six runs alternating order, medians (search_harness --float16-study --full): at 100K the file goes from 154.0 to 80.8 MiB (-48%), checkpoint 752 -> 512 ms, open 300 -> 252 ms; vector recall@10 against an exact scan and hybrid_search latency do not change. At 10K open is 3 ms slower. Also a test that h5py opens a whole agent store, f32 and float16, and decodes every dataset. Docs: README, BENCHMARKS.md ("float16 embedding storage"), CHANGELOG (including the h5py interop fixes in the previous commit), CLAUDE.md. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-24 12:00:38 -05:00
osobhandClaude Opus 5.5	5e4aa1c6bf	fix(format): files we write now open in h5py and libhdf5 Two write-side bugs, both present in every release (the first at least since v2.1.0), made libhdf5 refuse files written by clawhdf5. Our own reader ignores both fields, and the interop suites only ever wrote f64 from our side, so nothing here caught them. - Every f32 dataset: "sign bit position out of bounds". The float datatype encoder hard-coded the sign bit's position (bits 8-15 of the class bit field) to 63, which is right only for f64. It is now derived from the type: bit_offset + bit_precision - 1. This covered every agent store's embeddings, norms and activation weights. - Every empty dataset: "invalid dataset size, likely file corruption". It was written with a real address and size 0, which trips libhdf5's `addr + size <= addr` overflow check. An empty contiguous dataset now gets the undefined address, as libhdf5 writes it. This covered every agent store without sessions or a knowledge graph. Agent stores are rewritten in full at each checkpoint, so they become readable at their next checkpoint on a fixed build; other files with f32 or empty datasets need rewriting. Both are recorded in docs/known-issues.md. Tests: the sign position byte for f32/f64, and h5py reading our f32 datasets (plain and chunked + deflate) bit for bit. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-24 12:00:26 -05:00
osobh	1cceb930b2	Merge pull request 'Feat/pure rust default and msrv' (#3 ) from feat/pure-rust-default-and-msrv into main CI / test-arm64 (push) Successful in 1m20s Details CI / test (push) Successful in 5m59s Details Reviewed-on: #3	2026-09-23 16:29:33 +00:00
osobhandClaude Opus 5.5	fc7ae6549a	build: declare Rust 1.92 as the MSRV and check it in CI CI / test-arm64 (pull_request) Successful in 1m19s Details CI / test (pull_request) Successful in 5m23s Details rust-version = "1.92" in [workspace.package], inherited by every crate. 1.92 is the floor: wgpu (clawhdf5-gpu) requires it, and the whole workspace, Python bindings included, checks cleanly on it. ci-test.sh reads the version from Cargo.toml and checks the workspace on exactly that toolchain, so the manifests and the README badge cannot drift from what actually builds. The badge said 1.75, below edition 2024's own floor. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-23 11:05:26 -05:00
osobhandClaude Opus 5.5	735db117a7	build: pure-Rust zlib-rs as the default deflate backend The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel, -netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs, a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in `fast-deflate`, which overrides zlib-rs wherever it is enabled. A default build no longer needs cmake or a C compiler. Measured on tank, both builds run alternately, three rounds, medians: zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512 deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4 vs 65.2 ms), and compressed output is byte-identical. Details in BENCHMARKS.md, "Deflate backend". Getting there took two fixes the first measurement exposed: - zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via its default `runtime_detection`, which `default-features = false` had switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs` features now enable it. - Both deflate paths streamed through flate2's 32 KiB read/write wrappers. They now hand the codec the whole chunk in one call, into a buffer sized up front (~5% on chunked writes). This also fixes a silent short read: the streaming reader returned a truncated stream's bytes without an error; a truncated chunk is now DecompressionError. In clawhdf5-filters, output longer than the stated size is now an error rather than silently cut off. CI: ci-test.sh lints and tests the zlib-ng path, and fails if a C-building crate (*-sys, cc, cmake) enters a core crate's default dependency tree. The arm64 job no longer installs cmake. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-23 11:05:22 -05:00
osobhandClaude Opus 5.5	e9b37a9602	docs: bring the README up to date with the last five releases The README had fallen behind v2.3.0-v2.7.0, and parts of it were not true. Checked every claim against the code and BENCHMARKS.md: - Three of the six Quick Start snippets no longer compiled (Agent Memory, Consolidation, OpenClaw); all six now do. - Hybrid search was described as RRF throughout. The default has been weighted 0.4/0.6 fusion since v2.5.0; re-ranking and confidence rejection run only in the OpenClaw backend. - The `float16` feature does not halve embedding storage (the store always writes f32), `--features agent` enables nothing, "Source Isolation" is not wired in, and nothing backs "billion-scale" IVF-PQ. - "Cryptographically verifiable" overstated an unkeyed, session-scoped FNV-1a ledger; "Zero C dependencies" was false while zlib-ng was the default deflate backend. - Stale numbers: tests (1,650 -> 1,868), Rust badge (1.75 is below edition 2024's floor), 6.5 KB/record on disk (BENCHMARKS.md: 1.7 KB), consolidation and hybrid-search latency, and a feature-flag table broken by a paragraph pasted into it. - The file schema, module table and crate map now match the code. Adds a "What's new (v2.2 -> v2.7)" section for collaborators, leading with the silent Extensible Array read bug fixed in v2.7.0. Footer links point at git.redclaw.dev. CLAUDE.md: clawhdf5-migrate is the SQLite migration tool, and MemoryConfig::compression is off by default. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>	2026-09-23 11:05:10 -05:00
osobhandClaude Opus 5	4bed8b3765	Merge docs/ci: record how CI is set up CI / test-arm64 (push) Successful in 1m6s Details CI / test (push) Successful in 3m28s Details Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-22 04:48:17 -07:00
osobhandClaude Opus 5	36d689bc2c	docs: record how CI is set up, and what broke it Two jobs, which runners serve them, and the two constraints that kept the x86 job failing on every push until today: no JavaScript actions (`rust:latest` has no `node`, and GitHub is not reachable from every runner) and `cmake` for libz-ng-sys. Also notes that the Docker Hub `latest` tag for the runner is frozen at 0.6.1, so it is not a way to stay current. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-22 04:48:17 -07:00
osobhandClaude Opus 5	e7c08e06b4	Merge ci/fix-jobs: make both CI jobs actually run CI / test-arm64 (push) Successful in 1m55s Details CI / test (push) Successful in 4m8s Details Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-21 20:37:51 -07:00
osobhandClaude Opus 5	c5049eb734	ci: stop depending on node, and install cmake where the build needs it Read from the job logs of the first run with the arm64 job, rather than guessed: - The x86 `test` job has been failing on every push, in about three seconds. It runs in `rust:latest` and starts with actions/checkout, a JavaScript action, and `rust:latest` has no `node`: exit 127 before a line of code was built. It is replaced with a plain git checkout (the image has git), and actions/cache, JavaScript for the same reason, is dropped. - `test-arm64` got through checkout, toolchain, the aarch64 check and clippy, then failed building libz-ng-sys — pulled in by clawhdf5-format's default `fast-deflate` — because the host runner had no cmake. The x86 image lacks it too, so the x86 job would have hit the same wall one step later. Both jobs now install cmake where they can (the Docker job and the x86 container run as root) and say plainly when they cannot (a host runner), instead of failing inside a build script. vision-01 now has cmake. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-21 20:37:51 -07:00
osobhandClaude Opus 5	6f6bc97850	Merge ci/arm64: run the aarch64 kernels in CI CI / test (push) Failing after 3s Details CI / test-arm64 (push) Failing after 37s Details Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-21 20:31:05 -07:00
osobhandClaude Opus 5	0cb72e8a60	ci: test the aarch64 kernels on an arm64 runner The NEON kernels in clawhdf5-accel — `dot_i8` including its SDOT path, and the f32 NEON kernels that predate it — are cfg'd out on x86, so the existing job has never compiled, linted or tested a line of them. They were verified once, by hand, on a Raspberry Pi 5. `test-arm64` runs on `linux_arm64`, which two runners serve in different ways: vision-01 executes steps on the host with Rust preinstalled, and vision-02 executes them in docker.gitea.com/runner-images. The job is written to work in both: no `container:`, no JavaScript actions (those are fetched from GitHub, which not every runner reliably reaches), and an explicit `+stable` toolchain so a host's default — vision-01's is a January nightly — is neither relied on nor changed. Fetches retry, since one runner's outbound network was seen failing intermittently. It lints the accel crate and tests accel, ann and format. It reports rather than requires the dot-product extension: on a core without it the plain-NEON kernel is the one that runs, and the tests cover whichever is present. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-21 20:31:05 -07:00
osobhandClaude Opus 5	e338d58ad5	Merge feat/int8-default: new stores use the int8 index CI / test (push) Failing after 2s Details quantized_index defaults to true for new stores — smaller and faster at equal recall on every configuration measured. Existing stores keep their setting, and stores predating it stay f32, guarded by a real v2.5.0 store committed as a test fixture. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-21 18:27:59 -07:00
osobhandClaude Opus 5	8b85d9364b	feat(agent): new stores use the int8 vector index by default `MemoryConfig::quantized_index` now defaults to `true`. It holds a quarter of the index memory and, with the exact re-score, is faster at equal recall on every configuration measured: 1.63x the queries per second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with builds 1.8x and 2.3x faster. The one argument for keeping it off — that int8 search was slower on ARM — did not survive being measured. Existing stores do not change. A store written with v2.6.0 or later keeps its persisted setting. One written before the setting existed has no stored value, and it loads as `false` rather than as the new default, so reopening it never changes how its index is held. That case is guarded by a real store written with the v2.5.0 CLI, committed as `tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens with an f32 index and still searches, and it fails if the load default is changed to `true`. The CLI needed more than a new default. `create --quantized-index` assigned its value straight into the config, so under the new default every CLI-created store would have been forced back to f32 unless the caller knew to ask for int8. It is replaced by `--f32-index`, which only ever switches the default off; `--quantized-index` is still accepted, hidden, as a no-op, and the two conflict. The whole agent suite passes under the new default, including the brute-force recall oracle, now running on int8 plus re-score without being asked to. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-21 18:27:59 -07:00
osobhandClaude Opus 5	6598a7d02f	Merge feat/neon-int8: aarch64 int8 dot product, verified on a Pi 5 CI / test (push) Failing after 2s Details SDOT and plain-NEON kernels for dot_i8, tested bit-exact against scalar on real ARM. At equal recall the quantised index is 1.18x f32 on a Pi 5 and builds 2.3x faster. Also corrects an unmeasured claim that it was slower than f32 on ARM. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-21 17:34:51 -07:00
osobhandClaude Opus 5	114a2dfcba	docs: measured ARM numbers, and a correction On a Raspberry Pi 5 at N = 100 000 and equal recall (0.9940 vs 0.9945), medians of three runs: f32 33 413 ms build 6 164 QPS int8 scalar 18 950 ms ~6 190 QPS (what v2.7.0 shipped) int8 NEON ~17 000 ms 6 640 QPS int8 SDOT 14 464 ms 7 267 QPS 1.18x f32, 2.3x build The docs said quantised search stayed off by default because aarch64 "falls back to the scalar loop, where the original trade still applies" — that it was ~13% slower than f32 there, as on x86. That was extrapolated rather than measured, and it was wrong: x86's portable baseline is SSE2 against hand-written AVX2 f32 kernels, but on aarch64 NEON is the baseline and the scalar loop vectorises well, so it already matched f32. Corrected in BENCHMARKS.md, README.md and CLAUDE.md; the released v2.7.0 changelog entry is left as it was and the correction is recorded in a new one. Labelled as Pi 5 figures throughout — a Pi's memory bandwidth and cache are far below an M-series or flagship phone, so the ratios will move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-21 17:34:51 -07:00
osobhandClaude Opus 5	56a8c2f3d0	feat(accel): aarch64 int8 dot product — SDOT and plain NEON `dot_i8` had an AVX2 kernel and a scalar fallback, so on aarch64 the quantised HNSW index ran the scalar loop. It now dispatches to one of two NEON kernels: - `dot_i8_dotprod`: the ARMv8.2 dot-product instruction, `SDOT`, which multiplies and accumulates sixteen i8 pairs into four i32 lanes per instruction. Present on Cortex-A76 and later (Raspberry Pi 5, current Android phones), Neoverse-N1 (Graviton2, Ampere Altra) and every Apple Silicon generation. Issued as inline assembly because the `vdotq_s32` intrinsic is still behind the unstable `stdarch_neon_dotprod` feature; inline asm is stable on aarch64. - `dot_i8`: plain NEON for cores without the extension — `vmull_s8` widens to i16 (even -128 * -128 fits) and `vpadalq_s16` folds adjacent pairs into i32 accumulators, so nothing overflows. Selected at runtime with `is_aarch64_feature_detected!("dotprod")`. Verified on a Raspberry Pi 5 (Cortex-A76, `asimddp` present), not just compiled — the aarch64 code is cfg'd out on x86, so x86 CI never builds or lints it: - both kernels bit-exact against scalar at every length, tails and extremes included. Each is tested directly rather than through dispatch, because dispatch only takes one path on a given CPU: on the Pi, testing through it alone would never have run the plain-NEON fallback at all. - mutation-checked: dropping the SDOT kernel's second accumulator fails at length 32, and using the low half twice in the NEON kernel fails at length 16 — the first lengths that exercise each. - the ANN suite passes, including int8 recall against ground truth. - clippy clean with -D warnings on aarch64. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-21 17:34:51 -07:00
`@@ -232,4 +232,4 @@ clawhdf5-agent = { version = "2.0", features = ["agent", "float16", "accelerate"`

	`---`	`---`

	`<p align="center"><em>Built by <a href="https://github.com/redclawsystems">RedClaw Systems</a></em></p>`	`<p align="center"><em>Built by <a href="https://git.redclaw.dev/quantumclaw">RedClaw Systems</a></em></p>`