chore(release): v2.7.0

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-20 18:24:07 -07:00
parent f4c6d43a3f
commit 4a5544da1d
20 changed files with 106 additions and 94 deletions
@@ -1,56 +1,20 @@
 # Changelog

-## Unreleased
+## v2.7.0 (2026-09-20)

-### Tuning
- `clawhdf5-agent`: **the HNSW parameters are configurable** —
-  `MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search`
-  (defaults 16, 64, and 0 meaning "scale with `k`", i.e. today's behaviour).
-  They were constants, so a deployment could not trade recall against memory
-  or query speed at all. All three are persisted with the store. Values are
-  clamped where the index requires it: `clawhdf5-ann` asserts a graph degree
-  of at least 2, so a configured 0 — from a file, or from a caller who took 0
-  to mean "default" — used to abort the process inside the builder. Lowering
-  `ef_search` also no longer narrows the candidate pool that fusion sees.
-  **Breaking:** `MemoryConfig` gained fields, so literal constructions need
-  updating; `..Default::default()` does not.
-
-### Performance
- `clawhdf5-agent`: **opening a store is ~28% faster** (455 ms -> 327 ms at
-  100k x 384). `read_from_disk` memory-mapped the file and then copied the
-  entire mapping into a `Vec` for `File::from_bytes`, when `File::open`
-  memory-maps it directly — so every open paid a full-file memcpy for nothing.
-  Process peak memory is unchanged: the peak falls after the parse, during the
-  index build, so the transient never reached the high-water mark. The
-  footprint harness now reports that peak next to the retained figure, which
-  is how this was checked rather than assumed.
-
-### Integrity
- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
-  their checksums** (the `checksum` feature, on by default). Every structure
-  in both — header, index block, super block, data block and each data block
-  page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
-  The consequence of skipping it is not a missing warning but wrong data: a
-  single flipped bit in a chunk address still parses, still points inside the
-  file, and the reader hands back whatever bytes now sit there as the chunk's
-  contents. Verified in both directions — the checksums accept files written
-  by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
-  and an interop test corrupts an address to confirm the read now fails
-  instead of returning data (it does return data when the check is removed).
-
-### Performance
- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
-  sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
-  elsewhere). The quantised HNSW index used a scalar loop while the `f32` path
-  it was measured against ran AVX2, so the ~13% throughput cost recorded for
-  `MemoryConfig::quantized_index` was a missing kernel rather than a property
-  of int8. With the kernel, at N = 100 000 x 384 and equal recall, the
-  quantised index answers **1.63x as many queries per second** (21 848 vs
-  13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778
-  vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three
-  alternating runs. It remains off by default only because the kernel is
-  AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so
-  the SIMD path is tested to agree with scalar bit for bit.
+### Upgrade Notes
+- **Two read-path bugs fixed, one of them silent.** Datasets indexed by an
+  Extensible Array (any dataset with one unlimited dimension) returned data
+  from the wrong chunks past their first few dozen. If you have readings taken
+  from such a dataset with an earlier release, they may be wrong; re-read them.
+- **A corrupt chunk index is now an error.** Fixed and Extensible Array
+  structures carry checksums that were previously ignored, so damage surfaced
+  as plausible data from the wrong offset. Code that read a damaged file and
+  got numbers will now get `ChecksumMismatch` instead. That is the point.
+- **Breaking:** `MemoryConfig` gained `hnsw_m`, `hnsw_ef_construction` and
+  `hnsw_ef_search`, so literal constructions need updating;
+  `..Default::default()` does not. All three default to the previous
+  behaviour.

 ### Correctness
 - `clawhdf5-format`: **datasets indexed by an Extensible Array returned wrong
@@ -100,6 +64,54 @@
  including a depth-2 HDF5 2.0 chunk index with 40 000 records, now covered by
  an interop test.

+### Integrity
+- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
+  their checksums** (the `checksum` feature, on by default). Every structure
+  in both — header, index block, super block, data block and each data block
+  page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
+  The consequence of skipping it is not a missing warning but wrong data: a
+  single flipped bit in a chunk address still parses, still points inside the
+  file, and the reader hands back whatever bytes now sit there as the chunk's
+  contents. Verified in both directions — the checksums accept files written
+  by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
+  and an interop test corrupts an address to confirm the read now fails
+  instead of returning data (it does return data when the check is removed).
+
+### Performance
+- `clawhdf5-agent`: **opening a store is ~28% faster** (455 ms -> 327 ms at
+  100k x 384). `read_from_disk` memory-mapped the file and then copied the
+  entire mapping into a `Vec` for `File::from_bytes`, when `File::open`
+  memory-maps it directly — so every open paid a full-file memcpy for nothing.
+  Process peak memory is unchanged: the peak falls after the parse, during the
+  index build, so the transient never reached the high-water mark. The
+  footprint harness now reports that peak next to the retained figure, which
+  is how this was checked rather than assumed.
+- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
+  sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
+  elsewhere). The quantised HNSW index used a scalar loop while the `f32` path
+  it was measured against ran AVX2, so the ~13% throughput cost recorded for
+  `MemoryConfig::quantized_index` was a missing kernel rather than a property
+  of int8. With the kernel, at N = 100 000 x 384 and equal recall, the
+  quantised index answers **1.63x as many queries per second** (21 848 vs
+  13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778
+  vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three
+  alternating runs. It remains off by default only because the kernel is
+  AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so
+  the SIMD path is tested to agree with scalar bit for bit.
+
+### Tuning
+- `clawhdf5-agent`: **the HNSW parameters are configurable** —
+  `MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search`
+  (defaults 16, 64, and 0 meaning "scale with `k`", i.e. today's behaviour).
+  They were constants, so a deployment could not trade recall against memory
+  or query speed at all. All three are persisted with the store. Values are
+  clamped where the index requires it: `clawhdf5-ann` asserts a graph degree
+  of at least 2, so a configured 0 — from a file, or from a caller who took 0
+  to mean "default" — used to abort the process inside the builder. Lowering
+  `ef_search` also no longer narrows the candidate pool that fusion sees.
+  **Breaking:** `MemoryConfig` gained fields, so literal constructions need
+  updating; `..Default::default()` does not.
+
 ### Documentation
 - `clawhdf5-agent`: `BM25Index::search` claimed to use Block-Max WAND for early
  termination. It never did; it scores every match exhaustively. It now says
@@ -21,7 +21,7 @@ members = [
 resolver = "2"

 [workspace.package]
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 license = "MIT"
 repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-accel"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "SIMD-accelerated operations for rustyhdf5"
 license = "MIT"
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-agent"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "HDF5-backed persistent memory store for on-device AI agents"
 license = "MIT"
@@ -10,12 +10,12 @@ keywords = ["agent", "memory", "hdf5", "vector-search", "embedding"]
 categories = ["database", "science", "algorithms"]

 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0", features = ["parallel", "fast-checksum"] }
-clawhdf5 = { path = "../clawhdf5", version = "2.6.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.6.0", features = ["mmap"] }
-clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.6.0" }
-clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.6.0", optional = true }
-clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.6.0", optional = true, default-features = false }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0", features = ["parallel", "fast-checksum"] }
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0", features = ["mmap"] }
+clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.7.0" }
+clawhdf5-ann = { path = "../clawhdf5-ann", version = "2.7.0", optional = true }
+clawhdf5-gpu = { path = "../clawhdf5-gpu", version = "2.7.0", optional = true, default-features = false }
 serde = { workspace = true }
 byteorder = "1"
 half = { workspace = true, optional = true }
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-android"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "Android JNI bridge for edgehdf5-memory HDF5 backend"
 license = "MIT"
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-ann"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "HNSW approximate nearest neighbor index stored as HDF5"
 license = "MIT"
@@ -10,9 +10,9 @@ keywords = ["hdf5", "ann", "hnsw", "nearest-neighbor"]
 categories = ["algorithms", "science"]

 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.6.0" }
-clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.6.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0" }
+clawhdf5-accel = { path = "../clawhdf5-accel", version = "2.7.0" }
 rayon = { version = "1", optional = true }

 [features]
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-bench"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "Benchmark harnesses for clawhdf5-agent (Track 8)"
 license = "MIT"
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-cli"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 license = "MIT"
 description = "CLI for clawhdf5 agent memory — create, save, search, recall, stats"
@@ -14,7 +14,7 @@ name = "clawhdf5"
 path = "src/main.rs"

 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.6.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
 clap = { version = "4", features = ["derive", "env"] }
 serde_json = "1"
 serde = { workspace = true }
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-derive"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "Derive macros for rustyhdf5 HDF5 traits"
 license = "MIT"
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-filters"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "Filter and compression pipeline for clawhdf5"
 license = "MIT"
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-format"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "Pure-Rust HDF5 binary format parsing and writing — no C dependencies"
 license = "MIT"
@@ -25,7 +25,7 @@ pco = { version = "1.0", optional = true }
 [dev-dependencies]
 serde_json = "1"
 criterion = { workspace = true }
-clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.6.0" }
+clawhdf5-derive = { path = "../clawhdf5-derive", version = "2.7.0" }

 [[bench]]
 name = "bench"
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-gpu"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "GPU-accelerated vector operations for rustyhdf5 using wgpu compute shaders"
 license = "MIT"
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-io"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "I/O abstraction layer for rustyhdf5"
 license = "MIT"
@@ -10,7 +10,7 @@ keywords = ["hdf5", "io", "science", "data"]
 categories = ["filesystem", "science"]

 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 memmap2 = { version = "0.9", optional = true }
 libc = { version = "0.2", optional = true }
 tokio = { version = "1", features = ["fs", "io-util"], optional = true }
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-migrate"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "CLI to migrate SQLite agent memory databases to HDF5 format"
 license = "MIT"
@@ -14,9 +14,9 @@ name = "clawhdf5-migrate"
 path = "src/main.rs"

 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.6.0" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
-clawhdf5 = { path = "../clawhdf5", version = "2.6.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
 rusqlite = { version = "0.31", features = ["bundled"] }
 clap = { version = "4", features = ["derive"] }
 half = { workspace = true }
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-napi"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "Node.js native addon (napi-rs) exposing clawhdf5-agent to TypeScript/JavaScript"
 license = "MIT"
@@ -10,7 +10,7 @@ repository = "https://git.redclaw.dev/quantumclaw/clawhdf5"
 crate-type = ["cdylib"]

 [dependencies]
-clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.6.0" }
+clawhdf5-agent = { path = "../clawhdf5-agent", version = "2.7.0" }
 napi = { version = "2", default-features = false, features = ["napi9"] }
 napi-derive = "2"

@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-netcdf4"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "NetCDF-4 read support built on rustyhdf5 — pure Rust, no C dependencies"
 license = "MIT"
@@ -10,8 +10,8 @@ keywords = ["netcdf", "netcdf4", "hdf5", "science", "climate"]
 categories = ["parser-implementations", "science"]

 [dependencies]
-clawhdf5 = { path = "../clawhdf5", version = "2.6.0" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
+clawhdf5 = { path = "../clawhdf5", version = "2.7.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }

 [dev-dependencies]
 tempfile = { workspace = true }
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5-py"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 license = "MIT"
@@ -14,8 +14,8 @@ name = "clawhdf5"
 crate-type = ["cdylib", "rlib"]

 [dependencies]
-clawhdf5_rs = { path = "../clawhdf5", version = "2.6.0", package = "clawhdf5" }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
+clawhdf5_rs = { path = "../clawhdf5", version = "2.7.0", package = "clawhdf5" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
 pyo3 = "0.29"
 numpy = "0.29"

@@ -4,7 +4,7 @@ build-backend = "maturin"

 [project]
 name = "rustyhdf5"
-version = "2.6.0"
+version = "2.7.0"
 description = "Python bindings for rustyhdf5 — a pure-Rust HDF5 library"
 requires-python = ">=3.8"
 license = { text = "MIT" }
@@ -1,6 +1,6 @@
 [package]
 name = "clawhdf5"
-version = "2.6.0"
+version = "2.7.0"
 edition = "2024"
 description = "Pure-Rust HDF5 reader/writer — no C dependencies"
 license = "MIT"
@@ -10,16 +10,16 @@ keywords = ["hdf5", "science", "data", "binary"]
 categories = ["parser-implementations", "science", "encoding"]

 [dependencies]
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0" }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.6.0" }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0" }
 rayon = { version = "1", optional = true }

 [dev-dependencies]
 tempfile = { workspace = true }
 criterion = { workspace = true }
-clawhdf5-io = { path = "../clawhdf5-io", version = "2.6.0", features = ["mmap"] }
-clawhdf5-format = { path = "../clawhdf5-format", version = "2.6.0", features = ["parallel", "fast-checksum"] }
-clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.6.0" }
+clawhdf5-io = { path = "../clawhdf5-io", version = "2.7.0", features = ["mmap"] }
+clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0", features = ["parallel", "fast-checksum"] }
+clawhdf5-filters = { path = "../clawhdf5-filters", version = "2.7.0" }

 [[bench]]
 name = "mmap_bench"
@@ -1,6 +1,6 @@
 {
  "name": "@redclaw/clawhdf5",
-  "version": "2.6.0",
+  "version": "2.7.0",
  "description": "Node.js bindings for clawhdf5 — HDF5-backed agent memory with hippocampal consolidation",
  "main": "index.js",
  "types": "index.d.ts",