chore(release): v2.7.0

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-20 18:24:07 -07:00
co-authored by Claude Opus 5
parent f4c6d43a3f
commit 4a5544da1d
20 changed files with 106 additions and 94 deletions
+62 -50
View File
@@ -1,56 +1,20 @@
# Changelog
## Unreleased
## v2.7.0 (2026-09-20)
### Tuning
- `clawhdf5-agent`: **the HNSW parameters are configurable**
`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search`
(defaults 16, 64, and 0 meaning "scale with `k`", i.e. today's behaviour).
They were constants, so a deployment could not trade recall against memory
or query speed at all. All three are persisted with the store. Values are
clamped where the index requires it: `clawhdf5-ann` asserts a graph degree
of at least 2, so a configured 0 — from a file, or from a caller who took 0
to mean "default" — used to abort the process inside the builder. Lowering
`ef_search` also no longer narrows the candidate pool that fusion sees.
**Breaking:** `MemoryConfig` gained fields, so literal constructions need
updating; `..Default::default()` does not.
### Performance
- `clawhdf5-agent`: **opening a store is ~28% faster** (455 ms -> 327 ms at
100k x 384). `read_from_disk` memory-mapped the file and then copied the
entire mapping into a `Vec` for `File::from_bytes`, when `File::open`
memory-maps it directly — so every open paid a full-file memcpy for nothing.
Process peak memory is unchanged: the peak falls after the parse, during the
index build, so the transient never reached the high-water mark. The
footprint harness now reports that peak next to the retained figure, which
is how this was checked rather than assumed.
### Integrity
- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
their checksums** (the `checksum` feature, on by default). Every structure
in both — header, index block, super block, data block and each data block
page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
The consequence of skipping it is not a missing warning but wrong data: a
single flipped bit in a chunk address still parses, still points inside the
file, and the reader hands back whatever bytes now sit there as the chunk's
contents. Verified in both directions — the checksums accept files written
by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
and an interop test corrupts an address to confirm the read now fails
instead of returning data (it does return data when the check is removed).
### Performance
- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
elsewhere). The quantised HNSW index used a scalar loop while the `f32` path
it was measured against ran AVX2, so the ~13% throughput cost recorded for
`MemoryConfig::quantized_index` was a missing kernel rather than a property
of int8. With the kernel, at N = 100 000 x 384 and equal recall, the
quantised index answers **1.63x as many queries per second** (21 848 vs
13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778
vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three
alternating runs. It remains off by default only because the kernel is
AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so
the SIMD path is tested to agree with scalar bit for bit.
### Upgrade Notes
- **Two read-path bugs fixed, one of them silent.** Datasets indexed by an
Extensible Array (any dataset with one unlimited dimension) returned data
from the wrong chunks past their first few dozen. If you have readings taken
from such a dataset with an earlier release, they may be wrong; re-read them.
- **A corrupt chunk index is now an error.** Fixed and Extensible Array
structures carry checksums that were previously ignored, so damage surfaced
as plausible data from the wrong offset. Code that read a damaged file and
got numbers will now get `ChecksumMismatch` instead. That is the point.
- **Breaking:** `MemoryConfig` gained `hnsw_m`, `hnsw_ef_construction` and
`hnsw_ef_search`, so literal constructions need updating;
`..Default::default()` does not. All three default to the previous
behaviour.
### Correctness
- `clawhdf5-format`: **datasets indexed by an Extensible Array returned wrong
@@ -100,6 +64,54 @@
including a depth-2 HDF5 2.0 chunk index with 40 000 records, now covered by
an interop test.
### Integrity
- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
their checksums** (the `checksum` feature, on by default). Every structure
in both — header, index block, super block, data block and each data block
page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
The consequence of skipping it is not a missing warning but wrong data: a
single flipped bit in a chunk address still parses, still points inside the
file, and the reader hands back whatever bytes now sit there as the chunk's
contents. Verified in both directions — the checksums accept files written
by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
and an interop test corrupts an address to confirm the read now fails
instead of returning data (it does return data when the check is removed).
### Performance
- `clawhdf5-agent`: **opening a store is ~28% faster** (455 ms -> 327 ms at
100k x 384). `read_from_disk` memory-mapped the file and then copied the
entire mapping into a `Vec` for `File::from_bytes`, when `File::open`
memory-maps it directly — so every open paid a full-file memcpy for nothing.
Process peak memory is unchanged: the peak falls after the parse, during the
index build, so the transient never reached the high-water mark. The
footprint harness now reports that peak next to the retained figure, which
is how this was checked rather than assumed.
- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
elsewhere). The quantised HNSW index used a scalar loop while the `f32` path
it was measured against ran AVX2, so the ~13% throughput cost recorded for
`MemoryConfig::quantized_index` was a missing kernel rather than a property
of int8. With the kernel, at N = 100 000 x 384 and equal recall, the
quantised index answers **1.63x as many queries per second** (21 848 vs
13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778
vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three
alternating runs. It remains off by default only because the kernel is
AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so
the SIMD path is tested to agree with scalar bit for bit.
### Tuning
- `clawhdf5-agent`: **the HNSW parameters are configurable**
`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search`
(defaults 16, 64, and 0 meaning "scale with `k`", i.e. today's behaviour).
They were constants, so a deployment could not trade recall against memory
or query speed at all. All three are persisted with the store. Values are
clamped where the index requires it: `clawhdf5-ann` asserts a graph degree
of at least 2, so a configured 0 — from a file, or from a caller who took 0
to mean "default" — used to abort the process inside the builder. Lowering
`ef_search` also no longer narrows the candidate pool that fusion sees.
**Breaking:** `MemoryConfig` gained fields, so literal constructions need
updating; `..Default::default()` does not.
### Documentation
- `clawhdf5-agent`: `BM25Index::search` claimed to use Block-Max WAND for early
termination. It never did; it scores every match exhaustively. It now says