chore(release): v2.7.0
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
+62
-50
@@ -1,56 +1,20 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased
|
||||
## v2.7.0 (2026-09-20)
|
||||
|
||||
### Tuning
|
||||
- `clawhdf5-agent`: **the HNSW parameters are configurable** —
|
||||
`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search`
|
||||
(defaults 16, 64, and 0 meaning "scale with `k`", i.e. today's behaviour).
|
||||
They were constants, so a deployment could not trade recall against memory
|
||||
or query speed at all. All three are persisted with the store. Values are
|
||||
clamped where the index requires it: `clawhdf5-ann` asserts a graph degree
|
||||
of at least 2, so a configured 0 — from a file, or from a caller who took 0
|
||||
to mean "default" — used to abort the process inside the builder. Lowering
|
||||
`ef_search` also no longer narrows the candidate pool that fusion sees.
|
||||
**Breaking:** `MemoryConfig` gained fields, so literal constructions need
|
||||
updating; `..Default::default()` does not.
|
||||
|
||||
### Performance
|
||||
- `clawhdf5-agent`: **opening a store is ~28% faster** (455 ms -> 327 ms at
|
||||
100k x 384). `read_from_disk` memory-mapped the file and then copied the
|
||||
entire mapping into a `Vec` for `File::from_bytes`, when `File::open`
|
||||
memory-maps it directly — so every open paid a full-file memcpy for nothing.
|
||||
Process peak memory is unchanged: the peak falls after the parse, during the
|
||||
index build, so the transient never reached the high-water mark. The
|
||||
footprint harness now reports that peak next to the retained figure, which
|
||||
is how this was checked rather than assumed.
|
||||
|
||||
### Integrity
|
||||
- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
|
||||
their checksums** (the `checksum` feature, on by default). Every structure
|
||||
in both — header, index block, super block, data block and each data block
|
||||
page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
|
||||
The consequence of skipping it is not a missing warning but wrong data: a
|
||||
single flipped bit in a chunk address still parses, still points inside the
|
||||
file, and the reader hands back whatever bytes now sit there as the chunk's
|
||||
contents. Verified in both directions — the checksums accept files written
|
||||
by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
|
||||
and an interop test corrupts an address to confirm the read now fails
|
||||
instead of returning data (it does return data when the check is removed).
|
||||
|
||||
### Performance
|
||||
- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
|
||||
sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
|
||||
elsewhere). The quantised HNSW index used a scalar loop while the `f32` path
|
||||
it was measured against ran AVX2, so the ~13% throughput cost recorded for
|
||||
`MemoryConfig::quantized_index` was a missing kernel rather than a property
|
||||
of int8. With the kernel, at N = 100 000 x 384 and equal recall, the
|
||||
quantised index answers **1.63x as many queries per second** (21 848 vs
|
||||
13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778
|
||||
vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three
|
||||
alternating runs. It remains off by default only because the kernel is
|
||||
AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so
|
||||
the SIMD path is tested to agree with scalar bit for bit.
|
||||
### Upgrade Notes
|
||||
- **Two read-path bugs fixed, one of them silent.** Datasets indexed by an
|
||||
Extensible Array (any dataset with one unlimited dimension) returned data
|
||||
from the wrong chunks past their first few dozen. If you have readings taken
|
||||
from such a dataset with an earlier release, they may be wrong; re-read them.
|
||||
- **A corrupt chunk index is now an error.** Fixed and Extensible Array
|
||||
structures carry checksums that were previously ignored, so damage surfaced
|
||||
as plausible data from the wrong offset. Code that read a damaged file and
|
||||
got numbers will now get `ChecksumMismatch` instead. That is the point.
|
||||
- **Breaking:** `MemoryConfig` gained `hnsw_m`, `hnsw_ef_construction` and
|
||||
`hnsw_ef_search`, so literal constructions need updating;
|
||||
`..Default::default()` does not. All three default to the previous
|
||||
behaviour.
|
||||
|
||||
### Correctness
|
||||
- `clawhdf5-format`: **datasets indexed by an Extensible Array returned wrong
|
||||
@@ -100,6 +64,54 @@
|
||||
including a depth-2 HDF5 2.0 chunk index with 40 000 records, now covered by
|
||||
an interop test.
|
||||
|
||||
### Integrity
|
||||
- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
|
||||
their checksums** (the `checksum` feature, on by default). Every structure
|
||||
in both — header, index block, super block, data block and each data block
|
||||
page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
|
||||
The consequence of skipping it is not a missing warning but wrong data: a
|
||||
single flipped bit in a chunk address still parses, still points inside the
|
||||
file, and the reader hands back whatever bytes now sit there as the chunk's
|
||||
contents. Verified in both directions — the checksums accept files written
|
||||
by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
|
||||
and an interop test corrupts an address to confirm the read now fails
|
||||
instead of returning data (it does return data when the check is removed).
|
||||
|
||||
### Performance
|
||||
- `clawhdf5-agent`: **opening a store is ~28% faster** (455 ms -> 327 ms at
|
||||
100k x 384). `read_from_disk` memory-mapped the file and then copied the
|
||||
entire mapping into a `Vec` for `File::from_bytes`, when `File::open`
|
||||
memory-maps it directly — so every open paid a full-file memcpy for nothing.
|
||||
Process peak memory is unchanged: the peak falls after the parse, during the
|
||||
index build, so the transient never reached the high-water mark. The
|
||||
footprint harness now reports that peak next to the retained figure, which
|
||||
is how this was checked rather than assumed.
|
||||
- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
|
||||
sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
|
||||
elsewhere). The quantised HNSW index used a scalar loop while the `f32` path
|
||||
it was measured against ran AVX2, so the ~13% throughput cost recorded for
|
||||
`MemoryConfig::quantized_index` was a missing kernel rather than a property
|
||||
of int8. With the kernel, at N = 100 000 x 384 and equal recall, the
|
||||
quantised index answers **1.63x as many queries per second** (21 848 vs
|
||||
13 399 at ef=64, recall 0.9940 vs 0.9945) and builds **1.8x faster** (1778
|
||||
vs 3197 ms) — on top of holding a quarter of the vectors. Medians of three
|
||||
alternating runs. It remains off by default only because the kernel is
|
||||
AVX2-only and aarch64 falls back to the scalar loop. Integer arithmetic, so
|
||||
the SIMD path is tested to agree with scalar bit for bit.
|
||||
|
||||
### Tuning
|
||||
- `clawhdf5-agent`: **the HNSW parameters are configurable** —
|
||||
`MemoryConfig::hnsw_m`, `hnsw_ef_construction` and `hnsw_ef_search`
|
||||
(defaults 16, 64, and 0 meaning "scale with `k`", i.e. today's behaviour).
|
||||
They were constants, so a deployment could not trade recall against memory
|
||||
or query speed at all. All three are persisted with the store. Values are
|
||||
clamped where the index requires it: `clawhdf5-ann` asserts a graph degree
|
||||
of at least 2, so a configured 0 — from a file, or from a caller who took 0
|
||||
to mean "default" — used to abort the process inside the builder. Lowering
|
||||
`ef_search` also no longer narrows the candidate pool that fusion sees.
|
||||
**Breaking:** `MemoryConfig` gained fields, so literal constructions need
|
||||
updating; `..Default::default()` does not.
|
||||
|
||||
### Documentation
|
||||
- `clawhdf5-agent`: `BM25Index::search` claimed to use Block-Max WAND for early
|
||||
termination. It never did; it scores every match exhaustively. It now says
|
||||
|
||||
Reference in New Issue
Block a user