filters: Fletcher-32 as libhdf5 computes it
Our checksum reduced its sums with `% 65535`; libhdf5's H5_checksum_fletcher32 folds them with `(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves 0. On about one chunk in 32768 libhdf5 refused the chunks we wrote and we refused the chunks it wrote. Every release since v2.1.0 is affected. clawhdf5_format::checksum::fletcher32 is a port of H5_checksum_fletcher32 and the filter's only implementation. Verification also accepts the byte-swapped form libhdf5 accepts (1.6.2 and earlier) and the `% 65535` form earlier releases wrote, so their files stay readable. The new interop test compares the checksum with libhdf5's own function (ctypes) on every 1- and 2-byte input and 40 000 random and fold-heavy inputs, and moves fold-case chunks between h5py and FileBuilder/FileEditor in both directions; with the old filters.rs the three file tests fail. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,32 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Correctness: Fletcher-32 (2026-09-26)
|
||||
- **Fletcher-32 checksums disagreed with libhdf5's on about one chunk in
|
||||
32768** (fixed 2026-09-26). **Every release is affected, v2.1.0 through
|
||||
v2.7.0**, both directions: `FileBuilder`/`FileWriter` (`with_fletcher32`)
|
||||
and, before release, `FileEditor` wrote chunks that h5py and libhdf5
|
||||
refuse ("filter returned failure during read"), and every reader
|
||||
rejected valid libhdf5-written chunks with `Fletcher32Mismatch`. Our
|
||||
checksum reduced its sums with `% 65535`; libhdf5's
|
||||
`H5_checksum_fletcher32` uses the ones'-complement fold
|
||||
`(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves
|
||||
0, so the two differ whenever a sum is a non-zero multiple of 65535.
|
||||
`clawhdf5_format::checksum::fletcher32` (new, public) is a port of
|
||||
`H5_checksum_fletcher32` and the only implementation; the filter writes
|
||||
and verifies with it, and, as libhdf5 does, also accepts a stored
|
||||
checksum with the bytes of each 16-bit half swapped (libhdf5 1.6.2 and
|
||||
earlier) and the `% 65535` form v2.7.0 and earlier wrote, so their files
|
||||
stay readable. Tests: `crates/clawhdf5/tests/fletcher32_interop.rs` compares
|
||||
it with libhdf5's own function (through ctypes) on every 1- and 2-byte
|
||||
input and 40 000 random and fold-heavy ones, and has h5py read
|
||||
fold-case chunks written by `FileBuilder` and `FileEditor` and us read
|
||||
h5py's. Files written by earlier releases read with a fixed build; to
|
||||
make one readable by libhdf5, rewrite its Fletcher-32 datasets with a
|
||||
fixed build (see `docs/known-issues.md`). `clawhdf5_accel::checksum_fletcher32` is a
|
||||
different, textbook Fletcher-32 (sums start at 0xffff) and is not used
|
||||
for HDF5.
|
||||
|
||||
### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
|
||||
- **`FileEditor` adds, moves and resizes chunks of datasets with two or
|
||||
more unlimited dimensions** (version-2 B-tree chunk index, record types
|
||||
|
||||
Reference in New Issue
Block a user