filters: Fletcher-32 as libhdf5 computes it

Our checksum reduced its sums with `% 65535`; libhdf5's
H5_checksum_fletcher32 folds them with `(s & 0xffff) + (s >> 16)`, which
leaves 0xffff where the modulo leaves 0. On about one chunk in 32768
libhdf5 refused the chunks we wrote and we refused the chunks it wrote.
Every release since v2.1.0 is affected.

clawhdf5_format::checksum::fletcher32 is a port of H5_checksum_fletcher32
and the filter's only implementation. Verification also accepts the
byte-swapped form libhdf5 accepts (1.6.2 and earlier) and the `% 65535`
form earlier releases wrote, so their files stay readable.

The new interop test compares the checksum with libhdf5's own function
(ctypes) on every 1- and 2-byte input and 40 000 random and fold-heavy
inputs, and moves fold-case chunks between h5py and FileBuilder/FileEditor
in both directions; with the old filters.rs the three file tests fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:50:23 -05:00
co-authored by Claude Opus 5.5
parent 4e8109770d
commit 159e588550
6 changed files with 451 additions and 54 deletions
+26
View File
@@ -2,6 +2,32 @@
## Unreleased
### Correctness: Fletcher-32 (2026-09-26)
- **Fletcher-32 checksums disagreed with libhdf5's on about one chunk in
32768** (fixed 2026-09-26). **Every release is affected, v2.1.0 through
v2.7.0**, both directions: `FileBuilder`/`FileWriter` (`with_fletcher32`)
and, before release, `FileEditor` wrote chunks that h5py and libhdf5
refuse ("filter returned failure during read"), and every reader
rejected valid libhdf5-written chunks with `Fletcher32Mismatch`. Our
checksum reduced its sums with `% 65535`; libhdf5's
`H5_checksum_fletcher32` uses the ones'-complement fold
`(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves
0, so the two differ whenever a sum is a non-zero multiple of 65535.
`clawhdf5_format::checksum::fletcher32` (new, public) is a port of
`H5_checksum_fletcher32` and the only implementation; the filter writes
and verifies with it, and, as libhdf5 does, also accepts a stored
checksum with the bytes of each 16-bit half swapped (libhdf5 1.6.2 and
earlier) and the `% 65535` form v2.7.0 and earlier wrote, so their files
stay readable. Tests: `crates/clawhdf5/tests/fletcher32_interop.rs` compares
it with libhdf5's own function (through ctypes) on every 1- and 2-byte
input and 40 000 random and fold-heavy ones, and has h5py read
fold-case chunks written by `FileBuilder` and `FileEditor` and us read
h5py's. Files written by earlier releases read with a fixed build; to
make one readable by libhdf5, rewrite its Fletcher-32 datasets with a
fixed build (see `docs/known-issues.md`). `clawhdf5_accel::checksum_fletcher32` is a
different, textbook Fletcher-32 (sums start at 0xffff) and is not used
for HDF5.
### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
- **`FileEditor` adds, moves and resizes chunks of datasets with two or
more unlimited dimensions** (version-2 B-tree chunk index, record types