filters: Fletcher-32 as libhdf5 computes it
Our checksum reduced its sums with `% 65535`; libhdf5's H5_checksum_fletcher32 folds them with `(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves 0. On about one chunk in 32768 libhdf5 refused the chunks we wrote and we refused the chunks it wrote. Every release since v2.1.0 is affected. clawhdf5_format::checksum::fletcher32 is a port of H5_checksum_fletcher32 and the filter's only implementation. Verification also accepts the byte-swapped form libhdf5 accepts (1.6.2 and earlier) and the `% 65535` form earlier releases wrote, so their files stay readable. The new interop test compares the checksum with libhdf5's own function (ctypes) on every 1- and 2-byte input and 40 000 random and fold-heavy inputs, and moves fold-case chunks between h5py and FileBuilder/FileEditor in both directions; with the old filters.rs the three file tests fail. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -7,6 +7,41 @@ deleting it.
|
||||
|
||||
---
|
||||
|
||||
## Fletcher-32 checksums disagreed with libhdf5 on about 1 chunk in 32768
|
||||
|
||||
**Status:** fixed 2026-09-26, after v2.7.0. **Every release (v2.1.0 to
|
||||
v2.7.0) is affected**, in both directions.
|
||||
|
||||
Our Fletcher-32 reduced its two running sums with `% 65535`; libhdf5's
|
||||
`H5_checksum_fletcher32` (H5checksum.c) folds them with
|
||||
`(s & 0xffff) + (s >> 16)`. Both are arithmetic mod 65535, but where a sum
|
||||
is a non-zero multiple of 65535 the fold leaves 0xffff and the modulo 0, so
|
||||
the checksums differ — for random data about one chunk in 32768 (each of
|
||||
the two sums hits it with probability about 1/65535). Found by the review
|
||||
of the editor work: a random-edit fuzzer with gzip + Fletcher-32 hit it on
|
||||
13 of about 100 seeds.
|
||||
|
||||
- Chunks we wrote (`FileBuilder`/`FileWriter` `with_fletcher32`, and the
|
||||
unreleased `FileEditor`) with such a sum are refused by h5py and libhdf5:
|
||||
"filter returned failure during read". h5py writing `[1, 0xfffe]` as
|
||||
big-endian `u2` stores checksum `0x0001ffff`; we computed `0x00010000`.
|
||||
- Chunks libhdf5 wrote with such a sum were refused by every reader here
|
||||
with `Fletcher32Mismatch`; the data itself was never wrong.
|
||||
|
||||
**Fix:** `clawhdf5_format::checksum::fletcher32`, a port of
|
||||
`H5_checksum_fletcher32`, used by the filter for writing and verifying. It
|
||||
also accepts a checksum whose 16-bit halves are byte-swapped, as libhdf5
|
||||
does for files from 1.6.2 and earlier, and the `% 65535` form clawhdf5
|
||||
v2.7.0 and earlier wrote (the two differ only in a half that is 0xffff).
|
||||
**Test:**
|
||||
`crates/clawhdf5/tests/fletcher32_interop.rs` (libhdf5's own function
|
||||
through ctypes on every 1- and 2-byte input plus 40 000 random and
|
||||
fold-heavy inputs; h5py reads fold-case chunks from `FileBuilder` and
|
||||
`FileEditor`; we read h5py's). **Existing data:** a Fletcher-32 dataset
|
||||
written by v2.7.0 or earlier may hold chunks libhdf5 cannot read; a fixed
|
||||
build reads them. Rewrite such datasets with a fixed build (read, then
|
||||
write them again) before handing the file to libhdf5 or h5py.
|
||||
|
||||
## LZF/Blosc chunks written with a stale filter mask
|
||||
|
||||
**Status:** fixed 2026-09-26, before any release (the LZF and Blosc writers
|
||||
|
||||
Reference in New Issue
Block a user