filters: Fletcher-32 as libhdf5 computes it

Our checksum reduced its sums with `% 65535`; libhdf5's
H5_checksum_fletcher32 folds them with `(s & 0xffff) + (s >> 16)`, which
leaves 0xffff where the modulo leaves 0. On about one chunk in 32768
libhdf5 refused the chunks we wrote and we refused the chunks it wrote.
Every release since v2.1.0 is affected.

clawhdf5_format::checksum::fletcher32 is a port of H5_checksum_fletcher32
and the filter's only implementation. Verification also accepts the
byte-swapped form libhdf5 accepts (1.6.2 and earlier) and the `% 65535`
form earlier releases wrote, so their files stay readable.

The new interop test compares the checksum with libhdf5's own function
(ctypes) on every 1- and 2-byte input and 40 000 random and fold-heavy
inputs, and moves fold-case chunks between h5py and FileBuilder/FileEditor
in both directions; with the old filters.rs the three file tests fail.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:50:23 -05:00
co-authored by Claude Opus 5.5
parent 4e8109770d
commit 159e588550
6 changed files with 451 additions and 54 deletions
+4 -1
View File
@@ -237,7 +237,10 @@ pub fn f16_to_f32_batch(input: &[u16], output: &mut [f32]) {
convert::f16_to_f32_batch(input, output);
}
/// Compute Fletcher-32 checksum.
/// Compute a textbook Fletcher-32 checksum (both sums start at 0xffff).
///
/// This is not HDF5's checksum; the Fletcher-32 I/O filter uses
/// `clawhdf5_format::checksum::fletcher32`.
pub fn checksum_fletcher32(data: &[u8]) -> u32 {
checksum::checksum_fletcher32(data)
}