filters: Fletcher-32 as libhdf5 computes it
Our checksum reduced its sums with `% 65535`; libhdf5's H5_checksum_fletcher32 folds them with `(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves 0. On about one chunk in 32768 libhdf5 refused the chunks we wrote and we refused the chunks it wrote. Every release since v2.1.0 is affected. clawhdf5_format::checksum::fletcher32 is a port of H5_checksum_fletcher32 and the filter's only implementation. Verification also accepts the byte-swapped form libhdf5 accepts (1.6.2 and earlier) and the `% 65535` form earlier releases wrote, so their files stay readable. The new interop test compares the checksum with libhdf5's own function (ctypes) on every 1- and 2-byte input and 40 000 random and fold-heavy inputs, and moves fold-case chunks between h5py and FileBuilder/FileEditor in both directions; with the old filters.rs the three file tests fail. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -237,7 +237,10 @@ pub fn f16_to_f32_batch(input: &[u16], output: &mut [f32]) {
|
||||
convert::f16_to_f32_batch(input, output);
|
||||
}
|
||||
|
||||
/// Compute Fletcher-32 checksum.
|
||||
/// Compute a textbook Fletcher-32 checksum (both sums start at 0xffff).
|
||||
///
|
||||
/// This is not HDF5's checksum; the Fletcher-32 I/O filter uses
|
||||
/// `clawhdf5_format::checksum::fletcher32`.
|
||||
pub fn checksum_fletcher32(data: &[u8]) -> u32 {
|
||||
checksum::checksum_fletcher32(data)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user