Remote reads (range-read M2/M3: open_storage, HTTP/S3) and near-complete in-place editing #18
@@ -2,6 +2,32 @@
|
|||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
|
### Correctness: Fletcher-32 (2026-09-26)
|
||||||
|
- **Fletcher-32 checksums disagreed with libhdf5's on about one chunk in
|
||||||
|
32768** (fixed 2026-09-26). **Every release is affected, v2.1.0 through
|
||||||
|
v2.7.0**, both directions: `FileBuilder`/`FileWriter` (`with_fletcher32`)
|
||||||
|
and, before release, `FileEditor` wrote chunks that h5py and libhdf5
|
||||||
|
refuse ("filter returned failure during read"), and every reader
|
||||||
|
rejected valid libhdf5-written chunks with `Fletcher32Mismatch`. Our
|
||||||
|
checksum reduced its sums with `% 65535`; libhdf5's
|
||||||
|
`H5_checksum_fletcher32` uses the ones'-complement fold
|
||||||
|
`(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves
|
||||||
|
0, so the two differ whenever a sum is a non-zero multiple of 65535.
|
||||||
|
`clawhdf5_format::checksum::fletcher32` (new, public) is a port of
|
||||||
|
`H5_checksum_fletcher32` and the only implementation; the filter writes
|
||||||
|
and verifies with it, and, as libhdf5 does, also accepts a stored
|
||||||
|
checksum with the bytes of each 16-bit half swapped (libhdf5 1.6.2 and
|
||||||
|
earlier) and the `% 65535` form v2.7.0 and earlier wrote, so their files
|
||||||
|
stay readable. Tests: `crates/clawhdf5/tests/fletcher32_interop.rs` compares
|
||||||
|
it with libhdf5's own function (through ctypes) on every 1- and 2-byte
|
||||||
|
input and 40 000 random and fold-heavy ones, and has h5py read
|
||||||
|
fold-case chunks written by `FileBuilder` and `FileEditor` and us read
|
||||||
|
h5py's. Files written by earlier releases read with a fixed build; to
|
||||||
|
make one readable by libhdf5, rewrite its Fletcher-32 datasets with a
|
||||||
|
fixed build (see `docs/known-issues.md`). `clawhdf5_accel::checksum_fletcher32` is a
|
||||||
|
different, textbook Fletcher-32 (sums start at 0xffff) and is not used
|
||||||
|
for HDF5.
|
||||||
|
|
||||||
### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
|
### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
|
||||||
- **`FileEditor` adds, moves and resizes chunks of datasets with two or
|
- **`FileEditor` adds, moves and resizes chunks of datasets with two or
|
||||||
more unlimited dimensions** (version-2 B-tree chunk index, record types
|
more unlimited dimensions** (version-2 B-tree chunk index, record types
|
||||||
|
|||||||
@@ -237,7 +237,10 @@ pub fn f16_to_f32_batch(input: &[u16], output: &mut [f32]) {
|
|||||||
convert::f16_to_f32_batch(input, output);
|
convert::f16_to_f32_batch(input, output);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Compute Fletcher-32 checksum.
|
/// Compute a textbook Fletcher-32 checksum (both sums start at 0xffff).
|
||||||
|
///
|
||||||
|
/// This is not HDF5's checksum; the Fletcher-32 I/O filter uses
|
||||||
|
/// `clawhdf5_format::checksum::fletcher32`.
|
||||||
pub fn checksum_fletcher32(data: &[u8]) -> u32 {
|
pub fn checksum_fletcher32(data: &[u8]) -> u32 {
|
||||||
checksum::checksum_fletcher32(data)
|
checksum::checksum_fletcher32(data)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,6 +14,45 @@ pub fn jenkins_lookup3(data: &[u8]) -> u32 {
|
|||||||
hashlittle(data, 0)
|
hashlittle(data, 0)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// HDF5's Fletcher-32 checksum, as the Fletcher-32 I/O filter (filter id 3)
|
||||||
|
/// stores it after each chunk.
|
||||||
|
///
|
||||||
|
/// A line-for-line port of `H5_checksum_fletcher32` (H5checksum.c, libhdf5
|
||||||
|
/// 1.8 through 1.14): big-endian 16-bit words summed in blocks of 360, each
|
||||||
|
/// sum reduced after a block by the ones'-complement fold
|
||||||
|
/// `(s & 0xffff) + (s >> 16)` rather than `% 65535`, an odd trailing byte
|
||||||
|
/// taken as the high byte of a last word, and a final fold of both sums.
|
||||||
|
/// The fold and `% 65535` differ whenever a sum is a non-zero multiple of
|
||||||
|
/// 65535: the fold leaves 0xffff where the modulo gives 0, so the two
|
||||||
|
/// disagree on about one chunk in 32768 and libhdf5 rejects the other's
|
||||||
|
/// checksum. This must stay the only implementation.
|
||||||
|
pub fn fletcher32(data: &[u8]) -> u32 {
|
||||||
|
let mut sum1: u32 = 0;
|
||||||
|
let mut sum2: u32 = 0;
|
||||||
|
// 360 words keep both sums inside 32 bits between folds (the bound
|
||||||
|
// libhdf5 uses: after a fold sum1 < 0x10200, so sum2 stays below
|
||||||
|
// 360 * 361 / 2 * 0xffff + 360 * 0x10200 + 0x1fffe < 2^32). The adds wrap
|
||||||
|
// like the C unsigned arithmetic all the same.
|
||||||
|
let (words, odd) = data.as_chunks::<2>();
|
||||||
|
for block in words.chunks(360) {
|
||||||
|
for w in block {
|
||||||
|
sum1 = sum1.wrapping_add((u32::from(w[0]) << 8) | u32::from(w[1]));
|
||||||
|
sum2 = sum2.wrapping_add(sum1);
|
||||||
|
}
|
||||||
|
sum1 = (sum1 & 0xffff) + (sum1 >> 16);
|
||||||
|
sum2 = (sum2 & 0xffff) + (sum2 >> 16);
|
||||||
|
}
|
||||||
|
if let [last] = odd {
|
||||||
|
sum1 = sum1.wrapping_add(u32::from(*last) << 8);
|
||||||
|
sum2 = sum2.wrapping_add(sum1);
|
||||||
|
sum1 = (sum1 & 0xffff) + (sum1 >> 16);
|
||||||
|
sum2 = (sum2 & 0xffff) + (sum2 >> 16);
|
||||||
|
}
|
||||||
|
sum1 = (sum1 & 0xffff) + (sum1 >> 16);
|
||||||
|
sum2 = (sum2 & 0xffff) + (sum2 >> 16);
|
||||||
|
(sum2 << 16) | sum1
|
||||||
|
}
|
||||||
|
|
||||||
/// Compute CRC32 (IEEE / ISO 3309) over data.
|
/// Compute CRC32 (IEEE / ISO 3309) over data.
|
||||||
///
|
///
|
||||||
/// When the `fast-checksum` feature is enabled, this uses hardware CRC32
|
/// When the `fast-checksum` feature is enabled, this uses hardware CRC32
|
||||||
@@ -207,6 +246,19 @@ fn hashlittle(data: &[u8], initval: u32) -> u32 {
|
|||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
||||||
|
/// Values of libhdf5's `H5_checksum_fletcher32` (h5py 3.x's bundled
|
||||||
|
/// libhdf5, called through ctypes). The first three are sums that are
|
||||||
|
/// multiples of 65535, where `% 65535` gave 0 instead of 0xffff.
|
||||||
|
#[test]
|
||||||
|
fn fletcher32_matches_libhdf5() {
|
||||||
|
assert_eq!(fletcher32(&[0x00, 0x01, 0xff, 0xfe]), 0x0001_ffff);
|
||||||
|
assert_eq!(fletcher32(&[0xff; 720]), 0xffff_ffff);
|
||||||
|
assert_eq!(fletcher32(&[0xff; 721]), 0xff00_ff00);
|
||||||
|
assert_eq!(fletcher32(&[0xff; 1441]), 0xff00_ff00);
|
||||||
|
assert_eq!(fletcher32(&[]), 0);
|
||||||
|
assert_eq!(fletcher32(&[7]), 0x0700_0700);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn empty_input() {
|
fn empty_input() {
|
||||||
// Empty input should return the initial state after no mixing
|
// Empty input should return the initial state after no mixing
|
||||||
|
|||||||
@@ -1678,56 +1678,6 @@ fn shuffle_compress_general(data: &[u8], n: usize, element_size: usize, result:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Compute HDF5 Fletcher32 checksum over data.
|
|
||||||
/// HDF5 uses a modified Fletcher32 that operates on 16-bit words.
|
|
||||||
///
|
|
||||||
/// Optimized with wider accumulators: processes blocks of 360 words before
|
|
||||||
/// taking the modulo, reducing the number of expensive modulo operations.
|
|
||||||
/// (360 is the maximum block size that avoids u32 overflow for sum2.)
|
|
||||||
fn fletcher32_compute(data: &[u8]) -> u32 {
|
|
||||||
let mut sum1: u32 = 0;
|
|
||||||
let mut sum2: u32 = 0;
|
|
||||||
|
|
||||||
// Process in blocks of 360 16-bit words (720 bytes) to delay modulo.
|
|
||||||
// Max sum1 before mod: 360 * 65535 = 23_592_600 < u32::MAX
|
|
||||||
// Max sum2 before mod: 360 * 23_592_600 ~ 8.5B > u32::MAX, but actual
|
|
||||||
// sum2 accumulates incrementally, so worst case is 360*360*65535/2 which
|
|
||||||
// fits in u64. We use u32 with block size 360 which is safe.
|
|
||||||
const BLOCK_WORDS: usize = 360;
|
|
||||||
const BLOCK_BYTES: usize = BLOCK_WORDS * 2;
|
|
||||||
|
|
||||||
let mut offset = 0;
|
|
||||||
let len = data.len();
|
|
||||||
|
|
||||||
while offset + BLOCK_BYTES <= len {
|
|
||||||
let end = offset + BLOCK_BYTES;
|
|
||||||
let mut i = offset;
|
|
||||||
while i < end {
|
|
||||||
let val = ((data[i] as u32) << 8) | (data[i + 1] as u32);
|
|
||||||
sum1 += val;
|
|
||||||
sum2 += sum1;
|
|
||||||
i += 2;
|
|
||||||
}
|
|
||||||
sum1 %= 65535;
|
|
||||||
sum2 %= 65535;
|
|
||||||
offset = end;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Handle remaining bytes
|
|
||||||
while offset < len {
|
|
||||||
let val = if offset + 1 < len {
|
|
||||||
((data[offset] as u32) << 8) | (data[offset + 1] as u32)
|
|
||||||
} else {
|
|
||||||
(data[offset] as u32) << 8
|
|
||||||
};
|
|
||||||
sum1 = (sum1 + val) % 65535;
|
|
||||||
sum2 = (sum2 + sum1) % 65535;
|
|
||||||
offset += 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
(sum2 << 16) | sum1
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Verify Fletcher32 checksum and strip it from the data.
|
/// Verify Fletcher32 checksum and strip it from the data.
|
||||||
/// The last 4 bytes are the stored checksum.
|
/// The last 4 bytes are the stored checksum.
|
||||||
fn fletcher32_verify(data: &[u8]) -> Result<Vec<u8>, FormatError> {
|
fn fletcher32_verify(data: &[u8]) -> Result<Vec<u8>, FormatError> {
|
||||||
@@ -1749,8 +1699,18 @@ fn fletcher32_payload(data: &[u8]) -> Result<usize, FormatError> {
|
|||||||
data[data.len() - 2],
|
data[data.len() - 2],
|
||||||
data[data.len() - 1],
|
data[data.len() - 1],
|
||||||
]);
|
]);
|
||||||
let computed = fletcher32_compute(payload);
|
let computed = crate::checksum::fletcher32(payload);
|
||||||
if stored != computed {
|
// libhdf5 also accepts the checksum with the bytes of each 16-bit half
|
||||||
|
// swapped, which is how 1.6.2 and earlier stored it
|
||||||
|
// (H5Z__filter_fletcher32's `reversed_fletcher`).
|
||||||
|
let reversed = ((computed & 0x00ff_00ff) << 8) | ((computed >> 8) & 0x00ff_00ff);
|
||||||
|
// clawhdf5 v2.7.0 and earlier reduced the sums `% 65535`, which gives 0
|
||||||
|
// where libhdf5's fold gives 0xffff; accept that form too, so that files
|
||||||
|
// those releases wrote can still be read (and rewritten for libhdf5).
|
||||||
|
// It differs from `computed` only in a half that is 0xffff.
|
||||||
|
let half = |h: u32| if h == 0xffff { 0 } else { h };
|
||||||
|
let legacy = (half(computed >> 16) << 16) | half(computed & 0xffff);
|
||||||
|
if stored != computed && stored != reversed && stored != legacy {
|
||||||
return Err(FormatError::Fletcher32Mismatch {
|
return Err(FormatError::Fletcher32Mismatch {
|
||||||
expected: stored,
|
expected: stored,
|
||||||
computed,
|
computed,
|
||||||
@@ -1761,7 +1721,7 @@ fn fletcher32_payload(data: &[u8]) -> Result<usize, FormatError> {
|
|||||||
|
|
||||||
/// Append Fletcher32 checksum to data.
|
/// Append Fletcher32 checksum to data.
|
||||||
fn fletcher32_append(data: &[u8]) -> Result<Vec<u8>, FormatError> {
|
fn fletcher32_append(data: &[u8]) -> Result<Vec<u8>, FormatError> {
|
||||||
let checksum = fletcher32_compute(data);
|
let checksum = crate::checksum::fletcher32(data);
|
||||||
let mut result = data.to_vec();
|
let mut result = data.to_vec();
|
||||||
result.extend_from_slice(&checksum.to_le_bytes());
|
result.extend_from_slice(&checksum.to_le_bytes());
|
||||||
Ok(result)
|
Ok(result)
|
||||||
|
|||||||
@@ -0,0 +1,321 @@
|
|||||||
|
//! Fletcher-32 against libhdf5.
|
||||||
|
//!
|
||||||
|
//! libhdf5's `H5_checksum_fletcher32` reduces its sums with the
|
||||||
|
//! ones'-complement fold `(s & 0xffff) + (s >> 16)`, which leaves 0xffff
|
||||||
|
//! where `% 65535` leaves 0. Our checksum once used `% 65535`, so on about
|
||||||
|
//! one chunk in 32768 (a sum that is a non-zero multiple of 65535) libhdf5
|
||||||
|
//! rejected the chunks we wrote and we rejected the chunks it wrote.
|
||||||
|
//!
|
||||||
|
//! - The checksum is compared with libhdf5's own `H5_checksum_fletcher32`,
|
||||||
|
//! called through ctypes from the library h5py loads, over every one-byte
|
||||||
|
//! and two-byte input and a large corpus of random and fold-heavy inputs.
|
||||||
|
//! - Chunks engineered to hit the fold are written by `FileBuilder` and by
|
||||||
|
//! `FileEditor` and read by h5py, and written by h5py and read by us.
|
||||||
|
//!
|
||||||
|
//! Skipped when python3 with h5py is unavailable, unless
|
||||||
|
//! `CLAWHDF5_REQUIRE_INTEROP=1`.
|
||||||
|
|
||||||
|
use std::process::Command;
|
||||||
|
|
||||||
|
use clawhdf5::{File, FileBuilder, FileEditor};
|
||||||
|
use clawhdf5_format::checksum::fletcher32;
|
||||||
|
|
||||||
|
fn python() -> String {
|
||||||
|
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn have_h5py() -> bool {
|
||||||
|
let ok = Command::new(python())
|
||||||
|
.args(["-c", "import h5py, numpy"])
|
||||||
|
.output()
|
||||||
|
.is_ok_and(|o| o.status.success());
|
||||||
|
if !ok {
|
||||||
|
assert!(
|
||||||
|
std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
|
||||||
|
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
|
||||||
|
);
|
||||||
|
eprintln!("SKIP: python3 with h5py not available");
|
||||||
|
}
|
||||||
|
ok
|
||||||
|
}
|
||||||
|
|
||||||
|
fn run_python(script: &str, args: &[&str]) -> String {
|
||||||
|
let out = Command::new(python())
|
||||||
|
.arg("-c")
|
||||||
|
.arg(script)
|
||||||
|
.args(args)
|
||||||
|
.output()
|
||||||
|
.expect("failed to run python");
|
||||||
|
assert!(
|
||||||
|
out.status.success(),
|
||||||
|
"python failed:\nSTDOUT: {}\nSTDERR: {}",
|
||||||
|
String::from_utf8_lossy(&out.stdout),
|
||||||
|
String::from_utf8_lossy(&out.stderr)
|
||||||
|
);
|
||||||
|
String::from_utf8_lossy(&out.stdout).trim().to_string()
|
||||||
|
}
|
||||||
|
|
||||||
|
fn tmp(name: &str) -> std::path::PathBuf {
|
||||||
|
let dir = std::env::temp_dir().join(format!("clawhdf5_fletcher32_{}", std::process::id()));
|
||||||
|
std::fs::create_dir_all(&dir).unwrap();
|
||||||
|
dir.join(name)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The checksum our code computed before it was fixed: each sum reduced
|
||||||
|
/// `% 65535`. Only used to show that the test data hits the disagreement.
|
||||||
|
fn fletcher32_mod(data: &[u8]) -> u32 {
|
||||||
|
let (mut s1, mut s2) = (0u64, 0u64);
|
||||||
|
for w in data.chunks(2) {
|
||||||
|
let v = (u64::from(w[0]) << 8) | w.get(1).map_or(0, |&b| u64::from(b));
|
||||||
|
s1 = (s1 + v) % 65535;
|
||||||
|
s2 = (s2 + s1) % 65535;
|
||||||
|
}
|
||||||
|
((s2 as u32) << 16) | s1 as u32
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Splitmix64, so the data is the same on every run.
|
||||||
|
struct Rng(u64);
|
||||||
|
impl Rng {
|
||||||
|
fn next(&mut self) -> u64 {
|
||||||
|
self.0 = self.0.wrapping_add(0x9e37_79b9_7f4a_7c15);
|
||||||
|
let mut z = self.0;
|
||||||
|
z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
|
||||||
|
z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
|
||||||
|
z ^ (z >> 31)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Writes every case of the file `argv[1]` (u32 LE length + bytes) back to
|
||||||
|
/// `argv[2]` as libhdf5's checksum of each, u32 LE.
|
||||||
|
const LIBHDF5_CHECKSUMS: &str = r#"
|
||||||
|
import ctypes, glob, os, struct, sys
|
||||||
|
import h5py
|
||||||
|
cands = glob.glob(os.path.join(os.path.dirname(h5py.__file__), os.pardir, 'h5py.libs', 'libhdf5-*.so*'))
|
||||||
|
cands += glob.glob(os.path.join(os.path.dirname(h5py.__file__), '.dylibs', 'libhdf5*.dylib'))
|
||||||
|
if cands:
|
||||||
|
lib = ctypes.CDLL(cands[0])
|
||||||
|
else:
|
||||||
|
# A system h5py links the system libhdf5, already loaded.
|
||||||
|
import h5py.h5
|
||||||
|
lib = ctypes.CDLL(h5py.h5.__file__)
|
||||||
|
f = lib.H5_checksum_fletcher32
|
||||||
|
f.restype = ctypes.c_uint32
|
||||||
|
f.argtypes = [ctypes.c_char_p, ctypes.c_size_t]
|
||||||
|
data = open(sys.argv[1], 'rb').read()
|
||||||
|
out = bytearray()
|
||||||
|
i = 0
|
||||||
|
while i < len(data):
|
||||||
|
(n,) = struct.unpack_from('<I', data, i)
|
||||||
|
i += 4
|
||||||
|
b = data[i:i + n]
|
||||||
|
i += n
|
||||||
|
out += struct.pack('<I', f(b, n))
|
||||||
|
open(sys.argv[2], 'wb').write(out)
|
||||||
|
"#;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn checksum_matches_libhdf5() {
|
||||||
|
if !have_h5py() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let mut cases: Vec<Vec<u8>> = Vec::new();
|
||||||
|
// Every one-byte input (the odd-length path alone) and every one-word
|
||||||
|
// input (65535 = 0xffff is the smallest fold).
|
||||||
|
cases.extend((0..=255u8).map(|b| vec![b]));
|
||||||
|
cases.extend((0..=u16::MAX).map(|w| w.to_be_bytes().to_vec()));
|
||||||
|
let mut rng = Rng(0x5eed_f1e7);
|
||||||
|
// Words drawn from values that make multiples of 65535 frequent, at
|
||||||
|
// lengths around the 360-word block boundaries, odd and even.
|
||||||
|
const FOLDY: [u16; 6] = [0, 1, 0xfffe, 0xffff, 0x8000, 0x7fff];
|
||||||
|
for _ in 0..40_000 {
|
||||||
|
let len = match rng.next() % 4 {
|
||||||
|
0 => (rng.next() % 16) as usize,
|
||||||
|
1 => 718 + (rng.next() % 6) as usize,
|
||||||
|
2 => 1438 + (rng.next() % 6) as usize,
|
||||||
|
_ => (rng.next() % 3000) as usize,
|
||||||
|
};
|
||||||
|
let foldy = rng.next().is_multiple_of(2);
|
||||||
|
let mut v = Vec::with_capacity(len + 1);
|
||||||
|
while v.len() < len {
|
||||||
|
let w = if foldy {
|
||||||
|
FOLDY[(rng.next() % 6) as usize]
|
||||||
|
} else {
|
||||||
|
rng.next() as u16
|
||||||
|
};
|
||||||
|
v.extend_from_slice(&w.to_be_bytes());
|
||||||
|
}
|
||||||
|
v.truncate(len);
|
||||||
|
cases.push(v);
|
||||||
|
}
|
||||||
|
// Long runs of 0xff: sums are multiples of 65535 at every block.
|
||||||
|
for len in [720, 721, 1440, 1441, 7200, 65536, 65537] {
|
||||||
|
cases.push(vec![0xff; len]);
|
||||||
|
}
|
||||||
|
let mut blob = Vec::new();
|
||||||
|
for c in &cases {
|
||||||
|
blob.extend_from_slice(&(c.len() as u32).to_le_bytes());
|
||||||
|
blob.extend_from_slice(c);
|
||||||
|
}
|
||||||
|
let input = tmp("cases.bin");
|
||||||
|
let output = tmp("sums.bin");
|
||||||
|
std::fs::write(&input, &blob).unwrap();
|
||||||
|
run_python(
|
||||||
|
LIBHDF5_CHECKSUMS,
|
||||||
|
&[input.to_str().unwrap(), output.to_str().unwrap()],
|
||||||
|
);
|
||||||
|
let sums = std::fs::read(&output).unwrap();
|
||||||
|
assert_eq!(sums.len(), cases.len() * 4);
|
||||||
|
let mut folds = 0;
|
||||||
|
for (c, s) in cases.iter().zip(sums.as_chunks::<4>().0) {
|
||||||
|
let want = u32::from_le_bytes(*s);
|
||||||
|
assert_eq!(
|
||||||
|
fletcher32(c),
|
||||||
|
want,
|
||||||
|
"checksum of {} bytes {:02x?}...",
|
||||||
|
c.len(),
|
||||||
|
&c[..c.len().min(16)]
|
||||||
|
);
|
||||||
|
if fletcher32_mod(c) != want {
|
||||||
|
folds += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// The corpus must exercise the case `% 65535` got wrong.
|
||||||
|
assert!(folds > 500, "only {folds} fold cases");
|
||||||
|
}
|
||||||
|
|
||||||
|
const CHUNK: usize = 8;
|
||||||
|
|
||||||
|
/// `n` chunks of `CHUNK` bytes, each one a chunk on which the old
|
||||||
|
/// `% 65535` checksum and libhdf5's differ (sum1, sum2 or both a non-zero
|
||||||
|
/// multiple of 65535), with an ordinary chunk between them.
|
||||||
|
fn fold_chunks(n: usize) -> Vec<u8> {
|
||||||
|
let mut rng = Rng(42);
|
||||||
|
let mut out = Vec::new();
|
||||||
|
let mut found = 0;
|
||||||
|
while found < n {
|
||||||
|
// Build a chunk whose sum1 is a multiple of 65535 half the time,
|
||||||
|
// otherwise search at random for a sum2 fold.
|
||||||
|
let mut c: Vec<u8> = (0..CHUNK).map(|_| rng.next() as u8).collect();
|
||||||
|
if found % 2 == 0 {
|
||||||
|
let words: u64 = c[..CHUNK - 2]
|
||||||
|
.chunks(2)
|
||||||
|
.map(|w| (u64::from(w[0]) << 8) | u64::from(w[1]))
|
||||||
|
.sum();
|
||||||
|
let last = ((65535 - words % 65535) % 65535) as u16;
|
||||||
|
c[CHUNK - 2..].copy_from_slice(&last.to_be_bytes());
|
||||||
|
}
|
||||||
|
if fletcher32(&c) != fletcher32_mod(&c) {
|
||||||
|
out.extend_from_slice(&c);
|
||||||
|
out.extend((0..CHUNK).map(|i| i as u8 + 1));
|
||||||
|
found += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn h5py_reads_fold_case_chunks_we_write() {
|
||||||
|
if !have_h5py() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let data = fold_chunks(32);
|
||||||
|
// FileBuilder.
|
||||||
|
let built = tmp("built.h5");
|
||||||
|
let mut b = FileBuilder::new();
|
||||||
|
b.create_dataset("d")
|
||||||
|
.with_u8_data(&data)
|
||||||
|
.with_chunks(&[CHUNK as u64])
|
||||||
|
.with_fletcher32();
|
||||||
|
b.write(&built).unwrap();
|
||||||
|
// FileEditor, into a dataset h5py created.
|
||||||
|
let edited = tmp("edited.h5");
|
||||||
|
run_python(
|
||||||
|
"import sys, h5py, numpy as np\n\
|
||||||
|
with h5py.File(sys.argv[1], 'w') as f:\n\
|
||||||
|
\x20 f.create_dataset('d', data=np.zeros(int(sys.argv[2]), 'u1'), chunks=(8,), fletcher32=True)",
|
||||||
|
&[edited.to_str().unwrap(), &data.len().to_string()],
|
||||||
|
);
|
||||||
|
FileEditor::open(&edited)
|
||||||
|
.unwrap()
|
||||||
|
.write_all("d", &data)
|
||||||
|
.unwrap();
|
||||||
|
for path in [&built, &edited] {
|
||||||
|
let got = run_python(
|
||||||
|
"import sys, h5py\n\
|
||||||
|
with h5py.File(sys.argv[1], 'r') as f:\n\
|
||||||
|
\x20 assert f['d'].fletcher32\n\
|
||||||
|
\x20 print(f['d'][:].tobytes().hex())",
|
||||||
|
&[path.to_str().unwrap()],
|
||||||
|
);
|
||||||
|
assert_eq!(got, hex(&data), "{}", path.display());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn we_read_fold_case_chunks_h5py_writes() {
|
||||||
|
if !have_h5py() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let data = fold_chunks(32);
|
||||||
|
let path = tmp("h5py.h5");
|
||||||
|
run_python(
|
||||||
|
"import sys, h5py, numpy as np\n\
|
||||||
|
with h5py.File(sys.argv[1], 'w') as f:\n\
|
||||||
|
\x20 f.create_dataset('d', data=np.frombuffer(bytes.fromhex(sys.argv[2]), 'u1'), chunks=(8,), fletcher32=True)",
|
||||||
|
&[path.to_str().unwrap(), &hex(&data)],
|
||||||
|
);
|
||||||
|
let file = File::open(&path).unwrap();
|
||||||
|
let ds = file.dataset("d").unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
ds.read_selection(&clawhdf5_format::selection::Selection::All)
|
||||||
|
.unwrap(),
|
||||||
|
data
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A checksum stored with the bytes of each 16-bit half swapped, as
|
||||||
|
/// libhdf5 1.6.2 and earlier wrote it, is accepted as libhdf5 accepts it;
|
||||||
|
/// so is the `% 65535` form clawhdf5 v2.7.0 and earlier wrote, so that
|
||||||
|
/// their files stay readable.
|
||||||
|
#[test]
|
||||||
|
fn legacy_checksums_are_accepted() {
|
||||||
|
use clawhdf5_format::filter_pipeline::{FILTER_FLETCHER32, FilterDescription, FilterPipeline};
|
||||||
|
let payload = [1u8, 2, 3, 4, 5];
|
||||||
|
let sum = fletcher32(&payload);
|
||||||
|
let swapped = ((sum & 0x00ff_00ff) << 8) | ((sum >> 8) & 0x00ff_00ff);
|
||||||
|
assert_ne!(sum, swapped);
|
||||||
|
let pipeline = FilterPipeline {
|
||||||
|
version: 2,
|
||||||
|
filters: vec![FilterDescription {
|
||||||
|
filter_id: FILTER_FLETCHER32,
|
||||||
|
name: None,
|
||||||
|
client_data: vec![],
|
||||||
|
flags: 0,
|
||||||
|
}],
|
||||||
|
};
|
||||||
|
for stored in [sum, swapped] {
|
||||||
|
let mut chunk = payload.to_vec();
|
||||||
|
chunk.extend_from_slice(&stored.to_le_bytes());
|
||||||
|
let out = clawhdf5_format::filters::decompress_chunk(&chunk, &pipeline, payload.len(), 1)
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(out, payload);
|
||||||
|
}
|
||||||
|
// Our old checksum of a fold-case chunk.
|
||||||
|
let fold = fold_chunks(1);
|
||||||
|
let fold = &fold[..CHUNK];
|
||||||
|
let old = fletcher32_mod(fold);
|
||||||
|
assert_ne!(old, fletcher32(fold));
|
||||||
|
let mut chunk = fold.to_vec();
|
||||||
|
chunk.extend_from_slice(&old.to_le_bytes());
|
||||||
|
let out = clawhdf5_format::filters::decompress_chunk(&chunk, &pipeline, CHUNK, 1).unwrap();
|
||||||
|
assert_eq!(out, fold);
|
||||||
|
let mut chunk = payload.to_vec();
|
||||||
|
chunk.extend_from_slice(&(sum ^ 1).to_le_bytes());
|
||||||
|
assert!(
|
||||||
|
clawhdf5_format::filters::decompress_chunk(&chunk, &pipeline, payload.len(), 1).is_err()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn hex(b: &[u8]) -> String {
|
||||||
|
b.iter().map(|x| format!("{x:02x}")).collect()
|
||||||
|
}
|
||||||
@@ -7,6 +7,41 @@ deleting it.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Fletcher-32 checksums disagreed with libhdf5 on about 1 chunk in 32768
|
||||||
|
|
||||||
|
**Status:** fixed 2026-09-26, after v2.7.0. **Every release (v2.1.0 to
|
||||||
|
v2.7.0) is affected**, in both directions.
|
||||||
|
|
||||||
|
Our Fletcher-32 reduced its two running sums with `% 65535`; libhdf5's
|
||||||
|
`H5_checksum_fletcher32` (H5checksum.c) folds them with
|
||||||
|
`(s & 0xffff) + (s >> 16)`. Both are arithmetic mod 65535, but where a sum
|
||||||
|
is a non-zero multiple of 65535 the fold leaves 0xffff and the modulo 0, so
|
||||||
|
the checksums differ — for random data about one chunk in 32768 (each of
|
||||||
|
the two sums hits it with probability about 1/65535). Found by the review
|
||||||
|
of the editor work: a random-edit fuzzer with gzip + Fletcher-32 hit it on
|
||||||
|
13 of about 100 seeds.
|
||||||
|
|
||||||
|
- Chunks we wrote (`FileBuilder`/`FileWriter` `with_fletcher32`, and the
|
||||||
|
unreleased `FileEditor`) with such a sum are refused by h5py and libhdf5:
|
||||||
|
"filter returned failure during read". h5py writing `[1, 0xfffe]` as
|
||||||
|
big-endian `u2` stores checksum `0x0001ffff`; we computed `0x00010000`.
|
||||||
|
- Chunks libhdf5 wrote with such a sum were refused by every reader here
|
||||||
|
with `Fletcher32Mismatch`; the data itself was never wrong.
|
||||||
|
|
||||||
|
**Fix:** `clawhdf5_format::checksum::fletcher32`, a port of
|
||||||
|
`H5_checksum_fletcher32`, used by the filter for writing and verifying. It
|
||||||
|
also accepts a checksum whose 16-bit halves are byte-swapped, as libhdf5
|
||||||
|
does for files from 1.6.2 and earlier, and the `% 65535` form clawhdf5
|
||||||
|
v2.7.0 and earlier wrote (the two differ only in a half that is 0xffff).
|
||||||
|
**Test:**
|
||||||
|
`crates/clawhdf5/tests/fletcher32_interop.rs` (libhdf5's own function
|
||||||
|
through ctypes on every 1- and 2-byte input plus 40 000 random and
|
||||||
|
fold-heavy inputs; h5py reads fold-case chunks from `FileBuilder` and
|
||||||
|
`FileEditor`; we read h5py's). **Existing data:** a Fletcher-32 dataset
|
||||||
|
written by v2.7.0 or earlier may hold chunks libhdf5 cannot read; a fixed
|
||||||
|
build reads them. Rewrite such datasets with a fixed build (read, then
|
||||||
|
write them again) before handing the file to libhdf5 or h5py.
|
||||||
|
|
||||||
## LZF/Blosc chunks written with a stale filter mask
|
## LZF/Blosc chunks written with a stale filter mask
|
||||||
|
|
||||||
**Status:** fixed 2026-09-26, before any release (the LZF and Blosc writers
|
**Status:** fixed 2026-09-26, before any release (the LZF and Blosc writers
|
||||||
|
|||||||
Reference in New Issue
Block a user