Remote reads (range-read M2/M3: open_storage, HTTP/S3) and near-complete in-place editing #18

Merged
osobh merged 52 commits from feat/p3-remote-editor into main 2026-09-27 11:15:23 +00:00
6 changed files with 451 additions and 54 deletions
Showing only changes of commit 159e588550 - Show all commits
+26
View File
@@ -2,6 +2,32 @@
## Unreleased ## Unreleased
### Correctness: Fletcher-32 (2026-09-26)
- **Fletcher-32 checksums disagreed with libhdf5's on about one chunk in
32768** (fixed 2026-09-26). **Every release is affected, v2.1.0 through
v2.7.0**, both directions: `FileBuilder`/`FileWriter` (`with_fletcher32`)
and, before release, `FileEditor` wrote chunks that h5py and libhdf5
refuse ("filter returned failure during read"), and every reader
rejected valid libhdf5-written chunks with `Fletcher32Mismatch`. Our
checksum reduced its sums with `% 65535`; libhdf5's
`H5_checksum_fletcher32` uses the ones'-complement fold
`(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves
0, so the two differ whenever a sum is a non-zero multiple of 65535.
`clawhdf5_format::checksum::fletcher32` (new, public) is a port of
`H5_checksum_fletcher32` and the only implementation; the filter writes
and verifies with it, and, as libhdf5 does, also accepts a stored
checksum with the bytes of each 16-bit half swapped (libhdf5 1.6.2 and
earlier) and the `% 65535` form v2.7.0 and earlier wrote, so their files
stay readable. Tests: `crates/clawhdf5/tests/fletcher32_interop.rs` compares
it with libhdf5's own function (through ctypes) on every 1- and 2-byte
input and 40 000 random and fold-heavy ones, and has h5py read
fold-case chunks written by `FileBuilder` and `FileEditor` and us read
h5py's. Files written by earlier releases read with a fixed build; to
make one readable by libhdf5, rewrite its Fletcher-32 datasets with a
fixed build (see `docs/known-issues.md`). `clawhdf5_accel::checksum_fletcher32` is a
different, textbook Fletcher-32 (sums start at 0xffff) and is not used
for HDF5.
### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26) ### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
- **`FileEditor` adds, moves and resizes chunks of datasets with two or - **`FileEditor` adds, moves and resizes chunks of datasets with two or
more unlimited dimensions** (version-2 B-tree chunk index, record types more unlimited dimensions** (version-2 B-tree chunk index, record types
+4 -1
View File
@@ -237,7 +237,10 @@ pub fn f16_to_f32_batch(input: &[u16], output: &mut [f32]) {
convert::f16_to_f32_batch(input, output); convert::f16_to_f32_batch(input, output);
} }
/// Compute Fletcher-32 checksum. /// Compute a textbook Fletcher-32 checksum (both sums start at 0xffff).
///
/// This is not HDF5's checksum; the Fletcher-32 I/O filter uses
/// `clawhdf5_format::checksum::fletcher32`.
pub fn checksum_fletcher32(data: &[u8]) -> u32 { pub fn checksum_fletcher32(data: &[u8]) -> u32 {
checksum::checksum_fletcher32(data) checksum::checksum_fletcher32(data)
} }
+52
View File
@@ -14,6 +14,45 @@ pub fn jenkins_lookup3(data: &[u8]) -> u32 {
hashlittle(data, 0) hashlittle(data, 0)
} }
/// HDF5's Fletcher-32 checksum, as the Fletcher-32 I/O filter (filter id 3)
/// stores it after each chunk.
///
/// A line-for-line port of `H5_checksum_fletcher32` (H5checksum.c, libhdf5
/// 1.8 through 1.14): big-endian 16-bit words summed in blocks of 360, each
/// sum reduced after a block by the ones'-complement fold
/// `(s & 0xffff) + (s >> 16)` rather than `% 65535`, an odd trailing byte
/// taken as the high byte of a last word, and a final fold of both sums.
/// The fold and `% 65535` differ whenever a sum is a non-zero multiple of
/// 65535: the fold leaves 0xffff where the modulo gives 0, so the two
/// disagree on about one chunk in 32768 and libhdf5 rejects the other's
/// checksum. This must stay the only implementation.
pub fn fletcher32(data: &[u8]) -> u32 {
let mut sum1: u32 = 0;
let mut sum2: u32 = 0;
// 360 words keep both sums inside 32 bits between folds (the bound
// libhdf5 uses: after a fold sum1 < 0x10200, so sum2 stays below
// 360 * 361 / 2 * 0xffff + 360 * 0x10200 + 0x1fffe < 2^32). The adds wrap
// like the C unsigned arithmetic all the same.
let (words, odd) = data.as_chunks::<2>();
for block in words.chunks(360) {
for w in block {
sum1 = sum1.wrapping_add((u32::from(w[0]) << 8) | u32::from(w[1]));
sum2 = sum2.wrapping_add(sum1);
}
sum1 = (sum1 & 0xffff) + (sum1 >> 16);
sum2 = (sum2 & 0xffff) + (sum2 >> 16);
}
if let [last] = odd {
sum1 = sum1.wrapping_add(u32::from(*last) << 8);
sum2 = sum2.wrapping_add(sum1);
sum1 = (sum1 & 0xffff) + (sum1 >> 16);
sum2 = (sum2 & 0xffff) + (sum2 >> 16);
}
sum1 = (sum1 & 0xffff) + (sum1 >> 16);
sum2 = (sum2 & 0xffff) + (sum2 >> 16);
(sum2 << 16) | sum1
}
/// Compute CRC32 (IEEE / ISO 3309) over data. /// Compute CRC32 (IEEE / ISO 3309) over data.
/// ///
/// When the `fast-checksum` feature is enabled, this uses hardware CRC32 /// When the `fast-checksum` feature is enabled, this uses hardware CRC32
@@ -207,6 +246,19 @@ fn hashlittle(data: &[u8], initval: u32) -> u32 {
mod tests { mod tests {
use super::*; use super::*;
/// Values of libhdf5's `H5_checksum_fletcher32` (h5py 3.x's bundled
/// libhdf5, called through ctypes). The first three are sums that are
/// multiples of 65535, where `% 65535` gave 0 instead of 0xffff.
#[test]
fn fletcher32_matches_libhdf5() {
assert_eq!(fletcher32(&[0x00, 0x01, 0xff, 0xfe]), 0x0001_ffff);
assert_eq!(fletcher32(&[0xff; 720]), 0xffff_ffff);
assert_eq!(fletcher32(&[0xff; 721]), 0xff00_ff00);
assert_eq!(fletcher32(&[0xff; 1441]), 0xff00_ff00);
assert_eq!(fletcher32(&[]), 0);
assert_eq!(fletcher32(&[7]), 0x0700_0700);
}
#[test] #[test]
fn empty_input() { fn empty_input() {
// Empty input should return the initial state after no mixing // Empty input should return the initial state after no mixing
+13 -53
View File
@@ -1678,56 +1678,6 @@ fn shuffle_compress_general(data: &[u8], n: usize, element_size: usize, result:
} }
} }
/// Compute HDF5 Fletcher32 checksum over data.
/// HDF5 uses a modified Fletcher32 that operates on 16-bit words.
///
/// Optimized with wider accumulators: processes blocks of 360 words before
/// taking the modulo, reducing the number of expensive modulo operations.
/// (360 is the maximum block size that avoids u32 overflow for sum2.)
fn fletcher32_compute(data: &[u8]) -> u32 {
let mut sum1: u32 = 0;
let mut sum2: u32 = 0;
// Process in blocks of 360 16-bit words (720 bytes) to delay modulo.
// Max sum1 before mod: 360 * 65535 = 23_592_600 < u32::MAX
// Max sum2 before mod: 360 * 23_592_600 ~ 8.5B > u32::MAX, but actual
// sum2 accumulates incrementally, so worst case is 360*360*65535/2 which
// fits in u64. We use u32 with block size 360 which is safe.
const BLOCK_WORDS: usize = 360;
const BLOCK_BYTES: usize = BLOCK_WORDS * 2;
let mut offset = 0;
let len = data.len();
while offset + BLOCK_BYTES <= len {
let end = offset + BLOCK_BYTES;
let mut i = offset;
while i < end {
let val = ((data[i] as u32) << 8) | (data[i + 1] as u32);
sum1 += val;
sum2 += sum1;
i += 2;
}
sum1 %= 65535;
sum2 %= 65535;
offset = end;
}
// Handle remaining bytes
while offset < len {
let val = if offset + 1 < len {
((data[offset] as u32) << 8) | (data[offset + 1] as u32)
} else {
(data[offset] as u32) << 8
};
sum1 = (sum1 + val) % 65535;
sum2 = (sum2 + sum1) % 65535;
offset += 2;
}
(sum2 << 16) | sum1
}
/// Verify Fletcher32 checksum and strip it from the data. /// Verify Fletcher32 checksum and strip it from the data.
/// The last 4 bytes are the stored checksum. /// The last 4 bytes are the stored checksum.
fn fletcher32_verify(data: &[u8]) -> Result<Vec<u8>, FormatError> { fn fletcher32_verify(data: &[u8]) -> Result<Vec<u8>, FormatError> {
@@ -1749,8 +1699,18 @@ fn fletcher32_payload(data: &[u8]) -> Result<usize, FormatError> {
data[data.len() - 2], data[data.len() - 2],
data[data.len() - 1], data[data.len() - 1],
]); ]);
let computed = fletcher32_compute(payload); let computed = crate::checksum::fletcher32(payload);
if stored != computed { // libhdf5 also accepts the checksum with the bytes of each 16-bit half
// swapped, which is how 1.6.2 and earlier stored it
// (H5Z__filter_fletcher32's `reversed_fletcher`).
let reversed = ((computed & 0x00ff_00ff) << 8) | ((computed >> 8) & 0x00ff_00ff);
// clawhdf5 v2.7.0 and earlier reduced the sums `% 65535`, which gives 0
// where libhdf5's fold gives 0xffff; accept that form too, so that files
// those releases wrote can still be read (and rewritten for libhdf5).
// It differs from `computed` only in a half that is 0xffff.
let half = |h: u32| if h == 0xffff { 0 } else { h };
let legacy = (half(computed >> 16) << 16) | half(computed & 0xffff);
if stored != computed && stored != reversed && stored != legacy {
return Err(FormatError::Fletcher32Mismatch { return Err(FormatError::Fletcher32Mismatch {
expected: stored, expected: stored,
computed, computed,
@@ -1761,7 +1721,7 @@ fn fletcher32_payload(data: &[u8]) -> Result<usize, FormatError> {
/// Append Fletcher32 checksum to data. /// Append Fletcher32 checksum to data.
fn fletcher32_append(data: &[u8]) -> Result<Vec<u8>, FormatError> { fn fletcher32_append(data: &[u8]) -> Result<Vec<u8>, FormatError> {
let checksum = fletcher32_compute(data); let checksum = crate::checksum::fletcher32(data);
let mut result = data.to_vec(); let mut result = data.to_vec();
result.extend_from_slice(&checksum.to_le_bytes()); result.extend_from_slice(&checksum.to_le_bytes());
Ok(result) Ok(result)
+321
View File
@@ -0,0 +1,321 @@
//! Fletcher-32 against libhdf5.
//!
//! libhdf5's `H5_checksum_fletcher32` reduces its sums with the
//! ones'-complement fold `(s & 0xffff) + (s >> 16)`, which leaves 0xffff
//! where `% 65535` leaves 0. Our checksum once used `% 65535`, so on about
//! one chunk in 32768 (a sum that is a non-zero multiple of 65535) libhdf5
//! rejected the chunks we wrote and we rejected the chunks it wrote.
//!
//! - The checksum is compared with libhdf5's own `H5_checksum_fletcher32`,
//! called through ctypes from the library h5py loads, over every one-byte
//! and two-byte input and a large corpus of random and fold-heavy inputs.
//! - Chunks engineered to hit the fold are written by `FileBuilder` and by
//! `FileEditor` and read by h5py, and written by h5py and read by us.
//!
//! Skipped when python3 with h5py is unavailable, unless
//! `CLAWHDF5_REQUIRE_INTEROP=1`.
use std::process::Command;
use clawhdf5::{File, FileBuilder, FileEditor};
use clawhdf5_format::checksum::fletcher32;
fn python() -> String {
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
}
fn have_h5py() -> bool {
let ok = Command::new(python())
.args(["-c", "import h5py, numpy"])
.output()
.is_ok_and(|o| o.status.success());
if !ok {
assert!(
std::env::var("CLAWHDF5_REQUIRE_INTEROP").as_deref() != Ok("1"),
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
);
eprintln!("SKIP: python3 with h5py not available");
}
ok
}
fn run_python(script: &str, args: &[&str]) -> String {
let out = Command::new(python())
.arg("-c")
.arg(script)
.args(args)
.output()
.expect("failed to run python");
assert!(
out.status.success(),
"python failed:\nSTDOUT: {}\nSTDERR: {}",
String::from_utf8_lossy(&out.stdout),
String::from_utf8_lossy(&out.stderr)
);
String::from_utf8_lossy(&out.stdout).trim().to_string()
}
fn tmp(name: &str) -> std::path::PathBuf {
let dir = std::env::temp_dir().join(format!("clawhdf5_fletcher32_{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
dir.join(name)
}
/// The checksum our code computed before it was fixed: each sum reduced
/// `% 65535`. Only used to show that the test data hits the disagreement.
fn fletcher32_mod(data: &[u8]) -> u32 {
let (mut s1, mut s2) = (0u64, 0u64);
for w in data.chunks(2) {
let v = (u64::from(w[0]) << 8) | w.get(1).map_or(0, |&b| u64::from(b));
s1 = (s1 + v) % 65535;
s2 = (s2 + s1) % 65535;
}
((s2 as u32) << 16) | s1 as u32
}
/// Splitmix64, so the data is the same on every run.
struct Rng(u64);
impl Rng {
fn next(&mut self) -> u64 {
self.0 = self.0.wrapping_add(0x9e37_79b9_7f4a_7c15);
let mut z = self.0;
z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
z ^ (z >> 31)
}
}
/// Writes every case of the file `argv[1]` (u32 LE length + bytes) back to
/// `argv[2]` as libhdf5's checksum of each, u32 LE.
const LIBHDF5_CHECKSUMS: &str = r#"
import ctypes, glob, os, struct, sys
import h5py
cands = glob.glob(os.path.join(os.path.dirname(h5py.__file__), os.pardir, 'h5py.libs', 'libhdf5-*.so*'))
cands += glob.glob(os.path.join(os.path.dirname(h5py.__file__), '.dylibs', 'libhdf5*.dylib'))
if cands:
lib = ctypes.CDLL(cands[0])
else:
# A system h5py links the system libhdf5, already loaded.
import h5py.h5
lib = ctypes.CDLL(h5py.h5.__file__)
f = lib.H5_checksum_fletcher32
f.restype = ctypes.c_uint32
f.argtypes = [ctypes.c_char_p, ctypes.c_size_t]
data = open(sys.argv[1], 'rb').read()
out = bytearray()
i = 0
while i < len(data):
(n,) = struct.unpack_from('<I', data, i)
i += 4
b = data[i:i + n]
i += n
out += struct.pack('<I', f(b, n))
open(sys.argv[2], 'wb').write(out)
"#;
#[test]
fn checksum_matches_libhdf5() {
if !have_h5py() {
return;
}
let mut cases: Vec<Vec<u8>> = Vec::new();
// Every one-byte input (the odd-length path alone) and every one-word
// input (65535 = 0xffff is the smallest fold).
cases.extend((0..=255u8).map(|b| vec![b]));
cases.extend((0..=u16::MAX).map(|w| w.to_be_bytes().to_vec()));
let mut rng = Rng(0x5eed_f1e7);
// Words drawn from values that make multiples of 65535 frequent, at
// lengths around the 360-word block boundaries, odd and even.
const FOLDY: [u16; 6] = [0, 1, 0xfffe, 0xffff, 0x8000, 0x7fff];
for _ in 0..40_000 {
let len = match rng.next() % 4 {
0 => (rng.next() % 16) as usize,
1 => 718 + (rng.next() % 6) as usize,
2 => 1438 + (rng.next() % 6) as usize,
_ => (rng.next() % 3000) as usize,
};
let foldy = rng.next().is_multiple_of(2);
let mut v = Vec::with_capacity(len + 1);
while v.len() < len {
let w = if foldy {
FOLDY[(rng.next() % 6) as usize]
} else {
rng.next() as u16
};
v.extend_from_slice(&w.to_be_bytes());
}
v.truncate(len);
cases.push(v);
}
// Long runs of 0xff: sums are multiples of 65535 at every block.
for len in [720, 721, 1440, 1441, 7200, 65536, 65537] {
cases.push(vec![0xff; len]);
}
let mut blob = Vec::new();
for c in &cases {
blob.extend_from_slice(&(c.len() as u32).to_le_bytes());
blob.extend_from_slice(c);
}
let input = tmp("cases.bin");
let output = tmp("sums.bin");
std::fs::write(&input, &blob).unwrap();
run_python(
LIBHDF5_CHECKSUMS,
&[input.to_str().unwrap(), output.to_str().unwrap()],
);
let sums = std::fs::read(&output).unwrap();
assert_eq!(sums.len(), cases.len() * 4);
let mut folds = 0;
for (c, s) in cases.iter().zip(sums.as_chunks::<4>().0) {
let want = u32::from_le_bytes(*s);
assert_eq!(
fletcher32(c),
want,
"checksum of {} bytes {:02x?}...",
c.len(),
&c[..c.len().min(16)]
);
if fletcher32_mod(c) != want {
folds += 1;
}
}
// The corpus must exercise the case `% 65535` got wrong.
assert!(folds > 500, "only {folds} fold cases");
}
const CHUNK: usize = 8;
/// `n` chunks of `CHUNK` bytes, each one a chunk on which the old
/// `% 65535` checksum and libhdf5's differ (sum1, sum2 or both a non-zero
/// multiple of 65535), with an ordinary chunk between them.
fn fold_chunks(n: usize) -> Vec<u8> {
let mut rng = Rng(42);
let mut out = Vec::new();
let mut found = 0;
while found < n {
// Build a chunk whose sum1 is a multiple of 65535 half the time,
// otherwise search at random for a sum2 fold.
let mut c: Vec<u8> = (0..CHUNK).map(|_| rng.next() as u8).collect();
if found % 2 == 0 {
let words: u64 = c[..CHUNK - 2]
.chunks(2)
.map(|w| (u64::from(w[0]) << 8) | u64::from(w[1]))
.sum();
let last = ((65535 - words % 65535) % 65535) as u16;
c[CHUNK - 2..].copy_from_slice(&last.to_be_bytes());
}
if fletcher32(&c) != fletcher32_mod(&c) {
out.extend_from_slice(&c);
out.extend((0..CHUNK).map(|i| i as u8 + 1));
found += 1;
}
}
out
}
#[test]
fn h5py_reads_fold_case_chunks_we_write() {
if !have_h5py() {
return;
}
let data = fold_chunks(32);
// FileBuilder.
let built = tmp("built.h5");
let mut b = FileBuilder::new();
b.create_dataset("d")
.with_u8_data(&data)
.with_chunks(&[CHUNK as u64])
.with_fletcher32();
b.write(&built).unwrap();
// FileEditor, into a dataset h5py created.
let edited = tmp("edited.h5");
run_python(
"import sys, h5py, numpy as np\n\
with h5py.File(sys.argv[1], 'w') as f:\n\
\x20 f.create_dataset('d', data=np.zeros(int(sys.argv[2]), 'u1'), chunks=(8,), fletcher32=True)",
&[edited.to_str().unwrap(), &data.len().to_string()],
);
FileEditor::open(&edited)
.unwrap()
.write_all("d", &data)
.unwrap();
for path in [&built, &edited] {
let got = run_python(
"import sys, h5py\n\
with h5py.File(sys.argv[1], 'r') as f:\n\
\x20 assert f['d'].fletcher32\n\
\x20 print(f['d'][:].tobytes().hex())",
&[path.to_str().unwrap()],
);
assert_eq!(got, hex(&data), "{}", path.display());
}
}
#[test]
fn we_read_fold_case_chunks_h5py_writes() {
if !have_h5py() {
return;
}
let data = fold_chunks(32);
let path = tmp("h5py.h5");
run_python(
"import sys, h5py, numpy as np\n\
with h5py.File(sys.argv[1], 'w') as f:\n\
\x20 f.create_dataset('d', data=np.frombuffer(bytes.fromhex(sys.argv[2]), 'u1'), chunks=(8,), fletcher32=True)",
&[path.to_str().unwrap(), &hex(&data)],
);
let file = File::open(&path).unwrap();
let ds = file.dataset("d").unwrap();
assert_eq!(
ds.read_selection(&clawhdf5_format::selection::Selection::All)
.unwrap(),
data
);
}
/// A checksum stored with the bytes of each 16-bit half swapped, as
/// libhdf5 1.6.2 and earlier wrote it, is accepted as libhdf5 accepts it;
/// so is the `% 65535` form clawhdf5 v2.7.0 and earlier wrote, so that
/// their files stay readable.
#[test]
fn legacy_checksums_are_accepted() {
use clawhdf5_format::filter_pipeline::{FILTER_FLETCHER32, FilterDescription, FilterPipeline};
let payload = [1u8, 2, 3, 4, 5];
let sum = fletcher32(&payload);
let swapped = ((sum & 0x00ff_00ff) << 8) | ((sum >> 8) & 0x00ff_00ff);
assert_ne!(sum, swapped);
let pipeline = FilterPipeline {
version: 2,
filters: vec![FilterDescription {
filter_id: FILTER_FLETCHER32,
name: None,
client_data: vec![],
flags: 0,
}],
};
for stored in [sum, swapped] {
let mut chunk = payload.to_vec();
chunk.extend_from_slice(&stored.to_le_bytes());
let out = clawhdf5_format::filters::decompress_chunk(&chunk, &pipeline, payload.len(), 1)
.unwrap();
assert_eq!(out, payload);
}
// Our old checksum of a fold-case chunk.
let fold = fold_chunks(1);
let fold = &fold[..CHUNK];
let old = fletcher32_mod(fold);
assert_ne!(old, fletcher32(fold));
let mut chunk = fold.to_vec();
chunk.extend_from_slice(&old.to_le_bytes());
let out = clawhdf5_format::filters::decompress_chunk(&chunk, &pipeline, CHUNK, 1).unwrap();
assert_eq!(out, fold);
let mut chunk = payload.to_vec();
chunk.extend_from_slice(&(sum ^ 1).to_le_bytes());
assert!(
clawhdf5_format::filters::decompress_chunk(&chunk, &pipeline, payload.len(), 1).is_err()
);
}
fn hex(b: &[u8]) -> String {
b.iter().map(|x| format!("{x:02x}")).collect()
}
+35
View File
@@ -7,6 +7,41 @@ deleting it.
--- ---
## Fletcher-32 checksums disagreed with libhdf5 on about 1 chunk in 32768
**Status:** fixed 2026-09-26, after v2.7.0. **Every release (v2.1.0 to
v2.7.0) is affected**, in both directions.
Our Fletcher-32 reduced its two running sums with `% 65535`; libhdf5's
`H5_checksum_fletcher32` (H5checksum.c) folds them with
`(s & 0xffff) + (s >> 16)`. Both are arithmetic mod 65535, but where a sum
is a non-zero multiple of 65535 the fold leaves 0xffff and the modulo 0, so
the checksums differ — for random data about one chunk in 32768 (each of
the two sums hits it with probability about 1/65535). Found by the review
of the editor work: a random-edit fuzzer with gzip + Fletcher-32 hit it on
13 of about 100 seeds.
- Chunks we wrote (`FileBuilder`/`FileWriter` `with_fletcher32`, and the
unreleased `FileEditor`) with such a sum are refused by h5py and libhdf5:
"filter returned failure during read". h5py writing `[1, 0xfffe]` as
big-endian `u2` stores checksum `0x0001ffff`; we computed `0x00010000`.
- Chunks libhdf5 wrote with such a sum were refused by every reader here
with `Fletcher32Mismatch`; the data itself was never wrong.
**Fix:** `clawhdf5_format::checksum::fletcher32`, a port of
`H5_checksum_fletcher32`, used by the filter for writing and verifying. It
also accepts a checksum whose 16-bit halves are byte-swapped, as libhdf5
does for files from 1.6.2 and earlier, and the `% 65535` form clawhdf5
v2.7.0 and earlier wrote (the two differ only in a half that is 0xffff).
**Test:**
`crates/clawhdf5/tests/fletcher32_interop.rs` (libhdf5's own function
through ctypes on every 1- and 2-byte input plus 40 000 random and
fold-heavy inputs; h5py reads fold-case chunks from `FileBuilder` and
`FileEditor`; we read h5py's). **Existing data:** a Fletcher-32 dataset
written by v2.7.0 or earlier may hold chunks libhdf5 cannot read; a fixed
build reads them. Rewrite such datasets with a fixed build (read, then
write them again) before handing the file to libhdf5 or h5py.
## LZF/Blosc chunks written with a stale filter mask ## LZF/Blosc chunks written with a stale filter mask
**Status:** fixed 2026-09-26, before any release (the LZF and Blosc writers **Status:** fixed 2026-09-26, before any release (the LZF and Blosc writers