fix(format): verify Fixed and Extensible Array checksums

Every structure in both chunk indexes — header, index block, super
block, data block and each data block page — carries a Jenkins lookup3
checksum, and all of them were parsed past and ignored.

What that costs is not a warning but correct data. Flip one low bit of a
chunk address and the index still has the right shape, the address still
lands inside the file, and the reader returns whatever bytes now sit
there as that chunk's contents. Nothing else in the parse can tell.

Verified in both directions. The checksums accept files written by
HDF5 2.0 from 100 to 200 000 chunks — dense, sparse, gzip-filtered and
paged — which also confirms the block layouts byte for byte, since a
wrong offset would fail every file. And an interop test corrupts an
address to check the read fails instead of returning data: removing the
verification makes that test fail with "corruption produced data instead
of an error", which is what it is there to prove.

The first version of that test passed with verification disabled — it
corrupted a byte a structural check already rejected, so it proved
nothing. Worth recording, since a test that passes for the wrong reason
looks exactly like coverage.

Hand-built fixtures now stamp real checksums, as HDF5 writers do, and
the Extensible Array ones no longer describe the superseded layout.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-20 17:41:41 -07:00
co-authored by Claude Opus 5
parent 0bc7a293ae
commit b41272487a
4 changed files with 349 additions and 61 deletions
@@ -1318,3 +1318,72 @@ with h5py.File("{sparse_str}", "w", libver="latest") as f:
.count();
assert_eq!(wrong, 0, "sparse: {wrong} of {n} elements read back wrong");
}
#[test]
fn corrupting_a_chunk_index_is_an_error_not_wrong_data() {
// Every Fixed/Extensible Array structure carries a Jenkins checksum, and
// the reader now verifies it. The point is not the checksum itself but
// what it prevents: a damaged index otherwise yields addresses pointing
// at the wrong bytes, and the caller receives another chunk's data as if
// it were the one asked for.
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
for (name, maxshape) in [("fixed", "None"), ("extensible", "(None,)")] {
let path = dir.path().join(format!("{name}.h5"));
let path_str = path.display().to_string();
let shape_arg = if maxshape == "None" {
String::new()
} else {
format!(", maxshape={maxshape}")
};
run_python(&format!(
r#"
import h5py, numpy as np
with h5py.File("{path_str}", "w", libver="latest") as f:
d = f.create_dataset("x", shape=(400,), chunks=(1,), dtype="i4"{shape_arg})
d[...] = np.arange(400, dtype="i4")
"#
));
let clean = std::fs::read(&path).unwrap();
assert_eq!(
File::open(&path)
.unwrap()
.dataset("x")
.unwrap()
.read_i32()
.unwrap()
.len(),
400,
"{name}: the intact file must read"
);
// Flip a low bit of a chunk address inside a data block. Structurally
// everything still parses — the index still has the right shape and
// the address still lands inside the file — so nothing but the
// checksum can notice. Without it the read succeeds and hands back
// whatever bytes now sit at that address.
let sig: &[u8] = if name == "fixed" { b"FADB" } else { b"EADB" };
let block = clean
.windows(4)
.position(|w| w == sig)
.unwrap_or_else(|| panic!("{name}: no data block in the fixture"));
// Past the prefix (signature, version, client id, header address, and
// for the Extensible Array a block offset), into the first address.
let at = block + 4 + 1 + 1 + 8 + if name == "fixed" { 0 } else { 4 } + 1;
let mut damaged = clean.clone();
damaged[at] ^= 0x10;
let damaged_path = dir.path().join(format!("{name}_damaged.h5"));
std::fs::write(&damaged_path, &damaged).unwrap();
let result = File::open(&damaged_path)
.unwrap()
.dataset("x")
.and_then(|d| d.read_i32());
assert!(
result.is_err(),
"{name}: corruption produced data instead of an error"
);
}
}