fix(format): verify Fixed and Extensible Array checksums

Every structure in both chunk indexes — header, index block, super
block, data block and each data block page — carries a Jenkins lookup3
checksum, and all of them were parsed past and ignored.

What that costs is not a warning but correct data. Flip one low bit of a
chunk address and the index still has the right shape, the address still
lands inside the file, and the reader returns whatever bytes now sit
there as that chunk's contents. Nothing else in the parse can tell.

Verified in both directions. The checksums accept files written by
HDF5 2.0 from 100 to 200 000 chunks — dense, sparse, gzip-filtered and
paged — which also confirms the block layouts byte for byte, since a
wrong offset would fail every file. And an interop test corrupts an
address to check the read fails instead of returning data: removing the
verification makes that test fail with "corruption produced data instead
of an error", which is what it is there to prove.

The first version of that test passed with verification disabled — it
corrupted a byte a structural check already rejected, so it proved
nothing. Worth recording, since a test that passes for the wrong reason
looks exactly like coverage.

Hand-built fixtures now stamp real checksums, as HDF5 writers do, and
the Extensible Array ones no longer describe the superseded layout.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-20 17:41:41 -07:00
co-authored by Claude Opus 5
parent 0bc7a293ae
commit b41272487a
4 changed files with 349 additions and 61 deletions
+13
View File
@@ -2,6 +2,19 @@
## Unreleased
### Integrity
- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
their checksums** (the `checksum` feature, on by default). Every structure
in both — header, index block, super block, data block and each data block
page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
The consequence of skipping it is not a missing warning but wrong data: a
single flipped bit in a chunk address still parses, still points inside the
file, and the reader hands back whatever bytes now sit there as the chunk's
contents. Verified in both directions — the checksums accept files written
by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
and an interop test corrupts an address to confirm the read now fails
instead of returning data (it does return data when the check is removed).
### Performance
- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
sign-extend each half to `i16`, then `madd_epi16`; scalar fallback