fix(format): verify Fixed and Extensible Array checksums
Every structure in both chunk indexes — header, index block, super block, data block and each data block page — carries a Jenkins lookup3 checksum, and all of them were parsed past and ignored. What that costs is not a warning but correct data. Flip one low bit of a chunk address and the index still has the right shape, the address still lands inside the file, and the reader returns whatever bytes now sit there as that chunk's contents. Nothing else in the parse can tell. Verified in both directions. The checksums accept files written by HDF5 2.0 from 100 to 200 000 chunks — dense, sparse, gzip-filtered and paged — which also confirms the block layouts byte for byte, since a wrong offset would fail every file. And an interop test corrupts an address to check the read fails instead of returning data: removing the verification makes that test fail with "corruption produced data instead of an error", which is what it is there to prove. The first version of that test passed with verification disabled — it corrupted a byte a structural check already rejected, so it proved nothing. Worth recording, since a test that passes for the wrong reason looks exactly like coverage. Hand-built fixtures now stamp real checksums, as HDF5 writers do, and the Extensible Array ones no longer describe the superseded layout. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,19 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Integrity
|
||||
- `clawhdf5-format`: **Fixed and Extensible Array chunk indexes now verify
|
||||
their checksums** (the `checksum` feature, on by default). Every structure
|
||||
in both — header, index block, super block, data block and each data block
|
||||
page — carries a Jenkins lookup3 checksum that was parsed past and ignored.
|
||||
The consequence of skipping it is not a missing warning but wrong data: a
|
||||
single flipped bit in a chunk address still parses, still points inside the
|
||||
file, and the reader hands back whatever bytes now sit there as the chunk's
|
||||
contents. Verified in both directions — the checksums accept files written
|
||||
by HDF5 2.0 at 100 to 200 000 chunks, dense, sparse, filtered and paged,
|
||||
and an interop test corrupts an address to confirm the read now fails
|
||||
instead of returning data (it does return data when the check is removed).
|
||||
|
||||
### Performance
|
||||
- `clawhdf5-accel`: **`dot_i8`, a runtime-dispatched int8 dot product** (AVX2:
|
||||
sign-extend each half to `i16`, then `madd_epi16`; scalar fallback
|
||||
|
||||
Reference in New Issue
Block a user