quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(format): write layout v4 chunk dimensions in the fewest bytes

Browse Source 
libhdf5 encodes a version-4 layout's chunk dimensions in (log2(max) +
8) / 8 bytes, and HDF5 2.0.0 (h5py 3.16) refuses any other width:
"stored chunk dimension encoding length does not match value calculated
from chunk dimensions". The writer rounded 3 bytes up to 4, so h5py
could not open a dataset we wrote with a chunk dimension from 65 536 to
16 777 215, for every chunk index (single chunk, fixed and extensible
array, v2 B-tree). The three encoders now share push_v4_chunk_dims,
which writes the exact width.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 01:27:22 -05:00
co-authored by Claude Opus 5.5
parent f713847e65
commit afae86f3ea
4 changed files with 57 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+7
View File
@@ -197,6 +197,13 @@
takes `--f32`; it had kept printing "f32" after the default changed.
### Interop
- **h5py could not open chunked datasets we wrote with a chunk dimension
from 65 536 to 16 777 215.** A version-4 layout must store its chunk
dimensions in the fewest bytes that hold the largest (3 for 70 000);
the writer rounded 3 up to 4, and HDF5 2.0.0 (h5py 3.16) refuses that
("stored chunk dimension encoding length does not match value calculated
from chunk dimensions"). Newer libhdf5 and clawhdf5 read those files; new
files use the exact width. Test: `we_write_chunk_dimensions_in_the_fewest_bytes`.
- **Conformance sweep in the repo** (`conformance/`, report in
`CONFORMANCE.md`). `conformance/run.sh` fetches eight public HDF5 corpora
pinned by commit (libhdf5's test files, the HDF Group's CVE reproducers,