quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(format): write layout v4 chunk dimensions in the fewest bytes

Browse Source 
libhdf5 encodes a version-4 layout's chunk dimensions in (log2(max) +
8) / 8 bytes, and HDF5 2.0.0 (h5py 3.16) refuses any other width:
"stored chunk dimension encoding length does not match value calculated
from chunk dimensions". The writer rounded 3 bytes up to 4, so h5py
could not open a dataset we wrote with a chunk dimension from 65 536 to
16 777 215, for every chunk index (single chunk, fixed and extensible
array, v2 B-tree). The three encoders now share push_v4_chunk_dims,
which writes the exact width.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 01:27:22 -05:00
co-authored by Claude Opus 5.5
parent f713847e65
commit afae86f3ea
4 changed files with 57 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5/tests/chunk_index_interop.rs
+25
View File
@@ -515,6 +515,31 @@ fn we_write_maxshape_larger_than_shape() {
check_we_write(&cases);
}
/// A version-4 layout must encode every chunk dimension in the fewest bytes
/// that hold the largest one, as libhdf5 does: HDF5 2.0.0 (h5py 3.16)
/// refuses a wider encoding ("stored chunk dimension encoding length does
/// not match value calculated from chunk dimensions"). We rounded 3 bytes
/// up to 4, so h5py could not open any dataset we wrote with a chunk
/// dimension from 65 536 to 16 777 215.
#[test]
fn we_write_chunk_dimensions_in_the_fewest_bytes() {
const U: u64 = u64::MAX;
let mut cases = vec![
// Single chunk, Fixed Array, Extensible Array, v2 B-tree.
wcase("single_70000", &[70_000], &[70_000], None),
wcase("fa_70000", &[140_000], &[70_000], None),
wcase("ea_70000", &[140_000], &[70_000], Some(&[U])),
wcase("bt2_70000", &[2, 70_000], &[1, 70_000], Some(&[U, U])),
// 2 bytes and 1 byte still, with the element size (4) the largest.
wcase("fa_300", &[600], &[300], None),
wcase("fa_3", &[6], &[3], None),
];
let mut filtered = wcase("single_70000_deflate", &[70_000], &[70_000], None);
filtered.deflate = true;
cases.push(filtered);
check_we_write(&cases);
}
/// More than one unlimited dimension needs a version-2 B-tree chunk index,
/// as the library uses; an Extensible Array for `(None, None)` made libhdf5
/// refuse the whole file ("already found unlimited dimension").