fix(format): write chunk indexes over the max extent, swizzled for EA

The writer indexed chunks by their position in the current shape, the
same mistake the reader had. With a finite maxshape larger than the shape
the Fixed Array was sized for the shape, so libhdf5 looked up chunks past
its end ("addr overflow"); with the unlimited dimension anywhere but first,
e.g. maxshape (20, None), libhdf5 swizzles that dimension to the slowest
position and read our Extensible Array scrambled. Two unlimited dimensions
produced a file libhdf5 refused to open ("already found unlimited
dimension").

Chunks are now placed with the shared chunk_grid linearisation: Fixed
Array slots cover every chunk of the maximum extent (unwritten ones
undefined), Extensible Array indexes are swizzled, Single Chunk is only
used when the maximum extent is one chunk, and a maxshape that is smaller
than the shape, has more than one unlimited dimension, or would need an
absurd Fixed Array is an error instead of a bad file.
build_chunked_data_from_precompressed now returns a Result.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 21:12:47 -05:00
co-authored by Claude Opus 5.5
parent 44f5f8b5c5
commit 540fa08907
4 changed files with 243 additions and 16 deletions
-1
View File
@@ -147,7 +147,6 @@ impl ChunkGrid {
/// Linear index of the chunk with scaled coordinates `scaled`
/// (`offset / chunk_dim` per dimension, in dataset order).
#[allow(dead_code)] // used by the writer
pub(crate) fn linear_index(&self, scaled: &[u64]) -> u64 {
self.order
.iter()