fix(format): write chunk indexes over the max extent, swizzled for EA
The writer indexed chunks by their position in the current shape, the
same mistake the reader had. With a finite maxshape larger than the shape
the Fixed Array was sized for the shape, so libhdf5 looked up chunks past
its end ("addr overflow"); with the unlimited dimension anywhere but first,
e.g. maxshape (20, None), libhdf5 swizzles that dimension to the slowest
position and read our Extensible Array scrambled. Two unlimited dimensions
produced a file libhdf5 refused to open ("already found unlimited
dimension").
Chunks are now placed with the shared chunk_grid linearisation: Fixed
Array slots cover every chunk of the maximum extent (unwritten ones
undefined), Extensible Array indexes are swizzled, Single Chunk is only
used when the maximum extent is one chunk, and a maxshape that is smaller
than the shape, has more than one unlimited dimension, or would need an
absurd Fixed Array is an error instead of a bad file.
build_chunked_data_from_precompressed now returns a Result.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -147,7 +147,6 @@ impl ChunkGrid {
|
||||
|
||||
/// Linear index of the chunk with scaled coordinates `scaled`
|
||||
/// (`offset / chunk_dim` per dimension, in dataset order).
|
||||
#[allow(dead_code)] // used by the writer
|
||||
pub(crate) fn linear_index(&self, scaled: &[u64]) -> u64 {
|
||||
self.order
|
||||
.iter()
|
||||
|
||||
Reference in New Issue
Block a user