feat(format): choose chunk dimensions automatically for large datasets

Requesting a filter without chunk dimensions made the whole dataset a single chunk. Any read, even one row, then decompresses everything, and a large dataset cannot be decoded in parallel — which also made the new partial reads pointless for such files. auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour) and splits larger ones by halving the dimensions in turn, so chunks keep roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's approach. An empty (unlimited, unwritten) dimension is treated as 1024. The writer passes the element size through resolve_chunk_dims_for; the old resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins. Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks between 128 KiB and 1 MiB, and a small dataset still has one chunk. Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-09-19 14:21:04 -07:00
parent b36c6ec2af
commit 05c665a898
4 changed files with 152 additions and 6 deletions
@@ -1221,8 +1221,10 @@ impl FileWriter {
                    precompressed: None,
                });
            } else if is_chunked[i] {
-                let chunk_dims = d.chunk_options.resolve_chunk_dims(&d.ds.dimensions);
                let elem_size = d.dt.type_size() as usize;
+                let chunk_dims = d
+                    .chunk_options
+                    .resolve_chunk_dims_for(&d.ds.dimensions, elem_size);
                // Compress once in Pass 1; cache the result so Pass 2 can skip
                // re-compression and just rebuild the index with real addresses.
                let pre = precompress_chunks(