feat(format): choose chunk dimensions automatically for large datasets
Requesting a filter without chunk dimensions made the whole dataset a single chunk. Any read, even one row, then decompresses everything, and a large dataset cannot be decoded in parallel — which also made the new partial reads pointless for such files. auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour) and splits larger ones by halving the dimensions in turn, so chunks keep roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's approach. An empty (unlimited, unwritten) dimension is treated as 1024. The writer passes the element size through resolve_chunk_dims_for; the old resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins. Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks between 128 KiB and 1 MiB, and a small dataset still has one chunk. Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
b36c6ec2af
commit
05c665a898
@@ -22,6 +22,14 @@
|
||||
type 5 — what `libver='latest'` uses for two or more unlimited dimensions;
|
||||
previously "unsupported chunked layout"). The four copies of the chunk-index
|
||||
dispatch are now one shared function, so every read path gets it.
|
||||
- **Automatic chunk sizes.** Asking for compression (or any filter) without
|
||||
`with_chunks` used to store the whole dataset as one chunk, so any read had
|
||||
to decompress everything and nothing could be decoded in parallel. Datasets up
|
||||
to 1 MiB stay a single chunk, as before; larger ones are split by halving the
|
||||
dimensions in turn until a chunk is at most 1 MiB (the approach h5py takes).
|
||||
**Behaviour change:** large compressed datasets written without explicit
|
||||
chunk dimensions get a different (standard, h5py-readable) layout. Explicit
|
||||
`with_chunks` is unaffected.
|
||||
- **Out-of-range selections are errors.** They used to return data: a hyperslab
|
||||
past an edge came back padded with zeros, and a point whose column was out of
|
||||
range wrapped into the next row and returned that element. Now
|
||||
|
||||
Reference in New Issue
Block a user