feat(format): choose chunk dimensions automatically for large datasets

Requesting a filter without chunk dimensions made the whole dataset a single
chunk. Any read, even one row, then decompresses everything, and a large
dataset cannot be decoded in parallel — which also made the new partial reads
pointless for such files.

auto_chunk_dims keeps datasets up to 1 MiB as one chunk (unchanged behaviour)
and splits larger ones by halving the dimensions in turn, so chunks keep
roughly the dataset's proportions, until a chunk is at most 1 MiB — h5py's
approach. An empty (unlimited, unwritten) dimension is treated as 1024. The
writer passes the element size through resolve_chunk_dims_for; the old
resolve_chunk_dims assumes 8-byte elements. Explicit with_chunks always wins.

Interop test: h5py reads an auto-chunked 13 MB deflate dataset, sees chunks
between 128 KiB and 1 MiB, and a small dataset still has one chunk.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
osobh
2026-09-19 14:21:04 -07:00
co-authored by Claude Fable 5.1
parent b36c6ec2af
commit 05c665a898
4 changed files with 152 additions and 6 deletions
+8
View File
@@ -22,6 +22,14 @@
type 5 — what `libver='latest'` uses for two or more unlimited dimensions;
previously "unsupported chunked layout"). The four copies of the chunk-index
dispatch are now one shared function, so every read path gets it.
- **Automatic chunk sizes.** Asking for compression (or any filter) without
`with_chunks` used to store the whole dataset as one chunk, so any read had
to decompress everything and nothing could be decoded in parallel. Datasets up
to 1 MiB stay a single chunk, as before; larger ones are split by halving the
dimensions in turn until a chunk is at most 1 MiB (the approach h5py takes).
**Behaviour change:** large compressed datasets written without explicit
chunk dimensions get a different (standard, h5py-readable) layout. Explicit
`with_chunks` is unaffected.
- **Out-of-range selections are errors.** They used to return data: a hyperslab
past an edge came back padded with zeros, and a point whose column was out of
range wrapped into the next row and returned that element. Now