fix(format): limit chunks to 4 GiB only under a v1 B-tree index

libhdf5 refuses a chunk of 4 GiB or more only when a version-1 B-tree
indexes it (H5D__chunk_init: "chunk size must be < 4GB with v1 b-tree
index"). HDF5 2.0 writes larger chunks with layout version 5, and h5py
reads them; these were refused. chunk_geometry now takes the layout
version and applies the limit to layout version 3 and earlier only.

The interop test is ignored by default: h5py writes a 4 GiB chunk and
both libraries hold it in memory.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 01:17:36 -05:00
co-authored by Claude Opus 5.5
parent 6a8ee3ec7f
commit a14ccc36bf
5 changed files with 104 additions and 42 deletions
+3 -1
View File
@@ -323,7 +323,9 @@
VAX byte order; libhdf5 ignores it before version 3, and so does this.
- chunked layouts (`FormatError::InvalidChunkDimensions`): a zero chunk
dimension, a chunk rank that does not match the dataspace, a chunk of
4 GiB or more (0x80000000-sized chunks hung the reader), and v1 B-tree
4 GiB or more indexed by a v1 B-tree (layout version 3 or earlier;
0x80000000-sized chunks hung the reader — layout versions 4 and 5 allow
larger chunks, and HDF5 2.0 writes them), and v1 B-tree
chunk keys whose offsets are not multiples of the chunk dimensions,
including the keys that only bound a node
(`chunked_read::collect_chunk_info_checked`).