Read chunks of 4 GiB or more in every chunk index

ChunkInfo::chunk_size (and ChunkMapping::file_size) are u64: sizes past
u32 were truncated for Single Chunk, Implicit, Fixed and Extensible Array
indexes, and a v2 B-tree index refused them. A selection of a chunked
dataset with a non-default fill value is read over a box of fill values
instead of a full read, an unfiltered chunk of a file that is not in memory
is read row by row, and an intermediate deflate stage no longer reserves the
chunk's whole bound.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

This commit is contained in:

osobh

2026-09-28 23:44:48 -05:00

co-authored by

Claude Opus 5.5

parent 01d2a5dc5d

commit ac7871fff5

16 changed files with 604 additions and 43 deletions

.gitignore

View File

@@ -7,3 +7,6 @@ weights/
 .venv
 __pycache__/
 .pytest_cache/
 # Scratch files the heavy tests generate (huge_chunks_interop)
 crates/*/tests/scratch/