format: read a contiguous selection's runs merged across small gaps

gather_storage merged only runs that touch, so a strided selection of a
contiguous dataset over a Storage became one range (and one owned Vec) per
element: a stride-2 read of 32M f32 through File::open_storage made
16,777,232 read_at calls, took 2.0 s and peaked at 2.09 GB.

The selection is now walked twice. The first walk checks the runs and plans
spans: runs in increasing order at most 4 KiB apart (GATHER_GAP_BYTES) are
read as one span up to 8 MiB (GATHER_SPAN_BYTES; a longer run is split), so
nothing is stored per run. The spans are fetched in RAW_BATCH_BYTES batches
while the second walk copies each run out of its span. Same checks and
errors as before.

The same read is now 32 reads and 0.31 s (File::open: 0.08 s).
contiguous_read_interop: every h5py-checked selection is also read through
File::open_storage and must give libhdf5's bytes; a new test bounds the
range reads of strided, blocked, column and point selections (stride 2: at
most 1 data read; 563,200 before).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:34:31 -05:00
co-authored by Claude Opus 5.5
parent 7d629f49e3
commit b086dc3c2b
4 changed files with 294 additions and 81 deletions
+4 -1
View File
@@ -61,7 +61,10 @@
`chunks x extent` bytes (`tests/raw_fetch_bounds.rs`). Chunks the
file's chunk cache already holds are not fetched. A selection fetches
only the chunks its bounding box overlaps; a contiguous selection only
its runs (adjacent ones merged). A global-heap collection is read once
its runs, merged into reads of up to 8 MiB across gaps of up to 4 KiB
(a stride-2 selection of 32M `f32` is 32 reads and 0.3 s over a
`CountingStorage`, where one read per element was 16.8M reads, 2.0 s
and 2.1 GB peak). A global-heap collection is read once
per resolver and kept (within the resolver's 32 MiB budget).
- Each extent's bounds error is the one the slice readers gave, reported
when the read reaches that extent, so a damaged file fails with the