format: bound and batch every chunk fetch over Storage

Only the full read split its chunk fetches into 64 MiB batches. The
selection path, the indexed read and the parallel_read decoders fetched
every chunk's stored bytes in one read_ranges call, each extent bounded only
by the file length, so a crafted chunk index pointing many chunks at one
large extent made File::open_storage hold chunks x extent bytes (3.3 GB from
a 16.8 MB file) before the first decode error.

- storage::for_each_extent_batch is now the one way raw-data reads fetch
  chunk bytes: batches of at most RAW_BATCH_BYTES (now pub), each decoded
  before the next is fetched. Used by the full, cached, indexed, selection
  and parallel_read paths; the sweep read uses read_extent per chunk.
- ExtentReq carries each chunk's claimed extent (bounds-checked as before,
  same errors) and the prefix actually fetched:
  filters::stored_chunk_limit — the chunk size if unfiltered, else each
  applied filter's worst-case growth (n + n/4 + 4096 per codec; unbounded
  only for an application-registered codec). The in-memory path cuts the
  slice it decodes the same way, so both paths still agree.
- tests/raw_fetch_bounds.rs: a crafted chunked_large.h5 (ten chunks all
  claiming 20 MiB at one padding blob) read through every path over a
  storage that records the largest single fetch; and 160 MiB of legitimate
  unfiltered chunks fetched batch by batch. Before: one 80 MiB fetch
  (selection) and one 160 MiB fetch; after: within the budget.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:30:30 -05:00
co-authored by Claude Opus 5.5
parent f191dc09d5
commit 7d629f49e3
10 changed files with 825 additions and 285 deletions
+12 -3
View File
@@ -47,9 +47,18 @@
`read_vl_strings_in`, `read_vl_bytes_in`),
`AttributeMessage::read_vl_strings_in`, `provenance::verify_dataset_in`.
- A chunked read first lists the chunks it needs, then fetches all their
stored bytes with **one `read_ranges` call** (per 64 MiB of stored
data), so a remote backend can coalesce and parallelise them, then
decodes as before (in parallel with the `parallel` feature). Chunks the
stored bytes with **one `read_ranges` call** per batch of at most 64 MiB
(`storage::RAW_BATCH_BYTES`), so a remote backend can coalesce and
parallelise them, then decodes each batch as before (in parallel with
the `parallel` feature) before fetching the next. Every path that reads
chunks — full, cached, indexed, sweep, selection and the
`parallel_read` decoders — goes through the same batching, and no chunk
fetches more of its stored bytes than its decoded size can need (the
chunk size if unfiltered; else each applied filter's worst-case growth,
generously: `n + n/4 + 4096` per codec, unbounded only for a codec the
application registered). A crafted chunk index that points every chunk
at one huge extent therefore costs a bounded fetch, not
`chunks x extent` bytes (`tests/raw_fetch_bounds.rs`). Chunks the
file's chunk cache already holds are not fetched. A selection fetches
only the chunks its bounding box overlaps; a contiguous selection only
its runs (adjacent ones merged). A global-heap collection is read once