format: raw data, VDS and VL data over Storage

Every raw-data path has a generic *_in core, with the &[u8] functions as
thin wrappers: data_read (read_raw_data*, read_raw_data_selection,
read_chunked_native), chunked_read (the v1 B-tree chunk index, list_chunks,
the full, cached, sweep and indexed reads), parallel_read, partial_read,
fill_value (read_full_with_fill, apply_to_unallocated_chunks; and
dataset_fill_value_from_storage is now generic), vds (the virtual file
through Storage, external sources still through the resolver),
vl_data (VlResolver<'a, S = [u8]>, read_vl_strings_in, read_vl_bytes_in),
AttributeMessage::read_vl_strings_in and provenance::verify_dataset_in.

With the whole file in memory nothing changes: chunks and contiguous data
are sliced from it as before. Otherwise a chunked read lists its chunks,
fetches their stored bytes with one Storage::read_ranges call per 64 MiB
batch (chunks the cache already holds are not fetched), then decodes as
today; a selection fetches only the chunks it overlaps, and a contiguous
selection only its runs. Each extent's bounds error is the one the slice
code gave, reported when that extent is reached, so errors keep their
order.

Tests: the equivalence harness now reads every dataset's values (whole,
fill-aware, cached, indexed, three selections, VDS, VL strings and
sequences) through the read_at-only storage and requires the slice
results (all 653 corpus files agree); a misbehaving storage (a failing
Nth read, short reads) only ever yields errors or the right values; and
chunked reads are checked to use one read_ranges call.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

crates/clawhdf5-format/src/storage.rs

@@ -336,6 +336,159 @@ pub fn read_upto<S: Storage + ?Sized>(
     Ok(bytes)
 }
 
+/// Most stored bytes fetched by one [`Storage::read_ranges`] call when a
+/// read gathers many extents (a chunked dataset's chunks): a larger read is
+/// fetched and decoded batch by batch, so a remote backend never holds more
+/// than this much undecoded data per read.
+pub(crate) const RAW_BATCH_BYTES: usize = 64 << 20;
+
+/// The stored bytes of a list of extents (chunks, contiguous runs), fetched
+/// together: [`Storage::read_ranges`] is called once for all of them, so a
+/// remote backend can coalesce and parallelise the requests.
+///
+/// With the whole file in memory nothing is fetched: [`Self::get`] slices
+/// it, as the slice readers did. Either way an extent that does not lie in
+/// the file is the error the slice readers gave for it
+/// ([`FormatError::UnexpectedEof`] with its end and the file length, or
+/// [`FormatError::Overflow`] for an address past this platform's `usize`),
+/// reported when that extent is asked for — so a read reports the first
+/// failing extent in its own order, whatever fails after it.
+pub(crate) enum ExtentBytes<'a> {
+    /// The whole file.
+    Contiguous(&'a [u8]),
+    /// Each extent's bytes, or its bounds error.
+    Fetched(Vec<Extent<'a>>),
+}
+
+/// One extent of [`ExtentBytes::Fetched`].
+pub(crate) enum Extent<'a> {
+    /// Its bytes.
+    Bytes(Cow<'a, [u8]>),
+    /// In the file, but not fetched (the caller did not want its bytes).
+    NotFetched,
+    /// The error reading it gives.
+    Err(FormatError),
+}
+
+impl<'a> ExtentBytes<'a> {
+    /// Fetch `extents` (`(address, length, wanted)`): the bytes of those
+    /// `wanted`, and the bounds check of all of them.
+    pub(crate) fn fetch<S: Storage + ?Sized>(
+        file: &'a S,
+        extents: &[(u64, usize, bool)],
+    ) -> Result<Self, FormatError> {
+        if let Some(all) = file.as_contiguous() {
+            return Ok(ExtentBytes::Contiguous(all));
+        }
+        let file_len = len_usize(file);
+        let mut ranges = Vec::new();
+        let mut out = Vec::with_capacity(extents.len());
+        // Positions in `out` of the extents being read, in `ranges` order.
+        let mut slots = Vec::new();
+        for &(addr, len, wanted) in extents {
+            let checked =
+                crate::addr::to_usize(addr).and_then(|start| match start.checked_add(len) {
+                    Some(end) if end <= file_len => Ok(()),
+                    _ => Err(FormatError::UnexpectedEof {
+                        expected: start.saturating_add(len),
+                        available: file_len,
+                    }),
+                });
+            match checked {
+                Ok(()) if wanted => {
+                    slots.push(out.len());
+                    ranges.push(addr..addr + len as u64);
+                    out.push(Extent::NotFetched);
+                }
+                Ok(()) => out.push(Extent::NotFetched),
+                Err(e) => out.push(Extent::Err(e)),
+            }
+        }
+        if !ranges.is_empty() {
+            let got = file.read_ranges(&ranges)?;
+            if got.len() != ranges.len() {
+                return Err(FormatError::Storage(
+                    "read_ranges returned the wrong number of ranges".into(),
+                ));
+            }
+            for ((slot, bytes), r) in slots.into_iter().zip(got).zip(&ranges) {
+                if (bytes.len() as u64) < r.end - r.start {
+                    return Err(short_read());
+                }
+                out[slot] = Extent::Bytes(bytes);
+            }
+        }
+        Ok(ExtentBytes::Fetched(out))
+    }
+
+    /// Whether extent `i` (at `addr`, `len` bytes, as passed to
+    /// [`Self::fetch`]) lies in the file: its bounds error if not.
+    pub(crate) fn check(&self, i: usize, addr: u64, len: usize) -> Result<(), FormatError> {
+        match self {
+            ExtentBytes::Contiguous(_) => self.get(i, addr, len).map(|_| ()),
+            ExtentBytes::Fetched(v) => match v.get(i) {
+                Some(Extent::Err(e)) => Err(e.clone()),
+                Some(_) => Ok(()),
+                None => Err(not_fetched()),
+            },
+        }
+    }
+
+    /// Extent `i`'s bytes (at `addr`, `len` bytes, as passed to
+    /// [`Self::fetch`]).
+    pub(crate) fn get(&self, i: usize, addr: u64, len: usize) -> Result<&[u8], FormatError> {
+        match self {
+            ExtentBytes::Contiguous(all) => {
+                let start = crate::addr::to_usize(addr)?;
+                start
+                    .checked_add(len)
+                    .and_then(|end| all.get(start..end))
+                    .ok_or(FormatError::UnexpectedEof {
+                        expected: start.saturating_add(len),
+                        available: <[u8]>::len(all),
+                    })
+            }
+            ExtentBytes::Fetched(v) => match v.get(i) {
+                Some(Extent::Bytes(b)) => Ok(b),
+                Some(Extent::Err(e)) => Err(e.clone()),
+                _ => Err(not_fetched()),
+            },
+        }
+    }
+}
+
+#[cold]
+fn not_fetched() -> FormatError {
+    FormatError::Storage("an extent that was not fetched was asked for".into())
+}
+
+/// Split `n` extents, whose sizes `size(i)` gives, into consecutive batches
+/// of at most [`RAW_BATCH_BYTES`] (at least one extent each): the ranges of
+/// `0..n` to fetch together. With the whole file in memory (`contiguous`)
+/// there is nothing to fetch, and one batch.
+pub(crate) fn raw_batches(
+    n: usize,
+    contiguous: bool,
+    size: impl Fn(usize) -> usize,
+) -> Vec<Range<usize>> {
+    if contiguous || n == 0 {
+        return core::iter::once(0..n).collect();
+    }
+    let mut out = Vec::new();
+    let (mut start, mut bytes) = (0, 0usize);
+    for i in 0..n {
+        let s = size(i);
+        if i > start && bytes.saturating_add(s) > RAW_BATCH_BYTES {
+            out.push(start..i);
+            start = i;
+            bytes = 0;
+        }
+        bytes = bytes.saturating_add(s);
+    }
+    out.push(start..n);
+    out
+}
+
 /// Borrow the whole file for a code path that has not been converted to
 /// [`Storage`] yet. On a backend without a contiguous view this is the
 /// clean [`FormatError::ContiguousStorageRequired`] error, never a guess.