format: raw data, VDS and VL data over Storage

Every raw-data path has a generic *_in core, with the &[u8] functions as
thin wrappers: data_read (read_raw_data*, read_raw_data_selection,
read_chunked_native), chunked_read (the v1 B-tree chunk index, list_chunks,
the full, cached, sweep and indexed reads), parallel_read, partial_read,
fill_value (read_full_with_fill, apply_to_unallocated_chunks; and
dataset_fill_value_from_storage is now generic), vds (the virtual file
through Storage, external sources still through the resolver),
vl_data (VlResolver<'a, S = [u8]>, read_vl_strings_in, read_vl_bytes_in),
AttributeMessage::read_vl_strings_in and provenance::verify_dataset_in.

With the whole file in memory nothing changes: chunks and contiguous data
are sliced from it as before. Otherwise a chunked read lists its chunks,
fetches their stored bytes with one Storage::read_ranges call per 64 MiB
batch (chunks the cache already holds are not fetched), then decodes as
today; a selection fetches only the chunks it overlaps, and a contiguous
selection only its runs. Each extent's bounds error is the one the slice
code gave, reported when that extent is reached, so errors keep their
order.

Tests: the equivalence harness now reads every dataset's values (whole,
fill-aware, cached, indexed, three selections, VDS, VL strings and
sequences) through the read_at-only storage and requires the slice
results (all 653 corpus files agree); a misbehaving storage (a failing
Nth read, short reads) only ever yields errors or the right values; and
chunked reads are checked to use one read_ranges call.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

crates/clawhdf5-format/src/fill_value.rs

@@ -13,7 +13,7 @@
 use alloc::{format, vec, vec::Vec};
 
 use crate::addr::to_usize;
-use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks};
+use crate::chunked_read::{alloc_output, checked_byte_len, list_chunks_in};
 use crate::data_layout::DataLayout;
 use crate::dataspace::Dataspace;
 use crate::error::FormatError;
@@ -122,10 +122,11 @@ pub fn dataset_fill_value_in(
 }
 
 /// [`dataset_fill_value_in`] with the file behind any
-/// [`Storage`](crate::storage::Storage). (The trait is not imported here:
-/// its `len` would shadow the slice method in this module.)
-pub fn dataset_fill_value_from_storage(
-    file: &dyn crate::storage::Storage,
+/// [`Storage`](crate::storage::Storage) (a `&dyn Storage` too). (The trait
+/// is not imported here: its `len` would shadow the slice method in this
+/// module.)
+pub fn dataset_fill_value_from_storage<S: crate::storage::Storage + ?Sized>(
+    file: &S,
     messages: &[HeaderMessage],
     offset_size: u8,
     length_size: u8,
@@ -212,6 +213,30 @@ pub fn read_full_with_fill<E: From<FormatError>>(
     offset_size: u8,
     length_size: u8,
     read: impl FnOnce() -> Result<Vec<u8>, E>,
+) -> Result<Vec<u8>, E> {
+    read_full_with_fill_in(
+        messages,
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        offset_size,
+        length_size,
+        read,
+    )
+}
+
+/// [`read_full_with_fill`] over any [`Storage`](crate::storage::Storage).
+#[allow(clippy::too_many_arguments)]
+pub fn read_full_with_fill_in<E: From<FormatError>, S: crate::storage::Storage + ?Sized>(
+    messages: &[HeaderMessage],
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    offset_size: u8,
+    length_size: u8,
+    read: impl FnOnce() -> Result<Vec<u8>, E>,
 ) -> Result<Vec<u8>, E> {
     // A dataset with external raw data also has no data address in this
     // file. It is NOT unallocated — its values live elsewhere — so it must
@@ -222,12 +247,12 @@ pub fn read_full_with_fill<E: From<FormatError>>(
     {
         return Err(FormatError::ExternalDataFilesUnsupported.into());
     }
-    let fill = dataset_fill_value_in(file_data, messages, offset_size, length_size)?;
+    let fill = dataset_fill_value_from_storage(file_data, messages, offset_size, length_size)?;
     if !has_storage(layout) {
         return Ok(filled_dataset(dataspace, elem_size, fill.as_deref())?);
     }
     let mut output = read()?;
-    apply_to_unallocated_chunks(
+    apply_to_unallocated_chunks_in(
         &mut output,
         file_data,
         layout,
@@ -253,6 +278,30 @@ pub fn apply_to_unallocated_chunks(
     fill: Option<&[u8]>,
     offset_size: u8,
     length_size: u8,
+) -> Result<(), FormatError> {
+    apply_to_unallocated_chunks_in(
+        output,
+        file_data,
+        layout,
+        dataspace,
+        elem_size,
+        fill,
+        offset_size,
+        length_size,
+    )
+}
+
+/// [`apply_to_unallocated_chunks`] over any [`Storage`](crate::storage::Storage).
+#[allow(clippy::too_many_arguments)]
+pub fn apply_to_unallocated_chunks_in<S: crate::storage::Storage + ?Sized>(
+    output: &mut [u8],
+    file_data: &S,
+    layout: &DataLayout,
+    dataspace: &Dataspace,
+    elem_size: usize,
+    fill: Option<&[u8]>,
+    offset_size: u8,
+    length_size: u8,
 ) -> Result<(), FormatError> {
     let Some(fill) = fill.filter(|f| f.len() == elem_size && !is_default(Some(f))) else {
         return Ok(());
@@ -260,7 +309,7 @@ pub fn apply_to_unallocated_chunks(
     if !matches!(layout, DataLayout::Chunked { .. }) || elem_size == 0 {
         return Ok(());
     }
-    let (chunks, chunk_dims) = list_chunks(
+    let (chunks, chunk_dims) = list_chunks_in(
         file_data,
         layout,
         dataspace,