format: raw data, VDS and VL data over Storage
Every raw-data path has a generic *_in core, with the &[u8] functions as thin wrappers: data_read (read_raw_data*, read_raw_data_selection, read_chunked_native), chunked_read (the v1 B-tree chunk index, list_chunks, the full, cached, sweep and indexed reads), parallel_read, partial_read, fill_value (read_full_with_fill, apply_to_unallocated_chunks; and dataset_fill_value_from_storage is now generic), vds (the virtual file through Storage, external sources still through the resolver), vl_data (VlResolver<'a, S = [u8]>, read_vl_strings_in, read_vl_bytes_in), AttributeMessage::read_vl_strings_in and provenance::verify_dataset_in. With the whole file in memory nothing changes: chunks and contiguous data are sliced from it as before. Otherwise a chunked read lists its chunks, fetches their stored bytes with one Storage::read_ranges call per 64 MiB batch (chunks the cache already holds are not fetched), then decodes as today; a selection fetches only the chunks it overlaps, and a contiguous selection only its runs. Each extent's bounds error is the one the slice code gave, reported when that extent is reached, so errors keep their order. Tests: the equivalence harness now reads every dataset's values (whole, fill-aware, cached, indexed, three selections, VDS, VL strings and sequences) through the read_at-only storage and requires the slice results (all 653 corpus files agree); a misbehaving storage (a failing Nth read, short reads) only ever yields errors or the right values; and chunked reads are checked to use one read_ranges call. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -336,9 +336,19 @@ impl AttributeMessage {
|
||||
file_data: &[u8],
|
||||
offset_size: u8,
|
||||
length_size: u8,
|
||||
) -> Result<Vec<String>, FormatError> {
|
||||
self.read_vl_strings_in(file_data, offset_size, length_size)
|
||||
}
|
||||
|
||||
/// [`Self::read_vl_strings`] over any [`Storage`].
|
||||
pub fn read_vl_strings_in<S: Storage + ?Sized>(
|
||||
&self,
|
||||
file_data: &S,
|
||||
offset_size: u8,
|
||||
length_size: u8,
|
||||
) -> Result<Vec<String>, FormatError> {
|
||||
let num_elements = self.dataspace.num_elements();
|
||||
vl_data::read_vl_strings(
|
||||
vl_data::read_vl_strings_in(
|
||||
file_data,
|
||||
&self.raw_data,
|
||||
num_elements,
|
||||
|
||||
Reference in New Issue
Block a user