quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases 1 Wiki Activity

format: raw data, VDS and VL data over Storage

Browse Source 
Every raw-data path has a generic *_in core, with the &[u8] functions as
thin wrappers: data_read (read_raw_data*, read_raw_data_selection,
read_chunked_native), chunked_read (the v1 B-tree chunk index, list_chunks,
the full, cached, sweep and indexed reads), parallel_read, partial_read,
fill_value (read_full_with_fill, apply_to_unallocated_chunks; and
dataset_fill_value_from_storage is now generic), vds (the virtual file
through Storage, external sources still through the resolver),
vl_data (VlResolver<'a, S = [u8]>, read_vl_strings_in, read_vl_bytes_in),
AttributeMessage::read_vl_strings_in and provenance::verify_dataset_in.

With the whole file in memory nothing changes: chunks and contiguous data
are sliced from it as before. Otherwise a chunked read lists its chunks,
fetches their stored bytes with one Storage::read_ranges call per 64 MiB
batch (chunks the cache already holds are not fetched), then decodes as
today; a selection fetches only the chunks it overlaps, and a contiguous
selection only its runs. Each extent's bounds error is the one the slice
code gave, reported when that extent is reached, so errors keep their
order.

Tests: the equivalence harness now reads every dataset's values (whole,
fill-aware, cached, indexed, three selections, VDS, VL strings and
sequences) through the read_at-only storage and requires the slice
results (all 653 corpus files agree); a misbehaving storage (a failing
Nth read, short reads) only ever yields errors or the right values; and
chunked reads are checked to use one read_ranges call.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 16:28:01 -05:00
co-authored by Claude Opus 5.5
parent 42894bf93b
commit 3fa5ed1dda
13 changed files with 1803 additions and 279 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-format/src/vl_data.rs
+138 -66
View File
@@ -5,9 +5,9 @@
//! `sequence_length(4 LE) + collection_address(offset_size LE) + object_index(4 LE)`.
#[cfg(not(feature = "std"))]
use alloc::{collections::BTreeMap, format, string::String, vec, vec::Vec};
use alloc::{borrow::Cow, collections::BTreeMap, format, string::String, vec, vec::Vec};
#[cfg(feature = "std")]
use std::collections::BTreeMap;
use std::{borrow::Cow, collections::BTreeMap};
use crate::addr::to_usize;
use crate::error::FormatError;
@@ -137,13 +137,15 @@ pub fn check_element_size(stored_size: u32, offset_size: u8) -> Result<(), Forma
/// A collection's objects, located in the file data but not copied:
/// `(index, offset, size)` of the first object with each index, sorted by
/// index.
struct CachedCollection {
/// index. Over a storage without the whole file in memory, also the
/// collection's bytes (`(offset, bytes)`), read once when it is indexed.
struct CachedCollection<'a> {
objects: Vec<(u16, usize, usize)>,
bytes: Option<(usize, Cow<'a, [u8]>)>,
}
impl CachedCollection {
fn new(index: GlobalHeapIndex) -> Self {
impl<'a> CachedCollection<'a> {
fn new(index: GlobalHeapIndex, bytes: Option<(usize, Cow<'a, [u8]>)>) -> Self {
let mut objects: Vec<(u16, usize, usize)> = index
.objects
.iter()
@@ -152,12 +154,16 @@ impl CachedCollection {
// Stable, so the first object with a repeated index is kept.
objects.sort_by_key(|o| o.0);
objects.dedup_by_key(|o| o.0);
Self { objects }
Self { objects, bytes }
}
/// What this entry costs to keep, in bytes (roughly).
fn cost(&self) -> usize {
64 + self.objects.len() * core::mem::size_of::<(u16, usize, usize)>()
let held = match &self.bytes {
Some((_, Cow::Owned(b))) => b.len(),
_ => 0,
};
64 + self.objects.len() * core::mem::size_of::<(u16, usize, usize)>() + held
}
fn get(&self, index: u32) -> Option<(usize, usize)> {
@@ -170,6 +176,8 @@ impl CachedCollection {
/// How many bytes of collection indexes a [`VlResolver`] keeps before it
/// drops them and starts again. Values are never copied into the cache, so
/// this bounds what a read retains however many collections it visits.
/// (Over a storage without the whole file in memory the collections' bytes
/// are kept too, and count against this.)
const CACHE_BUDGET: usize = 32 << 20;
/// Resolves variable-length elements against a file's global heap, parsing
@@ -185,11 +193,18 @@ const CACHE_BUDGET: usize = 32 << 20;
/// that overlap one another are refused (libhdf5 never writes them), so a
/// file cannot make the resolver parse the same bytes as the objects of
/// many collections.
pub struct VlResolver<'a> {
file_data: &'a [u8],
///
/// The file is any [`Storage`](crate::storage::Storage) (`S`, a slice by default). Over one without
/// the whole file in memory each collection is read once, when first used,
/// and kept (within the budget above); [`Self::strings`],
/// [`Self::string_bytes`] and [`Self::sequences`] work over any storage,
/// [`Self::element`] and [`Self::string_element`], which borrow from the
/// file, over a slice.
pub struct VlResolver<'a, S: crate::storage::Storage + ?Sized = [u8]> {
file_data: &'a S,
offset_size: u8,
length_size: u8,
cache: BTreeMap<u64, CachedCollection>,
cache: BTreeMap<u64, CachedCollection<'a>>,
cached_bytes: usize,
budget: usize,
/// Start → end of every collection parsed so far (kept when the cache
@@ -201,6 +216,56 @@ impl<'a> VlResolver<'a> {
/// A resolver over `file_data` (the file from its superblock on), with
/// the superblock's offset and length sizes.
pub fn new(file_data: &'a [u8], offset_size: u8, length_size: u8) -> Self {
Self::new_in(file_data, offset_size, length_size)
}
/// One element (the first [`element_size`](Self::element_size) bytes of
/// `elem`) of a variable-length sequence whose base type is `base_size`
/// bytes: its `length × base_size` bytes, or `None` for a null element
/// (heap address 0).
pub fn element(
&mut self,
elem: &[u8],
base_size: usize,
) -> Result<Option<&'a [u8]>, FormatError> {
let vl = parse_vl_references(elem, 1, self.offset_size)?;
let vl = &vl[0];
if vl.collection_address == 0 {
return Ok(None);
}
let (start, size) = self.locate(vl)?;
let data = &self.file_data[start..start + size];
check_object_size(vl, data.len(), base_size)?;
Ok(Some(data))
}
/// One variable-length string element: its bytes up to the first NUL,
/// or `None` for a null element (h5dump prints it as `NULL`, h5py
/// returns it as empty).
pub fn string_element(&mut self, elem: &[u8]) -> Result<Option<&'a [u8]>, FormatError> {
Ok(self.element(elem, 1)?.map(cut_at_nul))
}
}
/// `data_len`, the size of `vl`'s heap object, against the `length ×
/// base_size` bytes the element says it holds.
fn check_object_size(vl: &VlElement, data_len: usize, base_size: usize) -> Result<(), FormatError> {
let expected = (vl.length as usize)
.checked_mul(base_size)
.ok_or_else(|| FormatError::Overflow("variable-length element size".into()))?;
if data_len != expected {
return Err(FormatError::VlDataError(format!(
"global heap object {} in the collection at {} holds {data_len} bytes; the element \
says {} × {base_size}",
vl.object_index, vl.collection_address, vl.length
)));
}
Ok(())
}
impl<'a, S: crate::storage::Storage + ?Sized> VlResolver<'a, S> {
/// [`VlResolver::new`] over any [`Storage`](crate::storage::Storage).
pub fn new_in(file_data: &'a S, offset_size: u8, length_size: u8) -> Self {
Self {
file_data,
offset_size,
@@ -231,51 +296,15 @@ impl<'a> VlResolver<'a> {
/// The bytes of one element: `length × base_size` bytes from the heap,
/// or `None` for a null element.
fn resolve(
&mut self,
vl: &VlElement,
base_size: usize,
) -> Result<Option<&'a [u8]>, FormatError> {
let addr = vl.collection_address;
if addr == 0 {
fn resolve(&mut self, vl: &VlElement, base_size: usize) -> Result<Option<&[u8]>, FormatError> {
if vl.collection_address == 0 {
return Ok(None);
}
let data = self.object(vl)?;
let expected = (vl.length as usize)
.checked_mul(base_size)
.ok_or_else(|| FormatError::Overflow("variable-length element size".into()))?;
if data.len() != expected {
return Err(FormatError::VlDataError(format!(
"global heap object {} in the collection at {addr} holds {} bytes; the element \
says {} × {base_size}",
vl.object_index,
data.len(),
vl.length
)));
}
check_object_size(vl, data.len(), base_size)?;
Ok(Some(data))
}
/// One element (the first [`element_size`](Self::element_size) bytes of
/// `elem`) of a variable-length sequence whose base type is `base_size`
/// bytes: its `length × base_size` bytes, or `None` for a null element
/// (heap address 0).
pub fn element(
&mut self,
elem: &[u8],
base_size: usize,
) -> Result<Option<&'a [u8]>, FormatError> {
let vl = parse_vl_references(elem, 1, self.offset_size)?;
self.resolve(&vl[0], base_size)
}
/// One variable-length string element: its bytes up to the first NUL,
/// or `None` for a null element (h5dump prints it as `NULL`, h5py
/// returns it as empty).
pub fn string_element(&mut self, elem: &[u8]) -> Result<Option<&'a [u8]>, FormatError> {
Ok(self.element(elem, 1)?.map(cut_at_nul))
}
/// The strings of the variable-length string elements in `raw`, as
/// bytes. A string ends at its first NUL, as libhdf5 returns it (it
/// converts each to a C string); a null element is empty.
@@ -330,9 +359,20 @@ pub fn read_vl_strings(
num_elements: u64,
offset_size: u8,
length_size: u8,
) -> Result<Vec<String>, FormatError> {
read_vl_strings_in(file_data, raw_data, num_elements, offset_size, length_size)
}
/// [`read_vl_strings`] over any [`Storage`](crate::storage::Storage).
pub fn read_vl_strings_in<S: crate::storage::Storage + ?Sized>(
file_data: &S,
raw_data: &[u8],
num_elements: u64,
offset_size: u8,
length_size: u8,
) -> Result<Vec<String>, FormatError> {
let raw = first_elements(raw_data, num_elements, offset_size)?;
VlResolver::new(file_data, offset_size, length_size).strings(raw)
VlResolver::new_in(file_data, offset_size, length_size).strings(raw)
}
/// The first `num_elements` elements of `raw`, or an error if it is shorter.
@@ -364,9 +404,20 @@ pub fn read_vl_bytes(
num_elements: u64,
offset_size: u8,
length_size: u8,
) -> Result<Vec<Vec<u8>>, FormatError> {
read_vl_bytes_in(file_data, raw_data, num_elements, offset_size, length_size)
}
/// [`read_vl_bytes`] over any [`Storage`](crate::storage::Storage).
pub fn read_vl_bytes_in<S: crate::storage::Storage + ?Sized>(
file_data: &S,
raw_data: &[u8],
num_elements: u64,
offset_size: u8,
length_size: u8,
) -> Result<Vec<Vec<u8>>, FormatError> {
let refs = parse_vl_references(raw_data, num_elements, offset_size)?;
let mut resolver = VlResolver::new(file_data, offset_size, length_size);
let mut resolver = VlResolver::new_in(file_data, offset_size, length_size);
let mut result = Vec::with_capacity(refs.len());
for vl in &refs {
@@ -385,10 +436,10 @@ pub fn read_vl_bytes(
Ok(result)
}
impl<'a> VlResolver<'a> {
/// The heap object `vl` points to, whatever its size; its collection is
/// parsed on first use.
fn object(&mut self, vl: &VlElement) -> Result<&'a [u8], FormatError> {
impl<'a, S: crate::storage::Storage + ?Sized> VlResolver<'a, S> {
/// Where the heap object `vl` points to lies in the file, whatever its
/// size (`(offset, size)`); its collection is parsed on first use.
fn locate(&mut self, vl: &VlElement) -> Result<(usize, usize), FormatError> {
let addr = vl.collection_address;
// libhdf5 writes a null element with address 0, never the undefined
// address, and fails to read one ("addr undefined") even when its
@@ -402,14 +453,20 @@ impl<'a> VlResolver<'a> {
if !self.cache.contains_key(&addr) {
let offset = usize::try_from(addr).map_err(|_| FormatError::UnexpectedEof {
expected: usize::MAX,
available: self.file_data.len(),
available: crate::storage::len_usize(self.file_data),
})?;
let index =
GlobalHeapCollection::parse_index(self.file_data, offset, self.length_size)?;
// parse_index checked that the collection lies in the file.
let (bytes, base, index) =
GlobalHeapCollection::read_collection(self.file_data, addr, self.length_size)?;
// read_collection checked that the collection lies in the file.
let end = offset + to_usize(index.collection_size)?;
self.check_overlap(offset, end)?;
let coll = CachedCollection::new(index);
// With the whole file in memory the objects are sliced from it;
// otherwise the collection's bytes are kept.
let bytes = match self.file_data.as_contiguous() {
Some(_) => None,
None => Some((base, bytes)),
};
let coll = CachedCollection::new(index, bytes);
if self.cached_bytes.saturating_add(coll.cost()) > self.budget {
self.cache.clear();
self.cached_bytes = 0;
@@ -417,13 +474,27 @@ impl<'a> VlResolver<'a> {
self.cached_bytes += coll.cost();
self.cache.insert(addr, coll);
}
let (start, size) = self.cache[&addr].get(vl.object_index).ok_or(
FormatError::GlobalHeapObjectNotFound {
self.cache[&addr]
.get(vl.object_index)
.ok_or(FormatError::GlobalHeapObjectNotFound {
collection_address: addr,
index: vl.object_index as u16,
},
)?;
Ok(&self.file_data[start..start + size])
})
}
/// The heap object `vl` points to, whatever its size; its collection is
/// parsed on first use.
fn object(&mut self, vl: &VlElement) -> Result<&[u8], FormatError> {
let (start, size) = self.locate(vl)?;
if let Some(all) = self.file_data.as_contiguous() {
return Ok(&all[start..start + size]);
}
match &self.cache[&vl.collection_address].bytes {
Some((base, bytes)) => Ok(&bytes[start - base..start - base + size]),
None => Err(FormatError::Storage(
"global heap collection bytes were not kept".into(),
)),
}
}
/// Record the collection at `start..end`, refusing one that overlaps a
@@ -709,6 +780,7 @@ mod tests {
let mut r = VlResolver::new(&file_data, 8, 8);
let one = CachedCollection {
objects: vec![(0, 0, 0); 3],
bytes: None,
}
.cost();
r.budget = 2 * one + 1;