format, clawhdf5: cut every Storage read to the range asked for

ExtentBytes and read_exact_at/read_upto rejected short results but passed
longer-than-asked ones through, and FileData forwarded them too, so a
Storage that broke read_at's contract by returning extra bytes had them
decoded or returned as data (a contiguous dataset read gained 37 junk
bytes). gather_storage alone trimmed.

- storage::exact_len (new, pub): a read of len bytes as exactly len — cut
  when longer, an error when short. read_exact_at, read_upto and
  ExtentBytes (so chunk fetches and selection gathers) go through it.
- FileData cuts a backend's answer to what it asked for before laying the
  cache image over it.
- Tests: over a storage that appends 37 junk bytes to every read, every
  format-crate fixture reads exactly as from the slice
  (overlong_reads_are_cut_to_the_range_asked_for), and every facade
  fixture opens and reads through File::open_storage as through File::open
  (overlong_storage_reads_identically). Both failed before.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:43:05 -05:00
co-authored by Claude Opus 5.5
parent 89e7977943
commit 6185874f9c
5 changed files with 225 additions and 90 deletions
+28 -15
View File
@@ -43,6 +43,9 @@ pub trait Storage {
/// end of the storage (and empty when `offset` is at or past the end);
/// a backend that cannot serve a range returns an error instead of a
/// short read.
/// It is never longer than `len`; the parsers cut a longer result to
/// `len` (see [`exact_len`]) rather than read bytes from outside the
/// range.
fn read_at(&self, offset: u64, len: usize) -> Result<Cow<'_, [u8]>, FormatError>;
/// Current length of the storage in bytes.
@@ -218,13 +221,30 @@ pub fn read_exact_at<S: Storage + ?Sized>(
Some(end) if end <= file.len() => {}
_ => return Err(eof()),
}
let bytes = file.read_at(offset, len)?;
if bytes.len() < len {
// The storage shrank or the backend served a short read inside the
// file: never parse a partial structure.
return Err(short_read());
// A short read (the storage shrank, or the backend served less inside
// the file) is an error: never parse a partial structure.
exact_len(file.read_at(offset, len)?, len)
}
/// `bytes`, the result of asking a [`Storage`] for `len` bytes, as exactly
/// `len` bytes: a longer result (a backend that broke
/// [`Storage::read_at`]'s contract) is cut to `len`, so bytes from outside
/// the range asked for are never parsed or returned; a shorter one is an
/// error (the storage shrank, or the backend failed), never a partial
/// structure.
#[inline]
pub fn exact_len(bytes: Cow<'_, [u8]>, len: usize) -> Result<Cow<'_, [u8]>, FormatError> {
match bytes.len().cmp(&len) {
core::cmp::Ordering::Equal => Ok(bytes),
core::cmp::Ordering::Less => Err(short_read()),
core::cmp::Ordering::Greater => Ok(match bytes {
Cow::Borrowed(b) => Cow::Borrowed(&b[..len]),
Cow::Owned(mut v) => {
v.truncate(len);
Cow::Owned(v)
}
}),
}
Ok(bytes)
}
#[cold]
@@ -329,11 +349,7 @@ pub fn read_upto<S: Storage + ?Sized>(
}
let avail = file.len().saturating_sub(offset);
let len = usize::try_from(avail).map_or(max, |a| a.min(max));
let bytes = file.read_at(offset, len)?;
if bytes.len() < len {
return Err(short_read());
}
Ok(bytes)
exact_len(file.read_at(offset, len)?, len)
}
/// Most stored bytes fetched by one [`Storage::read_ranges`] call when a
@@ -461,10 +477,7 @@ impl<'a> ExtentBytes<'a> {
));
}
for ((slot, bytes), r) in slots.into_iter().zip(got).zip(&ranges) {
if (bytes.len() as u64) < r.end - r.start {
return Err(short_read());
}
out[slot] = Extent::Bytes(bytes);
out[slot] = Extent::Bytes(exact_len(bytes, (r.end - r.start) as usize)?);
}
}
Ok(ExtentBytes::Fetched { base, extents: out })