fix: apply the superblock extension and cache image in every opener
File, MmapFile and LazyFile decoded the superblock extension and laid a metadata cache image over the file's metadata; the other readers did not, so the same file read differently by entry point: NativeVol, AsyncHDF5File and MpiVol (clawhdf5-io) and the external source files of a virtual dataset (clawhdf5-format vds.rs) read a file with an image from its own bytes, which libhdf5 does not (they may be stale, or zeros: h5clear_mdc_image.h5 failed with InvalidObjectHeaderVersion(0)), and skipped the extension checks File::open makes (cve-2020-10810/10812). Each of them owns its buffer, so each now calls the shared superblock_ext::apply_cache_image_in_place, which checks the extension and writes the image's entries in place (only the image block is copied). These readers read whole datasets and cannot open a file and fail each object, so an image libhdf5 cannot load is refused with the image's error, never read around. clawhdf5-io's vol::load_hdf5 wraps it for NativeVol (at open; for from_bytes the error is reported on read, as a truncated file already was) and MpiVol. The MpiVol edit is minimal and was not compiled: the mpi-io feature needs an MPI installation this machine does not have (mpi-sys's build script panics). Tests: NativeVol (open_path and from_bytes), AsyncHDF5File and a VDS whose source file is h5clear_mdc_image.h5 (vds_interop.rs, against h5py) read the fixture's values; the corrupted-image variants are refused. Each fails without its fix. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -803,7 +803,11 @@ impl<'a, 'r> Sources<'a, 'r> {
|
||||
let resolver = self.resolver.ok_or_else(|| {
|
||||
vds_err("external-file virtual dataset sources require a file resolver")
|
||||
})?;
|
||||
self.cached_file = Some((String::from(name), resolver(name)?));
|
||||
let mut bytes = resolver(name)?;
|
||||
if let Some(b) = bytes.as_mut() {
|
||||
load_source_file(b)?;
|
||||
}
|
||||
self.cached_file = Some((String::from(name), bytes));
|
||||
}
|
||||
// An external file is handed over whole; its addresses are relative
|
||||
// to its superblock, so skip any user block.
|
||||
@@ -851,6 +855,23 @@ impl<'a, 'r> Sources<'a, 'r> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Check an external source file's superblock extension as libhdf5 does
|
||||
/// when it opens the file, and write any metadata cache image over its
|
||||
/// metadata in place: libhdf5 reads the image's entries instead of the
|
||||
/// file's own, possibly stale, bytes (`crate::superblock_ext`). A source
|
||||
/// file whose image cannot be loaded is an error, as other corrupt source
|
||||
/// files are here.
|
||||
fn load_source_file(whole: &mut [u8]) -> Result<(), FormatError> {
|
||||
let base = crate::signature::find_signature(whole)?;
|
||||
let sb = crate::superblock::Superblock::parse(&whole[base..], 0)?;
|
||||
// The end of file the superblock records; a truncated source file is
|
||||
// read as before, up to its length.
|
||||
let end = sb
|
||||
.data_end(base as u64, whole.len() as u64)
|
||||
.map_or(whole.len(), |e| base + e as usize);
|
||||
crate::superblock_ext::apply_cache_image_in_place(&mut whole[base..end], &sb)
|
||||
}
|
||||
|
||||
/// Whether elements of `dt` contain addresses into their own file:
|
||||
/// variable-length data (global-heap IDs) or references.
|
||||
fn holds_file_addresses(dt: &Datatype) -> bool {
|
||||
|
||||
Reference in New Issue
Block a user