osobhandClaude Opus 5.5
60502593b7
apply_cache_image returned a copy of the whole file with the image's
entries written in, and File (mmap by default), MmapFile and LazyFile
used that copy for every read: opening a 1 GiB sparse file with an image
needed 2 GB of memory, and an 8 GiB one aborted the process, where
de2a53f (which ignored the image) opened them in a few MB.
The metadata parsers read one contiguous slice, so the image still has
to be laid over the file's bytes; it is now laid over a private copy
that costs only the pages it touches:
- clawhdf5_format::superblock_ext::CacheImage decodes the image into an
entry list (address, offset in the block, length) and applies it to
any destination; cache_image_state tells an opener whether the file
has no image, a loadable one, or one libhdf5 cannot load;
apply_cache_image_in_place is for readers that own their buffer.
apply_cache_image and metadata_view (which copied) are gone.
- clawhdf5_io::HDF5Read::private_copy returns a writable private copy
of a reader's bytes: MmapReader gives a MAP_PRIVATE copy-on-write
mapping (memmap2 map_copy), so only the pages the entries land on are
copied; the default copies the bytes (in-memory readers).
- File, MmapFile and LazyFile write the image into that mapping
(crate::cache_image). File::from_bytes / open_buffered patch their own
buffer in place, copying only the image block, as libhdf5 does. A
file without an image is read straight from the mapping, unchanged.
An image entry that runs past the end of file is now refused: libhdf5
checks only that it starts inside the file, and the images libhdf5
writes never do this, but those bytes have nowhere to go in a view of
the file.
Tests: tests/cache_image_memory.rs has libhdf5 (through ctypes) add an
image to a 1 GiB sparse file and bounds resident-memory growth for all
three openers at 256 MiB; it fails on the previous commit (File::open
grew 2,148,720,640 bytes). reader.rs zero_copy_tests check that a file
without an image is read from the mapping itself and that an image goes
into a copy-on-write mapping, not a heap copy; clawhdf5-io checks that
private_copy writes never reach the reader or the file.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
813 lines
31 KiB
Rust
813 lines
31 KiB
Rust
//! The superblock extension of a version 2 or 3 superblock, and the
|
|
//! metadata cache image it can point to.
|
|
//!
|
|
//! libhdf5 reads the extension when it opens a file (`H5F__super_read`) and
|
|
//! decodes the messages that configure the file: v1 B-tree "K" values, File
|
|
//! Space Info, and the Metadata Cache Image. A message that does not decode
|
|
//! makes the file fail to open, so [`read_superblock_extension`] decodes and
|
|
//! checks them the way libhdf5 does.
|
|
//!
|
|
//! A metadata cache image (written with `H5Pset_mdc_image_config`) is a
|
|
//! block holding serialized metadata cache entries — object headers, B-tree
|
|
//! nodes, heaps — each with its file address. libhdf5 loads it into its
|
|
//! cache before it reads any other metadata (`H5C__load_cache_image`,
|
|
//! `H5C__reconstruct_cache_contents`), and the entries take the place of
|
|
//! the file's bytes at their addresses: the file itself may hold stale or
|
|
//! no metadata there (in `h5clear_mdc_image.h5` the root group's header is
|
|
//! only in the image). [`CacheImage::apply`] does the same with bytes: it
|
|
//! writes every entry at its address, so every parser reads what libhdf5
|
|
//! reads. It writes into whatever the opener gives it — a private
|
|
//! copy-on-write mapping of the file, or a buffer the opener owns — so the
|
|
//! file is never copied whole.
|
|
|
|
#[cfg(not(feature = "std"))]
|
|
use alloc::{collections::BTreeSet, vec::Vec};
|
|
#[cfg(feature = "std")]
|
|
use std::collections::BTreeSet;
|
|
|
|
use crate::error::FormatError;
|
|
use crate::message_type::MessageType;
|
|
use crate::object_header::ObjectHeader;
|
|
use crate::superblock::Superblock;
|
|
|
|
/// Message type of the File Space Info message.
|
|
const MSG_FSINFO: u16 = 0x0017;
|
|
/// Message type of the Metadata Cache Image message.
|
|
const MSG_MDCI: u16 = 0x0018;
|
|
/// Header message flag: the library did not know the message when it wrote
|
|
/// it back (`H5O_MSG_FLAG_WAS_UNKNOWN`); libhdf5 then ignores its contents.
|
|
const MSG_FLAG_WAS_UNKNOWN: u8 = 0x20;
|
|
|
|
/// `H5F_FILE_SPACE_PAGE_SIZE_MIN` / `_MAX`.
|
|
const PAGE_SIZE_MIN: u64 = 512;
|
|
const PAGE_SIZE_MAX: u64 = 1024 * 1024 * 1024;
|
|
/// libhdf5's default file space page size, used for a version 0 message.
|
|
const PAGE_SIZE_DEFAULT: u64 = 4096;
|
|
/// Free-space managers whose addresses a persisting version 1 File Space
|
|
/// Info message lists (`H5F_MEM_PAGE_SUPER` .. `H5F_MEM_PAGE_NTYPES`), and
|
|
/// a version 0 one (`H5FD_MEM_SUPER` .. `H5FD_MEM_NTYPES`).
|
|
const FSM_ADDRS_V1: usize = 12;
|
|
const FSM_ADDRS_V0: usize = 6;
|
|
|
|
/// Metadata cache image block limits (`H5Cimage.c`, `H5ACprivate.h`).
|
|
const MDCI_SIGNATURE: &[u8; 4] = b"MDCI";
|
|
const MDCI_HAVE_RESIZE_STATUS: u8 = 0x01;
|
|
const MDCI_ENTRY_IS_FD_PARENT: u8 = 0x04;
|
|
const MDCI_ENTRY_IS_FD_CHILD: u8 = 0x08;
|
|
/// `H5AC_NTYPES`: entry type ids are below this.
|
|
const MDCI_NTYPES: u8 = 30;
|
|
/// `H5C_RING_NTYPES`.
|
|
const MDCI_RING_NTYPES: u8 = 6;
|
|
/// `H5AC__CACHE_IMAGE__ENTRY_AGEOUT__MAX`.
|
|
const MDCI_AGE_MAX: u8 = 100;
|
|
|
|
/// A decoded File Space Info message (0x0017), mapped to version 1 as
|
|
/// libhdf5 maps a version 0 one.
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
pub struct FileSpaceInfo {
|
|
/// Message version as stored (0 or 1).
|
|
pub version: u8,
|
|
/// File space strategy (`H5F_fspace_strategy_t`).
|
|
pub strategy: u8,
|
|
/// Whether free space is persisted.
|
|
pub persist: bool,
|
|
/// Free-space section threshold.
|
|
pub threshold: u64,
|
|
/// File space page size.
|
|
pub page_size: u64,
|
|
}
|
|
|
|
/// Where a metadata cache image block is (Metadata Cache Image message,
|
|
/// 0x0018).
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub struct CacheImageLocation {
|
|
/// Address of the image block.
|
|
pub address: u64,
|
|
/// Length of the image block in bytes.
|
|
pub length: u64,
|
|
}
|
|
|
|
/// The messages of a superblock extension that libhdf5 decodes at open.
|
|
#[derive(Debug, Clone, Default, PartialEq, Eq)]
|
|
pub struct SuperblockExtension {
|
|
/// v1 B-tree "K" values (chunk index, symbol table node, symbol table
|
|
/// leaf), when the extension overrides the defaults.
|
|
pub btree_k: Option<(u16, u16, u16)>,
|
|
/// The File Space Info message.
|
|
pub file_space_info: Option<FileSpaceInfo>,
|
|
/// The metadata cache image, when the file has one.
|
|
pub cache_image: Option<CacheImageLocation>,
|
|
}
|
|
|
|
fn ext_err(why: &'static str) -> FormatError {
|
|
FormatError::InvalidSuperblockExtension(why)
|
|
}
|
|
|
|
const RAN_OFF: &str = "ran off end of input buffer while decoding";
|
|
|
|
/// A little-endian cursor over one message or block, failing with
|
|
/// `overrun` when it runs off the end.
|
|
struct Cursor<'a> {
|
|
data: &'a [u8],
|
|
pos: usize,
|
|
overrun: FormatError,
|
|
}
|
|
|
|
impl<'a> Cursor<'a> {
|
|
fn new(data: &'a [u8], overrun: FormatError) -> Self {
|
|
Cursor {
|
|
data,
|
|
pos: 0,
|
|
overrun,
|
|
}
|
|
}
|
|
|
|
fn take(&mut self, n: usize) -> Result<&'a [u8], FormatError> {
|
|
let end = self
|
|
.pos
|
|
.checked_add(n)
|
|
.filter(|&e| e <= self.data.len())
|
|
.ok_or_else(|| self.overrun.clone())?;
|
|
let s = &self.data[self.pos..end];
|
|
self.pos = end;
|
|
Ok(s)
|
|
}
|
|
|
|
fn u8(&mut self) -> Result<u8, FormatError> {
|
|
Ok(self.take(1)?[0])
|
|
}
|
|
|
|
fn uint(&mut self, width: u8) -> Result<u64, FormatError> {
|
|
let b = self.take(width as usize)?;
|
|
Ok(b.iter()
|
|
.rev()
|
|
.fold(0u64, |acc, &x| (acc << 8) | u64::from(x)))
|
|
}
|
|
|
|
/// An address of `width` bytes; `None` when undefined (all ones).
|
|
fn addr(&mut self, width: u8) -> Result<Option<u64>, FormatError> {
|
|
let v = self.uint(width)?;
|
|
let undef = if width >= 8 {
|
|
u64::MAX
|
|
} else {
|
|
(1u64 << (8 * u32::from(width))) - 1
|
|
};
|
|
Ok((v != undef).then_some(v))
|
|
}
|
|
}
|
|
|
|
/// Decode and check the superblock extension of `sb`, as libhdf5 does when
|
|
/// it opens the file. `data` is the file from the superblock on, up to the
|
|
/// end of file the superblock records (its end is libhdf5's "eoa").
|
|
///
|
|
/// Returns `Ok(None)` for a superblock without an extension (versions 0
|
|
/// and 1 have none). A message libhdf5 fails to decode, or a cache image
|
|
/// that does not lie inside the file, is an error: libhdf5 refuses to open
|
|
/// such a file (`cve-2020-10810`: a File Space Info message too short for
|
|
/// the free-space manager addresses it announces; `cve-2020-10812`: a cache
|
|
/// image past the end of the file).
|
|
pub fn read_superblock_extension(
|
|
data: &[u8],
|
|
sb: &Superblock,
|
|
) -> Result<Option<SuperblockExtension>, FormatError> {
|
|
let os = sb.offset_size;
|
|
let ls = sb.length_size;
|
|
let undef = if os >= 8 {
|
|
u64::MAX
|
|
} else {
|
|
(1u64 << (8 * u32::from(os))) - 1
|
|
};
|
|
let Some(addr) = sb.superblock_extension_address.filter(|&a| a != undef) else {
|
|
return Ok(None);
|
|
};
|
|
let addr = usize::try_from(addr).map_err(|_| ext_err("address out of range"))?;
|
|
let header = ObjectHeader::parse(data, addr, os, ls)?;
|
|
let eoa = data.len() as u64;
|
|
|
|
let mut ext = SuperblockExtension::default();
|
|
for msg in &header.messages {
|
|
match msg.msg_type {
|
|
MessageType::BTreeKValues => {
|
|
let mut c = Cursor::new(&msg.data, ext_err(RAN_OFF));
|
|
if c.u8()? != 0 {
|
|
return Err(ext_err("bad version number for v1 B-tree 'K' message"));
|
|
}
|
|
let chunk = c.uint(2)? as u16;
|
|
let snode = c.uint(2)? as u16;
|
|
let leaf = c.uint(2)? as u16;
|
|
ext.btree_k = Some((chunk, snode, leaf));
|
|
}
|
|
MessageType::Unknown(MSG_FSINFO) if msg.flags & MSG_FLAG_WAS_UNKNOWN == 0 => {
|
|
ext.file_space_info = Some(decode_fsinfo(&msg.data, os, ls)?);
|
|
}
|
|
MessageType::Unknown(MSG_MDCI) => {
|
|
let mut c = Cursor::new(&msg.data, ext_err(RAN_OFF));
|
|
if c.u8()? != 0 {
|
|
return Err(ext_err(
|
|
"bad version number for metadata cache image message",
|
|
));
|
|
}
|
|
let address = c.addr(os)?;
|
|
let length = c.uint(ls)?;
|
|
let Some(address) = address else {
|
|
return Err(ext_err("metadata cache image address is undefined"));
|
|
};
|
|
if address.checked_add(length).is_none_or(|end| end > eoa) {
|
|
return Err(ext_err(
|
|
"metadata cache image: address plus size exceeds file eoa",
|
|
));
|
|
}
|
|
ext.cache_image = Some(CacheImageLocation { address, length });
|
|
}
|
|
_ => {}
|
|
}
|
|
}
|
|
Ok(Some(ext))
|
|
}
|
|
|
|
/// `H5O__fsinfo_decode` plus the checks `H5F__super_read` makes on it.
|
|
fn decode_fsinfo(data: &[u8], os: u8, ls: u8) -> Result<FileSpaceInfo, FormatError> {
|
|
let mut c = Cursor::new(data, ext_err(RAN_OFF));
|
|
let version = c.u8()?;
|
|
let info = if version == 0 {
|
|
let old_strategy = c.u8()?;
|
|
let threshold = c.uint(ls)?;
|
|
// H5F_file_space_type_t: 1 ALL_PERSIST, 2 ALL, 3 AGGR_VFD, 4 VFD.
|
|
let (strategy, persist) = match old_strategy {
|
|
1 => {
|
|
for _ in 0..FSM_ADDRS_V0 {
|
|
c.addr(os)?;
|
|
}
|
|
(0, true)
|
|
}
|
|
2 => (0, false),
|
|
3 => (2, false),
|
|
4 => (3, false),
|
|
_ => return Err(ext_err("invalid file space strategy")),
|
|
};
|
|
FileSpaceInfo {
|
|
version,
|
|
strategy,
|
|
persist,
|
|
threshold,
|
|
page_size: PAGE_SIZE_DEFAULT,
|
|
}
|
|
} else {
|
|
if version > 1 {
|
|
return Err(ext_err("File space info message's version out of bounds"));
|
|
}
|
|
let strategy = c.u8()?;
|
|
let persist = c.u8()? != 0;
|
|
let threshold = c.uint(ls)?;
|
|
let page_size = c.uint(ls)?;
|
|
if page_size == 0 || page_size > PAGE_SIZE_MAX {
|
|
return Err(ext_err("invalid page size in file space info"));
|
|
}
|
|
c.uint(2)?; // page end metadata threshold
|
|
c.addr(os)?; // EOA before the free-space managers
|
|
if persist {
|
|
for _ in 0..FSM_ADDRS_V1 {
|
|
c.addr(os)?;
|
|
}
|
|
}
|
|
FileSpaceInfo {
|
|
version,
|
|
strategy,
|
|
persist,
|
|
threshold,
|
|
page_size,
|
|
}
|
|
};
|
|
if info.page_size < PAGE_SIZE_MIN {
|
|
return Err(ext_err("file space page size too small"));
|
|
}
|
|
Ok(info)
|
|
}
|
|
|
|
/// One entry of a metadata cache image: `len` bytes at `image_offset` in
|
|
/// the image block, belonging at file address `address`.
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
struct ImageEntry {
|
|
address: u64,
|
|
image_offset: usize,
|
|
len: usize,
|
|
}
|
|
|
|
/// A decoded metadata cache image: where its block is, and the entries it
|
|
/// holds. [`CacheImage::apply`] writes the entries over a file's bytes.
|
|
///
|
|
/// Only the entry list is kept, never a copy of the file: an opener that
|
|
/// maps the file applies the image to a private copy-on-write mapping, so
|
|
/// only the pages the entries land on are copied.
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
pub struct CacheImage {
|
|
location: CacheImageLocation,
|
|
entries: Vec<ImageEntry>,
|
|
}
|
|
|
|
/// What an opener must do about a file's metadata cache image.
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
pub enum CacheImageState {
|
|
/// The file has no image: its bytes are its metadata.
|
|
Absent,
|
|
/// The file has an image that loads: apply it with [`CacheImage::apply`].
|
|
Loaded(CacheImage),
|
|
/// The file has an image libhdf5 fails to load. libhdf5 still opens the
|
|
/// file (the image loads at the first metadata read), and that read
|
|
/// fails with this error.
|
|
Unloadable(FormatError),
|
|
}
|
|
|
|
impl CacheImage {
|
|
/// Decode the metadata cache image at `location` in `data` (the file
|
|
/// from the superblock on, up to its recorded end of file). The image is
|
|
/// checked as libhdf5 checks it (`H5C__decode_cache_image_header`,
|
|
/// `H5C__reconstruct_cache_entry`): signature and version, the image
|
|
/// length it records, entry types, rings and ages in range, entry
|
|
/// addresses inside the file and not repeated, flush-dependency parents
|
|
/// already in the cache.
|
|
///
|
|
/// One check is stricter than libhdf5's: an entry must end inside the
|
|
/// file. libhdf5 checks only that it starts there, and serves the rest
|
|
/// from the image; the images libhdf5 writes never do this (every entry
|
|
/// lies below the image block, which is written last), and the bytes an
|
|
/// entry would put past the end of file have nowhere to go in a view of
|
|
/// the file.
|
|
///
|
|
/// libhdf5 does not verify the block's trailing checksum when it loads
|
|
/// an image, so neither does this.
|
|
pub fn decode(
|
|
data: &[u8],
|
|
location: CacheImageLocation,
|
|
sb: &Superblock,
|
|
) -> Result<Self, FormatError> {
|
|
let (offset_size, length_size) = (sb.offset_size, sb.length_size);
|
|
let bad = FormatError::InvalidCacheImage;
|
|
let block = image_block(data, location)?;
|
|
let eoa = data.len() as u64;
|
|
let mut c = Cursor::new(block, bad(RAN_OFF));
|
|
|
|
// Header: signature, version, flags, image data length, entry count.
|
|
if c.take(4)? != MDCI_SIGNATURE {
|
|
return Err(bad("bad metadata cache image header signature"));
|
|
}
|
|
if c.u8()? != 0 {
|
|
return Err(bad("bad metadata cache image version"));
|
|
}
|
|
if c.u8()? & MDCI_HAVE_RESIZE_STATUS != 0 {
|
|
return Err(bad("MDC resize status not yet supported"));
|
|
}
|
|
if c.uint(length_size)? != location.length {
|
|
return Err(bad("bad metadata cache image data length"));
|
|
}
|
|
let n_entries = c.uint(4)?;
|
|
if n_entries == 0 {
|
|
return Err(bad("bad metadata cache entry count"));
|
|
}
|
|
|
|
let mut entries = Vec::new();
|
|
// What is in libhdf5's cache when it loads the image: the superblock
|
|
// and the superblock extension's object header (read to find the
|
|
// image). Each entry's flush-dependency parents are looked up in the
|
|
// cache as the entry is inserted (`H5C__reconstruct_cache_contents`
|
|
// searches the index inside the loop that inserts the entries, in
|
|
// HDF5 1.14.6 and 2.0.0 alike), so a parent must be one of those or
|
|
// an earlier entry.
|
|
let mut cached = BTreeSet::new();
|
|
cached.insert(0);
|
|
if let Some(ext) = sb.superblock_extension_address {
|
|
cached.insert(ext);
|
|
}
|
|
let mut seen = BTreeSet::new();
|
|
for _ in 0..n_entries {
|
|
let type_id = c.u8()?;
|
|
if type_id >= MDCI_NTYPES {
|
|
return Err(bad("type id is out of valid range"));
|
|
}
|
|
let flags = c.u8()?;
|
|
if c.u8()? >= MDCI_RING_NTYPES {
|
|
return Err(bad("ring is out of valid range"));
|
|
}
|
|
if c.u8()? > MDCI_AGE_MAX {
|
|
return Err(bad("entry age is out of policy range"));
|
|
}
|
|
let children = c.uint(2)?;
|
|
// libhdf5 checks the parent flag against the child count only in
|
|
// debug builds (release builds refuse any entry with children);
|
|
// the image format's own rule is checked here.
|
|
if (flags & MDCI_ENTRY_IS_FD_PARENT != 0) != (children > 0) {
|
|
return Err(bad("flush dependency parent flag and child count disagree"));
|
|
}
|
|
c.uint(2)?; // dirty dependency children: reset for a read-only open
|
|
let parents = c.uint(2)?;
|
|
if (flags & MDCI_ENTRY_IS_FD_CHILD != 0) != (parents > 0) {
|
|
return Err(bad("flush dependency child flag and parent count disagree"));
|
|
}
|
|
c.uint(4)?; // LRU rank
|
|
let address = c
|
|
.addr(offset_size)?
|
|
.filter(|&a| a < eoa)
|
|
.ok_or(bad("invalid entry address range"))?;
|
|
let size = c.uint(length_size)?;
|
|
if size == 0 {
|
|
return Err(bad("invalid entry size"));
|
|
}
|
|
for _ in 0..parents {
|
|
let parent = c
|
|
.addr(offset_size)?
|
|
.ok_or(bad("invalid flush dependency parent offset"))?;
|
|
if !seen.contains(&parent) && !cached.contains(&parent) {
|
|
return Err(bad("fd parent not in cache"));
|
|
}
|
|
}
|
|
let len = usize::try_from(size).map_err(|_| bad(RAN_OFF))?;
|
|
let image_offset = c.pos;
|
|
c.take(len)?;
|
|
if address.checked_add(size).is_none_or(|end| end > eoa) {
|
|
return Err(bad("entry extends past the end of file"));
|
|
}
|
|
if !seen.insert(address) {
|
|
return Err(bad("duplicate addresses in cache"));
|
|
}
|
|
entries.push(ImageEntry {
|
|
address,
|
|
image_offset,
|
|
len,
|
|
});
|
|
}
|
|
Ok(CacheImage { location, entries })
|
|
}
|
|
|
|
/// Where the image block is.
|
|
pub fn location(&self) -> CacheImageLocation {
|
|
self.location
|
|
}
|
|
|
|
/// The number of entries in the image.
|
|
pub fn len(&self) -> usize {
|
|
self.entries.len()
|
|
}
|
|
|
|
/// Whether the image has no entries (a decoded image always has some).
|
|
pub fn is_empty(&self) -> bool {
|
|
self.entries.is_empty()
|
|
}
|
|
|
|
/// The file ranges (address, length) the image's entries replace.
|
|
pub fn entry_ranges(&self) -> impl Iterator<Item = (u64, usize)> + '_ {
|
|
self.entries.iter().map(|e| (e.address, e.len))
|
|
}
|
|
|
|
/// The image block in `data`, the bytes [`Self::decode`] read it from.
|
|
pub fn block<'a>(&self, data: &'a [u8]) -> Result<&'a [u8], FormatError> {
|
|
image_block(data, self.location)
|
|
}
|
|
|
|
/// Write every entry over `dst`, the file's bytes from the superblock
|
|
/// on (as long as the `data` the image was decoded from), taking the
|
|
/// entries from `block` (the image block, see [`Self::block`]). `block`
|
|
/// must not alias `dst`: an entry may land on the block itself.
|
|
pub fn apply(&self, block: &[u8], dst: &mut [u8]) -> Result<(), FormatError> {
|
|
let short = || FormatError::InvalidCacheImage("image applied to the wrong file");
|
|
for e in &self.entries {
|
|
let src = block
|
|
.get(e.image_offset..e.image_offset + e.len)
|
|
.ok_or_else(short)?;
|
|
let at = usize::try_from(e.address).map_err(|_| short())?;
|
|
dst.get_mut(at..at + e.len)
|
|
.ok_or_else(short)?
|
|
.copy_from_slice(src);
|
|
}
|
|
Ok(())
|
|
}
|
|
}
|
|
|
|
fn image_block(data: &[u8], location: CacheImageLocation) -> Result<&[u8], FormatError> {
|
|
let bad = FormatError::InvalidCacheImage;
|
|
let start = usize::try_from(location.address).map_err(|_| bad("address out of range"))?;
|
|
let len = usize::try_from(location.length).map_err(|_| bad("length out of range"))?;
|
|
start
|
|
.checked_add(len)
|
|
.and_then(|end| data.get(start..end))
|
|
.ok_or(bad("image block extends past the end of the file"))
|
|
}
|
|
|
|
/// What an opener must do before reading a file's metadata: check the
|
|
/// superblock extension ([`read_superblock_extension`]; an error means
|
|
/// libhdf5 refuses to open the file) and decode any metadata cache image
|
|
/// ([`CacheImage::decode`]). `data` is the file from the superblock on, up
|
|
/// to its recorded end of file.
|
|
pub fn cache_image_state(data: &[u8], sb: &Superblock) -> Result<CacheImageState, FormatError> {
|
|
match read_superblock_extension(data, sb)? {
|
|
Some(SuperblockExtension {
|
|
cache_image: Some(location),
|
|
..
|
|
}) => Ok(match CacheImage::decode(data, location, sb) {
|
|
Ok(image) => CacheImageState::Loaded(image),
|
|
Err(e) => CacheImageState::Unloadable(e),
|
|
}),
|
|
_ => Ok(CacheImageState::Absent),
|
|
}
|
|
}
|
|
|
|
/// [`cache_image_state`] for a reader that holds the file's bytes in a
|
|
/// buffer of its own: check the superblock extension and write any cache
|
|
/// image over `data` in place (only the image block is copied). An image
|
|
/// libhdf5 cannot load is an error here: such a reader has no way to open
|
|
/// the file and fail each object instead.
|
|
pub fn apply_cache_image_in_place(data: &mut [u8], sb: &Superblock) -> Result<(), FormatError> {
|
|
match cache_image_state(data, sb)? {
|
|
CacheImageState::Absent => Ok(()),
|
|
CacheImageState::Unloadable(e) => Err(e),
|
|
CacheImageState::Loaded(image) => {
|
|
let block = image.block(data)?.to_vec();
|
|
image.apply(&block, data)
|
|
}
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
/// The file's bytes with the image at `loc` applied.
|
|
fn apply_cache_image(
|
|
data: &[u8],
|
|
loc: CacheImageLocation,
|
|
sb: &Superblock,
|
|
) -> Result<Vec<u8>, FormatError> {
|
|
let image = CacheImage::decode(data, loc, sb)?;
|
|
let mut out = data.to_vec();
|
|
image.apply(image.block(data)?, &mut out)?;
|
|
Ok(out)
|
|
}
|
|
|
|
fn sb_v2(ext: u64) -> Superblock {
|
|
Superblock {
|
|
version: 2,
|
|
offset_size: 8,
|
|
length_size: 8,
|
|
base_address: 0,
|
|
eof_address: 0,
|
|
root_group_address: 0,
|
|
group_leaf_node_k: None,
|
|
group_internal_node_k: None,
|
|
indexed_storage_internal_node_k: None,
|
|
free_space_address: None,
|
|
driver_info_address: None,
|
|
consistency_flags: 0,
|
|
superblock_extension_address: Some(ext),
|
|
checksum: None,
|
|
page_size: None,
|
|
}
|
|
}
|
|
|
|
/// A file whose superblock extension (a version 1 object header at 48)
|
|
/// holds the given messages, padded to `len` bytes.
|
|
fn file_with_ext(messages: &[(u16, &[u8])], len: usize) -> Vec<u8> {
|
|
let mut body = Vec::new();
|
|
for (t, d) in messages {
|
|
let padded = d.len().div_ceil(8) * 8;
|
|
body.extend_from_slice(&t.to_le_bytes());
|
|
body.extend_from_slice(&(padded as u16).to_le_bytes());
|
|
body.extend_from_slice(&[0x14, 0, 0, 0]);
|
|
body.extend_from_slice(d);
|
|
body.resize(body.len() + padded - d.len(), 0);
|
|
}
|
|
let mut f = vec![0u8; 48];
|
|
f.push(1);
|
|
f.push(0);
|
|
f.extend_from_slice(&(messages.len() as u16).to_le_bytes());
|
|
f.extend_from_slice(&1u32.to_le_bytes());
|
|
f.extend_from_slice(&(body.len() as u32).to_le_bytes());
|
|
f.extend_from_slice(&[0; 4]);
|
|
f.extend_from_slice(&body);
|
|
f.resize(len, 0);
|
|
f
|
|
}
|
|
|
|
fn fsinfo_v1(page_size: u64, persist: bool, n_addrs: usize) -> Vec<u8> {
|
|
let mut m = vec![1, 1, u8::from(persist)];
|
|
m.extend_from_slice(&1u64.to_le_bytes());
|
|
m.extend_from_slice(&page_size.to_le_bytes());
|
|
m.extend_from_slice(&0u16.to_le_bytes());
|
|
m.extend_from_slice(&u64::MAX.to_le_bytes());
|
|
for _ in 0..n_addrs {
|
|
m.extend_from_slice(&u64::MAX.to_le_bytes());
|
|
}
|
|
m
|
|
}
|
|
|
|
fn mdci(address: u64, length: u64) -> Vec<u8> {
|
|
let mut m = vec![0];
|
|
m.extend_from_slice(&address.to_le_bytes());
|
|
m.extend_from_slice(&length.to_le_bytes());
|
|
m
|
|
}
|
|
|
|
#[test]
|
|
fn no_extension() {
|
|
assert_eq!(
|
|
read_superblock_extension(&[0; 64], &sb_v2(u64::MAX)).unwrap(),
|
|
None
|
|
);
|
|
}
|
|
|
|
#[test]
|
|
fn file_space_info_as_libhdf5_decodes_it() {
|
|
// What FileWriter::with_page_size writes.
|
|
let f = file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, false, 0))], 256);
|
|
let ext = read_superblock_extension(&f, &sb_v2(48)).unwrap().unwrap();
|
|
assert_eq!(ext.file_space_info.unwrap().page_size, 4096);
|
|
let f = file_with_ext(&[(MSG_FSINFO, &fsinfo_v1(4096, true, 12))], 512);
|
|
assert!(read_superblock_extension(&f, &sb_v2(48)).is_ok());
|
|
|
|
let refused = |m: Vec<u8>| {
|
|
let f = file_with_ext(&[(MSG_FSINFO, &m)], 512);
|
|
read_superblock_extension(&f, &sb_v2(48)).unwrap_err()
|
|
};
|
|
// Persisting, but too short for the manager addresses.
|
|
let mut short = fsinfo_v1(4096, true, 12);
|
|
short.truncate(short.len() - 8);
|
|
assert_eq!(refused(short), ext_err(RAN_OFF));
|
|
assert!(matches!(
|
|
refused(fsinfo_v1(256, false, 0)),
|
|
FormatError::InvalidSuperblockExtension(_)
|
|
));
|
|
assert!(matches!(
|
|
refused(fsinfo_v1(0, false, 0)),
|
|
FormatError::InvalidSuperblockExtension(_)
|
|
));
|
|
let mut v2 = fsinfo_v1(4096, false, 0);
|
|
v2[0] = 2;
|
|
assert!(matches!(
|
|
refused(v2),
|
|
FormatError::InvalidSuperblockExtension(_)
|
|
));
|
|
// cve-2020-10810: version 0, strategy ALL_PERSIST, and a message of
|
|
// 32 bytes that cannot hold the six addresses that follow.
|
|
let mut v0 = vec![0u8, 1];
|
|
v0.extend_from_slice(&[0, 1, 0, 0, 0, 0, 0, 0]);
|
|
v0.resize(32, 0xff);
|
|
assert_eq!(refused(v0), ext_err(RAN_OFF));
|
|
// A version 0 message without persistence is fine.
|
|
let mut v0 = vec![0u8, 2];
|
|
v0.extend_from_slice(&[0; 8]);
|
|
let f = file_with_ext(&[(MSG_FSINFO, &v0)], 256);
|
|
assert!(read_superblock_extension(&f, &sb_v2(48)).is_ok());
|
|
}
|
|
|
|
#[test]
|
|
fn cache_image_location_must_be_inside_the_file() {
|
|
let f = file_with_ext(&[(MSG_MDCI, &mdci(128, 64))], 192);
|
|
let ext = read_superblock_extension(&f, &sb_v2(48)).unwrap().unwrap();
|
|
assert_eq!(
|
|
ext.cache_image,
|
|
Some(CacheImageLocation {
|
|
address: 128,
|
|
length: 64
|
|
})
|
|
);
|
|
// cve-2020-10812: 256 MiB at 0x10100 in a 2565-byte file.
|
|
let f = file_with_ext(&[(MSG_MDCI, &mdci(0x10100, 0x1000_0000))], 2565);
|
|
assert!(matches!(
|
|
read_superblock_extension(&f, &sb_v2(48)),
|
|
Err(FormatError::InvalidSuperblockExtension(_))
|
|
));
|
|
let f = file_with_ext(&[(MSG_MDCI, &mdci(u64::MAX, 8))], 256);
|
|
assert!(read_superblock_extension(&f, &sb_v2(48)).is_err());
|
|
}
|
|
|
|
/// A cache image block with `entries` of (address, bytes).
|
|
fn image(entries: &[(u64, &[u8])]) -> Vec<u8> {
|
|
let with_deps: Vec<_> = entries.iter().map(|&(a, b)| (a, b, 0, None)).collect();
|
|
image_with_deps(&with_deps)
|
|
}
|
|
|
|
/// A cache image block with `entries` of (address, bytes, flush
|
|
/// dependency children, flush dependency parent).
|
|
fn image_with_deps(entries: &[(u64, &[u8], u16, Option<u64>)]) -> Vec<u8> {
|
|
let mut b = Vec::new();
|
|
b.extend_from_slice(MDCI_SIGNATURE);
|
|
b.push(0);
|
|
b.push(0);
|
|
b.extend_from_slice(&0u64.to_le_bytes()); // length, patched below
|
|
b.extend_from_slice(&(entries.len() as u32).to_le_bytes());
|
|
for &(addr, bytes, children, parent) in entries {
|
|
let mut flags = 0x02; // in LRU
|
|
if children > 0 {
|
|
flags |= MDCI_ENTRY_IS_FD_PARENT;
|
|
}
|
|
if parent.is_some() {
|
|
flags |= MDCI_ENTRY_IS_FD_CHILD;
|
|
}
|
|
b.extend_from_slice(&[5, flags, 1, 0]); // type, flags, ring, age
|
|
b.extend_from_slice(&children.to_le_bytes());
|
|
b.extend_from_slice(&0u16.to_le_bytes()); // dirty children
|
|
b.extend_from_slice(&u16::from(parent.is_some()).to_le_bytes());
|
|
b.extend_from_slice(&0i32.to_le_bytes());
|
|
b.extend_from_slice(&addr.to_le_bytes());
|
|
b.extend_from_slice(&(bytes.len() as u64).to_le_bytes());
|
|
if let Some(p) = parent {
|
|
b.extend_from_slice(&p.to_le_bytes());
|
|
}
|
|
b.extend_from_slice(bytes);
|
|
}
|
|
b.extend_from_slice(&[0; 4]); // checksum (not verified, as in libhdf5)
|
|
let n = b.len() as u64;
|
|
b[6..14].copy_from_slice(&n.to_le_bytes());
|
|
b
|
|
}
|
|
|
|
#[test]
|
|
fn cache_image_entries_replace_the_file_bytes() {
|
|
let img = image(&[(16, b"HEADER"), (40, b"NODE")]);
|
|
let mut f = vec![0u8; 64];
|
|
let at = f.len() as u64;
|
|
f.extend_from_slice(&img);
|
|
let loc = CacheImageLocation {
|
|
address: at,
|
|
length: img.len() as u64,
|
|
};
|
|
let out = apply_cache_image(&f, loc, &sb_v2(u64::MAX)).unwrap();
|
|
assert_eq!(out.len(), f.len());
|
|
assert_eq!(&out[16..22], b"HEADER");
|
|
assert_eq!(&out[40..44], b"NODE");
|
|
assert_eq!(&out[..16], &f[..16]);
|
|
|
|
let bad = |img: Vec<u8>| {
|
|
let mut f = vec![0u8; 64];
|
|
f.extend_from_slice(&img);
|
|
let loc = CacheImageLocation {
|
|
address: 64,
|
|
length: img.len() as u64,
|
|
};
|
|
apply_cache_image(&f, loc, &sb_v2(u64::MAX)).unwrap_err()
|
|
};
|
|
let mut sig = image(&[(16, b"x")]);
|
|
sig[0] = b'X';
|
|
assert!(matches!(bad(sig), FormatError::InvalidCacheImage(_)));
|
|
assert!(matches!(
|
|
bad(image(&[(16, b"a"), (16, b"b")])),
|
|
FormatError::InvalidCacheImage("duplicate addresses in cache")
|
|
));
|
|
assert!(matches!(
|
|
bad(image(&[(1 << 20, b"far")])),
|
|
FormatError::InvalidCacheImage("invalid entry address range")
|
|
));
|
|
let mut len = image(&[(16, b"x")]);
|
|
len[6] ^= 1;
|
|
assert!(matches!(bad(len), FormatError::InvalidCacheImage(_)));
|
|
// An entry that starts inside the file (64 bytes, then a 60-byte
|
|
// image) but runs past its end.
|
|
assert!(matches!(
|
|
bad(image(&[(123, b"8 bytes!")])),
|
|
FormatError::InvalidCacheImage("entry extends past the end of file")
|
|
));
|
|
let mut cut = image(&[(16, b"abcdef")]);
|
|
let n = cut.len() as u64 - 8;
|
|
cut.truncate(cut.len() - 8);
|
|
cut[6..14].copy_from_slice(&n.to_le_bytes());
|
|
assert!(matches!(bad(cut), FormatError::InvalidCacheImage(_)));
|
|
}
|
|
|
|
/// libhdf5 resolves an entry's flush-dependency parents as it inserts
|
|
/// the entry (`H5C__reconstruct_cache_contents`): a parent must be an
|
|
/// earlier entry, or the superblock or its extension's object header,
|
|
/// which are cached before the image loads. A parent listed after its
|
|
/// child fails ("fd parent not in cache?!?").
|
|
#[test]
|
|
fn flush_dependency_parents_must_already_be_cached() {
|
|
let load = |img: Vec<u8>| {
|
|
let mut f = vec![0u8; 64];
|
|
f.extend_from_slice(&img);
|
|
let loc = CacheImageLocation {
|
|
address: 64,
|
|
length: img.len() as u64,
|
|
};
|
|
apply_cache_image(&f, loc, &sb_v2(48))
|
|
};
|
|
// Parent first, as libhdf5 writes images.
|
|
assert!(
|
|
load(image_with_deps(&[
|
|
(16, b"P", 1, None),
|
|
(40, b"C", 0, Some(16))
|
|
]))
|
|
.is_ok()
|
|
);
|
|
// Child first: libhdf5 does not find the parent.
|
|
assert_eq!(
|
|
load(image_with_deps(&[
|
|
(40, b"C", 0, Some(16)),
|
|
(16, b"P", 1, None)
|
|
]))
|
|
.unwrap_err(),
|
|
FormatError::InvalidCacheImage("fd parent not in cache")
|
|
);
|
|
// The superblock extension's header (at 48 here) is in the cache.
|
|
assert!(load(image_with_deps(&[(40, b"C", 0, Some(48))])).is_ok());
|
|
// An entry cannot be its own parent.
|
|
assert!(load(image_with_deps(&[(40, b"C", 1, Some(40))])).is_err());
|
|
}
|
|
}
|