quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases 1 Wiki Activity

Merge branch 'fix/p1-vds' into fix/p1-read-gaps

Browse Source 
# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-format/src/data_read.rs
This commit is contained in:
osobh
2026-09-25 22:42:04 -05:00
parent 61f36516d7 b4a44a2e66
commit f2e704abf3
23 changed files with 2321 additions and 287 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+41
View File
@@ -331,6 +331,47 @@
The legacy per-member dimension fields are now decoded into an array type, The legacy per-member dimension fields are now decoded into an array type,
as libhdf5 does; more than four dimensions, or a zero-sized one, is an as libhdf5 does; more than four dimensions, or a zero-sized one, is an
error. error.
- `clawhdf5-format` virtual datasets (VDS), checked against HDF5 2.0 through
h5py (`crates/clawhdf5/tests/vds_interop.rs`):
- **Wrong data:** elements no mapping supplies — unmapped regions, and
mappings whose source file or dataset is missing — read as 0 instead of
the virtual dataset's fill value (e.g. h5py `fillvalue=-1`). Assembly moved
to the new `vds` module: `vds::read_virtual_dataset` takes the fill value
and a resolver that can refuse a name (`VdsFileResolver`), and `File`
passes the dataset's fill value. A missing source *dataset* read as an
error; it is fill now, as in libhdf5. Source datasets are read with their
own fill value for unallocated chunks, and a source whose datatype differs
from the virtual dataset's is an error (libhdf5 converts; we do not).
`File` now refuses a source name that leaves the virtual file's directory
(`../x.h5`, absolute paths), or any external source of a `File::from_bytes`
file, with an error — these used to read as fill.
**Behaviour change:** the raw-read API (`read_raw_data_full*`), which has
no fill value, now returns an error for a virtual dataset with unmapped
elements instead of zeros.
- Unlimited and printf-style mappings are supported (all 7 VDS files in the
libhdf5 test set are such mappings, e.g. Eiger/Percival detector layouts).
`%b` in a source file or dataset name is the block number and `%%` a
literal `%` (other `%` sequences are an error, as in libhdf5); block `j`
is read from the source named with `j`, probing from 0 up to the first
missing source. Unlimited source/virtual selections cover as much as the
source's current extent fills, including a partial last block. As
libhdf5 does on `H5Dget_space`, the extent is recomputed from the sources
present (default "last available" view, printf gap 0) —
`vds::virtual_dataset_extent`, used by `Dataset::shape()` — so e.g.
`vds-eiger.h5` is `[5, 10, 10]`, not its stored `[20, 10, 10]`. A source
stored in the other byte order is byte-swapped (libhdf5 converts);
other type conversions remain an error.
- Hyperslab selection versions 1 and 2 were refused ("only version-3
hyperslab selections are supported"). Version 1 is what libhdf5 writes for
every VDS created with the default format bounds (h5py's default), so
those could not be read at all; version 2 is its encoding of an unlimited
selection. Both are decoded now, as are irregular hyperslabs (a union of
blocks, read in row-major order as libhdf5 iterates them).
`SerializedSelection` exposes the raw form, including unlimited counts.
- The version-1 mapping list HDF5 2.0 writes (low version bound 2.0) was
misparsed: each entry's flags byte was read as the start of the source
file name, and names shared with an earlier entry (stored as that entry's
index) were not followed. Now decoded as `H5D__virtual_load_layout` does.
- `clawhdf5-format` reader — **values returned wrong with no error:** - `clawhdf5-format` reader — **values returned wrong with no error:**
- Fixed Array and Extensible Array chunk indexes were laid out by the - Fixed Array and Extensible Array chunk indexes were laid out by the
dataset's current shape instead of its max shape (23 libhdf5 test files, dataset's current shape instead of its max shape (23 libhdf5 test files,
crates/clawhdf5-format/src/data_layout.rs
+120 -12
View File
@@ -74,21 +74,33 @@ pub enum DataLayout {
}, },
} }
/// Version-1 VDS mapping flag: the source file name is stored by an earlier
/// entry, whose index follows in place of the name.
const VDS_SOURCE_FILE_SHARED: u8 = 0x01;
/// Version-1 VDS mapping flag: likewise for the source dataset name.
const VDS_SOURCE_DSET_SHARED: u8 = 0x02;
/// Version-1 VDS mapping flag: the source is in the virtual file itself
/// (`"."`); no file name is stored.
const VDS_SOURCE_SAME_FILE: u8 = 0x04;
const VDS_ALL_FLAGS: u8 = VDS_SOURCE_FILE_SHARED | VDS_SOURCE_DSET_SHARED | VDS_SOURCE_SAME_FILE;
/// Parse VDS mappings from global-heap object data. /// Parse VDS mappings from global-heap object data.
/// ///
/// The global-heap block holding a VDS mapping list is laid out as /// The global-heap block holding a VDS mapping list is laid out as
/// (reverse-engineered and validated against HDF5 2.0): /// (`H5D__virtual_store_layout` / `H5D__virtual_load_layout` in libhdf5):
/// ///
/// ```text /// ```text
/// version(1) · nused(length_size, LE) · entry[nused] · checksum(4) /// version(1) · nused(length_size, LE) · entry[nused] · checksum(4)
/// ``` /// ```
/// ///
/// Each entry is: /// Each entry is:
/// - source file name — a null-terminated string in **block version 0**; in /// - **block version 1 only:** a flags byte. `0x04`: the source is in the
/// **block version 1** a same-file reference is encoded as a single `0x04` /// virtual file itself and no file name is stored; `0x01`/`0x02`: the
/// marker byte (the source file is the virtual file itself) in place of the /// source file/dataset name is that of an earlier entry, whose index
/// name; /// (`length_size` bytes) is stored instead of the name. libhdf5 2.0 writes
/// - source dataset name (null-terminated string); /// version 1 when the file's low version bound is 2.0 and it saves space;
/// - source file name (null-terminated string, unless flagged above);
/// - source dataset name (null-terminated string, unless flagged above);
/// - source selection (serialized `H5S` dataspace selection — self-describing /// - source selection (serialized `H5S` dataspace selection — self-describing
/// in length); /// in length);
/// - virtual selection (serialized `H5S` dataspace selection). /// - virtual selection (serialized `H5S` dataspace selection).
@@ -114,7 +126,7 @@ pub fn parse_vds_mappings(
// `nused` is untrusted; don't pre-allocate from it. Each entry consumes at // `nused` is untrusted; don't pre-allocate from it. Each entry consumes at
// least a few bytes, so the loop is naturally bounded by the heap data and // least a few bytes, so the loop is naturally bounded by the heap data and
// a bogus `nused` simply errors out on the first short read. // a bogus `nused` simply errors out on the first short read.
let mut mappings = Vec::new(); let mut mappings: Vec<VdsMapping> = Vec::new();
// Reads one self-describing selection at `pos`, returning its raw bytes and // Reads one self-describing selection at `pos`, returning its raw bytes and
// advancing past it — bounds-checked so a corrupt selection can't overrun. // advancing past it — bounds-checked so a corrupt selection can't overrun.
let read_selection = |heap_data: &[u8], pos: &mut usize| -> Result<Vec<u8>, FormatError> { let read_selection = |heap_data: &[u8], pos: &mut usize| -> Result<Vec<u8>, FormatError> {
@@ -134,17 +146,57 @@ pub fn parse_vds_mappings(
Ok(bytes) Ok(bytes)
}; };
for _ in 0..nused { if version > 1 {
// Source file name (with the version-1 same-file marker handled). return Err(FormatError::ChunkedReadError(
let source_file = if version >= 1 && heap_data.get(pos) == Some(&0x04) { "unsupported VDS mapping block version".into(),
));
}
for i in 0..nused {
// Version 1 prefixes each entry with a flags byte; a name may then be
// omitted (same file) or replaced by the index of an earlier entry
// holding the same name (`H5D__virtual_load_layout`).
let flags = if version >= 1 {
let f = *heap_data.get(pos).ok_or(FormatError::UnexpectedEof {
expected: pos + 1,
available: heap_data.len(),
})?;
pos += 1; pos += 1;
if f & !VDS_ALL_FLAGS != 0 {
return Err(FormatError::ChunkedReadError(
"unknown VDS mapping flags".into(),
));
}
f
} else {
0
};
// Index of an earlier entry, for a shared name.
let earlier = |pos: &mut usize| -> Result<usize, FormatError> {
let idx = read_length(heap_data, *pos, length_size)?;
*pos += ls;
if idx >= i {
return Err(FormatError::ChunkedReadError(
"VDS mapping shares a name with a later entry".into(),
));
}
Ok(idx as usize)
};
let source_file = if flags & VDS_SOURCE_SAME_FILE != 0 {
String::from(".") String::from(".")
} else if flags & VDS_SOURCE_FILE_SHARED != 0 {
let idx = earlier(&mut pos)?;
mappings[idx].source_file.clone()
} else { } else {
read_null_terminated_string(heap_data, &mut pos)? read_null_terminated_string(heap_data, &mut pos)?
}; };
// Source dataset name. let source_dataset = if flags & VDS_SOURCE_DSET_SHARED != 0 {
let source_dataset = read_null_terminated_string(heap_data, &mut pos)?; let idx = earlier(&mut pos)?;
mappings[idx].source_dataset.clone()
} else {
read_null_terminated_string(heap_data, &mut pos)?
};
// Source selection, then virtual selection (both self-describing length). // Source selection, then virtual selection (both self-describing length).
let source_selection = read_selection(heap_data, &mut pos)?; let source_selection = read_selection(heap_data, &mut pos)?;
@@ -1035,6 +1087,62 @@ mod tests {
assert_eq!(v1.iter_linear_1d(8).unwrap(), vec![4, 5, 6, 7]); assert_eq!(v1.iter_linear_1d(8).unwrap(), vec![4, 5, 6, 7]);
} }
#[test]
fn parse_vds_mappings_v1_shared_names() {
// Written by HDF5 2.0 (h5py, libver=("v200", "v200")) for three
// mappings from `a_rather_long_source_file.h5:a_rather_long_dataset_name`
// and one from the same file: the entries carry flags 0x00, 0x03, 0x03
// and 0x06, so names after the first are stored as entry indices.
let blob: &[u8] = &[
0x01, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x61, 0x5f, 0x72, 0x61,
0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x73, 0x6f, 0x75, 0x72,
0x63, 0x65, 0x5f, 0x66, 0x69, 0x6c, 0x65, 0x2e, 0x68, 0x35, 0x00, 0x61, 0x5f, 0x72,
0x61, 0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x64, 0x61, 0x74,
0x61, 0x73, 0x65, 0x74, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x00, 0x02, 0x00, 0x00, 0x00,
0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00,
0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02,
0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00,
0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03,
0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x04, 0x00, 0x01, 0x00, 0x01,
0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02,
0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00,
0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x00, 0x01, 0x00,
0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02, 0x00,
0x00, 0x00, 0x02, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00,
0x01, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02,
0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00,
0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00,
0x00, 0x01, 0x02, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01,
0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x8e, 0xa7, 0xea, 0x7a,
];
let mappings = parse_vds_mappings(blob, 8).unwrap();
let names: Vec<(&str, &str)> = mappings
.iter()
.map(|m| (m.source_file.as_str(), m.source_dataset.as_str()))
.collect();
let (file, dset) = ("a_rather_long_source_file.h5", "a_rather_long_dataset_name");
assert_eq!(
names,
vec![(file, dset), (file, dset), (file, dset), (".", dset)]
);
}
#[test]
fn parse_vds_mappings_v1_forward_reference_is_error() {
// Entry 0 claiming to share entry 0's file name must not index past
// the entries decoded so far.
let mut blob = vec![0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x01];
blob.extend_from_slice(&[0u8; 8]);
blob.extend_from_slice(b"d\0");
assert!(parse_vds_mappings(&blob, 8).is_err());
// Unknown flag bits are refused.
let blob = [0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x08, b'd', 0];
assert!(parse_vds_mappings(&blob, 8).is_err());
}
#[test] #[test]
fn parse_vds_mappings_external_v0() { fn parse_vds_mappings_external_v0() {
// Block version 0 with an explicit (external) source file name. // Block version 0 with an explicit (external) source file name.
crates/clawhdf5-format/src/data_read.rs
+36 -147
View File
@@ -191,14 +191,9 @@ fn read_raw_data_full_impl(
offset_size, offset_size,
length_size, length_size,
), ),
DataLayout::Virtual { DataLayout::Virtual { .. } => read_virtual_data(
global_heap_address,
global_heap_index,
..
} => read_virtual_data(
file_data, file_data,
*global_heap_address, layout,
*global_heap_index,
dataspace, dataspace,
datatype, datatype,
offset_size, offset_size,
@@ -465,160 +460,54 @@ pub fn read_raw_data_selection(
} }
} }
/// Assemble a **Virtual Dataset (VDS)** from its source mappings. /// Assemble a **Virtual Dataset (VDS)** through the raw-read API, which has no
/// access to the dataset's fill value message.
/// ///
/// Supports virtual datasets of any rank. Same-file sources are read directly; /// Delegates to [`crate::vds::read_virtual_dataset`]. Because the fill value
/// **external-file** sources are read through the caller-supplied `resolver`, /// is unknown here, a virtual dataset with any element no mapping supplies
/// which maps a stored source file name to that file's bytes. Each mapping's /// (an unmapped region, or a missing source file or dataset) is an error
/// selected source elements are scattered into the virtual buffer at the /// rather than a guess at the fill value; so is one whose extent libhdf5
/// positions given by the virtual selection (both enumerated in row-major /// would report differently from the stored dataspace (unlimited mappings).
/// order, as HDF5 pairs them). Unmapped regions are left at the zero fill value. /// Use [`crate::vds::read_virtual_dataset`] to read those.
///
/// A mapping whose external source file the resolver cannot supply (`None`) is
/// skipped, leaving its region at fill — matching HDF5's tolerance of missing
/// sources. An external source with no resolver at all is a hard error.
#[allow(clippy::too_many_arguments)] #[allow(clippy::too_many_arguments)]
fn read_virtual_data( fn read_virtual_data(
file_data: &[u8], file_data: &[u8],
global_heap_address: Option<u64>, layout: &DataLayout,
global_heap_index: u32,
dataspace: &Dataspace, dataspace: &Dataspace,
datatype: &Datatype, datatype: &Datatype,
offset_size: u8, offset_size: u8,
length_size: u8, length_size: u8,
resolver: Option<&VdsSourceResolver>, resolver: Option<&VdsSourceResolver>,
) -> Result<Vec<u8>, FormatError> { ) -> Result<Vec<u8>, FormatError> {
use crate::data_layout::parse_vds_mappings; let wrapped =
use crate::global_heap::GlobalHeapCollection; resolver.map(|r| move |name: &str| -> Result<Option<Vec<u8>>, FormatError> { Ok(r(name)) });
use crate::selection::Selection; let wrapped_ref = wrapped.as_ref().map(|w| w as &crate::vds::VdsFileResolver);
let v = crate::vds::read_virtual_dataset(
let elem_size = datatype.type_size() as usize; file_data,
let mut out = crate::chunked_read::alloc_output(crate::chunked_read::checked_byte_len( layout,
dataspace.checked_num_elements()?, dataspace,
elem_size, datatype,
)?)?; None,
offset_size,
let virtual_dims = &dataspace.dimensions; length_size,
wrapped_ref,
let addr = global_heap_address.ok_or_else(|| { )?;
FormatError::ChunkedReadError("virtual dataset has no mapping global heap".into()) if v.dims != dataspace.dimensions {
})?;
let coll = GlobalHeapCollection::parse(file_data, addr as usize, length_size)?;
let obj =
coll.get_object(global_heap_index as u16)
.ok_or(FormatError::GlobalHeapObjectNotFound {
collection_address: addr,
index: global_heap_index as u16,
})?;
let mappings = parse_vds_mappings(&obj.data, length_size)?;
for m in &mappings {
let same_file = m.source_file.is_empty() || m.source_file == ".";
// Resolve the bytes of the file holding this source dataset.
let external;
let src_file_data: &[u8] = if same_file {
file_data
} else {
let r = resolver.ok_or_else(|| {
FormatError::ChunkedReadError(
"external-file virtual dataset sources require a file resolver".into(),
)
})?;
match r(&m.source_file) {
Some(bytes) => {
external = bytes;
&external
}
// Source file unavailable: leave this region at fill value.
None => continue,
}
};
let (vsel, _) = Selection::decode_serialized(&m.virtual_selection)?;
let (ssel, _) = Selection::decode_serialized(&m.source_selection)?;
let (src_raw, src_dims) =
read_named_dataset_raw(src_file_data, &m.source_dataset, offset_size, length_size)?;
let vidx = vsel.iter_linear(virtual_dims)?;
let sidx = ssel.iter_linear(&src_dims)?;
if vidx.len() != sidx.len() {
return Err(FormatError::ChunkedReadError(
"virtual/source selection element counts differ".into(),
));
}
for (&v, &s) in vidx.iter().zip(sidx.iter()) {
let (vo, so) = (v as usize * elem_size, s as usize * elem_size);
if vo + elem_size > out.len() || so + elem_size > src_raw.len() {
return Err(FormatError::ChunkedReadError(
"virtual dataset selection out of bounds".into(),
));
}
out[vo..vo + elem_size].copy_from_slice(&src_raw[so..so + elem_size]);
}
}
Ok(out)
}
/// Read a named dataset's raw (decoded) bytes and its dimensions, navigating
/// from the superblock. Used to pull VDS source datasets out of the same file.
fn read_named_dataset_raw(
file_data: &[u8],
path: &str,
_offset_size: u8,
_length_size: u8,
) -> Result<(Vec<u8>, Vec<u64>), FormatError> {
use crate::filter_pipeline::FilterPipeline;
use crate::group_v2::resolve_path_any;
use crate::message_type::MessageType;
use crate::object_header::ObjectHeader;
use crate::signature::split_user_block;
use crate::superblock::Superblock;
// An external source file is handed over whole, user block included;
// its addresses are relative to its superblock.
let (_, file_data) = split_user_block(file_data)?;
let sb = Superblock::parse(file_data, 0)?;
let addr = resolve_path_any(file_data, &sb, path)?;
let hdr = ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size)?;
let find = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t);
let ds_msg = find(MessageType::Dataspace)
.ok_or_else(|| FormatError::ChunkedReadError("VDS source has no dataspace".into()))?;
let dataspace = Dataspace::parse(&ds_msg.data, sb.length_size)?;
let dt_msg = find(MessageType::Datatype)
.ok_or_else(|| FormatError::ChunkedReadError("VDS source has no datatype".into()))?;
let (datatype, _) = Datatype::parse(&dt_msg.data)?;
let dl_msg = find(MessageType::DataLayout)
.ok_or_else(|| FormatError::ChunkedReadError("VDS source has no data layout".into()))?;
let layout = DataLayout::parse(&dl_msg.data, sb.offset_size, sb.length_size)?;
// A virtual dataset whose source is itself another virtual dataset could
// form a cycle (A -> B -> A) and recurse into a stack overflow. Nested
// virtual sources are exotic and unsupported, so stop here cleanly.
if matches!(layout, DataLayout::Virtual { .. }) {
return Err(FormatError::ChunkedReadError( return Err(FormatError::ChunkedReadError(
"virtual dataset source is itself virtual (unsupported)".into(), "virtual dataset extent differs from its stored dataspace; \
read it with vds::read_virtual_dataset"
.into(),
)); ));
} }
let pipeline = find(MessageType::FilterPipeline) if v.unmapped > 0 {
.map(|m| FilterPipeline::parse(&m.data)) return Err(FormatError::ChunkedReadError(
.transpose()?; "virtual dataset has elements no source supplies, which read as its \
fill value; read it with vds::read_virtual_dataset and the fill value"
let raw = read_raw_data_full( .into(),
file_data, ));
&layout, }
&dataspace, Ok(v.data)
&datatype,
pipeline.as_ref(),
sb.offset_size,
sb.length_size,
)?;
Ok((raw, dataspace.dimensions.clone()))
} }
/// Extract selected elements from a full dataset buffer. /// Extract selected elements from a full dataset buffer.
pub fn extract_selection_from_buffer( pub fn extract_selection_from_buffer(
full_data: &[u8], full_data: &[u8],
crates/clawhdf5-format/src/lib.rs
+1
View File
@@ -101,6 +101,7 @@ pub mod signature;
pub mod superblock; pub mod superblock;
pub mod symbol_table; pub mod symbol_table;
pub mod type_builders; pub mod type_builders;
pub mod vds;
pub mod vl_data; pub mod vl_data;
#[cfg(feature = "provenance")] #[cfg(feature = "provenance")]
crates/clawhdf5-format/src/selection.rs
+395 -103
View File
@@ -229,44 +229,47 @@ impl Selection {
/// self-describing in length, so the count lets a caller walk a packed list /// self-describing in length, so the count lets a caller walk a packed list
/// of selections — as the Virtual Dataset global-heap block does). /// of selections — as the Virtual Dataset global-heap block does).
/// ///
/// Only the forms needed for VDS assembly are decoded: `ALL`, `NONE`, and /// Decodes `ALL`, `NONE`, and hyperslabs at every version libhdf5 writes
/// **regular** hyperslabs serialized at **version 3** (the encoding HDF5 /// (1: irregular, 4-byte coordinates — the default-format encoding; 2:
/// 1.10+/2.0 emit). Point selections, irregular hyperslabs, and older /// regular, 8-byte; 3: either, variable width). A regular hyperslab maps
/// hyperslab versions return an error rather than mis-decoding. /// to [`Selection::Hyperslab`]; an *irregular* one (a union of blocks)
/// maps to a single-block hyperslab when it has one block, and otherwise to
/// [`Selection::Points`] listing the union in row-major order (the order
/// libhdf5 iterates it in). Unlimited counts/blocks decode as `u64::MAX`
/// (see [`SerializedSelection::decode`] for the raw form). Point
/// selections are refused: libhdf5 does not allow them in virtual datasets
/// either.
pub fn decode_serialized(data: &[u8]) -> Result<(Selection, usize), FormatError> { pub fn decode_serialized(data: &[u8]) -> Result<(Selection, usize), FormatError> {
if data.len() < 8 { let (raw, len) = SerializedSelection::decode(data)?;
return Err(FormatError::UnexpectedEof { let sel = match raw {
expected: 8, SerializedSelection::All => Selection::All,
available: data.len(), SerializedSelection::None => Selection::None,
}); SerializedSelection::Regular {
} start,
let sel_type = u32::from_le_bytes([data[0], data[1], data[2], data[3]]); stride,
let version = u32::from_le_bytes([data[4], data[5], data[6], data[7]]); count,
block,
match sel_type { } => Selection::Hyperslab {
// ALL / NONE: type(4) + version(4) + reserved(4) + length(4) = 16 bytes. start,
3 | 0 => { stride,
if data.len() < 16 { count,
return Err(FormatError::UnexpectedEof { block,
expected: 16, },
available: data.len(), SerializedSelection::Blocks { rank, starts, ends } => {
}); if starts.len() == rank {
} let block = starts.iter().zip(&ends).map(|(&s, &e)| e - s + 1).collect();
let sel = if sel_type == 3 { Selection::Hyperslab {
Selection::All start: starts,
stride: vec![1; rank],
count: vec![1; rank],
block,
}
} else { } else {
Selection::None Selection::Points(blocks_union_coords(rank, &starts, &ends)?)
}; }
Ok((sel, 16))
} }
2 => decode_hyperslab_serialized(data, version), };
1 => Err(FormatError::ChunkedReadError( Ok((sel, len))
"VDS point selections are not supported".into(),
)),
_ => Err(FormatError::ChunkedReadError(
"unknown dataspace selection type".into(),
)),
}
} }
/// Enumerate the selected element indices of a **1-D** dataspace of the /// Enumerate the selected element indices of a **1-D** dataspace of the
@@ -314,6 +317,11 @@ impl Selection {
"VDS selection rank does not match dataspace rank".into(), "VDS selection rank does not match dataspace rank".into(),
)); ));
} }
if count.iter().chain(block.iter()).any(|&v| v == UNLIMITED) {
return Err(FormatError::ChunkedReadError(
"unlimited selection must be clipped before it is enumerated".into(),
));
}
// Selected coordinates along each dimension, in order. // Selected coordinates along each dimension, in order.
let mut per_dim: Vec<Vec<u64>> = Vec::with_capacity(rank); let mut per_dim: Vec<Vec<u64>> = Vec::with_capacity(rank);
for d in 0..rank { for d in 0..rank {
@@ -400,84 +408,279 @@ impl Selection {
} }
} }
/// Decode an `H5S_SEL_HYPER` selection in its serialized form. Only version-3 /// Hyperslab count/block value meaning "unlimited" (`H5S_UNLIMITED`).
/// **regular** hyperslabs are supported. pub const UNLIMITED: u64 = u64::MAX;
fn decode_hyperslab_serialized(
data: &[u8], /// Largest number of elements an irregular selection is expanded to when it
version: u32, /// is converted to a point list by [`Selection::decode_serialized`].
) -> Result<(Selection, usize), FormatError> { const MAX_EXPANDED_POINTS: u64 = 1 << 26;
if version != 3 {
return Err(FormatError::ChunkedReadError( /// A selection exactly as `H5S_select_serialize` stores it, before it is
"only version-3 hyperslab selections are supported".into(), /// applied to any dataspace.
)); ///
/// Unlike [`Selection`] this keeps an irregular hyperslab as its list of
/// blocks, and a regular hyperslab's count/block may be [`UNLIMITED`] (the
/// unlimited selections used by unlimited and "printf" virtual dataset
/// mappings).
#[derive(Debug, Clone, PartialEq)]
pub enum SerializedSelection {
/// `H5S_SEL_ALL`.
All,
/// `H5S_SEL_NONE`.
None,
/// A regular hyperslab. `count[d]` or `block[d]` may be [`UNLIMITED`].
Regular {
start: Vec<u64>,
stride: Vec<u64>,
count: Vec<u64>,
block: Vec<u64>,
},
/// An irregular hyperslab: the union of `starts.len() / rank` blocks, each
/// given by its first (`starts`) and last (`ends`, inclusive) coordinate,
/// flattened block-major.
Blocks {
rank: usize,
starts: Vec<u64>,
ends: Vec<u64>,
},
}
fn sel_err(msg: &str) -> FormatError {
FormatError::ChunkedReadError(msg.into())
}
/// Bounds-checked little-endian reader over a serialized selection.
struct SelReader<'a> {
data: &'a [u8],
pos: usize,
}
impl SelReader<'_> {
fn take(&mut self, n: usize) -> Result<&[u8], FormatError> {
let end = self.pos.checked_add(n).filter(|&e| e <= self.data.len());
let end = end.ok_or(FormatError::UnexpectedEof {
expected: self.pos.saturating_add(n),
available: self.data.len(),
})?;
let s = &self.data[self.pos..end];
self.pos = end;
Ok(s)
} }
// type(4) ver(4) flags(1) enc_size(1) rank(4) [start,stride,count,block]*rank
if data.len() < 14 { fn uint(&mut self, size: usize) -> Result<u64, FormatError> {
return Err(FormatError::UnexpectedEof { let bytes = self.take(size)?;
expected: 14, Ok(bytes
available: data.len(), .iter()
}); .enumerate()
.fold(0u64, |v, (i, &b)| v | (b as u64) << (i * 8)))
} }
let flags = data[8];
let enc_size = data[9] as usize; fn remaining(&self) -> usize {
// Bit 0 set => regular hyperslab. Irregular hyperslabs list explicit blocks. self.data.len() - self.pos
if flags & 0x01 == 0 {
return Err(FormatError::ChunkedReadError(
"irregular VDS hyperslab selections are not supported".into(),
));
} }
if enc_size != 2 && enc_size != 4 && enc_size != 8 { }
return Err(FormatError::ChunkedReadError(
"unsupported hyperslab coordinate encoding size".into(), impl SerializedSelection {
)); /// Decode a serialized selection, returning it and the number of bytes it
} /// occupies. Mirrors libhdf5's `H5S_select_deserialize`: `ALL`/`NONE` and
let rank = u32::from_le_bytes([data[10], data[11], data[12], data[13]]) as usize; /// hyperslab versions 1-3 are decoded; point selections (which libhdf5
// HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything larger so a /// refuses in virtual datasets) and malformed input are errors.
// corrupt rank can't drive a huge allocation or read loop. pub fn decode(data: &[u8]) -> Result<(SerializedSelection, usize), FormatError> {
if rank > 32 { let mut r = SelReader { data, pos: 0 };
return Err(FormatError::ChunkedReadError( let sel_type = r.uint(4)?;
"hyperslab selection rank exceeds maximum (32)".into(), let version = r.uint(4)?;
)); match sel_type {
} // ALL / NONE: type(4) + version(4) + reserved(4) + length(4).
let mut pos = 14; 0 | 3 => {
let read_coord = |data: &[u8], pos: usize| -> Result<u64, FormatError> { r.take(8)?;
if pos + enc_size > data.len() { let sel = if sel_type == 3 {
return Err(FormatError::UnexpectedEof { SerializedSelection::All
expected: pos + enc_size, } else {
available: data.len(), SerializedSelection::None
}); };
Ok((sel, r.pos))
}
2 => {
let sel = decode_hyperslab(&mut r, version)?;
Ok((sel, r.pos))
}
1 => Err(sel_err(
"VDS point selections are not supported (libhdf5 rejects them too)",
)),
_ => Err(sel_err("unknown dataspace selection type")),
} }
let mut v = 0u64; }
for (i, &b) in data[pos..pos + enc_size].iter().enumerate() {
v |= (b as u64) << (i * 8); /// The single dimension in which this selection is unlimited, if any.
pub fn unlimited_dim(&self) -> Option<usize> {
match self {
SerializedSelection::Regular { count, block, .. } => count
.iter()
.zip(block)
.position(|(&c, &b)| c == UNLIMITED || b == UNLIMITED),
_ => None,
} }
Ok(v) }
/// The rank the selection was serialized with (`None` for ALL/NONE, which
/// carry no rank).
pub fn rank(&self) -> Option<usize> {
match self {
SerializedSelection::Regular { start, .. } => Some(start.len()),
SerializedSelection::Blocks { rank, .. } => Some(*rank),
_ => None,
}
}
}
/// `H5S__hyper_deserialize`: after the type and version words.
fn decode_hyperslab(r: &mut SelReader, version: u64) -> Result<SerializedSelection, FormatError> {
const REGULAR: u8 = 0x01;
let (flags, enc_size) = match version {
// v1: reserved(4) + length(4), always irregular, 4-byte coordinates.
1 => {
r.take(8)?;
(0u8, 4usize)
}
// v2: flags(1) + length(4), 8-byte coordinates.
2 => {
let flags = r.take(1)?[0];
r.take(4)?;
(flags, 8)
}
// v3: flags(1) + encoding size(1).
3 => {
let flags = r.take(1)?[0];
let enc = r.take(1)?[0] as usize;
(flags, enc)
}
_ => return Err(sel_err("unsupported hyperslab selection version")),
}; };
let (mut start, mut stride, mut count, mut block) = ( if flags & !REGULAR != 0 {
Vec::with_capacity(rank), return Err(sel_err("unknown hyperslab selection flags"));
Vec::with_capacity(rank),
Vec::with_capacity(rank),
Vec::with_capacity(rank),
);
for _ in 0..rank {
start.push(read_coord(data, pos)?);
pos += enc_size;
stride.push(read_coord(data, pos)?);
pos += enc_size;
count.push(read_coord(data, pos)?);
pos += enc_size;
block.push(read_coord(data, pos)?);
pos += enc_size;
} }
Ok(( if !matches!(enc_size, 2 | 4 | 8) {
Selection::Hyperslab { return Err(sel_err("unsupported hyperslab coordinate encoding size"));
}
let rank = r.uint(4)? as usize;
// HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything else so a
// corrupt rank can't drive a huge allocation or read loop.
if rank == 0 || rank > 32 {
return Err(sel_err("hyperslab selection rank must be 1..=32"));
}
// The all-ones value of the encoding width means "unlimited".
let unlim_raw = if enc_size == 8 {
u64::MAX
} else {
(1u64 << (enc_size * 8)) - 1
};
if flags & REGULAR != 0 {
let (mut start, mut stride, mut count, mut block) = (
Vec::with_capacity(rank),
Vec::with_capacity(rank),
Vec::with_capacity(rank),
Vec::with_capacity(rank),
);
for _ in 0..rank {
start.push(r.uint(enc_size)?);
stride.push(r.uint(enc_size)?);
let c = r.uint(enc_size)?;
count.push(if c == unlim_raw { UNLIMITED } else { c });
let b = r.uint(enc_size)?;
block.push(if b == unlim_raw { UNLIMITED } else { b });
}
let unlimited = count
.iter()
.zip(&block)
.filter(|&(&c, &b)| c == UNLIMITED || b == UNLIMITED)
.count();
if unlimited > 1 {
return Err(sel_err(
"hyperslab selection is unlimited in more than one dimension",
));
}
for d in 0..rank {
// Overlapping blocks are not a valid regular hyperslab.
if count[d] > 1 && block[d] != UNLIMITED && block[d] > stride[d] {
return Err(sel_err("regular hyperslab blocks overlap"));
}
}
return Ok(SerializedSelection::Regular {
start, start,
stride, stride,
count, count,
block, block,
}, });
pos, }
))
// Irregular: number of blocks, then each block's start and end corners.
let nblocks = r.uint(enc_size)?;
let per_block = (rank * 2 * enc_size) as u64;
// Untrusted count: it must fit in what is left of the buffer.
if nblocks
.checked_mul(per_block)
.is_none_or(|need| need > r.remaining() as u64)
{
return Err(FormatError::UnexpectedEof {
expected: r
.pos
.saturating_add(nblocks.saturating_mul(per_block) as usize),
available: r.data.len(),
});
}
let n = nblocks as usize * rank;
let (mut starts, mut ends) = (Vec::with_capacity(n), Vec::with_capacity(n));
for _ in 0..nblocks {
for _ in 0..rank {
starts.push(r.uint(enc_size)?);
}
for _ in 0..rank {
ends.push(r.uint(enc_size)?);
}
}
if starts.iter().zip(&ends).any(|(s, e)| e < s) {
return Err(sel_err("hyperslab block ends before it starts"));
}
Ok(SerializedSelection::Blocks { rank, starts, ends })
}
/// The coordinates of the union of the given blocks, in row-major order.
fn blocks_union_coords(
rank: usize,
starts: &[u64],
ends: &[u64],
) -> Result<Vec<Vec<u64>>, FormatError> {
let mut total = 0u64;
for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
let vol = s
.iter()
.zip(e)
.try_fold(1u64, |acc, (&s, &e)| acc.checked_mul(e - s + 1));
total = vol
.and_then(|v| total.checked_add(v))
.filter(|&t| t <= MAX_EXPANDED_POINTS)
.ok_or_else(|| sel_err("irregular hyperslab selection is too large to expand"))?;
}
let mut out = Vec::with_capacity(total as usize);
for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
let mut cur = s.to_vec();
'block: loop {
out.push(cur.clone());
for d in (0..rank).rev() {
if cur[d] < e[d] {
cur[d] += 1;
continue 'block;
}
cur[d] = s[d];
}
break;
}
}
// Lexicographic order of coordinates is row-major order.
out.sort_unstable();
out.dedup();
Ok(out)
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -642,11 +845,100 @@ mod tests {
} }
#[test] #[test]
fn decode_irregular_hyperslab_rejected() { fn decode_truncated_irregular_hyperslab_is_error() {
// Irregular, rank 1, but the block count is missing.
let bytes = [0x02u8, 0, 0, 0, 0x03, 0, 0, 0, 0x00, 0x02, 0x01, 0, 0, 0]; let bytes = [0x02u8, 0, 0, 0, 0x03, 0, 0, 0, 0x00, 0x02, 0x01, 0, 0, 0];
assert!(Selection::decode_serialized(&bytes).is_err()); assert!(Selection::decode_serialized(&bytes).is_err());
} }
/// Version 1 as libhdf5 writes it for the default (earliest) format bounds:
/// type, version, reserved(4), length(4), rank(4), nblocks(4), then each
/// block's start and inclusive end corner as 4-byte values.
fn v1_blocks(rank: u32, blocks: &[(&[u32], &[u32])]) -> Vec<u8> {
let mut b = Vec::new();
for w in [2u32, 1, 0, 0, rank, blocks.len() as u32] {
b.extend_from_slice(&w.to_le_bytes());
}
for (s, e) in blocks {
for v in s.iter().chain(e.iter()) {
b.extend_from_slice(&v.to_le_bytes());
}
}
b
}
#[test]
fn decode_v1_irregular_single_block() {
// Exactly what h5py/HDF5 2.0 writes for `[0:4]` with default libver.
let bytes = v1_blocks(1, &[(&[0], &[3])]);
let (sel, used) = Selection::decode_serialized(&bytes).unwrap();
assert_eq!(used, bytes.len());
assert_eq!(sel.iter_linear_1d(8).unwrap(), vec![0, 1, 2, 3]);
}
#[test]
fn decode_v1_irregular_union_is_row_major() {
// Blocks given out of order and overlapping still enumerate once each,
// in row-major order (libhdf5 iterates the union, not the list).
let bytes = v1_blocks(2, &[(&[1, 0], &[1, 1]), (&[0, 2], &[1, 2])]);
let (sel, used) = Selection::decode_serialized(&bytes).unwrap();
assert_eq!(used, bytes.len());
// (0,2) (1,0) (1,1) (1,2) in a 2x3 space.
assert_eq!(sel.iter_linear(&[2, 3]).unwrap(), vec![2, 3, 4, 5]);
}
#[test]
fn decode_v2_regular_with_unlimited_count() {
// v2: flags(1) + length(4), then 8-byte start/stride/count/block.
let mut b = Vec::new();
b.extend_from_slice(&2u32.to_le_bytes());
b.extend_from_slice(&2u32.to_le_bytes());
b.push(0x01);
b.extend_from_slice(&36u32.to_le_bytes());
b.extend_from_slice(&1u32.to_le_bytes());
for v in [0u64, 10, u64::MAX, 10] {
b.extend_from_slice(&v.to_le_bytes());
}
let (raw, used) = SerializedSelection::decode(&b).unwrap();
assert_eq!(used, b.len());
assert_eq!(raw.unlimited_dim(), Some(0));
assert_eq!(
raw,
SerializedSelection::Regular {
start: vec![0],
stride: vec![10],
count: vec![UNLIMITED],
block: vec![10],
}
);
// An unclipped unlimited selection cannot be enumerated.
let (sel, _) = Selection::decode_serialized(&b).unwrap();
assert!(sel.iter_linear_1d(100).is_err());
}
#[test]
fn decode_v3_two_byte_all_ones_is_unlimited() {
let bytes = [
0x02, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0x01, 0, 0, 0, //
0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0xFF, 0xFF,
];
let (raw, _) = SerializedSelection::decode(&bytes).unwrap();
assert_eq!(raw.unlimited_dim(), Some(0));
}
#[test]
fn decode_irregular_block_count_beyond_buffer_is_error() {
let mut b = v1_blocks(1, &[(&[0], &[3])]);
b[20..24].copy_from_slice(&u32::MAX.to_le_bytes());
assert!(Selection::decode_serialized(&b).is_err());
}
#[test]
fn decode_point_selection_is_refused() {
let bytes = [1u8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
assert!(Selection::decode_serialized(&bytes).is_err());
}
#[test] #[test]
fn iter_linear_2d_block_row_major() { fn iter_linear_2d_block_row_major() {
// A 2x2 block at the top-left of a 4x4 space => linear 0,1,4,5. // A 2x2 block at the top-left of a 4x4 space => linear 0,1,4,5.
crates/clawhdf5-format/src/vds.rs
+1100
View File
File diff suppressed because it is too large Load Diff
crates/clawhdf5-format/tests/integration_test.rs
+73 -2
View File
@@ -83,6 +83,45 @@ fn read_chunked_dataset(file_data: &[u8], dataset_path: &str) -> (Vec<u8>, Datat
(raw, datatype, dataspace) (raw, datatype, dataspace)
} }
/// Helper: read a virtual dataset with `vds::read_virtual_dataset`, giving it
/// the dataset's own fill value (same-file sources only).
fn read_virtual_fixture(file_data: &[u8], path: &str) -> (Vec<u8>, Datatype) {
let sig = find_signature(file_data).unwrap();
let sb = Superblock::parse(file_data, sig).unwrap();
let addr = resolve_path_any(file_data, &sb, path).unwrap();
let hdr =
ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
let msg = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t).unwrap();
let ds = Dataspace::parse(&msg(MessageType::Dataspace).data, sb.length_size).unwrap();
let (dt, _) = Datatype::parse(&msg(MessageType::Datatype).data).unwrap();
let layout = DataLayout::parse(
&msg(MessageType::DataLayout).data,
sb.offset_size,
sb.length_size,
)
.unwrap();
let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
file_data,
&hdr.messages,
sb.offset_size,
sb.length_size,
)
.unwrap();
let v = clawhdf5_format::vds::read_virtual_dataset(
file_data,
&layout,
&ds,
&dt,
fill.as_deref(),
sb.offset_size,
sb.length_size,
None,
)
.unwrap();
assert_eq!(v.dims, ds.dimensions);
(v.data, dt)
}
/// Helper: read any dataset (contiguous or chunked) as f64. /// Helper: read any dataset (contiguous or chunked) as f64.
fn read_dataset_f64_any(bytes: &[u8], path: &str) -> Vec<f64> { fn read_dataset_f64_any(bytes: &[u8], path: &str) -> Vec<f64> {
let sig = find_signature(bytes).unwrap(); let sig = find_signature(bytes).unwrap();
@@ -672,7 +711,7 @@ fn v4_virtual_dataset_same_file_read() {
// virt[4:8] <- (unmapped) => fill 0 // virt[4:8] <- (unmapped) => fill 0
// virt[8:12] <- src_b[0:4] (ALL) => 20,21,22,23 // virt[8:12] <- src_b[0:4] (ALL) => 20,21,22,23
let file_data = include_bytes!("fixtures/vds_same_file.h5"); let file_data = include_bytes!("fixtures/vds_same_file.h5");
let (raw, datatype, _) = read_chunked_dataset(file_data, "virt"); let (raw, datatype) = read_virtual_fixture(file_data, "virt");
let values = read_as_i32(&raw, &datatype).unwrap(); let values = read_as_i32(&raw, &datatype).unwrap();
assert_eq!( assert_eq!(
values, values,
@@ -681,6 +720,38 @@ fn v4_virtual_dataset_same_file_read() {
); );
} }
#[test]
fn v4_virtual_dataset_raw_api_refuses_to_guess_the_fill_value() {
// The raw read API has no fill value message, so a virtual dataset with an
// unmapped region is an error there instead of zeros that may be wrong.
let file_data = include_bytes!("fixtures/vds_same_file.h5");
let sig = find_signature(file_data).unwrap();
let sb = Superblock::parse(file_data, sig).unwrap();
let addr = resolve_path_any(file_data, &sb, "virt").unwrap();
let hdr =
ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
let msg = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t).unwrap();
let ds = Dataspace::parse(&msg(MessageType::Dataspace).data, sb.length_size).unwrap();
let (dt, _) = Datatype::parse(&msg(MessageType::Datatype).data).unwrap();
let layout = DataLayout::parse(
&msg(MessageType::DataLayout).data,
sb.offset_size,
sb.length_size,
)
.unwrap();
let err = read_raw_data_full(
file_data,
&layout,
&ds,
&dt,
None,
sb.offset_size,
sb.length_size,
)
.unwrap_err();
assert!(err.to_string().contains("fill value"), "{err}");
}
#[test] #[test]
fn v4_virtual_dataset_2d_same_file_read() { fn v4_virtual_dataset_2d_same_file_read() {
// A 4x4 virtual dataset assembled from two 2x2 same-file sources placed as // A 4x4 virtual dataset assembled from two 2x2 same-file sources placed as
@@ -689,7 +760,7 @@ fn v4_virtual_dataset_2d_same_file_read() {
// virt[2:4,2:4] <- src_b = [[5,6],[7,8]] // virt[2:4,2:4] <- src_b = [[5,6],[7,8]]
// everything else -> fill 0 // everything else -> fill 0
let file_data = include_bytes!("fixtures/vds_2d_same_file.h5"); let file_data = include_bytes!("fixtures/vds_2d_same_file.h5");
let (raw, datatype, _) = read_chunked_dataset(file_data, "virt"); let (raw, datatype) = read_virtual_fixture(file_data, "virt");
let values = read_as_i32(&raw, &datatype).unwrap(); let values = read_as_i32(&raw, &datatype).unwrap();
assert_eq!( assert_eq!(
values, values,
crates/clawhdf5/src/reader.rs
+75 -18
View File
@@ -522,6 +522,7 @@ impl<'f> Dataset<'f> {
self.file.length_size(), self.file.length_size(),
)?; )?;
let fill_matters = !clawhdf5_format::fill_value::has_storage(&dl) let fill_matters = !clawhdf5_format::fill_value::has_storage(&dl)
|| matches!(dl, DataLayout::Virtual { .. })
|| (matches!(dl, DataLayout::Chunked { .. }) || (matches!(dl, DataLayout::Chunked { .. })
&& !clawhdf5_format::fill_value::is_default(fill.as_deref())); && !clawhdf5_format::fill_value::is_default(fill.as_deref()));
if fill_matters { if fill_matters {
@@ -855,7 +856,22 @@ impl<'f> Dataset<'f> {
fn dataspace(&self) -> Result<Dataspace, Error> { fn dataspace(&self) -> Result<Dataspace, Error> {
let data = self.required_payload(MessageType::Dataspace)?; let data = self.required_payload(MessageType::Dataspace)?;
Ok(Dataspace::parse(&data, self.file.length_size())?) let mut ds = Dataspace::parse(&data, self.file.length_size())?;
// libhdf5 reports a virtual dataset with unlimited or printf-style
// mappings at the extent its sources currently fill, not the stored
// one (`H5Dget_space`).
if let Ok(dl @ DataLayout::Virtual { .. }) = self.data_layout() {
let resolver = self.vds_resolver();
ds.dimensions = clawhdf5_format::vds::virtual_dataset_extent(
self.file.data.as_bytes(),
&dl,
&ds,
self.file.offset_size(),
self.file.length_size(),
Some(&resolver),
)?;
}
Ok(ds)
} }
fn data_layout(&self) -> Result<DataLayout, Error> { fn data_layout(&self) -> Result<DataLayout, Error> {
@@ -884,24 +900,9 @@ impl<'f> Dataset<'f> {
let pipeline = self.filter_pipeline()?; let pipeline = self.filter_pipeline()?;
// Virtual datasets are assembled from source datasets; the per-file // Virtual datasets are assembled from source datasets; the per-file
// chunk cache does not apply. Route them through the resolver path so // chunk cache does not apply.
// external sibling files resolve relative to this file's directory.
if matches!(dl, DataLayout::Virtual { .. }) { if matches!(dl, DataLayout::Virtual { .. }) {
let base_dir = self.file.base_dir.clone(); return self.read_virtual(&dl, &ds, &dt);
let resolver = move |name: &str| -> Option<Vec<u8>> {
let dir = base_dir.as_ref()?;
std::fs::read(dir.join(sibling_file_name(name)?)).ok()
};
return Ok(data_read::read_raw_data_full_with_resolver(
self.file.data.as_bytes(),
&dl,
&ds,
&dt,
pipeline.as_ref(),
self.file.offset_size(),
self.file.length_size(),
Some(&resolver),
)?);
} }
// Unallocated storage reads as the dataset's fill value. // Unallocated storage reads as the dataset's fill value.
@@ -927,6 +928,62 @@ impl<'f> Dataset<'f> {
}, },
) )
} }
/// Resolver for external Virtual Dataset source files: names are
/// resolved against the directory of the file that holds the virtual
/// dataset, as libhdf5 does. A missing file is `Ok(None)` (its mappings
/// read as the fill value); a name that would leave that directory is
/// refused with an error rather than read as fill.
fn vds_resolver(&self) -> impl Fn(&str) -> Result<Option<Vec<u8>>, FormatError> + use<> {
let base_dir = self.file.base_dir.clone();
move |name: &str| {
let Some(dir) = base_dir.as_ref() else {
return Err(FormatError::ChunkedReadError(format!(
"virtual dataset source file {name:?} cannot be resolved for an in-memory file"
)));
};
let rel = sibling_file_name(name).ok_or_else(|| {
FormatError::ChunkedReadError(format!(
"virtual dataset source file {name:?} is outside the virtual file's \
directory and is not followed"
))
})?;
match std::fs::read(dir.join(rel)) {
Ok(bytes) => Ok(Some(bytes)),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(e) => Err(FormatError::ChunkedReadError(format!(
"cannot read virtual dataset source file {name:?}: {e}"
))),
}
}
}
/// Read a whole virtual dataset; unmapped elements hold its fill value.
fn read_virtual(
&self,
dl: &DataLayout,
ds: &Dataspace,
dt: &Datatype,
) -> Result<Vec<u8>, Error> {
let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
self.file.data.as_bytes(),
&self.header.messages,
self.file.offset_size(),
self.file.length_size(),
)?;
let resolver = self.vds_resolver();
let v = clawhdf5_format::vds::read_virtual_dataset(
self.file.data.as_bytes(),
dl,
ds,
dt,
fill.as_deref(),
self.file.offset_size(),
self.file.length_size(),
Some(&resolver),
)?;
Ok(v.data)
}
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
crates/clawhdf5/tests/fixtures/vds/4_0.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/4_1.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/4_2.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/4_vds.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/README.md
Vendored
+14
View File
@@ -0,0 +1,14 @@
# VDS test files from libhdf5
Copied unchanged from the HDF Group's HDF5 repository,
`tools/test/testfiles/vds/` (the h5dump/h5ls VDS test data). HDF5 is
distributed under a BSD-style license (see `COPYING` in the HDF5 source).
| File | What it exercises |
|---|---|
| `vds-eiger.h5` + `f-0.h5`, `f-3.h5` | printf-style source name `f-%b.h5`; `f-3.h5` lies past the first missing source and must be ignored (extent 5, not 20) |
| `4_vds.h5` + `4_0.h5`..`4_2.h5` | printf-style `4_%b.h5` with version-2 (1.10 format) hyperslab selections |
| `vds-percival-unlim-maxmin.h5` + `a.h5`..`d.h5` | four interleaved unlimited mappings whose sources have different lengths |
Used by `crates/clawhdf5/tests/vds_interop.rs::vds_libhdf5_test_files`,
which compares our reads with h5py's.
crates/clawhdf5/tests/fixtures/vds/a.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/b.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/c.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/d.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/f-0.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/f-3.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/vds-eiger.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/fixtures/vds/vds-percival-unlim-maxmin.h5
Vendored
BIN
View File
Binary file not shown.
crates/clawhdf5/tests/vds_interop.rs
+450
View File
@@ -0,0 +1,450 @@
//! Virtual Dataset (VDS) reads checked against libhdf5 (through h5py).
//!
//! Each test has h5py build virtual datasets and their source files in a temp
//! directory, record what libhdf5 reads back (shape and values) next to them,
//! and then compares that with what clawhdf5 reads from the same files.
//!
//! Skipped when python3 or h5py is unavailable, unless
//! `CLAWHDF5_REQUIRE_INTEROP=1`.
use std::path::Path;
use std::process::Command;
use clawhdf5::File;
/// The Python interpreter to drive interop checks with (`CLAWHDF5_PYTHON`
/// lets these run against a virtualenv holding h5py).
fn python() -> String {
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
}
/// When `CLAWHDF5_REQUIRE_INTEROP=1` (set in CI), a missing Python dependency
/// is a test failure instead of a silent skip.
fn interop_required() -> bool {
std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
}
fn python_available() -> bool {
Command::new(python())
.args(["-c", "import h5py, numpy"])
.output()
.map(|o| o.status.success())
.unwrap_or(false)
}
macro_rules! skip_if_no_python {
() => {
if !python_available() {
assert!(
!interop_required(),
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
);
eprintln!("SKIP: python3 with h5py not available");
return;
}
};
}
/// Prelude for every generator script: `expect(file, dset, tag)` records what
/// libhdf5 reads for `file:dset` as `<tag>.expect` (shape line, values line).
const PRELUDE: &str = r#"
import h5py, numpy as np
def expect(fn, dset, tag):
with h5py.File(fn, "r") as f:
d = f[dset]
a = d[...]
with open(tag + ".expect", "w") as out:
out.write(" ".join(str(n) for n in d.shape) + "\n")
out.write(" ".join(repr(float(v)) for v in a.ravel()) + "\n")
"#;
/// Run `body` (after [`PRELUDE`]) with `dir` as the working directory, so
/// relative source file names land next to the virtual file.
fn generate(dir: &Path, body: &str) {
let script = format!("{PRELUDE}\n{body}");
let out = Command::new(python())
.args(["-c", &script])
.current_dir(dir)
.output()
.expect("failed to run python");
assert!(
out.status.success(),
"generator failed:\nSTDOUT: {}\nSTDERR: {}",
String::from_utf8_lossy(&out.stdout),
String::from_utf8_lossy(&out.stderr)
);
}
/// What libhdf5 read for `tag`: (shape, values as f64).
fn expected(dir: &Path, tag: &str) -> (Vec<u64>, Vec<f64>) {
let text = std::fs::read_to_string(dir.join(format!("{tag}.expect"))).unwrap();
let mut lines = text.lines();
let parse_line = |l: Option<&str>| -> Vec<String> {
l.unwrap_or("")
.split_whitespace()
.map(str::to_string)
.collect()
};
let shape = parse_line(lines.next())
.iter()
.map(|s| s.parse().unwrap())
.collect();
let values = parse_line(lines.next())
.iter()
.map(|s| s.parse().unwrap())
.collect();
(shape, values)
}
/// Assert clawhdf5 reads `file:dset` exactly as libhdf5 did for `tag`.
fn assert_matches_libhdf5(dir: &Path, file: &str, dset: &str, tag: &str) {
let (shape, values) = expected(dir, tag);
let f = File::open(dir.join(file)).unwrap();
let ds = f.dataset(dset).unwrap();
assert_eq!(
ds.shape().unwrap(),
shape,
"{tag}: shape differs from libhdf5"
);
let got = ds
.read_f64()
.unwrap_or_else(|e| panic!("{tag}: read failed: {e}"));
assert_eq!(got.len(), values.len(), "{tag}: element count differs");
for (i, (g, e)) in got.iter().zip(&values).enumerate() {
assert!(
g == e || (g.is_nan() && e.is_nan()),
"{tag}: element {i} is {g}, libhdf5 reads {e}\n ours: {got:?}\n libhdf5: {values:?}"
);
}
}
// ---------------------------------------------------------------------------
// Selection encodings
// ---------------------------------------------------------------------------
/// Files written with the default (earliest) format bounds serialize every
/// VDS hyperslab as a version-1 *irregular* selection (4-byte block corners),
/// and a strided selection as many blocks. These were refused outright.
#[test]
fn vds_version1_irregular_hyperslab_selections() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
generate(
dir.path(),
r#"
with h5py.File("src.h5", "w") as s:
s.create_dataset("a", data=np.arange(12.0))
s.create_dataset("m", data=np.arange(20.0).reshape(4, 5))
with h5py.File("v1.h5", "w") as f: # default libver: hyperslab version 1
f.create_dataset("local", data=np.arange(10.0) * -1)
lay = h5py.VirtualLayout(shape=(12,), dtype="f8")
lay[0:4] = h5py.VirtualSource(".", "local", shape=(10,))[2:6]
lay[4:10] = h5py.VirtualSource("src.h5", "a", shape=(12,))[::2]
lay[10:12] = h5py.VirtualSource("src.h5", "a", shape=(12,))[10:12]
f.create_virtual_dataset("strided", lay)
lay = h5py.VirtualLayout(shape=(4, 6), dtype="f8")
lay[:, 0:2] = h5py.VirtualSource("src.h5", "m", shape=(4, 5))[:, 3:5]
lay[:, 2:6:2] = h5py.VirtualSource("src.h5", "m", shape=(4, 5))[:, 0:2]
lay[:, 3:6:2] = h5py.VirtualSource("src.h5", "m", shape=(4, 5))[:, 2:4]
f.create_virtual_dataset("grid", lay)
expect("v1.h5", "strided", "strided")
expect("v1.h5", "grid", "grid")
"#,
);
assert_matches_libhdf5(dir.path(), "v1.h5", "strided", "strided");
assert_matches_libhdf5(dir.path(), "v1.h5", "grid", "grid");
}
// ---------------------------------------------------------------------------
// Mapping list encoding
// ---------------------------------------------------------------------------
/// With a 2.0 low version bound libhdf5 writes the mapping list as block
/// version 1: a flags byte per entry, and repeated names stored as the index
/// of the entry that first spelled them out. The flags byte was mistaken for
/// an empty (same-file) name.
#[test]
fn vds_mapping_block_version1_shared_names() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
generate(
dir.path(),
r#"
name = "a_rather_long_dataset_name"
with h5py.File("a_rather_long_source_file.h5", "w") as s:
s.create_dataset(name, data=np.arange(12.0) + 100)
with h5py.File("shared.h5", "w", libver=("v200", "v200")) as f:
f.create_dataset(name, data=np.arange(12.0) * -1)
lay = h5py.VirtualLayout(shape=(4, 4), dtype="f8")
for i in range(3):
src = h5py.VirtualSource("a_rather_long_source_file.h5", name, shape=(12,))
lay[i] = src[4 * i:4 * i + 4]
lay[3] = h5py.VirtualSource(".", name, shape=(12,))[0:4]
f.create_virtual_dataset("v", lay)
# the heap block must really be version 1 for this test to mean anything
raw = open("shared.h5", "rb").read()
gcol = raw.index(b"GCOL")
assert raw[gcol + 32] == 1, "expected a version-1 VDS mapping block"
expect("shared.h5", "v", "shared")
"#,
);
assert_matches_libhdf5(dir.path(), "shared.h5", "v", "shared");
}
// ---------------------------------------------------------------------------
// Fill value
// ---------------------------------------------------------------------------
/// Elements no mapping supplies read as the virtual dataset's fill value, not
/// as 0: unmapped regions, a missing source file, a missing source dataset.
/// A source's own unallocated chunks read as *its* fill value.
#[test]
fn vds_unmapped_regions_read_as_fill_value() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
generate(
dir.path(),
r#"
for i in range(3):
with h5py.File(f"src_{i}.h5", "w") as s:
s.create_dataset("data", data=np.arange(10.0) + i * 100)
with h5py.File("sparse_src.h5", "w") as s:
d = s.create_dataset("data", shape=(10,), chunks=(5,), dtype="f8", fillvalue=42.0)
d[0:5] = np.arange(5.0) + 1000 # the second chunk is never written
for libver in ["earliest", "latest"]:
with h5py.File(f"fill_{libver}.h5", "w", libver=libver) as f:
f.create_dataset("local", data=np.arange(10.0) * -1)
lay = h5py.VirtualLayout(shape=(6, 10), dtype="f8")
for i in range(3):
lay[i] = h5py.VirtualSource(f"src_{i}.h5", "data", shape=(10,))
lay[3] = h5py.VirtualSource("no_such_file.h5", "data", shape=(10,))
lay[4] = h5py.VirtualSource("src_0.h5", "no_such_dataset", shape=(10,))
# row 5 is not mapped at all
f.create_virtual_dataset("files", lay, fillvalue=-1.0)
lay = h5py.VirtualLayout(shape=(20,), dtype="f8")
lay[0:10] = h5py.VirtualSource(".", "local", shape=(10,))
f.create_virtual_dataset("same_file", lay, fillvalue=7.0)
lay = h5py.VirtualLayout(shape=(12,), dtype="f8")
lay[1:11] = h5py.VirtualSource("sparse_src.h5", "data", shape=(10,))
f.create_virtual_dataset("sparse_source", lay, fillvalue=-3.5)
lay = h5py.VirtualLayout(shape=(3, 4), dtype="i4")
lay[1, :] = h5py.VirtualSource(".", "ints", shape=(4,))
f.create_dataset("ints", data=np.arange(4, dtype="i4") + 1)
f.create_virtual_dataset("int_fill", lay, fillvalue=-99)
for name in ["files", "same_file", "sparse_source", "int_fill"]:
expect(f"fill_{libver}.h5", name, f"{name}_{libver}")
"#,
);
for libver in ["earliest", "latest"] {
let file = format!("fill_{libver}.h5");
for name in ["files", "same_file", "sparse_source", "int_fill"] {
assert_matches_libhdf5(dir.path(), &file, name, &format!("{name}_{libver}"));
}
}
// A selection read goes through the same fill-aware assembly.
let f = File::open(dir.path().join("fill_latest.h5")).unwrap();
let sel = clawhdf5::Selection::slice(std::slice::from_ref(&(8..14)));
let got = f
.dataset("same_file")
.unwrap()
.read_f64_selection(&sel)
.unwrap();
assert_eq!(got, vec![-8.0, -9.0, 7.0, 7.0, 7.0, 7.0]);
}
/// A source name that would leave the virtual file's directory is refused
/// with an error; it used to be skipped and read silently as fill.
#[test]
fn vds_source_outside_directory_is_an_error_not_fill() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
std::fs::create_dir(dir.path().join("sub")).unwrap();
generate(
dir.path(),
r#"
with h5py.File("src.h5", "w") as s:
s.create_dataset("data", data=np.arange(4.0))
with h5py.File("sub/up.h5", "w", libver="latest") as f:
lay = h5py.VirtualLayout(shape=(4,), dtype="f8")
lay[:] = h5py.VirtualSource("../src.h5", "data", shape=(4,))
f.create_virtual_dataset("v", lay, fillvalue=-1.0)
with h5py.File("nested.h5", "w", libver="latest") as f:
lay = h5py.VirtualLayout(shape=(4,), dtype="f8")
lay[:] = h5py.VirtualSource("sub/inner.h5", "data", shape=(4,))
f.create_virtual_dataset("v", lay, fillvalue=-1.0)
with h5py.File("sub/inner.h5", "w") as s:
s.create_dataset("data", data=np.arange(4.0) + 10)
expect("nested.h5", "v", "nested")
"#,
);
// libhdf5 resolves "../src.h5" (and would read [0, 1, 2, 3]); we refuse
// to leave the directory, and say so.
let f = File::open(dir.path().join("sub/up.h5")).unwrap();
let err = f.dataset("v").unwrap().read_f64().unwrap_err();
assert!(
err.to_string().contains("not followed"),
"unexpected error: {err}"
);
// A relative name below the virtual file's directory resolves there.
assert_matches_libhdf5(dir.path(), "nested.h5", "v", "nested");
}
// ---------------------------------------------------------------------------
// Unlimited and printf-style mappings
// ---------------------------------------------------------------------------
/// Helpers for building unlimited VDS mappings through h5py's low-level API.
const UNLIMITED_HELPERS: &str = r#"
U = h5py.h5s.UNLIMITED
def space(dims, maxdims, start=None, count=None, stride=None, block=None):
s = h5py.h5s.create_simple(dims, maxdims)
if start is not None:
s.select_hyperslab(start, count, stride, block)
return s
def make_vds(fn, name, dims, maxdims, maps, fill, libver="latest", mode="w"):
# maps: [(vsel_kwargs, source_file, source_dataset, source_space)]
with h5py.File(fn, mode, libver=libver) as f:
dcpl = h5py.h5p.create(h5py.h5p.DATASET_CREATE)
for vsel, sfile, sdset, sspace in maps:
dcpl.set_virtual(space(dims, maxdims, **vsel), sfile.encode(), sdset.encode(), sspace)
dcpl.set_fill_value(np.array(fill, dtype="f8"))
h5py.h5d.create(f.id, name.encode(), h5py.h5t.IEEE_F64LE,
h5py.h5s.create_simple(dims, maxdims), dcpl=dcpl)
"#;
/// printf-style names: block `j` of the virtual selection comes from the
/// source named with `j` in place of `%b` (`%%` is a literal `%`), probing
/// j = 0, 1, ... until the first missing source. libhdf5 also recomputes the
/// extent from what it finds, so the stored dataspace is not the shape.
#[test]
fn vds_printf_source_names() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
let body = format!(
"{UNLIMITED_HELPERS}{}",
r#"
for i in [0, 1, 2, 4]: # 3 is missing: 4 is past the first gap and unused
with h5py.File(f"vds_src_{i}.h5", "w") as s:
s.create_dataset("data", data=np.arange(10.0) + i * 100)
with h5py.File(f"p%c_{i}.h5", "w") as s:
s.create_dataset("data", data=np.arange(10.0) - i * 100)
for libver in ["earliest", "latest"]:
fn = f"printf_{libver}.h5"
make_vds(fn, "files", (10,), (U,),
[(dict(start=(0,), count=(U,), stride=(10,), block=(10,)), "vds_src_%b.h5", "data",
space((10,), (10,), (0,), (1,), (1,), (10,)))], -1.0, libver)
# interleaved blocks with gaps between them, and an escaped percent sign
make_vds(fn, "escaped", (4,), (U,),
[(dict(start=(1,), count=(U,), stride=(6,), block=(4,)), "p%%c_%b.h5", "data",
space((10,), (10,), (2,), (1,), (1,), (4,)))], -5.0, libver, "a")
# printf in the dataset name, same file, 2-D frames
with h5py.File(fn, "a") as f:
for i in range(3):
f.create_dataset(f"frame_{i}", data=np.arange(6.0).reshape(2, 3) + 10 * i)
with h5py.File(fn, "a", libver=libver) as f:
dcpl = h5py.h5p.create(h5py.h5p.DATASET_CREATE)
dcpl.set_virtual(space((1, 2, 3), (U, 2, 3), (0, 0, 0), (U, 1, 1), (1, 1, 1), (1, 2, 3)),
b".", b"frame_%b", space((2, 3), (2, 3)))
h5py.h5d.create(f.id, b"frames", h5py.h5t.IEEE_F64LE,
h5py.h5s.create_simple((1, 2, 3), (U, 2, 3)), dcpl=dcpl)
for name in ["files", "escaped", "frames"]:
expect(fn, name, f"{name}_{libver}")
"#
);
generate(dir.path(), &body);
for libver in ["earliest", "latest"] {
for name in ["files", "escaped", "frames"] {
let file = format!("printf_{libver}.h5");
assert_matches_libhdf5(dir.path(), &file, name, &format!("{name}_{libver}"));
}
}
// The shape libhdf5 reports: three 10-element blocks.
let f = File::open(dir.path().join("printf_latest.h5")).unwrap();
assert_eq!(f.dataset("files").unwrap().shape().unwrap(), vec![30]);
}
/// Unlimited source and virtual selections: each mapping covers as much as
/// its source's current extent fills (a partial last block included), the
/// extent is the largest of them but never smaller than the limited
/// mappings need, and a missing source contributes nothing.
#[test]
fn vds_unlimited_mappings_follow_source_extents() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
let body = format!(
"{UNLIMITED_HELPERS}{}",
r#"
with h5py.File("grow.h5", "w") as s:
s.create_dataset("a", data=np.arange(7.0) + 1, maxshape=(None,))
s.create_dataset("b", data=np.arange(5.0) + 100, maxshape=(None,))
s.create_dataset("rows", data=np.arange(12.0).reshape(4, 3) + 50, maxshape=(None, 3))
unlim_src = lambda: space((1,), (U,), (0,), (U,), (1,), (1,))
for libver in ["earliest", "latest"]:
fn = f"unlim_{libver}.h5"
make_vds(fn, "interleaved", (1,), (U,), [
# blocks of 3 every 4: 7 source elements end mid-block
(dict(start=(0,), count=(U,), stride=(4,), block=(3,)), "grow.h5", "a", unlim_src()),
(dict(start=(3,), count=(U,), stride=(4,), block=(1,)), "grow.h5", "b", unlim_src()),
(dict(start=(0,), count=(U,), stride=(1,), block=(1,)), "missing.h5", "a", unlim_src()),
], -2.0, libver)
make_vds(fn, "rows", (6, 3), (U, 3), [
# an unlimited *block*, plus a limited mapping reaching row 5
(dict(start=(0, 0), count=(1, 1), stride=(1, 1), block=(U, 3)), "grow.h5", "rows",
space((1, 3), (U, 3), (0, 0), (1, 1), (1, 1), (U, 3))),
(dict(start=(5, 0), count=(1, 1), stride=(1, 1), block=(1, 3)), "grow.h5", "rows",
space((4, 3), (U, 3), (1, 0), (1, 1), (1, 1), (1, 3))),
], -4.0, libver, "a")
for name in ["interleaved", "rows"]:
expect(fn, name, f"{name}_{libver}")
"#
);
generate(dir.path(), &body);
for libver in ["earliest", "latest"] {
for name in ["interleaved", "rows"] {
let file = format!("unlim_{libver}.h5");
assert_matches_libhdf5(dir.path(), &file, name, &format!("{name}_{libver}"));
}
}
// "a" (7 elements, blocks of 3 every 4) ends at 9; "b" (5 elements from
// 3, every 4) at 20. "rows" fills 4 rows but a limited mapping needs 6.
let f = File::open(dir.path().join("unlim_latest.h5")).unwrap();
assert_eq!(f.dataset("interleaved").unwrap().shape().unwrap(), vec![20]);
assert_eq!(f.dataset("rows").unwrap().shape().unwrap(), vec![6, 3]);
}
/// libhdf5's own VDS test files (HDF5 `tools/test/testfiles/vds`):
/// printf-style Eiger frames (with a source past the first gap that must be
/// ignored), a printf mapping in the 1.10 format, and Percival's four
/// interleaved unlimited sources of different lengths.
#[test]
fn vds_libhdf5_test_files() {
skip_if_no_python!();
let dir = tempfile::tempdir().unwrap();
let fixtures = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/vds");
for entry in std::fs::read_dir(&fixtures).unwrap() {
let path = entry.unwrap().path();
if path.extension().is_some_and(|e| e == "h5") {
std::fs::copy(&path, dir.path().join(path.file_name().unwrap())).unwrap();
}
}
let cases = [
("vds-eiger.h5", "/VDS-Eiger"),
("4_vds.h5", "/vds_dset"),
("vds-percival-unlim-maxmin.h5", "/VDS-Percival-unlim-maxmin"),
];
let mut body = String::new();
for (i, (file, dset)) in cases.iter().enumerate() {
body.push_str(&format!("expect({file:?}, {dset:?}, \"case{i}\")\n"));
}
generate(dir.path(), &body);
for (i, (file, dset)) in cases.iter().enumerate() {
assert_matches_libhdf5(dir.path(), file, dset, &format!("case{i}"));
}
// Stored as 20 frames; only f-0.h5 is found before the first gap.
let f = File::open(dir.path().join("vds-eiger.h5")).unwrap();
assert_eq!(
f.dataset("VDS-Eiger").unwrap().shape().unwrap(),
vec![5, 10, 10]
);
}
docs/known-issues.md
+16 -5
View File
@@ -73,8 +73,8 @@ segfault or abort.
## Gaps found by the 2026-09-25 HDF5 audit (open) ## Gaps found by the 2026-09-25 HDF5 audit (open)
**Status:** open. These fail with an error; none returns wrong data, except **Status:** open. These fail with an error; none returns wrong data (the VDS
the VDS item, which is marked. fill-value item that did is fixed).
- **Layout message versions 1 and 2** (HDF5 1.6-era files): 84 of the 686 - **Layout message versions 1 and 2** (HDF5 1.6-era files): 84 of the 686
sweep files, `InvalidLayoutVersion`. This is the largest single gap. sweep files, `InvalidLayoutVersion`. This is the largest single gap.
@@ -85,9 +85,20 @@ the VDS item, which is marked.
per-member dimensions were skipped, so an array member read as one scalar. per-member dimensions were skipped, so an array member read as one scalar.
**Fixed 2026-09-25.** **Fixed 2026-09-25.**
- **Virtual datasets:** - **Virtual datasets:**
- **Wrong data:** unmapped regions read as 0 instead of the fill value. - ~~**Wrong data:** unmapped regions read as 0 instead of the fill value.~~
- `%b` printf-style source names are not expanded. Fixed 2026-09-25: unmapped elements and missing sources read as the
- Hyperslab selection versions 1 and 2 are refused. virtual dataset's fill value.
- ~~`%b` printf-style source names are not expanded.~~ Fixed 2026-09-25:
printf-style and unlimited mappings are read, and the extent is
recomputed from the sources as libhdf5 does. Still open: the
"first missing" view and a printf gap other than 0 (libhdf5 access
properties we always read at their defaults), source-to-virtual type
conversion other than a byte swap, nested virtual sources, and source
files outside the virtual file's directory (refused with an error).
- ~~Hyperslab selection versions 1 and 2 are refused.~~ Fixed 2026-09-25:
versions 1-3 and irregular hyperslabs are decoded.
- ~~The version-1 mapping list written with a 2.0 low bound (flags byte,
shared names) was misparsed.~~ Found and fixed 2026-09-25.
- **Files with a user block:** the base address is not applied. - **Files with a user block:** the base address is not applied.
**Fixed 2026-09-25:** every reader views the file from the superblock on **Fixed 2026-09-25:** every reader views the file from the superblock on
(`twithub.h5`, `twithub513.h5`, `h5clear_fsm_persist_user_*.h5`; the (`twithub.h5`, `twithub513.h5`, `h5clear_fsm_persist_user_*.h5`; the