Merge branch 'fix/p1-vds' into fix/p1-read-gaps
# Conflicts: # CHANGELOG.md # crates/clawhdf5-format/src/data_read.rs
This commit is contained in:
osobh
@@ -331,6 +331,47 @@
|
||||
The legacy per-member dimension fields are now decoded into an array type,
|
||||
as libhdf5 does; more than four dimensions, or a zero-sized one, is an
|
||||
error.
|
||||
- `clawhdf5-format` virtual datasets (VDS), checked against HDF5 2.0 through
|
||||
h5py (`crates/clawhdf5/tests/vds_interop.rs`):
|
||||
- **Wrong data:** elements no mapping supplies — unmapped regions, and
|
||||
mappings whose source file or dataset is missing — read as 0 instead of
|
||||
the virtual dataset's fill value (e.g. h5py `fillvalue=-1`). Assembly moved
|
||||
to the new `vds` module: `vds::read_virtual_dataset` takes the fill value
|
||||
and a resolver that can refuse a name (`VdsFileResolver`), and `File`
|
||||
passes the dataset's fill value. A missing source *dataset* read as an
|
||||
error; it is fill now, as in libhdf5. Source datasets are read with their
|
||||
own fill value for unallocated chunks, and a source whose datatype differs
|
||||
from the virtual dataset's is an error (libhdf5 converts; we do not).
|
||||
`File` now refuses a source name that leaves the virtual file's directory
|
||||
(`../x.h5`, absolute paths), or any external source of a `File::from_bytes`
|
||||
file, with an error — these used to read as fill.
|
||||
**Behaviour change:** the raw-read API (`read_raw_data_full*`), which has
|
||||
no fill value, now returns an error for a virtual dataset with unmapped
|
||||
elements instead of zeros.
|
||||
- Unlimited and printf-style mappings are supported (all 7 VDS files in the
|
||||
libhdf5 test set are such mappings, e.g. Eiger/Percival detector layouts).
|
||||
`%b` in a source file or dataset name is the block number and `%%` a
|
||||
literal `%` (other `%` sequences are an error, as in libhdf5); block `j`
|
||||
is read from the source named with `j`, probing from 0 up to the first
|
||||
missing source. Unlimited source/virtual selections cover as much as the
|
||||
source's current extent fills, including a partial last block. As
|
||||
libhdf5 does on `H5Dget_space`, the extent is recomputed from the sources
|
||||
present (default "last available" view, printf gap 0) —
|
||||
`vds::virtual_dataset_extent`, used by `Dataset::shape()` — so e.g.
|
||||
`vds-eiger.h5` is `[5, 10, 10]`, not its stored `[20, 10, 10]`. A source
|
||||
stored in the other byte order is byte-swapped (libhdf5 converts);
|
||||
other type conversions remain an error.
|
||||
- Hyperslab selection versions 1 and 2 were refused ("only version-3
|
||||
hyperslab selections are supported"). Version 1 is what libhdf5 writes for
|
||||
every VDS created with the default format bounds (h5py's default), so
|
||||
those could not be read at all; version 2 is its encoding of an unlimited
|
||||
selection. Both are decoded now, as are irregular hyperslabs (a union of
|
||||
blocks, read in row-major order as libhdf5 iterates them).
|
||||
`SerializedSelection` exposes the raw form, including unlimited counts.
|
||||
- The version-1 mapping list HDF5 2.0 writes (low version bound 2.0) was
|
||||
misparsed: each entry's flags byte was read as the start of the source
|
||||
file name, and names shared with an earlier entry (stored as that entry's
|
||||
index) were not followed. Now decoded as `H5D__virtual_load_layout` does.
|
||||
- `clawhdf5-format` reader — **values returned wrong with no error:**
|
||||
- Fixed Array and Extensible Array chunk indexes were laid out by the
|
||||
dataset's current shape instead of its max shape (23 libhdf5 test files,
|
||||
|
||||
@@ -74,21 +74,33 @@ pub enum DataLayout {
|
||||
},
|
||||
}
|
||||
|
||||
/// Version-1 VDS mapping flag: the source file name is stored by an earlier
|
||||
/// entry, whose index follows in place of the name.
|
||||
const VDS_SOURCE_FILE_SHARED: u8 = 0x01;
|
||||
/// Version-1 VDS mapping flag: likewise for the source dataset name.
|
||||
const VDS_SOURCE_DSET_SHARED: u8 = 0x02;
|
||||
/// Version-1 VDS mapping flag: the source is in the virtual file itself
|
||||
/// (`"."`); no file name is stored.
|
||||
const VDS_SOURCE_SAME_FILE: u8 = 0x04;
|
||||
const VDS_ALL_FLAGS: u8 = VDS_SOURCE_FILE_SHARED | VDS_SOURCE_DSET_SHARED | VDS_SOURCE_SAME_FILE;
|
||||
|
||||
/// Parse VDS mappings from global-heap object data.
|
||||
///
|
||||
/// The global-heap block holding a VDS mapping list is laid out as
|
||||
/// (reverse-engineered and validated against HDF5 2.0):
|
||||
/// (`H5D__virtual_store_layout` / `H5D__virtual_load_layout` in libhdf5):
|
||||
///
|
||||
/// ```text
|
||||
/// version(1) · nused(length_size, LE) · entry[nused] · checksum(4)
|
||||
/// ```
|
||||
///
|
||||
/// Each entry is:
|
||||
/// - source file name — a null-terminated string in **block version 0**; in
|
||||
/// **block version 1** a same-file reference is encoded as a single `0x04`
|
||||
/// marker byte (the source file is the virtual file itself) in place of the
|
||||
/// name;
|
||||
/// - source dataset name (null-terminated string);
|
||||
/// - **block version 1 only:** a flags byte. `0x04`: the source is in the
|
||||
/// virtual file itself and no file name is stored; `0x01`/`0x02`: the
|
||||
/// source file/dataset name is that of an earlier entry, whose index
|
||||
/// (`length_size` bytes) is stored instead of the name. libhdf5 2.0 writes
|
||||
/// version 1 when the file's low version bound is 2.0 and it saves space;
|
||||
/// - source file name (null-terminated string, unless flagged above);
|
||||
/// - source dataset name (null-terminated string, unless flagged above);
|
||||
/// - source selection (serialized `H5S` dataspace selection — self-describing
|
||||
/// in length);
|
||||
/// - virtual selection (serialized `H5S` dataspace selection).
|
||||
@@ -114,7 +126,7 @@ pub fn parse_vds_mappings(
|
||||
// `nused` is untrusted; don't pre-allocate from it. Each entry consumes at
|
||||
// least a few bytes, so the loop is naturally bounded by the heap data and
|
||||
// a bogus `nused` simply errors out on the first short read.
|
||||
let mut mappings = Vec::new();
|
||||
let mut mappings: Vec<VdsMapping> = Vec::new();
|
||||
// Reads one self-describing selection at `pos`, returning its raw bytes and
|
||||
// advancing past it — bounds-checked so a corrupt selection can't overrun.
|
||||
let read_selection = |heap_data: &[u8], pos: &mut usize| -> Result<Vec<u8>, FormatError> {
|
||||
@@ -134,17 +146,57 @@ pub fn parse_vds_mappings(
|
||||
Ok(bytes)
|
||||
};
|
||||
|
||||
for _ in 0..nused {
|
||||
// Source file name (with the version-1 same-file marker handled).
|
||||
let source_file = if version >= 1 && heap_data.get(pos) == Some(&0x04) {
|
||||
if version > 1 {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"unsupported VDS mapping block version".into(),
|
||||
));
|
||||
}
|
||||
for i in 0..nused {
|
||||
// Version 1 prefixes each entry with a flags byte; a name may then be
|
||||
// omitted (same file) or replaced by the index of an earlier entry
|
||||
// holding the same name (`H5D__virtual_load_layout`).
|
||||
let flags = if version >= 1 {
|
||||
let f = *heap_data.get(pos).ok_or(FormatError::UnexpectedEof {
|
||||
expected: pos + 1,
|
||||
available: heap_data.len(),
|
||||
})?;
|
||||
pos += 1;
|
||||
if f & !VDS_ALL_FLAGS != 0 {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"unknown VDS mapping flags".into(),
|
||||
));
|
||||
}
|
||||
f
|
||||
} else {
|
||||
0
|
||||
};
|
||||
// Index of an earlier entry, for a shared name.
|
||||
let earlier = |pos: &mut usize| -> Result<usize, FormatError> {
|
||||
let idx = read_length(heap_data, *pos, length_size)?;
|
||||
*pos += ls;
|
||||
if idx >= i {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"VDS mapping shares a name with a later entry".into(),
|
||||
));
|
||||
}
|
||||
Ok(idx as usize)
|
||||
};
|
||||
|
||||
let source_file = if flags & VDS_SOURCE_SAME_FILE != 0 {
|
||||
String::from(".")
|
||||
} else if flags & VDS_SOURCE_FILE_SHARED != 0 {
|
||||
let idx = earlier(&mut pos)?;
|
||||
mappings[idx].source_file.clone()
|
||||
} else {
|
||||
read_null_terminated_string(heap_data, &mut pos)?
|
||||
};
|
||||
|
||||
// Source dataset name.
|
||||
let source_dataset = read_null_terminated_string(heap_data, &mut pos)?;
|
||||
let source_dataset = if flags & VDS_SOURCE_DSET_SHARED != 0 {
|
||||
let idx = earlier(&mut pos)?;
|
||||
mappings[idx].source_dataset.clone()
|
||||
} else {
|
||||
read_null_terminated_string(heap_data, &mut pos)?
|
||||
};
|
||||
|
||||
// Source selection, then virtual selection (both self-describing length).
|
||||
let source_selection = read_selection(heap_data, &mut pos)?;
|
||||
@@ -1035,6 +1087,62 @@ mod tests {
|
||||
assert_eq!(v1.iter_linear_1d(8).unwrap(), vec![4, 5, 6, 7]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_vds_mappings_v1_shared_names() {
|
||||
// Written by HDF5 2.0 (h5py, libver=("v200", "v200")) for three
|
||||
// mappings from `a_rather_long_source_file.h5:a_rather_long_dataset_name`
|
||||
// and one from the same file: the entries carry flags 0x00, 0x03, 0x03
|
||||
// and 0x06, so names after the first are stored as entry indices.
|
||||
let blob: &[u8] = &[
|
||||
0x01, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x61, 0x5f, 0x72, 0x61,
|
||||
0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x73, 0x6f, 0x75, 0x72,
|
||||
0x63, 0x65, 0x5f, 0x66, 0x69, 0x6c, 0x65, 0x2e, 0x68, 0x35, 0x00, 0x61, 0x5f, 0x72,
|
||||
0x61, 0x74, 0x68, 0x65, 0x72, 0x5f, 0x6c, 0x6f, 0x6e, 0x67, 0x5f, 0x64, 0x61, 0x74,
|
||||
0x61, 0x73, 0x65, 0x74, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x00, 0x02, 0x00, 0x00, 0x00,
|
||||
0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00,
|
||||
0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02,
|
||||
0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00,
|
||||
0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
|
||||
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03,
|
||||
0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x04, 0x00, 0x01, 0x00, 0x01,
|
||||
0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02,
|
||||
0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01,
|
||||
0x00, 0x01, 0x00, 0x04, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
|
||||
0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00,
|
||||
0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x00, 0x01, 0x00,
|
||||
0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x02, 0x00,
|
||||
0x00, 0x00, 0x02, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00,
|
||||
0x01, 0x00, 0x04, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02,
|
||||
0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x02, 0x01, 0x00, 0x00, 0x00, 0x00,
|
||||
0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00,
|
||||
0x00, 0x01, 0x02, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01,
|
||||
0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x04, 0x00, 0x8e, 0xa7, 0xea, 0x7a,
|
||||
];
|
||||
let mappings = parse_vds_mappings(blob, 8).unwrap();
|
||||
let names: Vec<(&str, &str)> = mappings
|
||||
.iter()
|
||||
.map(|m| (m.source_file.as_str(), m.source_dataset.as_str()))
|
||||
.collect();
|
||||
let (file, dset) = ("a_rather_long_source_file.h5", "a_rather_long_dataset_name");
|
||||
assert_eq!(
|
||||
names,
|
||||
vec![(file, dset), (file, dset), (file, dset), (".", dset)]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_vds_mappings_v1_forward_reference_is_error() {
|
||||
// Entry 0 claiming to share entry 0's file name must not index past
|
||||
// the entries decoded so far.
|
||||
let mut blob = vec![0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x01];
|
||||
blob.extend_from_slice(&[0u8; 8]);
|
||||
blob.extend_from_slice(b"d\0");
|
||||
assert!(parse_vds_mappings(&blob, 8).is_err());
|
||||
// Unknown flag bits are refused.
|
||||
let blob = [0x01u8, 1, 0, 0, 0, 0, 0, 0, 0, 0x08, b'd', 0];
|
||||
assert!(parse_vds_mappings(&blob, 8).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parse_vds_mappings_external_v0() {
|
||||
// Block version 0 with an explicit (external) source file name.
|
||||
|
||||
@@ -191,14 +191,9 @@ fn read_raw_data_full_impl(
|
||||
offset_size,
|
||||
length_size,
|
||||
),
|
||||
DataLayout::Virtual {
|
||||
global_heap_address,
|
||||
global_heap_index,
|
||||
..
|
||||
} => read_virtual_data(
|
||||
DataLayout::Virtual { .. } => read_virtual_data(
|
||||
file_data,
|
||||
*global_heap_address,
|
||||
*global_heap_index,
|
||||
layout,
|
||||
dataspace,
|
||||
datatype,
|
||||
offset_size,
|
||||
@@ -465,160 +460,54 @@ pub fn read_raw_data_selection(
|
||||
}
|
||||
}
|
||||
|
||||
/// Assemble a **Virtual Dataset (VDS)** from its source mappings.
|
||||
/// Assemble a **Virtual Dataset (VDS)** through the raw-read API, which has no
|
||||
/// access to the dataset's fill value message.
|
||||
///
|
||||
/// Supports virtual datasets of any rank. Same-file sources are read directly;
|
||||
/// **external-file** sources are read through the caller-supplied `resolver`,
|
||||
/// which maps a stored source file name to that file's bytes. Each mapping's
|
||||
/// selected source elements are scattered into the virtual buffer at the
|
||||
/// positions given by the virtual selection (both enumerated in row-major
|
||||
/// order, as HDF5 pairs them). Unmapped regions are left at the zero fill value.
|
||||
///
|
||||
/// A mapping whose external source file the resolver cannot supply (`None`) is
|
||||
/// skipped, leaving its region at fill — matching HDF5's tolerance of missing
|
||||
/// sources. An external source with no resolver at all is a hard error.
|
||||
/// Delegates to [`crate::vds::read_virtual_dataset`]. Because the fill value
|
||||
/// is unknown here, a virtual dataset with any element no mapping supplies
|
||||
/// (an unmapped region, or a missing source file or dataset) is an error
|
||||
/// rather than a guess at the fill value; so is one whose extent libhdf5
|
||||
/// would report differently from the stored dataspace (unlimited mappings).
|
||||
/// Use [`crate::vds::read_virtual_dataset`] to read those.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn read_virtual_data(
|
||||
file_data: &[u8],
|
||||
global_heap_address: Option<u64>,
|
||||
global_heap_index: u32,
|
||||
layout: &DataLayout,
|
||||
dataspace: &Dataspace,
|
||||
datatype: &Datatype,
|
||||
offset_size: u8,
|
||||
length_size: u8,
|
||||
resolver: Option<&VdsSourceResolver>,
|
||||
) -> Result<Vec<u8>, FormatError> {
|
||||
use crate::data_layout::parse_vds_mappings;
|
||||
use crate::global_heap::GlobalHeapCollection;
|
||||
use crate::selection::Selection;
|
||||
|
||||
let elem_size = datatype.type_size() as usize;
|
||||
let mut out = crate::chunked_read::alloc_output(crate::chunked_read::checked_byte_len(
|
||||
dataspace.checked_num_elements()?,
|
||||
elem_size,
|
||||
)?)?;
|
||||
|
||||
let virtual_dims = &dataspace.dimensions;
|
||||
|
||||
let addr = global_heap_address.ok_or_else(|| {
|
||||
FormatError::ChunkedReadError("virtual dataset has no mapping global heap".into())
|
||||
})?;
|
||||
let coll = GlobalHeapCollection::parse(file_data, addr as usize, length_size)?;
|
||||
let obj =
|
||||
coll.get_object(global_heap_index as u16)
|
||||
.ok_or(FormatError::GlobalHeapObjectNotFound {
|
||||
collection_address: addr,
|
||||
index: global_heap_index as u16,
|
||||
})?;
|
||||
let mappings = parse_vds_mappings(&obj.data, length_size)?;
|
||||
|
||||
for m in &mappings {
|
||||
let same_file = m.source_file.is_empty() || m.source_file == ".";
|
||||
|
||||
// Resolve the bytes of the file holding this source dataset.
|
||||
let external;
|
||||
let src_file_data: &[u8] = if same_file {
|
||||
file_data
|
||||
} else {
|
||||
let r = resolver.ok_or_else(|| {
|
||||
FormatError::ChunkedReadError(
|
||||
"external-file virtual dataset sources require a file resolver".into(),
|
||||
)
|
||||
})?;
|
||||
match r(&m.source_file) {
|
||||
Some(bytes) => {
|
||||
external = bytes;
|
||||
&external
|
||||
}
|
||||
// Source file unavailable: leave this region at fill value.
|
||||
None => continue,
|
||||
}
|
||||
};
|
||||
|
||||
let (vsel, _) = Selection::decode_serialized(&m.virtual_selection)?;
|
||||
let (ssel, _) = Selection::decode_serialized(&m.source_selection)?;
|
||||
|
||||
let (src_raw, src_dims) =
|
||||
read_named_dataset_raw(src_file_data, &m.source_dataset, offset_size, length_size)?;
|
||||
|
||||
let vidx = vsel.iter_linear(virtual_dims)?;
|
||||
let sidx = ssel.iter_linear(&src_dims)?;
|
||||
if vidx.len() != sidx.len() {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"virtual/source selection element counts differ".into(),
|
||||
));
|
||||
}
|
||||
|
||||
for (&v, &s) in vidx.iter().zip(sidx.iter()) {
|
||||
let (vo, so) = (v as usize * elem_size, s as usize * elem_size);
|
||||
if vo + elem_size > out.len() || so + elem_size > src_raw.len() {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"virtual dataset selection out of bounds".into(),
|
||||
));
|
||||
}
|
||||
out[vo..vo + elem_size].copy_from_slice(&src_raw[so..so + elem_size]);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Read a named dataset's raw (decoded) bytes and its dimensions, navigating
|
||||
/// from the superblock. Used to pull VDS source datasets out of the same file.
|
||||
fn read_named_dataset_raw(
|
||||
file_data: &[u8],
|
||||
path: &str,
|
||||
_offset_size: u8,
|
||||
_length_size: u8,
|
||||
) -> Result<(Vec<u8>, Vec<u64>), FormatError> {
|
||||
use crate::filter_pipeline::FilterPipeline;
|
||||
use crate::group_v2::resolve_path_any;
|
||||
use crate::message_type::MessageType;
|
||||
use crate::object_header::ObjectHeader;
|
||||
use crate::signature::split_user_block;
|
||||
use crate::superblock::Superblock;
|
||||
|
||||
// An external source file is handed over whole, user block included;
|
||||
// its addresses are relative to its superblock.
|
||||
let (_, file_data) = split_user_block(file_data)?;
|
||||
let sb = Superblock::parse(file_data, 0)?;
|
||||
let addr = resolve_path_any(file_data, &sb, path)?;
|
||||
let hdr = ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size)?;
|
||||
|
||||
let find = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t);
|
||||
let ds_msg = find(MessageType::Dataspace)
|
||||
.ok_or_else(|| FormatError::ChunkedReadError("VDS source has no dataspace".into()))?;
|
||||
let dataspace = Dataspace::parse(&ds_msg.data, sb.length_size)?;
|
||||
let dt_msg = find(MessageType::Datatype)
|
||||
.ok_or_else(|| FormatError::ChunkedReadError("VDS source has no datatype".into()))?;
|
||||
let (datatype, _) = Datatype::parse(&dt_msg.data)?;
|
||||
let dl_msg = find(MessageType::DataLayout)
|
||||
.ok_or_else(|| FormatError::ChunkedReadError("VDS source has no data layout".into()))?;
|
||||
let layout = DataLayout::parse(&dl_msg.data, sb.offset_size, sb.length_size)?;
|
||||
// A virtual dataset whose source is itself another virtual dataset could
|
||||
// form a cycle (A -> B -> A) and recurse into a stack overflow. Nested
|
||||
// virtual sources are exotic and unsupported, so stop here cleanly.
|
||||
if matches!(layout, DataLayout::Virtual { .. }) {
|
||||
let wrapped =
|
||||
resolver.map(|r| move |name: &str| -> Result<Option<Vec<u8>>, FormatError> { Ok(r(name)) });
|
||||
let wrapped_ref = wrapped.as_ref().map(|w| w as &crate::vds::VdsFileResolver);
|
||||
let v = crate::vds::read_virtual_dataset(
|
||||
file_data,
|
||||
layout,
|
||||
dataspace,
|
||||
datatype,
|
||||
None,
|
||||
offset_size,
|
||||
length_size,
|
||||
wrapped_ref,
|
||||
)?;
|
||||
if v.dims != dataspace.dimensions {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"virtual dataset source is itself virtual (unsupported)".into(),
|
||||
"virtual dataset extent differs from its stored dataspace; \
|
||||
read it with vds::read_virtual_dataset"
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
let pipeline = find(MessageType::FilterPipeline)
|
||||
.map(|m| FilterPipeline::parse(&m.data))
|
||||
.transpose()?;
|
||||
|
||||
let raw = read_raw_data_full(
|
||||
file_data,
|
||||
&layout,
|
||||
&dataspace,
|
||||
&datatype,
|
||||
pipeline.as_ref(),
|
||||
sb.offset_size,
|
||||
sb.length_size,
|
||||
)?;
|
||||
Ok((raw, dataspace.dimensions.clone()))
|
||||
if v.unmapped > 0 {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"virtual dataset has elements no source supplies, which read as its \
|
||||
fill value; read it with vds::read_virtual_dataset and the fill value"
|
||||
.into(),
|
||||
));
|
||||
}
|
||||
Ok(v.data)
|
||||
}
|
||||
|
||||
/// Extract selected elements from a full dataset buffer.
|
||||
pub fn extract_selection_from_buffer(
|
||||
full_data: &[u8],
|
||||
|
||||
@@ -101,6 +101,7 @@ pub mod signature;
|
||||
pub mod superblock;
|
||||
pub mod symbol_table;
|
||||
pub mod type_builders;
|
||||
pub mod vds;
|
||||
pub mod vl_data;
|
||||
|
||||
#[cfg(feature = "provenance")]
|
||||
|
||||
@@ -229,44 +229,47 @@ impl Selection {
|
||||
/// self-describing in length, so the count lets a caller walk a packed list
|
||||
/// of selections — as the Virtual Dataset global-heap block does).
|
||||
///
|
||||
/// Only the forms needed for VDS assembly are decoded: `ALL`, `NONE`, and
|
||||
/// **regular** hyperslabs serialized at **version 3** (the encoding HDF5
|
||||
/// 1.10+/2.0 emit). Point selections, irregular hyperslabs, and older
|
||||
/// hyperslab versions return an error rather than mis-decoding.
|
||||
/// Decodes `ALL`, `NONE`, and hyperslabs at every version libhdf5 writes
|
||||
/// (1: irregular, 4-byte coordinates — the default-format encoding; 2:
|
||||
/// regular, 8-byte; 3: either, variable width). A regular hyperslab maps
|
||||
/// to [`Selection::Hyperslab`]; an *irregular* one (a union of blocks)
|
||||
/// maps to a single-block hyperslab when it has one block, and otherwise to
|
||||
/// [`Selection::Points`] listing the union in row-major order (the order
|
||||
/// libhdf5 iterates it in). Unlimited counts/blocks decode as `u64::MAX`
|
||||
/// (see [`SerializedSelection::decode`] for the raw form). Point
|
||||
/// selections are refused: libhdf5 does not allow them in virtual datasets
|
||||
/// either.
|
||||
pub fn decode_serialized(data: &[u8]) -> Result<(Selection, usize), FormatError> {
|
||||
if data.len() < 8 {
|
||||
return Err(FormatError::UnexpectedEof {
|
||||
expected: 8,
|
||||
available: data.len(),
|
||||
});
|
||||
}
|
||||
let sel_type = u32::from_le_bytes([data[0], data[1], data[2], data[3]]);
|
||||
let version = u32::from_le_bytes([data[4], data[5], data[6], data[7]]);
|
||||
|
||||
match sel_type {
|
||||
// ALL / NONE: type(4) + version(4) + reserved(4) + length(4) = 16 bytes.
|
||||
3 | 0 => {
|
||||
if data.len() < 16 {
|
||||
return Err(FormatError::UnexpectedEof {
|
||||
expected: 16,
|
||||
available: data.len(),
|
||||
});
|
||||
}
|
||||
let sel = if sel_type == 3 {
|
||||
Selection::All
|
||||
let (raw, len) = SerializedSelection::decode(data)?;
|
||||
let sel = match raw {
|
||||
SerializedSelection::All => Selection::All,
|
||||
SerializedSelection::None => Selection::None,
|
||||
SerializedSelection::Regular {
|
||||
start,
|
||||
stride,
|
||||
count,
|
||||
block,
|
||||
} => Selection::Hyperslab {
|
||||
start,
|
||||
stride,
|
||||
count,
|
||||
block,
|
||||
},
|
||||
SerializedSelection::Blocks { rank, starts, ends } => {
|
||||
if starts.len() == rank {
|
||||
let block = starts.iter().zip(&ends).map(|(&s, &e)| e - s + 1).collect();
|
||||
Selection::Hyperslab {
|
||||
start: starts,
|
||||
stride: vec![1; rank],
|
||||
count: vec![1; rank],
|
||||
block,
|
||||
}
|
||||
} else {
|
||||
Selection::None
|
||||
};
|
||||
Ok((sel, 16))
|
||||
Selection::Points(blocks_union_coords(rank, &starts, &ends)?)
|
||||
}
|
||||
}
|
||||
2 => decode_hyperslab_serialized(data, version),
|
||||
1 => Err(FormatError::ChunkedReadError(
|
||||
"VDS point selections are not supported".into(),
|
||||
)),
|
||||
_ => Err(FormatError::ChunkedReadError(
|
||||
"unknown dataspace selection type".into(),
|
||||
)),
|
||||
}
|
||||
};
|
||||
Ok((sel, len))
|
||||
}
|
||||
|
||||
/// Enumerate the selected element indices of a **1-D** dataspace of the
|
||||
@@ -314,6 +317,11 @@ impl Selection {
|
||||
"VDS selection rank does not match dataspace rank".into(),
|
||||
));
|
||||
}
|
||||
if count.iter().chain(block.iter()).any(|&v| v == UNLIMITED) {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"unlimited selection must be clipped before it is enumerated".into(),
|
||||
));
|
||||
}
|
||||
// Selected coordinates along each dimension, in order.
|
||||
let mut per_dim: Vec<Vec<u64>> = Vec::with_capacity(rank);
|
||||
for d in 0..rank {
|
||||
@@ -400,84 +408,279 @@ impl Selection {
|
||||
}
|
||||
}
|
||||
|
||||
/// Decode an `H5S_SEL_HYPER` selection in its serialized form. Only version-3
|
||||
/// **regular** hyperslabs are supported.
|
||||
fn decode_hyperslab_serialized(
|
||||
data: &[u8],
|
||||
version: u32,
|
||||
) -> Result<(Selection, usize), FormatError> {
|
||||
if version != 3 {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"only version-3 hyperslab selections are supported".into(),
|
||||
));
|
||||
/// Hyperslab count/block value meaning "unlimited" (`H5S_UNLIMITED`).
|
||||
pub const UNLIMITED: u64 = u64::MAX;
|
||||
|
||||
/// Largest number of elements an irregular selection is expanded to when it
|
||||
/// is converted to a point list by [`Selection::decode_serialized`].
|
||||
const MAX_EXPANDED_POINTS: u64 = 1 << 26;
|
||||
|
||||
/// A selection exactly as `H5S_select_serialize` stores it, before it is
|
||||
/// applied to any dataspace.
|
||||
///
|
||||
/// Unlike [`Selection`] this keeps an irregular hyperslab as its list of
|
||||
/// blocks, and a regular hyperslab's count/block may be [`UNLIMITED`] (the
|
||||
/// unlimited selections used by unlimited and "printf" virtual dataset
|
||||
/// mappings).
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub enum SerializedSelection {
|
||||
/// `H5S_SEL_ALL`.
|
||||
All,
|
||||
/// `H5S_SEL_NONE`.
|
||||
None,
|
||||
/// A regular hyperslab. `count[d]` or `block[d]` may be [`UNLIMITED`].
|
||||
Regular {
|
||||
start: Vec<u64>,
|
||||
stride: Vec<u64>,
|
||||
count: Vec<u64>,
|
||||
block: Vec<u64>,
|
||||
},
|
||||
/// An irregular hyperslab: the union of `starts.len() / rank` blocks, each
|
||||
/// given by its first (`starts`) and last (`ends`, inclusive) coordinate,
|
||||
/// flattened block-major.
|
||||
Blocks {
|
||||
rank: usize,
|
||||
starts: Vec<u64>,
|
||||
ends: Vec<u64>,
|
||||
},
|
||||
}
|
||||
|
||||
fn sel_err(msg: &str) -> FormatError {
|
||||
FormatError::ChunkedReadError(msg.into())
|
||||
}
|
||||
|
||||
/// Bounds-checked little-endian reader over a serialized selection.
|
||||
struct SelReader<'a> {
|
||||
data: &'a [u8],
|
||||
pos: usize,
|
||||
}
|
||||
|
||||
impl SelReader<'_> {
|
||||
fn take(&mut self, n: usize) -> Result<&[u8], FormatError> {
|
||||
let end = self.pos.checked_add(n).filter(|&e| e <= self.data.len());
|
||||
let end = end.ok_or(FormatError::UnexpectedEof {
|
||||
expected: self.pos.saturating_add(n),
|
||||
available: self.data.len(),
|
||||
})?;
|
||||
let s = &self.data[self.pos..end];
|
||||
self.pos = end;
|
||||
Ok(s)
|
||||
}
|
||||
// type(4) ver(4) flags(1) enc_size(1) rank(4) [start,stride,count,block]*rank
|
||||
if data.len() < 14 {
|
||||
return Err(FormatError::UnexpectedEof {
|
||||
expected: 14,
|
||||
available: data.len(),
|
||||
});
|
||||
|
||||
fn uint(&mut self, size: usize) -> Result<u64, FormatError> {
|
||||
let bytes = self.take(size)?;
|
||||
Ok(bytes
|
||||
.iter()
|
||||
.enumerate()
|
||||
.fold(0u64, |v, (i, &b)| v | (b as u64) << (i * 8)))
|
||||
}
|
||||
let flags = data[8];
|
||||
let enc_size = data[9] as usize;
|
||||
// Bit 0 set => regular hyperslab. Irregular hyperslabs list explicit blocks.
|
||||
if flags & 0x01 == 0 {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"irregular VDS hyperslab selections are not supported".into(),
|
||||
));
|
||||
|
||||
fn remaining(&self) -> usize {
|
||||
self.data.len() - self.pos
|
||||
}
|
||||
if enc_size != 2 && enc_size != 4 && enc_size != 8 {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"unsupported hyperslab coordinate encoding size".into(),
|
||||
));
|
||||
}
|
||||
let rank = u32::from_le_bytes([data[10], data[11], data[12], data[13]]) as usize;
|
||||
// HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything larger so a
|
||||
// corrupt rank can't drive a huge allocation or read loop.
|
||||
if rank > 32 {
|
||||
return Err(FormatError::ChunkedReadError(
|
||||
"hyperslab selection rank exceeds maximum (32)".into(),
|
||||
));
|
||||
}
|
||||
let mut pos = 14;
|
||||
let read_coord = |data: &[u8], pos: usize| -> Result<u64, FormatError> {
|
||||
if pos + enc_size > data.len() {
|
||||
return Err(FormatError::UnexpectedEof {
|
||||
expected: pos + enc_size,
|
||||
available: data.len(),
|
||||
});
|
||||
}
|
||||
|
||||
impl SerializedSelection {
|
||||
/// Decode a serialized selection, returning it and the number of bytes it
|
||||
/// occupies. Mirrors libhdf5's `H5S_select_deserialize`: `ALL`/`NONE` and
|
||||
/// hyperslab versions 1-3 are decoded; point selections (which libhdf5
|
||||
/// refuses in virtual datasets) and malformed input are errors.
|
||||
pub fn decode(data: &[u8]) -> Result<(SerializedSelection, usize), FormatError> {
|
||||
let mut r = SelReader { data, pos: 0 };
|
||||
let sel_type = r.uint(4)?;
|
||||
let version = r.uint(4)?;
|
||||
match sel_type {
|
||||
// ALL / NONE: type(4) + version(4) + reserved(4) + length(4).
|
||||
0 | 3 => {
|
||||
r.take(8)?;
|
||||
let sel = if sel_type == 3 {
|
||||
SerializedSelection::All
|
||||
} else {
|
||||
SerializedSelection::None
|
||||
};
|
||||
Ok((sel, r.pos))
|
||||
}
|
||||
2 => {
|
||||
let sel = decode_hyperslab(&mut r, version)?;
|
||||
Ok((sel, r.pos))
|
||||
}
|
||||
1 => Err(sel_err(
|
||||
"VDS point selections are not supported (libhdf5 rejects them too)",
|
||||
)),
|
||||
_ => Err(sel_err("unknown dataspace selection type")),
|
||||
}
|
||||
let mut v = 0u64;
|
||||
for (i, &b) in data[pos..pos + enc_size].iter().enumerate() {
|
||||
v |= (b as u64) << (i * 8);
|
||||
}
|
||||
|
||||
/// The single dimension in which this selection is unlimited, if any.
|
||||
pub fn unlimited_dim(&self) -> Option<usize> {
|
||||
match self {
|
||||
SerializedSelection::Regular { count, block, .. } => count
|
||||
.iter()
|
||||
.zip(block)
|
||||
.position(|(&c, &b)| c == UNLIMITED || b == UNLIMITED),
|
||||
_ => None,
|
||||
}
|
||||
Ok(v)
|
||||
}
|
||||
|
||||
/// The rank the selection was serialized with (`None` for ALL/NONE, which
|
||||
/// carry no rank).
|
||||
pub fn rank(&self) -> Option<usize> {
|
||||
match self {
|
||||
SerializedSelection::Regular { start, .. } => Some(start.len()),
|
||||
SerializedSelection::Blocks { rank, .. } => Some(*rank),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `H5S__hyper_deserialize`: after the type and version words.
|
||||
fn decode_hyperslab(r: &mut SelReader, version: u64) -> Result<SerializedSelection, FormatError> {
|
||||
const REGULAR: u8 = 0x01;
|
||||
let (flags, enc_size) = match version {
|
||||
// v1: reserved(4) + length(4), always irregular, 4-byte coordinates.
|
||||
1 => {
|
||||
r.take(8)?;
|
||||
(0u8, 4usize)
|
||||
}
|
||||
// v2: flags(1) + length(4), 8-byte coordinates.
|
||||
2 => {
|
||||
let flags = r.take(1)?[0];
|
||||
r.take(4)?;
|
||||
(flags, 8)
|
||||
}
|
||||
// v3: flags(1) + encoding size(1).
|
||||
3 => {
|
||||
let flags = r.take(1)?[0];
|
||||
let enc = r.take(1)?[0] as usize;
|
||||
(flags, enc)
|
||||
}
|
||||
_ => return Err(sel_err("unsupported hyperslab selection version")),
|
||||
};
|
||||
let (mut start, mut stride, mut count, mut block) = (
|
||||
Vec::with_capacity(rank),
|
||||
Vec::with_capacity(rank),
|
||||
Vec::with_capacity(rank),
|
||||
Vec::with_capacity(rank),
|
||||
);
|
||||
for _ in 0..rank {
|
||||
start.push(read_coord(data, pos)?);
|
||||
pos += enc_size;
|
||||
stride.push(read_coord(data, pos)?);
|
||||
pos += enc_size;
|
||||
count.push(read_coord(data, pos)?);
|
||||
pos += enc_size;
|
||||
block.push(read_coord(data, pos)?);
|
||||
pos += enc_size;
|
||||
if flags & !REGULAR != 0 {
|
||||
return Err(sel_err("unknown hyperslab selection flags"));
|
||||
}
|
||||
Ok((
|
||||
Selection::Hyperslab {
|
||||
if !matches!(enc_size, 2 | 4 | 8) {
|
||||
return Err(sel_err("unsupported hyperslab coordinate encoding size"));
|
||||
}
|
||||
let rank = r.uint(4)? as usize;
|
||||
// HDF5 caps dataspace rank at 32 (H5S_MAX_RANK). Reject anything else so a
|
||||
// corrupt rank can't drive a huge allocation or read loop.
|
||||
if rank == 0 || rank > 32 {
|
||||
return Err(sel_err("hyperslab selection rank must be 1..=32"));
|
||||
}
|
||||
// The all-ones value of the encoding width means "unlimited".
|
||||
let unlim_raw = if enc_size == 8 {
|
||||
u64::MAX
|
||||
} else {
|
||||
(1u64 << (enc_size * 8)) - 1
|
||||
};
|
||||
|
||||
if flags & REGULAR != 0 {
|
||||
let (mut start, mut stride, mut count, mut block) = (
|
||||
Vec::with_capacity(rank),
|
||||
Vec::with_capacity(rank),
|
||||
Vec::with_capacity(rank),
|
||||
Vec::with_capacity(rank),
|
||||
);
|
||||
for _ in 0..rank {
|
||||
start.push(r.uint(enc_size)?);
|
||||
stride.push(r.uint(enc_size)?);
|
||||
let c = r.uint(enc_size)?;
|
||||
count.push(if c == unlim_raw { UNLIMITED } else { c });
|
||||
let b = r.uint(enc_size)?;
|
||||
block.push(if b == unlim_raw { UNLIMITED } else { b });
|
||||
}
|
||||
let unlimited = count
|
||||
.iter()
|
||||
.zip(&block)
|
||||
.filter(|&(&c, &b)| c == UNLIMITED || b == UNLIMITED)
|
||||
.count();
|
||||
if unlimited > 1 {
|
||||
return Err(sel_err(
|
||||
"hyperslab selection is unlimited in more than one dimension",
|
||||
));
|
||||
}
|
||||
for d in 0..rank {
|
||||
// Overlapping blocks are not a valid regular hyperslab.
|
||||
if count[d] > 1 && block[d] != UNLIMITED && block[d] > stride[d] {
|
||||
return Err(sel_err("regular hyperslab blocks overlap"));
|
||||
}
|
||||
}
|
||||
return Ok(SerializedSelection::Regular {
|
||||
start,
|
||||
stride,
|
||||
count,
|
||||
block,
|
||||
},
|
||||
pos,
|
||||
))
|
||||
});
|
||||
}
|
||||
|
||||
// Irregular: number of blocks, then each block's start and end corners.
|
||||
let nblocks = r.uint(enc_size)?;
|
||||
let per_block = (rank * 2 * enc_size) as u64;
|
||||
// Untrusted count: it must fit in what is left of the buffer.
|
||||
if nblocks
|
||||
.checked_mul(per_block)
|
||||
.is_none_or(|need| need > r.remaining() as u64)
|
||||
{
|
||||
return Err(FormatError::UnexpectedEof {
|
||||
expected: r
|
||||
.pos
|
||||
.saturating_add(nblocks.saturating_mul(per_block) as usize),
|
||||
available: r.data.len(),
|
||||
});
|
||||
}
|
||||
let n = nblocks as usize * rank;
|
||||
let (mut starts, mut ends) = (Vec::with_capacity(n), Vec::with_capacity(n));
|
||||
for _ in 0..nblocks {
|
||||
for _ in 0..rank {
|
||||
starts.push(r.uint(enc_size)?);
|
||||
}
|
||||
for _ in 0..rank {
|
||||
ends.push(r.uint(enc_size)?);
|
||||
}
|
||||
}
|
||||
if starts.iter().zip(&ends).any(|(s, e)| e < s) {
|
||||
return Err(sel_err("hyperslab block ends before it starts"));
|
||||
}
|
||||
Ok(SerializedSelection::Blocks { rank, starts, ends })
|
||||
}
|
||||
|
||||
/// The coordinates of the union of the given blocks, in row-major order.
|
||||
fn blocks_union_coords(
|
||||
rank: usize,
|
||||
starts: &[u64],
|
||||
ends: &[u64],
|
||||
) -> Result<Vec<Vec<u64>>, FormatError> {
|
||||
let mut total = 0u64;
|
||||
for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
|
||||
let vol = s
|
||||
.iter()
|
||||
.zip(e)
|
||||
.try_fold(1u64, |acc, (&s, &e)| acc.checked_mul(e - s + 1));
|
||||
total = vol
|
||||
.and_then(|v| total.checked_add(v))
|
||||
.filter(|&t| t <= MAX_EXPANDED_POINTS)
|
||||
.ok_or_else(|| sel_err("irregular hyperslab selection is too large to expand"))?;
|
||||
}
|
||||
let mut out = Vec::with_capacity(total as usize);
|
||||
for (s, e) in starts.chunks_exact(rank).zip(ends.chunks_exact(rank)) {
|
||||
let mut cur = s.to_vec();
|
||||
'block: loop {
|
||||
out.push(cur.clone());
|
||||
for d in (0..rank).rev() {
|
||||
if cur[d] < e[d] {
|
||||
cur[d] += 1;
|
||||
continue 'block;
|
||||
}
|
||||
cur[d] = s[d];
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
// Lexicographic order of coordinates is row-major order.
|
||||
out.sort_unstable();
|
||||
out.dedup();
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -642,11 +845,100 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_irregular_hyperslab_rejected() {
|
||||
fn decode_truncated_irregular_hyperslab_is_error() {
|
||||
// Irregular, rank 1, but the block count is missing.
|
||||
let bytes = [0x02u8, 0, 0, 0, 0x03, 0, 0, 0, 0x00, 0x02, 0x01, 0, 0, 0];
|
||||
assert!(Selection::decode_serialized(&bytes).is_err());
|
||||
}
|
||||
|
||||
/// Version 1 as libhdf5 writes it for the default (earliest) format bounds:
|
||||
/// type, version, reserved(4), length(4), rank(4), nblocks(4), then each
|
||||
/// block's start and inclusive end corner as 4-byte values.
|
||||
fn v1_blocks(rank: u32, blocks: &[(&[u32], &[u32])]) -> Vec<u8> {
|
||||
let mut b = Vec::new();
|
||||
for w in [2u32, 1, 0, 0, rank, blocks.len() as u32] {
|
||||
b.extend_from_slice(&w.to_le_bytes());
|
||||
}
|
||||
for (s, e) in blocks {
|
||||
for v in s.iter().chain(e.iter()) {
|
||||
b.extend_from_slice(&v.to_le_bytes());
|
||||
}
|
||||
}
|
||||
b
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_v1_irregular_single_block() {
|
||||
// Exactly what h5py/HDF5 2.0 writes for `[0:4]` with default libver.
|
||||
let bytes = v1_blocks(1, &[(&[0], &[3])]);
|
||||
let (sel, used) = Selection::decode_serialized(&bytes).unwrap();
|
||||
assert_eq!(used, bytes.len());
|
||||
assert_eq!(sel.iter_linear_1d(8).unwrap(), vec![0, 1, 2, 3]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_v1_irregular_union_is_row_major() {
|
||||
// Blocks given out of order and overlapping still enumerate once each,
|
||||
// in row-major order (libhdf5 iterates the union, not the list).
|
||||
let bytes = v1_blocks(2, &[(&[1, 0], &[1, 1]), (&[0, 2], &[1, 2])]);
|
||||
let (sel, used) = Selection::decode_serialized(&bytes).unwrap();
|
||||
assert_eq!(used, bytes.len());
|
||||
// (0,2) (1,0) (1,1) (1,2) in a 2x3 space.
|
||||
assert_eq!(sel.iter_linear(&[2, 3]).unwrap(), vec![2, 3, 4, 5]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_v2_regular_with_unlimited_count() {
|
||||
// v2: flags(1) + length(4), then 8-byte start/stride/count/block.
|
||||
let mut b = Vec::new();
|
||||
b.extend_from_slice(&2u32.to_le_bytes());
|
||||
b.extend_from_slice(&2u32.to_le_bytes());
|
||||
b.push(0x01);
|
||||
b.extend_from_slice(&36u32.to_le_bytes());
|
||||
b.extend_from_slice(&1u32.to_le_bytes());
|
||||
for v in [0u64, 10, u64::MAX, 10] {
|
||||
b.extend_from_slice(&v.to_le_bytes());
|
||||
}
|
||||
let (raw, used) = SerializedSelection::decode(&b).unwrap();
|
||||
assert_eq!(used, b.len());
|
||||
assert_eq!(raw.unlimited_dim(), Some(0));
|
||||
assert_eq!(
|
||||
raw,
|
||||
SerializedSelection::Regular {
|
||||
start: vec![0],
|
||||
stride: vec![10],
|
||||
count: vec![UNLIMITED],
|
||||
block: vec![10],
|
||||
}
|
||||
);
|
||||
// An unclipped unlimited selection cannot be enumerated.
|
||||
let (sel, _) = Selection::decode_serialized(&b).unwrap();
|
||||
assert!(sel.iter_linear_1d(100).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_v3_two_byte_all_ones_is_unlimited() {
|
||||
let bytes = [
|
||||
0x02, 0, 0, 0, 0x03, 0, 0, 0, 0x01, 0x02, 0x01, 0, 0, 0, //
|
||||
0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0xFF, 0xFF,
|
||||
];
|
||||
let (raw, _) = SerializedSelection::decode(&bytes).unwrap();
|
||||
assert_eq!(raw.unlimited_dim(), Some(0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_irregular_block_count_beyond_buffer_is_error() {
|
||||
let mut b = v1_blocks(1, &[(&[0], &[3])]);
|
||||
b[20..24].copy_from_slice(&u32::MAX.to_le_bytes());
|
||||
assert!(Selection::decode_serialized(&b).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decode_point_selection_is_refused() {
|
||||
let bytes = [1u8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
|
||||
assert!(Selection::decode_serialized(&bytes).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn iter_linear_2d_block_row_major() {
|
||||
// A 2x2 block at the top-left of a 4x4 space => linear 0,1,4,5.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -83,6 +83,45 @@ fn read_chunked_dataset(file_data: &[u8], dataset_path: &str) -> (Vec<u8>, Datat
|
||||
(raw, datatype, dataspace)
|
||||
}
|
||||
|
||||
/// Helper: read a virtual dataset with `vds::read_virtual_dataset`, giving it
|
||||
/// the dataset's own fill value (same-file sources only).
|
||||
fn read_virtual_fixture(file_data: &[u8], path: &str) -> (Vec<u8>, Datatype) {
|
||||
let sig = find_signature(file_data).unwrap();
|
||||
let sb = Superblock::parse(file_data, sig).unwrap();
|
||||
let addr = resolve_path_any(file_data, &sb, path).unwrap();
|
||||
let hdr =
|
||||
ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
|
||||
let msg = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t).unwrap();
|
||||
let ds = Dataspace::parse(&msg(MessageType::Dataspace).data, sb.length_size).unwrap();
|
||||
let (dt, _) = Datatype::parse(&msg(MessageType::Datatype).data).unwrap();
|
||||
let layout = DataLayout::parse(
|
||||
&msg(MessageType::DataLayout).data,
|
||||
sb.offset_size,
|
||||
sb.length_size,
|
||||
)
|
||||
.unwrap();
|
||||
let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
|
||||
file_data,
|
||||
&hdr.messages,
|
||||
sb.offset_size,
|
||||
sb.length_size,
|
||||
)
|
||||
.unwrap();
|
||||
let v = clawhdf5_format::vds::read_virtual_dataset(
|
||||
file_data,
|
||||
&layout,
|
||||
&ds,
|
||||
&dt,
|
||||
fill.as_deref(),
|
||||
sb.offset_size,
|
||||
sb.length_size,
|
||||
None,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(v.dims, ds.dimensions);
|
||||
(v.data, dt)
|
||||
}
|
||||
|
||||
/// Helper: read any dataset (contiguous or chunked) as f64.
|
||||
fn read_dataset_f64_any(bytes: &[u8], path: &str) -> Vec<f64> {
|
||||
let sig = find_signature(bytes).unwrap();
|
||||
@@ -672,7 +711,7 @@ fn v4_virtual_dataset_same_file_read() {
|
||||
// virt[4:8] <- (unmapped) => fill 0
|
||||
// virt[8:12] <- src_b[0:4] (ALL) => 20,21,22,23
|
||||
let file_data = include_bytes!("fixtures/vds_same_file.h5");
|
||||
let (raw, datatype, _) = read_chunked_dataset(file_data, "virt");
|
||||
let (raw, datatype) = read_virtual_fixture(file_data, "virt");
|
||||
let values = read_as_i32(&raw, &datatype).unwrap();
|
||||
assert_eq!(
|
||||
values,
|
||||
@@ -681,6 +720,38 @@ fn v4_virtual_dataset_same_file_read() {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn v4_virtual_dataset_raw_api_refuses_to_guess_the_fill_value() {
|
||||
// The raw read API has no fill value message, so a virtual dataset with an
|
||||
// unmapped region is an error there instead of zeros that may be wrong.
|
||||
let file_data = include_bytes!("fixtures/vds_same_file.h5");
|
||||
let sig = find_signature(file_data).unwrap();
|
||||
let sb = Superblock::parse(file_data, sig).unwrap();
|
||||
let addr = resolve_path_any(file_data, &sb, "virt").unwrap();
|
||||
let hdr =
|
||||
ObjectHeader::parse(file_data, addr as usize, sb.offset_size, sb.length_size).unwrap();
|
||||
let msg = |t: MessageType| hdr.messages.iter().find(|m| m.msg_type == t).unwrap();
|
||||
let ds = Dataspace::parse(&msg(MessageType::Dataspace).data, sb.length_size).unwrap();
|
||||
let (dt, _) = Datatype::parse(&msg(MessageType::Datatype).data).unwrap();
|
||||
let layout = DataLayout::parse(
|
||||
&msg(MessageType::DataLayout).data,
|
||||
sb.offset_size,
|
||||
sb.length_size,
|
||||
)
|
||||
.unwrap();
|
||||
let err = read_raw_data_full(
|
||||
file_data,
|
||||
&layout,
|
||||
&ds,
|
||||
&dt,
|
||||
None,
|
||||
sb.offset_size,
|
||||
sb.length_size,
|
||||
)
|
||||
.unwrap_err();
|
||||
assert!(err.to_string().contains("fill value"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn v4_virtual_dataset_2d_same_file_read() {
|
||||
// A 4x4 virtual dataset assembled from two 2x2 same-file sources placed as
|
||||
@@ -689,7 +760,7 @@ fn v4_virtual_dataset_2d_same_file_read() {
|
||||
// virt[2:4,2:4] <- src_b = [[5,6],[7,8]]
|
||||
// everything else -> fill 0
|
||||
let file_data = include_bytes!("fixtures/vds_2d_same_file.h5");
|
||||
let (raw, datatype, _) = read_chunked_dataset(file_data, "virt");
|
||||
let (raw, datatype) = read_virtual_fixture(file_data, "virt");
|
||||
let values = read_as_i32(&raw, &datatype).unwrap();
|
||||
assert_eq!(
|
||||
values,
|
||||
|
||||
@@ -522,6 +522,7 @@ impl<'f> Dataset<'f> {
|
||||
self.file.length_size(),
|
||||
)?;
|
||||
let fill_matters = !clawhdf5_format::fill_value::has_storage(&dl)
|
||||
|| matches!(dl, DataLayout::Virtual { .. })
|
||||
|| (matches!(dl, DataLayout::Chunked { .. })
|
||||
&& !clawhdf5_format::fill_value::is_default(fill.as_deref()));
|
||||
if fill_matters {
|
||||
@@ -855,7 +856,22 @@ impl<'f> Dataset<'f> {
|
||||
|
||||
fn dataspace(&self) -> Result<Dataspace, Error> {
|
||||
let data = self.required_payload(MessageType::Dataspace)?;
|
||||
Ok(Dataspace::parse(&data, self.file.length_size())?)
|
||||
let mut ds = Dataspace::parse(&data, self.file.length_size())?;
|
||||
// libhdf5 reports a virtual dataset with unlimited or printf-style
|
||||
// mappings at the extent its sources currently fill, not the stored
|
||||
// one (`H5Dget_space`).
|
||||
if let Ok(dl @ DataLayout::Virtual { .. }) = self.data_layout() {
|
||||
let resolver = self.vds_resolver();
|
||||
ds.dimensions = clawhdf5_format::vds::virtual_dataset_extent(
|
||||
self.file.data.as_bytes(),
|
||||
&dl,
|
||||
&ds,
|
||||
self.file.offset_size(),
|
||||
self.file.length_size(),
|
||||
Some(&resolver),
|
||||
)?;
|
||||
}
|
||||
Ok(ds)
|
||||
}
|
||||
|
||||
fn data_layout(&self) -> Result<DataLayout, Error> {
|
||||
@@ -884,24 +900,9 @@ impl<'f> Dataset<'f> {
|
||||
let pipeline = self.filter_pipeline()?;
|
||||
|
||||
// Virtual datasets are assembled from source datasets; the per-file
|
||||
// chunk cache does not apply. Route them through the resolver path so
|
||||
// external sibling files resolve relative to this file's directory.
|
||||
// chunk cache does not apply.
|
||||
if matches!(dl, DataLayout::Virtual { .. }) {
|
||||
let base_dir = self.file.base_dir.clone();
|
||||
let resolver = move |name: &str| -> Option<Vec<u8>> {
|
||||
let dir = base_dir.as_ref()?;
|
||||
std::fs::read(dir.join(sibling_file_name(name)?)).ok()
|
||||
};
|
||||
return Ok(data_read::read_raw_data_full_with_resolver(
|
||||
self.file.data.as_bytes(),
|
||||
&dl,
|
||||
&ds,
|
||||
&dt,
|
||||
pipeline.as_ref(),
|
||||
self.file.offset_size(),
|
||||
self.file.length_size(),
|
||||
Some(&resolver),
|
||||
)?);
|
||||
return self.read_virtual(&dl, &ds, &dt);
|
||||
}
|
||||
|
||||
// Unallocated storage reads as the dataset's fill value.
|
||||
@@ -927,6 +928,62 @@ impl<'f> Dataset<'f> {
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
/// Resolver for external Virtual Dataset source files: names are
|
||||
/// resolved against the directory of the file that holds the virtual
|
||||
/// dataset, as libhdf5 does. A missing file is `Ok(None)` (its mappings
|
||||
/// read as the fill value); a name that would leave that directory is
|
||||
/// refused with an error rather than read as fill.
|
||||
fn vds_resolver(&self) -> impl Fn(&str) -> Result<Option<Vec<u8>>, FormatError> + use<> {
|
||||
let base_dir = self.file.base_dir.clone();
|
||||
move |name: &str| {
|
||||
let Some(dir) = base_dir.as_ref() else {
|
||||
return Err(FormatError::ChunkedReadError(format!(
|
||||
"virtual dataset source file {name:?} cannot be resolved for an in-memory file"
|
||||
)));
|
||||
};
|
||||
let rel = sibling_file_name(name).ok_or_else(|| {
|
||||
FormatError::ChunkedReadError(format!(
|
||||
"virtual dataset source file {name:?} is outside the virtual file's \
|
||||
directory and is not followed"
|
||||
))
|
||||
})?;
|
||||
match std::fs::read(dir.join(rel)) {
|
||||
Ok(bytes) => Ok(Some(bytes)),
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
|
||||
Err(e) => Err(FormatError::ChunkedReadError(format!(
|
||||
"cannot read virtual dataset source file {name:?}: {e}"
|
||||
))),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Read a whole virtual dataset; unmapped elements hold its fill value.
|
||||
fn read_virtual(
|
||||
&self,
|
||||
dl: &DataLayout,
|
||||
ds: &Dataspace,
|
||||
dt: &Datatype,
|
||||
) -> Result<Vec<u8>, Error> {
|
||||
let fill = clawhdf5_format::fill_value::dataset_fill_value_in(
|
||||
self.file.data.as_bytes(),
|
||||
&self.header.messages,
|
||||
self.file.offset_size(),
|
||||
self.file.length_size(),
|
||||
)?;
|
||||
let resolver = self.vds_resolver();
|
||||
let v = clawhdf5_format::vds::read_virtual_dataset(
|
||||
self.file.data.as_bytes(),
|
||||
dl,
|
||||
ds,
|
||||
dt,
|
||||
fill.as_deref(),
|
||||
self.file.offset_size(),
|
||||
self.file.length_size(),
|
||||
Some(&resolver),
|
||||
)?;
|
||||
Ok(v.data)
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
+14
@@ -0,0 +1,14 @@
|
||||
# VDS test files from libhdf5
|
||||
|
||||
Copied unchanged from the HDF Group's HDF5 repository,
|
||||
`tools/test/testfiles/vds/` (the h5dump/h5ls VDS test data). HDF5 is
|
||||
distributed under a BSD-style license (see `COPYING` in the HDF5 source).
|
||||
|
||||
| File | What it exercises |
|
||||
|---|---|
|
||||
| `vds-eiger.h5` + `f-0.h5`, `f-3.h5` | printf-style source name `f-%b.h5`; `f-3.h5` lies past the first missing source and must be ignored (extent 5, not 20) |
|
||||
| `4_vds.h5` + `4_0.h5`..`4_2.h5` | printf-style `4_%b.h5` with version-2 (1.10 format) hyperslab selections |
|
||||
| `vds-percival-unlim-maxmin.h5` + `a.h5`..`d.h5` | four interleaved unlimited mappings whose sources have different lengths |
|
||||
|
||||
Used by `crates/clawhdf5/tests/vds_interop.rs::vds_libhdf5_test_files`,
|
||||
which compares our reads with h5py's.
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,450 @@
|
||||
//! Virtual Dataset (VDS) reads checked against libhdf5 (through h5py).
|
||||
//!
|
||||
//! Each test has h5py build virtual datasets and their source files in a temp
|
||||
//! directory, record what libhdf5 reads back (shape and values) next to them,
|
||||
//! and then compares that with what clawhdf5 reads from the same files.
|
||||
//!
|
||||
//! Skipped when python3 or h5py is unavailable, unless
|
||||
//! `CLAWHDF5_REQUIRE_INTEROP=1`.
|
||||
|
||||
use std::path::Path;
|
||||
use std::process::Command;
|
||||
|
||||
use clawhdf5::File;
|
||||
|
||||
/// The Python interpreter to drive interop checks with (`CLAWHDF5_PYTHON`
|
||||
/// lets these run against a virtualenv holding h5py).
|
||||
fn python() -> String {
|
||||
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
|
||||
}
|
||||
|
||||
/// When `CLAWHDF5_REQUIRE_INTEROP=1` (set in CI), a missing Python dependency
|
||||
/// is a test failure instead of a silent skip.
|
||||
fn interop_required() -> bool {
|
||||
std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
|
||||
}
|
||||
|
||||
fn python_available() -> bool {
|
||||
Command::new(python())
|
||||
.args(["-c", "import h5py, numpy"])
|
||||
.output()
|
||||
.map(|o| o.status.success())
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
macro_rules! skip_if_no_python {
|
||||
() => {
|
||||
if !python_available() {
|
||||
assert!(
|
||||
!interop_required(),
|
||||
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py is not available"
|
||||
);
|
||||
eprintln!("SKIP: python3 with h5py not available");
|
||||
return;
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/// Prelude for every generator script: `expect(file, dset, tag)` records what
|
||||
/// libhdf5 reads for `file:dset` as `<tag>.expect` (shape line, values line).
|
||||
const PRELUDE: &str = r#"
|
||||
import h5py, numpy as np
|
||||
def expect(fn, dset, tag):
|
||||
with h5py.File(fn, "r") as f:
|
||||
d = f[dset]
|
||||
a = d[...]
|
||||
with open(tag + ".expect", "w") as out:
|
||||
out.write(" ".join(str(n) for n in d.shape) + "\n")
|
||||
out.write(" ".join(repr(float(v)) for v in a.ravel()) + "\n")
|
||||
"#;
|
||||
|
||||
/// Run `body` (after [`PRELUDE`]) with `dir` as the working directory, so
|
||||
/// relative source file names land next to the virtual file.
|
||||
fn generate(dir: &Path, body: &str) {
|
||||
let script = format!("{PRELUDE}\n{body}");
|
||||
let out = Command::new(python())
|
||||
.args(["-c", &script])
|
||||
.current_dir(dir)
|
||||
.output()
|
||||
.expect("failed to run python");
|
||||
assert!(
|
||||
out.status.success(),
|
||||
"generator failed:\nSTDOUT: {}\nSTDERR: {}",
|
||||
String::from_utf8_lossy(&out.stdout),
|
||||
String::from_utf8_lossy(&out.stderr)
|
||||
);
|
||||
}
|
||||
|
||||
/// What libhdf5 read for `tag`: (shape, values as f64).
|
||||
fn expected(dir: &Path, tag: &str) -> (Vec<u64>, Vec<f64>) {
|
||||
let text = std::fs::read_to_string(dir.join(format!("{tag}.expect"))).unwrap();
|
||||
let mut lines = text.lines();
|
||||
let parse_line = |l: Option<&str>| -> Vec<String> {
|
||||
l.unwrap_or("")
|
||||
.split_whitespace()
|
||||
.map(str::to_string)
|
||||
.collect()
|
||||
};
|
||||
let shape = parse_line(lines.next())
|
||||
.iter()
|
||||
.map(|s| s.parse().unwrap())
|
||||
.collect();
|
||||
let values = parse_line(lines.next())
|
||||
.iter()
|
||||
.map(|s| s.parse().unwrap())
|
||||
.collect();
|
||||
(shape, values)
|
||||
}
|
||||
|
||||
/// Assert clawhdf5 reads `file:dset` exactly as libhdf5 did for `tag`.
|
||||
fn assert_matches_libhdf5(dir: &Path, file: &str, dset: &str, tag: &str) {
|
||||
let (shape, values) = expected(dir, tag);
|
||||
let f = File::open(dir.join(file)).unwrap();
|
||||
let ds = f.dataset(dset).unwrap();
|
||||
assert_eq!(
|
||||
ds.shape().unwrap(),
|
||||
shape,
|
||||
"{tag}: shape differs from libhdf5"
|
||||
);
|
||||
let got = ds
|
||||
.read_f64()
|
||||
.unwrap_or_else(|e| panic!("{tag}: read failed: {e}"));
|
||||
assert_eq!(got.len(), values.len(), "{tag}: element count differs");
|
||||
for (i, (g, e)) in got.iter().zip(&values).enumerate() {
|
||||
assert!(
|
||||
g == e || (g.is_nan() && e.is_nan()),
|
||||
"{tag}: element {i} is {g}, libhdf5 reads {e}\n ours: {got:?}\n libhdf5: {values:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Selection encodings
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Files written with the default (earliest) format bounds serialize every
|
||||
/// VDS hyperslab as a version-1 *irregular* selection (4-byte block corners),
|
||||
/// and a strided selection as many blocks. These were refused outright.
|
||||
#[test]
|
||||
fn vds_version1_irregular_hyperslab_selections() {
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
generate(
|
||||
dir.path(),
|
||||
r#"
|
||||
with h5py.File("src.h5", "w") as s:
|
||||
s.create_dataset("a", data=np.arange(12.0))
|
||||
s.create_dataset("m", data=np.arange(20.0).reshape(4, 5))
|
||||
with h5py.File("v1.h5", "w") as f: # default libver: hyperslab version 1
|
||||
f.create_dataset("local", data=np.arange(10.0) * -1)
|
||||
lay = h5py.VirtualLayout(shape=(12,), dtype="f8")
|
||||
lay[0:4] = h5py.VirtualSource(".", "local", shape=(10,))[2:6]
|
||||
lay[4:10] = h5py.VirtualSource("src.h5", "a", shape=(12,))[::2]
|
||||
lay[10:12] = h5py.VirtualSource("src.h5", "a", shape=(12,))[10:12]
|
||||
f.create_virtual_dataset("strided", lay)
|
||||
lay = h5py.VirtualLayout(shape=(4, 6), dtype="f8")
|
||||
lay[:, 0:2] = h5py.VirtualSource("src.h5", "m", shape=(4, 5))[:, 3:5]
|
||||
lay[:, 2:6:2] = h5py.VirtualSource("src.h5", "m", shape=(4, 5))[:, 0:2]
|
||||
lay[:, 3:6:2] = h5py.VirtualSource("src.h5", "m", shape=(4, 5))[:, 2:4]
|
||||
f.create_virtual_dataset("grid", lay)
|
||||
expect("v1.h5", "strided", "strided")
|
||||
expect("v1.h5", "grid", "grid")
|
||||
"#,
|
||||
);
|
||||
assert_matches_libhdf5(dir.path(), "v1.h5", "strided", "strided");
|
||||
assert_matches_libhdf5(dir.path(), "v1.h5", "grid", "grid");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Mapping list encoding
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// With a 2.0 low version bound libhdf5 writes the mapping list as block
|
||||
/// version 1: a flags byte per entry, and repeated names stored as the index
|
||||
/// of the entry that first spelled them out. The flags byte was mistaken for
|
||||
/// an empty (same-file) name.
|
||||
#[test]
|
||||
fn vds_mapping_block_version1_shared_names() {
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
generate(
|
||||
dir.path(),
|
||||
r#"
|
||||
name = "a_rather_long_dataset_name"
|
||||
with h5py.File("a_rather_long_source_file.h5", "w") as s:
|
||||
s.create_dataset(name, data=np.arange(12.0) + 100)
|
||||
with h5py.File("shared.h5", "w", libver=("v200", "v200")) as f:
|
||||
f.create_dataset(name, data=np.arange(12.0) * -1)
|
||||
lay = h5py.VirtualLayout(shape=(4, 4), dtype="f8")
|
||||
for i in range(3):
|
||||
src = h5py.VirtualSource("a_rather_long_source_file.h5", name, shape=(12,))
|
||||
lay[i] = src[4 * i:4 * i + 4]
|
||||
lay[3] = h5py.VirtualSource(".", name, shape=(12,))[0:4]
|
||||
f.create_virtual_dataset("v", lay)
|
||||
# the heap block must really be version 1 for this test to mean anything
|
||||
raw = open("shared.h5", "rb").read()
|
||||
gcol = raw.index(b"GCOL")
|
||||
assert raw[gcol + 32] == 1, "expected a version-1 VDS mapping block"
|
||||
expect("shared.h5", "v", "shared")
|
||||
"#,
|
||||
);
|
||||
assert_matches_libhdf5(dir.path(), "shared.h5", "v", "shared");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Fill value
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Elements no mapping supplies read as the virtual dataset's fill value, not
|
||||
/// as 0: unmapped regions, a missing source file, a missing source dataset.
|
||||
/// A source's own unallocated chunks read as *its* fill value.
|
||||
#[test]
|
||||
fn vds_unmapped_regions_read_as_fill_value() {
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
generate(
|
||||
dir.path(),
|
||||
r#"
|
||||
for i in range(3):
|
||||
with h5py.File(f"src_{i}.h5", "w") as s:
|
||||
s.create_dataset("data", data=np.arange(10.0) + i * 100)
|
||||
with h5py.File("sparse_src.h5", "w") as s:
|
||||
d = s.create_dataset("data", shape=(10,), chunks=(5,), dtype="f8", fillvalue=42.0)
|
||||
d[0:5] = np.arange(5.0) + 1000 # the second chunk is never written
|
||||
for libver in ["earliest", "latest"]:
|
||||
with h5py.File(f"fill_{libver}.h5", "w", libver=libver) as f:
|
||||
f.create_dataset("local", data=np.arange(10.0) * -1)
|
||||
lay = h5py.VirtualLayout(shape=(6, 10), dtype="f8")
|
||||
for i in range(3):
|
||||
lay[i] = h5py.VirtualSource(f"src_{i}.h5", "data", shape=(10,))
|
||||
lay[3] = h5py.VirtualSource("no_such_file.h5", "data", shape=(10,))
|
||||
lay[4] = h5py.VirtualSource("src_0.h5", "no_such_dataset", shape=(10,))
|
||||
# row 5 is not mapped at all
|
||||
f.create_virtual_dataset("files", lay, fillvalue=-1.0)
|
||||
lay = h5py.VirtualLayout(shape=(20,), dtype="f8")
|
||||
lay[0:10] = h5py.VirtualSource(".", "local", shape=(10,))
|
||||
f.create_virtual_dataset("same_file", lay, fillvalue=7.0)
|
||||
lay = h5py.VirtualLayout(shape=(12,), dtype="f8")
|
||||
lay[1:11] = h5py.VirtualSource("sparse_src.h5", "data", shape=(10,))
|
||||
f.create_virtual_dataset("sparse_source", lay, fillvalue=-3.5)
|
||||
lay = h5py.VirtualLayout(shape=(3, 4), dtype="i4")
|
||||
lay[1, :] = h5py.VirtualSource(".", "ints", shape=(4,))
|
||||
f.create_dataset("ints", data=np.arange(4, dtype="i4") + 1)
|
||||
f.create_virtual_dataset("int_fill", lay, fillvalue=-99)
|
||||
for name in ["files", "same_file", "sparse_source", "int_fill"]:
|
||||
expect(f"fill_{libver}.h5", name, f"{name}_{libver}")
|
||||
"#,
|
||||
);
|
||||
for libver in ["earliest", "latest"] {
|
||||
let file = format!("fill_{libver}.h5");
|
||||
for name in ["files", "same_file", "sparse_source", "int_fill"] {
|
||||
assert_matches_libhdf5(dir.path(), &file, name, &format!("{name}_{libver}"));
|
||||
}
|
||||
}
|
||||
|
||||
// A selection read goes through the same fill-aware assembly.
|
||||
let f = File::open(dir.path().join("fill_latest.h5")).unwrap();
|
||||
let sel = clawhdf5::Selection::slice(std::slice::from_ref(&(8..14)));
|
||||
let got = f
|
||||
.dataset("same_file")
|
||||
.unwrap()
|
||||
.read_f64_selection(&sel)
|
||||
.unwrap();
|
||||
assert_eq!(got, vec![-8.0, -9.0, 7.0, 7.0, 7.0, 7.0]);
|
||||
}
|
||||
|
||||
/// A source name that would leave the virtual file's directory is refused
|
||||
/// with an error; it used to be skipped and read silently as fill.
|
||||
#[test]
|
||||
fn vds_source_outside_directory_is_an_error_not_fill() {
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
std::fs::create_dir(dir.path().join("sub")).unwrap();
|
||||
generate(
|
||||
dir.path(),
|
||||
r#"
|
||||
with h5py.File("src.h5", "w") as s:
|
||||
s.create_dataset("data", data=np.arange(4.0))
|
||||
with h5py.File("sub/up.h5", "w", libver="latest") as f:
|
||||
lay = h5py.VirtualLayout(shape=(4,), dtype="f8")
|
||||
lay[:] = h5py.VirtualSource("../src.h5", "data", shape=(4,))
|
||||
f.create_virtual_dataset("v", lay, fillvalue=-1.0)
|
||||
with h5py.File("nested.h5", "w", libver="latest") as f:
|
||||
lay = h5py.VirtualLayout(shape=(4,), dtype="f8")
|
||||
lay[:] = h5py.VirtualSource("sub/inner.h5", "data", shape=(4,))
|
||||
f.create_virtual_dataset("v", lay, fillvalue=-1.0)
|
||||
with h5py.File("sub/inner.h5", "w") as s:
|
||||
s.create_dataset("data", data=np.arange(4.0) + 10)
|
||||
expect("nested.h5", "v", "nested")
|
||||
"#,
|
||||
);
|
||||
// libhdf5 resolves "../src.h5" (and would read [0, 1, 2, 3]); we refuse
|
||||
// to leave the directory, and say so.
|
||||
let f = File::open(dir.path().join("sub/up.h5")).unwrap();
|
||||
let err = f.dataset("v").unwrap().read_f64().unwrap_err();
|
||||
assert!(
|
||||
err.to_string().contains("not followed"),
|
||||
"unexpected error: {err}"
|
||||
);
|
||||
// A relative name below the virtual file's directory resolves there.
|
||||
assert_matches_libhdf5(dir.path(), "nested.h5", "v", "nested");
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Unlimited and printf-style mappings
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Helpers for building unlimited VDS mappings through h5py's low-level API.
|
||||
const UNLIMITED_HELPERS: &str = r#"
|
||||
U = h5py.h5s.UNLIMITED
|
||||
def space(dims, maxdims, start=None, count=None, stride=None, block=None):
|
||||
s = h5py.h5s.create_simple(dims, maxdims)
|
||||
if start is not None:
|
||||
s.select_hyperslab(start, count, stride, block)
|
||||
return s
|
||||
def make_vds(fn, name, dims, maxdims, maps, fill, libver="latest", mode="w"):
|
||||
# maps: [(vsel_kwargs, source_file, source_dataset, source_space)]
|
||||
with h5py.File(fn, mode, libver=libver) as f:
|
||||
dcpl = h5py.h5p.create(h5py.h5p.DATASET_CREATE)
|
||||
for vsel, sfile, sdset, sspace in maps:
|
||||
dcpl.set_virtual(space(dims, maxdims, **vsel), sfile.encode(), sdset.encode(), sspace)
|
||||
dcpl.set_fill_value(np.array(fill, dtype="f8"))
|
||||
h5py.h5d.create(f.id, name.encode(), h5py.h5t.IEEE_F64LE,
|
||||
h5py.h5s.create_simple(dims, maxdims), dcpl=dcpl)
|
||||
"#;
|
||||
|
||||
/// printf-style names: block `j` of the virtual selection comes from the
|
||||
/// source named with `j` in place of `%b` (`%%` is a literal `%`), probing
|
||||
/// j = 0, 1, ... until the first missing source. libhdf5 also recomputes the
|
||||
/// extent from what it finds, so the stored dataspace is not the shape.
|
||||
#[test]
|
||||
fn vds_printf_source_names() {
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let body = format!(
|
||||
"{UNLIMITED_HELPERS}{}",
|
||||
r#"
|
||||
for i in [0, 1, 2, 4]: # 3 is missing: 4 is past the first gap and unused
|
||||
with h5py.File(f"vds_src_{i}.h5", "w") as s:
|
||||
s.create_dataset("data", data=np.arange(10.0) + i * 100)
|
||||
with h5py.File(f"p%c_{i}.h5", "w") as s:
|
||||
s.create_dataset("data", data=np.arange(10.0) - i * 100)
|
||||
for libver in ["earliest", "latest"]:
|
||||
fn = f"printf_{libver}.h5"
|
||||
make_vds(fn, "files", (10,), (U,),
|
||||
[(dict(start=(0,), count=(U,), stride=(10,), block=(10,)), "vds_src_%b.h5", "data",
|
||||
space((10,), (10,), (0,), (1,), (1,), (10,)))], -1.0, libver)
|
||||
# interleaved blocks with gaps between them, and an escaped percent sign
|
||||
make_vds(fn, "escaped", (4,), (U,),
|
||||
[(dict(start=(1,), count=(U,), stride=(6,), block=(4,)), "p%%c_%b.h5", "data",
|
||||
space((10,), (10,), (2,), (1,), (1,), (4,)))], -5.0, libver, "a")
|
||||
# printf in the dataset name, same file, 2-D frames
|
||||
with h5py.File(fn, "a") as f:
|
||||
for i in range(3):
|
||||
f.create_dataset(f"frame_{i}", data=np.arange(6.0).reshape(2, 3) + 10 * i)
|
||||
with h5py.File(fn, "a", libver=libver) as f:
|
||||
dcpl = h5py.h5p.create(h5py.h5p.DATASET_CREATE)
|
||||
dcpl.set_virtual(space((1, 2, 3), (U, 2, 3), (0, 0, 0), (U, 1, 1), (1, 1, 1), (1, 2, 3)),
|
||||
b".", b"frame_%b", space((2, 3), (2, 3)))
|
||||
h5py.h5d.create(f.id, b"frames", h5py.h5t.IEEE_F64LE,
|
||||
h5py.h5s.create_simple((1, 2, 3), (U, 2, 3)), dcpl=dcpl)
|
||||
for name in ["files", "escaped", "frames"]:
|
||||
expect(fn, name, f"{name}_{libver}")
|
||||
"#
|
||||
);
|
||||
generate(dir.path(), &body);
|
||||
for libver in ["earliest", "latest"] {
|
||||
for name in ["files", "escaped", "frames"] {
|
||||
let file = format!("printf_{libver}.h5");
|
||||
assert_matches_libhdf5(dir.path(), &file, name, &format!("{name}_{libver}"));
|
||||
}
|
||||
}
|
||||
// The shape libhdf5 reports: three 10-element blocks.
|
||||
let f = File::open(dir.path().join("printf_latest.h5")).unwrap();
|
||||
assert_eq!(f.dataset("files").unwrap().shape().unwrap(), vec![30]);
|
||||
}
|
||||
|
||||
/// Unlimited source and virtual selections: each mapping covers as much as
|
||||
/// its source's current extent fills (a partial last block included), the
|
||||
/// extent is the largest of them but never smaller than the limited
|
||||
/// mappings need, and a missing source contributes nothing.
|
||||
#[test]
|
||||
fn vds_unlimited_mappings_follow_source_extents() {
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let body = format!(
|
||||
"{UNLIMITED_HELPERS}{}",
|
||||
r#"
|
||||
with h5py.File("grow.h5", "w") as s:
|
||||
s.create_dataset("a", data=np.arange(7.0) + 1, maxshape=(None,))
|
||||
s.create_dataset("b", data=np.arange(5.0) + 100, maxshape=(None,))
|
||||
s.create_dataset("rows", data=np.arange(12.0).reshape(4, 3) + 50, maxshape=(None, 3))
|
||||
unlim_src = lambda: space((1,), (U,), (0,), (U,), (1,), (1,))
|
||||
for libver in ["earliest", "latest"]:
|
||||
fn = f"unlim_{libver}.h5"
|
||||
make_vds(fn, "interleaved", (1,), (U,), [
|
||||
# blocks of 3 every 4: 7 source elements end mid-block
|
||||
(dict(start=(0,), count=(U,), stride=(4,), block=(3,)), "grow.h5", "a", unlim_src()),
|
||||
(dict(start=(3,), count=(U,), stride=(4,), block=(1,)), "grow.h5", "b", unlim_src()),
|
||||
(dict(start=(0,), count=(U,), stride=(1,), block=(1,)), "missing.h5", "a", unlim_src()),
|
||||
], -2.0, libver)
|
||||
make_vds(fn, "rows", (6, 3), (U, 3), [
|
||||
# an unlimited *block*, plus a limited mapping reaching row 5
|
||||
(dict(start=(0, 0), count=(1, 1), stride=(1, 1), block=(U, 3)), "grow.h5", "rows",
|
||||
space((1, 3), (U, 3), (0, 0), (1, 1), (1, 1), (U, 3))),
|
||||
(dict(start=(5, 0), count=(1, 1), stride=(1, 1), block=(1, 3)), "grow.h5", "rows",
|
||||
space((4, 3), (U, 3), (1, 0), (1, 1), (1, 1), (1, 3))),
|
||||
], -4.0, libver, "a")
|
||||
for name in ["interleaved", "rows"]:
|
||||
expect(fn, name, f"{name}_{libver}")
|
||||
"#
|
||||
);
|
||||
generate(dir.path(), &body);
|
||||
for libver in ["earliest", "latest"] {
|
||||
for name in ["interleaved", "rows"] {
|
||||
let file = format!("unlim_{libver}.h5");
|
||||
assert_matches_libhdf5(dir.path(), &file, name, &format!("{name}_{libver}"));
|
||||
}
|
||||
}
|
||||
// "a" (7 elements, blocks of 3 every 4) ends at 9; "b" (5 elements from
|
||||
// 3, every 4) at 20. "rows" fills 4 rows but a limited mapping needs 6.
|
||||
let f = File::open(dir.path().join("unlim_latest.h5")).unwrap();
|
||||
assert_eq!(f.dataset("interleaved").unwrap().shape().unwrap(), vec![20]);
|
||||
assert_eq!(f.dataset("rows").unwrap().shape().unwrap(), vec![6, 3]);
|
||||
}
|
||||
|
||||
/// libhdf5's own VDS test files (HDF5 `tools/test/testfiles/vds`):
|
||||
/// printf-style Eiger frames (with a source past the first gap that must be
|
||||
/// ignored), a printf mapping in the 1.10 format, and Percival's four
|
||||
/// interleaved unlimited sources of different lengths.
|
||||
#[test]
|
||||
fn vds_libhdf5_test_files() {
|
||||
skip_if_no_python!();
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let fixtures = Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/vds");
|
||||
for entry in std::fs::read_dir(&fixtures).unwrap() {
|
||||
let path = entry.unwrap().path();
|
||||
if path.extension().is_some_and(|e| e == "h5") {
|
||||
std::fs::copy(&path, dir.path().join(path.file_name().unwrap())).unwrap();
|
||||
}
|
||||
}
|
||||
let cases = [
|
||||
("vds-eiger.h5", "/VDS-Eiger"),
|
||||
("4_vds.h5", "/vds_dset"),
|
||||
("vds-percival-unlim-maxmin.h5", "/VDS-Percival-unlim-maxmin"),
|
||||
];
|
||||
let mut body = String::new();
|
||||
for (i, (file, dset)) in cases.iter().enumerate() {
|
||||
body.push_str(&format!("expect({file:?}, {dset:?}, \"case{i}\")\n"));
|
||||
}
|
||||
generate(dir.path(), &body);
|
||||
for (i, (file, dset)) in cases.iter().enumerate() {
|
||||
assert_matches_libhdf5(dir.path(), file, dset, &format!("case{i}"));
|
||||
}
|
||||
// Stored as 20 frames; only f-0.h5 is found before the first gap.
|
||||
let f = File::open(dir.path().join("vds-eiger.h5")).unwrap();
|
||||
assert_eq!(
|
||||
f.dataset("VDS-Eiger").unwrap().shape().unwrap(),
|
||||
vec![5, 10, 10]
|
||||
);
|
||||
}
|
||||
+16
-5
@@ -73,8 +73,8 @@ segfault or abort.
|
||||
|
||||
## Gaps found by the 2026-09-25 HDF5 audit (open)
|
||||
|
||||
**Status:** open. These fail with an error; none returns wrong data, except
|
||||
the VDS item, which is marked.
|
||||
**Status:** open. These fail with an error; none returns wrong data (the VDS
|
||||
fill-value item that did is fixed).
|
||||
|
||||
- **Layout message versions 1 and 2** (HDF5 1.6-era files): 84 of the 686
|
||||
sweep files, `InvalidLayoutVersion`. This is the largest single gap.
|
||||
@@ -85,9 +85,20 @@ the VDS item, which is marked.
|
||||
per-member dimensions were skipped, so an array member read as one scalar.
|
||||
**Fixed 2026-09-25.**
|
||||
- **Virtual datasets:**
|
||||
- **Wrong data:** unmapped regions read as 0 instead of the fill value.
|
||||
- `%b` printf-style source names are not expanded.
|
||||
- Hyperslab selection versions 1 and 2 are refused.
|
||||
- ~~**Wrong data:** unmapped regions read as 0 instead of the fill value.~~
|
||||
Fixed 2026-09-25: unmapped elements and missing sources read as the
|
||||
virtual dataset's fill value.
|
||||
- ~~`%b` printf-style source names are not expanded.~~ Fixed 2026-09-25:
|
||||
printf-style and unlimited mappings are read, and the extent is
|
||||
recomputed from the sources as libhdf5 does. Still open: the
|
||||
"first missing" view and a printf gap other than 0 (libhdf5 access
|
||||
properties we always read at their defaults), source-to-virtual type
|
||||
conversion other than a byte swap, nested virtual sources, and source
|
||||
files outside the virtual file's directory (refused with an error).
|
||||
- ~~Hyperslab selection versions 1 and 2 are refused.~~ Fixed 2026-09-25:
|
||||
versions 1-3 and irregular hyperslabs are decoded.
|
||||
- ~~The version-1 mapping list written with a 2.0 low bound (flags byte,
|
||||
shared names) was misparsed.~~ Found and fixed 2026-09-25.
|
||||
- **Files with a user block:** the base address is not applied.
|
||||
**Fixed 2026-09-25:** every reader views the file from the superblock on
|
||||
(`twithub.h5`, `twithub513.h5`, `h5clear_fsm_persist_user_*.h5`; the
|
||||
|
||||
Reference in New Issue
Block a user