feat(format): parse H5T_STD_REF references and decode object references

HDF5 1.12 revised the reference datatype (class 7) in datatype message version
4: reference types 2-4 are the new H5T_STD_REF object / dataset-region /
attribute references. Datatype::parse rejected them with
InvalidReferenceType, so any dataset of that type was unreadable.

h5py cannot write this type, which is why it had never been tested. A real
file was produced by calling the libhdf5 bundled in the h5py wheel through
ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is
committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator,
gen_std_ref.py.

- ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only
  from datatype version 4.
- read_object_references decodes Object2 elements: type(1) flags(1)
  token_size(1) token, zero-padded to the element size; the token is the
  target's object header address. A null reference decodes to the undefined
  address; an external reference, a wrong type byte or a token that doesn't
  fit is an error.

The fixture test follows both references and checks they resolve to the
objects they were created from.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
osobh
2026-09-19 14:24:04 -07:00
co-authored by Claude Fable 5.1
parent 05c665a898
commit 52cfcf20b2
7 changed files with 258 additions and 9 deletions
+60
View File
@@ -1427,6 +1427,26 @@ pub fn read_object_references(
}
Ok(result)
}
Datatype::Reference {
ref_type: crate::datatype::ReferenceType::Object2,
size,
} => {
let elem_size = *size as usize;
if elem_size == 0 {
return Ok(Vec::new());
}
if !raw.len().is_multiple_of(elem_size) {
return Err(FormatError::DataSizeMismatch {
expected: 0,
actual: raw.len(),
});
}
raw.chunks_exact(elem_size)
.map(|element| {
decode_std_object_ref(element).map(|address| ObjectReference { address })
})
.collect()
}
_ => Err(FormatError::TypeMismatch {
expected: "Reference(Object)",
actual: datatype_name(datatype),
@@ -1434,6 +1454,46 @@ pub fn read_object_references(
}
}
/// Decode one `H5T_STD_REF` object reference as stored in a dataset:
/// `type(1) flags(1) token_size(1) token(token_size)`, zero-padded to the
/// element size. For a reference within the same file the token is the target
/// object's header address. An all-zero element is a null reference and
/// decodes to the undefined address (`u64::MAX`).
fn decode_std_object_ref(element: &[u8]) -> Result<u64, FormatError> {
const STD_REF_OBJECT: u8 = 2;
const FLAG_EXTERNAL: u8 = 0x01;
if element.iter().all(|&b| b == 0) {
return Ok(u64::MAX);
}
let [ref_type, flags, token_size, token @ ..] = element else {
return Err(FormatError::UnexpectedEof {
expected: 3,
available: element.len(),
});
};
if *ref_type != STD_REF_OBJECT {
return Err(FormatError::InvalidReferenceType(*ref_type));
}
if flags & FLAG_EXTERNAL != 0 {
// Carries a file name as well; nothing here follows those.
return Err(FormatError::TypeMismatch {
expected: "object reference within this file",
actual: "external object reference",
});
}
let n = *token_size as usize;
if n == 0 || n > 8 || n > token.len() {
return Err(FormatError::UnexpectedEof {
expected: 3 + n,
available: element.len(),
});
}
Ok(token[..n]
.iter()
.rev()
.fold(0u64, |addr, &byte| (addr << 8) | u64::from(byte)))
}
/// Read region references from raw bytes.
///
/// Region references encode a dataset selection (hyperslab, point list, etc.)