feat(format): parse H5T_STD_REF references and decode object references

HDF5 1.12 revised the reference datatype (class 7) in datatype message version
4: reference types 2-4 are the new H5T_STD_REF object / dataset-region /
attribute references. Datatype::parse rejected them with
InvalidReferenceType, so any dataset of that type was unreadable.

h5py cannot write this type, which is why it had never been tested. A real
file was produced by calling the libhdf5 bundled in the h5py wheel through
ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is
committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator,
gen_std_ref.py.

- ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only
  from datatype version 4.
- read_object_references decodes Object2 elements: type(1) flags(1)
  token_size(1) token, zero-padded to the element size; the token is the
  target's object header address. A null reference decodes to the undefined
  address; an external reference, a wrong type byte or a token that doesn't
  fit is an error.

The fixture test follows both references and checks they resolve to the
objects they were created from.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
osobh
2026-09-19 14:24:04 -07:00
co-authored by Claude Fable 5.1
parent 05c665a898
commit 52cfcf20b2
7 changed files with 258 additions and 9 deletions
+14 -6
View File
@@ -62,13 +62,21 @@ only files using the native type through the C API / h5py low-level API hit this
## Revised reference datatype (class 7, version 4) is not parsed
**Status:** open, unconfirmed against a real file.
**Status:** fixed 2026-09-19 for object references; region and attribute
references are recognised but not decoded.
**Summary:** HDF5 1.12+ `H5T_STD_REF` references use datatype version 4 with
reference types 24 (object2 / region2 / attribute), which `Datatype::parse`
rejects with `InvalidReferenceType`. h5py still writes the legacy v1
object/region references, which read correctly, so no reproducing file has been
generated yet; one written with the C API (`H5T_STD_REF`) is needed.
**Summary:** HDF5 1.12+ `H5T_STD_REF` references use datatype message version 4
with reference types 2-4 (object / region / attribute), which `Datatype::parse`
rejected with `InvalidReferenceType`. h5py still writes the legacy references,
so no file had been available to test against.
**Fix:** a real file was produced by driving the libhdf5 bundled in the h5py
wheel through ctypes (`tests/fixtures/gen_std_ref.py` ->
`std_ref_hdf5_2_0.h5`). The three new types parse as
`ReferenceType::{Object2, DatasetRegion2, Attribute}`, and
`read_object_references` decodes `Object2` elements (type, flags, token size,
token = target object header address). External references (flag bit 0) and
the region/attribute payloads are errors rather than misreads.
## `clawhdf5-gpu` `gpu_tests` can hang under the default parallel test runner