feat(format): parse H5T_STD_REF references and decode object references

HDF5 1.12 revised the reference datatype (class 7) in datatype message version
4: reference types 2-4 are the new H5T_STD_REF object / dataset-region /
attribute references. Datatype::parse rejected them with
InvalidReferenceType, so any dataset of that type was unreadable.

h5py cannot write this type, which is why it had never been tested. A real
file was produced by calling the libhdf5 bundled in the h5py wheel through
ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is
committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator,
gen_std_ref.py.

- ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only
  from datatype version 4.
- read_object_references decodes Object2 elements: type(1) flags(1)
  token_size(1) token, zero-padded to the element size; the token is the
  target's object header address. A null reference decodes to the undefined
  address; an external reference, a wrong type byte or a token that doesn't
  fit is an error.

The fixture test follows both references and checks they resolve to the
objects they were created from.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
osobh
2026-09-19 14:24:04 -07:00
co-authored by Claude Fable 5.1
parent 05c665a898
commit 52cfcf20b2
7 changed files with 258 additions and 9 deletions
+5
View File
@@ -22,6 +22,11 @@
type 5 — what `libver='latest'` uses for two or more unlimited dimensions;
previously "unsupported chunked layout"). The four copies of the chunk-index
dispatch are now one shared function, so every read path gets it.
- **`H5T_STD_REF` references** (HDF5 1.12+, datatype message version 4) parse:
`ReferenceType` gains `Object2`, `DatasetRegion2` and `Attribute`, and
`read_object_references` decodes the new object references. Previously any
dataset of this type failed with `InvalidReferenceType(2)`. Tested against a
file written by HDF5 2.0 itself (fixture + generator script committed).
- **Automatic chunk sizes.** Asking for compression (or any filter) without
`with_chunks` used to store the whole dataset as one chunk, so any read had
to decompress everything and nothing could be decoded in parallel. Datasets up