HDF5 1.12 revised the reference datatype (class 7) in datatype message version
4: reference types 2-4 are the new H5T_STD_REF object / dataset-region /
attribute references. Datatype::parse rejected them with
InvalidReferenceType, so any dataset of that type was unreadable.
h5py cannot write this type, which is why it had never been tested. A real
file was produced by calling the libhdf5 bundled in the h5py wheel through
ctypes (H5T_STD_REF_g, H5Rcreate_object, H5Dwrite); the 2 KB result is
committed as tests/fixtures/std_ref_hdf5_2_0.h5 with its generator,
gen_std_ref.py.
- ReferenceType gains Object2, DatasetRegion2 and Attribute, accepted only
from datatype version 4.
- read_object_references decodes Object2 elements: type(1) flags(1)
token_size(1) token, zero-padded to the element size; the token is the
target's object header address. A null reference decodes to the undefined
address; an external reference, a wrong type byte or a token that doesn't
fit is an error.
The fixture test follows both references and checks they resolve to the
objects they were created from.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
With libver='latest', a chunked dataset with two or more unlimited dimensions
indexes its chunks with a v2 B-tree (layout v4, index type 5). Reading one
failed with "unsupported chunked layout version=4, index_type=Some(5)".
read_btree_v2_chunks decodes record types 10 (address + scaled offsets) and 11
(address, stored size, filter mask, scaled offsets). The width of the
stored-size field is taken from the record size the tree header declares
rather than re-deriving the library's formula. Scaled offsets are multiplied
back by the chunk dimensions with overflow checks.
The chunk-index dispatch existed four times (uncached, cached, sweep and
indexed readers). The three copies outside list_chunks now call it, so every
read path — and fill-value handling and partial reads — supports every index
type from one place.
h5py interop test: plain, gzip+shuffle, a 2500-chunk tree with internal nodes,
a sparse dataset with a fill value, and a strided hyperslab.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
attrs() silently omitted any attribute whose datatype had no AttrValue variant
— including every Python bool, which h5py stores as an enum — plus complex,
compound and reference attributes, and cast unsigned 64-bit arrays to
I64Array so values above i64::MAX came back negative.
- numpy/h5py-style booleans (an enum of exactly FALSE=0 / TRUE=1 over an
integer base) decode as I64 / I64Array of 0/1.
- AttrValue::U64Array keeps unsigned arrays unsigned. Behaviour change: an
unsigned array attribute no longer arrives as I64Array; the netCDF-4 CF
helpers (_FillValue, valid_range) and the Python bindings handle it.
- AttrValue::Raw { datatype, shape, data } carries any other attribute
verbatim (also used when a value fails to decode as its declared type), so
the attribute list is always complete. Decodable with data_read against the
datatype; Python receives {"dtype", "shape", "data"}.
- Both new variants are writable, so attributes round-trip between files.
h5py interop tests cover reading 13 attribute kinds and h5py reading back a
compound and a u64 attribute written by clawhdf5.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Class 11 (datatype version 5) properties are a single base floating-point
datatype message, not a compound-style member list. The old parser read the
base type's bytes as member names, yielding a garbage datatype, and failed
with UnexpectedEof when a complex type was nested in a compound.
Parse the base type and surface the type as the equivalent {r, i} compound
(the shape h5py writes for numpy complex dtypes), with a size check against
the base type. Covered by byte-level tests taken from HDF5 2.0 output and an
h5py end-to-end test (writer_h5py_tests is now 27/27 against HDF5 2.0.0).
Found while validating a user report of InvalidDatatypeVersion
{ class: 6, version: 5 } against v2.1.0 (already fixed on main in a13ff51,
never released). Add docs/known-issues.md recording that report, this bug,
the open reference-v4 gap and a gpu_tests parallel-run hang; credit the
reporter in the changelog.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
