feat: check HDF5 2.x small floats against libhdf5 2.2.0
CI / test-arm64 (pull_request) Successful in 1m38s
CI / test (pull_request) Successful in 31m22s

Fixture written by libhdf5 2.2.0 (built from tag 2.2.0) through ctypes:
every bit pattern of FP4 E2M1, FP6 E2M3/E3M2, FP8 E4M3/E5M2 and a
bfloat16 LE/BE set, as datasets and attributes, with what H5Dread/H5Aread
return into double and float and the conversion exceptions libhdf5
raises. clawhdf5 already decoded every value as libhdf5 does, including
an all-ones exponent as inf/NaN in the OCP formats that have none
(documented as a deliberate match in known-issues).

- data_read: NaNs of non-native float layouts get libhdf5's bits (sign
  kept, every mantissa bit set) in f64 and f32.
- h5rs dump/ls name these types as h5dump/h5ls 2.x do
  (H5T_FLOAT_F4E2M1, "FP4 E2M1 4-bit float", float4-e2m1 ...), checked
  against h5dump 2.2.0's output of the fixture.
- Python bindings read them as h5py 3.16 does (float32 for bfloat16,
  float16 for the 1-byte formats, file byte order, same bytes as h5py);
  writing them in 'r+' is refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-28 21:25:03 -05:00
co-authored by Claude Opus 5.5
parent 31efac1ae1
commit bce07e9cb9
16 changed files with 1207 additions and 13 deletions
+6 -1
View File
@@ -37,7 +37,12 @@ with clawhdf5.File("data.h5", "r") as f:
`dtype.metadata['enum']`), complex, `S<n>` fixed strings, `object` for
variable-length strings (`bytes` values) and sequences (array values),
`V<n>` opaque, array types, and compounds as structured dtypes.
Other types raise `TypeError`.
Non-IEEE floats (HDF5 2.x's bfloat16, FP8, FP6 and FP4) read as h5py
3.16 reads them: as the narrowest IEEE float that holds them (`float32`
for bfloat16, `float16` for the 1-byte formats) in the file's byte order,
with libhdf5's values (`tests/test_small_floats.py`); inside a compound or
array type they raise `TypeError`, and writing them raises
`NotImplementedError`. Other types raise `TypeError`.
- Keys are h5py's: integers, slices with a positive step, `...`, one
increasing list of integers, compound field names. Each maps onto a
hyperslab selection. `None` and negative steps are refused