feat: check HDF5 2.x small floats against libhdf5 2.2.0
Fixture written by libhdf5 2.2.0 (built from tag 2.2.0) through ctypes: every bit pattern of FP4 E2M1, FP6 E2M3/E3M2, FP8 E4M3/E5M2 and a bfloat16 LE/BE set, as datasets and attributes, with what H5Dread/H5Aread return into double and float and the conversion exceptions libhdf5 raises. clawhdf5 already decoded every value as libhdf5 does, including an all-ones exponent as inf/NaN in the OCP formats that have none (documented as a deliberate match in known-issues). - data_read: NaNs of non-native float layouts get libhdf5's bits (sign kept, every mantissa bit set) in f64 and f32. - h5rs dump/ls name these types as h5dump/h5ls 2.x do (H5T_FLOAT_F4E2M1, "FP4 E2M1 4-bit float", float4-e2m1 ...), checked against h5dump 2.2.0's output of the fixture. - Python bindings read them as h5py 3.16 does (float32 for bfloat16, float16 for the 1-byte formats, file byte order, same bytes as h5py); writing them in 'r+' is refused. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -16,6 +16,9 @@ Checked against `main` at `9b5803f` on 2026-09-28.
|
||||
| Issue | Kind | Since |
|
||||
|---|---|---|
|
||||
| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 |
|
||||
|
||||
| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 |
|
||||
| [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 |
|
||||
| [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 |
|
||||
| [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 |
|
||||
| [Selection reads that decode more than the selection](#selection-reads-that-decode-more-than-the-selection) | speed only | 2026-09-26 |
|
||||
@@ -420,6 +423,26 @@ unwritten records read as fill); `Variable::shape` is the HDF5 extent,
|
||||
Workaround: read the variable's `_Netcdf4Coordinates` attribute (dimension
|
||||
ids, matching each scale's `_Netcdf4Dimid`).
|
||||
|
||||
## Small floats decode as libhdf5 does, not as the OCP MX specification
|
||||
|
||||
**Status:** open, deliberate (documented 2026-09-28). HDF5 2.x predefines
|
||||
bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2 and FP4 E2M1. clawhdf5 decodes every
|
||||
bit pattern of these types exactly as libhdf5 2.2.0 does (all 16/64/256
|
||||
patterns and a bfloat16 set compared bit for bit, NaN bits included:
|
||||
`crates/clawhdf5/tests/mx_floats_interop.rs` over a fixture libhdf5 2.2.0
|
||||
wrote, `fixtures/gen_mx_floats.py`), and h5py 3.16 (libhdf5 2.0.0) returns
|
||||
the same values. libhdf5 treats an all-ones exponent as infinity or NaN in
|
||||
every one of them, IEEE-style, whereas the OCP Microscaling formats
|
||||
specification gives FP4 and FP6 no infinity or NaN and FP8 E4M3 a single NaN
|
||||
(`S.1111.111`). So values that are finite under the OCP specification read
|
||||
as ±inf or NaN in both libraries: FP4 E2M1 4.0 and 6.0 (`0b0110`, `0b0111`),
|
||||
FP6 E2M3 above 3.75, FP6 E3M2 above 14, and FP8 E4M3 256 to 448 (`0x78` to
|
||||
`0x7E`). libhdf5's own release notes (2.2.0, *Known Problems*) say its
|
||||
conversions of these types "may improperly convert some data values"; if it
|
||||
changes that, clawhdf5 should follow. Workaround for data written under the
|
||||
OCP meaning: read the element bytes (`Dataset::read_selection(&Selection::All)`)
|
||||
and the type's fields (`Dataset::raw_datatype`) and decode them yourself.
|
||||
|
||||
## The Node.js package (`packages/clawhdf5-node`) does not work
|
||||
|
||||
**Status:** open (found 2026-09-25; re-checked 2026-09-28, unchanged).
|
||||
|
||||
Reference in New Issue
Block a user