diff --git a/CHANGELOG.md b/CHANGELOG.md index 189ab99..b2a578c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,35 @@ Affected v2.1.0 to v2.7.0. `docs/known-issues.md` also gains an open entry found meanwhile: variables' dimensions are matched by size. +### HDF5 2.x small floats checked against libhdf5 2.2.0 (2026-09-28) +- New fixture written by libhdf5 2.2.0 itself (built from tag `2.2.0`, + `crates/clawhdf5/tests/fixtures/gen_mx_floats.py`): a dataset and an + attribute of each of `H5T_FLOAT_F4E2M1`, `H5T_FLOAT_F6E2M3`, + `H5T_FLOAT_F6E3M2` (every bit pattern, also with the unused high bits set), + `H5T_FLOAT_F8E4M3`, `H5T_FLOAT_F8E5M2` (all 256) and + `H5T_FLOAT_BFLOAT16LE`/`BE` (zeros, subnormals, max, ±inf, NaNs), with + what libhdf5 returns for each element into `double` and `float`. + `tests/mx_floats_interop.rs` compares `read_f64`, `read_f32` and the + attributes bit for bit. No value was wrong: clawhdf5 already decoded every + pattern as libhdf5 does, including an all-ones exponent as ±inf/NaN in the + formats the OCP MX specification makes finite (FP4, FP6, FP8 E4M3 — a + deliberate match, recorded in `docs/known-issues.md`). +- NaNs of these formats now have libhdf5's bits: sign kept, every mantissa + bit set (`0x7FFF_FFFF_FFFF_FFFF` as `f64`, `0x7FFF_FFFF` as `f32`); they + were `f64::NAN`'s bits and an `as f32` cast's. +- `h5rs dump` names these types as h5dump 2.x does (`H5T_FLOAT_F4E2M1`, + `H5T_FLOAT_BFLOAT16LE`, ...) instead of printing an `H5T_FLOAT { ... }` + block; `h5rs ls -v` describes them as h5ls 2.x does (`FP4 E2M1 4-bit + float`) and `h5rs ls` lists them as `bfloat16`, `float8-e4m3`, + `float6-e2m3`, `float4-e2m1`, ... instead of `float16`/`float8`. Checked + against h5dump 2.2.0's output of the fixture (`tests/mx_floats_dump.rs`). +- Python bindings: datasets and attributes of these types used to raise + `TypeError`; they now read as h5py 3.16 reads them — `float32` for + bfloat16, `float16` for the 1-byte formats, in the file's byte order — with + the same bytes h5py returns (`tests/test_small_floats.py`). Writing them in + `'r+'` raises `NotImplementedError` before anything is written; inside a + compound or array type they still raise `TypeError`. + ### `ObjectHeader::parse` back at its pre-M2/M3 speed (2026-09-27) - Parsing a version-1 object header was 4% slower than before range-read M2/M3 (`docs/known-issues.md`). The cause was the call to the per-chunk diff --git a/README.md b/README.md index 55b6e48..7c34ff5 100644 --- a/README.md +++ b/README.md @@ -114,7 +114,7 @@ Limits and open issues, with dates, are in |---|---|---|---| | **File format** | Superblock v0–v3, user blocks, v1/v2 object headers | Metadata cache images | Writing files HDF5 1.8 can read | | **Groups and links** | Symbol-table, compact and dense groups (tested to 100 000 links), creation order, soft and hard links; writing external links | | Following external links (explicit error); user-defined links are skipped | -| **Datatypes** | Integers and IEEE floats of every width and byte order (incl. `f16`), enums, compounds (every version, incl. HDF5 2.0's v5), arrays, fixed-length strings, opaque, complex (HDF5 2.0 class 11) | Variable-length strings and sequences, object references | Writing variable-length data; decoding region and attribute references; x87 long double and binary128 | +| **Datatypes** | Integers and IEEE floats of every width and byte order (incl. `f16`), enums, compounds (every version, incl. HDF5 2.0's v5), arrays, fixed-length strings, opaque, complex (HDF5 2.0 class 11) | Variable-length strings and sequences, object references; HDF5 2.x's small floats (bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2, FP4 E2M1: every bit pattern decoded as libhdf5 2.2.0 decodes it) and other non-IEEE floats up to 64 bits | Writing variable-length data; writing non-IEEE floats; decoding region and attribute references; x87 long double and binary128 | | **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does); fill values; resizable datasets; virtual datasets (read limits in known-issues) | Chunk indexes v1 B-tree and implicit (the editor also changes them) | External raw data files (explicit error) | | **Filters** | deflate (pure-Rust zlib-rs), shuffle, Fletcher-32, LZ4 (opt-in), Zstd (C, opt-in); plugins LZF, bitshuffle, bzip2, Blosc 1 | N-Bit, scale-offset, SZIP (C, opt-in); plugins Blosc2 and ZFP | Other filter IDs, unless you register a codec (`filter_registry::register_filter`) | | **Editing in place** | `FileEditor`: overwrite values, grow and shrink chunked datasets (every index), set attributes (compact and dense), in files from h5py or clawhdf5 | | Creating or deleting objects in an existing file; deleting attributes; new chunks in implicit indexes; VL data; filters this build cannot encode (refused before any write) | diff --git a/crates/clawhdf5-format/src/data_read.rs b/crates/clawhdf5-format/src/data_read.rs index 40826c1..588d568 100644 --- a/crates/clawhdf5-format/src/data_read.rs +++ b/crates/clawhdf5-format/src/data_read.rs @@ -1419,9 +1419,11 @@ pub fn read_as_f32(raw: &[u8], datatype: &Datatype) -> Result, FormatEr result.push(match format { FloatFormat::Single => read_f32_bytes(chunk, &order), FloatFormat::Half => read_f16_bytes(chunk, &order), - // Double rounds; every other supported layout (bfloat16, FP8) - // is exact in f32. - _ => format.decode(chunk, &order) as f32, + // Double rounds. + FloatFormat::Double => format.decode(chunk, &order) as f32, + // Every other supported layout (bfloat16, FP8, FP6, FP4) is + // exact in f32. + FloatFormat::Other(_) => narrow_decoded(format.decode(chunk, &order)), }); } return Ok(result); @@ -1947,7 +1949,12 @@ enum FloatFormat { Double, /// Any other IEEE-style layout (implied leading mantissa bit, all-ones /// exponent for infinity/NaN) whose values are all exact in `f64`: - /// bfloat16, the FP8 formats, and similar. + /// bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2, FP4 E2M1, and similar. + /// + /// libhdf5 (checked against 2.2.0) decodes all of them this way, also the + /// OCP MX formats whose specification has no infinity (FP6, FP4) or a + /// single NaN (FP8 E4M3): an all-ones exponent is infinity or NaN, not a + /// finite value. clawhdf5 follows libhdf5 so both read a file alike. Other(FloatLayout), } @@ -2046,7 +2053,9 @@ impl FloatLayout { if mantissa == 0 { f64::INFINITY } else { - f64::NAN + // The NaN libhdf5 converts every NaN to: all mantissa bits + // set (the sign is applied below). + LIBHDF5_NAN } } else { let bias = i64::from(self.exponent_bias); @@ -2067,6 +2076,27 @@ impl FloatLayout { } } +/// The `f64` NaN libhdf5's conversion (`H5T__conv_f_f`) produces from a NaN +/// of a non-native float layout: sign clear, every mantissa bit set. +const LIBHDF5_NAN: f64 = f64::from_bits(0x7FFF_FFFF_FFFF_FFFF); + +/// Narrow a value decoded from a non-native float layout to `f32`. Every such +/// value is exact in `f32`; a NaN becomes the NaN libhdf5 gives for +/// `H5T_NATIVE_FLOAT` (sign kept, every mantissa bit set) rather than +/// whatever payload an `as` cast leaves. +fn narrow_decoded(value: f64) -> f32 { + if value.is_nan() { + let sign = if value.is_sign_negative() { + 1u32 << 31 + } else { + 0 + }; + f32::from_bits(sign | 0x7FFF_FFFF) + } else { + value as f32 + } +} + /// `x * 2^power` without `std` (no `powi`/`libm`). `x` is a non-negative /// integer below 2^53, so it is exact. fn scale_by_pow2(x: f64, power: i64) -> f64 { @@ -2425,6 +2455,37 @@ mod tests { assert!(got[4].is_nan()); } + #[test] + fn fp4_decodes_as_libhdf5_does() { + // H5T_FLOAT_F4E2M1: 4 significant bits in a byte; the high bits are + // padding. libhdf5 2.2.0 reads 0b0110 as +inf and 0b0111 as NaN + // (IEEE-style, although OCP MX FP4 has neither), and returns NaNs + // with every mantissa bit set. + let fp4 = Datatype::FloatingPoint { + size: 1, + byte_order: DatatypeByteOrder::LittleEndian, + bit_offset: 0, + bit_precision: 4, + exponent_location: 1, + exponent_size: 2, + mantissa_location: 0, + mantissa_size: 1, + exponent_bias: 1, + }; + let raw = [0x01, 0x05, 0xF5, 0x06, 0x0E, 0x07, 0x0F]; + let got = read_as_f64(&raw, &fp4).unwrap(); + assert_eq!( + &got[..5], + &[0.5, 3.0, 3.0, f64::INFINITY, f64::NEG_INFINITY] + ); + assert_eq!(got[5].to_bits(), 0x7FFF_FFFF_FFFF_FFFF); + assert_eq!(got[6].to_bits(), 0xFFFF_FFFF_FFFF_FFFF); + let got = read_as_f32(&raw, &fp4).unwrap(); + assert_eq!(&got[..3], &[0.5, 3.0, 3.0]); + assert_eq!(got[5].to_bits(), 0x7FFF_FFFF); + assert_eq!(got[6].to_bits(), 0xFFFF_FFFF); + } + #[test] fn full_width_signed_unchanged() { // Regression: full-width 32-bit signed must be unaffected. diff --git a/crates/clawhdf5-py/README.md b/crates/clawhdf5-py/README.md index 79db9ac..3a62261 100644 --- a/crates/clawhdf5-py/README.md +++ b/crates/clawhdf5-py/README.md @@ -37,7 +37,12 @@ with clawhdf5.File("data.h5", "r") as f: `dtype.metadata['enum']`), complex, `S` fixed strings, `object` for variable-length strings (`bytes` values) and sequences (array values), `V` opaque, array types, and compounds as structured dtypes. - Other types raise `TypeError`. + Non-IEEE floats (HDF5 2.x's bfloat16, FP8, FP6 and FP4) read as h5py + 3.16 reads them: as the narrowest IEEE float that holds them (`float32` + for bfloat16, `float16` for the 1-byte formats) in the file's byte order, + with libhdf5's values (`tests/test_small_floats.py`); inside a compound or + array type they raise `TypeError`, and writing them raises + `NotImplementedError`. Other types raise `TypeError`. - Keys are h5py's: integers, slices with a positive step, `...`, one increasing list of integers, compound field names. Each maps onto a hyperslab selection. `None` and negative steps are refused diff --git a/crates/clawhdf5-py/src/convert.rs b/crates/clawhdf5-py/src/convert.rs index c8c3d68..b93675b 100644 --- a/crates/clawhdf5-py/src/convert.rs +++ b/crates/clawhdf5-py/src/convert.rs @@ -8,10 +8,16 @@ //! returns become the array's buffer as they are: the `Vec` is handed to //! numpy without a copy and viewed as the dtype. //! -//! Anything this mapping cannot describe exactly — non-IEEE floats, integers -//! with padding bits, VAX byte order, references, bitfields, time, and -//! variable-length members inside compounds or arrays — is a `TypeError`, -//! never a best-effort guess. +//! The one exception is a dataset or attribute of a non-IEEE float +//! (bfloat16, FP8 E4M3/E5M2, FP6, FP4, ...): like h5py, it reads as the +//! narrowest of `float16`/`float32`/`float64` that holds every value of the +//! format exactly, in the file's byte order, and the values are converted +//! (as libhdf5 converts them, NaN bits included). +//! +//! Anything this mapping cannot describe exactly — non-IEEE floats inside +//! compounds or arrays, integers with padding bits, VAX byte order, +//! references, bitfields, time, and variable-length members inside compounds +//! or arrays — is a `TypeError`, never a best-effort guess. use std::collections::HashMap; @@ -35,6 +41,13 @@ pub(crate) enum Layout { VlString { utf8: bool }, /// Variable-length sequence of a fixed-size base type. VlSequence, + /// A non-IEEE float (`source`), converted to the IEEE float of `size` + /// bytes numpy reports (see [`widened_float`]). + Float { + source: Datatype, + size: usize, + big_endian: bool, + }, } /// Everything needed to turn a dataset's or attribute's bytes into numpy. @@ -139,6 +152,74 @@ fn float_format(dt: &Datatype) -> PyResult { Ok(format!("{}f{size}", byte_order_char(byte_order, *size)?)) } +/// Whether `dt` is an IEEE 754 binary16/32/64 float numpy reads as it is. +pub(crate) fn is_ieee_float(dt: &Datatype) -> bool { + float_format(dt).is_ok() +} + +/// The IEEE float h5py reads a non-IEEE float as (`TypeFloatID.py_dtype` in +/// h5py 3.16): the first of `float16`, `float32`, `float64` at least as wide +/// whose mantissa holds the format's and whose normal exponent range covers +/// it — bfloat16 as `float32`, FP8/FP6/FP4 as `float16` — in the file's byte +/// order. Returns `(size in bytes, big endian)`, or `None` when no IEEE type +/// holds it (or the byte order is VAX). +fn widened_float(dt: &Datatype) -> Option<(usize, bool)> { + let Datatype::FloatingPoint { + size, + byte_order, + exponent_size, + mantissa_size, + exponent_bias, + .. + } = dt + else { + return None; + }; + let big_endian = match byte_order { + DatatypeByteOrder::LittleEndian => false, + DatatypeByteOrder::BigEndian => true, + DatatypeByteOrder::Vax => return None, + }; + if *exponent_size >= 32 { + return None; + } + let max_exp = (1i64 << exponent_size) - i64::from(*exponent_bias) - 1; + let min_exp = 1 - i64::from(*exponent_bias); + // numpy's finfo: (itemsize, nmant, maxexp, minexp). + [(2, 10, 16, -14), (4, 23, 128, -126), (8, 52, 1024, -1022)] + .into_iter() + .find(|&(bytes, nmant, maxexp, minexp)| { + bytes >= *size && *mantissa_size <= nmant && max_exp <= maxexp && min_exp >= minexp + }) + .map(|(bytes, ..)| (bytes as usize, big_endian)) +} + +/// Encode `values` as IEEE floats of `size` bytes. Every value is exact in +/// the target (`widened_float` chose it so); a NaN keeps its sign and gets +/// every mantissa bit set, as libhdf5 converts NaNs. +fn encode_floats(values: &[f64], size: usize, big_endian: bool) -> Vec { + let mut out = Vec::with_capacity(values.len() * size); + for &v in values { + let bits: u64 = if v.is_nan() { + let sign = u64::from(v.is_sign_negative()) << (size * 8 - 1); + sign | ((1u64 << (size * 8 - 1)) - 1) + } else { + match size { + 2 => u64::from(clawhdf5_format::float16::f32_to_f16_bits(v as f32)), + 4 => u64::from((v as f32).to_bits()), + _ => v.to_bits(), + } + }; + let bytes = bits.to_le_bytes(); + if big_endian { + out.extend(bytes[..size].iter().rev()); + } else { + out.extend_from_slice(&bytes[..size]); + } + } + out +} + /// `r`/`i` compounds of two identical IEEE floats are complex numbers in h5py. fn complex_format( size: u32, @@ -366,6 +447,25 @@ impl Converter { vl_unit: 0, }) } + Datatype::FloatingPoint { .. } if !is_ieee_float(dt) => { + let Some((size, big_endian)) = widened_float(dt) else { + // The error names the layout. + return Err(float_format(dt).expect_err("not IEEE")); + }; + let order = if big_endian { '>' } else { '<' }; + let dtype = np_dtype(py, format!("{order}f{size}"))?; + Ok(Self { + view: dtype.clone().unbind(), + dtype: dtype.unbind(), + layout: Layout::Float { + source: dt.clone(), + size, + big_endian, + }, + elem_size: dt.type_size() as usize, + vl_unit: 0, + }) + } _ => { let dtype = fixed_dtype(py, dt)?; Ok(Self { @@ -416,6 +516,19 @@ impl Converter { (Elements::Bytes(bytes), Layout::Fixed) => { bytes_as_array(py, bytes, self.view.bind(py), shape) } + ( + Elements::Bytes(bytes), + Layout::Float { + source, + size, + big_endian, + }, + ) => { + let values = clawhdf5_format::data_read::read_as_f64(&bytes, source) + .map_err(|e| unsupported(e.to_string()))?; + let converted = encode_floats(&values, *size, *big_endian); + bytes_as_array(py, converted, self.view.bind(py), shape) + } (Elements::Bytes(bytes), Layout::Subarray(dims)) => { let mut full = shape.to_vec(); full.extend_from_slice(dims); diff --git a/crates/clawhdf5-py/src/edit.rs b/crates/clawhdf5-py/src/edit.rs index 05dd536..6646a4d 100644 --- a/crates/clawhdf5-py/src/edit.rs +++ b/crates/clawhdf5-py/src/edit.rs @@ -48,7 +48,12 @@ fn not_implemented(what: impl std::fmt::Display) -> PyErr { pub(crate) fn category(dt: &Datatype) -> PyResult<&'static str> { match dt { Datatype::FixedPoint { .. } => Ok("int"), - Datatype::FloatingPoint { .. } => Ok("float"), + Datatype::FloatingPoint { .. } if crate::convert::is_ieee_float(dt) => Ok("float"), + // Read as a wider IEEE float; writing would need the reverse + // conversion (rounding into bfloat16, FP8, ...). + Datatype::FloatingPoint { .. } => Err(not_implemented( + "writing non-IEEE floats (bfloat16, FP8, FP6, FP4, ...)", + )), Datatype::Enumeration { base_type, members, .. } => { diff --git a/crates/clawhdf5-py/tests/test_small_floats.py b/crates/clawhdf5-py/tests/test_small_floats.py new file mode 100644 index 0000000..f66cf8c --- /dev/null +++ b/crates/clawhdf5-py/tests/test_small_floats.py @@ -0,0 +1,92 @@ +"""Non-IEEE floats (bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2, FP4 E2M1) read +as h5py 3.16 reads them: as the narrowest IEEE float that holds every value +(bfloat16 as float32, the 1-byte formats as float16), in the file's byte +order, with the values libhdf5 converts them to. + +The fixture was written by libhdf5 2.2.0; the JSON next to it holds what +libhdf5 2.2.0 itself returns for every element (see gen_mx_floats.py).""" + +import json +import os +import shutil + +import numpy as np +import pytest + +import clawhdf5 + +FIXTURES = os.path.join(os.path.dirname(__file__), "..", "..", "clawhdf5", "tests", "fixtures") +FILE = os.path.join(FIXTURES, "mx_floats_hdf5_2_2.h5") +REFERENCE = json.load(open(os.path.join(FIXTURES, "mx_floats_hdf5_2_2.json")))["objects"] + +# What h5py 3.16 reports for each (checked 2026-09-28). +DTYPES = { + "bf16le": "f4", + "f8e4m3": "> 63 + bits = dtype.itemsize * 8 + word = (negative << (bits - 1)) | ((1 << (bits - 1)) - 1) + out.append(np.frombuffer(word.to_bytes(dtype.itemsize, "little"), dtype.newbyteorder("<"))[0]) + else: + out.append(float(text)) + return np.array(out, dtype=dtype.newbyteorder("<")).astype(dtype) + + +def test_values_match_libhdf5_2_2(): + assert set(DTYPES) == set(REFERENCE) + with clawhdf5.File(FILE, "r") as f: + for name, dtype in DTYPES.items(): + ds = f[name] + assert ds.dtype == np.dtype(dtype), name + assert ds.dtype.str == dtype, name + want = _expected(name, np.dtype(dtype)) + got = ds[()] + assert got.dtype.str == dtype + assert got.tobytes() == want.tobytes(), name + # Selections convert the same way. + assert ds[3:9].tobytes() == want[3:9].tobytes(), name + assert ds[[0, 5, 7]].tobytes() == want[[0, 5, 7]].tobytes(), name + assert ds[5].tobytes() == want[5].tobytes(), name + if REFERENCE[name]["attribute"]: + attr = f.attrs[name] + assert attr.dtype.str == dtype + assert attr.tobytes() == want.tobytes(), name + + +def test_matches_h5py(h5py): + with clawhdf5.File(FILE, "r") as ours, h5py.File(FILE, "r") as theirs: + for name in DTYPES: + assert ours[name].dtype == theirs[name].dtype, name + assert ours[name][()].tobytes() == theirs[name][()].tobytes(), name + if name in theirs.attrs: + assert ours.attrs[name].dtype == theirs.attrs[name].dtype + assert ours.attrs[name].tobytes() == theirs.attrs[name].tobytes() + + +def test_writing_is_refused(tmp_path): + path = tmp_path / "copy.h5" + shutil.copyfile(FILE, path) + with clawhdf5.File(str(path), "r+") as f: + with pytest.raises(NotImplementedError, match="non-IEEE"): + f["bf16le"][0] = 1.0 + with pytest.raises(NotImplementedError, match="non-IEEE"): + f["f4e2m1"][:] = np.zeros(16) + with open(path, "rb") as a, open(FILE, "rb") as b: + assert a.read() == b.read() diff --git a/crates/clawhdf5-tools/README.md b/crates/clawhdf5-tools/README.md index 5c1cd99..d47224e 100644 --- a/crates/clawhdf5-tools/README.md +++ b/crates/clawhdf5-tools/README.md @@ -109,7 +109,16 @@ virtual datasets. Known differences from h5dump: - Floats print at their own precision (a `float32` 0.1 prints as `0.1`), - which for some values is more digits than h5dump's `%g`. + which for some values is more digits than h5dump's `%g`; non-finite + values print as `Inf`, `-Inf` and `NaN` (h5dump: `inf`, `-inf`, `nan`, + `-nan`). +- HDF5 2.x's predefined small floats print under h5dump 2.x's names + (`H5T_FLOAT_BFLOAT16LE`, `H5T_FLOAT_F8E4M3`, ..., `H5T_FLOAT_F4E2M1`; + `ls -v` as h5ls 2.x does: `FP4 E2M1 4-bit float`). + `tests/mx_floats_dump.rs` checks this, and the values, against the output + of h5dump 2.2.0 stored with the fixture. h5dump 1.14 describes these types + instead (`8-bit floating-point 4-bit precision`); other non-IEEE floats + print as an `H5T_FLOAT { ... }` block. - A compound nested in a compound prints inline (`{ 1, 2.5 }`) where h5dump prints it as an indented block, one member per line; only the outer compound is a block. diff --git a/crates/clawhdf5-tools/src/dtype.rs b/crates/clawhdf5-tools/src/dtype.rs index c42c4eb..2c1a1fd 100644 --- a/crates/clawhdf5-tools/src/dtype.rs +++ b/crates/clawhdf5-tools/src/dtype.rs @@ -59,8 +59,81 @@ pub fn is_ieee(dt: &Datatype) -> bool { ) == std } +/// A float datatype predefined by libhdf5 2.x besides the IEEE ones. +struct SmallFloat { + /// h5dump's name (`H5T_FLOAT_F8E4M3`). + ddl: &'static str, + /// h5ls's description (`FP8 E4M3 8-bit float`). + long: &'static str, + /// The short name `ls` lists (`float8-e4m3`). + short: &'static str, +} + +/// The libhdf5 2.x predefined float `dt` is equal to (as `H5Tequal` sees it: +/// the same size, byte order, precision, offset, fields and bias), if any. +/// The 1-byte types are predefined little-endian only. +fn small_float(dt: &Datatype) -> Option { + let Datatype::FloatingPoint { + size, + byte_order, + bit_offset, + bit_precision, + exponent_location, + exponent_size, + mantissa_location, + mantissa_size, + exponent_bias, + } = dt + else { + return None; + }; + if *bit_offset != 0 || *mantissa_location != 0 { + return None; + } + let f = |ddl, long, short| Some(SmallFloat { ddl, long, short }); + let fields = ( + *size, + *bit_precision, + *exponent_location, + *exponent_size, + *mantissa_size, + *exponent_bias, + ); + match (fields, byte_order) { + ((2, 16, 7, 8, 7, 127), DatatypeByteOrder::LittleEndian) => f( + "H5T_FLOAT_BFLOAT16LE", + "bfloat16 16-bit little-endian float", + "bfloat16", + ), + ((2, 16, 7, 8, 7, 127), DatatypeByteOrder::BigEndian) => f( + "H5T_FLOAT_BFLOAT16BE", + "bfloat16 16-bit big-endian float", + "bfloat16-be", + ), + ((1, 8, 3, 4, 3, 7), DatatypeByteOrder::LittleEndian) => { + f("H5T_FLOAT_F8E4M3", "FP8 E4M3 8-bit float", "float8-e4m3") + } + ((1, 8, 2, 5, 2, 15), DatatypeByteOrder::LittleEndian) => { + f("H5T_FLOAT_F8E5M2", "FP8 E5M2 8-bit float", "float8-e5m2") + } + ((1, 6, 3, 2, 3, 1), DatatypeByteOrder::LittleEndian) => { + f("H5T_FLOAT_F6E2M3", "FP6 E2M3 6-bit float", "float6-e2m3") + } + ((1, 6, 2, 3, 2, 3), DatatypeByteOrder::LittleEndian) => { + f("H5T_FLOAT_F6E3M2", "FP6 E3M2 6-bit float", "float6-e3m2") + } + ((1, 4, 1, 2, 1, 1), DatatypeByteOrder::LittleEndian) => { + f("H5T_FLOAT_F4E2M1", "FP4 E2M1 4-bit float", "float4-e2m1") + } + _ => None, + } +} + /// Short name used by `ls`: `int32`, `float64-be`, `string[3]`, ... pub fn short(dt: &Datatype) -> String { + if let Some(f) = small_float(dt) { + return f.short.into(); + } match dt { Datatype::FixedPoint { size, @@ -136,6 +209,9 @@ fn cset_word(c: &CharacterSet) -> &'static str { /// h5ls -v style description. pub fn long(dt: &Datatype) -> String { + if let Some(f) = small_float(dt) { + return f.long.into(); + } match dt { Datatype::FixedPoint { size, @@ -277,6 +353,8 @@ fn atomic_ddl(dt: &Datatype) -> Option { u64::from(*size) * 8, order_suffix(byte_order) )), + // As h5dump 2.x names them (checked against h5dump 2.2.0). + Datatype::FloatingPoint { .. } => small_float(dt).map(|f| f.ddl.to_string()), Datatype::BitField { size, byte_order, .. } => Some(format!( diff --git a/crates/clawhdf5-tools/tests/mx_floats_dump.rs b/crates/clawhdf5-tools/tests/mx_floats_dump.rs new file mode 100644 index 0000000..51f859f --- /dev/null +++ b/crates/clawhdf5-tools/tests/mx_floats_dump.rs @@ -0,0 +1,111 @@ +//! `h5rs dump` and `h5rs ls` of the small floats libhdf5 2.x predefines +//! (bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2, FP4 E2M1), against the output +//! of h5dump 2.2.0 and h5ls 2.2.0 stored next to the fixture (the Debian +//! h5dump CI installs, 1.14.x, predates these types). See +//! `crates/clawhdf5/tests/fixtures/gen_mx_floats.py`. + +use std::path::PathBuf; +use std::process::Command; + +fn fixture(name: &str) -> PathBuf { + PathBuf::from(env!("CARGO_MANIFEST_DIR")) + .join("../clawhdf5/tests/fixtures") + .join(name) +} + +fn h5rs(args: &[&str]) -> String { + let out = Command::new(env!("CARGO_BIN_EXE_h5rs")) + .args(args) + .output() + .unwrap(); + assert!(out.status.success(), "h5rs {args:?}: {out:?}"); + String::from_utf8(out.stdout).unwrap() +} + +/// Split a dump into the lines outside `DATA { ... }` blocks and the values +/// inside them, per block. +fn split(ddl: &str) -> (Vec<&str>, Vec>) { + let (mut frame, mut data) = (Vec::new(), Vec::new()); + let mut values: Option> = None; + for line in ddl.lines() { + match &mut values { + None if line.trim() == "DATA {" => values = Some(Vec::new()), + None => frame.push(line), + Some(v) if line.trim() == "}" => { + data.push(std::mem::take(v)); + values = None; + } + Some(v) => { + let (_, body) = line.split_once("):").expect("an indexed data line"); + v.extend( + body.split(',') + .map(str::trim) + .filter(|s| !s.is_empty()) + .map(String::from), + ); + } + } + } + (frame, data) +} + +#[test] +fn dump_matches_h5dump_2_2() { + let path = fixture("mx_floats_hdf5_2_2.h5"); + let ours = h5rs(&["dump", path.to_str().unwrap()]); + let reference = std::fs::read_to_string(fixture("mx_floats_hdf5_2_2.ddl")).unwrap(); + let (our_frame, our_data) = split(&ours); + let (ref_frame, ref_data) = split(&reference); + // Everything but the values is h5dump's byte for byte: the datatypes + // print as H5T_FLOAT_BFLOAT16LE, H5T_FLOAT_F4E2M1, ... + assert_eq!(our_frame, ref_frame); + assert_eq!(our_data.len(), 17); + assert_eq!(our_data.len(), ref_data.len()); + // Values: h5dump prints `%g` (6 significant digits) and inf/-inf/nan/ + // -nan; h5rs prints the shortest string that round-trips and Inf/-Inf/NaN. + for (block, (ours, theirs)) in our_data.iter().zip(&ref_data).enumerate() { + assert_eq!(ours.len(), theirs.len(), "block {block}"); + for (i, (o, r)) in ours.iter().zip(theirs).enumerate() { + let o: f64 = o.parse().unwrap(); + let r: f64 = r.parse().unwrap(); + let same = if r.is_nan() || r.is_infinite() { + o.is_nan() == r.is_nan() && (r.is_nan() || o == r) + } else { + // Both round to the same f32 (a bfloat16 subnormal prints + // as its shortest f32 string, `9.1835e-41`, fewer digits + // than h5dump's), or agree to h5dump's 6 digits. + o as f32 == r as f32 || (o - r).abs() <= 5e-6 * r.abs() + }; + assert!(same, "block {block}[{i}]: h5rs {o}, h5dump {r}"); + } + } +} + +#[test] +fn ls_names_small_floats_like_h5ls_2_2() { + let path = fixture("mx_floats_hdf5_2_2.h5"); + // h5ls 2.2.0 -v, `Type:` line of each dataset. + for (name, long, short) in [ + ("bf16le", "bfloat16 16-bit little-endian float", "bfloat16"), + ("bf16be", "bfloat16 16-bit big-endian float", "bfloat16-be"), + ("f8e4m3", "FP8 E4M3 8-bit float", "float8-e4m3"), + ("f8e5m2", "FP8 E5M2 8-bit float", "float8-e5m2"), + ("f6e2m3", "FP6 E2M3 6-bit float", "float6-e2m3"), + ("f6e3m2", "FP6 E3M2 6-bit float", "float6-e3m2"), + ("f4e2m1", "FP4 E2M1 4-bit float", "float4-e2m1"), + ] { + let target = format!("{}/{name}", path.display()); + let verbose = h5rs(&["ls", "-v", &target]); + assert!( + verbose.contains(&format!(" Type: {long}\n")), + "{name}:\n{verbose}" + ); + let listing = h5rs(&["ls", path.to_str().unwrap()]); + assert!( + listing + .lines() + .any(|l| l.starts_with(&format!("{name} ")) && l.ends_with(&format!(" {short}"))), + "{name}:\n{listing}" + ); + } +} diff --git a/crates/clawhdf5/tests/fixtures/gen_mx_floats.py b/crates/clawhdf5/tests/fixtures/gen_mx_floats.py new file mode 100644 index 0000000..0d03b2e --- /dev/null +++ b/crates/clawhdf5/tests/fixtures/gen_mx_floats.py @@ -0,0 +1,218 @@ +"""Generate mx_floats_hdf5_2_2.h5 and mx_floats_hdf5_2_2.json. + +The file holds one dataset and one root attribute per small-float datatype +predefined by libhdf5 2.x: + + H5T_FLOAT_F4E2M1 all 16 bit patterns + H5T_FLOAT_F6E2M3 all 64 bit patterns + H5T_FLOAT_F6E3M2 all 64 bit patterns + H5T_FLOAT_F8E4M3 all 256 bit patterns + H5T_FLOAT_F8E5M2 all 256 bit patterns + H5T_FLOAT_BFLOAT16LE a representative set (zeros, subnormals, normals, + H5T_FLOAT_BFLOAT16BE max, +-inf, quiet/signalling/negative NaN) + +plus `_pad` datasets for the 4- and 6-bit types whose bytes also have +every unused high bit set (libhdf5 ignores bits outside the precision). + +The raw bit patterns are written with the file type as the memory type (no +conversion). The JSON is the reference: for every element, the bits of the +f64 and f32 values that libhdf5 itself returns from H5Dread/H5Aread into +H5T_NATIVE_DOUBLE/H5T_NATIVE_FLOAT, and each conversion exception libhdf5 +raised through H5Pset_type_conv_cb (the callback returns UNHANDLED, so the +values are libhdf5's defaults). + +It talks to libhdf5 2.2.0 through ctypes only (no h5py: h5py 3.16 bundles +libhdf5 2.0.0, which predates these types). libhdf5 was built from source: + + git clone --depth 1 --branch 2.2.0 https://github.com/HDFGroup/hdf5 \ + ~/.cache/hdf5-2.2.0-src # commit 49df1b4e5ca4108d56e80f06321a749da1535268 + cmake -S ~/.cache/hdf5-2.2.0-src -B ~/.cache/hdf5-2.2.0-build \ + -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$HOME/.cache/hdf5-2.2.0 \ + -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=OFF -DHDF5_ENABLE_ZLIB_SUPPORT=ON \ + -DHDF5_BUILD_FORTRAN=OFF -DHDF5_BUILD_JAVA=OFF -DHDF5_BUILD_CPP_LIB=OFF \ + -DBUILD_TESTING=OFF -DHDF5_BUILD_EXAMPLES=OFF -DHDF5_BUILD_TOOLS=ON \ + -DHDF5_BUILD_HL_LIB=OFF + cmake --build ~/.cache/hdf5-2.2.0-build -j8 + cmake --install ~/.cache/hdf5-2.2.0-build + +Re-run only if the fixture ever needs regenerating (generated 2026-09-28): + + python gen_mx_floats.py ~/.cache/hdf5-2.2.0/lib/libhdf5.so mx_floats_hdf5_2_2 + +(h5dump from the same build writes mx_floats_hdf5_2_2.ddl: +`h5dump mx_floats_hdf5_2_2.h5 > mx_floats_hdf5_2_2.ddl`.) +""" +import ctypes +import json +import struct +import sys + +lib = ctypes.CDLL(sys.argv[1]) +out = sys.argv[2] +hid = ctypes.c_int64 +herr = ctypes.c_int +assert lib.H5open() >= 0 + +majnum, minnum, relnum = ctypes.c_uint(), ctypes.c_uint(), ctypes.c_uint() +lib.H5get_libversion(ctypes.byref(majnum), ctypes.byref(minnum), ctypes.byref(relnum)) +version = f"{majnum.value}.{minnum.value}.{relnum.value}" +assert version == "2.2.0", version + + +def g(name): + return hid.in_dll(lib, name + "_g").value + + +for fn, res, args in [ + ("H5Fcreate", hid, [ctypes.c_char_p, ctypes.c_uint, hid, hid]), + ("H5Fclose", herr, [hid]), + ("H5Fopen", hid, [ctypes.c_char_p, ctypes.c_uint, hid]), + ("H5Screate_simple", hid, [ctypes.c_int, ctypes.POINTER(ctypes.c_uint64), ctypes.c_void_p]), + ("H5Sclose", herr, [hid]), + ("H5Dcreate2", hid, [hid, ctypes.c_char_p, hid, hid, hid, hid, hid]), + ("H5Dopen2", hid, [hid, ctypes.c_char_p, hid]), + ("H5Dwrite", herr, [hid, hid, hid, hid, hid, ctypes.c_void_p]), + ("H5Dread", herr, [hid, hid, hid, hid, hid, ctypes.c_void_p]), + ("H5Dclose", herr, [hid]), + ("H5Acreate2", hid, [hid, ctypes.c_char_p, hid, hid, hid, hid]), + ("H5Aopen", hid, [hid, ctypes.c_char_p, hid]), + ("H5Awrite", herr, [hid, hid, ctypes.c_void_p]), + ("H5Aread", herr, [hid, hid, ctypes.c_void_p]), + ("H5Aclose", herr, [hid]), + ("H5Pcreate", hid, [hid]), + ("H5Pclose", herr, [hid]), + ("H5Pset_type_conv_cb", herr, [hid, ctypes.c_void_p, ctypes.c_void_p]), + ("H5Tget_fields", herr, [hid] + [ctypes.POINTER(ctypes.c_size_t)] * 5), + ("H5Tget_ebias", ctypes.c_size_t, [hid]), + ("H5Tget_precision", ctypes.c_size_t, [hid]), + ("H5Tget_offset", ctypes.c_int, [hid]), + ("H5Tget_size", ctypes.c_size_t, [hid]), + ("H5Tget_order", ctypes.c_int, [hid]), +]: + getattr(lib, fn).restype = res + getattr(lib, fn).argtypes = args + +H5F_ACC_TRUNC, H5F_ACC_RDONLY = 0x0002, 0x0000 +H5P_DEFAULT, H5S_ALL = 0, 0 + +NATIVE_DOUBLE = g("H5T_NATIVE_DOUBLE") +NATIVE_FLOAT = g("H5T_NATIVE_FLOAT") +BF16 = [0x0000, 0x8000, 0x3F80, 0xBF80, 0x3FC0, 0xC010, 0x4049, 0x3F81, + 0x0001, 0x8001, 0x007F, 0x0080, 0x7F7F, 0xFF7F, 0x7F80, 0xFF80, + 0x7FC0, 0x7F81, 0xFFC1, 0x7FFF] + +# name -> (libhdf5 global, raw element bytes) +TYPES = { + "f4e2m1": ("H5T_FLOAT_F4E2M1", [bytes([b]) for b in range(16)]), + "f4e2m1_pad": ("H5T_FLOAT_F4E2M1", [bytes([0xF0 | b]) for b in range(16)]), + "f6e2m3": ("H5T_FLOAT_F6E2M3", [bytes([b]) for b in range(64)]), + "f6e2m3_pad": ("H5T_FLOAT_F6E2M3", [bytes([0xC0 | b]) for b in range(64)]), + "f6e3m2": ("H5T_FLOAT_F6E3M2", [bytes([b]) for b in range(64)]), + "f6e3m2_pad": ("H5T_FLOAT_F6E3M2", [bytes([0xC0 | b]) for b in range(64)]), + "f8e4m3": ("H5T_FLOAT_F8E4M3", [bytes([b]) for b in range(256)]), + "f8e5m2": ("H5T_FLOAT_F8E5M2", [bytes([b]) for b in range(256)]), + "bf16le": ("H5T_FLOAT_BFLOAT16LE", [struct.pack("H", v) for v in BF16]), +} +EXCEPT_NAMES = ["RANGE_HI", "RANGE_LOW", "PRECISION", "TRUNCATE", "PINF", "NINF", "NAN"] + +fid = lib.H5Fcreate((out + ".h5").encode(), H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT) +assert fid >= 0 +for name, (tname, raw) in TYPES.items(): + t = g(tname) + buf = ctypes.create_string_buffer(b"".join(raw), len(raw) * len(raw[0])) + sid = lib.H5Screate_simple(1, (ctypes.c_uint64 * 1)(len(raw)), None) + did = lib.H5Dcreate2(fid, name.encode(), t, sid, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT) + assert did >= 0 and lib.H5Dwrite(did, t, H5S_ALL, H5S_ALL, H5P_DEFAULT, buf) >= 0 + lib.H5Dclose(did) + if not name.endswith("_pad"): + aid = lib.H5Acreate2(fid, name.encode(), t, sid, H5P_DEFAULT, H5P_DEFAULT) + assert aid >= 0 and lib.H5Awrite(aid, t, buf) >= 0 + lib.H5Aclose(aid) + lib.H5Sclose(sid) +assert lib.H5Fclose(fid) >= 0 + +# --- reference: what libhdf5 converts each element to -------------------- +CB = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int, hid, hid, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p) +events = [] + + +@CB +def on_except(kind, src_id, dst_id, src_buf, dst_buf, user): + events.append((EXCEPT_NAMES[kind], ctypes.string_at(src_buf, 1 if lib.H5Tget_size(src_id) == 1 else 2).hex())) + return 0 # H5T_CONV_UNHANDLED: let libhdf5 apply its default + + +def f64_text(b): + """The f64 as Python's round-tripping repr ("0.125", "inf", "-inf"), or + "nan:" for a NaN, whose sign and payload are kept exactly.""" + (bits,) = struct.unpack("= 0 + else: + # H5Aread takes no transfer property list, so no exception callback. + assert lib.H5Aread(obj, mem, buf) >= 0 + return [buf.raw[i * width:(i + 1) * width] for i in range(n)], sorted(events) + + +fid = lib.H5Fopen((out + ".h5").encode(), H5F_ACC_RDONLY, H5P_DEFAULT) +dxpl = lib.H5Pcreate(g("H5P_CLS_DATASET_XFER_ID")) +assert lib.H5Pset_type_conv_cb(dxpl, ctypes.cast(on_except, ctypes.c_void_p), None) >= 0 +ref = { + "libhdf5": version, + "generator": "gen_mx_floats.py", + "note": "f64: what H5Dread into H5T_NATIVE_DOUBLE returns per element; " + "H5Aread (attribute of the same name) and H5T_NATIVE_FLOAT return the same values " + "(f32 NaN bits in f32_nan); exceptions: H5Pset_type_conv_cb events by source bytes", + "objects": {}, +} +for name, (tname, raw) in TYPES.items(): + t = g(tname) + sizes = [ctypes.c_size_t() for _ in range(5)] + lib.H5Tget_fields(t, *[ctypes.byref(s) for s in sizes]) + n = len(raw) + did = lib.H5Dopen2(fid, name.encode(), H5P_DEFAULT) + d64, exc64 = read_all(did, True, NATIVE_DOUBLE, 8, n) + d32, exc32 = read_all(did, True, NATIVE_FLOAT, 4, n) + lib.H5Dclose(did) + assert exc32 == exc64, name + for a, b in zip(d64, d32): + (x,) = struct.unpack(", + /// Bits of the `f32` NaNs libhdf5 returns (every non-NaN value is the + /// `f64` value, exact in `f32`). + f32_nan: Vec, +} + +fn reference() -> Vec { + let json: serde_json::Value = serde_json::from_str(REFERENCE).unwrap(); + assert_eq!(json["libhdf5"], "2.2.0"); + json["objects"] + .as_object() + .unwrap() + .iter() + .map(|(name, obj)| Expected { + name: name.clone(), + attribute: obj["attribute"].as_bool().unwrap(), + f64_bits: obj["f64"] + .as_array() + .unwrap() + .iter() + .map(|v| { + let text = v.as_str().unwrap(); + match text.strip_prefix("nan:") { + Some(bits) => u64::from_str_radix(bits, 16).unwrap(), + None => text.parse::().unwrap().to_bits(), + } + }) + .collect(), + f32_nan: obj["f32_nan"] + .as_array() + .unwrap() + .iter() + .map(|v| u32::from_str_radix(v.as_str().unwrap(), 16).unwrap()) + .collect(), + }) + .collect() +} + +fn check_f64(what: &str, got: &[f64], want: &[u64]) { + assert_eq!(got.len(), want.len(), "{what}: element count"); + for (i, (g, w)) in got.iter().zip(want).enumerate() { + assert_eq!( + g.to_bits(), + *w, + "{what}[{i}]: clawhdf5 {g:?} ({:#018x}), libhdf5 {:?} ({w:#018x})", + g.to_bits(), + f64::from_bits(*w) + ); + } +} + +fn check_f32(what: &str, got: &[f32], want: &Expected) { + assert_eq!(got.len(), want.f64_bits.len(), "{what}: element count"); + for (i, (g, w)) in got.iter().zip(&want.f64_bits).enumerate() { + let w = f64::from_bits(*w); + if w.is_nan() { + // libhdf5 keeps the sign and sets every mantissa bit. + let expected = if w.is_sign_negative() { + 0xFFFF_FFFF + } else { + 0x7FFF_FFFF + }; + assert!(want.f32_nan.contains(&expected)); + assert_eq!(g.to_bits(), expected, "{what}[{i}]: NaN bits"); + } else { + assert_eq!(g.to_bits(), (w as f32).to_bits(), "{what}[{i}]: {g} vs {w}"); + } + } +} + +#[test] +fn small_floats_decode_as_libhdf5_2_2_does() { + let file = File::open(FIXTURE).unwrap(); + let root_attrs = file.root().attrs().unwrap(); + let expected = reference(); + assert_eq!(expected.len(), 10); + for want in &expected { + let name = &want.name; + let ds = file.dataset(name).unwrap(); + check_f64(name, &ds.read_f64().unwrap(), &want.f64_bits); + check_f32(name, &ds.read_f32().unwrap(), want); + if want.attribute { + match &root_attrs[name.as_str()] { + AttrValue::F64Array(values) => { + check_f64(&format!("attribute {name}"), values, &want.f64_bits) + } + other => panic!("attribute {name}: {other:?}"), + } + } + } +} + +#[test] +fn small_float_datatypes_keep_their_fields() { + use clawhdf5_format::datatype::Datatype; + let json: serde_json::Value = serde_json::from_str(REFERENCE).unwrap(); + let file = File::open(FIXTURE).unwrap(); + for (name, obj) in json["objects"].as_object().unwrap() { + let dt = file.dataset(name).unwrap().raw_datatype().unwrap(); + let Datatype::FloatingPoint { + size, + bit_offset, + bit_precision, + exponent_location, + exponent_size, + mantissa_location, + mantissa_size, + exponent_bias, + .. + } = dt + else { + panic!("{name}: {dt:?}"); + }; + let field = |i: usize| obj["fields"][i].as_u64().unwrap(); + assert_eq!(u64::from(size), obj["size"].as_u64().unwrap(), "{name}"); + assert_eq!(u64::from(bit_offset), obj["offset"].as_u64().unwrap()); + assert_eq!( + u64::from(bit_precision), + obj["precision"].as_u64().unwrap(), + "{name}" + ); + // Fields: sign position, exponent position and size, mantissa + // position and size. + assert_eq!( + u64::from(exponent_location) + u64::from(exponent_size), + field(0) + ); + assert_eq!(u64::from(exponent_location), field(1), "{name}"); + assert_eq!(u64::from(exponent_size), field(2), "{name}"); + assert_eq!(u64::from(mantissa_location), field(3), "{name}"); + assert_eq!(u64::from(mantissa_size), field(4), "{name}"); + assert_eq!(u64::from(exponent_bias), obj["ebias"].as_u64().unwrap()); + } +} diff --git a/docs/known-issues.md b/docs/known-issues.md index 8cc1266..947c356 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -16,6 +16,9 @@ Checked against `main` at `9b5803f` on 2026-09-28. | Issue | Kind | Since | |---|---|---| | [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 | + +| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 | +| [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 | | [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 | | [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 | | [Selection reads that decode more than the selection](#selection-reads-that-decode-more-than-the-selection) | speed only | 2026-09-26 | @@ -420,6 +423,26 @@ unwritten records read as fill); `Variable::shape` is the HDF5 extent, Workaround: read the variable's `_Netcdf4Coordinates` attribute (dimension ids, matching each scale's `_Netcdf4Dimid`). +## Small floats decode as libhdf5 does, not as the OCP MX specification + +**Status:** open, deliberate (documented 2026-09-28). HDF5 2.x predefines +bfloat16, FP8 E4M3/E5M2, FP6 E2M3/E3M2 and FP4 E2M1. clawhdf5 decodes every +bit pattern of these types exactly as libhdf5 2.2.0 does (all 16/64/256 +patterns and a bfloat16 set compared bit for bit, NaN bits included: +`crates/clawhdf5/tests/mx_floats_interop.rs` over a fixture libhdf5 2.2.0 +wrote, `fixtures/gen_mx_floats.py`), and h5py 3.16 (libhdf5 2.0.0) returns +the same values. libhdf5 treats an all-ones exponent as infinity or NaN in +every one of them, IEEE-style, whereas the OCP Microscaling formats +specification gives FP4 and FP6 no infinity or NaN and FP8 E4M3 a single NaN +(`S.1111.111`). So values that are finite under the OCP specification read +as ±inf or NaN in both libraries: FP4 E2M1 4.0 and 6.0 (`0b0110`, `0b0111`), +FP6 E2M3 above 3.75, FP6 E3M2 above 14, and FP8 E4M3 256 to 448 (`0x78` to +`0x7E`). libhdf5's own release notes (2.2.0, *Known Problems*) say its +conversions of these types "may improperly convert some data values"; if it +changes that, clawhdf5 should follow. Workaround for data written under the +OCP meaning: read the element bytes (`Dataset::read_selection(&Selection::All)`) +and the type's fields (`Dataset::raw_datatype`) and decode them yourself. + ## The Node.js package (`packages/clawhdf5-node`) does not work **Status:** open (found 2026-09-25; re-checked 2026-09-28, unchanged).