quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases 1 Wiki Activity

HDF5 2.x small floats (FP4/FP6/FP8/bfloat16) checked against libhdf5 2.2.0 #25

Merged
osobh merged 1 commits from feat/mx-floats into main 2026-09-29 03:11:09 +00:00
Conversation 0 Commits 1 Files Changed 16
+1207 -13
osobh commented 2026-09-29 02:38:32 +00:00
Owner
Copy Link
Copy Source

Merge order: 3 of 4. This PR is stacked on #24.

This checks HDF5 2.x's small floats against libhdf5 2.2.0 itself. The types are bfloat16 LE/BE, FP8 E4M3/E5M2, FP6 E2M3/E3M2 (added in 2.1.0) and FP4 E2M1 (2.1.0).

Fixtures: libhdf5 2.2.0 was built from source, and gen_mx_floats.py (ctypes) writes every bit pattern of the FP4/FP6/FP8 types plus 20 bfloat16 values. It also records what libhdf5's H5Dread/H5Aread returns for each value into NATIVE_DOUBLE/NATIVE_FLOAT (JSON), and the h5dump 2.2 output.

Results:

  • Every value already matched libhdf5. The only difference was the NaN payload bits, which now match too.
  • libhdf5 treats an all-ones exponent as inf/NaN even in FP4, FP6 and FP8 E4M3, where the OCP MX spec says those values are finite. We follow libhdf5, and a known-issues entry documents the difference.

Other changes:

  • h5rs dump uses h5dump 2.2's type names (H5T_FLOAT_F4E2M1, …); all 1536 values match h5dump 2.2.
  • h5rs ls shows names like bfloat16 and float8-e4m3.
  • Python reads these types the way h5py 3.16 does, byte for byte (they used to raise TypeError).

Tests: mx_floats_interop.rs compares bit for bit against the JSON; mx_floats_dump.rs checks h5rs output; test_small_floats.py covers Python.

🤖 Generated with Claude Code

**Merge order: 3 of 4.** This PR is stacked on #24. This checks HDF5 2.x's small floats against libhdf5 2.2.0 itself. The types are bfloat16 LE/BE, FP8 E4M3/E5M2, FP6 E2M3/E3M2 (added in 2.1.0) and FP4 E2M1 (2.1.0). **Fixtures:** libhdf5 2.2.0 was built from source, and `gen_mx_floats.py` (ctypes) writes every bit pattern of the FP4/FP6/FP8 types plus 20 bfloat16 values. It also records what libhdf5's `H5Dread`/`H5Aread` returns for each value into `NATIVE_DOUBLE`/`NATIVE_FLOAT` (JSON), and the h5dump 2.2 output. **Results:** - Every value already matched libhdf5. The only difference was the NaN payload bits, which now match too. - libhdf5 treats an all-ones exponent as inf/NaN even in FP4, FP6 and FP8 E4M3, where the OCP MX spec says those values are finite. We follow libhdf5, and a known-issues entry documents the difference. **Other changes:** - `h5rs dump` uses h5dump 2.2's type names (`H5T_FLOAT_F4E2M1`, …); all 1536 values match h5dump 2.2. - `h5rs ls` shows names like `bfloat16` and `float8-e4m3`. - Python reads these types the way h5py 3.16 does, byte for byte (they used to raise `TypeError`). **Tests:** `mx_floats_interop.rs` compares bit for bit against the JSON; `mx_floats_dump.rs` checks `h5rs` output; `test_small_floats.py` covers Python. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
osobh added 3 commits 2026-09-29 02:38:32 +00:00
clawhdf5: a dropped FileEditor releases its lock at once
CI / test-arm64 (pull_request) Successful in 1m38s
Details
CI / test (pull_request) Successful in 28m11s
Details
3eca5d8334 
FileEditor's flock belongs to the open file description. When another
thread forks to spawn a process, the child shares the locked descriptor
until it execs, so a reopen right after the drop could be refused with
Error::Locked (a one-off failure of edit_interop::editor_locks_the_file in
a parallel test run). Drop now unlocks before closing, which releases the
lock for every descriptor sharing it.

Reproducer edit_tests::drop_releases_the_lock_while_other_threads_spawn_processes
(4 threads running `true`, 2000 open/drop rounds): 1483 of 2000 reopens
refused before, 0 in 30 runs after (tank). An OFD lock would not help: it
is inherited across fork the same way and does not conflict with
libhdf5's flock. The agent store's lock file unlocks on drop too.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-netcdf4: an unlimited dimension reports its current length
CI / test-arm64 (pull_request) Successful in 1m34s
Details
CI / test (pull_request) Successful in 30m38s
Details
31efac1ae1 
Dimension::size of an unlimited dimension was its dimension scale's
extent, which netCDF-C leaves at 0, so it read 0 for a dimension holding
records. It is now what netCDF-C reports (nc4_find_dim_len): the largest
current extent of the variables using it in any group, found through the
scale's REFERENCE_LIST, a coordinate variable's own extent included; 0
when nothing has been written.

interop_tests::unlimited_dimension_lengths_match_netcdf4_python compares
with netCDF4-python (variables of different lengths, one in a subgroup,
an unwritten dimension, a coordinate variable shorter than another
variable on its dimension, a subgroup's own dimension); before the fix it
got time 0/6, rec 3/5, srec 0/1.

The known-issues entry moves to Fixed; the crate README's warning goes.
A new open entry records a related bug found meanwhile: variables'
dimensions are matched by size, not DIMENSION_LIST.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
feat: check HDF5 2.x small floats against libhdf5 2.2.0
CI / test-arm64 (pull_request) Successful in 1m38s
Details
CI / test (pull_request) Successful in 31m22s
Details
bce07e9cb9 
Fixture written by libhdf5 2.2.0 (built from tag 2.2.0) through ctypes:
every bit pattern of FP4 E2M1, FP6 E2M3/E3M2, FP8 E4M3/E5M2 and a
bfloat16 LE/BE set, as datasets and attributes, with what H5Dread/H5Aread
return into double and float and the conversion exceptions libhdf5
raises. clawhdf5 already decoded every value as libhdf5 does, including
an all-ones exponent as inf/NaN in the OCP formats that have none
(documented as a deliberate match in known-issues).

- data_read: NaNs of non-native float layouts get libhdf5's bits (sign
  kept, every mantissa bit set) in f64 and f32.
- h5rs dump/ls name these types as h5dump/h5ls 2.x do
  (H5T_FLOAT_F4E2M1, "FP4 E2M1 4-bit float", float4-e2m1 ...), checked
  against h5dump 2.2.0's output of the fixture.
- Python bindings read them as h5py 3.16 does (float32 for bfloat16,
  float16 for the 1-byte formats, file byte order, same bytes as h5py);
  writing them in 'r+' is refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
osobh merged commit 56684ff147 into main 2026-09-29 03:11:09 +00:00
osobh deleted branch feat/mx-floats 2026-09-29 03:11:13 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
nsobh osobh scottb (Scott B.)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#25