Write complex numbers, incl. HDF5 2.0 native complex (class 11) #26

Merged

osobh merged 3 commits from feat/write-complex into main 2026-09-29 03:11:22 +00:00

Conversation 0 Commits 3 Files Changed 21

+791 -27

osobh commented 2026-09-29 02:38:33 +00:00

Owner

Copy Link

Copy Source

Merge order: 4 of 4. This PR is stacked on #25.

clawhdf5 can now write complex numbers, in datasets and attributes:

• Default: h5py's {r, i} compound (with_complex_f32_data / with_complex_f64_data, make_complex_f*_type).
• Opt-in: HDF5 2.0's native complex type, class 11 (with_native_complex_f*_data, make_native_complex_f*_type). Only libhdf5 2.0+ can read it.

Checks:

• The native encoding is byte-identical to libhdf5 2.2.0's.
• h5py 3.16 (libhdf5 2.0.0) reads it back as complex64/complex128, in contiguous, chunked+deflate, compound-member, array and attribute forms.
• h5dump 2.2.0 prints H5T_COMPLEX_IEEE_F*, and h5rs check --data finds no problems.

New read API: Dataset::read_complex_f64 / read_complex_f32 read either form. Python's create_dataset accepts complex64/complex128 arrays and writes the compound form, as h5py does.

API note: this adds a new variant, Datatype::Complex, to the public Datatype enum. That is a breaking change for any downstream exhaustive match. ClawBrainHub doesn't match on Datatype, and its 204 tests pass against this stack.

Limitation: a class 11 type still reads as its {r, i} compound view (dtype(), h5rs dump, wasm), so h5rs dump differs from h5dump 2.x on these files. Values are correct. This is recorded in known-issues.

Also here: the README datatypes row is merged across #25 and #26, and the two Fixed entries in known-issues now cite #23 and #24.

Verification:

• scripts/ci-test.sh on this stack (all four PRs): 27/27 steps pass (tank, 2026-09-28).
• ClawBrainHub: 204/204 tests pass.

🤖 Generated with Claude Code

**Merge order: 4 of 4.** This PR is stacked on #25. clawhdf5 can now write complex numbers, in datasets and attributes: - **Default:** h5py's `{r, i}` compound (`with_complex_f32_data` / `with_complex_f64_data`, `make_complex_f*_type`). - **Opt-in:** HDF5 2.0's native complex type, class 11 (`with_native_complex_f*_data`, `make_native_complex_f*_type`). Only libhdf5 2.0+ can read it. **Checks:** - The native encoding is byte-identical to libhdf5 2.2.0's. - h5py 3.16 (libhdf5 2.0.0) reads it back as complex64/complex128, in contiguous, chunked+deflate, compound-member, array and attribute forms. - h5dump 2.2.0 prints `H5T_COMPLEX_IEEE_F*`, and `h5rs check --data` finds no problems. **New read API:** `Dataset::read_complex_f64` / `read_complex_f32` read either form. Python's `create_dataset` accepts complex64/complex128 arrays and writes the compound form, as h5py does. **API note:** this adds a new variant, `Datatype::Complex`, to the public `Datatype` enum. That is a breaking change for any downstream exhaustive `match`. ClawBrainHub doesn't match on `Datatype`, and its 204 tests pass against this stack. **Limitation:** a class 11 type still reads as its `{r, i}` compound view (`dtype()`, `h5rs dump`, wasm), so `h5rs dump` differs from h5dump 2.x on these files. Values are correct. This is recorded in known-issues. Also here: the README datatypes row is merged across #25 and #26, and the two Fixed entries in known-issues now cite #23 and #24. **Verification:** - `scripts/ci-test.sh` on this stack (all four PRs): 27/27 steps pass (tank, 2026-09-28). - ClawBrainHub: 204/204 tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

osobh added 5 commits 2026-09-29 02:38:33 +00:00

clawhdf5: a dropped FileEditor releases its lock at once ... 3eca5d8334

FileEditor's flock belongs to the open file description. When another
thread forks to spawn a process, the child shares the locked descriptor
until it execs, so a reopen right after the drop could be refused with
Error::Locked (a one-off failure of edit_interop::editor_locks_the_file in
a parallel test run). Drop now unlocks before closing, which releases the
lock for every descriptor sharing it.

Reproducer edit_tests::drop_releases_the_lock_while_other_threads_spawn_processes
(4 threads running `true`, 2000 open/drop rounds): 1483 of 2000 reopens
refused before, 0 in 30 runs after (tank). An OFD lock would not help: it
is inherited across fork the same way and does not conflict with
libhdf5's flock. The agent store's lock file unlocks on drop too.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

clawhdf5-netcdf4: an unlimited dimension reports its current length ... 31efac1ae1

Dimension::size of an unlimited dimension was its dimension scale's
extent, which netCDF-C leaves at 0, so it read 0 for a dimension holding
records. It is now what netCDF-C reports (nc4_find_dim_len): the largest
current extent of the variables using it in any group, found through the
scale's REFERENCE_LIST, a coordinate variable's own extent included; 0
when nothing has been written.

interop_tests::unlimited_dimension_lengths_match_netcdf4_python compares
with netCDF4-python (variables of different lengths, one in a subgroup,
an unwritten dimension, a coordinate variable shorter than another
variable on its dimension, a subgroup's own dimension); before the fix it
got time 0/6, rec 3/5, srec 0/1.

The known-issues entry moves to Fixed; the crate README's warning goes.
A new open entry records a related bug found meanwhile: variables'
dimensions are matched by size, not DIMENSION_LIST.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

feat: check HDF5 2.x small floats against libhdf5 2.2.0 ... bce07e9cb9

Fixture written by libhdf5 2.2.0 (built from tag 2.2.0) through ctypes:
every bit pattern of FP4 E2M1, FP6 E2M3/E3M2, FP8 E4M3/E5M2 and a
bfloat16 LE/BE set, as datasets and attributes, with what H5Dread/H5Aread
return into double and float and the conversion exceptions libhdf5
raises. clawhdf5 already decoded every value as libhdf5 does, including
an all-ones exponent as inf/NaN in the OCP formats that have none
(documented as a deliberate match in known-issues).

- data_read: NaNs of non-native float layouts get libhdf5's bits (sign
  kept, every mantissa bit set) in f64 and f32.
- h5rs dump/ls name these types as h5dump/h5ls 2.x do
  (H5T_FLOAT_F4E2M1, "FP4 E2M1 4-bit float", float4-e2m1 ...), checked
  against h5dump 2.2.0's output of the fixture.
- Python bindings read them as h5py 3.16 does (float32 for bfloat16,
  float16 for the 1-byte formats, file byte order, same bytes as h5py);
  writing them in 'r+' is refused.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

feat: write complex numbers, incl. HDF5 2.0 native complex (class 11) ... d0d5347cd9

- Datatype::Complex serializes class 11 version 5 byte-identically to
  libhdf5 2.2.0; containers holding it are written as version 5.
- DatasetBuilder::with_complex_f32/f64_data (h5py's {r, i} compound,
  default) and with_native_complex_f32/f64_data (class 11, opt-in);
  make_(native_)complex_f32/f64_type for attributes.
- Dataset::read_complex_f64/f32 read either form.
- Python create_dataset accepts complex64/complex128 (compound form).
- Parsing unchanged: class 11 still surfaces as {r, i}.
- Tests vs h5py 3.16 / libhdf5 2.0.0 and h5dump 2.2.0; docs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: README — one datatypes row for complex writing and small floats ... 5e9160f82c

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

osobh added 1 commit 2026-09-29 02:38:39 +00:00

docs: known-issues — cite PRs #23 and #24 ... c033660d0d

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

osobh merged commit f9edf4d6ad into main 2026-09-29 03:11:22 +00:00

osobh deleted branch feat/write-complex 2026-09-29 03:11:26 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

nsobh osobh scottb (Scott B.)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#26