quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases 1 Wiki Activity

HDF5 2.0 native complex as a first-class type; Python libver= #31

Open
osobh wants to merge 2 commits from feat/complex-first-class into main
pull from: feat/complex-first-class
merge into: :main
Conversation 0 Commits 2 Files Changed 37
+3251 -605
osobh commented 2026-09-30 02:47:26 +00:00
Owner
Copy Link
Copy Source

Merge order: 2 of 3. This PR is stacked on #30.

HDF5 2.0's native complex type (class 11) is now a type of its own when read. Before, it showed up as its {r, i} compound view.

  • Format crate: Datatype::parse returns Datatype::Complex, including inside compounds, arrays and variable-length types.
  • Facade: DType::Complex(Box<DType>), where Complex(F32) is complex64 and Complex(F64) is complex128.
  • h5rs:
    • dump prints h5dump 2.2.0's type names (H5T_COMPLEX_IEEE_F64LE, …) and values like 1.5-2i. It is compared against h5dump 2.2's stored output.
    • ls shows complex128, and ls -v shows h5ls 2.2's text.
    • diff compares both parts, as h5diff 2.2 does.
    • dump --json keeps {r, i}, because h5json 2.0.0 has no complex type (checked in its source).
  • wasm: native complex datasets now read as [re, im] pairs (they used to be refused), and the viewer renders them. Checked under Node and in headless Chromium.
  • Python:
    • Native complex still reads as complex64/complex128.
    • New File(path, 'w', libver=...) takes h5py's names and (low, high) tuples.
    • With 'earliest' as the low bound it writes the 1.8 format with a UserWarning.
    • libver='v108' output opens in HDF5 1.8.23's h5dump and in h5py.

API break: raw_datatype() and Datatype::parse return Datatype::Complex for class 11, and DType gains a variant. h5rs output for class 11 changes.

Not done: h5py's {r, i} compound complex is still refused by the wasm reader, as every compound is. Native complex with binary16 parts goes to Python as a structured float16 dtype, because numpy has no complex32.

🤖 Generated with Claude Code

**Merge order: 2 of 3.** This PR is stacked on #30. HDF5 2.0's native complex type (class 11) is now a type of its own when read. Before, it showed up as its `{r, i}` compound view. - **Format crate:** `Datatype::parse` returns `Datatype::Complex`, including inside compounds, arrays and variable-length types. - **Facade:** `DType::Complex(Box<DType>)`, where `Complex(F32)` is complex64 and `Complex(F64)` is complex128. - **`h5rs`:** - `dump` prints h5dump 2.2.0's type names (`H5T_COMPLEX_IEEE_F64LE`, …) and values like `1.5-2i`. It is compared against h5dump 2.2's stored output. - `ls` shows `complex128`, and `ls -v` shows h5ls 2.2's text. - `diff` compares both parts, as h5diff 2.2 does. - `dump --json` keeps `{r, i}`, because h5json 2.0.0 has no complex type (checked in its source). - **wasm:** native complex datasets now read as `[re, im]` pairs (they used to be refused), and the viewer renders them. Checked under Node and in headless Chromium. - **Python:** - Native complex still reads as complex64/complex128. - New `File(path, 'w', libver=...)` takes h5py's names and `(low, high)` tuples. - With `'earliest'` as the low bound it writes the 1.8 format with a `UserWarning`. - `libver='v108'` output opens in HDF5 1.8.23's h5dump and in h5py. **API break:** `raw_datatype()` and `Datatype::parse` return `Datatype::Complex` for class 11, and `DType` gains a variant. `h5rs` output for class 11 changes. **Not done:** h5py's `{r, i}` compound complex is still refused by the wasm reader, as every compound is. Native complex with binary16 parts goes to Python as a structured float16 dtype, because numpy has no complex32. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
osobh added 2 commits 2026-09-30 02:47:27 +00:00
clawhdf5-netcdf4: phony dimensions, skipped types and order as netCDF-C
CI / test-arm64 (pull_request) Successful in 1m43s
Details
CI / test (pull_request) Successful in 20m28s
Details
e5d6f59e12 
Read a file's metadata the way netCDF-C 4.9.3 does (libhdf5/hdf5open.c),
for the whole file on first use (src/model.rs, replacing src/scope.rs):

- links in creation order when the group tracks it, else name order;
  a group's datasets before its subgroups; dimension ids file-wide;
- variables' dimensions from _Netcdf4Coordinates (file-wide ids), else
  the scales DIMENSION_LIST attaches when the first axis has one, else
  netCDF-C's phony dimensions phony_dim_<id> (create_phony_dims: shared
  by length and unlimitedness within a group, not between two axes of
  one variable, numbered subgroups first, a zero length unlimited);
- datasets of types netCDF-C cannot represent are not variables
  (references, bit fields, time, arrays, compounds/enums/VLENs over
  them), replaying netCDF-C's file-wide type list, failed types
  included;
- unlimited lengths as nc4_find_dim_len (its group and below).

NcType gains Enum, Compound, VLen, Opaque and is #[non_exhaustive];
Variable::nc_type is netCDF-C's type (1-byte strings NC_CHAR). New
clawhdf5_format::group_v2::links_in_creation_order_in.

Tests compare with netCDF-C itself (tests/netcdf_c_view.py calls the
libnetcdf netCDF4-python bundles through ctypes): new interop cases for
h5py files without dimension scales, every type class, link order; and
the gated corpus_vs_netcdf_c (CLAWHDF5_NETCDF_CORPUS): 420 of the 429
conformance-corpus files netCDF-C opens match (main: 68); the other 9
are explained in tests/corpus_known_differences.txt and known-issues.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
HDF5 2.0 native complex as a first-class type on read; Python libver=
CI / test-arm64 (pull_request) Successful in 1m38s
Details
CI / test (pull_request) Successful in 21m11s
Details
e4ba09946f 
- Datatype::parse returns Datatype::Complex for class 11 (also inside
  compounds, arrays and VL types) instead of the {r, i} compound view.
- Facade: DType::Complex(Box<DType>); read_complex_f32/f64 accept it.
- h5rs dump/ls/diff print native complex as h5dump/h5ls/h5diff 2.2.0 do
  (checked against a fixture written by h5py 3.16 / libhdf5 2.0.0);
  dump --json keeps the {r, i} compound (hdf5-json has no complex class).
- clawhdf5-wasm reads native complex datasets as [re, im] pairs.
- Python: clawhdf5.File(path, 'w', libver=...) with h5py's values,
  mapped to FileBuilder::libver_bounds; 'v108' output opens in HDF5 1.8.23.
- Docs: known-issues entry moved to Fixed (history), CHANGELOG, READMEs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
All checks were successful
CI / test-arm64 (pull_request) Successful in 1m38s
Details
CI / test (pull_request) Successful in 21m11s
Details
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/complex-first-class:feat/complex-first-class
git checkout feat/complex-first-class
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
nsobh osobh scottb (Scott B.)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#31