clawhdf5-netcdf4: phony dimensions, skipped types and order as netCDF-C #30

2026-09-30T02:47:25Z

osobh commented

2026-09-30 02:47:25 +00:00

Merge order: this is 1 of 3 (#30 → #31 → #32). The later PRs are stacked on this one.

clawhdf5-netcdf4 now presents a file the way netCDF-C does.

Reference: a new script, tests/netcdf_c_view.py, calls the libnetcdf that netCDF4-python bundles through ctypes. netCDF4-python's own objects hide opaque variables and some compounds that netCDF-C shows, so it can't be the reference itself.

Changes (checked against netCDF-C 4.9.3's hdf5open.c and real files):

Phony dimensions: phony_dim_<n> names, file-wide ids, subgroups numbered before their parent, and reuse of the group's first dimension with the same length and unlimited flag. They replace dim_<size>.
Order: groups, variables and dimensions come in netCDF-C's order: creation order when tracked, else by name, with datasets before subgroups.
Skipped types: reference, bit-field, time and array datasets are not variables, and neither are types built on them. netCDF-C's quirk of listing a second dataset of a failed compound is replayed.
NcType gains Enum, Compound, VLen and Opaque.

Corpus test tests/corpus_vs_netcdf_c.rs (gated on CLAWHDF5_NETCDF_CORPUS): of the 429 corpus files netCDF-C opens, 420 match (main: 68). The other 9 are listed and explained in tests/corpus_known_differences.txt: 5 use external links, and 4 have values our HDF5 reader refuses. The test fails on any unexplained difference, and also when a listed file starts matching.

API break: NcType gains variants and becomes #[non_exhaustive]. variables() and dimensions() now use netCDF-C's names and order. Nothing else in the workspace uses this crate, and neither does ClawBrainHub.

Deliberate differences, documented in known-issues:

Half and small floats stay numeric. netCDF-C labels them NC_STRING.
For axes netCDF-C leaves without a dimension (it reads uninitialised memory there), we use the phony rule.

Found, not fixed: clawhdf5 refuses bad_nbit_decompress.h5 ("nbit: element count exceeds chunk size"), while libhdf5 1.14.6 and 2.0 read it.

🤖 Generated with Claude Code

**Merge order: this is 1 of 3 (#30 → #31 → #32).** The later PRs are stacked on this one. `clawhdf5-netcdf4` now presents a file the way netCDF-C does. **Reference:** a new script, `tests/netcdf_c_view.py`, calls the libnetcdf that netCDF4-python bundles through ctypes. netCDF4-python's own objects hide opaque variables and some compounds that netCDF-C shows, so it can't be the reference itself. **Changes** (checked against netCDF-C 4.9.3's `hdf5open.c` and real files): - **Phony dimensions:** `phony_dim_<n>` names, file-wide ids, subgroups numbered before their parent, and reuse of the group's first dimension with the same length and unlimited flag. They replace `dim_<size>`. - **Order:** groups, variables and dimensions come in netCDF-C's order: creation order when tracked, else by name, with datasets before subgroups. - **Skipped types:** reference, bit-field, time and array datasets are not variables, and neither are types built on them. netCDF-C's quirk of listing a second dataset of a failed compound is replayed. - **`NcType`** gains `Enum`, `Compound`, `VLen` and `Opaque`. **Corpus test** `tests/corpus_vs_netcdf_c.rs` (gated on `CLAWHDF5_NETCDF_CORPUS`): of the 429 corpus files netCDF-C opens, **420 match** (main: 68). The other 9 are listed and explained in `tests/corpus_known_differences.txt`: 5 use external links, and 4 have values our HDF5 reader refuses. The test fails on any unexplained difference, and also when a listed file starts matching. **API break:** `NcType` gains variants and becomes `#[non_exhaustive]`. `variables()` and `dimensions()` now use netCDF-C's names and order. Nothing else in the workspace uses this crate, and neither does ClawBrainHub. **Deliberate differences, documented in known-issues:** - Half and small floats stay numeric. netCDF-C labels them `NC_STRING`. - For axes netCDF-C leaves without a dimension (it reads uninitialised memory there), we use the phony rule. **Found, not fixed:** clawhdf5 refuses `bad_nbit_decompress.h5` ("nbit: element count exceeds chunk size"), while libhdf5 1.14.6 and 2.0 read it. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

osobh added 1 commit 2026-09-30 02:47:25 +00:00

clawhdf5-netcdf4: phony dimensions, skipped types and order as netCDF-C

CI / test-arm64 (pull_request) Successful in 1m43s

Details

CI / test (pull_request) Successful in 20m28s

Details

e5d6f59e12

Read a file's metadata the way netCDF-C 4.9.3 does (libhdf5/hdf5open.c),
for the whole file on first use (src/model.rs, replacing src/scope.rs):

- links in creation order when the group tracks it, else name order;
  a group's datasets before its subgroups; dimension ids file-wide;
- variables' dimensions from _Netcdf4Coordinates (file-wide ids), else
  the scales DIMENSION_LIST attaches when the first axis has one, else
  netCDF-C's phony dimensions phony_dim_<id> (create_phony_dims: shared
  by length and unlimitedness within a group, not between two axes of
  one variable, numbered subgroups first, a zero length unlimited);
- datasets of types netCDF-C cannot represent are not variables
  (references, bit fields, time, arrays, compounds/enums/VLENs over
  them), replaying netCDF-C's file-wide type list, failed types
  included;
- unlimited lengths as nc4_find_dim_len (its group and below).

NcType gains Enum, Compound, VLen, Opaque and is #[non_exhaustive];
Variable::nc_type is netCDF-C's type (1-byte strings NC_CHAR). New
clawhdf5_format::group_v2::links_in_creation_order_in.

Tests compare with netCDF-C itself (tests/netcdf_c_view.py calls the
libnetcdf netCDF4-python bundles through ctypes): new interop cases for
h5py files without dimension scales, every type class, link order; and
the gated corpus_vs_netcdf_c (CLAWHDF5_NETCDF_CORPUS): 420 of the 429
conformance-corpus files netCDF-C opens match (main: 68); the other 9
are explained in tests/corpus_known_differences.txt and known-issues.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

osobh referenced this pull request

2026-09-30 02:47:26 +00:00

HDF5 2.0 native complex as a first-class type; Python libver= #31

All checks were successful

CI / test-arm64 (pull_request) Successful in 1m43s

Details

CI / test (pull_request) Successful in 20m28s

Details

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/netcdf-phony-dims:fix/netcdf-phony-dims

git checkout fix/netcdf-phony-dims

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#30