clawhdf5-netcdf4: phony dimensions, skipped types and order as netCDF-C

Read a file's metadata the way netCDF-C 4.9.3 does (libhdf5/hdf5open.c),
for the whole file on first use (src/model.rs, replacing src/scope.rs):

- links in creation order when the group tracks it, else name order;
  a group's datasets before its subgroups; dimension ids file-wide;
- variables' dimensions from _Netcdf4Coordinates (file-wide ids), else
  the scales DIMENSION_LIST attaches when the first axis has one, else
  netCDF-C's phony dimensions phony_dim_<id> (create_phony_dims: shared
  by length and unlimitedness within a group, not between two axes of
  one variable, numbered subgroups first, a zero length unlimited);
- datasets of types netCDF-C cannot represent are not variables
  (references, bit fields, time, arrays, compounds/enums/VLENs over
  them), replaying netCDF-C's file-wide type list, failed types
  included;
- unlimited lengths as nc4_find_dim_len (its group and below).

NcType gains Enum, Compound, VLen, Opaque and is #[non_exhaustive];
Variable::nc_type is netCDF-C's type (1-byte strings NC_CHAR). New
clawhdf5_format::group_v2::links_in_creation_order_in.

Tests compare with netCDF-C itself (tests/netcdf_c_view.py calls the
libnetcdf netCDF4-python bundles through ctypes): new interop cases for
h5py files without dimension scales, every type class, link order; and
the gated corpus_vs_netcdf_c (CLAWHDF5_NETCDF_CORPUS): 420 of the 429
conformance-corpus files netCDF-C opens match (main: 68); the other 9
are explained in tests/corpus_known_differences.txt and known-issues.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs/known-issues.md

@@ -15,10 +15,8 @@ Checked against `main` at `9b5803f` on 2026-09-28.
 
 | Issue | Kind | Since |
 |---|---|---|
-
-| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 |
+| [NetCDF-4: differences from netCDF-C](#netcdf-4-differences-from-netcdf-c) | deliberate (floats of other than 4 or 8 bytes, axes netCDF-C leaves without a dimension), and 9 corpus files (external links, values the HDF5 reader refuses) | 2026-09-29 |
 | [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (chunk dimensions of 2^32 or more, five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 |
-| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 |
 | [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 |
 | [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 |
 | [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 |
@@ -33,6 +31,83 @@ Checked against `main` at `9b5803f` on 2026-09-28.
 
 ---
 
+## NetCDF-4: differences from netCDF-C
+
+**Status:** open (documented 2026-09-29): deliberate differences, and the
+corpus files that still read differently. `clawhdf5-netcdf4` reports a
+file's groups, dimensions and variables — names, order, types, shapes — as
+netCDF-C 4.9.3 does (`libhdf5/hdf5open.c`; see the crate README), and
+`crates/clawhdf5-netcdf4/tests/corpus_vs_netcdf_c.rs` compares it with
+netCDF-C itself (the libnetcdf netCDF4-python bundles, called through
+ctypes by `tests/netcdf_c_view.py`) over a corpus. Over the conformance
+corpus (tank, 2026-09-29, netCDF4-python 1.7.4: netCDF-C 4.9.3 on libhdf5
+1.14.6; `CLAWHDF5_NETCDF_CORPUS=<main checkout>/conformance/.cache/corpus
+CLAWHDF5_PYTHON=$PWD/.venv/bin/python cargo test -p clawhdf5-netcdf4 --test
+corpus_vs_netcdf_c -- --nocapture`): of 692 files, netCDF-C opens 429;
+420 match in groups, dimensions (names, lengths, unlimited, order),
+variables (names, order, types, dimensions, shapes) and the values of
+every numeric variable of up to 5000 elements; the other 9 are listed
+below and in `tests/corpus_known_differences.txt`, which the test holds to.
+
+Deliberate:
+
+- **Floats of other than 4 or 8 bytes** (half, bfloat16, the 4-, 6- and
+  8-bit floats, `long double`) are `NcType::Float` (up to 4 bytes) or
+  `Double`, and read as numbers. netCDF-C 4.9.3 on libhdf5 1.14.6 labels
+  them `NC_STRING`: their native type (libhdf5 1.14.6 has a native
+  `_Float16`) matches none of its own. 27 corpus variables; the comparison
+  shows them as netCDF-C does and counts them.
+- **An axis netCDF-C leaves without a dimension** gets one by the phony
+  rule (the group's first dimension of its length, else a new
+  `phony_dim_<n>`): a `_Netcdf4Coordinates` id or a `DIMENSION_LIST` scale
+  it cannot find, and an axis without a scale of a variable whose first
+  axis has one — there netCDF-C 4.9.3 reads uninitialised memory
+  (`get_attached_info` mallocs the object ids and `dimscale_visitor` never
+  fills that axis's): in one run it gave a 2-long axis the group's first
+  phony dimension, 3 long; in another netCDF4-python could not open the
+  file. No corpus file has such an axis.
+- **Types netCDF-C remembers after failing to read them.** netCDF-C adds a
+  type to its list before reading its members, and keeps it when that
+  fails, so the second dataset of such a type is a variable. For a
+  compound, enum or variable-length type it has a class and is shown here
+  too (`interop_tests::types_netcdf_c_skips_are_not_variables`); a bit
+  field, time, array or complex type has none, and netCDF-C lists the
+  second dataset with an invalid type (class 0): it stays hidden here.
+- **Files netCDF-C refuses** are read as well as they can be: a
+  multi-dimensional dimension scale without `_Netcdf4Coordinates`, a
+  `_Netcdf4Coordinates` of the wrong length, a named datatype netCDF-C
+  cannot represent, an unreadable dataset or attribute (netCDF-C fails
+  `nc_open`; this crate skips it).
+- **Cost:** the first call that needs the metadata reads that of the whole
+  file (every group, and every dataset's attributes), as `nc_open` does;
+  it was one group's per call. Not measured; worth measuring on a file
+  with many groups and variables before a release.
+- A whole-variable read of a variable shorter than an unlimited dimension
+  that is not its first: see [the entry of #27](#netcdf-4-variables-dimensions-are-guessed-from-sizes).
+
+Corpus files that differ (2026-09-29):
+
+- **External links** (`hdf5/test/testfiles/be_extlink1.h5`,
+  `le_extlink1.h5`, `hdf5/tools/test/testfiles/h5diff_ext2softlink_src.h5`,
+  `h5diff_grp_recurse_ext2-1.h5`, `h5diff_grp_recurse_ext2-2.h5`): libhdf5
+  follows them into the other file; clawhdf5 does not
+  ([External links](#external-links-and-external-raw-data-are-not-followed)),
+  so the linked objects are missing and the phony dimension numbers after
+  them shift.
+- **Values the HDF5 reader refuses** and libhdf5 1.14.6 returns (metadata
+  matches): `cve_hdf5/cvefiles/cve-2025-2308.h5`, `cve-2025-44904.h5` and
+  `hdf5/test/testfiles/bad_nbit_parms_walk.h5`, the scale-offset and N-Bit
+  files of CONFORMANCE.md's ref-bug/our-error list (libhdf5 reads past the
+  stored data); and `hdf5/test/testfiles/bad_nbit_decompress.h5`, whose
+  `/Nbit_float_data_le` clawhdf5 refuses ("nbit: element count exceeds
+  chunk size", also `h5rs dump`) while libhdf5 1.14.6 and h5py 3.16
+  (libhdf5 2.0.0) return values. That file is not in the conformance
+  report and has not been investigated.
+- Not compared: the values of 18 variables whose filter (SZIP, Blosc,
+  bzip2) this crate's default build of the HDF5 reader lacks. Blosc and
+  bzip2 are pure-Rust cargo features of `clawhdf5` (`blosc`, `bzip2`) that
+  a dependent can turn on; SZIP needs libaec.
+
 ## In-place modification (`FileEditor`) limits
 
 **Status:** open (documented 2026-09-26, updated when version-2 B-tree
@@ -539,6 +614,61 @@ Newest first. "Before any release" means no tagged release (v2.7.0 and
 earlier) contains the bug. Full detail is in `CHANGELOG.md` under the date
 given.
 
+## NetCDF-4: files without dimension scales, types and order differed from netCDF-C
+
+**Status:** fixed 2026-09-29 (branch `fix/netcdf-phony-dims`). Affected
+every release (v2.1.0 to v2.7.0) and `main` after #27. Wrong metadata only
+(names, order and types of dimensions and variables); values were read
+right. Users: the dimensions of a file without dimension scales are now
+netCDF-C's `phony_dim_<n>`, so code that matched `dim_<size>` or took a
+1-D dataset's name for a dimension must use the new names; datasets of
+types netCDF-C skips (references, bit fields, array types, compounds of
+those, ...) are no longer variables; lists come in netCDF-C's order.
+`NcType` has four new variants (`Enum`, `Compound`, `VLen`, `Opaque`) and
+is `#[non_exhaustive]`: an exhaustive `match` needs a wildcard arm.
+
+Found 2026-09-28 by the one-off corpus comparison of #27 (52 of the 78
+files netCDF4-python opened matched; netCDF4-python itself hides variables
+of opaque and other types it does not support, so the comparison now asks
+netCDF-C through its C API). Under that comparison `main` at `4260af4`
+matched 68 of the 429 corpus files netCDF-C opens (tank, 2026-09-29, the
+command above run on a copy of `4260af4` with the new test files). What
+differed:
+
+- **Files without dimension scales.** netCDF-C gives every axis a phony
+  dimension `phony_dim_<id>` (`create_phony_dims`), ids file-wide after the
+  real dimensions, subgroups numbered before their parent's variables,
+  shared by length and unlimitedness within a group but not between two
+  axes of one variable, a length of 0 always unlimited. The crate reported
+  one dimension per 1-D dataset, named after it (an h5py file with `x(5)`
+  and `v(5)` had dimensions `x` and `v`, and `v` on `x`), and other axes as
+  `dim_<size>`.
+- **Datasets netCDF-C skips** (`NC_EBADTYPID`: references, bit fields,
+  time and array types, a compound with such a member or a half-float
+  member, an enum or VLEN over one) were variables, with `NcType::Char`;
+  so were enums, compounds, VLENs and opaque types, and 1-byte strings
+  were `String` where netCDF-C says `NC_CHAR` (longer ones `NC_STRING`).
+- **Order.** Groups and variables came in the order of the HDF5 links as
+  stored (hash order in a dense group); netCDF-C uses creation order when
+  the group tracks it, else name order. Dimensions without
+  `_Netcdf4Dimid` came after the others instead of taking the next id in
+  reading order; a zero-length dimension scale was not unlimited.
+- `_Netcdf4Coordinates` ids were looked up only in the variable's group
+  and its parents (netCDF-C: file-wide), and an unlimited dimension's
+  length came from its scale's `REFERENCE_LIST` in any group
+  (`nc4_find_dim_len`: the variables in its group and below).
+
+The crate now reads the metadata of the whole file in netCDF-C's two
+passes (`crates/clawhdf5-netcdf4/src/model.rs`), with a new
+`clawhdf5_format::group_v2::links_in_creation_order_in` for the order.
+Tests: `interop_tests::phony_dimensions_match_netcdf_c`,
+`types_netcdf_c_skips_are_not_variables`,
+`group_and_variable_order_match_netcdf_c` and
+`netcdf4_python_files_match_netcdf_c` compare h5py- and
+netCDF4-python-written files with netCDF-C itself, and
+`corpus_vs_netcdf_c` the corpus (420 of 429 match; the rest are in
+[NetCDF-4: differences from netCDF-C](#netcdf-4-differences-from-netcdf-c)).
+
 ## NetCDF-4: variables' dimensions are guessed from sizes
 
 
@@ -590,7 +720,8 @@ dimension scales, and h5netcdf 1.8.1 and xarray files (tank, 2026-09-28,
 `CLAWHDF5_PYTHON=<venv with h5netcdf> cargo test -p clawhdf5-netcdf4`).
 Files without dimension scales still get dimensions by size (netCDF-C
 gives them `phony_dim_<n>`), as before; see `CHANGELOG.md` for a
-comparison over the conformance corpus's netCDF-readable files.
+comparison over the conformance corpus's netCDF-readable files. (Fixed
+2026-09-29: see the entry above.)
 
 ## HDF5 1.8 could not read the files we wrote