clawhdf5-netcdf4: variables' dimensions come from the file
CI / test-arm64 (pull_request) Successful in 1m33s
CI / test (pull_request) Successful in 18m24s

Variables got the first unused dimension of equal size, so a variable on
an unlimited dimension with fewer records got an anonymous dim_<n>, and
dimensions of one size could be swapped. Resolve them as netCDF-C does
(libhdf5/hdf5open.c): _Netcdf4Coordinates ids, else the scales
DIMENSION_LIST references (the last one attached to an axis), searched in
the variable's group and its parents; a coordinate variable is on its own
scale. Size matching remains only for axes the file names nothing for.

variables()/variable_names() leave out dimension scales that are only
dimensions, and _nc4_non_coord_<name> is the variable <name>.
Variable::shape is the netCDF shape (an unlimited dimension's length) and
the reads pad unwritten records with the fill value (_FillValue, else
NC_FILL_*; NaN from read_f64); Variable::stored_shape is the HDF5 extent.
New NetCDF4File::variable_names.

Tests compare with netCDF4-python variable by variable: the known-issues
reproducer, equal sizes, (p, p), scalars, inherited dimensions, unwritten
records, h5py dimension scales, h5netcdf and xarray files. CI installs
h5netcdf. known-issues entry moved to Fixed (history); stale open-table
row for the unlimited-size fix removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-28 22:52:04 -05:00
co-authored by Claude Opus 5.5
parent f9edf4d6ad
commit 00b6f76ee0
12 changed files with 1191 additions and 237 deletions
+53
View File
@@ -2,6 +2,59 @@
## Unreleased
### NetCDF-4: variables' dimensions come from the file (2026-09-28)
- `clawhdf5-netcdf4` gave each variable the first unused dimension of
equal size (else an anonymous `dim_<n>`), so a variable on an unlimited
dimension it had written fewer records of got `dim_<n>`, and dimensions
of one size could be swapped. It now resolves them as netCDF-C does
(`libhdf5/hdf5open.c`): the ids in the variable's `_Netcdf4Coordinates`
(each scale's `_Netcdf4Dimid`), else the scales its `DIMENSION_LIST`
references (object references, also the revised `H5T_STD_REF`; of
several scales attached to one axis, the last, as netCDF-C's
`dimscale_visitor` keeps), looked up in its group and then each parent
group; a coordinate variable is on
its own scale. Only an axis with neither (a file not written by a netCDF
library) is still matched by size. A variable may use one dimension
twice (`(p, p)`).
- `variables()` and `variable_names()` leave out the dimension scales that
are only dimensions (`NAME` "This is a netCDF dimension but not a netCDF
variable."), as netCDF-C does, and `variable()` refuses them
(`VariableNotFound`). A variable stored as `_nc4_non_coord_<name>`
(netCDF-C's name for a variable sharing a dimension's name without being
its coordinate variable) is listed and found as `<name>`.
`NetCDF4File::variable_names` is new; `NetCDF4Group::variable_names` used
to list every dataset.
- A variable along an unlimited dimension has the dimension's length, as in
netCDF: `Variable::shape` is that length and the reads return that many
values, the records it has not written as the fill value (`_FillValue`,
else netCDF's `NC_FILL_*` for the type; `""` for strings; NaN from
`read_f64`). It was the HDF5 extent. `Variable::stored_shape` (new) is the
HDF5 extent. Where the unlimited dimension is not a variable's first,
unwritten values are placed per row, as netCDF-C's element and row reads
return them; a whole-variable read through netCDF-C 4.9.3
(netCDF4-python 1.7.4) instead returns the written values first, then the
fill.
- A variable's attributes are read when it is opened (they were read on
first use).
- Tests, compared with netCDF4-python 1.7.4 (netCDF-C 4.9.3) variable by
variable (dimensions, shape, every value): the known-issues reproducer;
two dimensions of one size in either order, one dimension used twice, a
scalar, a non-coordinate variable named like a dimension, a subgroup and
a sub-subgroup on their ancestors' dimensions; unwritten records of
`i1`/`i4`/`u8`/`f4`/`f8`/string variables, with and without
`_FillValue`; a file of h5py dimension scales (no netCDF attributes);
h5netcdf 1.8.1 and xarray 2026.7.0 files (engines netcdf4 and h5netcdf).
The h5netcdf cases skip when h5netcdf is not installed; CI now installs
it. A one-off comparison over the 78 conformance-corpus files
netCDF4-python opens (tank, 2026-09-28; a throwaway test, not committed)
found 52 files with the same variables, dimensions and shapes, and the
same values in every numeric variable of up to 5000 elements (enum
variables were not compared). The other 26 have no dimension scales
(netCDF-C names their axes `phony_dim_<n>`; this crate still matches by
size or names them `dim_<size>`) or hold datasets of types netCDF-C
skips (opaque, references). Affected v2.1.0 to v2.7.0.
`docs/known-issues.md`.
### A dropped `FileEditor` releases its lock at once (2026-09-28)
- `FileEditor`'s `flock` could outlive the editor for a moment when
another thread forked to spawn a process: the child shared the locked