clawhdf5-netcdf4: variables' dimensions come from the file
Variables got the first unused dimension of equal size, so a variable on an unlimited dimension with fewer records got an anonymous dim_<n>, and dimensions of one size could be swapped. Resolve them as netCDF-C does (libhdf5/hdf5open.c): _Netcdf4Coordinates ids, else the scales DIMENSION_LIST references (the last one attached to an axis), searched in the variable's group and its parents; a coordinate variable is on its own scale. Size matching remains only for axes the file names nothing for. variables()/variable_names() leave out dimension scales that are only dimensions, and _nc4_non_coord_<name> is the variable <name>. Variable::shape is the netCDF shape (an unlimited dimension's length) and the reads pad unwritten records with the fill value (_FillValue, else NC_FILL_*; NaN from read_f64); Variable::stored_shape is the HDF5 extent. New NetCDF4File::variable_names. Tests compare with netCDF4-python variable by variable: the known-issues reproducer, equal sizes, (p, p), scalars, inherited dimensions, unwritten records, h5py dimension scales, h5netcdf and xarray files. CI installs h5netcdf. known-issues entry moved to Fixed (history); stale open-table row for the unlimited-size fix removed. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,59 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### NetCDF-4: variables' dimensions come from the file (2026-09-28)
|
||||
- `clawhdf5-netcdf4` gave each variable the first unused dimension of
|
||||
equal size (else an anonymous `dim_<n>`), so a variable on an unlimited
|
||||
dimension it had written fewer records of got `dim_<n>`, and dimensions
|
||||
of one size could be swapped. It now resolves them as netCDF-C does
|
||||
(`libhdf5/hdf5open.c`): the ids in the variable's `_Netcdf4Coordinates`
|
||||
(each scale's `_Netcdf4Dimid`), else the scales its `DIMENSION_LIST`
|
||||
references (object references, also the revised `H5T_STD_REF`; of
|
||||
several scales attached to one axis, the last, as netCDF-C's
|
||||
`dimscale_visitor` keeps), looked up in its group and then each parent
|
||||
group; a coordinate variable is on
|
||||
its own scale. Only an axis with neither (a file not written by a netCDF
|
||||
library) is still matched by size. A variable may use one dimension
|
||||
twice (`(p, p)`).
|
||||
- `variables()` and `variable_names()` leave out the dimension scales that
|
||||
are only dimensions (`NAME` "This is a netCDF dimension but not a netCDF
|
||||
variable."), as netCDF-C does, and `variable()` refuses them
|
||||
(`VariableNotFound`). A variable stored as `_nc4_non_coord_<name>`
|
||||
(netCDF-C's name for a variable sharing a dimension's name without being
|
||||
its coordinate variable) is listed and found as `<name>`.
|
||||
`NetCDF4File::variable_names` is new; `NetCDF4Group::variable_names` used
|
||||
to list every dataset.
|
||||
- A variable along an unlimited dimension has the dimension's length, as in
|
||||
netCDF: `Variable::shape` is that length and the reads return that many
|
||||
values, the records it has not written as the fill value (`_FillValue`,
|
||||
else netCDF's `NC_FILL_*` for the type; `""` for strings; NaN from
|
||||
`read_f64`). It was the HDF5 extent. `Variable::stored_shape` (new) is the
|
||||
HDF5 extent. Where the unlimited dimension is not a variable's first,
|
||||
unwritten values are placed per row, as netCDF-C's element and row reads
|
||||
return them; a whole-variable read through netCDF-C 4.9.3
|
||||
(netCDF4-python 1.7.4) instead returns the written values first, then the
|
||||
fill.
|
||||
- A variable's attributes are read when it is opened (they were read on
|
||||
first use).
|
||||
- Tests, compared with netCDF4-python 1.7.4 (netCDF-C 4.9.3) variable by
|
||||
variable (dimensions, shape, every value): the known-issues reproducer;
|
||||
two dimensions of one size in either order, one dimension used twice, a
|
||||
scalar, a non-coordinate variable named like a dimension, a subgroup and
|
||||
a sub-subgroup on their ancestors' dimensions; unwritten records of
|
||||
`i1`/`i4`/`u8`/`f4`/`f8`/string variables, with and without
|
||||
`_FillValue`; a file of h5py dimension scales (no netCDF attributes);
|
||||
h5netcdf 1.8.1 and xarray 2026.7.0 files (engines netcdf4 and h5netcdf).
|
||||
The h5netcdf cases skip when h5netcdf is not installed; CI now installs
|
||||
it. A one-off comparison over the 78 conformance-corpus files
|
||||
netCDF4-python opens (tank, 2026-09-28; a throwaway test, not committed)
|
||||
found 52 files with the same variables, dimensions and shapes, and the
|
||||
same values in every numeric variable of up to 5000 elements (enum
|
||||
variables were not compared). The other 26 have no dimension scales
|
||||
(netCDF-C names their axes `phony_dim_<n>`; this crate still matches by
|
||||
size or names them `dim_<size>`) or hold datasets of types netCDF-C
|
||||
skips (opaque, references). Affected v2.1.0 to v2.7.0.
|
||||
`docs/known-issues.md`.
|
||||
|
||||
### A dropped `FileEditor` releases its lock at once (2026-09-28)
|
||||
- `FileEditor`'s `flock` could outlive the editor for a moment when
|
||||
another thread forked to spawn a process: the child shared the locked
|
||||
|
||||
Reference in New Issue
Block a user