clawhdf5-netcdf4: variables' dimensions come from the file
Variables got the first unused dimension of equal size, so a variable on an unlimited dimension with fewer records got an anonymous dim_<n>, and dimensions of one size could be swapped. Resolve them as netCDF-C does (libhdf5/hdf5open.c): _Netcdf4Coordinates ids, else the scales DIMENSION_LIST references (the last one attached to an axis), searched in the variable's group and its parents; a coordinate variable is on its own scale. Size matching remains only for axes the file names nothing for. variables()/variable_names() leave out dimension scales that are only dimensions, and _nc4_non_coord_<name> is the variable <name>. Variable::shape is the netCDF shape (an unlimited dimension's length) and the reads pad unwritten records with the fill value (_FillValue, else NC_FILL_*; NaN from read_f64); Variable::stored_shape is the HDF5 extent. New NetCDF4File::variable_names. Tests compare with netCDF4-python variable by variable: the known-issues reproducer, equal sizes, (p, p), scalars, inherited dimensions, unwritten records, h5py dimension scales, h5netcdf and xarray files. CI installs h5netcdf. known-issues entry moved to Fixed (history); stale open-table row for the unlimited-size fix removed. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+53
-22
@@ -15,9 +15,6 @@ Checked against `main` at `9b5803f` on 2026-09-28.
|
||||
|
||||
| Issue | Kind | Since |
|
||||
|---|---|---|
|
||||
| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 |
|
||||
|
||||
| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 |
|
||||
| [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 |
|
||||
| [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 |
|
||||
| [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 |
|
||||
@@ -413,25 +410,6 @@ always expose it. A fix belongs in `conformance/ref_bugs.py` (more or more
|
||||
varied reads for this object) or in documenting the file as a known
|
||||
refusal; neither is done.
|
||||
|
||||
## NetCDF-4: variables' dimensions are guessed from sizes
|
||||
|
||||
**Status:** open (found 2026-09-28 while fixing unlimited dimension sizes).
|
||||
`clawhdf5-netcdf4` gives each variable the dimensions it finds by size
|
||||
(`match_dimensions_to_variable`: the first unused dimension of equal size,
|
||||
else an anonymous `dim_<n>`), not the ones its `DIMENSION_LIST` names, and
|
||||
`variables()` also lists the dimension scales that are only dimensions
|
||||
(netCDF-C hides them). With netCDF4-python: unlimited `time` and `empty`,
|
||||
`x` (3), `a(time)` with 2 records, `b(time, x)` with 5, `e(empty)` — netCDF4
|
||||
reports `time` = 5, `a` on `time`, and variables `a`, `b`, `e` and a `c(x)`;
|
||||
clawhdf5-netcdf4 reports `time` = 5 (right) but `a` on `dim_2`, and also
|
||||
variables `time` and `empty` (the scales, put on `empty`). Two dimensions of
|
||||
one size can be swapped the same way. Related: netCDF4 gives `a` the shape
|
||||
(5,) (a variable along an unlimited dimension has the dimension's length,
|
||||
unwritten records read as fill); `Variable::shape` is the HDF5 extent,
|
||||
`[2]`, and reads return those 2 values. `dimensions()` is right.
|
||||
Workaround: read the variable's `_Netcdf4Coordinates` attribute (dimension
|
||||
ids, matching each scale's `_Netcdf4Dimid`).
|
||||
|
||||
## Small floats decode as libhdf5 does, not as the OCP MX specification
|
||||
|
||||
**Status:** open, deliberate (documented 2026-09-28). HDF5 2.x predefines
|
||||
@@ -484,6 +462,59 @@ Newest first. "Before any release" means no tagged release (v2.7.0 and
|
||||
earlier) contains the bug. Full detail is in `CHANGELOG.md` under the date
|
||||
given.
|
||||
|
||||
## NetCDF-4: variables' dimensions are guessed from sizes
|
||||
|
||||
|
||||
**Status:** fixed 2026-09-28 (branch `fix/netcdf-dimension-list`). Affected
|
||||
every release (v2.1.0 to v2.7.0: size matching dates from the crate's
|
||||
first version). Wrong metadata only: stored values were always read right.
|
||||
Users who read `_Netcdf4Coordinates` themselves can use
|
||||
`Variable::dimensions` again; code that relied on `Variable::shape` being
|
||||
the HDF5 extent, or on the reads returning only the written records, should
|
||||
use `Variable::stored_shape` (new) — `shape` and the reads now follow
|
||||
netCDF (below). `variables()` no longer lists pure dimension scales.
|
||||
|
||||
Found 2026-09-28 while fixing unlimited dimension sizes.
|
||||
`clawhdf5-netcdf4` gave each variable the dimensions it found by size
|
||||
(`match_dimensions_to_variable`: the first unused dimension of equal size,
|
||||
else an anonymous `dim_<n>`), not the ones its `DIMENSION_LIST` names, and
|
||||
`variables()` also listed the dimension scales that are only dimensions
|
||||
(netCDF-C hides them). With netCDF4-python: unlimited `time` and `empty`,
|
||||
`x` (3), `a(time)` with 2 records, `b(time, x)` with 5, `e(empty)` — netCDF4
|
||||
reports `time` = 5, `a` on `time`, and variables `a`, `b`, `e` and a `c(x)`;
|
||||
clawhdf5-netcdf4 reported `time` = 5 (right) but `a` on `dim_2`, and also
|
||||
variables `time` and `empty` (the scales, put on `empty`). Two dimensions of
|
||||
one size could be swapped the same way. Related: netCDF4 gives `a` the shape
|
||||
(5,) (a variable along an unlimited dimension has the dimension's length,
|
||||
unwritten records read as fill); `Variable::shape` was the HDF5 extent,
|
||||
`[2]`, and reads returned those 2 values. `dimensions()` was right. The
|
||||
workaround was to read the variable's `_Netcdf4Coordinates` attribute
|
||||
(dimension ids, matching each scale's `_Netcdf4Dimid`).
|
||||
|
||||
Variables now get their dimensions as netCDF-C resolves them
|
||||
(`libhdf5/hdf5open.c`): `_Netcdf4Coordinates` ids, else the scales
|
||||
`DIMENSION_LIST` references, looked up in the variable's group and its
|
||||
parents; size matching remains only for axes with neither (files not
|
||||
written by a netCDF library). Pure dimension scales are not variables, and
|
||||
`_nc4_non_coord_<name>` datasets are the variables `<name>`. A variable
|
||||
along an unlimited dimension has the dimension's length, and its unwritten
|
||||
records read as the fill value. One difference from netCDF-C 4.9.3 is
|
||||
deliberate: when the unlimited dimension is not a variable's first
|
||||
(`f(x, t)` with 1 of 4 records), a whole-variable read through netCDF-C
|
||||
returns the written values first and then the fill
|
||||
(`[[1, 2, fill, fill], [fill, ...]]` for rows `[1]` and `[2]`), while its
|
||||
element and row reads — and clawhdf5-netcdf4 — place each row's values in
|
||||
their row (`[[1, fill, fill, fill], [2, fill, fill, fill]]`).
|
||||
`interop_tests` compares variables, dimensions, shapes and every value
|
||||
with netCDF4-python 1.7.4 for the reproducer, dimensions of equal size,
|
||||
one dimension used twice, a scalar, subgroups on their parents'
|
||||
dimensions, unwritten records with and without `_FillValue`, h5py
|
||||
dimension scales, and h5netcdf 1.8.1 and xarray files (tank, 2026-09-28,
|
||||
`CLAWHDF5_PYTHON=<venv with h5netcdf> cargo test -p clawhdf5-netcdf4`).
|
||||
Files without dimension scales still get dimensions by size (netCDF-C
|
||||
gives them `phony_dim_<n>`), as before; see `CHANGELOG.md` for a
|
||||
comparison over the conformance corpus's netCDF-readable files.
|
||||
|
||||
## A dropped `FileEditor` could keep its file locked for a moment
|
||||
|
||||
**Status:** fixed 2026-09-28 (#23), before any release
|
||||
|
||||
Reference in New Issue
Block a user