quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases 1 Wiki Activity

clawhdf5-netcdf4: variables' dimensions come from the file
CI / test-arm64 (pull_request) Successful in 1m33s
Details
CI / test (pull_request) Successful in 18m24s
Details

Browse Source 
Variables got the first unused dimension of equal size, so a variable on
an unlimited dimension with fewer records got an anonymous dim_<n>, and
dimensions of one size could be swapped. Resolve them as netCDF-C does
(libhdf5/hdf5open.c): _Netcdf4Coordinates ids, else the scales
DIMENSION_LIST references (the last one attached to an axis), searched in
the variable's group and its parents; a coordinate variable is on its own
scale. Size matching remains only for axes the file names nothing for.

variables()/variable_names() leave out dimension scales that are only
dimensions, and _nc4_non_coord_<name> is the variable <name>.
Variable::shape is the netCDF shape (an unlimited dimension's length) and
the reads pad unwritten records with the fill value (_FillValue, else
NC_FILL_*; NaN from read_f64); Variable::stored_shape is the HDF5 extent.
New NetCDF4File::variable_names.

Tests compare with netCDF4-python variable by variable: the known-issues
reproducer, equal sizes, (p, p), scalars, inherited dimensions, unwritten
records, h5py dimension scales, h5netcdf and xarray files. CI installs
h5netcdf. known-issues entry moved to Fixed (history); stale open-table
row for the unlimited-size fix removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-28 22:52:04 -05:00
co-authored by Claude Opus 5.5
parent f9edf4d6ad
commit 00b6f76ee0
12 changed files with 1191 additions and 237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/known-issues.md
+53 -22
View File
@@ -15,9 +15,6 @@ Checked against `main` at `9b5803f` on 2026-09-28.
| Issue | Kind | Since |
|---|---|---|
| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 |
| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 |
| [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 |
| [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 |
| [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 |
@@ -413,25 +410,6 @@ always expose it. A fix belongs in `conformance/ref_bugs.py` (more or more
varied reads for this object) or in documenting the file as a known
refusal; neither is done.
## NetCDF-4: variables' dimensions are guessed from sizes
**Status:** open (found 2026-09-28 while fixing unlimited dimension sizes).
`clawhdf5-netcdf4` gives each variable the dimensions it finds by size
(`match_dimensions_to_variable`: the first unused dimension of equal size,
else an anonymous `dim_<n>`), not the ones its `DIMENSION_LIST` names, and
`variables()` also lists the dimension scales that are only dimensions
(netCDF-C hides them). With netCDF4-python: unlimited `time` and `empty`,
`x` (3), `a(time)` with 2 records, `b(time, x)` with 5, `e(empty)` — netCDF4
reports `time` = 5, `a` on `time`, and variables `a`, `b`, `e` and a `c(x)`;
clawhdf5-netcdf4 reports `time` = 5 (right) but `a` on `dim_2`, and also
variables `time` and `empty` (the scales, put on `empty`). Two dimensions of
one size can be swapped the same way. Related: netCDF4 gives `a` the shape
(5,) (a variable along an unlimited dimension has the dimension's length,
unwritten records read as fill); `Variable::shape` is the HDF5 extent,
`[2]`, and reads return those 2 values. `dimensions()` is right.
Workaround: read the variable's `_Netcdf4Coordinates` attribute (dimension
ids, matching each scale's `_Netcdf4Dimid`).
## Small floats decode as libhdf5 does, not as the OCP MX specification
**Status:** open, deliberate (documented 2026-09-28). HDF5 2.x predefines
@@ -484,6 +462,59 @@ Newest first. "Before any release" means no tagged release (v2.7.0 and
earlier) contains the bug. Full detail is in `CHANGELOG.md` under the date
given.
## NetCDF-4: variables' dimensions are guessed from sizes
**Status:** fixed 2026-09-28 (branch `fix/netcdf-dimension-list`). Affected
every release (v2.1.0 to v2.7.0: size matching dates from the crate's
first version). Wrong metadata only: stored values were always read right.
Users who read `_Netcdf4Coordinates` themselves can use
`Variable::dimensions` again; code that relied on `Variable::shape` being
the HDF5 extent, or on the reads returning only the written records, should
use `Variable::stored_shape` (new) — `shape` and the reads now follow
netCDF (below). `variables()` no longer lists pure dimension scales.
Found 2026-09-28 while fixing unlimited dimension sizes.
`clawhdf5-netcdf4` gave each variable the dimensions it found by size
(`match_dimensions_to_variable`: the first unused dimension of equal size,
else an anonymous `dim_<n>`), not the ones its `DIMENSION_LIST` names, and
`variables()` also listed the dimension scales that are only dimensions
(netCDF-C hides them). With netCDF4-python: unlimited `time` and `empty`,
`x` (3), `a(time)` with 2 records, `b(time, x)` with 5, `e(empty)` — netCDF4
reports `time` = 5, `a` on `time`, and variables `a`, `b`, `e` and a `c(x)`;
clawhdf5-netcdf4 reported `time` = 5 (right) but `a` on `dim_2`, and also
variables `time` and `empty` (the scales, put on `empty`). Two dimensions of
one size could be swapped the same way. Related: netCDF4 gives `a` the shape
(5,) (a variable along an unlimited dimension has the dimension's length,
unwritten records read as fill); `Variable::shape` was the HDF5 extent,
`[2]`, and reads returned those 2 values. `dimensions()` was right. The
workaround was to read the variable's `_Netcdf4Coordinates` attribute
(dimension ids, matching each scale's `_Netcdf4Dimid`).
Variables now get their dimensions as netCDF-C resolves them
(`libhdf5/hdf5open.c`): `_Netcdf4Coordinates` ids, else the scales
`DIMENSION_LIST` references, looked up in the variable's group and its
parents; size matching remains only for axes with neither (files not
written by a netCDF library). Pure dimension scales are not variables, and
`_nc4_non_coord_<name>` datasets are the variables `<name>`. A variable
along an unlimited dimension has the dimension's length, and its unwritten
records read as the fill value. One difference from netCDF-C 4.9.3 is
deliberate: when the unlimited dimension is not a variable's first
(`f(x, t)` with 1 of 4 records), a whole-variable read through netCDF-C
returns the written values first and then the fill
(`[[1, 2, fill, fill], [fill, ...]]` for rows `[1]` and `[2]`), while its
element and row reads — and clawhdf5-netcdf4 — place each row's values in
their row (`[[1, fill, fill, fill], [2, fill, fill, fill]]`).
`interop_tests` compares variables, dimensions, shapes and every value
with netCDF4-python 1.7.4 for the reproducer, dimensions of equal size,
one dimension used twice, a scalar, subgroups on their parents'
dimensions, unwritten records with and without `_FillValue`, h5py
dimension scales, and h5netcdf 1.8.1 and xarray files (tank, 2026-09-28,
`CLAWHDF5_PYTHON=<venv with h5netcdf> cargo test -p clawhdf5-netcdf4`).
Files without dimension scales still get dimensions by size (netCDF-C
gives them `phony_dim_<n>`), as before; see `CHANGELOG.md` for a
comparison over the conformance corpus's netCDF-readable files.
## A dropped `FileEditor` could keep its file locked for a moment
**Status:** fixed 2026-09-28 (#23), before any release