clawhdf5-netcdf4: phony dimensions, skipped types and order as netCDF-C
Read a file's metadata the way netCDF-C 4.9.3 does (libhdf5/hdf5open.c), for the whole file on first use (src/model.rs, replacing src/scope.rs): - links in creation order when the group tracks it, else name order; a group's datasets before its subgroups; dimension ids file-wide; - variables' dimensions from _Netcdf4Coordinates (file-wide ids), else the scales DIMENSION_LIST attaches when the first axis has one, else netCDF-C's phony dimensions phony_dim_<id> (create_phony_dims: shared by length and unlimitedness within a group, not between two axes of one variable, numbered subgroups first, a zero length unlimited); - datasets of types netCDF-C cannot represent are not variables (references, bit fields, time, arrays, compounds/enums/VLENs over them), replaying netCDF-C's file-wide type list, failed types included; - unlimited lengths as nc4_find_dim_len (its group and below). NcType gains Enum, Compound, VLen, Opaque and is #[non_exhaustive]; Variable::nc_type is netCDF-C's type (1-byte strings NC_CHAR). New clawhdf5_format::group_v2::links_in_creation_order_in. Tests compare with netCDF-C itself (tests/netcdf_c_view.py calls the libnetcdf netCDF4-python bundles through ctypes): new interop cases for h5py files without dimension scales, every type class, link order; and the gated corpus_vs_netcdf_c (CLAWHDF5_NETCDF_CORPUS): 420 of the 429 conformance-corpus files netCDF-C opens match (main: 68); the other 9 are explained in tests/corpus_known_differences.txt and known-issues. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
+135
-4
@@ -15,10 +15,8 @@ Checked against `main` at `9b5803f` on 2026-09-28.
|
||||
|
||||
| Issue | Kind | Since |
|
||||
|---|---|---|
|
||||
|
||||
| [NetCDF-4: variables' dimensions are guessed from sizes](#netcdf-4-variables-dimensions-are-guessed-from-sizes) | **wrong metadata** (`Variable::dimensions`, pure dimension scales listed as variables; values and dimension sizes are right) | 2026-09-28 |
|
||||
| [NetCDF-4: differences from netCDF-C](#netcdf-4-differences-from-netcdf-c) | deliberate (floats of other than 4 or 8 bytes, axes netCDF-C leaves without a dimension), and 9 corpus files (external links, values the HDF5 reader refuses) | 2026-09-29 |
|
||||
| [Chunks of 4 GiB or more: limits](#chunks-of-4-gib-or-more-limits) | refused (chunk dimensions of 2^32 or more, five filters, editor rewrites), memory (a decoded chunk is held whole) | 2026-09-28 |
|
||||
| [NetCDF-4: an unlimited dimension reports size 0](#netcdf-4-an-unlimited-dimension-reports-size-0) | **wrong metadata** (dimension size; variable shapes and values are right) | 2026-09-28 |
|
||||
| [Small floats decode as libhdf5 does, not as the OCP MX specification](#small-floats-decode-as-libhdf5-does-not-as-the-ocp-mx-specification) | deliberate: libhdf5's values (FP4/FP6/FP8 E4M3 all-ones exponent is inf/NaN) | 2026-09-28 |
|
||||
| [In-place modification (`FileEditor`) limits](#in-place-modification-fileeditor-limits) | refused edits (`Error::Unsupported`), space reuse per editor, no journal | 2026-09-26 |
|
||||
| [Python in-place editing limits](#python-in-place-editing-clawhdf5filepath-r-limits) | refused writes (`NotImplementedError`), deliberate conversion differences | 2026-09-27 |
|
||||
@@ -33,6 +31,83 @@ Checked against `main` at `9b5803f` on 2026-09-28.
|
||||
|
||||
---
|
||||
|
||||
## NetCDF-4: differences from netCDF-C
|
||||
|
||||
**Status:** open (documented 2026-09-29): deliberate differences, and the
|
||||
corpus files that still read differently. `clawhdf5-netcdf4` reports a
|
||||
file's groups, dimensions and variables — names, order, types, shapes — as
|
||||
netCDF-C 4.9.3 does (`libhdf5/hdf5open.c`; see the crate README), and
|
||||
`crates/clawhdf5-netcdf4/tests/corpus_vs_netcdf_c.rs` compares it with
|
||||
netCDF-C itself (the libnetcdf netCDF4-python bundles, called through
|
||||
ctypes by `tests/netcdf_c_view.py`) over a corpus. Over the conformance
|
||||
corpus (tank, 2026-09-29, netCDF4-python 1.7.4: netCDF-C 4.9.3 on libhdf5
|
||||
1.14.6; `CLAWHDF5_NETCDF_CORPUS=<main checkout>/conformance/.cache/corpus
|
||||
CLAWHDF5_PYTHON=$PWD/.venv/bin/python cargo test -p clawhdf5-netcdf4 --test
|
||||
corpus_vs_netcdf_c -- --nocapture`): of 692 files, netCDF-C opens 429;
|
||||
420 match in groups, dimensions (names, lengths, unlimited, order),
|
||||
variables (names, order, types, dimensions, shapes) and the values of
|
||||
every numeric variable of up to 5000 elements; the other 9 are listed
|
||||
below and in `tests/corpus_known_differences.txt`, which the test holds to.
|
||||
|
||||
Deliberate:
|
||||
|
||||
- **Floats of other than 4 or 8 bytes** (half, bfloat16, the 4-, 6- and
|
||||
8-bit floats, `long double`) are `NcType::Float` (up to 4 bytes) or
|
||||
`Double`, and read as numbers. netCDF-C 4.9.3 on libhdf5 1.14.6 labels
|
||||
them `NC_STRING`: their native type (libhdf5 1.14.6 has a native
|
||||
`_Float16`) matches none of its own. 27 corpus variables; the comparison
|
||||
shows them as netCDF-C does and counts them.
|
||||
- **An axis netCDF-C leaves without a dimension** gets one by the phony
|
||||
rule (the group's first dimension of its length, else a new
|
||||
`phony_dim_<n>`): a `_Netcdf4Coordinates` id or a `DIMENSION_LIST` scale
|
||||
it cannot find, and an axis without a scale of a variable whose first
|
||||
axis has one — there netCDF-C 4.9.3 reads uninitialised memory
|
||||
(`get_attached_info` mallocs the object ids and `dimscale_visitor` never
|
||||
fills that axis's): in one run it gave a 2-long axis the group's first
|
||||
phony dimension, 3 long; in another netCDF4-python could not open the
|
||||
file. No corpus file has such an axis.
|
||||
- **Types netCDF-C remembers after failing to read them.** netCDF-C adds a
|
||||
type to its list before reading its members, and keeps it when that
|
||||
fails, so the second dataset of such a type is a variable. For a
|
||||
compound, enum or variable-length type it has a class and is shown here
|
||||
too (`interop_tests::types_netcdf_c_skips_are_not_variables`); a bit
|
||||
field, time, array or complex type has none, and netCDF-C lists the
|
||||
second dataset with an invalid type (class 0): it stays hidden here.
|
||||
- **Files netCDF-C refuses** are read as well as they can be: a
|
||||
multi-dimensional dimension scale without `_Netcdf4Coordinates`, a
|
||||
`_Netcdf4Coordinates` of the wrong length, a named datatype netCDF-C
|
||||
cannot represent, an unreadable dataset or attribute (netCDF-C fails
|
||||
`nc_open`; this crate skips it).
|
||||
- **Cost:** the first call that needs the metadata reads that of the whole
|
||||
file (every group, and every dataset's attributes), as `nc_open` does;
|
||||
it was one group's per call. Not measured; worth measuring on a file
|
||||
with many groups and variables before a release.
|
||||
- A whole-variable read of a variable shorter than an unlimited dimension
|
||||
that is not its first: see [the entry of #27](#netcdf-4-variables-dimensions-are-guessed-from-sizes).
|
||||
|
||||
Corpus files that differ (2026-09-29):
|
||||
|
||||
- **External links** (`hdf5/test/testfiles/be_extlink1.h5`,
|
||||
`le_extlink1.h5`, `hdf5/tools/test/testfiles/h5diff_ext2softlink_src.h5`,
|
||||
`h5diff_grp_recurse_ext2-1.h5`, `h5diff_grp_recurse_ext2-2.h5`): libhdf5
|
||||
follows them into the other file; clawhdf5 does not
|
||||
([External links](#external-links-and-external-raw-data-are-not-followed)),
|
||||
so the linked objects are missing and the phony dimension numbers after
|
||||
them shift.
|
||||
- **Values the HDF5 reader refuses** and libhdf5 1.14.6 returns (metadata
|
||||
matches): `cve_hdf5/cvefiles/cve-2025-2308.h5`, `cve-2025-44904.h5` and
|
||||
`hdf5/test/testfiles/bad_nbit_parms_walk.h5`, the scale-offset and N-Bit
|
||||
files of CONFORMANCE.md's ref-bug/our-error list (libhdf5 reads past the
|
||||
stored data); and `hdf5/test/testfiles/bad_nbit_decompress.h5`, whose
|
||||
`/Nbit_float_data_le` clawhdf5 refuses ("nbit: element count exceeds
|
||||
chunk size", also `h5rs dump`) while libhdf5 1.14.6 and h5py 3.16
|
||||
(libhdf5 2.0.0) return values. That file is not in the conformance
|
||||
report and has not been investigated.
|
||||
- Not compared: the values of 18 variables whose filter (SZIP, Blosc,
|
||||
bzip2) this crate's default build of the HDF5 reader lacks. Blosc and
|
||||
bzip2 are pure-Rust cargo features of `clawhdf5` (`blosc`, `bzip2`) that
|
||||
a dependent can turn on; SZIP needs libaec.
|
||||
|
||||
## In-place modification (`FileEditor`) limits
|
||||
|
||||
**Status:** open (documented 2026-09-26, updated when version-2 B-tree
|
||||
@@ -539,6 +614,61 @@ Newest first. "Before any release" means no tagged release (v2.7.0 and
|
||||
earlier) contains the bug. Full detail is in `CHANGELOG.md` under the date
|
||||
given.
|
||||
|
||||
## NetCDF-4: files without dimension scales, types and order differed from netCDF-C
|
||||
|
||||
**Status:** fixed 2026-09-29 (branch `fix/netcdf-phony-dims`). Affected
|
||||
every release (v2.1.0 to v2.7.0) and `main` after #27. Wrong metadata only
|
||||
(names, order and types of dimensions and variables); values were read
|
||||
right. Users: the dimensions of a file without dimension scales are now
|
||||
netCDF-C's `phony_dim_<n>`, so code that matched `dim_<size>` or took a
|
||||
1-D dataset's name for a dimension must use the new names; datasets of
|
||||
types netCDF-C skips (references, bit fields, array types, compounds of
|
||||
those, ...) are no longer variables; lists come in netCDF-C's order.
|
||||
`NcType` has four new variants (`Enum`, `Compound`, `VLen`, `Opaque`) and
|
||||
is `#[non_exhaustive]`: an exhaustive `match` needs a wildcard arm.
|
||||
|
||||
Found 2026-09-28 by the one-off corpus comparison of #27 (52 of the 78
|
||||
files netCDF4-python opened matched; netCDF4-python itself hides variables
|
||||
of opaque and other types it does not support, so the comparison now asks
|
||||
netCDF-C through its C API). Under that comparison `main` at `4260af4`
|
||||
matched 68 of the 429 corpus files netCDF-C opens (tank, 2026-09-29, the
|
||||
command above run on a copy of `4260af4` with the new test files). What
|
||||
differed:
|
||||
|
||||
- **Files without dimension scales.** netCDF-C gives every axis a phony
|
||||
dimension `phony_dim_<id>` (`create_phony_dims`), ids file-wide after the
|
||||
real dimensions, subgroups numbered before their parent's variables,
|
||||
shared by length and unlimitedness within a group but not between two
|
||||
axes of one variable, a length of 0 always unlimited. The crate reported
|
||||
one dimension per 1-D dataset, named after it (an h5py file with `x(5)`
|
||||
and `v(5)` had dimensions `x` and `v`, and `v` on `x`), and other axes as
|
||||
`dim_<size>`.
|
||||
- **Datasets netCDF-C skips** (`NC_EBADTYPID`: references, bit fields,
|
||||
time and array types, a compound with such a member or a half-float
|
||||
member, an enum or VLEN over one) were variables, with `NcType::Char`;
|
||||
so were enums, compounds, VLENs and opaque types, and 1-byte strings
|
||||
were `String` where netCDF-C says `NC_CHAR` (longer ones `NC_STRING`).
|
||||
- **Order.** Groups and variables came in the order of the HDF5 links as
|
||||
stored (hash order in a dense group); netCDF-C uses creation order when
|
||||
the group tracks it, else name order. Dimensions without
|
||||
`_Netcdf4Dimid` came after the others instead of taking the next id in
|
||||
reading order; a zero-length dimension scale was not unlimited.
|
||||
- `_Netcdf4Coordinates` ids were looked up only in the variable's group
|
||||
and its parents (netCDF-C: file-wide), and an unlimited dimension's
|
||||
length came from its scale's `REFERENCE_LIST` in any group
|
||||
(`nc4_find_dim_len`: the variables in its group and below).
|
||||
|
||||
The crate now reads the metadata of the whole file in netCDF-C's two
|
||||
passes (`crates/clawhdf5-netcdf4/src/model.rs`), with a new
|
||||
`clawhdf5_format::group_v2::links_in_creation_order_in` for the order.
|
||||
Tests: `interop_tests::phony_dimensions_match_netcdf_c`,
|
||||
`types_netcdf_c_skips_are_not_variables`,
|
||||
`group_and_variable_order_match_netcdf_c` and
|
||||
`netcdf4_python_files_match_netcdf_c` compare h5py- and
|
||||
netCDF4-python-written files with netCDF-C itself, and
|
||||
`corpus_vs_netcdf_c` the corpus (420 of 429 match; the rest are in
|
||||
[NetCDF-4: differences from netCDF-C](#netcdf-4-differences-from-netcdf-c)).
|
||||
|
||||
## NetCDF-4: variables' dimensions are guessed from sizes
|
||||
|
||||
|
||||
@@ -590,7 +720,8 @@ dimension scales, and h5netcdf 1.8.1 and xarray files (tank, 2026-09-28,
|
||||
`CLAWHDF5_PYTHON=<venv with h5netcdf> cargo test -p clawhdf5-netcdf4`).
|
||||
Files without dimension scales still get dimensions by size (netCDF-C
|
||||
gives them `phony_dim_<n>`), as before; see `CHANGELOG.md` for a
|
||||
comparison over the conformance corpus's netCDF-readable files.
|
||||
comparison over the conformance corpus's netCDF-readable files. (Fixed
|
||||
2026-09-29: see the entry above.)
|
||||
|
||||
## HDF5 1.8 could not read the files we wrote
|
||||
|
||||
|
||||
Reference in New Issue
Block a user