osobhandClaude Opus 5.5
e5d6f59e12
Read a file's metadata the way netCDF-C 4.9.3 does (libhdf5/hdf5open.c), for the whole file on first use (src/model.rs, replacing src/scope.rs): - links in creation order when the group tracks it, else name order; a group's datasets before its subgroups; dimension ids file-wide; - variables' dimensions from _Netcdf4Coordinates (file-wide ids), else the scales DIMENSION_LIST attaches when the first axis has one, else netCDF-C's phony dimensions phony_dim_<id> (create_phony_dims: shared by length and unlimitedness within a group, not between two axes of one variable, numbered subgroups first, a zero length unlimited); - datasets of types netCDF-C cannot represent are not variables (references, bit fields, time, arrays, compounds/enums/VLENs over them), replaying netCDF-C's file-wide type list, failed types included; - unlimited lengths as nc4_find_dim_len (its group and below). NcType gains Enum, Compound, VLen, Opaque and is #[non_exhaustive]; Variable::nc_type is netCDF-C's type (1-byte strings NC_CHAR). New clawhdf5_format::group_v2::links_in_creation_order_in. Tests compare with netCDF-C itself (tests/netcdf_c_view.py calls the libnetcdf netCDF4-python bundles through ctypes): new interop cases for h5py files without dimension scales, every type class, link order; and the gated corpus_vs_netcdf_c (CLAWHDF5_NETCDF_CORPUS): 420 of the 429 conformance-corpus files netCDF-C opens match (main: 68); the other 9 are explained in tests/corpus_known_differences.txt and known-issues. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
107 lines
5.7 KiB
Markdown
107 lines
5.7 KiB
Markdown
# clawhdf5-netcdf4
|
|
|
|
Read NetCDF-4 files in pure Rust. NetCDF-4 files are HDF5 files with
|
|
conventions for dimensions, coordinate variables and attributes; this crate
|
|
reads them through the [`clawhdf5`](../clawhdf5/README.md) facade, with no
|
|
libnetcdf or libhdf5. Read-only: NetCDF-3 (classic) files are not HDF5 and
|
|
are not supported.
|
|
|
|
Not on crates.io yet; depend on it from git:
|
|
|
|
```toml
|
|
[dependencies]
|
|
clawhdf5-netcdf4 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
|
|
```
|
|
|
|
## Usage
|
|
|
|
```rust,no_run
|
|
use clawhdf5_netcdf4::NetCDF4File;
|
|
|
|
let nc = NetCDF4File::open("climate.nc")?;
|
|
for dim in nc.dimensions()? {
|
|
println!("{}: {} (unlimited = {})", dim.name, dim.size, dim.is_unlimited);
|
|
}
|
|
let mut temp = nc.variable("temperature")?;
|
|
let dims: Vec<&str> = temp.dimensions().iter().map(|d| d.name.as_str()).collect();
|
|
println!("{:?} over {:?}", temp.shape()?, dims);
|
|
let cf = temp.cf_attributes()?;
|
|
println!("units: {:?}", cf.units);
|
|
// scale_factor/add_offset applied; _FillValue and missing_value become NaN
|
|
let values: Vec<f64> = temp.read_f64()?;
|
|
# Ok::<(), clawhdf5_netcdf4::Error>(())
|
|
```
|
|
|
|
## API
|
|
|
|
| Item | What |
|
|
|---|---|
|
|
| `NetCDF4File` | `open`, `from_bytes`, `dimensions`, `variables`, `variable_names`, `variable` (a name or a path into a subgroup), `global_attrs`, `group` (a name or a path), `group_names`, `nc_properties`, and `hdf5_file` for the underlying `clawhdf5::File` |
|
|
| `NetCDF4Group` | the same for a sub-group (`dimensions`, `variables`, `variable_names`, `attrs`, nested `group`) |
|
|
| `Variable` | `name`, `shape`, `stored_shape`, `dimensions`, `nc_type`, `is_coordinate`, `attrs`, `cf_attributes`; `read_f64` (CF scale/offset and fill applied), `read_raw_f32`/`_f64`/`_i32`/`_i64`/`_u64`, `read_string`, `read_raw` |
|
|
| `Dimension` | `name`, `size`, `is_unlimited` (an unlimited dimension's `size` is its current length as netCDF-C reports it: the largest extent of the variables using it) |
|
|
| `CfAttributes` | CF convention attributes: `units`, `long_name`, `standard_name`, `fill_value` (`_FillValue`), `missing_value`, `scale_factor`, `add_offset`, `valid_range`, `calendar`, `axis` |
|
|
| `NcType` | the NetCDF type of a variable: an atomic type (`Byte` ... `UInt64`, `Float`, `Double`, `Char`, `String`) or the class of a user-defined one (`Enum`, `Compound`, `VLen`, `Opaque`); `#[non_exhaustive]` |
|
|
|
|
Groups, dimensions and variables are what netCDF-C 4.9.3 reports
|
|
(`libhdf5/hdf5open.c`), names, order and types included. The first call
|
|
that needs them reads the metadata of the whole file, as `nc_open` does:
|
|
|
|
- Order: a group's links in creation order when it tracks it (netCDF-4
|
|
files do), else in name order (h5py's default); groups and variables in
|
|
that order, dimensions by id.
|
|
- A dimension scale defines a dimension (its `_Netcdf4Dimid`, else the
|
|
next free id; unlimited when its first axis is or its length is 0);
|
|
scales that are only dimensions are not variables, and a dataset
|
|
`_nc4_non_coord_<name>` is the variable `<name>`.
|
|
- A variable's dimensions are the ones the file names: the ids in its
|
|
`_Netcdf4Coordinates` attribute (any group), else — when its first axis
|
|
has one — the dimension scales its `DIMENSION_LIST` attaches, found in
|
|
its group or a parent group.
|
|
- Otherwise (an HDF5 file not written by a netCDF library) its axes get
|
|
netCDF-C's **phony dimensions** `phony_dim_<n>`: per axis, the first
|
|
dimension of the variable's group with the same length and
|
|
unlimitedness that an earlier axis of the variable does not use, else a
|
|
new one. The numbers run file-wide, a group's subgroups before its own
|
|
variables.
|
|
- Datasets of types netCDF-C cannot represent are not variables:
|
|
references, bit fields, time and array types, and compounds, enums and
|
|
VLENs over members or base types that are not netCDF atomic types (4-
|
|
and 8-byte floats only) or types netCDF-C has read before in the file.
|
|
Enum, compound, VLEN and opaque datasets are variables of those classes
|
|
(netCDF4-python itself leaves out opaque ones).
|
|
- Two deliberate differences: floats of other than 4 or 8 bytes (half,
|
|
bfloat16, 4/6/8-bit floats, `long double`) are `Float`/`Double` and
|
|
read as numbers, where netCDF-C 4.9.3 on libhdf5 1.14.6 labels them
|
|
`NC_STRING`; and an axis netCDF-C leaves without a dimension (an id or
|
|
scale it cannot find, or no scale on an axis after a first one that has
|
|
one, where it reads uninitialised memory) gets one by the phony rule.
|
|
- A variable along an unlimited dimension has the dimension's length:
|
|
`shape` is that length, and the reads return that many values, the
|
|
records the variable has not written as its fill value (`_FillValue`,
|
|
else netCDF's default for the type; NaN from `read_f64`).
|
|
`stored_shape` is the HDF5 dataset's extent.
|
|
|
|
No cargo features. Tests compare against files written by netCDF4-python,
|
|
h5py (with and without dimension scales, every HDF5 type class), h5netcdf
|
|
and xarray, with what netCDF4-python reads and with what netCDF-C itself
|
|
reports (`tests/interop_tests.rs`; `tests/netcdf_c_view.py` calls the
|
|
libnetcdf netCDF4-python bundles; the CI job requires them with
|
|
`CLAWHDF5_REQUIRE_INTEROP=1`; the h5netcdf cases skip when h5netcdf is not
|
|
installed). `tests/corpus_vs_netcdf_c.rs` compares every file of a corpus
|
|
netCDF-C opens, when `CLAWHDF5_NETCDF_CORPUS` names one: over the
|
|
conformance corpus, 420 of 429 files match (tank, 2026-09-29), and the
|
|
other 9 are explained in `tests/corpus_known_differences.txt`:
|
|
|
|
```sh
|
|
CLAWHDF5_NETCDF_CORPUS=conformance/.cache/corpus CLAWHDF5_PYTHON=$PWD/.venv/bin/python \
|
|
cargo test -p clawhdf5-netcdf4 --test corpus_vs_netcdf_c -- --nocapture
|
|
```
|
|
|
|
The differences from netCDF-C, and what the HDF5 reader underneath cannot
|
|
read, are listed in [`docs/known-issues.md`](../../docs/known-issues.md).
|
|
|
|
## License
|
|
|
|
MIT
|