Read a file's metadata the way netCDF-C 4.9.3 does (libhdf5/hdf5open.c),
for the whole file on first use (src/model.rs, replacing src/scope.rs):
- links in creation order when the group tracks it, else name order;
a group's datasets before its subgroups; dimension ids file-wide;
- variables' dimensions from _Netcdf4Coordinates (file-wide ids), else
the scales DIMENSION_LIST attaches when the first axis has one, else
netCDF-C's phony dimensions phony_dim_<id> (create_phony_dims: shared
by length and unlimitedness within a group, not between two axes of
one variable, numbered subgroups first, a zero length unlimited);
- datasets of types netCDF-C cannot represent are not variables
(references, bit fields, time, arrays, compounds/enums/VLENs over
them), replaying netCDF-C's file-wide type list, failed types
included;
- unlimited lengths as nc4_find_dim_len (its group and below).
NcType gains Enum, Compound, VLen, Opaque and is #[non_exhaustive];
Variable::nc_type is netCDF-C's type (1-byte strings NC_CHAR). New
clawhdf5_format::group_v2::links_in_creation_order_in.
Tests compare with netCDF-C itself (tests/netcdf_c_view.py calls the
libnetcdf netCDF4-python bundles through ctypes): new interop cases for
h5py files without dimension scales, every type class, link order; and
the gated corpus_vs_netcdf_c (CLAWHDF5_NETCDF_CORPUS): 420 of the 429
conformance-corpus files netCDF-C opens match (main: 68); the other 9
are explained in tests/corpus_known_differences.txt and known-issues.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Variables got the first unused dimension of equal size, so a variable on
an unlimited dimension with fewer records got an anonymous dim_<n>, and
dimensions of one size could be swapped. Resolve them as netCDF-C does
(libhdf5/hdf5open.c): _Netcdf4Coordinates ids, else the scales
DIMENSION_LIST references (the last one attached to an axis), searched in
the variable's group and its parents; a coordinate variable is on its own
scale. Size matching remains only for axes the file names nothing for.
variables()/variable_names() leave out dimension scales that are only
dimensions, and _nc4_non_coord_<name> is the variable <name>.
Variable::shape is the netCDF shape (an unlimited dimension's length) and
the reads pad unwritten records with the fill value (_FillValue, else
NC_FILL_*; NaN from read_f64); Variable::stored_shape is the HDF5 extent.
New NetCDF4File::variable_names.
Tests compare with netCDF4-python variable by variable: the known-issues
reproducer, equal sizes, (p, p), scalars, inherited dimensions, unwritten
records, h5py dimension scales, h5netcdf and xarray files. CI installs
h5netcdf. known-issues entry moved to Fixed (history); stale open-table
row for the unlimited-size fix removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Dimension::size of an unlimited dimension was its dimension scale's
extent, which netCDF-C leaves at 0, so it read 0 for a dimension holding
records. It is now what netCDF-C reports (nc4_find_dim_len): the largest
current extent of the variables using it in any group, found through the
scale's REFERENCE_LIST, a coordinate variable's own extent included; 0
when nothing has been written.
interop_tests::unlimited_dimension_lengths_match_netcdf4_python compares
with netCDF4-python (variables of different lengths, one in a subgroup,
an unwritten dimension, a coordinate variable shorter than another
variable on its dimension, a subgroup's own dimension); before the fix it
got time 0/6, rec 3/5, srec 0/1.
The known-issues entry moves to Fixed; the crate README's warning goes.
A new open entry records a related bug found meanwhile: variables'
dimensions are matched by size, not DIMENSION_LIST.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
VL-string datasets (h5py's default str dtype) failed read_string with
"type mismatch: expected String, got VariableLength". read_string now
reads fixed- and variable-length strings, with h5py's values (a string
ends at a NUL, a null element is ""). New:
- Dataset::read_string_bytes: each VL string's exact bytes;
- Dataset::read_string_selection: hyperslabs/points of either kind;
- Dataset::read_vlen::<T>() and read_vlen_selection::<T>(): VL sequences
of numbers as Vec<Vec<T>>, T in f64/f32/i64/i32/u64, converted like the
other typed readers;
- File::decode_strings / decode_string_bytes / decode_vlen: VL values in
compound fields and AttrValue::Raw attributes;
- MmapDataset and LazyDataset: read_string for VL strings,
read_string_bytes and read_vlen.
tests/vl_data_interop.rs checks every path against h5py with 8- and
4-byte offsets: scalar, 1-D and 2-D, ASCII and UTF-8, empty strings,
contiguous, compact, chunked with gzip and shuffle, unwritten and partly
written chunks, hyperslabs, compound members, attributes, a big-endian
base type, and a patched file with an embedded NUL and mis-sized heap
objects. NetCDF-4 string variables read too (netCDF4-python test).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Every Python interop suite had stopped running on this machine: the h5py
writer round-trips, the facade suite, netCDF4 and the reference files.
`python3` is 3.14, nothing on the box has h5py, and PEP 668 refuses to
install it into a system interpreter at all — so the availability probes
all returned false and each suite skipped without failing.
A silent skip here is exactly how the v5 compound-datatype bug reached a
release, so the probes now read `CLAWHDF5_PYTHON` and `ci-test.sh` picks
up `.venv/bin/python` on its own. The detection sits at the top of the
script rather than beside the interop step, because the non-ignored
suites run in the earlier `cargo test` step and would otherwise still
miss it. `CLAWHDF5_REQUIRE_INTEROP=1` continues to turn a skip into a
failure.
Verified against a venv with h5py 3.16 / HDF5 2.0.0: 94 interop tests
across the four suites, all passing.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
