- clippy --all-targets plus a clawhdf5-format feature matrix (parallel, lz4,
zstd, pcodec, fast-checksum); fix the accumulated lint backlog in test,
bench and feature-gated code (no behaviour changes).
- Install python3 + h5py/numpy/netCDF4/xarray in the CI container and set
CLAWHDF5_REQUIRE_INTEROP=1, which makes a missing interop dependency a test
failure. Every h5py/netCDF4 interop test used to skip silently in CI. Run
the #[ignore]d writer_h5py_tests suite explicitly.
- cargo bench --no-run so benches can't rot; fix bench.rs and memory_bench.rs,
which no longer compiled against the current strategy/consolidation APIs.
- Optional fuzz smoke run via CLAWHDF5_FUZZ_SECONDS.
- CHANGELOG and docs/known-issues.md updated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
- Add .gitea/workflows/ci.yml running scripts/ci-test.sh (fmt, clippy,
test, no_std check) on push/PR to main.
- Fix stale rustyhdf5-py/rustyhdf5-format package names in
ci-test.sh/check-nostd.sh, which had been silently no-op'ing those
checks (cargo warns but doesn't fail on an unknown --exclude/-p
target).
- With those checks actually running, fix the real issues they surface:
- clippy: useless_conversion in chunked_write.rs, byte_char_slices in
global_heap.rs/object_header.rs.
- cargo fmt: apply formatting across the workspace (whitespace only).
- no_std (thumbv7em-none-eabihf) build errors in clawhdf5-format:
core::sync::atomic::AtomicU64 doesn't exist on that target (no
native 64-bit atomics) — switch profiling.rs's counters to
portable-atomic, which falls back to a CAS-based emulation there
and is a no-op wrapper elsewhere. Add missing alloc imports for
Box (filters.rs), Vec (filters_szip.rs), and format! (dict_encoding.rs)
on no_std paths. Replace f64::powi (std/libm-only) with a small
local exponentiation-by-squaring helper in the scale-offset filter.
The readers added this cycle parse untrusted bytes, so malformed/hostile
input must produce errors — never a panic, OOM, or unbounded recursion.
Audited each new surface and fixed the concrete vectors, each covered by an
adversarial regression test:
- Paged Fixed Array: `1 << max_nelmts_bits` shift overflow (u8 up to 255);
element-count bounded by file size; element/page offset multiplies checked.
- H5S selection decoder: ALL/NONE validate they have the 16 bytes they claim
to consume; hyperslab rank capped at 32 (H5S_MAX_RANK); iter_linear
coordinate/stride/product arithmetic uses checked ops.
- VDS mapping parser: drop pre-allocation from the untrusted `nused`;
bounds-check all selection slicing.
- scale-offset / N-Bit filters: `1 << minbits` overflow at minbits==64; N-Bit
`bit_offset + precision` overflow; N-Bit type-tree recursion depth capped to
stop a crafted nested tree from overflowing the stack; element counts bounded
by the chunk's expected decompressed size (threaded the previously-unused
chunk_size into both decoders) so a bogus count can't over-allocate.
- VDS assembly: a virtual dataset whose source is itself virtual (a cycle) now
errors instead of recursing into a stack overflow.
16 new adversarial tests; full format suite (482 lib) + agent + facade green;
clippy clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Generalize Selection iteration from 1-D to arbitrary rank: iter_linear(dims)
enumerates a selection's row-major linear indices over a dataspace of the
given shape (ALL, NONE, regular hyperslabs, points), which is the order HDF5
uses to pair virtual and source selections.
read_virtual_data now passes the full virtual/source dimensions instead of a
single extent, so multi-dimensional block mappings scatter to the correct
non-contiguous linear positions. read_named_dataset_raw returns the source
dataset's dimensions. The rank-1 restriction is removed; only external-file
sources remain unsupported.
Tests: 2-D integration fixture (vds_2d_same_file.h5: two 2x2 sources placed
as non-contiguous blocks in a 4x4 virtual) plus N-D iter_linear unit tests
(block, strided, ALL, rank-mismatch). The 1-D path is unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A virtual layout previously returned UnsupportedVersion. Implement reading
for the common 1-D, same-file case, reverse-engineered and validated against
HDF5 2.0.
- Rewrite parse_vds_mappings to the real global-heap block format
(version(1) · nused(length_size) · entries · checksum(4)), where each
entry is source-file(null) · source-dataset(null) · source-selection ·
virtual-selection. Block version 1 encodes a same-file source as a single
0x04 marker in place of the file name; version 0 stores an explicit file
name. The selections are H5S-serialized and self-describing in length, so
they are decoded to find entry boundaries. The previous parser used a
guessed layout that did not match real files.
- Extend Selection with decode_serialized() (H5S_select_serialize: ALL,
NONE, and version-3 regular hyperslabs) and iter_linear_1d().
- Add read_virtual_data: resolve the mapping block from the global heap,
read each same-file source dataset, and scatter its selected elements into
the virtual buffer; unmapped regions stay at the zero fill value.
External-file sources and N-D selections return a clean unsupported error.
Tests: real-file integration test (vds_same_file.h5: partial source slice +
fill gap), selection decoder unit tests built from the fixture bytes, and
same-file/external mapping-parser unit tests.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
