build: pure-Rust zlib-rs as the default deflate backend

The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.

Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".

Getting there took two fixes the first measurement exposed:

- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
  its default `runtime_detection`, which `default-features = false` had
  switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
  features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
  wrappers. They now hand the codec the whole chunk in one call, into a
  buffer sized up front (~5% on chunked writes). This also fixes a
  silent short read: the streaming reader returned a truncated stream's
  bytes without an error; a truncated chunk is now DecompressionError.
  In clawhdf5-filters, output longer than the stated size is now an
  error rather than silently cut off.

CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-23 11:05:22 -05:00
co-authored by Claude Opus 5.5
parent e9b37a9602
commit 735db117a7
13 changed files with 471 additions and 113 deletions
+3 -15
View File
@@ -28,8 +28,9 @@ jobs:
# dependency a failure (CLAWHDF5_REQUIRE_INTEROP below).
run: |
apt-get update
# cmake builds libz-ng-sys (clawhdf5-format's default `fast-deflate`);
# rust:latest does not ship it.
# cmake builds libz-ng-sys for the opt-in `fast-deflate` (zlib-ng)
# steps in ci-test.sh; rust:latest does not ship it. The default
# build (pure-Rust zlib-rs) does not need it.
apt-get install -y --no-install-recommends python3 python3-venv cmake
python3 -m venv /opt/interop
/opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray
@@ -75,19 +76,6 @@ jobs:
command -v rustup >/dev/null || curl -sSf --retry 5 https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain none
rustup toolchain install stable --profile minimal --component clippy
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
- name: Build dependencies
# libz-ng-sys (clawhdf5-format's default `fast-deflate`) needs cmake.
# In the Docker runner the job is root and can install it; a host
# runner cannot, so say what is missing rather than fail inside a
# build script.
run: |
if command -v cmake >/dev/null; then cmake --version | head -1; exit 0; fi
if [ "$(id -u)" = 0 ]; then
apt-get update -qq && apt-get install -y -qq cmake
else
echo "::error::cmake is not installed on this runner host (needed by libz-ng-sys)"
exit 1
fi
- name: Confirm aarch64
run: |
test "$(uname -m)" = aarch64