build: pure-Rust zlib-rs as the default deflate backend
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel, -netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs, a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in `fast-deflate`, which overrides zlib-rs wherever it is enabled. A default build no longer needs cmake or a C compiler. Measured on tank, both builds run alternately, three rounds, medians: zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512 deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4 vs 65.2 ms), and compressed output is byte-identical. Details in BENCHMARKS.md, "Deflate backend". Getting there took two fixes the first measurement exposed: - zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via its default `runtime_detection`, which `default-features = false` had switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs` features now enable it. - Both deflate paths streamed through flate2's 32 KiB read/write wrappers. They now hand the codec the whole chunk in one call, into a buffer sized up front (~5% on chunked writes). This also fixes a silent short read: the streaming reader returned a truncated stream's bytes without an error; a truncated chunk is now DecompressionError. In clawhdf5-filters, output longer than the stated size is now an error rather than silently cut off. CI: ci-test.sh lints and tests the zlib-ng path, and fails if a C-building crate (*-sys, cc, cmake) enters a core crate's default dependency tree. The arm64 job no longer installs cmake. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+3
-15
@@ -28,8 +28,9 @@ jobs:
|
||||
# dependency a failure (CLAWHDF5_REQUIRE_INTEROP below).
|
||||
run: |
|
||||
apt-get update
|
||||
# cmake builds libz-ng-sys (clawhdf5-format's default `fast-deflate`);
|
||||
# rust:latest does not ship it.
|
||||
# cmake builds libz-ng-sys for the opt-in `fast-deflate` (zlib-ng)
|
||||
# steps in ci-test.sh; rust:latest does not ship it. The default
|
||||
# build (pure-Rust zlib-rs) does not need it.
|
||||
apt-get install -y --no-install-recommends python3 python3-venv cmake
|
||||
python3 -m venv /opt/interop
|
||||
/opt/interop/bin/pip install --no-cache-dir h5py numpy netCDF4 xarray
|
||||
@@ -75,19 +76,6 @@ jobs:
|
||||
command -v rustup >/dev/null || curl -sSf --retry 5 https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain none
|
||||
rustup toolchain install stable --profile minimal --component clippy
|
||||
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"
|
||||
- name: Build dependencies
|
||||
# libz-ng-sys (clawhdf5-format's default `fast-deflate`) needs cmake.
|
||||
# In the Docker runner the job is root and can install it; a host
|
||||
# runner cannot, so say what is missing rather than fail inside a
|
||||
# build script.
|
||||
run: |
|
||||
if command -v cmake >/dev/null; then cmake --version | head -1; exit 0; fi
|
||||
if [ "$(id -u)" = 0 ]; then
|
||||
apt-get update -qq && apt-get install -y -qq cmake
|
||||
else
|
||||
echo "::error::cmake is not installed on this runner host (needed by libz-ng-sys)"
|
||||
exit 1
|
||||
fi
|
||||
- name: Confirm aarch64
|
||||
run: |
|
||||
test "$(uname -m)" = aarch64
|
||||
|
||||
Reference in New Issue
Block a user