build: pure-Rust zlib-rs as the default deflate backend
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel, -netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs, a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in `fast-deflate`, which overrides zlib-rs wherever it is enabled. A default build no longer needs cmake or a C compiler. Measured on tank, both builds run alternately, three rounds, medians: zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512 deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4 vs 65.2 ms), and compressed output is byte-identical. Details in BENCHMARKS.md, "Deflate backend". Getting there took two fixes the first measurement exposed: - zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via its default `runtime_detection`, which `default-features = false` had switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs` features now enable it. - Both deflate paths streamed through flate2's 32 KiB read/write wrappers. They now hand the codec the whole chunk in one call, into a buffer sized up front (~5% on chunked writes). This also fixes a silent short read: the streaming reader returned a truncated stream's bytes without an error; a truncated chunk is now DecompressionError. In clawhdf5-filters, output longer than the stated size is now an error rather than silently cut off. CI: ci-test.sh lints and tests the zlib-ng path, and fails if a C-building crate (*-sys, cc, cmake) enters a core crate's default dependency tree. The arm64 job no longer installs cmake. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -77,6 +77,35 @@ run_step "cargo clippy (ann parallel)" cargo clippy \
|
||||
--features parallel \
|
||||
-- -D warnings
|
||||
|
||||
# zlib-ng is opt-in (`fast-deflate`; the default is pure-Rust zlib-rs), so
|
||||
# nothing above builds it. Keep it compiling and passing.
|
||||
run_step "cargo clippy (fast-deflate / zlib-ng)" cargo clippy \
|
||||
-p clawhdf5-format -p clawhdf5-filters -p clawhdf5 \
|
||||
--all-targets \
|
||||
--features clawhdf5-format/fast-deflate,clawhdf5-filters/fast-deflate \
|
||||
-- -D warnings
|
||||
|
||||
# The README promises that the core crates build no C by default. Hold it to
|
||||
# that: fail if a crate that compiles C (a *-sys crate, cc or cmake) enters the
|
||||
# default dependency tree of any of them. clawhdf5-migrate (bundled SQLite),
|
||||
# clawhdf5-napi (Node) and clawhdf5-gpu (graphics drivers) are exempt.
|
||||
no_c_in_default_build() {
|
||||
local crate found=0
|
||||
for crate in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \
|
||||
clawhdf5-agent clawhdf5-ann clawhdf5-accel clawhdf5-netcdf4 clawhdf5-cli; do
|
||||
local c_deps
|
||||
c_deps=$(cargo tree -q -p "$crate" -e normal,build --prefix none \
|
||||
| grep -E '^([a-z0-9_-]+-sys|cc|cmake) v' | sort -u)
|
||||
if [ -n "$c_deps" ]; then
|
||||
echo "$crate pulls in C by default:"
|
||||
echo "$c_deps" | sed 's/^/ /'
|
||||
found=1
|
||||
fi
|
||||
done
|
||||
return $found
|
||||
}
|
||||
run_step "no C in the default build (core crates)" no_c_in_default_build
|
||||
|
||||
# 4. Tests (exclude clawhdf5-py)
|
||||
run_step "cargo test" cargo test \
|
||||
--workspace \
|
||||
@@ -90,6 +119,10 @@ run_step "cargo test (ann parallel)" cargo test \
|
||||
-p clawhdf5-ann \
|
||||
--features parallel
|
||||
|
||||
run_step "cargo test (fast-deflate / zlib-ng)" cargo test \
|
||||
-p clawhdf5-format -p clawhdf5-filters -p clawhdf5 \
|
||||
--features clawhdf5-format/fast-deflate,clawhdf5-filters/fast-deflate
|
||||
|
||||
# 5. Python interop suites. The h5py writer tests are #[ignore]d so a plain
|
||||
# `cargo test` stays hermetic; run them explicitly here.
|
||||
# On a PEP 668 "externally managed" system h5py can only live in a
|
||||
|
||||
Reference in New Issue
Block a user