build: pure-Rust zlib-rs as the default deflate backend

The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.

Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".

Getting there took two fixes the first measurement exposed:

- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
  its default `runtime_detection`, which `default-features = false` had
  switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
  features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
  wrappers. They now hand the codec the whole chunk in one call, into a
  buffer sized up front (~5% on chunked writes). This also fixes a
  silent short read: the streaming reader returned a truncated stream's
  bytes without an error; a truncated chunk is now DecompressionError.
  In clawhdf5-filters, output longer than the stated size is now an
  error rather than silently cut off.

CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-23 11:05:22 -05:00
co-authored by Claude Opus 5.5
parent e9b37a9602
commit 735db117a7
13 changed files with 471 additions and 113 deletions
+33
View File
@@ -77,6 +77,35 @@ run_step "cargo clippy (ann parallel)" cargo clippy \
--features parallel \
-- -D warnings
# zlib-ng is opt-in (`fast-deflate`; the default is pure-Rust zlib-rs), so
# nothing above builds it. Keep it compiling and passing.
run_step "cargo clippy (fast-deflate / zlib-ng)" cargo clippy \
-p clawhdf5-format -p clawhdf5-filters -p clawhdf5 \
--all-targets \
--features clawhdf5-format/fast-deflate,clawhdf5-filters/fast-deflate \
-- -D warnings
# The README promises that the core crates build no C by default. Hold it to
# that: fail if a crate that compiles C (a *-sys crate, cc or cmake) enters the
# default dependency tree of any of them. clawhdf5-migrate (bundled SQLite),
# clawhdf5-napi (Node) and clawhdf5-gpu (graphics drivers) are exempt.
no_c_in_default_build() {
local crate found=0
for crate in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \
clawhdf5-agent clawhdf5-ann clawhdf5-accel clawhdf5-netcdf4 clawhdf5-cli; do
local c_deps
c_deps=$(cargo tree -q -p "$crate" -e normal,build --prefix none \
| grep -E '^([a-z0-9_-]+-sys|cc|cmake) v' | sort -u)
if [ -n "$c_deps" ]; then
echo "$crate pulls in C by default:"
echo "$c_deps" | sed 's/^/ /'
found=1
fi
done
return $found
}
run_step "no C in the default build (core crates)" no_c_in_default_build
# 4. Tests (exclude clawhdf5-py)
run_step "cargo test" cargo test \
--workspace \
@@ -90,6 +119,10 @@ run_step "cargo test (ann parallel)" cargo test \
-p clawhdf5-ann \
--features parallel
run_step "cargo test (fast-deflate / zlib-ng)" cargo test \
-p clawhdf5-format -p clawhdf5-filters -p clawhdf5 \
--features clawhdf5-format/fast-deflate,clawhdf5-filters/fast-deflate
# 5. Python interop suites. The h5py writer tests are #[ignore]d so a plain
# `cargo test` stays hermetic; run them explicitly here.
# On a PEP 668 "externally managed" system h5py can only live in a