quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

build: pure-Rust zlib-rs as the default deflate backend

Browse Source 
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.

Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".

Getting there took two fixes the first measurement exposed:

- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
  its default `runtime_detection`, which `default-features = false` had
  switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
  features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
  wrappers. They now hand the codec the whole chunk in one call, into a
  buffer sized up front (~5% on chunked writes). This also fixes a
  silent short read: the streaming reader returned a truncated stream's
  bytes without an error; a truncated chunk is now DecompressionError.
  In clawhdf5-filters, output longer than the stated size is now an
  error rather than silently cut off.

CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-23 11:05:22 -05:00
co-authored by Claude Opus 5.5
parent e9b37a9602
commit 735db117a7
13 changed files with 471 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-filters/Cargo.toml
+6 -2
View File
@@ -25,8 +25,12 @@ name = "compression_bench"
harness = false
[features]
default = ["fast-deflate"]
# Pure-Rust zlib-rs by default; `fast-deflate` (zlib-ng, C) overrides it.
default = ["zlib-rs"]
fast-deflate = ["flate2/zlib-ng"]
system-zlib = ["flate2/zlib-default"]
zlib-rs = ["flate2/zlib-rs"]
# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
# SIMD at runtime. flate2 enables it by default, but we build flate2 with
# default-features = false, and without it zlib-rs inflates 3.5x slower.
zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
apple-compression = []