quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

build: pure-Rust zlib-rs as the default deflate backend

Browse Source 
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.

Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".

Getting there took two fixes the first measurement exposed:

- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
  its default `runtime_detection`, which `default-features = false` had
  switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
  features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
  wrappers. They now hand the codec the whole chunk in one call, into a
  buffer sized up front (~5% on chunked writes). This also fixes a
  silent short read: the streaming reader returned a truncated stream's
  bytes without an error; a truncated chunk is now DecompressionError.
  In clawhdf5-filters, output longer than the stated size is now an
  error rather than silently cut off.

CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-23 11:05:22 -05:00
co-authored by Claude Opus 5.5
parent e9b37a9602
commit 735db117a7
13 changed files with 471 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files
BENCHMARKS.md
+63
View File
@@ -1185,6 +1185,69 @@ cargo run --release --bin ephemeral_perf
---
## Deflate backend: zlib-rs vs zlib-ng
Measured 2026-09-23 on tank (AMD Ryzen 7 7800X3D, 8C/16T). The default
deflate backend is now **zlib-rs**, a pure-Rust port of zlib-ng; zlib-ng (C,
built with cmake) was the default before and is still available as
`fast-deflate`. Both builds were compiled once into separate target
directories and run **alternately, three rounds each**; figures are medians.
```bash
# zlib-rs (default)
cargo bench -p clawhdf5-filters --bench deflate_bench
cargo bench -p clawhdf5-bench --bench h5bench_write --features libhdf5-compare -- '^write_2d_chunked/'
cargo run --release -p clawhdf5-bench --bin read_harness
# zlib-ng: add --features fast-deflate (filters) or clawhdf5-format/fast-deflate (bench)
```
| Workload | zlib-rs | zlib-ng | rs / ng |
|---|---:|---:|---:|
| HDF5 chunked write, deflate-6, 512×512 f32 | 1.458 ms | 1.484 ms | 0.98 |
| HDF5 chunked write, deflate-6, 128×128 f32 | 157.6 µs | 152.8 µs | 1.03 |
| HDF5 chunked write, deflate-6, 32×32 f32 | 62.9 µs | 60.2 µs | 1.05 |
| HDF5 read, 64 MB chunked + deflate, full | 64.4 ms | 65.2 ms | 0.99 |
| HDF5 read, 64×64 window (1 chunk) | 0.18 ms | 0.17 ms | 1.06 |
| HDF5 read, 512×512 window (4–9 chunks) | 4.10 ms | 4.10 ms | 1.00 |
| HDF5 read, one row / one column | 1.00 / 2.01 ms | 0.95 / 1.95 ms | 1.05 / 1.03 |
| Raw inflate, 8 MB f64 | 5.92 ms | 6.06 ms | 0.98 |
| Raw inflate, 1 MB sine | 82.8 µs | 68.6 µs | 1.21 |
| Raw deflate-6, 8 MB f64 / 1 MB sine | 92.2 / 2.01 ms | 83.1 / 1.84 ms | 1.11 / 1.09 |
Compressed output is **byte-identical** between the two at levels 1, 6 and 9
on all three inputs, so files do not change size. libhdf5 1.14.6 took 51.4 ms
for the 512×512 write in the same session (35× the zlib-rs figure).
On the HDF5 paths zlib-rs is within 6% of zlib-ng everywhere, and ahead on the
largest write. The raw codec loops show zlib-ng still slightly faster at
compression (~10%), which chunked writes do not expose because encoding runs
in parallel across chunks.
**Two findings along the way.** The first measurement had zlib-rs 1.2–1.9×
slower on single-chunk reads and 3.7× slower on a 1 MB inflate — slower even
than miniz_oxide. Neither was zlib-rs's fault:
1. **Runtime CPU detection was off.** zlib-rs needs its `std` feature to
detect and use SIMD at runtime; flate2 turns it on through its default
`runtime_detection` feature, which our `default-features = false` flate2
dependency was disabling. With it, a 1 MB inflate goes 282 → 83 µs.
`clawhdf5-format/zlib-rs` and `clawhdf5-filters/zlib-rs` now enable it.
2. **The codec was fed through a 32 KiB buffer.** Both deflate paths used
flate2's streaming `read::ZlibDecoder` / `write::ZlibEncoder`. A chunk's
decompressed size is known, so they now hand the codec the whole input in
one call, into an output buffer sized up front. Worth ~5% on chunked
writes and ~10% on zlib-ng's 1 MB inflate. It also closed a hole: the
streaming reader returned a truncated stream's bytes without an error, so
a truncated chunk read back short; it is now an error.
| 1 MB inflate, same build otherwise | zlib-rs | zlib-ng |
|---|---:|---:|
| streaming reader, no runtime detection | 284.1 µs | 76.7 µs |
| one-shot, no runtime detection | 282.3 µs | 68.5 µs |
| one-shot + runtime detection (shipped) | **82.8 µs** | **68.6 µs** |
---
## h5bench-Equivalent I/O Benchmarks
Criterion harness mirroring h5bench serial workloads. clawhdf5 benchmarks dated 2026-07-01;