quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

build: pure-Rust zlib-rs as the default deflate backend

Browse Source 
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.

Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".

Getting there took two fixes the first measurement exposed:

- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
  its default `runtime_detection`, which `default-features = false` had
  switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
  features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
  wrappers. They now hand the codec the whole chunk in one call, into a
  buffer sized up front (~5% on chunked writes). This also fixes a
  silent short read: the streaming reader returned a truncated stream's
  bytes without an error; a truncated chunk is now DecompressionError.
  In clawhdf5-filters, output longer than the stated size is now an
  error rather than silently cut off.

CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-23 11:05:22 -05:00
co-authored by Claude Opus 5.5
parent e9b37a9602
commit 735db117a7
13 changed files with 471 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-filters/Cargo.toml
+6 -2
View File
@@ -25,8 +25,12 @@ name = "compression_bench"
harness = false
[features]
default = ["fast-deflate"]
# Pure-Rust zlib-rs by default; `fast-deflate` (zlib-ng, C) overrides it.
default = ["zlib-rs"]
fast-deflate = ["flate2/zlib-ng"]
system-zlib = ["flate2/zlib-default"]
zlib-rs = ["flate2/zlib-rs"]
# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
# SIMD at runtime. flate2 enables it by default, but we build flate2 with
# default-features = false, and without it zlib-rs inflates 3.5x slower.
zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
apple-compression = []
crates/clawhdf5-filters/README.md
+6 -4
View File
@@ -8,16 +8,18 @@ Filter and compression pipeline for clawhdf5.
## Features
- DEFLATE compression/decompression
- Fast deflate via zlib-ng (`fast-deflate` feature)
- Pure-Rust deflate via zlib-rs (default, `zlib-rs` feature)
- zlib-ng instead, if you want it (`fast-deflate` feature; C, needs cmake)
- Apple Compression framework support (`apple-compression` feature)
## Usage
```rust
use clawhdf5_filters::{deflate_decode, deflate_encode};
use clawhdf5_filters::{deflate_compress, deflate_decompress};
let compressed = deflate_encode(&data, 6).unwrap();
let decompressed = deflate_decode(&compressed).unwrap();
let compressed = deflate_compress(&data, 6).unwrap();
// The second argument bounds the output: the expected decompressed size.
let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
```
## License
crates/clawhdf5-filters/src/fast_deflate.rs
+114 -51
View File
@@ -1,12 +1,13 @@
//! Fast deflate backends: Apple Compression Framework and zlib-ng.
//! Deflate backends: Apple Compression Framework, zlib-ng and zlib-rs.
//!
//! Backend selection priority (decompression & compression):
//! 1. Apple Compression Framework (macOS only, `apple-compression` feature)
//! 2. flate2 with zlib-ng backend (`fast-deflate` feature) or miniz_oxide (default)
//! 2. flate2 with zlib-ng (`fast-deflate`), else zlib-rs (`zlib-rs`, the
//! default), else miniz_oxide
//!
//! The Apple Compression Framework uses hardware-accelerated zlib on Apple Silicon
//! and is typically the fastest option on macOS. zlib-ng is the fastest portable
//! option and what C HDF5 uses internally.
//! and is typically the fastest option on macOS. zlib-rs is a pure-Rust port of
//! zlib-ng; see `BENCHMARKS.md` for how the two compare.
// ---------------------------------------------------------------------------
// Apple Compression Framework FFI (macOS only)
@@ -243,65 +244,117 @@ mod apple {
}
// ---------------------------------------------------------------------------
// Streaming decompression via flate2 (uses zlib-ng when fast-deflate enabled)
// One-shot (de)compression via flate2 (whichever backend flate2 was built with)
//
// The whole input goes to the codec in one call, into an output buffer sized
// up front. `flate2::read::ZlibDecoder` / `write::ZlibEncoder` stream through a
// 32 KiB buffer instead, which cost zlib-rs up to 3.7x against zlib-ng on a
// 1 MB chunk. clawhdf5-format's deflate filter does the same; see
// `BENCHMARKS.md`, "Deflate backend".
// ---------------------------------------------------------------------------
/// Streaming decompress with pre-allocated output buffer.
///
/// When the output size is known (typical for HDF5 chunks), this avoids
/// dynamic reallocation by writing directly into a pre-sized buffer.
/// Decompress into a buffer pre-sized to `output_size`, the expected
/// decompressed length (known for HDF5 chunks). Output longer than that is an
/// error, as is a stream that ends early.
pub(crate) fn flate2_decompress_preallocated(
data: &[u8],
output_size: usize,
) -> Result<Vec<u8>, String> {
use std::io::Read;
let mut decoder = flate2::read::ZlibDecoder::new(data);
let mut output = vec![0u8; output_size];
let mut total_read = 0;
loop {
match decoder.read(&mut output[total_read..]) {
Ok(0) => break,
Ok(n) => total_read += n,
Err(e) => return Err(e.to_string()),
}
}
output.truncate(total_read);
Ok(output)
inflate_bounded(data, output_size, output_size)
}
/// Absolute ceiling on decompressed output when the caller has no size hint,
/// preventing unbounded allocation from a hostile/corrupted zlib stream.
const MAX_DECOMPRESS_SIZE: usize = 256 * 1024 * 1024;
/// Streaming decompress with dynamic sizing (when output size is unknown).
///
/// Bounded by [`MAX_DECOMPRESS_SIZE`] since there is no chunk-size hint to
/// validate against here — an unbounded `read_to_end` would let a hostile
/// zlib stream force arbitrarily large allocation (a "zlib bomb").
/// Decompress with no size hint, bounded by [`MAX_DECOMPRESS_SIZE`] so a
/// hostile zlib stream cannot force arbitrarily large allocation (a "zlib
/// bomb").
pub(crate) fn flate2_decompress_streaming(data: &[u8]) -> Result<Vec<u8>, String> {
use std::io::Read;
let decoder = flate2::read::ZlibDecoder::new(data);
let mut result = Vec::new();
decoder
.take(MAX_DECOMPRESS_SIZE as u64 + 1)
.read_to_end(&mut result)
.map_err(|e| e.to_string())?;
if result.len() > MAX_DECOMPRESS_SIZE {
return Err(format!(
"decompressed output exceeds {} MiB limit",
MAX_DECOMPRESS_SIZE / 1024 / 1024
));
}
Ok(result)
let hint = data.len().saturating_mul(4).min(1 << 20);
inflate_bounded(data, hint, MAX_DECOMPRESS_SIZE).map_err(|e| {
if e.ends_with("exceeds size limit") {
format!(
"decompressed output exceeds {} MiB limit",
MAX_DECOMPRESS_SIZE / 1024 / 1024
)
} else {
e
}
})
}
/// Compress data using flate2 (zlib-ng when fast-deflate enabled, else miniz_oxide).
/// Inflate a zlib stream, starting from `size_hint` bytes of output and
/// failing past `limit`.
fn inflate_bounded(data: &[u8], size_hint: usize, limit: usize) -> Result<Vec<u8>, String> {
use flate2::{Decompress, FlushDecompress, Status};
// One byte of headroom past the limit distinguishes an over-size stream
// from one that legitimately ends exactly at the limit.
let max_capacity = limit.saturating_add(1);
let mut out = Vec::new();
out.try_reserve_exact(size_hint.clamp(1, max_capacity))
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
let mut inflater = Decompress::new(true);
loop {
let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
let status = inflater
.decompress_vec(
&data[in_before as usize..],
&mut out,
FlushDecompress::Finish,
)
.map_err(|e| format!("deflate: {e}"))?;
if out.len() > limit {
return Err("deflate: output exceeds size limit".into());
}
match status {
Status::StreamEnd => return Ok(out),
Status::Ok | Status::BufError if out.len() == out.capacity() => {
let grow = out.capacity().min(max_capacity - out.capacity()).max(1);
out.try_reserve_exact(grow)
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
}
Status::Ok | Status::BufError => {
if inflater.total_in() as usize >= data.len()
|| (inflater.total_in(), inflater.total_out()) == (in_before, out_before)
{
return Err("deflate: truncated stream".into());
}
}
}
}
}
/// Compress data using flate2 (zlib-ng, zlib-rs or miniz_oxide; see module docs).
pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
use std::io::Write;
let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(level));
encoder.write_all(data).map_err(|e| e.to_string())?;
encoder.finish().map_err(|e| e.to_string())
use flate2::{Compress, Compression, FlushCompress, Status};
// zlib's compressBound, plus the zlib header and trailer.
let bound = data.len() + (data.len() >> 12) + (data.len() >> 14) + (data.len() >> 25) + 13 + 6;
let mut out = Vec::new();
out.try_reserve_exact(bound)
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
let mut deflater = Compress::new(Compression::new(level), true);
loop {
let (in_before, out_before) = (deflater.total_in(), deflater.total_out());
let status = deflater
.compress_vec(&data[in_before as usize..], &mut out, FlushCompress::Finish)
.map_err(|e| format!("deflate: {e}"))?;
match status {
Status::StreamEnd => return Ok(out),
Status::Ok | Status::BufError if out.len() == out.capacity() => out
.try_reserve(out.capacity().max(4096))
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?,
Status::Ok | Status::BufError => {
if (deflater.total_in(), deflater.total_out()) == (in_before, out_before) {
return Err("deflate: encoder made no progress".into());
}
}
}
}
}
// ---------------------------------------------------------------------------
@@ -312,7 +365,7 @@ pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String
///
/// Selection order:
/// 1. Apple Compression Framework (macOS + `apple-compression` feature)
/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
///
/// When `output_hint` > 0, pre-allocates the output buffer for zero-copy
/// decompression (avoids reallocation).
@@ -344,7 +397,7 @@ pub fn decompress(data: &[u8], output_hint: usize) -> Result<Vec<u8>, String> {
///
/// Selection order:
/// 1. Apple Compression Framework (macOS + `apple-compression` feature)
/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
pub fn compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
#[cfg(all(target_os = "macos", feature = "apple-compression"))]
{
@@ -377,9 +430,19 @@ pub fn active_backend() -> &'static str {
{
"zlib-ng"
}
// flate2 prefers a C zlib over zlib-rs when both are enabled.
#[cfg(all(
not(all(target_os = "macos", feature = "apple-compression")),
not(feature = "fast-deflate"),
feature = "zlib-rs"
))]
{
"zlib-rs"
}
#[cfg(not(any(
all(target_os = "macos", feature = "apple-compression"),
feature = "fast-deflate"
feature = "fast-deflate",
feature = "zlib-rs"
)))]
{
"miniz_oxide"
@@ -436,7 +499,7 @@ mod tests {
fn backend_name_is_set() {
let name = active_backend();
assert!(
["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
"unexpected backend: {name}"
);
}
crates/clawhdf5-filters/src/lib.rs
+6 -4
View File
@@ -2,12 +2,14 @@
//!
//! Provides deflate (zlib) decompression/compression with multiple backend options:
//!
//! - **Default**: `miniz_oxide` (pure Rust, no C dependencies)
//! - **`fast-deflate` feature**: `zlib-ng` via flate2 (~2-3x faster, matches C HDF5)
//! - **Default (`zlib-rs` feature)**: `zlib-rs` via flate2 (pure Rust, no C
//! dependencies)
//! - **`fast-deflate` feature**: `zlib-ng` via flate2 (C, built with cmake)
//! - **`apple-compression` feature**: Apple Compression Framework on macOS
//! (hardware-accelerated on Apple Silicon)
//! - With none of the above: `miniz_oxide` (pure Rust, slower)
//!
//! Backend priority: apple-compression > zlib-ng > miniz_oxide.
//! Backend priority: apple-compression > zlib-ng > zlib-rs > miniz_oxide.
pub mod fast_deflate;
@@ -115,7 +117,7 @@ mod tests {
fn backend_reports_name() {
let name = deflate_backend();
assert!(
["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
"unexpected backend: {name}"
);
}
crates/clawhdf5-format/Cargo.toml
+8 -2
View File
@@ -32,7 +32,10 @@ name = "bench"
harness = false
[features]
default = ["std", "checksum", "deflate", "provenance", "fast-deflate", "system-zlib-decompress"]
# Deflate backend: `zlib-rs` (pure Rust) by default. `fast-deflate` selects
# zlib-ng instead (C, built with cmake); flate2 prefers a C zlib whenever one
# is enabled, so turning it on anywhere in the build overrides the default.
default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress"]
std = []
checksum = []
deflate = ["flate2"]
@@ -42,7 +45,10 @@ fast-checksum = ["crc32fast"]
fast-deflate = ["flate2/zlib-ng"]
system-zlib = ["flate2/zlib-default"]
system-zlib-decompress = []
zlib-rs = ["flate2/zlib-rs"]
# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
# SIMD at runtime. flate2 enables it by default, but we build flate2 with
# default-features = false, and without it zlib-rs inflates 3.5x slower.
zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
lz4 = ["lz4_flex"]
zstd = ["dep:zstd"]
blake3_hash = ["blake3"]
crates/clawhdf5-format/src/filters.rs
+165 -21
View File
@@ -629,21 +629,70 @@ fn deflate_decompress(data: &[u8], expected_bytes: usize) -> Result<Vec<u8>, For
// Fall through to flate2 on error
}
use std::io::Read;
let decoder = flate2::read::ZlibDecoder::new(data);
let mut result = Vec::with_capacity(limit.min(1 << 20));
// Read one byte past the limit so an over-size stream is distinguishable
// A chunk's decompressed size is known, so allocate it once; without one,
// start from a multiple of the input and grow.
let size_hint = if expected_bytes != 0 {
expected_bytes
} else {
data.len().saturating_mul(4).min(1 << 20)
};
inflate_bounded(data, size_hint, limit).map_err(FormatError::DecompressionError)
}
/// Inflate a zlib stream into a buffer sized up front, handing the decoder the
/// whole input at once.
///
/// `flate2::read::ZlibDecoder` feeds its input through a 32 KiB buffer and
/// grows the output as it goes; on single chunks that cost zlib-rs up to 3.7x
/// against zlib-ng (`BENCHMARKS.md`, "Deflate backend"). Output beyond `limit`
/// is an error, as is a stream that ends before its end-of-stream marker (the
/// streaming reader returned the bytes it had and no error).
#[cfg(feature = "deflate")]
pub(crate) fn inflate_bounded(
data: &[u8],
size_hint: usize,
limit: usize,
) -> Result<Vec<u8>, String> {
use flate2::{Decompress, FlushDecompress, Status};
// One byte of headroom past the limit distinguishes an over-size stream
// from one that legitimately ends exactly at the limit.
decoder
.take(limit as u64 + 1)
.read_to_end(&mut result)
.map_err(|e| FormatError::DecompressionError(e.to_string()))?;
if result.len() > limit {
return Err(FormatError::DecompressionError(
"deflate: output exceeds size limit".into(),
));
let max_capacity = limit.saturating_add(1);
let mut out = Vec::new();
out.try_reserve_exact(size_hint.clamp(1, max_capacity))
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
let mut inflater = Decompress::new(true);
loop {
let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
let status = inflater
.decompress_vec(
&data[in_before as usize..],
&mut out,
FlushDecompress::Finish,
)
.map_err(|e| format!("deflate: {e}"))?;
if out.len() > limit {
return Err("deflate: output exceeds size limit".into());
}
match status {
Status::StreamEnd => return Ok(out),
Status::Ok | Status::BufError if out.len() == out.capacity() => {
// Out of room: double, up to the limit.
let grow = out.capacity().min(max_capacity - out.capacity()).max(1);
out.try_reserve_exact(grow)
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
}
Status::Ok | Status::BufError => {
// Room left, so the decoder stopped for want of input.
if inflater.total_in() as usize >= data.len()
|| (inflater.total_in(), inflater.total_out()) == (in_before, out_before)
{
return Err("deflate: truncated stream".into());
}
}
}
}
Ok(result)
}
/// Direct FFI to Apple's system libz for fast decompression.
@@ -722,14 +771,41 @@ fn deflate_decompress(_data: &[u8], _expected_bytes: usize) -> Result<Vec<u8>, F
/// Compress data with zlib.
#[cfg(feature = "deflate")]
fn deflate_compress(data: &[u8], level: u32) -> Result<Vec<u8>, FormatError> {
use std::io::Write;
let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(level));
encoder
.write_all(data)
.map_err(|e| FormatError::CompressionError(e.to_string()))?;
encoder
.finish()
.map_err(|e| FormatError::CompressionError(e.to_string()))
deflate_bounded(data, level).map_err(FormatError::CompressionError)
}
/// Deflate `data` into a zlib stream in one pass, into a buffer sized for the
/// worst case up front (the same reasoning as [`inflate_bounded`]).
#[cfg(feature = "deflate")]
pub(crate) fn deflate_bounded(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
use flate2::{Compress, Compression, FlushCompress, Status};
// zlib's compressBound, plus the zlib header and trailer.
let bound = data.len() + (data.len() >> 12) + (data.len() >> 14) + (data.len() >> 25) + 13 + 6;
let mut out = Vec::new();
out.try_reserve_exact(bound)
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
let mut deflater = Compress::new(Compression::new(level), true);
loop {
let (in_before, out_before) = (deflater.total_in(), deflater.total_out());
let status = deflater
.compress_vec(&data[in_before as usize..], &mut out, FlushCompress::Finish)
.map_err(|e| format!("deflate: {e}"))?;
match status {
Status::StreamEnd => return Ok(out),
// The bound should make running out of room unreachable; grow
// rather than fail if it happens.
Status::Ok | Status::BufError if out.len() == out.capacity() => out
.try_reserve(out.capacity().max(4096))
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?,
Status::Ok | Status::BufError => {
if (deflater.total_in(), deflater.total_out()) == (in_before, out_before) {
return Err("deflate: encoder made no progress".into());
}
}
}
}
}
#[cfg(not(feature = "deflate"))]
@@ -1833,6 +1909,74 @@ mod tests {
assert!(deflate_decompress(&compressed, 64).is_err());
}
#[cfg(feature = "deflate")]
fn noisy_bytes(n: usize) -> Vec<u8> {
// Compressible but not trivially so.
(0..n)
.map(|i| ((i as f64 * 0.01).sin() * 127.0 + 128.0) as u8 ^ (i as u8 & 3))
.collect()
}
#[test]
#[cfg(feature = "deflate")]
fn deflate_decompress_accepts_output_exactly_at_chunk_size() {
let data = noisy_bytes(100_000);
let compressed = deflate_compress(&data, 6).unwrap();
assert_eq!(deflate_decompress(&compressed, data.len()).unwrap(), data);
// One byte short of the real size is over the limit.
assert!(deflate_decompress(&compressed, data.len() - 1).is_err());
}
#[test]
#[cfg(feature = "deflate")]
fn deflate_decompress_without_size_grows_the_buffer() {
// No chunk size: the output starts at 4x the input and has to grow.
let data = vec![7u8; 3 * 1024 * 1024];
let compressed = deflate_compress(&data, 6).unwrap();
assert!(compressed.len() * 4 < data.len());
assert_eq!(deflate_decompress(&compressed, 0).unwrap(), data);
}
#[test]
#[cfg(feature = "deflate")]
fn deflate_decompress_rejects_truncated_stream() {
// The streaming reader this replaced returned the bytes it had and no
// error, so a truncated chunk read back short.
let data = noisy_bytes(100_000);
let compressed = deflate_compress(&data, 6).unwrap();
for cut in [compressed.len() - 1, compressed.len() / 2, 3] {
assert!(
deflate_decompress(&compressed[..cut], data.len()).is_err(),
"truncated to {cut} of {} bytes",
compressed.len()
);
}
}
#[test]
#[cfg(feature = "deflate")]
fn deflate_compress_roundtrips_incompressible_data() {
// Random-looking input compresses to slightly more than it started
// as; the output must still fit the pre-sized buffer (or grow).
let mut x = 0x9E37_79B9_7F4A_7C15u64;
let data: Vec<u8> = (0..200_000)
.map(|_| {
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
x as u8
})
.collect();
for level in [0, 1, 6, 9] {
let compressed = deflate_compress(&data, level).unwrap();
assert_eq!(deflate_decompress(&compressed, data.len()).unwrap(), data);
}
assert_eq!(
deflate_decompress(&deflate_compress(&[], 6).unwrap(), 0).unwrap(),
Vec::<u8>::new()
);
}
#[test]
#[cfg(feature = "zstd")]
fn zstd_decompress_rejects_output_exceeding_chunk_size() {
crates/clawhdf5/Cargo.toml
+2 -1
View File
@@ -30,9 +30,10 @@ name = "parallel_bench"
harness = false
[features]
default = ["mmap", "fast-deflate", "provenance"]
default = ["mmap", "provenance"]
mmap = ["clawhdf5-io/mmap"]
parallel = ["clawhdf5-format/parallel", "rayon"]
# zlib-ng (C, needs cmake) instead of the default pure-Rust zlib-rs.
fast-deflate = ["clawhdf5-format/fast-deflate"]
apple-compression = []
zstd = ["clawhdf5-format/zstd"]