build: pure-Rust zlib-rs as the default deflate backend
The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel, -netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs, a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in `fast-deflate`, which overrides zlib-rs wherever it is enabled. A default build no longer needs cmake or a C compiler. Measured on tank, both builds run alternately, three rounds, medians: zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512 deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4 vs 65.2 ms), and compressed output is byte-identical. Details in BENCHMARKS.md, "Deflate backend". Getting there took two fixes the first measurement exposed: - zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via its default `runtime_detection`, which `default-features = false` had switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs` features now enable it. - Both deflate paths streamed through flate2's 32 KiB read/write wrappers. They now hand the codec the whole chunk in one call, into a buffer sized up front (~5% on chunked writes). This also fixes a silent short read: the streaming reader returned a truncated stream's bytes without an error; a truncated chunk is now DecompressionError. In clawhdf5-filters, output longer than the stated size is now an error rather than silently cut off. CI: ci-test.sh lints and tests the zlib-ng path, and fails if a C-building crate (*-sys, cc, cmake) enters a core crate's default dependency tree. The arm64 job no longer installs cmake. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -25,8 +25,12 @@ name = "compression_bench"
|
||||
harness = false
|
||||
|
||||
[features]
|
||||
default = ["fast-deflate"]
|
||||
# Pure-Rust zlib-rs by default; `fast-deflate` (zlib-ng, C) overrides it.
|
||||
default = ["zlib-rs"]
|
||||
fast-deflate = ["flate2/zlib-ng"]
|
||||
system-zlib = ["flate2/zlib-default"]
|
||||
zlib-rs = ["flate2/zlib-rs"]
|
||||
# `runtime_detection` gives zlib-rs `std`, which it needs to detect and use
|
||||
# SIMD at runtime. flate2 enables it by default, but we build flate2 with
|
||||
# default-features = false, and without it zlib-rs inflates 3.5x slower.
|
||||
zlib-rs = ["flate2/zlib-rs", "flate2/runtime_detection"]
|
||||
apple-compression = []
|
||||
|
||||
@@ -8,16 +8,18 @@ Filter and compression pipeline for clawhdf5.
|
||||
## Features
|
||||
|
||||
- DEFLATE compression/decompression
|
||||
- Fast deflate via zlib-ng (`fast-deflate` feature)
|
||||
- Pure-Rust deflate via zlib-rs (default, `zlib-rs` feature)
|
||||
- zlib-ng instead, if you want it (`fast-deflate` feature; C, needs cmake)
|
||||
- Apple Compression framework support (`apple-compression` feature)
|
||||
|
||||
## Usage
|
||||
|
||||
```rust
|
||||
use clawhdf5_filters::{deflate_decode, deflate_encode};
|
||||
use clawhdf5_filters::{deflate_compress, deflate_decompress};
|
||||
|
||||
let compressed = deflate_encode(&data, 6).unwrap();
|
||||
let decompressed = deflate_decode(&compressed).unwrap();
|
||||
let compressed = deflate_compress(&data, 6).unwrap();
|
||||
// The second argument bounds the output: the expected decompressed size.
|
||||
let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
//! Fast deflate backends: Apple Compression Framework and zlib-ng.
|
||||
//! Deflate backends: Apple Compression Framework, zlib-ng and zlib-rs.
|
||||
//!
|
||||
//! Backend selection priority (decompression & compression):
|
||||
//! 1. Apple Compression Framework (macOS only, `apple-compression` feature)
|
||||
//! 2. flate2 with zlib-ng backend (`fast-deflate` feature) or miniz_oxide (default)
|
||||
//! 2. flate2 with zlib-ng (`fast-deflate`), else zlib-rs (`zlib-rs`, the
|
||||
//! default), else miniz_oxide
|
||||
//!
|
||||
//! The Apple Compression Framework uses hardware-accelerated zlib on Apple Silicon
|
||||
//! and is typically the fastest option on macOS. zlib-ng is the fastest portable
|
||||
//! option and what C HDF5 uses internally.
|
||||
//! and is typically the fastest option on macOS. zlib-rs is a pure-Rust port of
|
||||
//! zlib-ng; see `BENCHMARKS.md` for how the two compare.
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Apple Compression Framework FFI (macOS only)
|
||||
@@ -243,65 +244,117 @@ mod apple {
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Streaming decompression via flate2 (uses zlib-ng when fast-deflate enabled)
|
||||
// One-shot (de)compression via flate2 (whichever backend flate2 was built with)
|
||||
//
|
||||
// The whole input goes to the codec in one call, into an output buffer sized
|
||||
// up front. `flate2::read::ZlibDecoder` / `write::ZlibEncoder` stream through a
|
||||
// 32 KiB buffer instead, which cost zlib-rs up to 3.7x against zlib-ng on a
|
||||
// 1 MB chunk. clawhdf5-format's deflate filter does the same; see
|
||||
// `BENCHMARKS.md`, "Deflate backend".
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Streaming decompress with pre-allocated output buffer.
|
||||
///
|
||||
/// When the output size is known (typical for HDF5 chunks), this avoids
|
||||
/// dynamic reallocation by writing directly into a pre-sized buffer.
|
||||
/// Decompress into a buffer pre-sized to `output_size`, the expected
|
||||
/// decompressed length (known for HDF5 chunks). Output longer than that is an
|
||||
/// error, as is a stream that ends early.
|
||||
pub(crate) fn flate2_decompress_preallocated(
|
||||
data: &[u8],
|
||||
output_size: usize,
|
||||
) -> Result<Vec<u8>, String> {
|
||||
use std::io::Read;
|
||||
let mut decoder = flate2::read::ZlibDecoder::new(data);
|
||||
let mut output = vec![0u8; output_size];
|
||||
let mut total_read = 0;
|
||||
|
||||
loop {
|
||||
match decoder.read(&mut output[total_read..]) {
|
||||
Ok(0) => break,
|
||||
Ok(n) => total_read += n,
|
||||
Err(e) => return Err(e.to_string()),
|
||||
}
|
||||
}
|
||||
output.truncate(total_read);
|
||||
Ok(output)
|
||||
inflate_bounded(data, output_size, output_size)
|
||||
}
|
||||
|
||||
/// Absolute ceiling on decompressed output when the caller has no size hint,
|
||||
/// preventing unbounded allocation from a hostile/corrupted zlib stream.
|
||||
const MAX_DECOMPRESS_SIZE: usize = 256 * 1024 * 1024;
|
||||
|
||||
/// Streaming decompress with dynamic sizing (when output size is unknown).
|
||||
///
|
||||
/// Bounded by [`MAX_DECOMPRESS_SIZE`] since there is no chunk-size hint to
|
||||
/// validate against here — an unbounded `read_to_end` would let a hostile
|
||||
/// zlib stream force arbitrarily large allocation (a "zlib bomb").
|
||||
/// Decompress with no size hint, bounded by [`MAX_DECOMPRESS_SIZE`] so a
|
||||
/// hostile zlib stream cannot force arbitrarily large allocation (a "zlib
|
||||
/// bomb").
|
||||
pub(crate) fn flate2_decompress_streaming(data: &[u8]) -> Result<Vec<u8>, String> {
|
||||
use std::io::Read;
|
||||
let decoder = flate2::read::ZlibDecoder::new(data);
|
||||
let mut result = Vec::new();
|
||||
decoder
|
||||
.take(MAX_DECOMPRESS_SIZE as u64 + 1)
|
||||
.read_to_end(&mut result)
|
||||
.map_err(|e| e.to_string())?;
|
||||
if result.len() > MAX_DECOMPRESS_SIZE {
|
||||
return Err(format!(
|
||||
"decompressed output exceeds {} MiB limit",
|
||||
MAX_DECOMPRESS_SIZE / 1024 / 1024
|
||||
));
|
||||
}
|
||||
Ok(result)
|
||||
let hint = data.len().saturating_mul(4).min(1 << 20);
|
||||
inflate_bounded(data, hint, MAX_DECOMPRESS_SIZE).map_err(|e| {
|
||||
if e.ends_with("exceeds size limit") {
|
||||
format!(
|
||||
"decompressed output exceeds {} MiB limit",
|
||||
MAX_DECOMPRESS_SIZE / 1024 / 1024
|
||||
)
|
||||
} else {
|
||||
e
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// Compress data using flate2 (zlib-ng when fast-deflate enabled, else miniz_oxide).
|
||||
/// Inflate a zlib stream, starting from `size_hint` bytes of output and
|
||||
/// failing past `limit`.
|
||||
fn inflate_bounded(data: &[u8], size_hint: usize, limit: usize) -> Result<Vec<u8>, String> {
|
||||
use flate2::{Decompress, FlushDecompress, Status};
|
||||
|
||||
// One byte of headroom past the limit distinguishes an over-size stream
|
||||
// from one that legitimately ends exactly at the limit.
|
||||
let max_capacity = limit.saturating_add(1);
|
||||
let mut out = Vec::new();
|
||||
out.try_reserve_exact(size_hint.clamp(1, max_capacity))
|
||||
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
|
||||
|
||||
let mut inflater = Decompress::new(true);
|
||||
loop {
|
||||
let (in_before, out_before) = (inflater.total_in(), inflater.total_out());
|
||||
let status = inflater
|
||||
.decompress_vec(
|
||||
&data[in_before as usize..],
|
||||
&mut out,
|
||||
FlushDecompress::Finish,
|
||||
)
|
||||
.map_err(|e| format!("deflate: {e}"))?;
|
||||
if out.len() > limit {
|
||||
return Err("deflate: output exceeds size limit".into());
|
||||
}
|
||||
match status {
|
||||
Status::StreamEnd => return Ok(out),
|
||||
Status::Ok | Status::BufError if out.len() == out.capacity() => {
|
||||
let grow = out.capacity().min(max_capacity - out.capacity()).max(1);
|
||||
out.try_reserve_exact(grow)
|
||||
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
|
||||
}
|
||||
Status::Ok | Status::BufError => {
|
||||
if inflater.total_in() as usize >= data.len()
|
||||
|| (inflater.total_in(), inflater.total_out()) == (in_before, out_before)
|
||||
{
|
||||
return Err("deflate: truncated stream".into());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Compress data using flate2 (zlib-ng, zlib-rs or miniz_oxide; see module docs).
|
||||
pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
|
||||
use std::io::Write;
|
||||
let mut encoder = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::new(level));
|
||||
encoder.write_all(data).map_err(|e| e.to_string())?;
|
||||
encoder.finish().map_err(|e| e.to_string())
|
||||
use flate2::{Compress, Compression, FlushCompress, Status};
|
||||
|
||||
// zlib's compressBound, plus the zlib header and trailer.
|
||||
let bound = data.len() + (data.len() >> 12) + (data.len() >> 14) + (data.len() >> 25) + 13 + 6;
|
||||
let mut out = Vec::new();
|
||||
out.try_reserve_exact(bound)
|
||||
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?;
|
||||
|
||||
let mut deflater = Compress::new(Compression::new(level), true);
|
||||
loop {
|
||||
let (in_before, out_before) = (deflater.total_in(), deflater.total_out());
|
||||
let status = deflater
|
||||
.compress_vec(&data[in_before as usize..], &mut out, FlushCompress::Finish)
|
||||
.map_err(|e| format!("deflate: {e}"))?;
|
||||
match status {
|
||||
Status::StreamEnd => return Ok(out),
|
||||
Status::Ok | Status::BufError if out.len() == out.capacity() => out
|
||||
.try_reserve(out.capacity().max(4096))
|
||||
.map_err(|e| format!("deflate: cannot allocate output: {e}"))?,
|
||||
Status::Ok | Status::BufError => {
|
||||
if (deflater.total_in(), deflater.total_out()) == (in_before, out_before) {
|
||||
return Err("deflate: encoder made no progress".into());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -312,7 +365,7 @@ pub(crate) fn flate2_compress(data: &[u8], level: u32) -> Result<Vec<u8>, String
|
||||
///
|
||||
/// Selection order:
|
||||
/// 1. Apple Compression Framework (macOS + `apple-compression` feature)
|
||||
/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
|
||||
/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
|
||||
///
|
||||
/// When `output_hint` > 0, pre-allocates the output buffer for zero-copy
|
||||
/// decompression (avoids reallocation).
|
||||
@@ -344,7 +397,7 @@ pub fn decompress(data: &[u8], output_hint: usize) -> Result<Vec<u8>, String> {
|
||||
///
|
||||
/// Selection order:
|
||||
/// 1. Apple Compression Framework (macOS + `apple-compression` feature)
|
||||
/// 2. flate2 (zlib-ng with `fast-deflate`, otherwise miniz_oxide)
|
||||
/// 2. flate2 (zlib-ng with `fast-deflate`, else zlib-rs, else miniz_oxide)
|
||||
pub fn compress(data: &[u8], level: u32) -> Result<Vec<u8>, String> {
|
||||
#[cfg(all(target_os = "macos", feature = "apple-compression"))]
|
||||
{
|
||||
@@ -377,9 +430,19 @@ pub fn active_backend() -> &'static str {
|
||||
{
|
||||
"zlib-ng"
|
||||
}
|
||||
// flate2 prefers a C zlib over zlib-rs when both are enabled.
|
||||
#[cfg(all(
|
||||
not(all(target_os = "macos", feature = "apple-compression")),
|
||||
not(feature = "fast-deflate"),
|
||||
feature = "zlib-rs"
|
||||
))]
|
||||
{
|
||||
"zlib-rs"
|
||||
}
|
||||
#[cfg(not(any(
|
||||
all(target_os = "macos", feature = "apple-compression"),
|
||||
feature = "fast-deflate"
|
||||
feature = "fast-deflate",
|
||||
feature = "zlib-rs"
|
||||
)))]
|
||||
{
|
||||
"miniz_oxide"
|
||||
@@ -436,7 +499,7 @@ mod tests {
|
||||
fn backend_name_is_set() {
|
||||
let name = active_backend();
|
||||
assert!(
|
||||
["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
|
||||
["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
|
||||
"unexpected backend: {name}"
|
||||
);
|
||||
}
|
||||
|
||||
@@ -2,12 +2,14 @@
|
||||
//!
|
||||
//! Provides deflate (zlib) decompression/compression with multiple backend options:
|
||||
//!
|
||||
//! - **Default**: `miniz_oxide` (pure Rust, no C dependencies)
|
||||
//! - **`fast-deflate` feature**: `zlib-ng` via flate2 (~2-3x faster, matches C HDF5)
|
||||
//! - **Default (`zlib-rs` feature)**: `zlib-rs` via flate2 (pure Rust, no C
|
||||
//! dependencies)
|
||||
//! - **`fast-deflate` feature**: `zlib-ng` via flate2 (C, built with cmake)
|
||||
//! - **`apple-compression` feature**: Apple Compression Framework on macOS
|
||||
//! (hardware-accelerated on Apple Silicon)
|
||||
//! - With none of the above: `miniz_oxide` (pure Rust, slower)
|
||||
//!
|
||||
//! Backend priority: apple-compression > zlib-ng > miniz_oxide.
|
||||
//! Backend priority: apple-compression > zlib-ng > zlib-rs > miniz_oxide.
|
||||
|
||||
pub mod fast_deflate;
|
||||
|
||||
@@ -115,7 +117,7 @@ mod tests {
|
||||
fn backend_reports_name() {
|
||||
let name = deflate_backend();
|
||||
assert!(
|
||||
["miniz_oxide", "zlib-ng", "apple-compression"].contains(&name),
|
||||
["miniz_oxide", "zlib-rs", "zlib-ng", "apple-compression"].contains(&name),
|
||||
"unexpected backend: {name}"
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user