The core crates (clawhdf5, -agent, -format, -io, -filters, -ann, -accel,
-netcdf4, -cli) now build no C by default: deflate defaults to zlib-rs,
a pure-Rust port of zlib-ng, and zlib-ng becomes the opt-in
`fast-deflate`, which overrides zlib-rs wherever it is enabled. A default
build no longer needs cmake or a C compiler.
Measured on tank, both builds run alternately, three rounds, medians:
zlib-rs is within 6% of zlib-ng on every HDF5 read and write (512x512
deflate-6 chunked write 1.458 vs 1.484 ms; 64 MB compressed read 64.4
vs 65.2 ms), and compressed output is byte-identical. Details in
BENCHMARKS.md, "Deflate backend".
Getting there took two fixes the first measurement exposed:
- zlib-rs needs `std` to detect SIMD at runtime. flate2 enables it via
its default `runtime_detection`, which `default-features = false` had
switched off, leaving zlib-rs 3.5x slower on inflate. The `zlib-rs`
features now enable it.
- Both deflate paths streamed through flate2's 32 KiB read/write
wrappers. They now hand the codec the whole chunk in one call, into a
buffer sized up front (~5% on chunked writes). This also fixes a
silent short read: the streaming reader returned a truncated stream's
bytes without an error; a truncated chunk is now DecompressionError.
In clawhdf5-filters, output longer than the stated size is now an
error rather than silently cut off.
CI: ci-test.sh lints and tests the zlib-ng path, and fails if a
C-building crate (*-sys, cc, cmake) enters a core crate's default
dependency tree. The arm64 job no longer installs cmake.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
decompress_chunk() already threaded chunk_size (the pipeline's declared
decompressed size) into the scale-offset/nbit/szip decoders to bound their
output, but not into deflate/lz4/zstd/pcodec, all four of which allocated
based on attacker-controlled input with no cap:
- lz4: read a raw u32 "orig_size" straight from the compressed payload's
first 4 bytes and passed it directly to lz4_flex::block::decompress with
no upper bound — a 4-byte attacker-controlled field could request ~4 GiB.
- deflate (non-macOS path): unbounded flate2 read_to_end into a fresh Vec.
- zstd: zstd::decode_all with no output cap (classic decompression-bomb
vector, ratios can exceed 1000:1).
- pcodec: simple_decompress with no cap.
All four now take the expected chunk size and reject output that exceeds it
(or a 256 MiB absolute ceiling when the size is unavailable), matching the
pattern the other three filters already used. Also fixes the same unbounded
read_to_end in clawhdf5-filters' fast_deflate streaming fallback (used when
no size hint is available).
Added tests for each codec plus one exercising the actually-exploited path
through the public decompress_chunk() entrypoint.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
