osobhandClaude Opus 5.5
5a20cf04e8
The writer gives a chunk of more than u32::MAX bytes layout message version 5 and, filtered, index elements whose stored size takes the file's size of lengths, as libhdf5 2.x does (the Fixed and Extensible Array structures match libhdf5's byte for byte). Chunk dimensions of 2^32 or more and filters that cannot take such a chunk (LZF, bitshuffle, bzip2, Blosc, pcodec) are refused instead of truncated. Chunks are extracted row by row and one at a time; deflate no longer cuts input at 4 GiB - 1 bytes, nor holds the worst-case bound of a large chunk; an LZ4 chunk of 4 GiB or more is read as the registered framing. FileEditor refuses writing values into, or pruning/allocating, chunks of 4 GiB or more before anything is written; growing the extent and attributes still work. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-filters
Standalone deflate (zlib) compression and decompression with a choice of backend: pure-Rust zlib-rs (default), zlib-ng, Apple's Compression framework, or miniz_oxide.
This crate holds deflate backends only. The HDF5 filter pipeline, the
filter registry and every other codec (shuffle, Fletcher-32, N-Bit,
scale-offset, LZ4, Zstd, SZIP, pcodec, LZF, bitshuffle, bzip2, Blosc,
Blosc2, ZFP) live in clawhdf5-format,
which calls flate2 itself and selects its deflate backend with its own
features. No library crate of the workspace depends on this one (the
clawhdf5 facade uses it only in tests).
Not on crates.io yet; depend on it from git:
[dependencies]
clawhdf5-filters = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
API
use clawhdf5_filters::{deflate_backend, deflate_compress, deflate_decompress};
let data: Vec<u8> = (0..10_000u32).map(|i| (i % 251) as u8).collect();
let compressed = deflate_compress(&data, 6).unwrap();
// The second argument bounds the output: the expected decompressed size.
let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
assert_eq!(decompressed, data);
println!("backend: {}", deflate_backend()); // "zlib-rs" by default
Also deflate_compress_miniz/deflate_decompress_miniz (always
miniz_oxide) and fast_deflate::{compress, decompress, active_backend}.
Features
Backend priority: apple-compression (macOS only) > zlib-ng > zlib-rs >
miniz_oxide (with none enabled).
| Feature | Default | Backend | Builds C |
|---|---|---|---|
zlib-rs |
yes | zlib-rs through flate2, with runtime_detection (needed for its SIMD) |
no |
fast-deflate |
no | zlib-ng through flate2 | yes (cmake) |
system-zlib |
no | the system zlib through flate2 | yes (libz-sys) |
apple-compression |
no | Apple Compression framework, macOS only (ignored elsewhere) | no (links a system framework) |
zlib-rs matches zlib-ng on HDF5 reads and writes and produces
byte-identical output: see "Deflate backend" in
BENCHMARKS.md.
License
MIT