osobhandClaude Opus 5.5
b55b24b7ba
Every crate under crates/ now has a README (android, bench, cli, napi and wasm had none), each saying what the crate is, its main types and functions (names checked against the code), its cargo features with defaults and which ones build C (checked with `cargo tree`), and links to the top-level docs. Corrections to the old stubs: - clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs clawhdf5-format as a dependency. - clawhdf5-filters: deflate backends only, and no library crate depends on it; the filter pipeline and every other codec are in -format. - clawhdf5-gpu: vector distance compute, not I/O; not used by HDF5Memory::search. - clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO. - clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and search(q, k, ef). - clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm backends are reported but run the scalar kernels. - clawhdf5-gpu: the old example called l2_distances, which does not exist (l2_search). - clawhdf5-agent: it described a "vector store" with "GPU acceleration"; it now covers HDF5Memory, search options, WAL, signing, the graph. - crates.io/docs.rs badges removed and `cargo install <crate>` replaced: nothing is published; depend on git. - fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh. - tools: the FileEditor interop tests that live in this crate. - remote, py: license, other front ends, limits, File.mode/flush/chunks. The Rust examples of the facade, format, filters, accel, ann, derive and agent READMEs were compiled and run as tests (netcdf4, gpu and remote compiled only) in a scratch crate; the CLI example was run. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2.1 KiB
clawhdf5-filters
Standalone deflate (zlib) compression and decompression with a choice of backend: pure-Rust zlib-rs (default), zlib-ng, Apple's Compression framework, or miniz_oxide.
This crate holds deflate backends only. The HDF5 filter pipeline, the
filter registry and every other codec (shuffle, Fletcher-32, N-Bit,
scale-offset, LZ4, Zstd, SZIP, pcodec, LZF, bitshuffle, bzip2, Blosc,
Blosc2, ZFP) live in clawhdf5-format,
which calls flate2 itself and selects its deflate backend with its own
features. No library crate of the workspace depends on this one (the
clawhdf5 facade uses it only in tests).
Not on crates.io yet; depend on it from git:
[dependencies]
clawhdf5-filters = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
API
use clawhdf5_filters::{deflate_backend, deflate_compress, deflate_decompress};
let data: Vec<u8> = (0..10_000u32).map(|i| (i % 251) as u8).collect();
let compressed = deflate_compress(&data, 6).unwrap();
// The second argument bounds the output: the expected decompressed size.
let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
assert_eq!(decompressed, data);
println!("backend: {}", deflate_backend()); // "zlib-rs" by default
Also deflate_compress_miniz/deflate_decompress_miniz (always
miniz_oxide) and fast_deflate::{compress, decompress, active_backend}.
Features
Backend priority: apple-compression (macOS only) > zlib-ng > zlib-rs >
miniz_oxide (with none enabled).
| Feature | Default | Backend | Builds C |
|---|---|---|---|
zlib-rs |
yes | zlib-rs through flate2, with runtime_detection (needed for its SIMD) |
no |
fast-deflate |
no | zlib-ng through flate2 | yes (cmake) |
system-zlib |
no | the system zlib through flate2 | yes (libz-sys) |
apple-compression |
no | Apple Compression framework, macOS only (ignored elsewhere) | no (links a system framework) |
zlib-rs matches zlib-ng on HDF5 reads and writes and produces
byte-identical output: see "Deflate backend" in
BENCHMARKS.md.
License
MIT