feat(format): LZF filter (32000), read and write, pure Rust
h5py's built-in compression="lzf" failed with UnsupportedFilter(32000). The new `lzf` feature (no dependencies, on by default in clawhdf5-format and the facade) decodes the raw liblzf stream h5py's filter stores, bounded by the chunk size, and encodes it: DatasetBuilder::with_lzf() (or with_plugin_filter(PluginFilter::Lzf)) writes the filter with h5py's cd_values (filter version 4, liblzf 0x0105, chunk size in bytes), flagged optional as h5py does. ChunkOptions gains a `plugin` field for the plugin filters; build_pipeline_for_chunk passes the chunk size to filters that record it. tests/plugin_filters_interop.rs: h5py writes LZF (alone, with shuffle, with shuffle+fletcher32) over 12 dtype/shape/chunk/data cases with partial edge chunks and incompressible data, and every dataset reads byte for byte equal to its unfiltered twin; our LZF output (1-D and 2-D, edge chunks, with and without shuffle) reads back in h5py. Both fail with the decoder removed. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -37,7 +37,7 @@ harness = false
|
||||
# Deflate backend: `zlib-rs` (pure Rust) by default. `fast-deflate` selects
|
||||
# zlib-ng instead (C, built with cmake); flate2 prefers a C zlib whenever one
|
||||
# is enabled, so turning it on anywhere in the build overrides the default.
|
||||
default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress"]
|
||||
default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress", "lzf"]
|
||||
std = []
|
||||
checksum = []
|
||||
deflate = ["flate2"]
|
||||
@@ -56,6 +56,9 @@ zstd = ["dep:zstd"]
|
||||
blake3_hash = ["blake3"]
|
||||
szip = ["libaec-sys"]
|
||||
pcodec = ["dep:pco"]
|
||||
# Plugin filters, pure Rust. LZF (32000) is h5py's built-in compression; it
|
||||
# has no dependencies, so it is on by default.
|
||||
lzf = []
|
||||
|
||||
[[bench]]
|
||||
name = "parallel_decompress_bench"
|
||||
|
||||
Reference in New Issue
Block a user