feat(format): LZF filter (32000), read and write, pure Rust

h5py's built-in compression="lzf" failed with UnsupportedFilter(32000). The
new `lzf` feature (no dependencies, on by default in clawhdf5-format and
the facade) decodes the raw liblzf stream h5py's filter stores, bounded by
the chunk size, and encodes it: DatasetBuilder::with_lzf() (or
with_plugin_filter(PluginFilter::Lzf)) writes the filter with h5py's
cd_values (filter version 4, liblzf 0x0105, chunk size in bytes), flagged
optional as h5py does. ChunkOptions gains a `plugin` field for the plugin
filters; build_pipeline_for_chunk passes the chunk size to filters that
record it.

tests/plugin_filters_interop.rs: h5py writes LZF (alone, with shuffle, with
shuffle+fletcher32) over 12 dtype/shape/chunk/data cases with partial edge
chunks and incompressible data, and every dataset reads byte for byte equal
to its unfiltered twin; our LZF output (1-D and 2-D, edge chunks, with and
without shuffle) reads back in h5py. Both fail with the decoder removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 00:01:22 -05:00
co-authored by Claude Opus 5.5
parent d16544b928
commit e38f9123db
9 changed files with 629 additions and 9 deletions
+4 -1
View File
@@ -37,7 +37,7 @@ harness = false
# Deflate backend: `zlib-rs` (pure Rust) by default. `fast-deflate` selects
# zlib-ng instead (C, built with cmake); flate2 prefers a C zlib whenever one
# is enabled, so turning it on anywhere in the build overrides the default.
default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress"]
default = ["std", "checksum", "deflate", "provenance", "zlib-rs", "system-zlib-decompress", "lzf"]
std = []
checksum = []
deflate = ["flate2"]
@@ -56,6 +56,9 @@ zstd = ["dep:zstd"]
blake3_hash = ["blake3"]
szip = ["libaec-sys"]
pcodec = ["dep:pco"]
# Plugin filters, pure Rust. LZF (32000) is h5py's built-in compression; it
# has no dependencies, so it is on by default.
lzf = []
[[bench]]
name = "parallel_decompress_bench"