feat(format): bitshuffle filter (32008), with its LZ4 and Zstandard modes

hdf5plugin's Bitshuffle failed with UnsupportedFilter(32008). The new
`bitshuffle` feature (pure Rust: lz4_flex, and ruzstd for Zstandard — the
`zstd` feature's libzstd is not needed) decodes all three modes of
bshuf_h5filter.c — transpose only, LZ4 and Zstandard blocks behind the
12-byte header — including the default and explicit block sizes, the
shorter last block rounded down to a multiple of 8 elements, and the
untransposed trailing elements. Sizes read from the chunk are bounded by
the chunk size.

It also encodes: DatasetBuilder::with_bitshuffle(BitshuffleCompression)
or PluginFilter::Bitshuffle { block_size, compression } writes the filter
with hdf5plugin's cd_values and no automatic byte shuffle. ruzstd has one
compression level (about zstd's 1); the requested level is recorded.

The bit transpose is checked bit for bit against a one-bit-at-a-time model
(which matched hdf5plugin's output) and is shared with blosc next.
Interop: hdf5plugin writes none/LZ4/Zstandard at default and explicit
block sizes and levels over the 12-case matrix, read byte for byte; our
three modes at two block sizes read back through hdf5plugin. Both fail
with the decoder removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 00:04:05 -05:00
co-authored by Claude Opus 5.5
parent e38f9123db
commit 07094e34a9
8 changed files with 509 additions and 3 deletions
+5
View File
@@ -22,6 +22,9 @@ zstd = { version = "0.13", optional = true }
blake3 = { version = "1", optional = true }
libaec-sys = { path = "../libaec-sys", version = "0.1", optional = true }
pco = { version = "1.0", optional = true }
# Pure-Rust Zstandard, for the plugin filters that embed zstd (bitshuffle,
# blosc). The `zstd` feature (filter 32015) links libzstd instead.
ruzstd = { version = "0.9", optional = true }
[dev-dependencies]
half = { workspace = true }
@@ -59,6 +62,8 @@ pcodec = ["dep:pco"]
# Plugin filters, pure Rust. LZF (32000) is h5py's built-in compression; it
# has no dependencies, so it is on by default.
lzf = []
# Bitshuffle (32008), with its LZ4 and Zstandard modes.
bitshuffle = ["lz4_flex", "ruzstd"]
[[bench]]
name = "parallel_decompress_bench"