feat(format): bitshuffle filter (32008), with its LZ4 and Zstandard modes
hdf5plugin's Bitshuffle failed with UnsupportedFilter(32008). The new
`bitshuffle` feature (pure Rust: lz4_flex, and ruzstd for Zstandard — the
`zstd` feature's libzstd is not needed) decodes all three modes of
bshuf_h5filter.c — transpose only, LZ4 and Zstandard blocks behind the
12-byte header — including the default and explicit block sizes, the
shorter last block rounded down to a multiple of 8 elements, and the
untransposed trailing elements. Sizes read from the chunk are bounded by
the chunk size.
It also encodes: DatasetBuilder::with_bitshuffle(BitshuffleCompression)
or PluginFilter::Bitshuffle { block_size, compression } writes the filter
with hdf5plugin's cd_values and no automatic byte shuffle. ruzstd has one
compression level (about zstd's 1); the requested level is recorded.
The bit transpose is checked bit for bit against a one-bit-at-a-time model
(which matched hdf5plugin's output) and is shared with blosc next.
Interop: hdf5plugin writes none/LZ4/Zstandard at default and explicit
block sizes and levels over the 12-case matrix, read byte for byte; our
three modes at two block sizes read back through hdf5plugin. Both fail
with the decoder removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -22,6 +22,9 @@ zstd = { version = "0.13", optional = true }
|
||||
blake3 = { version = "1", optional = true }
|
||||
libaec-sys = { path = "../libaec-sys", version = "0.1", optional = true }
|
||||
pco = { version = "1.0", optional = true }
|
||||
# Pure-Rust Zstandard, for the plugin filters that embed zstd (bitshuffle,
|
||||
# blosc). The `zstd` feature (filter 32015) links libzstd instead.
|
||||
ruzstd = { version = "0.9", optional = true }
|
||||
|
||||
[dev-dependencies]
|
||||
half = { workspace = true }
|
||||
@@ -59,6 +62,8 @@ pcodec = ["dep:pco"]
|
||||
# Plugin filters, pure Rust. LZF (32000) is h5py's built-in compression; it
|
||||
# has no dependencies, so it is on by default.
|
||||
lzf = []
|
||||
# Bitshuffle (32008), with its LZ4 and Zstandard modes.
|
||||
bitshuffle = ["lz4_flex", "ruzstd"]
|
||||
|
||||
[[bench]]
|
||||
name = "parallel_decompress_bench"
|
||||
|
||||
Reference in New Issue
Block a user