feat(format): Blosc filter (32001), read and write, pure Rust

hdf5plugin's Blosc (hdf5-blosc) failed with UnsupportedFilter(32001). The
new `blosc` feature decodes the Blosc 1 frame c-blosc 1.x writes: the
16-byte header, raw ("memcpyed") frames, the block table, blocks split
into one stream per byte plane (and the "do not split" flag), streams
stored raw, the byte shuffle and bit shuffle (whole 8-element groups, the
rest copied) - and every codec hdf5plugin offers: BloscLZ (implemented
here from c-blosc 1.21's blosclz_decompress, including its rejection of
malformed and truncated streams), LZ4/LZ4HC (lz4_flex), Snappy (snap),
Zlib (flate2) and Zstandard (ruzstd). Every stream must decode to exactly
its size and the frame to at most the chunk size; a frame of another
format version (Blosc 2) is a clear error.

It also encodes (DatasetBuilder::with_blosc(codec, level, shuffle)):
LZ4, Snappy, Zlib or Zstandard, with c-blosc's split rule, raw streams
where compression does not pay, and a stored frame for level 0 or
incompressible data. It cannot write BloscLZ (asking for it is an
error). `plugin-filters` enables LZF, bitshuffle, bzip2 and Blosc.

Interop: hdf5plugin writes all six codecs x {no, byte, bit} shuffle at
levels 5/9/1, plus level 0, over the 12-case matrix, read byte for byte;
our four codecs x four shuffle/level settings read back through
hdf5plugin. Both fail with the decoder removed. `cargo tree` with
`plugin-filters` has no -sys crate other than libbz2-rs-sys (pure Rust),
no cc and no cmake.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 00:10:05 -05:00
co-authored by Claude Opus 5.5
parent 6dfd239011
commit 1f71f3bcbc
9 changed files with 758 additions and 6 deletions
+67 -2
View File
@@ -12,8 +12,8 @@ use crate::chunk_grid::ChunkGrid;
use crate::ea_writer;
use crate::error::FormatError;
use crate::filter_pipeline::{
FILTER_BITSHUFFLE, FILTER_BZIP2, FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZ4, FILTER_LZF,
FILTER_PCODEC, FILTER_PCODEC_NAME, FILTER_SHUFFLE, FILTER_ZSTD, FilterDescription,
FILTER_BITSHUFFLE, FILTER_BLOSC, FILTER_BZIP2, FILTER_DEFLATE, FILTER_FLETCHER32, FILTER_LZ4,
FILTER_LZF, FILTER_PCODEC, FILTER_PCODEC_NAME, FILTER_SHUFFLE, FILTER_ZSTD, FilterDescription,
FilterPipeline,
};
use crate::filters::compress_chunk;
@@ -77,6 +77,41 @@ pub enum PluginFilter {
/// Block size 1-9 (9 = hdf5plugin's default).
level: u32,
},
/// Blosc 1 (filter 32001): `codec` at `level` (0-9; 0 stores), after
/// `shuffle`. Needs the `blosc` feature.
Blosc {
/// The codec inside the Blosc frame.
codec: BloscCodec,
/// Compression level 0-9 (0 stores the data uncompressed).
level: u32,
/// The shuffle Blosc applies first.
shuffle: BloscShuffle,
},
}
/// The codec inside a Blosc frame that clawhdf5 can write. (It reads
/// BloscLZ too, but cannot write it.)
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum BloscCodec {
/// LZ4.
Lz4,
/// Snappy.
Snappy,
/// Zlib, at the Blosc level.
Zlib,
/// Zstandard (clawhdf5's pure-Rust encoder has one level, about zstd 1).
Zstd,
}
/// The shuffle Blosc applies before compressing.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum BloscShuffle {
/// None.
None,
/// Byte shuffle (Blosc's default).
Byte,
/// Bit shuffle.
Bit,
}
/// What bitshuffle compresses its blocks with.
@@ -102,6 +137,7 @@ impl PluginFilter {
PluginFilter::Lzf => false,
PluginFilter::Bitshuffle { .. } => true,
PluginFilter::Bzip2 { .. } => false,
PluginFilter::Blosc { .. } => true,
}
}
@@ -120,6 +156,35 @@ impl PluginFilter {
},
// bshuf_h5_set_local: version 0.4, element size, block size,
// compression (0 none, 2 LZ4, 3 Zstandard), Zstandard level.
// hdf5-blosc's blosc_set_local: filter revision 2, Blosc format
// 2, type size, chunk size, then level, shuffle, compressor.
PluginFilter::Blosc {
codec,
level,
shuffle,
} => FilterDescription {
filter_id: FILTER_BLOSC,
name: Some("blosc".into()),
flags: 1,
client_data: vec![
2,
2,
element_size,
chunk_bytes,
(*level).min(9),
match shuffle {
BloscShuffle::None => 0,
BloscShuffle::Byte => 1,
BloscShuffle::Bit => 2,
},
match codec {
BloscCodec::Lz4 => 1,
BloscCodec::Snappy => 3,
BloscCodec::Zlib => 4,
BloscCodec::Zstd => 5,
},
],
},
PluginFilter::Bzip2 { level } => FilterDescription {
filter_id: FILTER_BZIP2,
name: Some("bzip2".into()),