feat(format): read Blosc2 (filter 32026) in pure Rust
hdf5plugin's Blosc2 was a clear "not implemented" error. It stores each HDF5 chunk as a Blosc2 contiguous frame; for chunks of 2+ dimensions the frame holds a B2ND array whose data are cut into (padded) blocks stored one after another. filters_blosc2 (feature `blosc2`, in `plugin-filters`; the facade forwards both) decodes, following c-blosc2's decoder and hdf5-blosc2's blosc2_filter.c: - the frame: the msgpack header's fixed fields, the metalayer index, the offsets chunk (with the special zero/NaN/uninitialised offsets) and chunk lookup; - Blosc2 chunks: 16- and 32-byte headers, special chunks (zeros, NaN, uninitialised, one repeated value), split and unsplit streams, zero and run-length streams, and the filter pipeline run backwards (shuffle, shuffle with a byte-group size, bit shuffle including the version-2 and later handling of a partial group of 8, delta against the first block, truncated precision); - the codecs, shared with Blosc 1: BloscLZ, LZ4/LZ4HC, Zlib, Zstandard; - B2ND arrays: blocks gathered into C order, padding dropped, several chunks per array, and the array shape checked against the chunk shape in cd_values as the HDF5 filter does. Dictionaries, lazy chunks, variable-length blocks, user-defined codecs and registered filters (e.g. bytedelta) are errors. Uninitialised chunks read as zeros. No encoder. Tests: h5py + hdf5plugin write every codec x filter (none, shuffle, bitshuffle, delta) and levels 0-9 over the plugin-filter cases, then i1..u8/f4/f8 in 1-D to 5-D chunks with partial edge chunks, datasets of zeros, one value and NaN, and Fletcher32 before Blosc2 (plain frames for n-D chunks); clawhdf5 reads each exactly as its unfiltered twin, and truncated precision exactly as h5py reads it. Fixture frames from python-blosc2 (tests/fixtures/blosc2/generate.py) cover what hdf5plugin never writes: special chunks, delta over many blocks and odd type sizes, odd bit-shuffle blocks, forced splitting, multi-chunk B2ND arrays with a zero chunk, and the refused features. The decoder is fuzzed (random and mutated frames and chunks: no panic, output within the limit). Conformance: h5ex_d_blosc2.h5 now reads (576 of 697 ok, baseline 575). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+2
-2
@@ -67,7 +67,7 @@ run_step "cargo clippy --all-targets" cargo clippy \
|
||||
|
||||
# 3. Clippy over clawhdf5-format's optional features, which the default
|
||||
# workspace build never compiles (szip is left out: it needs libaec).
|
||||
# plugin-filters = bitshuffle, bzip2, blosc (and the default-on lzf).
|
||||
# plugin-filters = bitshuffle, bzip2, blosc, blosc2 (and the default-on lzf).
|
||||
run_step "cargo clippy (format feature matrix)" cargo clippy \
|
||||
-p clawhdf5-format \
|
||||
--all-targets \
|
||||
@@ -78,7 +78,7 @@ run_step "cargo clippy (format feature matrix)" cargo clippy \
|
||||
# dependencies (bitshuffle and blosc share code).
|
||||
plugin_filters_alone() {
|
||||
local f
|
||||
for f in bitshuffle bzip2 blosc; do
|
||||
for f in bitshuffle bzip2 blosc blosc2; do
|
||||
echo "--- $f"
|
||||
cargo clippy -p clawhdf5-format --all-targets --features "$f" -- -D warnings || return 1
|
||||
done
|
||||
|
||||
Reference in New Issue
Block a user