osobhandClaude Opus 5.5 7334e21c93 feat(format): read Blosc2 (filter 32026) in pure Rust ...

hdf5plugin's Blosc2 was a clear "not implemented" error. It stores each
HDF5 chunk as a Blosc2 contiguous frame; for chunks of 2+ dimensions the
frame holds a B2ND array whose data are cut into (padded) blocks stored
one after another.

filters_blosc2 (feature `blosc2`, in `plugin-filters`; the facade
forwards both) decodes, following c-blosc2's decoder and hdf5-blosc2's
blosc2_filter.c:
- the frame: the msgpack header's fixed fields, the metalayer index, the
  offsets chunk (with the special zero/NaN/uninitialised offsets) and
  chunk lookup;
- Blosc2 chunks: 16- and 32-byte headers, special chunks (zeros, NaN,
  uninitialised, one repeated value), split and unsplit streams, zero and
  run-length streams, and the filter pipeline run backwards (shuffle,
  shuffle with a byte-group size, bit shuffle including the version-2 and
  later handling of a partial group of 8, delta against the first block,
  truncated precision);
- the codecs, shared with Blosc 1: BloscLZ, LZ4/LZ4HC, Zlib, Zstandard;
- B2ND arrays: blocks gathered into C order, padding dropped, several
  chunks per array, and the array shape checked against the chunk shape
  in cd_values as the HDF5 filter does.
Dictionaries, lazy chunks, variable-length blocks, user-defined codecs
and registered filters (e.g. bytedelta) are errors. Uninitialised chunks
read as zeros. No encoder.

Tests: h5py + hdf5plugin write every codec x filter (none, shuffle,
bitshuffle, delta) and levels 0-9 over the plugin-filter cases, then
i1..u8/f4/f8 in 1-D to 5-D chunks with partial edge chunks, datasets of
zeros, one value and NaN, and Fletcher32 before Blosc2 (plain frames for
n-D chunks); clawhdf5 reads each exactly as its unfiltered twin, and
truncated precision exactly as h5py reads it. Fixture frames from
python-blosc2 (tests/fixtures/blosc2/generate.py) cover what hdf5plugin
never writes: special chunks, delta over many blocks and odd type sizes,
odd bit-shuffle blocks, forced splitting, multi-chunk B2ND arrays with a
zero chunk, and the refused features. The decoder is fuzzed (random and
mutated frames and chunks: no panic, output within the limit).

Conformance: h5ex_d_blosc2.h5 now reads (576 of 697 ok, baseline 575).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

2026-09-26 10:22:22 -05:00

3.9 KiB

Raw History

View Raw