docs: Blosc2 reads (read only); ZFP is the one plugin filter left

Record the `blosc2` feature in the changelog, the README's feature table
and the crate table, and mark the Blosc2 half of the known "Filters"
issue fixed (dated, with the conformance run that shows h5ex_d_blosc2
reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features
hdf5plugin never writes (dictionaries, lazy chunks, variable-length
blocks, user-defined codecs and registered filters), which are errors.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 10:22:52 -05:00
co-authored by Claude Opus 5.5
parent 7334e21c93
commit 56abaec75e
4 changed files with 41 additions and 13 deletions
+21
View File
@@ -2,6 +2,26 @@
## Unreleased ## Unreleased
### Blosc2 (2026-09-26)
- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with
hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature
`blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`)
decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the
B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered
back into C order), Blosc2 chunks with their special values (zeros, NaN,
uninitialised, one repeated value), and the shuffle, bit-shuffle, delta
and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and
Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2
encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined
codecs and registered Blosc2 filters (e.g. bytedelta) are errors;
uninitialised chunks read as zeros. Tested against h5py 3.16 +
hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with
partial edge chunks; every integer width and f4/f8; datasets of zeros,
one value and NaN; Fletcher32 before Blosc2) and against frames from
python-blosc2 4.13.1 for what hdf5plugin never writes
(`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is
fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
### Concurrent reads (2026-09-26) ### Concurrent reads (2026-09-26)
- **Full reads of chunked datasets scale with threads again when rayon's - **Full reads of chunked datasets scale with threads again when rayon's
pool has one thread.** Each full read handed its chunks to rayon to pool has one thread.** Each full read handed its chunks to rayon to
@@ -372,6 +392,7 @@
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
clawhdf5)"). clawhdf5)").
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error. - **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
(Blosc2 reads since the `blosc2` feature, see above.)
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every - **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
filter). HDF5 stores every chunk at the full chunk size, so a filter filter). HDF5 stores every chunk at the full chunk size, so a filter
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
+1 -1
View File
@@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
|-------|------| |-------|------|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants | | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
| `clawhdf5-io` | Read/write implementation | | `clawhdf5-io` | Read/write implementation |
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1) live in `clawhdf5-format`. No Blosc2 or ZFP. | | `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs | | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
| `clawhdf5` | Main facade crate | | `clawhdf5` | Main facade crate |
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer | | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
+9 -7
View File
@@ -668,7 +668,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
├── Core HDF5 ├── Core HDF5
│ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions │ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions
│ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI) │ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format │ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc/Blosc2 filters live in clawhdf5-format
│ ├── clawhdf5-derive — Proc macros │ ├── clawhdf5-derive — Proc macros
│ ├── clawhdf5 — High-level API │ ├── clawhdf5 — High-level API
│ ├── clawhdf5-netcdf4 — NetCDF-4 support │ ├── clawhdf5-netcdf4 — NetCDF-4 support
@@ -779,13 +779,15 @@ stores keep their setting. Opt out with `float16 = false` or
| `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) | | `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) |
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) | | `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust | | `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
| `plugin-filters` | no | All four above | | `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
| `plugin-filters` | no | All five above |
Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`,
`UnsupportedFilter`, whose message names the filter. Any other filter can be whose message names the filter. clawhdf5 cannot write Blosc2. Any other
supplied at run time with `filter_registry::register_filter` (a decoder filter can be supplied at run time with `filter_registry::register_filter` (a
closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`) decoder closure, or a `FilterCodec` that also encodes). The facade
forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write (`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and
`plugin-filters`. Write
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)` with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard `crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
+10 -5
View File
@@ -210,11 +210,16 @@ fill-value item that did is fixed).
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
`plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf, `plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf,
h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of
697 ok). **Still open:** Blosc2 (32026 — hdf5plugin stores each chunk as a 697 ok). **Fixed 2026-09-26** for Blosc2 (32026, `blosc2` feature, also
Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013); in `plugin-filters`), read only: hdf5plugin's frames and B2ND arrays, every
both fail with an `UnsupportedFilter` error that names the filter, and codec and filter it offers; h5ex_d_blosc2 now reads (conformance 576 of
either can be plugged in with `filter_registry::register_filter` (32023, 697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using
Granular BitRound, too, since 2026-09-26 even with the `pcodec` feature). dictionaries, lazy chunks, variable-length blocks, user-defined codecs or
registered filters (e.g. bytedelta) are refused with an error.
**Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that
names the filter, and can be plugged in with
`filter_registry::register_filter` (32023, Granular BitRound, too, since
2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2.
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk - **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
read with zeros for the missing bytes** (any filter; found reviewing the plugin read with zeros for the missing bytes** (any filter; found reviewing the plugin
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt