docs: Blosc2 reads (read only); ZFP is the one plugin filter left
Record the `blosc2` feature in the changelog, the README's feature table and the crate table, and mark the Blosc2 half of the known "Filters" issue fixed (dated, with the conformance run that shows h5ex_d_blosc2 reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features hdf5plugin never writes (dictionaries, lazy chunks, variable-length blocks, user-defined codecs and registered filters), which are errors. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,26 @@
|
|||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
|
### Blosc2 (2026-09-26)
|
||||||
|
- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with
|
||||||
|
hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature
|
||||||
|
`blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`)
|
||||||
|
decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the
|
||||||
|
B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered
|
||||||
|
back into C order), Blosc2 chunks with their special values (zeros, NaN,
|
||||||
|
uninitialised, one repeated value), and the shuffle, bit-shuffle, delta
|
||||||
|
and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and
|
||||||
|
Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2
|
||||||
|
encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined
|
||||||
|
codecs and registered Blosc2 filters (e.g. bytedelta) are errors;
|
||||||
|
uninitialised chunks read as zeros. Tested against h5py 3.16 +
|
||||||
|
hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with
|
||||||
|
partial edge chunks; every integer width and f4/f8; datasets of zeros,
|
||||||
|
one value and NaN; Fletcher32 before Blosc2) and against frames from
|
||||||
|
python-blosc2 4.13.1 for what hdf5plugin never writes
|
||||||
|
(`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is
|
||||||
|
fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
|
||||||
|
|
||||||
### Concurrent reads (2026-09-26)
|
### Concurrent reads (2026-09-26)
|
||||||
- **Full reads of chunked datasets scale with threads again when rayon's
|
- **Full reads of chunked datasets scale with threads again when rayon's
|
||||||
pool has one thread.** Each full read handed its chunks to rayon to
|
pool has one thread.** Each full read handed its chunks to rayon to
|
||||||
@@ -372,6 +392,7 @@
|
|||||||
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
|
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
|
||||||
clawhdf5)").
|
clawhdf5)").
|
||||||
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
|
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
|
||||||
|
(Blosc2 reads since the `blosc2` feature, see above.)
|
||||||
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
|
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
|
||||||
filter). HDF5 stores every chunk at the full chunk size, so a filter
|
filter). HDF5 stores every chunk at the full chunk size, so a filter
|
||||||
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
|
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
|
|||||||
|-------|------|
|
|-------|------|
|
||||||
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
||||||
| `clawhdf5-io` | Read/write implementation |
|
| `clawhdf5-io` | Read/write implementation |
|
||||||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1) live in `clawhdf5-format`. No Blosc2 or ZFP. |
|
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
|
||||||
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
||||||
| `clawhdf5` | Main facade crate |
|
| `clawhdf5` | Main facade crate |
|
||||||
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
||||||
|
|||||||
@@ -668,7 +668,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
|
|||||||
├── Core HDF5
|
├── Core HDF5
|
||||||
│ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions
|
│ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions
|
||||||
│ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
|
│ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
|
||||||
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format
|
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc/Blosc2 filters live in clawhdf5-format
|
||||||
│ ├── clawhdf5-derive — Proc macros
|
│ ├── clawhdf5-derive — Proc macros
|
||||||
│ ├── clawhdf5 — High-level API
|
│ ├── clawhdf5 — High-level API
|
||||||
│ ├── clawhdf5-netcdf4 — NetCDF-4 support
|
│ ├── clawhdf5-netcdf4 — NetCDF-4 support
|
||||||
@@ -779,13 +779,15 @@ stores keep their setting. Opt out with `float16 = false` or
|
|||||||
| `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) |
|
| `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) |
|
||||||
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
|
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
|
||||||
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
|
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
|
||||||
| `plugin-filters` | no | All four above |
|
| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
|
||||||
|
| `plugin-filters` | no | All five above |
|
||||||
|
|
||||||
Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with
|
ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`,
|
||||||
`UnsupportedFilter`, whose message names the filter. Any other filter can be
|
whose message names the filter. clawhdf5 cannot write Blosc2. Any other
|
||||||
supplied at run time with `filter_registry::register_filter` (a decoder
|
filter can be supplied at run time with `filter_registry::register_filter` (a
|
||||||
closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`)
|
decoder closure, or a `FilterCodec` that also encodes). The facade
|
||||||
forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write
|
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and
|
||||||
|
`plugin-filters`. Write
|
||||||
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
|
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
|
||||||
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
|
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
|
||||||
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
|
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
|
||||||
|
|||||||
+10
-5
@@ -210,11 +210,16 @@ fill-value item that did is fixed).
|
|||||||
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
|
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
|
||||||
`plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf,
|
`plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf,
|
||||||
h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of
|
h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of
|
||||||
697 ok). **Still open:** Blosc2 (32026 — hdf5plugin stores each chunk as a
|
697 ok). **Fixed 2026-09-26** for Blosc2 (32026, `blosc2` feature, also
|
||||||
Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013);
|
in `plugin-filters`), read only: hdf5plugin's frames and B2ND arrays, every
|
||||||
both fail with an `UnsupportedFilter` error that names the filter, and
|
codec and filter it offers; h5ex_d_blosc2 now reads (conformance 576 of
|
||||||
either can be plugged in with `filter_registry::register_filter` (32023,
|
697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using
|
||||||
Granular BitRound, too, since 2026-09-26 even with the `pcodec` feature).
|
dictionaries, lazy chunks, variable-length blocks, user-defined codecs or
|
||||||
|
registered filters (e.g. bytedelta) are refused with an error.
|
||||||
|
**Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that
|
||||||
|
names the filter, and can be plugged in with
|
||||||
|
`filter_registry::register_filter` (32023, Granular BitRound, too, since
|
||||||
|
2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2.
|
||||||
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
|
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
|
||||||
read with zeros for the missing bytes** (any filter; found reviewing the plugin
|
read with zeros for the missing bytes** (any filter; found reviewing the plugin
|
||||||
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
|
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
|
||||||
|
|||||||
Reference in New Issue
Block a user