From 56abaec75e81747a073a32f717fe922fa4bfd792 Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 10:22:52 -0500 Subject: [PATCH] docs: Blosc2 reads (read only); ZFP is the one plugin filter left Record the `blosc2` feature in the changelog, the README's feature table and the crate table, and mark the Blosc2 half of the known "Filters" issue fixed (dated, with the conformance run that shows h5ex_d_blosc2 reading). What stays open: ZFP, writing Blosc2, and the Blosc2 features hdf5plugin never writes (dictionaries, lazy chunks, variable-length blocks, user-defined codecs and registered filters), which are errors. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 21 +++++++++++++++++++++ CLAUDE.md | 2 +- README.md | 16 +++++++++------- docs/known-issues.md | 15 ++++++++++----- 4 files changed, 41 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 29804fc..aebeb93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,26 @@ ## Unreleased +### Blosc2 (2026-09-26) +- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with + hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature + `blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`) + decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the + B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered + back into C order), Blosc2 chunks with their special values (zeros, NaN, + uninitialised, one repeated value), and the shuffle, bit-shuffle, delta + and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and + Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2 + encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined + codecs and registered Blosc2 filters (e.g. bytedelta) are errors; + uninitialised chunks read as zeros. Tested against h5py 3.16 + + hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with + partial edge chunks; every integer width and f4/f8; datasets of zeros, + one value and NaN; Fletcher32 before Blosc2) and against frames from + python-blosc2 4.13.1 for what hdf5plugin never writes + (`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is + fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2. + ### Concurrent reads (2026-09-26) - **Full reads of chunked datasets scale with threads again when rayon's pool has one thread.** Each full read handed its chunks to rayon to @@ -372,6 +392,7 @@ missing feature ("unsupported filter: 32026 (Blosc2, not implemented by clawhdf5)"). - **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error. + (Blosc2 reads since the `blosc2` feature, see above.) - **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every filter). HDF5 stores every chunk at the full chunk size, so a filter pipeline that decodes to fewer bytes means a corrupt chunk; every chunk diff --git a/CLAUDE.md b/CLAUDE.md index 1cb7aba..b029c5d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F |-------|------| | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants | | `clawhdf5-io` | Read/write implementation | -| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1) live in `clawhdf5-format`. No Blosc2 or ZFP. | +| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. | | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs | | `clawhdf5` | Main facade crate | | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer | diff --git a/README.md b/README.md index 4ae4d1c..25f43a7 100644 --- a/README.md +++ b/README.md @@ -668,7 +668,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests ├── Core HDF5 │ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions │ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI) -│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format +│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc/Blosc2 filters live in clawhdf5-format │ ├── clawhdf5-derive — Proc macros │ ├── clawhdf5 — High-level API │ ├── clawhdf5-netcdf4 — NetCDF-4 support @@ -779,13 +779,15 @@ stores keep their setting. Opt out with `float16 = false` or | `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) | | `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) | | `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust | -| `plugin-filters` | no | All four above | +| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust | +| `plugin-filters` | no | All five above | -Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with -`UnsupportedFilter`, whose message names the filter. Any other filter can be -supplied at run time with `filter_registry::register_filter` (a decoder -closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`) -forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write +ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`, +whose message names the filter. clawhdf5 cannot write Blosc2. Any other +filter can be supplied at run time with `filter_registry::register_filter` (a +decoder closure, or a `FilterCodec` that also encodes). The facade +(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and +`plugin-filters`. Write with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)` and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in `crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard diff --git a/docs/known-issues.md b/docs/known-issues.md index 2657cdf..2c42b2f 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -210,11 +210,16 @@ fill-value item that did is fixed). bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or `plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf, h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of - 697 ok). **Still open:** Blosc2 (32026 — hdf5plugin stores each chunk as a - Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013); - both fail with an `UnsupportedFilter` error that names the filter, and - either can be plugged in with `filter_registry::register_filter` (32023, - Granular BitRound, too, since 2026-09-26 even with the `pcodec` feature). + 697 ok). **Fixed 2026-09-26** for Blosc2 (32026, `blosc2` feature, also + in `plugin-filters`), read only: hdf5plugin's frames and B2ND arrays, every + codec and filter it offers; h5ex_d_blosc2 now reads (conformance 576 of + 697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using + dictionaries, lazy chunks, variable-length blocks, user-defined codecs or + registered filters (e.g. bytedelta) are refused with an error. + **Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that + names the filter, and can be plugged in with + `filter_registry::register_filter` (32023, Granular BitRound, too, since + 2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2. - **Wrong data: a chunk whose filters decode to fewer bytes than the chunk read with zeros for the missing bytes** (any filter; found reviewing the plugin filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt