diff --git a/CHANGELOG.md b/CHANGELOG.md index 29804fc..aebeb93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,26 @@ ## Unreleased +### Blosc2 (2026-09-26) +- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with + hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature + `blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`) + decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the + B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered + back into C order), Blosc2 chunks with their special values (zeros, NaN, + uninitialised, one repeated value), and the shuffle, bit-shuffle, delta + and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and + Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2 + encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined + codecs and registered Blosc2 filters (e.g. bytedelta) are errors; + uninitialised chunks read as zeros. Tested against h5py 3.16 + + hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with + partial edge chunks; every integer width and f4/f8; datasets of zeros, + one value and NaN; Fletcher32 before Blosc2) and against frames from + python-blosc2 4.13.1 for what hdf5plugin never writes + (`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is + fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2. + ### Concurrent reads (2026-09-26) - **Full reads of chunked datasets scale with threads again when rayon's pool has one thread.** Each full read handed its chunks to rayon to @@ -372,6 +392,7 @@ missing feature ("unsupported filter: 32026 (Blosc2, not implemented by clawhdf5)"). - **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error. + (Blosc2 reads since the `blosc2` feature, see above.) - **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every filter). HDF5 stores every chunk at the full chunk size, so a filter pipeline that decodes to fewer bytes means a corrupt chunk; every chunk diff --git a/CLAUDE.md b/CLAUDE.md index 1cb7aba..b029c5d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F |-------|------| | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants | | `clawhdf5-io` | Read/write implementation | -| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1) live in `clawhdf5-format`. No Blosc2 or ZFP. | +| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. | | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs | | `clawhdf5` | Main facade crate | | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer | diff --git a/README.md b/README.md index 4ae4d1c..25f43a7 100644 --- a/README.md +++ b/README.md @@ -668,7 +668,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests ├── Core HDF5 │ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions │ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI) -│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format +│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc/Blosc2 filters live in clawhdf5-format │ ├── clawhdf5-derive — Proc macros │ ├── clawhdf5 — High-level API │ ├── clawhdf5-netcdf4 — NetCDF-4 support @@ -779,13 +779,15 @@ stores keep their setting. Opt out with `float16 = false` or | `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) | | `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) | | `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust | -| `plugin-filters` | no | All four above | +| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust | +| `plugin-filters` | no | All five above | -Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with -`UnsupportedFilter`, whose message names the filter. Any other filter can be -supplied at run time with `filter_registry::register_filter` (a decoder -closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`) -forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write +ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`, +whose message names the filter. clawhdf5 cannot write Blosc2. Any other +filter can be supplied at run time with `filter_registry::register_filter` (a +decoder closure, or a `FilterCodec` that also encodes). The facade +(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and +`plugin-filters`. Write with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)` and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in `crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard diff --git a/docs/known-issues.md b/docs/known-issues.md index 2657cdf..2c42b2f 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -210,11 +210,16 @@ fill-value item that did is fixed). bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or `plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf, h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of - 697 ok). **Still open:** Blosc2 (32026 — hdf5plugin stores each chunk as a - Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013); - both fail with an `UnsupportedFilter` error that names the filter, and - either can be plugged in with `filter_registry::register_filter` (32023, - Granular BitRound, too, since 2026-09-26 even with the `pcodec` feature). + 697 ok). **Fixed 2026-09-26** for Blosc2 (32026, `blosc2` feature, also + in `plugin-filters`), read only: hdf5plugin's frames and B2ND arrays, every + codec and filter it offers; h5ex_d_blosc2 now reads (conformance 576 of + 697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using + dictionaries, lazy chunks, variable-length blocks, user-defined codecs or + registered filters (e.g. bytedelta) are refused with an error. + **Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that + names the filter, and can be plugged in with + `filter_registry::register_filter` (32023, Granular BitRound, too, since + 2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2. - **Wrong data: a chunk whose filters decode to fewer bytes than the chunk read with zeros for the missing bytes** (any filter; found reviewing the plugin filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt