From 2b7065998adf1a552541a8ec1df6c391b481a1bd Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 13:08:47 -0500 Subject: [PATCH] docs: ZFP reads (changelog, README feature table, known issues) ZFP (32013) was the one plugin filter still listed as UnsupportedFilter. Conformance on tank, `conformance/run.sh --no-fetch` (2026-09-26): 600 of 697 files ok (baseline 599); h5ex_d_zfp.h5 is newly ok. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 24 ++++++++++++++++++++++++ CLAUDE.md | 2 +- README.md | 10 +++++----- docs/known-issues.md | 12 ++++++++---- 4 files changed, 38 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 76f4316..badf440 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,30 @@ ## Unreleased +### ZFP (2026-09-26) +- **ZFP (filter 32013, H5Z-ZFP) reads, in pure Rust.** It was the last + filter in the conformance corpus that failed with `UnsupportedFilter`. + New feature `zfp` (`clawhdf5-format` and `clawhdf5`, included in + `plugin-filters`, no dependencies) ports the zfp 1.0.1 decoder and the + decompression half of H5Z-ZFP 1.1.1: every mode (fixed rate, fixed + precision, fixed accuracy, reversible, expert), int32, int64, float and + double, 1-4-D fields with partial blocks, and headers written by + big-endian machines (values byte-swapped as H5Z-ZFP does). Read only: + there is no ZFP encoder. The decoder is deterministic, so lossy modes have + one right answer, and clawhdf5 returns exactly libzfp's values. +- Tests: `crates/clawhdf5/tests/zfp_interop.rs` has h5py + hdf5plugin 7.1 + (H5Z-ZFP 1.1.1, zfp 1.0.1) write 2205 datasets over 16 mode settings + (including expert parameters at their edges) x the four types x 1-4-D + shapes with partial edge chunks, partial blocks and unit chunk + dimensions x smooth, noisy, wide-range, zero and inf/NaN data; each must + read byte for byte as h5py reads it. `tests/zfp_alloc_bounds.rs` fuzzes + headers and streams under a counting allocator: no panics, and the output + is allocated only when it matches the chunk's size and the stream holds at + least a bit per block. A stream that ends before the decoder is done is an + error (libzfp reads past its buffer). +- Conformance on tank (2026-09-26, `conformance/run.sh --no-fetch`): 600 of + 697 files ok (599 before); `h5ex_d_zfp.h5` now reads. + ### Chunked full reads (2026-09-26) - **Chunks are decoded straight into the output, into reused buffers.** A full read of a chunked dataset faulted in about three times its size in diff --git a/CLAUDE.md b/CLAUDE.md index b029c5d..cf9f9ae 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F |-------|------| | `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants | | `clawhdf5-io` | Read/write implementation | -| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. | +| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. | | `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs | | `clawhdf5` | Main facade crate | | `clawhdf5-netcdf4` | NetCDF-4 compatibility layer | diff --git a/README.md b/README.md index 25f43a7..f74fc71 100644 --- a/README.md +++ b/README.md @@ -780,14 +780,14 @@ stores keep their setting. Opt out with `float16 = false` or | `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) | | `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust | | `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust | -| `plugin-filters` | no | All five above | +| `zfp` | no | ZFP filter (id 32013, H5Z-ZFP), read only: every mode (rate, precision, accuracy, reversible, expert) for int32, int64, float and double, 1-4-D, returning exactly libzfp's values. Pure Rust, no dependencies | +| `plugin-filters` | no | All six above | -ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`, -whose message names the filter. clawhdf5 cannot write Blosc2. Any other +clawhdf5 cannot write Blosc2 or ZFP. Any other filter can be supplied at run time with `filter_registry::register_filter` (a decoder closure, or a `FilterCodec` that also encodes). The facade -(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and -`plugin-filters`. Write +(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2`, `zfp` +and `plugin-filters`. Write with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)` and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in `crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard diff --git a/docs/known-issues.md b/docs/known-issues.md index 287e501..4cbd25b 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -293,10 +293,14 @@ fill-value item that did is fixed). 697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using dictionaries, lazy chunks, variable-length blocks, user-defined codecs or registered filters (e.g. bytedelta) are refused with an error. - **Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that - names the filter, and can be plugged in with - `filter_registry::register_filter` (32023, Granular BitRound, too, since - 2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2. + **Fixed 2026-09-26** for ZFP (32013, `zfp` feature, also in + `plugin-filters`), read only: every H5Z-ZFP mode and type, bit-exact + against h5py + hdf5plugin 7.1 (`crates/clawhdf5/tests/zfp_interop.rs`); + h5ex_d_zfp now reads (conformance 600 of 697 ok, tank, + `conformance/run.sh --no-fetch`). **Still open:** clawhdf5 cannot write + Blosc2 or ZFP. Other filters (32023, Granular BitRound, too, since + 2026-09-26 even with the `pcodec` feature) can be plugged in with + `filter_registry::register_filter`. - **Wrong data: a chunk whose filters decode to fewer bytes than the chunk read with zeros for the missing bytes** (any filter; found reviewing the plugin filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt