docs: ZFP reads (changelog, README feature table, known issues)
ZFP (32013) was the one plugin filter still listed as UnsupportedFilter. Conformance on tank, `conformance/run.sh --no-fetch` (2026-09-26): 600 of 697 files ok (baseline 599); h5ex_d_zfp.h5 is newly ok. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,30 @@
|
|||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
|
### ZFP (2026-09-26)
|
||||||
|
- **ZFP (filter 32013, H5Z-ZFP) reads, in pure Rust.** It was the last
|
||||||
|
filter in the conformance corpus that failed with `UnsupportedFilter`.
|
||||||
|
New feature `zfp` (`clawhdf5-format` and `clawhdf5`, included in
|
||||||
|
`plugin-filters`, no dependencies) ports the zfp 1.0.1 decoder and the
|
||||||
|
decompression half of H5Z-ZFP 1.1.1: every mode (fixed rate, fixed
|
||||||
|
precision, fixed accuracy, reversible, expert), int32, int64, float and
|
||||||
|
double, 1-4-D fields with partial blocks, and headers written by
|
||||||
|
big-endian machines (values byte-swapped as H5Z-ZFP does). Read only:
|
||||||
|
there is no ZFP encoder. The decoder is deterministic, so lossy modes have
|
||||||
|
one right answer, and clawhdf5 returns exactly libzfp's values.
|
||||||
|
- Tests: `crates/clawhdf5/tests/zfp_interop.rs` has h5py + hdf5plugin 7.1
|
||||||
|
(H5Z-ZFP 1.1.1, zfp 1.0.1) write 2205 datasets over 16 mode settings
|
||||||
|
(including expert parameters at their edges) x the four types x 1-4-D
|
||||||
|
shapes with partial edge chunks, partial blocks and unit chunk
|
||||||
|
dimensions x smooth, noisy, wide-range, zero and inf/NaN data; each must
|
||||||
|
read byte for byte as h5py reads it. `tests/zfp_alloc_bounds.rs` fuzzes
|
||||||
|
headers and streams under a counting allocator: no panics, and the output
|
||||||
|
is allocated only when it matches the chunk's size and the stream holds at
|
||||||
|
least a bit per block. A stream that ends before the decoder is done is an
|
||||||
|
error (libzfp reads past its buffer).
|
||||||
|
- Conformance on tank (2026-09-26, `conformance/run.sh --no-fetch`): 600 of
|
||||||
|
697 files ok (599 before); `h5ex_d_zfp.h5` now reads.
|
||||||
|
|
||||||
### Chunked full reads (2026-09-26)
|
### Chunked full reads (2026-09-26)
|
||||||
- **Chunks are decoded straight into the output, into reused buffers.** A
|
- **Chunks are decoded straight into the output, into reused buffers.** A
|
||||||
full read of a chunked dataset faulted in about three times its size in
|
full read of a chunked dataset faulted in about three times its size in
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
|
|||||||
|-------|------|
|
|-------|------|
|
||||||
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
||||||
| `clawhdf5-io` | Read/write implementation |
|
| `clawhdf5-io` | Read/write implementation |
|
||||||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
|
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. |
|
||||||
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
||||||
| `clawhdf5` | Main facade crate |
|
| `clawhdf5` | Main facade crate |
|
||||||
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
||||||
|
|||||||
@@ -780,14 +780,14 @@ stores keep their setting. Opt out with `float16 = false` or
|
|||||||
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
|
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
|
||||||
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
|
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
|
||||||
| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
|
| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
|
||||||
| `plugin-filters` | no | All five above |
|
| `zfp` | no | ZFP filter (id 32013, H5Z-ZFP), read only: every mode (rate, precision, accuracy, reversible, expert) for int32, int64, float and double, 1-4-D, returning exactly libzfp's values. Pure Rust, no dependencies |
|
||||||
|
| `plugin-filters` | no | All six above |
|
||||||
|
|
||||||
ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`,
|
clawhdf5 cannot write Blosc2 or ZFP. Any other
|
||||||
whose message names the filter. clawhdf5 cannot write Blosc2. Any other
|
|
||||||
filter can be supplied at run time with `filter_registry::register_filter` (a
|
filter can be supplied at run time with `filter_registry::register_filter` (a
|
||||||
decoder closure, or a `FilterCodec` that also encodes). The facade
|
decoder closure, or a `FilterCodec` that also encodes). The facade
|
||||||
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and
|
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2`, `zfp`
|
||||||
`plugin-filters`. Write
|
and `plugin-filters`. Write
|
||||||
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
|
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
|
||||||
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
|
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
|
||||||
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
|
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
|
||||||
|
|||||||
@@ -293,10 +293,14 @@ fill-value item that did is fixed).
|
|||||||
697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using
|
697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using
|
||||||
dictionaries, lazy chunks, variable-length blocks, user-defined codecs or
|
dictionaries, lazy chunks, variable-length blocks, user-defined codecs or
|
||||||
registered filters (e.g. bytedelta) are refused with an error.
|
registered filters (e.g. bytedelta) are refused with an error.
|
||||||
**Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that
|
**Fixed 2026-09-26** for ZFP (32013, `zfp` feature, also in
|
||||||
names the filter, and can be plugged in with
|
`plugin-filters`), read only: every H5Z-ZFP mode and type, bit-exact
|
||||||
`filter_registry::register_filter` (32023, Granular BitRound, too, since
|
against h5py + hdf5plugin 7.1 (`crates/clawhdf5/tests/zfp_interop.rs`);
|
||||||
2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2.
|
h5ex_d_zfp now reads (conformance 600 of 697 ok, tank,
|
||||||
|
`conformance/run.sh --no-fetch`). **Still open:** clawhdf5 cannot write
|
||||||
|
Blosc2 or ZFP. Other filters (32023, Granular BitRound, too, since
|
||||||
|
2026-09-26 even with the `pcodec` feature) can be plugged in with
|
||||||
|
`filter_registry::register_filter`.
|
||||||
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
|
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
|
||||||
read with zeros for the missing bytes** (any filter; found reviewing the plugin
|
read with zeros for the missing bytes** (any filter; found reviewing the plugin
|
||||||
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
|
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
|
||||||
|
|||||||
Reference in New Issue
Block a user