quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: plugin filters and the filter registry

Browse Source 
CHANGELOG (Unreleased): LZF, bitshuffle, bzip2 and Blosc read and write
in pure Rust, their features, the ChunkOptions::plugin field (breaking for
struct-literal construction), the filter registry, the named
UnsupportedFilter message, and Blosc2/ZFP still unimplemented.
README: the clawhdf5-format feature table gains lzf (default),
bitshuffle, bzip2, blosc and plugin-filters, with how to write them and
what is not implemented; no speed claims. docs/known-issues.md: the audit's
filter gap is marked fixed 2026-09-26 for LZF/bitshuffle/bzip2/Blosc,
Blosc2 and ZFP still open. CLAUDE.md: the clawhdf5-filters row no longer
says "No Blosc". clawhdf5-format's crate docs list the new features.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 00:14:47 -05:00
co-authored by Claude Opus 5.5
parent 056092b082
commit e01160299a
5 changed files with 64 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+17 -1
View File
@@ -594,7 +594,7 @@ clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
├── Core HDF5
│ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions
│ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); lz4/zstd/pcodec/szip filters live in clawhdf5-format
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format
│ ├── clawhdf5-derive — Proc macros
│ ├── clawhdf5 — High-level API
│ ├── clawhdf5-netcdf4 — NetCDF-4 support
@@ -700,6 +700,22 @@ stores keep their setting. Opt out with `float16 = false` or
| `system-zlib` | no | System zlib backend for deflate (C) |
| `blake3_hash` | no | BLAKE3 content hashing for provenance |
| `szip` | no | SZIP filter (id 4) via libaec (C, through the internal `libaec-sys` crate) |
| `lzf` | **yes** | LZF filter (id 32000), h5py's built-in `compression="lzf"`: read and write. No dependencies |
| `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) |
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
| `plugin-filters` | no | All four above |
Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with
`UnsupportedFilter`, whose message names the filter. Any other filter can be
supplied at run time with `filter_registry::register_filter` (a decoder
closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`)
forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
encoder has one level (about zstd's level 1); no speed or ratio claims are
made for these codecs.
### `clawhdf5-ann`