Chunked reads beat an h5py process pool; unlimited writer B-trees; Blosc2; 599/697 conformance #16
@@ -2,6 +2,26 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Blosc2 (2026-09-26)
|
||||
- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with
|
||||
hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature
|
||||
`blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`)
|
||||
decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the
|
||||
B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered
|
||||
back into C order), Blosc2 chunks with their special values (zeros, NaN,
|
||||
uninitialised, one repeated value), and the shuffle, bit-shuffle, delta
|
||||
and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and
|
||||
Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2
|
||||
encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined
|
||||
codecs and registered Blosc2 filters (e.g. bytedelta) are errors;
|
||||
uninitialised chunks read as zeros. Tested against h5py 3.16 +
|
||||
hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with
|
||||
partial edge chunks; every integer width and f4/f8; datasets of zeros,
|
||||
one value and NaN; Fletcher32 before Blosc2) and against frames from
|
||||
python-blosc2 4.13.1 for what hdf5plugin never writes
|
||||
(`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is
|
||||
fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
|
||||
|
||||
### Concurrent reads (2026-09-26)
|
||||
- **Full reads of chunked datasets scale with threads again when rayon's
|
||||
pool has one thread.** Each full read handed its chunks to rayon to
|
||||
@@ -372,6 +392,7 @@
|
||||
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
|
||||
clawhdf5)").
|
||||
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
|
||||
(Blosc2 reads since the `blosc2` feature, see above.)
|
||||
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
|
||||
filter). HDF5 stores every chunk at the full chunk size, so a filter
|
||||
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
|
||||
|
||||
@@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
|-------|------|
|
||||
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
||||
| `clawhdf5-io` | Read/write implementation |
|
||||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1) live in `clawhdf5-format`. No Blosc2 or ZFP. |
|
||||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
|
||||
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
||||
| `clawhdf5` | Main facade crate |
|
||||
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
||||
|
||||
@@ -668,7 +668,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
├── Core HDF5
|
||||
│ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions
|
||||
│ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
|
||||
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format
|
||||
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc/Blosc2 filters live in clawhdf5-format
|
||||
│ ├── clawhdf5-derive — Proc macros
|
||||
│ ├── clawhdf5 — High-level API
|
||||
│ ├── clawhdf5-netcdf4 — NetCDF-4 support
|
||||
@@ -779,13 +779,15 @@ stores keep their setting. Opt out with `float16 = false` or
|
||||
| `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) |
|
||||
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
|
||||
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
|
||||
| `plugin-filters` | no | All four above |
|
||||
| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
|
||||
| `plugin-filters` | no | All five above |
|
||||
|
||||
Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with
|
||||
`UnsupportedFilter`, whose message names the filter. Any other filter can be
|
||||
supplied at run time with `filter_registry::register_filter` (a decoder
|
||||
closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`)
|
||||
forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write
|
||||
ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`,
|
||||
whose message names the filter. clawhdf5 cannot write Blosc2. Any other
|
||||
filter can be supplied at run time with `filter_registry::register_filter` (a
|
||||
decoder closure, or a `FilterCodec` that also encodes). The facade
|
||||
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and
|
||||
`plugin-filters`. Write
|
||||
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
|
||||
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
|
||||
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
|
||||
|
||||
+10
-5
@@ -210,11 +210,16 @@ fill-value item that did is fixed).
|
||||
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
|
||||
`plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf,
|
||||
h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of
|
||||
697 ok). **Still open:** Blosc2 (32026 — hdf5plugin stores each chunk as a
|
||||
Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013);
|
||||
both fail with an `UnsupportedFilter` error that names the filter, and
|
||||
either can be plugged in with `filter_registry::register_filter` (32023,
|
||||
Granular BitRound, too, since 2026-09-26 even with the `pcodec` feature).
|
||||
697 ok). **Fixed 2026-09-26** for Blosc2 (32026, `blosc2` feature, also
|
||||
in `plugin-filters`), read only: hdf5plugin's frames and B2ND arrays, every
|
||||
codec and filter it offers; h5ex_d_blosc2 now reads (conformance 576 of
|
||||
697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using
|
||||
dictionaries, lazy chunks, variable-length blocks, user-defined codecs or
|
||||
registered filters (e.g. bytedelta) are refused with an error.
|
||||
**Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that
|
||||
names the filter, and can be plugged in with
|
||||
`filter_registry::register_filter` (32023, Granular BitRound, too, since
|
||||
2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2.
|
||||
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
|
||||
read with zeros for the missing bytes** (any filter; found reviewing the plugin
|
||||
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
|
||||
|
||||
Reference in New Issue
Block a user