Chunked reads beat an h5py process pool; unlimited writer B-trees; Blosc2; 599/697 conformance #16

Merged
osobh merged 48 commits from feat/p2b-scale into main 2026-09-26 17:42:16 +00:00
4 changed files with 41 additions and 13 deletions
Showing only changes of commit 56abaec75e - Show all commits
+21
View File
@@ -2,6 +2,26 @@
## Unreleased
### Blosc2 (2026-09-26)
- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with
hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature
`blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`)
decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the
B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered
back into C order), Blosc2 chunks with their special values (zeros, NaN,
uninitialised, one repeated value), and the shuffle, bit-shuffle, delta
and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and
Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2
encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined
codecs and registered Blosc2 filters (e.g. bytedelta) are errors;
uninitialised chunks read as zeros. Tested against h5py 3.16 +
hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with
partial edge chunks; every integer width and f4/f8; datasets of zeros,
one value and NaN; Fletcher32 before Blosc2) and against frames from
python-blosc2 4.13.1 for what hdf5plugin never writes
(`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is
fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
### Concurrent reads (2026-09-26)
- **Full reads of chunked datasets scale with threads again when rayon's
pool has one thread.** Each full read handed its chunks to rayon to
@@ -372,6 +392,7 @@
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
clawhdf5)").
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
(Blosc2 reads since the `blosc2` feature, see above.)
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
filter). HDF5 stores every chunk at the full chunk size, so a filter
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
+1 -1
View File
@@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
|-------|------|
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
| `clawhdf5-io` | Read/write implementation |
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1) live in `clawhdf5-format`. No Blosc2 or ZFP. |
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
| `clawhdf5` | Main facade crate |
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
+9 -7
View File
@@ -668,7 +668,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
├── Core HDF5
│ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions
│ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc/Blosc2 filters live in clawhdf5-format
│ ├── clawhdf5-derive — Proc macros
│ ├── clawhdf5 — High-level API
│ ├── clawhdf5-netcdf4 — NetCDF-4 support
@@ -779,13 +779,15 @@ stores keep their setting. Opt out with `float16 = false` or
| `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) |
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
| `plugin-filters` | no | All four above |
| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
| `plugin-filters` | no | All five above |
Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with
`UnsupportedFilter`, whose message names the filter. Any other filter can be
supplied at run time with `filter_registry::register_filter` (a decoder
closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`)
forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write
ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`,
whose message names the filter. clawhdf5 cannot write Blosc2. Any other
filter can be supplied at run time with `filter_registry::register_filter` (a
decoder closure, or a `FilterCodec` that also encodes). The facade
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and
`plugin-filters`. Write
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
+10 -5
View File
@@ -210,11 +210,16 @@ fill-value item that did is fixed).
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
`plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf,
h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of
697 ok). **Still open:** Blosc2 (32026 — hdf5plugin stores each chunk as a
Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013);
both fail with an `UnsupportedFilter` error that names the filter, and
either can be plugged in with `filter_registry::register_filter` (32023,
Granular BitRound, too, since 2026-09-26 even with the `pcodec` feature).
697 ok). **Fixed 2026-09-26** for Blosc2 (32026, `blosc2` feature, also
in `plugin-filters`), read only: hdf5plugin's frames and B2ND arrays, every
codec and filter it offers; h5ex_d_blosc2 now reads (conformance 576 of
697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using
dictionaries, lazy chunks, variable-length blocks, user-defined codecs or
registered filters (e.g. bytedelta) are refused with an error.
**Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that
names the filter, and can be plugged in with
`filter_registry::register_filter` (32023, Granular BitRound, too, since
2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2.
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
read with zeros for the missing bytes** (any filter; found reviewing the plugin
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt