Merge branch 'feat/p3-zfp' into feat/p3-range-zfp-edit
# Conflicts: # CHANGELOG.md # crates/clawhdf5-format/Cargo.toml
This commit is contained in:
osobh
@@ -149,6 +149,30 @@
|
||||
converted parser falling back to the whole file fails it. Milestones M2
|
||||
and M3 extend it.
|
||||
|
||||
### ZFP (2026-09-26)
|
||||
- **ZFP (filter 32013, H5Z-ZFP) reads, in pure Rust.** It was the last
|
||||
filter in the conformance corpus that failed with `UnsupportedFilter`.
|
||||
New feature `zfp` (`clawhdf5-format` and `clawhdf5`, included in
|
||||
`plugin-filters`, no dependencies) ports the zfp 1.0.1 decoder and the
|
||||
decompression half of H5Z-ZFP 1.1.1: every mode (fixed rate, fixed
|
||||
precision, fixed accuracy, reversible, expert), int32, int64, float and
|
||||
double, 1-4-D fields with partial blocks, and headers written by
|
||||
big-endian machines (values byte-swapped as H5Z-ZFP does). Read only:
|
||||
there is no ZFP encoder. The decoder is deterministic, so lossy modes have
|
||||
one right answer, and clawhdf5 returns exactly libzfp's values.
|
||||
- Tests: `crates/clawhdf5/tests/zfp_interop.rs` has h5py + hdf5plugin 7.1
|
||||
(H5Z-ZFP 1.1.1, zfp 1.0.1) write 2205 datasets over 16 mode settings
|
||||
(including expert parameters at their edges) x the four types x 1-4-D
|
||||
shapes with partial edge chunks, partial blocks and unit chunk
|
||||
dimensions x smooth, noisy, wide-range, zero and inf/NaN data; each must
|
||||
read byte for byte as h5py reads it. `tests/zfp_alloc_bounds.rs` fuzzes
|
||||
headers and streams under a counting allocator: no panics, and the output
|
||||
is allocated only when it matches the chunk's size and the stream holds at
|
||||
least a bit per block. A stream that ends before the decoder is done is an
|
||||
error (libzfp reads past its buffer).
|
||||
- Conformance on tank (2026-09-26, `conformance/run.sh --no-fetch`): 600 of
|
||||
697 files ok (599 before); `h5ex_d_zfp.h5` now reads.
|
||||
|
||||
### Chunked full reads (2026-09-26)
|
||||
- **Chunks are decoded straight into the output, into reused buffers.** A
|
||||
full read of a chunked dataset faulted in about three times its size in
|
||||
|
||||
@@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
|-------|------|
|
||||
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
||||
| `clawhdf5-io` | Read/write implementation |
|
||||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
|
||||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. |
|
||||
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
||||
| `clawhdf5` | Main facade crate |
|
||||
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
||||
|
||||
@@ -780,14 +780,14 @@ stores keep their setting. Opt out with `float16 = false` or
|
||||
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
|
||||
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
|
||||
| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
|
||||
| `plugin-filters` | no | All five above |
|
||||
| `zfp` | no | ZFP filter (id 32013, H5Z-ZFP), read only: every mode (rate, precision, accuracy, reversible, expert) for int32, int64, float and double, 1-4-D, returning exactly libzfp's values. Pure Rust, no dependencies |
|
||||
| `plugin-filters` | no | All six above |
|
||||
|
||||
ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`,
|
||||
whose message names the filter. clawhdf5 cannot write Blosc2. Any other
|
||||
clawhdf5 cannot write Blosc2 or ZFP. Any other
|
||||
filter can be supplied at run time with `filter_registry::register_filter` (a
|
||||
decoder closure, or a `FilterCodec` that also encodes). The facade
|
||||
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and
|
||||
`plugin-filters`. Write
|
||||
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2`, `zfp`
|
||||
and `plugin-filters`. Write
|
||||
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
|
||||
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
|
||||
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
|
||||
|
||||
@@ -78,8 +78,11 @@ bzip2 = ["dep:bzip2", "std"]
|
||||
blosc = ["lz4_flex", "ruzstd", "snap", "deflate", "std"]
|
||||
# Blosc2 (32026), read-only: frames, B2ND arrays, and the Blosc codecs above.
|
||||
blosc2 = ["blosc"]
|
||||
# ZFP (32013, H5Z-ZFP), read-only: every mode, for int32, int64, float and
|
||||
# double fields of 1 to 4 dimensions.
|
||||
zfp = []
|
||||
# Every plugin filter above.
|
||||
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2"]
|
||||
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2", "zfp"]
|
||||
# Test instrumentation: per-thread counts of heap objects read (see
|
||||
# `lookup_stats`), so tests can bound the cost of a name lookup.
|
||||
lookup-stats = ["std"]
|
||||
|
||||
@@ -27,7 +27,8 @@ pub const FILTER_LZF: u16 = 32000;
|
||||
pub const FILTER_BLOSC: u16 = 32001;
|
||||
/// Bitshuffle, optionally with LZ4 or Zstandard (hdf5plugin's `Bitshuffle`).
|
||||
pub const FILTER_BITSHUFFLE: u16 = 32008;
|
||||
/// ZFP lossy floating-point compression (hdf5plugin's `Zfp`). Not supported.
|
||||
/// ZFP lossy (and lossless) compression of numeric arrays (H5Z-ZFP;
|
||||
/// hdf5plugin's `Zfp`). Read-only, with the `zfp` feature.
|
||||
pub const FILTER_ZFP: u16 = 32013;
|
||||
/// Blosc 2 (hdf5plugin's `Blosc2`).
|
||||
pub const FILTER_BLOSC2: u16 = 32026;
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
//! build: the HDF5 standard filters (deflate, shuffle, Fletcher32, szip,
|
||||
//! N-Bit, scale-offset) and the plugin filters whose cargo features are
|
||||
//! enabled (LZ4, Zstandard, pcodec, LZF, bitshuffle, bzip2, blosc,
|
||||
//! blosc2).
|
||||
//! blosc2, zfp).
|
||||
//! [`builtin_filters`] lists them.
|
||||
//! * **Registered filters** (`std` only) — codecs the application supplies
|
||||
//! for any other ID with [`register_filter`] (a [`FilterCodec`], or just a
|
||||
@@ -162,7 +162,7 @@ pub fn known_filter(id: u16) -> Option<(&'static str, Option<&'static str>)> {
|
||||
32001 => ("Blosc", Some("blosc")),
|
||||
32004 => ("LZ4", Some("lz4")),
|
||||
32008 => ("bitshuffle", Some("bitshuffle")),
|
||||
32013 => ("ZFP", None),
|
||||
32013 => ("ZFP", Some("zfp")),
|
||||
32015 => ("Zstandard", Some("zstd")),
|
||||
32019 => ("JPEG", None),
|
||||
32022 => ("BitGroom", None),
|
||||
@@ -455,8 +455,10 @@ pub(crate) mod tests {
|
||||
let msg = FormatError::UnsupportedFilter(32026).to_string();
|
||||
assert!(msg.contains("Blosc2") && msg.contains("`blosc2`"), "{msg}");
|
||||
let msg = FormatError::UnsupportedFilter(32013).to_string();
|
||||
assert!(msg.contains("ZFP") && msg.contains("`zfp`"), "{msg}");
|
||||
let msg = FormatError::UnsupportedFilter(32019).to_string();
|
||||
assert!(
|
||||
msg.contains("ZFP") && msg.contains("not implemented"),
|
||||
msg.contains("JPEG") && msg.contains("not implemented"),
|
||||
"{msg}"
|
||||
);
|
||||
let msg = FormatError::UnsupportedFilter(32000).to_string();
|
||||
|
||||
@@ -434,6 +434,13 @@ pub(crate) static BUILTIN_FILTERS: &[BuiltinFilter] = &[
|
||||
decode: crate::filters_bitshuffle::bitshuffle_decode,
|
||||
encode: Some(crate::filters_bitshuffle::bitshuffle_encode),
|
||||
},
|
||||
#[cfg(feature = "zfp")]
|
||||
BuiltinFilter {
|
||||
id: crate::filter_pipeline::FILTER_ZFP,
|
||||
name: "zfp",
|
||||
decode: crate::filters_zfp::zfp_decode,
|
||||
encode: None,
|
||||
},
|
||||
#[cfg(feature = "zstd")]
|
||||
BuiltinFilter {
|
||||
id: FILTER_ZSTD,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -95,6 +95,8 @@ mod filters_bzip2;
|
||||
#[cfg(feature = "lzf")]
|
||||
pub mod filters_lzf;
|
||||
mod filters_szip;
|
||||
#[cfg(feature = "zfp")]
|
||||
pub mod filters_zfp;
|
||||
pub mod fixed_array;
|
||||
pub mod float16;
|
||||
pub mod fractal_heap;
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
//! Crafted ZFP filter parameters and streams cannot make the decoder panic
|
||||
//! or allocate out of proportion to the chunk it decodes.
|
||||
//!
|
||||
//! The field size comes from the filter's `cd_values` (up to 2^48 values),
|
||||
//! not from the chunk: the decoder allocates the output only when it matches
|
||||
//! the chunk's size (or, when that is unknown, is within the 256 MiB
|
||||
//! ceiling), and only when the stream is long enough to hold a bit per
|
||||
//! block. Peak heap use is measured with a counting global allocator; the
|
||||
//! tests share it, so each holds `SERIAL` for its whole run.
|
||||
#![cfg(feature = "zfp")]
|
||||
|
||||
use std::alloc::{GlobalAlloc, Layout, System};
|
||||
use std::sync::Mutex;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
|
||||
use clawhdf5_format::filters_zfp::zfp_decompress;
|
||||
|
||||
struct Counting;
|
||||
|
||||
static CURRENT: AtomicUsize = AtomicUsize::new(0);
|
||||
static PEAK: AtomicUsize = AtomicUsize::new(0);
|
||||
static SERIAL: Mutex<()> = Mutex::new(());
|
||||
|
||||
unsafe impl GlobalAlloc for Counting {
|
||||
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
|
||||
let p = unsafe { System.alloc(layout) };
|
||||
if !p.is_null() {
|
||||
let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
|
||||
PEAK.fetch_max(now, Ordering::Relaxed);
|
||||
}
|
||||
p
|
||||
}
|
||||
|
||||
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
|
||||
let p = unsafe { System.alloc_zeroed(layout) };
|
||||
if !p.is_null() {
|
||||
let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
|
||||
PEAK.fetch_max(now, Ordering::Relaxed);
|
||||
}
|
||||
p
|
||||
}
|
||||
|
||||
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
|
||||
unsafe { System.dealloc(ptr, layout) };
|
||||
CURRENT.fetch_sub(layout.size(), Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
|
||||
#[global_allocator]
|
||||
static ALLOC: Counting = Counting;
|
||||
|
||||
/// Bytes allocated at the peak of `f`, above what was live when it started.
|
||||
fn peak_during<T>(f: impl FnOnce() -> T) -> (T, usize) {
|
||||
let base = CURRENT.load(Ordering::Relaxed);
|
||||
PEAK.store(base, Ordering::Relaxed);
|
||||
let out = f();
|
||||
(out, PEAK.load(Ordering::Relaxed).saturating_sub(base))
|
||||
}
|
||||
|
||||
fn lock() -> std::sync::MutexGuard<'static, ()> {
|
||||
SERIAL.lock().unwrap_or_else(|e| e.into_inner())
|
||||
}
|
||||
|
||||
/// What decoding may hold: the output (at most the limit, and at most a
|
||||
/// 4-D block of doubles, 2 KiB, per bit of input), and a little more.
|
||||
fn bound(max_output: usize, input: &[u8]) -> usize {
|
||||
let limit = if max_output == 0 {
|
||||
256 << 20
|
||||
} else {
|
||||
max_output
|
||||
};
|
||||
limit.min(input.len() * 8 * 2048) + 4096
|
||||
}
|
||||
|
||||
/// An LSB-first bit writer.
|
||||
#[derive(Default)]
|
||||
struct Bits {
|
||||
v: Vec<u8>,
|
||||
n: usize,
|
||||
}
|
||||
|
||||
impl Bits {
|
||||
fn put(&mut self, x: u64, bits: usize) {
|
||||
for i in 0..bits {
|
||||
if self.n.is_multiple_of(8) {
|
||||
self.v.push(0);
|
||||
}
|
||||
if (x >> i) & 1 == 1 {
|
||||
*self.v.last_mut().unwrap() |= 1 << (self.n % 8);
|
||||
}
|
||||
self.n += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// H5Z-ZFP `cd_values`: a version word and a zfp header for a field of
|
||||
/// `ztype` (0 int32, 1 int64, 2 float, 3 double) and sizes `n` (fastest
|
||||
/// first), with `mode` (12 bits, or 64 when `long`).
|
||||
fn cd_values(ztype: u64, n: &[u64], mode: u64, long: bool) -> Vec<u32> {
|
||||
let mut b = Bits::default();
|
||||
for c in b"zfp" {
|
||||
b.put(*c as u64, 8);
|
||||
}
|
||||
b.put(5, 8);
|
||||
let dims = n.len();
|
||||
let mut meta = 0u64;
|
||||
let bits = [48, 24, 16, 12][dims - 1];
|
||||
for &v in n.iter().rev() {
|
||||
meta = (meta << bits) + v - 1;
|
||||
}
|
||||
meta = (meta << 2) + dims as u64 - 1;
|
||||
meta = (meta << 2) + ztype;
|
||||
b.put(meta, 52);
|
||||
b.put(mode, if long { 64 } else { 12 });
|
||||
b.v.resize(b.v.len().div_ceil(4) * 4, 0);
|
||||
let mut cd = vec![0x1001_1111u32];
|
||||
cd.extend(
|
||||
b.v.chunks(4)
|
||||
.map(|w| u32::from_le_bytes(w.try_into().unwrap())),
|
||||
);
|
||||
cd
|
||||
}
|
||||
|
||||
fn elem(ztype: u64) -> usize {
|
||||
if ztype & 1 == 0 { 4 } else { 8 }
|
||||
}
|
||||
|
||||
/// A 1-D field of 2^32 doubles (32 GiB) in a 1-byte chunk: refused for
|
||||
/// its size, with or without the chunk size known, before anything is
|
||||
/// allocated.
|
||||
#[test]
|
||||
fn huge_fields_are_refused_without_allocating() {
|
||||
let _g = lock();
|
||||
for (n, ztype) in [
|
||||
(vec![1u64 << 32], 3),
|
||||
(vec![1 << 24, 1 << 24], 3),
|
||||
(vec![4096; 4], 1),
|
||||
] {
|
||||
let cd = cd_values(ztype, &n, 2176, false);
|
||||
for max_output in [0usize, 1 << 20] {
|
||||
let (r, peak) = peak_during(|| zfp_decompress(&[0xff], &cd, max_output));
|
||||
assert!(r.is_err(), "{n:?}: decoded {:?} bytes", r.map(|v| v.len()));
|
||||
assert!(peak < 4096, "{n:?}: peak {peak} bytes");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// A field of the chunk's size whose stream is too short for its blocks
|
||||
/// is refused before the output is allocated.
|
||||
#[test]
|
||||
fn short_streams_are_refused_before_allocating() {
|
||||
let _g = lock();
|
||||
let n = [1u64 << 18];
|
||||
let cd = cd_values(2, &n, 2176, false);
|
||||
let size = (1 << 18) * 4;
|
||||
let (r, peak) = peak_during(|| zfp_decompress(&[0u8; 100], &cd, size));
|
||||
assert!(r.is_err());
|
||||
assert!(peak < 4096, "peak {peak} bytes");
|
||||
// A stream with a bit per block: all-zero blocks, which decode.
|
||||
let input = vec![0u8; (1 << 16) / 8];
|
||||
let out = zfp_decompress(&input, &cd, size).unwrap();
|
||||
assert_eq!(out, vec![0u8; size]);
|
||||
}
|
||||
|
||||
/// xorshift64*: deterministic, so a failure reproduces.
|
||||
struct Rng(u64);
|
||||
|
||||
impl Rng {
|
||||
fn next(&mut self) -> u64 {
|
||||
let mut x = self.0;
|
||||
x ^= x >> 12;
|
||||
x ^= x << 25;
|
||||
x ^= x >> 27;
|
||||
self.0 = x;
|
||||
x.wrapping_mul(0x2545_F491_4F6C_DD1D)
|
||||
}
|
||||
|
||||
fn below(&mut self, n: u64) -> u64 {
|
||||
self.next() % n.max(1)
|
||||
}
|
||||
}
|
||||
|
||||
/// A mode word: one of the four short forms, or the 64-bit expert form
|
||||
/// with parameters at and past their edges (maxbits below a float block's
|
||||
/// exponent, minbits past maxbits, precision 0, minexp at the reversible
|
||||
/// boundary).
|
||||
fn mode(rng: &mut Rng) -> (u64, bool) {
|
||||
match rng.below(6) {
|
||||
0 => (rng.below(2048), false),
|
||||
1 => (2048 + rng.below(128), false),
|
||||
2 => (2176, false),
|
||||
3 => (2177 + rng.below(4094 - 2177 + 1), false),
|
||||
_ => {
|
||||
fn pick(rng: &mut Rng, v: [u64; 6]) -> u64 {
|
||||
v[rng.below(6) as usize]
|
||||
}
|
||||
let r = [rng.below(2000), rng.below(0x8000), rng.below(40)];
|
||||
let minbits = pick(rng, [0, 1, 11, r[0], r[1], 1]);
|
||||
let r = [rng.below(0x8000), rng.below(40)];
|
||||
let maxbits = pick(rng, [minbits, minbits + r[1], 7, 11, r[0], 0x7fff]);
|
||||
let maxprec = rng.below(0x80);
|
||||
let r = [rng.below(0x8000), rng.below(400)];
|
||||
let minexp = pick(
|
||||
rng,
|
||||
[
|
||||
r[0],
|
||||
16495 - 1074,
|
||||
16495 - 1075,
|
||||
16495 + r[1] - 200,
|
||||
16495 - 1074,
|
||||
r[0],
|
||||
],
|
||||
);
|
||||
let m = ((((minexp << 7) + maxprec) << 15) + maxbits) << 15;
|
||||
((m + minbits) << 12 | 0xfff, true)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Random fields, modes and streams (random bytes, runs of ones, mostly
|
||||
/// zeros; of every length), decoded with the chunk size known and not,
|
||||
/// and with header words that are random too.
|
||||
#[test]
|
||||
fn fuzzed_headers_and_streams_stay_within_the_allocation_bound() {
|
||||
let _g = lock();
|
||||
let mut rng = Rng(0x2f9);
|
||||
let mut decoded = 0;
|
||||
for i in 0..20_000 {
|
||||
let ztype = rng.below(4);
|
||||
let dims = 1 + rng.below(4) as usize;
|
||||
let max = [300, 40, 14, 8][dims - 1];
|
||||
let n: Vec<u64> = (0..dims).map(|_| 1 + rng.below(max)).collect();
|
||||
let (m, long) = mode(&mut rng);
|
||||
let mut cd = cd_values(ztype, &n, m, long);
|
||||
if rng.below(10) == 0 {
|
||||
let at = rng.below(cd.len() as u64) as usize;
|
||||
cd[at] ^= 1 << rng.below(32);
|
||||
}
|
||||
if rng.below(20) == 0 {
|
||||
cd.truncate(rng.below(cd.len() as u64 + 1) as usize);
|
||||
}
|
||||
let len = match rng.below(4) {
|
||||
0 => rng.below(8),
|
||||
1 => rng.below(300),
|
||||
_ => rng.below(20_000),
|
||||
} as usize;
|
||||
let input: Vec<u8> = match rng.below(3) {
|
||||
0 => (0..len).map(|_| rng.next() as u8).collect(),
|
||||
1 => (0..len)
|
||||
.map(|_| [0, 0xff, rng.next() as u8][rng.below(3) as usize])
|
||||
.collect(),
|
||||
_ => (0..len)
|
||||
.map(|_| [0, 0, 0, 1, 0x80, rng.next() as u8][rng.below(6) as usize])
|
||||
.collect(),
|
||||
};
|
||||
let size = n.iter().product::<u64>() as usize * elem(ztype);
|
||||
let max_output = if rng.below(4) == 0 { 0 } else { size };
|
||||
let (r, peak) = peak_during(|| zfp_decompress(&input, &cd, max_output));
|
||||
if let Ok(out) = &r {
|
||||
decoded += 1;
|
||||
if max_output != 0 {
|
||||
assert_eq!(out.len(), max_output, "iteration {i}");
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
peak <= bound(max_output, &input),
|
||||
"iteration {i}: peak {peak} bytes for {n:?} from {} bytes ({:?})",
|
||||
input.len(),
|
||||
r.map(|v| v.len())
|
||||
);
|
||||
}
|
||||
// Most inputs are streams zfp decodes without running out.
|
||||
assert!(decoded > 5_000, "only {decoded} decoded");
|
||||
}
|
||||
@@ -53,8 +53,10 @@ bitshuffle = ["clawhdf5-format/bitshuffle"]
|
||||
bzip2 = ["clawhdf5-format/bzip2"]
|
||||
blosc = ["clawhdf5-format/blosc"]
|
||||
blosc2 = ["clawhdf5-format/blosc2"]
|
||||
# ZFP (32013), read-only.
|
||||
zfp = ["clawhdf5-format/zfp"]
|
||||
# Every plugin filter.
|
||||
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2"]
|
||||
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2", "zfp"]
|
||||
# Dataset::verify_provenance() — recompute a dataset's SHA-256 and compare
|
||||
# against its stored _provenance_sha256 attribute. On by default, matching
|
||||
# clawhdf5-format's own default-on `provenance` feature.
|
||||
|
||||
@@ -495,16 +495,15 @@ fn lzf_written_by_clawhdf5_reads_in_h5py() {
|
||||
});
|
||||
}
|
||||
|
||||
/// ZFP is not implemented, and Blosc2 is not in a build without the
|
||||
/// `blosc2` feature: reading them must be a clear error naming the filter,
|
||||
/// never data.
|
||||
/// Blosc2 and ZFP are not in a build without their features: reading them
|
||||
/// must be a clear error naming the filter, never data.
|
||||
#[test]
|
||||
fn unimplemented_filters_are_a_clear_error() {
|
||||
fn filters_left_out_of_the_build_are_a_clear_error() {
|
||||
if !have_python("h5py, hdf5plugin") {
|
||||
return;
|
||||
}
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("unimplemented.h5");
|
||||
let path = dir.path().join("left_out.h5");
|
||||
run_python(
|
||||
r#"
|
||||
import sys
|
||||
@@ -517,7 +516,10 @@ with h5py.File(sys.argv[1], 'w') as f:
|
||||
&[path.to_str().unwrap()],
|
||||
);
|
||||
let file = File::open(&path).unwrap();
|
||||
let mut missing = vec![("zfp", 32013u16, "ZFP")];
|
||||
let mut missing = Vec::new();
|
||||
if !cfg!(feature = "zfp") {
|
||||
missing.push(("zfp", 32013u16, "ZFP"));
|
||||
}
|
||||
if !cfg!(feature = "blosc2") {
|
||||
missing.push(("blosc2", 32026, "Blosc2"));
|
||||
}
|
||||
@@ -526,7 +528,7 @@ with h5py.File(sys.argv[1], 'w') as f:
|
||||
.dataset(name)
|
||||
.unwrap()
|
||||
.read_selection(&Selection::All)
|
||||
.expect_err("an unimplemented filter must not read");
|
||||
.expect_err("a filter left out of the build must not read");
|
||||
let msg = err.to_string();
|
||||
assert!(
|
||||
msg.contains(&id.to_string()) && msg.contains(label),
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
//! ZFP (H5Z-ZFP, filter 32013) against libhdf5 + libzfp.
|
||||
//!
|
||||
//! h5py with hdf5plugin (H5Z-ZFP 1.1.1, zfp 1.0.1) writes datasets in every
|
||||
//! ZFP mode (fixed rate, precision and accuracy, reversible, expert) for
|
||||
//! each type ZFP supports (int32, int64, float, double), in 1 to 4
|
||||
//! dimensions, with chunks that are partial at the dataset's edges, blocks
|
||||
//! that are partial at the chunks' edges, and chunks with unit dimensions
|
||||
//! (a lower-dimensional ZFP field). Next to each it stores what h5py reads
|
||||
//! back, unfiltered, and clawhdf5 must read the ZFP dataset bit for bit
|
||||
//! equal to that: the decoder is deterministic, so lossy modes have one
|
||||
//! right answer.
|
||||
//!
|
||||
//! Skipped when python3 with h5py and hdf5plugin is unavailable, unless
|
||||
//! `CLAWHDF5_REQUIRE_INTEROP=1`.
|
||||
#![cfg(feature = "zfp")]
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::process::Command;
|
||||
|
||||
use clawhdf5::File;
|
||||
use clawhdf5_format::selection::Selection;
|
||||
|
||||
fn python() -> String {
|
||||
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
|
||||
}
|
||||
|
||||
fn have_python() -> bool {
|
||||
let ok = Command::new(python())
|
||||
.args(["-c", "import h5py, hdf5plugin"])
|
||||
.output()
|
||||
.map(|o| o.status.success())
|
||||
.unwrap_or(false);
|
||||
if ok {
|
||||
return true;
|
||||
}
|
||||
assert!(
|
||||
!std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1"),
|
||||
"CLAWHDF5_REQUIRE_INTEROP=1 but python3 with h5py and hdf5plugin is not available"
|
||||
);
|
||||
eprintln!("SKIP: python3 with h5py and hdf5plugin not available");
|
||||
false
|
||||
}
|
||||
|
||||
fn run_python(script: &str, args: &[&str]) -> String {
|
||||
let output = Command::new(python())
|
||||
.arg("-c")
|
||||
.arg(script)
|
||||
.args(args)
|
||||
.output()
|
||||
.expect("failed to run python");
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"Python script failed:\nSTDOUT: {}\nSTDERR: {}",
|
||||
String::from_utf8_lossy(&output.stdout),
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
String::from_utf8_lossy(&output.stdout).trim().to_string()
|
||||
}
|
||||
|
||||
/// Writes `f{i}` (ZFP) and `r{i}` (h5py's reading of `f{i}`, unfiltered)
|
||||
/// for every mode x dtype x shape x data kind H5Z-ZFP accepts; each `f{i}`
|
||||
/// has a `case` attribute. Prints the number of pairs.
|
||||
const GENERATE: &str = r#"
|
||||
import sys
|
||||
import numpy as np, h5py, hdf5plugin
|
||||
path = sys.argv[1]
|
||||
MODES = [
|
||||
('rate 2.5', dict(rate=2.5)),
|
||||
('rate 8', dict(rate=8)),
|
||||
('rate 16', dict(rate=16)),
|
||||
('rate 31', dict(rate=31)),
|
||||
('rate 64', dict(rate=64)),
|
||||
('precision 6', dict(precision=6)),
|
||||
('precision 20', dict(precision=20)),
|
||||
('precision 64', dict(precision=64)),
|
||||
('accuracy 0.5', dict(accuracy=0.5)),
|
||||
('accuracy 1e-3', dict(accuracy=1e-3)),
|
||||
('accuracy 1e-12', dict(accuracy=1e-12)),
|
||||
('reversible', dict(reversible=True)),
|
||||
('expert', dict(minbits=24, maxbits=600, maxprec=30, minexp=-20)),
|
||||
# maxbits past ZFP_MAX_BITS: H5Z-ZFP stores it as fixed precision.
|
||||
('expert maxbits 20000', dict(minbits=1, maxbits=20000, maxprec=40, minexp=-1074)),
|
||||
# A budget of 10 bits a block: one bit left after a float's exponent.
|
||||
# (Doubles are left out: 12 bits of exponent and flag overrun it, and
|
||||
# libzfp's encoder then writes past its buffer.)
|
||||
('expert maxbits 10', dict(minbits=5, maxbits=10, maxprec=64, minexp=-100)),
|
||||
('expert minbits 900', dict(minbits=900, maxbits=4000, maxprec=64, minexp=-60)),
|
||||
]
|
||||
DTYPES = ['<f4', '<f8', '<i4', '<i8']
|
||||
SHAPES = [
|
||||
((37,), (16,)),
|
||||
((13, 22), (5, 8)),
|
||||
((9, 7, 6), (4, 5, 6)),
|
||||
((5, 6, 7, 3), (3, 5, 6, 3)),
|
||||
((4, 1, 30), (2, 1, 30)),
|
||||
((3, 10, 1, 9), (2, 10, 1, 9)),
|
||||
((64, 64), (64, 64)),
|
||||
]
|
||||
KINDS = ['smooth', 'noise', 'wide', 'zeros', 'special']
|
||||
rng = np.random.default_rng(13)
|
||||
|
||||
def data(dt, shape, kind):
|
||||
n = int(np.prod(shape))
|
||||
d = np.dtype(dt)
|
||||
idx = np.arange(n, dtype=np.float64)
|
||||
if d.kind == 'f':
|
||||
if kind == 'smooth':
|
||||
v = 100 * np.sin(idx / 5.0) + idx / 3.0
|
||||
elif kind == 'noise':
|
||||
v = rng.normal(size=n) * 10.0 ** rng.uniform(-3, 3, n)
|
||||
elif kind == 'wide':
|
||||
lo, hi = (-150, 120) if d.itemsize == 4 else (-1075, 1000)
|
||||
v = rng.choice([-1.0, 1.0], n) * np.exp2(rng.integers(lo, hi, n).astype(np.float64))
|
||||
v[rng.random(n) < 0.1] = 0.0
|
||||
elif kind == 'zeros':
|
||||
v = np.zeros(n)
|
||||
v[: n // 3] = 0.0
|
||||
else:
|
||||
v = rng.normal(size=n) * 1e3
|
||||
v[rng.random(n) < 0.05] = np.inf
|
||||
v[rng.random(n) < 0.05] = -np.inf
|
||||
v[rng.random(n) < 0.05] = np.nan
|
||||
v[rng.random(n) < 0.05] = -0.0
|
||||
with np.errstate(over='ignore'):
|
||||
return v.astype(dt).reshape(shape)
|
||||
info = np.iinfo(d)
|
||||
if kind == 'smooth':
|
||||
v = ((idx * 7) % 1000 - 500).astype(np.int64)
|
||||
elif kind == 'noise':
|
||||
v = rng.integers(info.min, info.max, n, dtype=np.int64, endpoint=True)
|
||||
elif kind == 'wide':
|
||||
bits = rng.integers(0, d.itemsize * 8 - 2, n)
|
||||
v = rng.integers(-(2 ** 20), 2 ** 20, n) << np.minimum(bits, d.itemsize * 8 - 22)
|
||||
elif kind == 'zeros':
|
||||
v = np.zeros(n, dtype=np.int64)
|
||||
else:
|
||||
v = rng.choice([info.min, info.max, 0, -1, 1], n)
|
||||
return v.astype(dt).reshape(shape)
|
||||
|
||||
# Written first and read back after the file is closed: h5py reads a chunk
|
||||
# still in libhdf5's chunk cache without decoding it.
|
||||
i = 0
|
||||
with h5py.File(path, 'w') as f:
|
||||
for label, kw in MODES:
|
||||
for dt in DTYPES:
|
||||
if label == 'expert maxbits 10' and dt == '<f8':
|
||||
continue
|
||||
for shape, chunks in SHAPES:
|
||||
for kind in KINDS:
|
||||
v = data(dt, shape, kind)
|
||||
name = f'f{i}'
|
||||
try:
|
||||
ds = f.create_dataset(name, data=v, chunks=chunks, **hdf5plugin.Zfp(**kw))
|
||||
except Exception:
|
||||
continue
|
||||
ds.attrs['case'] = f'{label}|{dt}|{shape}|{chunks}|{kind}'
|
||||
i += 1
|
||||
with h5py.File(path, 'a') as f:
|
||||
for k in range(i):
|
||||
f.create_dataset(f'r{k}', data=f[f'f{k}'][()])
|
||||
print(i)
|
||||
"#;
|
||||
|
||||
#[test]
|
||||
fn every_mode_reads_bit_exact() {
|
||||
if !have_python() {
|
||||
return;
|
||||
}
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("zfp.h5");
|
||||
let n: usize = run_python(GENERATE, &[path.to_str().unwrap()])
|
||||
.parse()
|
||||
.unwrap();
|
||||
let file = File::open(&path).unwrap();
|
||||
// mode -> dtypes read
|
||||
let mut seen: BTreeMap<String, Vec<String>> = BTreeMap::new();
|
||||
let mut failures = Vec::new();
|
||||
for i in 0..n {
|
||||
let ds = file.dataset(&format!("f{i}")).unwrap();
|
||||
let case = match ds.attrs().unwrap().get("case") {
|
||||
Some(clawhdf5::AttrValue::String(s)) => s.clone(),
|
||||
other => panic!("f{i}: case attribute {other:?}"),
|
||||
};
|
||||
let want = file
|
||||
.dataset(&format!("r{i}"))
|
||||
.unwrap()
|
||||
.read_selection(&Selection::All)
|
||||
.unwrap();
|
||||
match ds.read_selection(&Selection::All) {
|
||||
Ok(got) if got == want => {
|
||||
let mut parts = case.split('|');
|
||||
let mode = parts.next().unwrap().to_string();
|
||||
let dt = parts.next().unwrap().to_string();
|
||||
let e = seen.entry(mode).or_default();
|
||||
if !e.contains(&dt) {
|
||||
e.push(dt);
|
||||
}
|
||||
}
|
||||
Ok(got) => {
|
||||
let first = got.iter().zip(&want).position(|(a, b)| a != b);
|
||||
failures.push(format!("f{i} {case}: differs from byte {first:?}"));
|
||||
}
|
||||
Err(e) => failures.push(format!("f{i} {case}: {e}")),
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
failures.is_empty(),
|
||||
"{} of {n} datasets:\n{}",
|
||||
failures.len(),
|
||||
failures.join("\n")
|
||||
);
|
||||
// Every mode was exercised on every type H5Z-ZFP accepts it for.
|
||||
for (mode, dts) in &seen {
|
||||
assert!(dts.len() >= 2, "{mode}: only {dts:?}");
|
||||
}
|
||||
assert_eq!(seen.len(), 16, "{:?}", seen.keys());
|
||||
eprintln!("{n} ZFP datasets bit-exact; modes and types: {seen:?}");
|
||||
}
|
||||
|
||||
/// A header written on a big-endian machine: H5Z-ZFP finds the magic only
|
||||
/// after byte-swapping the `cd_values`, and then byte-swaps the decoded
|
||||
/// values, since the dataset's datatype is big-endian there. The file is
|
||||
/// made by swapping the header words of a little-endian one in place, so
|
||||
/// the datatype stays little-endian and libhdf5 reads the values swapped;
|
||||
/// clawhdf5 must read the same bytes.
|
||||
#[test]
|
||||
fn big_endian_header_swaps_the_values() {
|
||||
if !have_python() {
|
||||
return;
|
||||
}
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("zfp_be.h5");
|
||||
let n: usize = run_python(
|
||||
r#"
|
||||
import sys, struct
|
||||
import numpy as np, h5py, hdf5plugin
|
||||
path = sys.argv[1]
|
||||
cases = [('<f4', dict(rate=12)), ('<f8', dict(precision=30)), ('<i4', dict(reversible=True)),
|
||||
('<i8', dict(accuracy=4))]
|
||||
cds = []
|
||||
with h5py.File(path, 'w') as f:
|
||||
for k, (dt, kw) in enumerate(cases):
|
||||
v = (np.arange(150) * 13 % 97 - 40).reshape(10, 15).astype(dt)
|
||||
ds = f.create_dataset(f'f{k}', data=v, chunks=(4, 8), **hdf5plugin.Zfp(**kw))
|
||||
cds.append(ds.id.get_create_plist().get_filter(0)[2])
|
||||
raw = bytearray(open(path, 'rb').read())
|
||||
for cd in cds:
|
||||
le = struct.pack(f'<{len(cd)}I', *cd)
|
||||
swapped = le[:4] + struct.pack(f'>{len(cd) - 1}I', *cd[1:])
|
||||
at = raw.find(le)
|
||||
assert at > 0 and raw.find(le, at + 1) < 0, 'cd_values not found once'
|
||||
raw[at:at + len(le)] = swapped
|
||||
open(path, 'wb').write(raw)
|
||||
with h5py.File(path, 'a') as f:
|
||||
for k in range(len(cases)):
|
||||
f.create_dataset(f'r{k}', data=f[f'f{k}'][()])
|
||||
f.create_dataset(f'o{k}', data=f[f'f{k}'][()].byteswap())
|
||||
print(len(cases))
|
||||
"#,
|
||||
&[path.to_str().unwrap()],
|
||||
)
|
||||
.parse()
|
||||
.unwrap();
|
||||
let file = File::open(&path).unwrap();
|
||||
for k in 0..n {
|
||||
let read = |name: String| {
|
||||
file.dataset(&name)
|
||||
.unwrap()
|
||||
.read_selection(&Selection::All)
|
||||
.unwrap_or_else(|e| panic!("{name}: {e}"))
|
||||
};
|
||||
let got = read(format!("f{k}"));
|
||||
assert!(got == read(format!("r{k}")), "f{k}: differs from h5py");
|
||||
// Byte-swapped back, the values are the right ones.
|
||||
assert!(got != read(format!("o{k}")), "f{k}: not swapped");
|
||||
}
|
||||
}
|
||||
@@ -293,10 +293,14 @@ fill-value item that did is fixed).
|
||||
697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using
|
||||
dictionaries, lazy chunks, variable-length blocks, user-defined codecs or
|
||||
registered filters (e.g. bytedelta) are refused with an error.
|
||||
**Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that
|
||||
names the filter, and can be plugged in with
|
||||
`filter_registry::register_filter` (32023, Granular BitRound, too, since
|
||||
2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2.
|
||||
**Fixed 2026-09-26** for ZFP (32013, `zfp` feature, also in
|
||||
`plugin-filters`), read only: every H5Z-ZFP mode and type, bit-exact
|
||||
against h5py + hdf5plugin 7.1 (`crates/clawhdf5/tests/zfp_interop.rs`);
|
||||
h5ex_d_zfp now reads (conformance 600 of 697 ok, tank,
|
||||
`conformance/run.sh --no-fetch`). **Still open:** clawhdf5 cannot write
|
||||
Blosc2 or ZFP. Other filters (32023, Granular BitRound, too, since
|
||||
2026-09-26 even with the `pcodec` feature) can be plugged in with
|
||||
`filter_registry::register_filter`.
|
||||
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
|
||||
read with zeros for the missing bytes** (any filter; found reviewing the plugin
|
||||
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
|
||||
|
||||
+5
-4
@@ -67,7 +67,7 @@ run_step "cargo clippy --all-targets" cargo clippy \
|
||||
|
||||
# 3. Clippy over clawhdf5-format's optional features, which the default
|
||||
# workspace build never compiles (szip is left out: it needs libaec).
|
||||
# plugin-filters = bitshuffle, bzip2, blosc, blosc2 (and the default-on lzf).
|
||||
# plugin-filters = bitshuffle, bzip2, blosc, blosc2, zfp (and the default-on lzf).
|
||||
run_step "cargo clippy (format feature matrix)" cargo clippy \
|
||||
-p clawhdf5-format \
|
||||
--all-targets \
|
||||
@@ -78,7 +78,7 @@ run_step "cargo clippy (format feature matrix)" cargo clippy \
|
||||
# dependencies (bitshuffle and blosc share code).
|
||||
plugin_filters_alone() {
|
||||
local f
|
||||
for f in bitshuffle bzip2 blosc blosc2; do
|
||||
for f in bitshuffle bzip2 blosc blosc2 zfp; do
|
||||
echo "--- $f"
|
||||
cargo clippy -p clawhdf5-format --all-targets --features "$f" -- -D warnings || return 1
|
||||
done
|
||||
@@ -210,9 +210,10 @@ if "$PYTHON" -c "import h5py" >/dev/null 2>&1 || [ "${CLAWHDF5_REQUIRE_INTEROP:-
|
||||
# libhdf5's registered plugins) compile and run too.
|
||||
run_step "h5py interop (format, ignored tests)" cargo test \
|
||||
-p clawhdf5-format --features lz4,zstd --test writer_h5py_tests -- --include-ignored
|
||||
# LZF, bitshuffle, bzip2 and Blosc both ways against h5py + hdf5plugin.
|
||||
# LZF, bitshuffle, bzip2 and Blosc both ways against h5py + hdf5plugin;
|
||||
# Blosc2 and ZFP (read-only) against what h5py reads.
|
||||
run_step "h5py interop (plugin filters)" cargo test \
|
||||
-p clawhdf5 --features plugin-filters --test plugin_filters_interop
|
||||
-p clawhdf5 --features plugin-filters --test plugin_filters_interop --test zfp_interop
|
||||
else
|
||||
echo ""
|
||||
echo "==> [h5py interop] SKIPPED: no h5py in $PYTHON"
|
||||
|
||||
Reference in New Issue
Block a user