Merge branch 'feat/p2b-blosc2' into feat/p2b-scale
# Conflicts: # CHANGELOG.md
This commit is contained in:
osobh
@@ -89,6 +89,41 @@
|
||||
`names_whose_hashes_collide_are_found_by_name` in
|
||||
`crates/clawhdf5/tests/writer_groups_interop.rs`.
|
||||
|
||||
### Blosc2 (2026-09-26)
|
||||
- **Blosc2 (filter 32026) reads, in pure Rust.** Files written with
|
||||
hdf5plugin's `Blosc2` failed with `UnsupportedFilter`. New feature
|
||||
`blosc2` (`clawhdf5-format` and `clawhdf5`, included in `plugin-filters`)
|
||||
decodes the Blosc2 contiguous frame hdf5-blosc2 stores per chunk, the
|
||||
B2ND arrays it uses for chunks of 2 or more dimensions (blocks gathered
|
||||
back into C order), Blosc2 chunks with their special values (zeros, NaN,
|
||||
uninitialised, one repeated value), and the shuffle, bit-shuffle, delta
|
||||
and truncate-precision filters, over the BloscLZ, LZ4/LZ4HC, Zlib and
|
||||
Zstandard codecs shared with Blosc 1. Read only: there is no Blosc2
|
||||
encoder. Dictionaries, lazy chunks, variable-length blocks, user-defined
|
||||
codecs and registered Blosc2 filters (e.g. bytedelta) are errors;
|
||||
uninitialised chunks read as zeros. Tested against h5py 3.16 +
|
||||
hdf5plugin 7.1 (every codec, filter and level 0-9; 1- to 5-D chunks with
|
||||
partial edge chunks; every integer width and f4/f8; datasets of zeros,
|
||||
one value and NaN; Fletcher32 before Blosc2) and against frames from
|
||||
python-blosc2 4.13.1 for what hdf5plugin never writes
|
||||
(`crates/clawhdf5-format/tests/fixtures/blosc2/`); the decoder is
|
||||
fuzzed. Conformance: 576 of 697 files ok (was 575) — h5ex_d_blosc2.
|
||||
- **A crafted Blosc2 chunk cannot allocate more than a few times its HDF5
|
||||
chunk size.** Found in review before release: the sizes a frame declares
|
||||
sized the decoder's buffers. A 173-byte frame whose offsets chunk claimed
|
||||
2 GiB was decoded in full for a 1 MiB chunk; an empty chunk allocated its
|
||||
declared block size twice (about 1 GiB); a B2ND chunk was decoded whole
|
||||
with its padding (up to 16x the chunk); and a Zstandard stream's declared
|
||||
window (up to 100 MiB) was reserved as the decoder was reused, which also
|
||||
affected Blosc 1 and bitshuffle with Zstandard. Now the offsets chunk is
|
||||
capped at the chunk size, block sizes are clamped to the chunk, B2ND
|
||||
blocks are placed as they are decoded (padding is never held), and the
|
||||
Zstandard window is capped at twice the stream's output (at least
|
||||
128 KiB). A B2ND chunk may no longer be larger than its array, which
|
||||
hdf5-blosc2 never writes. `tests/blosc2_alloc_bounds.rs` measures peak
|
||||
allocation for these frames and for 45,000 fuzzed ones: at most 6x the
|
||||
chunk size, twice the input and 2 MiB of Zstandard state.
|
||||
|
||||
### Concurrent reads (2026-09-26)
|
||||
- **Full reads of chunked datasets scale with threads again when rayon's
|
||||
pool has one thread.** Each full read handed its chunks to rayon to
|
||||
@@ -459,6 +494,7 @@
|
||||
missing feature ("unsupported filter: 32026 (Blosc2, not implemented by
|
||||
clawhdf5)").
|
||||
- **Not implemented:** Blosc2 (32026) and ZFP (32013) remain a clear error.
|
||||
(Blosc2 reads since the `blosc2` feature, see above.)
|
||||
- **Wrong data: a chunk that decodes short read as zeros** (pre-existing, every
|
||||
filter). HDF5 stores every chunk at the full chunk size, so a filter
|
||||
pipeline that decodes to fewer bytes means a corrupt chunk; every chunk
|
||||
|
||||
@@ -11,7 +11,7 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
|-------|------|
|
||||
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
||||
| `clawhdf5-io` | Read/write implementation |
|
||||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1) live in `clawhdf5-format`. No Blosc2 or ZFP. |
|
||||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 read-only) live in `clawhdf5-format`. No ZFP. |
|
||||
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
||||
| `clawhdf5` | Main facade crate |
|
||||
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
||||
|
||||
@@ -668,7 +668,7 @@ clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
├── Core HDF5
|
||||
│ ├── clawhdf5-format — Binary parser/writer (no_std-capable), shared type definitions
|
||||
│ ├── clawhdf5-io — I/O abstraction (file/memory readers; optional mmap, async, HSDS, MPI)
|
||||
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc filters live in clawhdf5-format
|
||||
│ ├── clawhdf5-filters — Fast deflate path (zlib-ng); the filter registry and the lz4/zstd/pcodec/szip/LZF/bitshuffle/bzip2/Blosc/Blosc2 filters live in clawhdf5-format
|
||||
│ ├── clawhdf5-derive — Proc macros
|
||||
│ ├── clawhdf5 — High-level API
|
||||
│ ├── clawhdf5-netcdf4 — NetCDF-4 support
|
||||
@@ -779,13 +779,15 @@ stores keep their setting. Opt out with `float16 = false` or
|
||||
| `bitshuffle` | no | Bitshuffle filter (id 32008) with its LZ4 and Zstandard modes: read and write. Pure Rust (lz4_flex, ruzstd) |
|
||||
| `bzip2` | no | bzip2 filter (id 307): read and write. Pure Rust (the `bzip2` crate's libbz2-rs-sys backend compiles no C) |
|
||||
| `blosc` | no | Blosc 1 filter (id 32001): reads BloscLZ, LZ4/LZ4HC, Snappy, Zlib and Zstandard frames with byte or bit shuffle; writes LZ4, Snappy, Zlib or Zstandard (not BloscLZ). Pure Rust |
|
||||
| `plugin-filters` | no | All four above |
|
||||
| `blosc2` | no | Blosc2 filter (id 32026), read only: hdf5plugin's frames and B2ND (n-D) chunks, BloscLZ, LZ4/LZ4HC, Zlib and Zstandard, with shuffle, bit shuffle, delta or truncated precision. Pure Rust |
|
||||
| `plugin-filters` | no | All five above |
|
||||
|
||||
Blosc2 (32026) and ZFP (32013) are not implemented: reading them fails with
|
||||
`UnsupportedFilter`, whose message names the filter. Any other filter can be
|
||||
supplied at run time with `filter_registry::register_filter` (a decoder
|
||||
closure, or a `FilterCodec` that also encodes). The facade (`clawhdf5`)
|
||||
forwards `lzf`, `bitshuffle`, `bzip2`, `blosc` and `plugin-filters`. Write
|
||||
ZFP (32013) is not implemented: reading it fails with `UnsupportedFilter`,
|
||||
whose message names the filter. clawhdf5 cannot write Blosc2. Any other
|
||||
filter can be supplied at run time with `filter_registry::register_filter` (a
|
||||
decoder closure, or a `FilterCodec` that also encodes). The facade
|
||||
(`clawhdf5`) forwards `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2` and
|
||||
`plugin-filters`. Write
|
||||
with `DatasetBuilder::with_lzf()`, `with_bitshuffle(..)`, `with_bzip2(..)`
|
||||
and `with_blosc(..)`; h5py + hdf5plugin read the result (tested both ways in
|
||||
`crates/clawhdf5/tests/plugin_filters_interop.rs`). The pure-Rust Zstandard
|
||||
|
||||
@@ -76,8 +76,10 @@ bitshuffle = ["lz4_flex", "ruzstd"]
|
||||
bzip2 = ["dep:bzip2", "std"]
|
||||
# Blosc 1 (32001) with its BloscLZ, LZ4, Snappy, Zlib and Zstandard codecs.
|
||||
blosc = ["lz4_flex", "ruzstd", "snap", "deflate", "std"]
|
||||
# Blosc2 (32026), read-only: frames, B2ND arrays, and the Blosc codecs above.
|
||||
blosc2 = ["blosc"]
|
||||
# Every plugin filter above.
|
||||
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc"]
|
||||
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2"]
|
||||
|
||||
[[bench]]
|
||||
name = "parallel_decompress_bench"
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
//! * **Built-in filters** — a static table of the filters compiled into this
|
||||
//! build: the HDF5 standard filters (deflate, shuffle, Fletcher32, szip,
|
||||
//! N-Bit, scale-offset) and the plugin filters whose cargo features are
|
||||
//! enabled (LZ4, Zstandard, pcodec, LZF, bitshuffle, bzip2, blosc).
|
||||
//! enabled (LZ4, Zstandard, pcodec, LZF, bitshuffle, bzip2, blosc,
|
||||
//! blosc2).
|
||||
//! [`builtin_filters`] lists them.
|
||||
//! * **Registered filters** (`std` only) — codecs the application supplies
|
||||
//! for any other ID with [`register_filter`] (a [`FilterCodec`], or just a
|
||||
@@ -166,7 +167,7 @@ pub fn known_filter(id: u16) -> Option<(&'static str, Option<&'static str>)> {
|
||||
32019 => ("JPEG", None),
|
||||
32022 => ("BitGroom", None),
|
||||
32023 => ("Granular BitRound", None),
|
||||
32026 => ("Blosc2", None),
|
||||
32026 => ("Blosc2", Some("blosc2")),
|
||||
_ => return None,
|
||||
})
|
||||
}
|
||||
@@ -452,12 +453,12 @@ pub(crate) mod tests {
|
||||
#[test]
|
||||
fn unsupported_filter_error_names_the_filter() {
|
||||
let msg = FormatError::UnsupportedFilter(32026).to_string();
|
||||
assert!(msg.contains("Blosc2") && msg.contains("`blosc2`"), "{msg}");
|
||||
let msg = FormatError::UnsupportedFilter(32013).to_string();
|
||||
assert!(
|
||||
msg.contains("Blosc2") && msg.contains("not implemented"),
|
||||
msg.contains("ZFP") && msg.contains("not implemented"),
|
||||
"{msg}"
|
||||
);
|
||||
let msg = FormatError::UnsupportedFilter(32013).to_string();
|
||||
assert!(msg.contains("ZFP"), "{msg}");
|
||||
let msg = FormatError::UnsupportedFilter(32000).to_string();
|
||||
assert!(msg.contains("LZF") && msg.contains("`lzf`"), "{msg}");
|
||||
assert_eq!(
|
||||
|
||||
@@ -455,6 +455,13 @@ pub(crate) static BUILTIN_FILTERS: &[BuiltinFilter] = &[
|
||||
},
|
||||
encode: None,
|
||||
},
|
||||
#[cfg(feature = "blosc2")]
|
||||
BuiltinFilter {
|
||||
id: crate::filter_pipeline::FILTER_BLOSC2,
|
||||
name: "blosc2",
|
||||
decode: crate::filters_blosc2::blosc2_decode,
|
||||
encode: None,
|
||||
},
|
||||
];
|
||||
|
||||
/// Decode the HDF5 scale-offset filter (id 6).
|
||||
|
||||
@@ -218,12 +218,20 @@ pub(crate) fn bitshuffle_decode(
|
||||
}
|
||||
|
||||
/// Decode Zstandard frames into exactly `dst`, failing if they hold more.
|
||||
///
|
||||
/// ruzstd reserves a frame's declared window (by default up to 100 MiB)
|
||||
/// before decoding it, so the window is capped at what the output could
|
||||
/// need: twice `dst` (window sizes are rounded up), and at least 128 KiB.
|
||||
/// The encoders behind these filters (c-blosc, c-blosc2, bitshuffle)
|
||||
/// compress each block in one call with its size known, so libzstd's
|
||||
/// window never exceeds the block.
|
||||
#[cfg(any(feature = "bitshuffle", feature = "blosc"))]
|
||||
pub(crate) fn zstd_decode_into(
|
||||
decoder: &mut ruzstd::decoding::FrameDecoder,
|
||||
frames: &[u8],
|
||||
dst: &mut [u8],
|
||||
) -> Result<usize, FormatError> {
|
||||
decoder.set_max_window_size((2 * dst.len()).max(1 << 17) as u64);
|
||||
decoder
|
||||
.decode_all(frames, dst)
|
||||
.map_err(|e| FormatError::DecompressionError(format!("zstd: {e}")))
|
||||
|
||||
@@ -49,7 +49,7 @@ fn le32(b: &[u8], at: usize) -> Result<usize, FormatError> {
|
||||
|
||||
/// The codec inside a Blosc frame (flags bits 5-7).
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum Codec {
|
||||
pub(crate) enum Codec {
|
||||
BloscLz,
|
||||
Lz4,
|
||||
Snappy,
|
||||
@@ -58,7 +58,7 @@ enum Codec {
|
||||
}
|
||||
|
||||
impl Codec {
|
||||
fn from_flags(flags: u8) -> Result<Codec, FormatError> {
|
||||
pub(crate) fn from_flags(flags: u8) -> Result<Codec, FormatError> {
|
||||
match flags >> 5 {
|
||||
0 => Ok(Codec::BloscLz),
|
||||
1 => Ok(Codec::Lz4),
|
||||
@@ -71,7 +71,7 @@ impl Codec {
|
||||
}
|
||||
|
||||
/// Decode one codec stream into exactly `dst`.
|
||||
fn decode_stream(
|
||||
pub(crate) fn decode_stream(
|
||||
codec: Codec,
|
||||
src: &[u8],
|
||||
dst: &mut [u8],
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -87,6 +87,8 @@ pub mod filters;
|
||||
mod filters_bitshuffle;
|
||||
#[cfg(feature = "blosc")]
|
||||
pub mod filters_blosc;
|
||||
#[cfg(feature = "blosc2")]
|
||||
pub mod filters_blosc2;
|
||||
#[cfg(feature = "bzip2")]
|
||||
mod filters_bzip2;
|
||||
#[cfg(feature = "lzf")]
|
||||
|
||||
@@ -0,0 +1,641 @@
|
||||
//! Crafted Blosc2 frames and chunks cannot make the decoder allocate out of
|
||||
//! proportion to the HDF5 chunk it decodes.
|
||||
//!
|
||||
//! A frame's header, its offsets chunk and its chunk headers all declare
|
||||
//! sizes, and the decoder used to allocate what they declared: a 173-byte
|
||||
//! frame whose offsets chunk claimed 2 GiB was decoded in full before any
|
||||
//! check failed. Every allocation is now bounded by the output limit (the
|
||||
//! HDF5 chunk's size) and the input's length.
|
||||
//!
|
||||
//! Peak heap use is measured with a counting global allocator; the tests
|
||||
//! share it, so each holds `SERIAL` for its whole run.
|
||||
#![cfg(feature = "blosc2")]
|
||||
|
||||
use std::alloc::{GlobalAlloc, Layout, System};
|
||||
use std::sync::Mutex;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
|
||||
use clawhdf5_format::filters_blosc2::{blosc2_decompress, blosc2_decompress_chunk};
|
||||
|
||||
struct Counting;
|
||||
|
||||
static CURRENT: AtomicUsize = AtomicUsize::new(0);
|
||||
static PEAK: AtomicUsize = AtomicUsize::new(0);
|
||||
static SERIAL: Mutex<()> = Mutex::new(());
|
||||
|
||||
unsafe impl GlobalAlloc for Counting {
|
||||
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
|
||||
let p = unsafe { System.alloc(layout) };
|
||||
if !p.is_null() {
|
||||
let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
|
||||
PEAK.fetch_max(now, Ordering::Relaxed);
|
||||
}
|
||||
p
|
||||
}
|
||||
|
||||
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
|
||||
let p = unsafe { System.alloc_zeroed(layout) };
|
||||
if !p.is_null() {
|
||||
let now = CURRENT.fetch_add(layout.size(), Ordering::Relaxed) + layout.size();
|
||||
PEAK.fetch_max(now, Ordering::Relaxed);
|
||||
}
|
||||
p
|
||||
}
|
||||
|
||||
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
|
||||
unsafe { System.dealloc(ptr, layout) };
|
||||
CURRENT.fetch_sub(layout.size(), Ordering::Relaxed);
|
||||
}
|
||||
}
|
||||
|
||||
#[global_allocator]
|
||||
static ALLOC: Counting = Counting;
|
||||
|
||||
/// Bytes allocated at the peak of `f`, above what was live when it started.
|
||||
fn peak_during<T>(f: impl FnOnce() -> T) -> (T, usize) {
|
||||
let base = CURRENT.load(Ordering::Relaxed);
|
||||
PEAK.store(base, Ordering::Relaxed);
|
||||
let out = f();
|
||||
(out, PEAK.load(Ordering::Relaxed).saturating_sub(base))
|
||||
}
|
||||
|
||||
/// What decoding one HDF5 chunk of `limit` bytes from `input` may hold at
|
||||
/// once: the output, a few blocks of scratch (each no larger than the
|
||||
/// output), the offsets table, and the Zstandard decoder's state, which has
|
||||
/// a fixed ceiling: a window of at most 128 KiB (or twice the stream) and a
|
||||
/// block's table of sequences (up to 98,303 of 12 bytes, 1.2 MB).
|
||||
fn bound(limit: usize, input: &[u8]) -> usize {
|
||||
6 * limit + 2 * input.len() + (2 << 20)
|
||||
}
|
||||
|
||||
fn lock() -> std::sync::MutexGuard<'static, ()> {
|
||||
SERIAL.lock().unwrap_or_else(|e| e.into_inner())
|
||||
}
|
||||
|
||||
/// A 32-byte (extended) Blosc2 chunk header.
|
||||
fn chunk_header(ts: u8, nbytes: i32, blocksize: i32, cbytes: i32, special: u8) -> Vec<u8> {
|
||||
let mut c = vec![5u8, 1, 0x05, ts];
|
||||
for v in [nbytes, blocksize, cbytes] {
|
||||
c.extend_from_slice(&v.to_le_bytes());
|
||||
}
|
||||
c.resize(32, 0);
|
||||
c[31] = special << 4;
|
||||
c
|
||||
}
|
||||
|
||||
/// A chunk of `nbytes` bytes that repeats one value (special type 3).
|
||||
fn repeated(value: &[u8], nbytes: i32, blocksize: i32) -> Vec<u8> {
|
||||
let mut c = chunk_header(
|
||||
value.len() as u8,
|
||||
nbytes,
|
||||
blocksize,
|
||||
32 + value.len() as i32,
|
||||
3,
|
||||
);
|
||||
c.extend_from_slice(value);
|
||||
c
|
||||
}
|
||||
|
||||
/// A frame offset recording a special chunk of `kind` (1 zeros, 2 NaN).
|
||||
fn special_offset(kind: u8) -> [u8; 8] {
|
||||
(((0x80 | kind) as i64) << 56).to_le_bytes()
|
||||
}
|
||||
|
||||
/// A B2ND metalayer.
|
||||
fn nd_meta(shape: &[i64], chunks: &[i32], blocks: &[i32]) -> Vec<u8> {
|
||||
let n = shape.len() as u8;
|
||||
let mut m = vec![0x95, 0, n, 0x90 | n];
|
||||
for s in shape {
|
||||
m.push(0xd3);
|
||||
m.extend_from_slice(&s.to_be_bytes());
|
||||
}
|
||||
for dims in [chunks, blocks] {
|
||||
m.push(0x90 | n);
|
||||
for d in dims {
|
||||
m.push(0xd2);
|
||||
m.extend_from_slice(&d.to_be_bytes());
|
||||
}
|
||||
}
|
||||
m
|
||||
}
|
||||
|
||||
/// A contiguous frame: header (with a `b2nd` metalayer if given), the data
|
||||
/// chunks, then the offsets chunk.
|
||||
fn frame(
|
||||
meta: Option<&[u8]>,
|
||||
nbytes: i64,
|
||||
typesize: i32,
|
||||
chunksize: i32,
|
||||
data: &[u8],
|
||||
offsets: &[u8],
|
||||
) -> Vec<u8> {
|
||||
let mut h = vec![0u8; 91];
|
||||
h[0] = 0x9e;
|
||||
h[1] = 0xa8;
|
||||
h[2..10].copy_from_slice(b"b2frame\0");
|
||||
h[25] = 2;
|
||||
match meta {
|
||||
Some(m) => {
|
||||
h.extend_from_slice(&[0xde, 0, 1, 0xa4]);
|
||||
h.extend_from_slice(b"b2nd");
|
||||
let at = h.len() as i32 + 5;
|
||||
h.push(0xd2);
|
||||
h.extend_from_slice(&at.to_be_bytes());
|
||||
h.push(0xc6);
|
||||
h.extend_from_slice(&(m.len() as u32).to_be_bytes());
|
||||
h.extend_from_slice(m);
|
||||
}
|
||||
None => h.extend_from_slice(&[0xde, 0, 0]),
|
||||
}
|
||||
let header_len = h.len() as i32;
|
||||
h[11..15].copy_from_slice(&header_len.to_be_bytes());
|
||||
h[30..38].copy_from_slice(&nbytes.to_be_bytes());
|
||||
h[39..47].copy_from_slice(&(data.len() as i64).to_be_bytes());
|
||||
h[48..52].copy_from_slice(&typesize.to_be_bytes());
|
||||
h[58..62].copy_from_slice(&chunksize.to_be_bytes());
|
||||
h.extend_from_slice(data);
|
||||
h.extend_from_slice(offsets);
|
||||
let len = h.len() as u64;
|
||||
h[16..24].copy_from_slice(&len.to_be_bytes());
|
||||
h
|
||||
}
|
||||
|
||||
/// The frame header's own sizes must not size the offsets chunk: a frame
|
||||
/// declaring 32 Mi chunks of 4 bytes, whose offsets chunk (40 bytes) says
|
||||
/// "one repeated offset, 256 MiB of them", made the decoder build all
|
||||
/// 256 MiB of offsets for a 1 MiB HDF5 chunk and then return 4 bytes.
|
||||
#[test]
|
||||
fn offsets_chunk_is_bounded_by_the_output_limit() {
|
||||
let _g = lock();
|
||||
let limit = 1 << 20;
|
||||
let offsets_len: i32 = 256 << 20;
|
||||
let nchunks = offsets_len as i64 / 8;
|
||||
let offsets = repeated(&special_offset(1), offsets_len, 64 << 20);
|
||||
let f = frame(None, nchunks * 4, 4, 4, &[], &offsets);
|
||||
let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
|
||||
assert!(r.is_err(), "decoded {:?} bytes", r.map(|v| v.len()));
|
||||
assert!(
|
||||
peak <= bound(limit, &f),
|
||||
"peak {peak} bytes for a {}-byte frame",
|
||||
f.len()
|
||||
);
|
||||
// The same frame with a variable chunk size (0): the offsets chunk
|
||||
// alone says how many chunks there are.
|
||||
let f = frame(None, nchunks * 4, 4, 0, &[], &offsets);
|
||||
let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
|
||||
assert!(r.is_err());
|
||||
assert!(peak <= bound(limit, &f), "chunksize 0: peak {peak} bytes");
|
||||
}
|
||||
|
||||
/// A legitimate frame of this shape (one chunk, its offset special) still
|
||||
/// decodes.
|
||||
#[test]
|
||||
fn small_frames_still_decode() {
|
||||
let _g = lock();
|
||||
let offsets = repeated(&special_offset(1), 8, 8);
|
||||
let f = frame(None, 64, 4, 64, &[], &offsets);
|
||||
assert_eq!(blosc2_decompress(&f, 64).unwrap(), vec![0; 64]);
|
||||
let _ = blosc2_decompress_chunk;
|
||||
}
|
||||
|
||||
/// A chunk that decodes to nothing kept its declared block size (up to
|
||||
/// 512 MiB) and allocated two scratch blocks of it: about 1 GiB for a
|
||||
/// 20-byte chunk.
|
||||
#[test]
|
||||
fn empty_chunk_does_not_allocate_its_block_size() {
|
||||
let _g = lock();
|
||||
let mut c = vec![5u8, 1, 0x01, 1];
|
||||
for v in [0i32, 0x1FFF_F000, 20] {
|
||||
c.extend_from_slice(&v.to_le_bytes());
|
||||
}
|
||||
c.resize(20, 0);
|
||||
let (r, peak) = peak_during(|| blosc2_decompress_chunk(&c, 1 << 20));
|
||||
assert_eq!(r.map(|v| v.len()).unwrap_or(0), 0);
|
||||
assert!(
|
||||
peak <= bound(0, &c),
|
||||
"peak {peak} bytes for a 20-byte chunk"
|
||||
);
|
||||
// Inside a frame for a non-empty HDF5 chunk it is an error, not data.
|
||||
let offsets = repeated(&0i64.to_le_bytes(), 8, 8);
|
||||
let f = frame(None, 64, 4, 64, &c, &offsets);
|
||||
let (r, peak) = peak_during(|| blosc2_decompress(&f, 64));
|
||||
assert!(r.is_err(), "decoded {:?}", r.map(|v| v.len()));
|
||||
assert!(peak <= bound(64, &f), "in a frame: peak {peak} bytes");
|
||||
}
|
||||
|
||||
/// B2ND chunks were decoded whole, padding included, with up to 16x the
|
||||
/// HDF5 chunk size as their limit. Blocks are now placed as they are
|
||||
/// decoded, so the padding is never held.
|
||||
///
|
||||
/// Ten dimensions: nine of 3 split into blocks of 2 (padded to 4) and one
|
||||
/// of 4, so each chunk is 13x the array. One chunk, stored three ways: as a
|
||||
/// NaN chunk in the frame's offsets, as a repeated-value chunk, and as a
|
||||
/// chunk of stored (uncompressed) blocks.
|
||||
#[test]
|
||||
fn b2nd_padding_is_never_held() {
|
||||
let _g = lock();
|
||||
let ts = 4usize;
|
||||
let mut shape = vec![3i64; 9];
|
||||
shape.push(4);
|
||||
let chunks: Vec<i32> = shape.iter().map(|&s| s as i32).collect();
|
||||
let mut blocks = vec![2i32; 9];
|
||||
blocks.push(4);
|
||||
let meta = nd_meta(&shape, &chunks, &blocks);
|
||||
let items: usize = shape.iter().product::<i64>() as usize;
|
||||
let limit = items * ts;
|
||||
let block_bytes = ts * blocks.iter().product::<i32>() as usize;
|
||||
let ext_bytes = ts * 4usize.pow(9) * 4;
|
||||
assert!(ext_bytes > 13 * limit);
|
||||
let offsets = |off: [u8; 8]| repeated(&off, 8, 8);
|
||||
|
||||
let value = 1.5f32.to_le_bytes();
|
||||
let stored = {
|
||||
// Every block stored raw: block k holds the value k.
|
||||
let mut c = chunk_header(4, ext_bytes as i32, block_bytes as i32, 0, 0);
|
||||
c[2] = 0x02 | 0x10; // memcpyed, not split
|
||||
c.truncate(16);
|
||||
for k in 0..ext_bytes / block_bytes {
|
||||
c.extend((k as f32).to_le_bytes().repeat(block_bytes / 4));
|
||||
}
|
||||
let n = c.len() as i32;
|
||||
c[12..16].copy_from_slice(&n.to_le_bytes());
|
||||
c
|
||||
};
|
||||
let cases: Vec<(&str, Vec<u8>)> = vec![
|
||||
(
|
||||
"NaN offset",
|
||||
frame(
|
||||
Some(&meta),
|
||||
ext_bytes as i64,
|
||||
4,
|
||||
ext_bytes as i32,
|
||||
&[],
|
||||
&offsets(special_offset(2)),
|
||||
),
|
||||
),
|
||||
(
|
||||
"repeated value",
|
||||
frame(
|
||||
Some(&meta),
|
||||
ext_bytes as i64,
|
||||
4,
|
||||
ext_bytes as i32,
|
||||
&repeated(&value, ext_bytes as i32, block_bytes as i32),
|
||||
&offsets(0i64.to_le_bytes()),
|
||||
),
|
||||
),
|
||||
(
|
||||
"stored blocks",
|
||||
frame(
|
||||
Some(&meta),
|
||||
ext_bytes as i64,
|
||||
4,
|
||||
ext_bytes as i32,
|
||||
&stored,
|
||||
&offsets(0i64.to_le_bytes()),
|
||||
),
|
||||
),
|
||||
];
|
||||
for (name, f) in cases {
|
||||
let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
|
||||
let out = r.unwrap_or_else(|e| panic!("{name}: {e}"));
|
||||
assert_eq!(out.len(), limit, "{name}");
|
||||
match name {
|
||||
"NaN offset" => assert!(
|
||||
out.chunks(4)
|
||||
.all(|v| f32::from_le_bytes(v.try_into().unwrap()).is_nan())
|
||||
),
|
||||
"repeated value" => assert!(out.chunks(4).all(|v| v == value)),
|
||||
_ => {
|
||||
// Element (i0..i9) lies in block (i0/2, .., i8/2), numbered
|
||||
// in C order over a 2x..x2x1 grid of blocks.
|
||||
let mut idx = [0usize; 10];
|
||||
for (e, v) in out.chunks(4).enumerate() {
|
||||
let mut n = e;
|
||||
for d in (0..10).rev() {
|
||||
idx[d] = n % shape[d] as usize;
|
||||
n /= shape[d] as usize;
|
||||
}
|
||||
let k = idx[..9].iter().fold(0, |k, &i| k * 2 + i / 2);
|
||||
assert_eq!(
|
||||
f32::from_le_bytes(v.try_into().unwrap()),
|
||||
k as f32,
|
||||
"{name} {e}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
peak <= bound(limit, &f),
|
||||
"{name}: peak {peak} bytes for a {limit}-byte chunk ({}-byte frame)",
|
||||
f.len()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// A B2ND chunk larger than the array (here 16x, the old cap) is refused,
|
||||
/// or at least never allocated.
|
||||
#[test]
|
||||
fn b2nd_chunk_larger_than_the_array_is_not_allocated() {
|
||||
let _g = lock();
|
||||
let limit = 1 << 20;
|
||||
let c = 16 * limit as i32;
|
||||
let meta = nd_meta(&[limit as i64], &[c], &[c]);
|
||||
let offsets = repeated(&special_offset(1), 8, 8);
|
||||
let f = frame(Some(&meta), c as i64, 1, c, &[], &offsets);
|
||||
let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
|
||||
assert!(
|
||||
peak <= bound(limit, &f),
|
||||
"peak {peak} bytes ({:?})",
|
||||
r.map(|v| v.len())
|
||||
);
|
||||
}
|
||||
|
||||
/// ruzstd reserves a frame's declared window (up to 100 MiB) before it
|
||||
/// decodes a frame with a decoder it has used before: a Blosc2 chunk of
|
||||
/// two 16-byte Zstandard streams, each declaring a 96 MiB window,
|
||||
/// allocated 96 MiB. c-blosc2 compresses each block with its size known,
|
||||
/// so its windows never exceed the block.
|
||||
#[test]
|
||||
fn zstd_window_is_bounded_by_the_output() {
|
||||
let _g = lock();
|
||||
let mut z = 0xfd2f_b528u32.to_le_bytes().to_vec();
|
||||
// No single segment, no checksum; window 2^26 + 4/8 of it = 96 MiB.
|
||||
z.extend_from_slice(&[0x00, (16 << 3) | 4]);
|
||||
// One raw block, last, of 16 bytes.
|
||||
let h = 1 | (16 << 3);
|
||||
z.extend_from_slice(&[h as u8, (h >> 8) as u8, 0]);
|
||||
z.extend_from_slice(&[7; 16]);
|
||||
// Two blocks of 16 bytes, one stream each (not split), Zstandard
|
||||
// (codec 4).
|
||||
let chunk = |z: &[u8]| {
|
||||
let mut c = vec![5u8, 1, 0x10 | (4 << 5), 1];
|
||||
for v in [32i32, 16, 0] {
|
||||
c.extend_from_slice(&v.to_le_bytes());
|
||||
}
|
||||
let first = 24 + 4 + z.len();
|
||||
c.extend_from_slice(&24i32.to_le_bytes());
|
||||
c.extend_from_slice(&(first as i32).to_le_bytes());
|
||||
for _ in 0..2 {
|
||||
c.extend_from_slice(&(z.len() as i32).to_le_bytes());
|
||||
c.extend_from_slice(z);
|
||||
}
|
||||
let n = c.len() as i32;
|
||||
c[12..16].copy_from_slice(&n.to_le_bytes());
|
||||
c
|
||||
};
|
||||
let c = chunk(&z);
|
||||
let (r, peak) = peak_during(|| blosc2_decompress_chunk(&c, 32));
|
||||
assert!(peak <= bound(32, &c), "peak {peak} bytes ({r:?})");
|
||||
assert!(r.is_err(), "{r:?}");
|
||||
// The same streams with a window they can use read.
|
||||
z[5] = 0;
|
||||
assert_eq!(
|
||||
blosc2_decompress_chunk(&chunk(&z), 32).unwrap(),
|
||||
vec![7; 32]
|
||||
);
|
||||
}
|
||||
|
||||
/// xorshift64*: deterministic, so a failure reproduces.
|
||||
struct Rng(u64);
|
||||
|
||||
impl Rng {
|
||||
fn next(&mut self) -> u64 {
|
||||
let mut x = self.0;
|
||||
x ^= x >> 12;
|
||||
x ^= x << 25;
|
||||
x ^= x >> 27;
|
||||
self.0 = x;
|
||||
x.wrapping_mul(0x2545_F491_4F6C_DD1D)
|
||||
}
|
||||
|
||||
fn below(&mut self, n: usize) -> usize {
|
||||
(self.next() % n.max(1) as u64) as usize
|
||||
}
|
||||
|
||||
/// A size that tends to the edges: small, a power of two, huge.
|
||||
fn size(&mut self) -> i64 {
|
||||
match self.below(6) {
|
||||
0 => self.below(64) as i64,
|
||||
1 => 1 << self.below(31),
|
||||
2 => i32::MAX as i64 - self.below(4096) as i64,
|
||||
3 => (1i64 << self.below(62)) + self.below(8) as i64,
|
||||
4 => MAX_BLOCK - self.below(3) as i64,
|
||||
_ => self.next() as i32 as i64,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const MAX_BLOCK: i64 = 536_866_816;
|
||||
|
||||
/// One to four edits: bytes, or a size field written little-endian (chunk
|
||||
/// headers) or big-endian (frame headers), most often at a header's size
|
||||
/// fields.
|
||||
fn mutate(rng: &mut Rng, seed: &[u8], data_at: usize) -> Vec<u8> {
|
||||
let mut v = seed.to_vec();
|
||||
for _ in 0..1 + rng.below(4) {
|
||||
let len = v.len();
|
||||
if len < 16 {
|
||||
v.push(rng.next() as u8);
|
||||
continue;
|
||||
}
|
||||
match rng.below(8) {
|
||||
0 => {
|
||||
let i = rng.below(len);
|
||||
v[i] ^= 1 << rng.below(8);
|
||||
}
|
||||
1 => {
|
||||
let i = rng.below(len);
|
||||
v[i] = rng.next() as u8;
|
||||
}
|
||||
2 => {
|
||||
// Frame header: nbytes, cbytes (i64), typesize, chunksize.
|
||||
let x = rng.size();
|
||||
match rng.below(4) {
|
||||
0 if len >= 38 => v[30..38].copy_from_slice(&x.to_be_bytes()),
|
||||
1 if len >= 47 => v[39..47].copy_from_slice(&x.to_be_bytes()),
|
||||
2 if len >= 52 => v[48..52].copy_from_slice(&(x as i32).to_be_bytes()),
|
||||
_ if len >= 62 => v[58..62].copy_from_slice(&(x as i32).to_be_bytes()),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
3 | 4 => {
|
||||
// A chunk header's nbytes, blocksize or cbytes: in the first
|
||||
// data chunk, or anywhere (the offsets chunk comes last).
|
||||
let at = if rng.below(2) == 0 && data_at + 16 <= len {
|
||||
data_at + 4 * (1 + rng.below(3))
|
||||
} else {
|
||||
rng.below(len - 3)
|
||||
};
|
||||
let x = rng.size() as i32;
|
||||
v[at..at + 4].copy_from_slice(&x.to_le_bytes());
|
||||
}
|
||||
5 => v.truncate(rng.below(len)),
|
||||
6 => {
|
||||
let at = rng.below(len);
|
||||
v[at] = [0x10, 0x20, 0x30, 0x40, 0x05, 0x07, 0x02][rng.below(7)];
|
||||
}
|
||||
_ => {
|
||||
let i = rng.below(len - 3);
|
||||
let x = rng.size() as i32;
|
||||
v[i..i + 4].copy_from_slice(&x.to_be_bytes());
|
||||
}
|
||||
}
|
||||
}
|
||||
v
|
||||
}
|
||||
|
||||
/// Every fixture frame that decodes, with its decoded size.
|
||||
fn seeds() -> Vec<(Vec<u8>, usize)> {
|
||||
let dir = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("tests/fixtures/blosc2");
|
||||
let mut v = Vec::new();
|
||||
for e in std::fs::read_dir(dir).unwrap() {
|
||||
let p = e.unwrap().path();
|
||||
if p.extension().is_some_and(|x| x == "b2f")
|
||||
&& let Ok(out) = std::fs::read(p.with_extension("out"))
|
||||
{
|
||||
v.push((std::fs::read(&p).unwrap(), out.len()));
|
||||
}
|
||||
}
|
||||
v.sort();
|
||||
assert!(v.len() >= 20, "fixtures missing");
|
||||
v
|
||||
}
|
||||
|
||||
fn header_len(frame: &[u8]) -> usize {
|
||||
i32::from_be_bytes(frame[11..15].try_into().unwrap()) as usize
|
||||
}
|
||||
|
||||
/// Mutated fixture frames, decoded with their HDF5 chunk size as the
|
||||
/// limit, and their first chunks on their own: whatever they declare, no
|
||||
/// decode holds more than a small multiple of the output and the input.
|
||||
#[test]
|
||||
fn fuzzed_frames_and_chunks_stay_within_the_allocation_bound() {
|
||||
let _g = lock();
|
||||
let seeds = seeds();
|
||||
let mut rng = Rng(0xb2a1);
|
||||
let mut worst = (0.0f64, String::new());
|
||||
for i in 0..20_000 {
|
||||
let (seed, limit) = &seeds[rng.below(seeds.len())];
|
||||
let f = mutate(&mut rng, seed, header_len(seed));
|
||||
let (r, peak) = peak_during(|| blosc2_decompress(&f, *limit));
|
||||
if let Ok(out) = &r {
|
||||
assert!(out.len() <= *limit, "iteration {i}: output past the limit");
|
||||
}
|
||||
assert!(
|
||||
peak <= bound(*limit, &f),
|
||||
"iteration {i}: peak {peak} bytes for a {limit}-byte chunk from {} bytes ({:?})",
|
||||
f.len(),
|
||||
r.map(|v| v.len())
|
||||
);
|
||||
let ratio = peak as f64 / bound(*limit, &f) as f64;
|
||||
if ratio > worst.0 {
|
||||
worst = (
|
||||
ratio,
|
||||
format!(
|
||||
"frame iteration {i}: peak {peak}, limit {limit}, input {}",
|
||||
f.len()
|
||||
),
|
||||
);
|
||||
}
|
||||
}
|
||||
for i in 0..20_000 {
|
||||
let (seed, _) = &seeds[rng.below(seeds.len())];
|
||||
let at = header_len(seed);
|
||||
let chunk = &seed[at..];
|
||||
let c = mutate(&mut rng, chunk, 0);
|
||||
let limit = 1 << 16;
|
||||
let (r, peak) = peak_during(|| blosc2_decompress_chunk(&c, limit));
|
||||
assert!(
|
||||
peak <= bound(limit, &c),
|
||||
"chunk iteration {i}: peak {peak} bytes from {} bytes ({:?})",
|
||||
c.len(),
|
||||
r.map(|v| v.len())
|
||||
);
|
||||
}
|
||||
eprintln!("worst peak / bound: {:.2} ({})", worst.0, worst.1);
|
||||
}
|
||||
|
||||
/// Frames built from random header sizes, offsets chunks and B2ND shapes
|
||||
/// (chunk and block shapes that pad, special and repeated-value chunks).
|
||||
#[test]
|
||||
fn random_frames_stay_within_the_allocation_bound() {
|
||||
let _g = lock();
|
||||
let mut rng = Rng(0xb2a2);
|
||||
for i in 0..5_000 {
|
||||
let ts = [1usize, 2, 4, 8][rng.below(4)];
|
||||
let ndim = 1 + rng.below(8);
|
||||
let mut shape = Vec::new();
|
||||
let mut chunks = Vec::new();
|
||||
let mut blocks = Vec::new();
|
||||
for _ in 0..ndim {
|
||||
let s = 1 + rng.below(if ndim > 3 { 4 } else { 40 });
|
||||
let c = if rng.below(8) == 0 {
|
||||
s * (1 + rng.below(4))
|
||||
} else {
|
||||
1 + rng.below(s)
|
||||
};
|
||||
let b = 1 + rng.below(c);
|
||||
shape.push(s as i64);
|
||||
chunks.push(c as i32);
|
||||
blocks.push(b as i32);
|
||||
}
|
||||
let items: usize = shape.iter().product::<i64>() as usize;
|
||||
let limit = items * ts;
|
||||
let meta = nd_meta(&shape, &chunks, &blocks);
|
||||
let ext: usize = ts
|
||||
* chunks
|
||||
.iter()
|
||||
.zip(&blocks)
|
||||
.map(|(&c, &b)| (c as usize).div_ceil(b as usize) * b as usize)
|
||||
.product::<usize>();
|
||||
let nchunks: usize = shape
|
||||
.iter()
|
||||
.zip(&chunks)
|
||||
.map(|(&s, &c)| (s as usize).div_ceil(c as usize))
|
||||
.product();
|
||||
let block_bytes = ts * blocks.iter().product::<i32>() as usize;
|
||||
let chunksize = if rng.below(4) == 0 {
|
||||
rng.size()
|
||||
} else {
|
||||
ext as i64
|
||||
};
|
||||
let nbytes = if rng.below(4) == 0 {
|
||||
rng.size()
|
||||
} else {
|
||||
(nchunks * ext) as i64
|
||||
};
|
||||
let off_n = if rng.below(4) == 0 {
|
||||
rng.size() as i32
|
||||
} else {
|
||||
8 * nchunks as i32
|
||||
};
|
||||
let (data, off) = match rng.below(3) {
|
||||
0 => (Vec::new(), special_offset(1 + rng.below(2) as u8)),
|
||||
_ => {
|
||||
let bs = if rng.below(4) == 0 {
|
||||
rng.size() as i32
|
||||
} else {
|
||||
block_bytes as i32
|
||||
};
|
||||
let value: Vec<u8> = (0..ts).map(|_| rng.next() as u8).collect();
|
||||
let n = if rng.below(4) == 0 {
|
||||
rng.size() as i32
|
||||
} else {
|
||||
ext as i32
|
||||
};
|
||||
(repeated(&value, n, bs), 0i64.to_le_bytes())
|
||||
}
|
||||
};
|
||||
let offsets = repeated(&off, off_n, off_n.clamp(1, 8));
|
||||
let meta = (rng.below(4) != 0).then_some(meta.as_slice());
|
||||
let f = frame(meta, nbytes, ts as i32, chunksize as i32, &data, &offsets);
|
||||
let (r, peak) = peak_during(|| blosc2_decompress(&f, limit));
|
||||
assert!(
|
||||
peak <= bound(limit, &f),
|
||||
"iteration {i}: peak {peak} bytes for a {limit}-byte chunk ({:?}, shape {shape:?} \
|
||||
chunks {chunks:?} blocks {blocks:?})",
|
||||
r.map(|v| v.len())
|
||||
);
|
||||
}
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1 @@
|
||||
filter 35
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,129 @@
|
||||
"""Generate the Blosc2 frames the `filters_blosc2` unit tests decode.
|
||||
|
||||
Each case is `<name>.b2f` (a Blosc2 contiguous frame, what the HDF5 Blosc2
|
||||
filter stores per chunk) and `<name>.out` (what decoding it must give: the
|
||||
first chunk of a plain frame, or the whole array in C order for a B2ND
|
||||
frame), or `<name>.err` (a frame clawhdf5 must refuse; the file holds a word
|
||||
the error message must contain).
|
||||
|
||||
These cover what files written by h5py + hdf5plugin never contain, but a
|
||||
Blosc2 frame may: special chunks (repeated value, NaN, uninitialised), the
|
||||
delta filter over many blocks and odd type sizes, bit shuffle of blocks that
|
||||
are not a multiple of 8 elements, shuffle with a byte-group size, forced
|
||||
stream splitting, multi-chunk B2ND arrays with padded edge chunks and a
|
||||
chunk of zeros, and features clawhdf5 refuses (dictionaries, registered
|
||||
filters).
|
||||
|
||||
Written with python-blosc2 4.13.1 (c-blosc2 3.3.4) in a scratch venv
|
||||
(`pip install blosc2`). Re-run only to regenerate:
|
||||
|
||||
python generate.py <this directory>
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
import blosc2
|
||||
import numpy as np
|
||||
|
||||
out = sys.argv[1]
|
||||
|
||||
|
||||
def save(name, frame, expected):
|
||||
with open(os.path.join(out, name + ".b2f"), "wb") as f:
|
||||
f.write(frame)
|
||||
with open(os.path.join(out, name + ".out"), "wb") as f:
|
||||
f.write(expected)
|
||||
|
||||
|
||||
def save_err(name, frame, word):
|
||||
with open(os.path.join(out, name + ".b2f"), "wb") as f:
|
||||
f.write(frame)
|
||||
with open(os.path.join(out, name + ".err"), "w") as f:
|
||||
f.write(word)
|
||||
|
||||
|
||||
def plain(data, **cparams):
|
||||
"""A one-chunk super-chunk frame of `data`, as hdf5-blosc2 writes."""
|
||||
data = np.ascontiguousarray(data)
|
||||
cp = blosc2.CParams(typesize=data.dtype.itemsize, **cparams)
|
||||
sc = blosc2.SChunk(chunksize=data.nbytes, cparams=cp)
|
||||
sc.append_data(data)
|
||||
return sc.to_cframe(), data.tobytes()
|
||||
|
||||
|
||||
def special(nitems, dtype, kind, value=None):
|
||||
dt = np.dtype(dtype)
|
||||
sc = blosc2.SChunk(chunksize=nitems * dt.itemsize,
|
||||
cparams=blosc2.CParams(typesize=dt.itemsize))
|
||||
sc.fill_special(nitems, kind, value)
|
||||
return sc.to_cframe()
|
||||
|
||||
|
||||
# Special chunks. A repeated value stays in the frame as a 33+ byte chunk;
|
||||
# NaN and uninitialised chunks become special offsets.
|
||||
save("value_i4", special(300, "<i4", blosc2.SpecialValue.VALUE, 123456),
|
||||
np.full(300, 123456, "<i4").tobytes())
|
||||
save("value_f8", special(250, "<f8", blosc2.SpecialValue.VALUE, -2.5),
|
||||
np.full(250, -2.5, "<f8").tobytes())
|
||||
save("nan_f4", special(500, "<f4", blosc2.SpecialValue.NAN),
|
||||
np.full(500, np.nan, "<f4").tobytes())
|
||||
save("nan_f8", special(300, "<f8", blosc2.SpecialValue.NAN),
|
||||
np.full(300, np.nan, "<f8").tobytes())
|
||||
save("zero_u2", special(2000, "<u2", blosc2.SpecialValue.ZERO), bytes(4000))
|
||||
# Uninitialised values: libhdf5 would hand back whatever memory it had;
|
||||
# clawhdf5 returns zeros.
|
||||
save("uninit_i8", special(64, "<i8", blosc2.SpecialValue.UNINIT), bytes(512))
|
||||
|
||||
rng = np.random.default_rng(11)
|
||||
ramp = lambda n, dt: ((np.arange(n) * 7) % 1000 + rng.integers(0, 3, n)).astype(dt)
|
||||
# Slowly varying: what the delta filter is for (noise would be stored raw).
|
||||
smooth = lambda n, dt: (np.arange(n) // 3 + 1000).astype(dt)
|
||||
|
||||
# Delta over many blocks, for type sizes 1, 2, 4, 8, 3 (bytes) and 16 (u64
|
||||
# pairs).
|
||||
for dt, n, codec in [("<u1", 2000, blosc2.Codec.LZ4), ("<i2", 1000, blosc2.Codec.BLOSCLZ),
|
||||
("<i4", 700, blosc2.Codec.LZ4), ("<u8", 400, blosc2.Codec.BLOSCLZ)]:
|
||||
save(f"delta_{np.dtype(dt).name}_{codec.name.lower()}",
|
||||
*plain(smooth(n, dt), codec=codec, blocksize=256,
|
||||
filters=[blosc2.Filter.DELTA], filters_meta=[0]))
|
||||
rec3 = np.frombuffer(smooth(3 * 300, "<u1").tobytes(), dtype="V3")
|
||||
save("delta_v3", *plain(rec3, blocksize=300, filters=[blosc2.Filter.DELTA], filters_meta=[0]))
|
||||
rec16 = np.frombuffer(smooth(2 * 200, "<u8").tobytes(), dtype="V16")
|
||||
save("delta_shuffle_v16", *plain(rec16, blocksize=512,
|
||||
filters=[blosc2.Filter.DELTA, blosc2.Filter.SHUFFLE],
|
||||
filters_meta=[0, 0]))
|
||||
|
||||
# Bit shuffle of blocks whose element count is not a multiple of 8 (44-byte
|
||||
# blocks of 4-byte elements: 8 transposed, 3 copied).
|
||||
save("bitshuffle_odd_blocks", *plain(ramp(500, "<i4"), codec=blosc2.Codec.ZSTD, blocksize=44,
|
||||
filters=[blosc2.Filter.BITSHUFFLE], filters_meta=[0]))
|
||||
# Shuffle in groups of 2 bytes of an 8-byte type (filters_meta).
|
||||
save("shuffle_meta2", *plain(ramp(500, "<i8"), codec=blosc2.Codec.ZLIB,
|
||||
filters=[blosc2.Filter.SHUFFLE], filters_meta=[2]))
|
||||
# Streams split per byte, and never split.
|
||||
save("always_split", *plain(ramp(1000, "<f4"), codec=blosc2.Codec.LZ4HC,
|
||||
splitmode=blosc2.SplitMode.ALWAYS_SPLIT))
|
||||
save("never_split", *plain(ramp(1500, "<u2"), codec=blosc2.Codec.ZSTD,
|
||||
splitmode=blosc2.SplitMode.NEVER_SPLIT))
|
||||
|
||||
# B2ND arrays of several chunks whose edge chunks and blocks are padded, and
|
||||
# one whose middle chunk is all zeros (a special offset).
|
||||
for name, shape, chunks, blocks, dt in [
|
||||
("b2nd_2d", (37, 29), (10, 16), (4, 6), "<i4"),
|
||||
("b2nd_3d", (9, 10, 7), (4, 5, 3), (3, 2, 2), "<f4"),
|
||||
("b2nd_4d", (5, 7, 5, 6), (3, 2, 5, 4), (2, 2, 3, 3), "<u2"),
|
||||
]:
|
||||
a = ramp(int(np.prod(shape)), dt).reshape(shape)
|
||||
arr = blosc2.asarray(a, chunks=chunks, blocks=blocks)
|
||||
save(name, arr.to_cframe(), a.tobytes())
|
||||
a = ramp(60 * 20, "<i2").reshape(60, 20)
|
||||
a[20:40, :] = 0
|
||||
arr = blosc2.asarray(a, chunks=(20, 20), blocks=(8, 16))
|
||||
save("b2nd_zero_chunk", arr.to_cframe(), a.tobytes())
|
||||
|
||||
# Refused: a dictionary, and a registered filter (bytedelta).
|
||||
frame, _ = plain(ramp(4000, "<i4"), codec=blosc2.Codec.ZSTD, use_dict=True, blocksize=2048)
|
||||
save_err("zstd_dict", frame, "dictionar")
|
||||
frame, _ = plain(ramp(500, "<i4"), filters=[blosc2.Filter.SHUFFLE, blosc2.Filter.BYTEDELTA],
|
||||
filters_meta=[0, 4])
|
||||
save_err("bytedelta", frame, "filter 35")
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1 @@
|
||||
dictionar
|
||||
@@ -47,8 +47,9 @@ lzf = ["clawhdf5-format/lzf"]
|
||||
bitshuffle = ["clawhdf5-format/bitshuffle"]
|
||||
bzip2 = ["clawhdf5-format/bzip2"]
|
||||
blosc = ["clawhdf5-format/blosc"]
|
||||
blosc2 = ["clawhdf5-format/blosc2"]
|
||||
# Every plugin filter.
|
||||
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc"]
|
||||
plugin-filters = ["lzf", "bitshuffle", "bzip2", "blosc", "blosc2"]
|
||||
# Dataset::verify_provenance() — recompute a dataset's SHA-256 and compare
|
||||
# against its stored _provenance_sha256 attribute. On by default, matching
|
||||
# clawhdf5-format's own default-on `provenance` feature.
|
||||
|
||||
@@ -77,7 +77,7 @@ except ImportError:
|
||||
hdf5plugin = None
|
||||
path = sys.argv[1]
|
||||
FILTERS = eval('(' + sys.argv[2] + ')')
|
||||
cases = [
|
||||
cases = eval('(' + sys.argv[3] + ')') if len(sys.argv) > 3 else [
|
||||
('<u1', (1000,), (128,), 'ramp'),
|
||||
('<i2', (37, 53), (10, 16), 'ramp'),
|
||||
('<i4', (2000,), (300,), 'ramp'),
|
||||
@@ -97,7 +97,10 @@ with h5py.File(path, 'w') as f:
|
||||
for label, kw in FILTERS:
|
||||
for dt, shape, chunks, kind in cases:
|
||||
n = int(np.prod(shape))
|
||||
if kind == 'noise':
|
||||
if kind in ('zeros', 'const', 'nan'):
|
||||
fill = {'zeros': 0, 'const': 7, 'nan': np.nan}[kind]
|
||||
data = np.full(n, fill, dtype=dt)
|
||||
elif kind == 'noise':
|
||||
raw = rng.integers(0, 256, n * np.dtype(dt).itemsize, dtype=np.uint8)
|
||||
data = raw.view(dt)
|
||||
if np.dtype(dt).kind == 'f':
|
||||
@@ -117,11 +120,18 @@ print(i)
|
||||
/// `(label, create_dataset kwargs)`), then check that clawhdf5 reads each
|
||||
/// filtered dataset exactly as its unfiltered twin.
|
||||
fn check_h5py_written(tag: &str, filters: &str) {
|
||||
check_h5py_written_cases(tag, filters, None);
|
||||
}
|
||||
|
||||
/// [`check_h5py_written`] over `cases` (a Python list of `(dtype, shape,
|
||||
/// chunks, kind)`, kind one of ramp, noise, zeros, const, nan) instead of
|
||||
/// the default ones.
|
||||
fn check_h5py_written_cases(tag: &str, filters: &str, cases: Option<&str>) {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join(format!("{tag}.h5"));
|
||||
let n: usize = run_python(GENERATE, &[path.to_str().unwrap(), filters])
|
||||
.parse()
|
||||
.unwrap();
|
||||
let mut args = vec![path.to_str().unwrap(), filters];
|
||||
args.extend(cases);
|
||||
let n: usize = run_python(GENERATE, &args).parse().unwrap();
|
||||
assert!(n > 0);
|
||||
let file = File::open(&path).unwrap();
|
||||
for i in 0..n {
|
||||
@@ -353,6 +363,110 @@ fn blosc_written_by_clawhdf5_reads_in_hdf5plugin() {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "blosc2")]
|
||||
#[test]
|
||||
fn blosc2_written_by_hdf5plugin_reads_exactly() {
|
||||
if !have_python("h5py, hdf5plugin") {
|
||||
return;
|
||||
}
|
||||
// Every codec hdf5plugin's Blosc2 offers, each lossless filter, and
|
||||
// levels from "store" to maximum. Truncating precision is lossy, so it
|
||||
// is checked separately against h5py's own reading.
|
||||
check_h5py_written(
|
||||
"blosc2",
|
||||
r#"[(f'{c} {f} {l}', hdf5plugin.Blosc2(cname=c, clevel=l, filters=f))
|
||||
for c in ['blosclz', 'lz4', 'lz4hc', 'zlib', 'zstd']
|
||||
for f, l in [(hdf5plugin.Blosc2.NOFILTER, 5),
|
||||
(hdf5plugin.Blosc2.SHUFFLE, 9),
|
||||
(hdf5plugin.Blosc2.BITSHUFFLE, 1),
|
||||
(hdf5plugin.Blosc2.DELTA, 5)]]
|
||||
+ [('blosclz level 0', hdf5plugin.Blosc2(cname='blosclz', clevel=0)),
|
||||
('zstd level 0 bitshuffle',
|
||||
hdf5plugin.Blosc2(cname='zstd', clevel=0, filters=hdf5plugin.Blosc2.BITSHUFFLE))]"#,
|
||||
);
|
||||
}
|
||||
|
||||
/// Blosc2 over more shapes and dtypes: every integer and float width,
|
||||
/// 1-D to 5-D chunks (B2ND arrays from 2-D on) with partial edge chunks and
|
||||
/// block shapes that pad the chunk, datasets of zeros, of one repeated value
|
||||
/// and of NaN (Blosc2's "special" chunks), and Fletcher32 before Blosc2
|
||||
/// (which makes hdf5-blosc2 fall back from B2ND to a plain frame).
|
||||
#[cfg(feature = "blosc2")]
|
||||
#[test]
|
||||
fn blosc2_shapes_and_special_chunks_read_exactly() {
|
||||
if !have_python("h5py, hdf5plugin") {
|
||||
return;
|
||||
}
|
||||
let cases = r#"[(dt, shape, chunks, kind)
|
||||
for dt in ['<i1', '<u1', '<i2', '>u2', '<i4', '<u4', '<i8', '<u8', '<f4', '>f8']
|
||||
for shape, chunks in [((777,), (100,)),
|
||||
((37, 53), (10, 16)),
|
||||
((9, 10, 11), (4, 5, 3)),
|
||||
((6, 7, 5, 9), (3, 2, 5, 4))]
|
||||
for kind in ['ramp', 'noise']]
|
||||
+ [('<f8', (300, 30), (64, 7), 'zeros'), ('<i4', (50, 40, 3), (7, 9, 3), 'zeros'),
|
||||
('<u2', (5000,), (1024,), 'const'), ('<i8', (40, 40), (16, 16), 'const'),
|
||||
('<f4', (100, 20), (32, 8), 'nan'), ('<f8', (3000,), (1000,), 'nan'),
|
||||
('<f4', (1000, 1000), (256, 512), 'ramp'),
|
||||
('<i2', (3, 3, 3, 3, 3), (2, 2, 2, 2, 2), 'ramp')]"#;
|
||||
check_h5py_written_cases(
|
||||
"blosc2_shapes",
|
||||
r#"[('lz4 shuffle', hdf5plugin.Blosc2(cname='lz4')),
|
||||
('zstd bitshuffle',
|
||||
hdf5plugin.Blosc2(cname='zstd', clevel=7, filters=hdf5plugin.Blosc2.BITSHUFFLE)),
|
||||
('blosclz delta', hdf5plugin.Blosc2(cname='blosclz', filters=hdf5plugin.Blosc2.DELTA)),
|
||||
('zlib + fletcher32', dict(**hdf5plugin.Blosc2(cname='zlib'), fletcher32=True))]"#,
|
||||
Some(cases),
|
||||
);
|
||||
}
|
||||
|
||||
/// Truncating precision is lossy: clawhdf5 must read exactly what h5py
|
||||
/// (libhdf5 with hdf5plugin) reads.
|
||||
#[cfg(feature = "blosc2")]
|
||||
#[test]
|
||||
fn blosc2_truncated_precision_reads_as_h5py() {
|
||||
if !have_python("h5py, hdf5plugin") {
|
||||
return;
|
||||
}
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("blosc2_trunc.h5");
|
||||
let n: usize = run_python(
|
||||
r#"
|
||||
import sys
|
||||
import numpy as np, h5py, hdf5plugin
|
||||
rng = np.random.default_rng(3)
|
||||
i = 0
|
||||
with h5py.File(sys.argv[1], 'w') as f:
|
||||
for dt in ['<f4', '<f8']:
|
||||
for shape, chunks in [((1000,), (300,)), ((37, 53), (10, 16))]:
|
||||
d = (rng.standard_normal(shape) * 1000).astype(dt)
|
||||
ds = f.create_dataset(f'f{i}', data=d, chunks=chunks,
|
||||
**hdf5plugin.Blosc2(cname='lz4',
|
||||
filters=hdf5plugin.Blosc2.TRUNC_PREC))
|
||||
f.create_dataset(f'r{i}', data=ds[()])
|
||||
i += 1
|
||||
print(i)
|
||||
"#,
|
||||
&[path.to_str().unwrap()],
|
||||
)
|
||||
.parse()
|
||||
.unwrap();
|
||||
let file = File::open(&path).unwrap();
|
||||
for i in 0..n {
|
||||
let got = file
|
||||
.dataset(&format!("f{i}"))
|
||||
.unwrap()
|
||||
.read_selection(&Selection::All)
|
||||
.unwrap();
|
||||
let want = file
|
||||
.dataset(&format!("r{i}"))
|
||||
.unwrap()
|
||||
.read_selection(&Selection::All)
|
||||
.unwrap();
|
||||
assert!(got == want, "trunc_prec f{i}: data differs from h5py");
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "lzf")]
|
||||
#[test]
|
||||
fn lzf_written_by_h5py_reads_exactly() {
|
||||
@@ -381,8 +495,9 @@ fn lzf_written_by_clawhdf5_reads_in_h5py() {
|
||||
});
|
||||
}
|
||||
|
||||
/// Blosc2 and ZFP are not implemented: reading them must be a clear error
|
||||
/// naming the filter, never data.
|
||||
/// ZFP is not implemented, and Blosc2 is not in a build without the
|
||||
/// `blosc2` feature: reading them must be a clear error naming the filter,
|
||||
/// never data.
|
||||
#[test]
|
||||
fn unimplemented_filters_are_a_clear_error() {
|
||||
if !have_python("h5py, hdf5plugin") {
|
||||
@@ -402,7 +517,11 @@ with h5py.File(sys.argv[1], 'w') as f:
|
||||
&[path.to_str().unwrap()],
|
||||
);
|
||||
let file = File::open(&path).unwrap();
|
||||
for (name, id, label) in [("blosc2", 32026u16, "Blosc2"), ("zfp", 32013, "ZFP")] {
|
||||
let mut missing = vec![("zfp", 32013u16, "ZFP")];
|
||||
if !cfg!(feature = "blosc2") {
|
||||
missing.push(("blosc2", 32026, "Blosc2"));
|
||||
}
|
||||
for (name, id, label) in missing {
|
||||
let err = file
|
||||
.dataset(name)
|
||||
.unwrap()
|
||||
|
||||
+10
-5
@@ -221,11 +221,16 @@ fill-value item that did is fixed).
|
||||
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
|
||||
`plugin-filters` for all), read and write, pure Rust; h5ex_d_lzf,
|
||||
h5ex_d_bshuf, h5ex_d_bzip2 and h5ex_d_blosc now read (conformance 573 of
|
||||
697 ok). **Still open:** Blosc2 (32026 — hdf5plugin stores each chunk as a
|
||||
Blosc2 super-chunk frame, and n-D chunks as B2ND arrays) and ZFP (32013);
|
||||
both fail with an `UnsupportedFilter` error that names the filter, and
|
||||
either can be plugged in with `filter_registry::register_filter` (32023,
|
||||
Granular BitRound, too, since 2026-09-26 even with the `pcodec` feature).
|
||||
697 ok). **Fixed 2026-09-26** for Blosc2 (32026, `blosc2` feature, also
|
||||
in `plugin-filters`), read only: hdf5plugin's frames and B2ND arrays, every
|
||||
codec and filter it offers; h5ex_d_blosc2 now reads (conformance 576 of
|
||||
697 ok, tank, `conformance/run.sh --no-fetch`). Blosc2 frames using
|
||||
dictionaries, lazy chunks, variable-length blocks, user-defined codecs or
|
||||
registered filters (e.g. bytedelta) are refused with an error.
|
||||
**Still open:** ZFP (32013) fails with an `UnsupportedFilter` error that
|
||||
names the filter, and can be plugged in with
|
||||
`filter_registry::register_filter` (32023, Granular BitRound, too, since
|
||||
2026-09-26 even with the `pcodec` feature). clawhdf5 cannot write Blosc2.
|
||||
- **Wrong data: a chunk whose filters decode to fewer bytes than the chunk
|
||||
read with zeros for the missing bytes** (any filter; found reviewing the plugin
|
||||
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
|
||||
|
||||
+2
-2
@@ -67,7 +67,7 @@ run_step "cargo clippy --all-targets" cargo clippy \
|
||||
|
||||
# 3. Clippy over clawhdf5-format's optional features, which the default
|
||||
# workspace build never compiles (szip is left out: it needs libaec).
|
||||
# plugin-filters = bitshuffle, bzip2, blosc (and the default-on lzf).
|
||||
# plugin-filters = bitshuffle, bzip2, blosc, blosc2 (and the default-on lzf).
|
||||
run_step "cargo clippy (format feature matrix)" cargo clippy \
|
||||
-p clawhdf5-format \
|
||||
--all-targets \
|
||||
@@ -78,7 +78,7 @@ run_step "cargo clippy (format feature matrix)" cargo clippy \
|
||||
# dependencies (bitshuffle and blosc share code).
|
||||
plugin_filters_alone() {
|
||||
local f
|
||||
for f in bitshuffle bzip2 blosc; do
|
||||
for f in bitshuffle bzip2 blosc blosc2; do
|
||||
echo "--- $f"
|
||||
cargo clippy -p clawhdf5-format --all-targets --features "$f" -- -D warnings || return 1
|
||||
done
|
||||
|
||||
Reference in New Issue
Block a user