quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(format): honour each bit of a chunk's filter mask

Browse Source 
A chunk's filter mask has one bit per pipeline filter; bit i set means
filter i was not applied to that chunk (an optional filter that
declined, or a direct chunk write). Every read path treated any nonzero
mask as "no filters applied" and returned the stored bytes, so a chunk
that skipped only gzip in a shuffle+gzip pipeline came back still
shuffled (h5py write_direct_chunk with filter_mask=0b10: 8 of 32 values
wrong).

decompress_chunk_masked undoes the filters the mask leaves set and skips
the rest; an unsupported filter is no longer an error when the chunk
skipped it. The full, cached, sweep, indexed, parallel and selection
(partial_read) paths all use it, and a chunk is copied straight from the
file only when every filter was skipped. decompress_chunk is the mask-0
case.

Regression: h5py_partial_filter_mask_skips_only_masked_filters (1-D
shuffle+gzip with masks 0, 0b10 and 0b11; 2-D with 0b01; full and
hyperslab reads) and filter_mask_skips_only_the_masked_filters.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-25 21:05:07 -05:00
co-authored by Claude Opus 5.5
parent 9ea44d473d
commit 585e14d5e2
5 changed files with 244 additions and 59 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-format/src/partial_read.rs
+11 -5
View File
@@ -22,7 +22,7 @@ use crate::data_read::extract_selection_from_buffer;
use crate::dataspace::Dataspace;
use crate::error::FormatError;
use crate::filter_pipeline::FilterPipeline;
use crate::filters::decompress_chunk;
use crate::filters::{all_filters_skipped, decompress_chunk_masked};
use crate::selection::Selection;
/// The smallest axis-aligned box containing every selected element, as
@@ -325,12 +325,18 @@ pub fn read_selection(
expected: at.saturating_add(chunk.chunk_size as usize),
available: file_data.len(),
})?;
// Mirrors the full-read path: a non-zero filter mask means the
// chunk was stored unfiltered.
// Mirrors the full-read path: filter-mask bit i set means
// filter i was not applied to this chunk.
let decoded;
let data: &[u8] = match pipeline {
Some(pl) if chunk.filter_mask == 0 => {
decoded = decompress_chunk(raw, pl, chunk_bytes, elem_size as u32)?;
Some(pl) if !all_filters_skipped(pl, chunk.filter_mask) => {
decoded = decompress_chunk_masked(
raw,
pl,
chunk_bytes,
elem_size as u32,
chunk.filter_mask,
)?;
&decoded
}
_ => raw,