clawhdf5: FileEditor skips optional filters that fail, as libhdf5 does

The editor stored every chunk through the whole pipeline with filter mask 0.
For LZF that did not shrink a chunk, h5py instead stores it raw with the
filter's mask bit set. A chunk the editor stored LZF-encoded at exactly the
raw size was then rewritten raw by libhdf5 at the same size; libhdf5 does not
touch the index entry when the size is unchanged, so the stale mask 0 stayed
and h5py (and h5dump) could no longer read the dataset.

clawhdf5_format::filters::compress_chunk_masked runs the pipeline as
H5Z_pipeline does: an optional filter (H5Z_FLAG_OPTIONAL) that fails is
skipped and its bit set, a mandatory one fails the write, and LZF/Blosc
output no smaller than the input counts as failure, as in the reference
filters (their output buffer is the input's size). Deflate, LZ4, Zstd,
bitshuffle and bzip2 never fail on size in libhdf5 and are kept as before.

Test: edit_interop optional_filters_that_fail_are_skipped — the reviewer's
repro at every libver: the editor stores the chunk exactly as h5py does
(mask 1, size 5; shuffle+LZF+fletcher32 mask 2), h5py r+ rewrites and
extends the datasets, and h5py, h5dump and our reader read every value.
Fails on the previous editor (mask 0; h5dump cannot read /u8).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 14:09:34 -05:00
co-authored by Claude Opus 5.5
parent 677dc5ec7c
commit f7e2ab12f2
4 changed files with 260 additions and 5 deletions
+9 -1
View File
@@ -19,7 +19,15 @@
index, its data blocks, super blocks and pages, and splitting B-tree
nodes, as libhdf5 does: after the same sequence of writes the B-tree has
the same number of nodes per level and the Extensible Array header the
same block statistics as libhdf5's (tested).
same block statistics as libhdf5's (tested). Filters run as libhdf5's
`H5Z_pipeline` runs them (new
`clawhdf5_format::filters::compress_chunk_masked`): an optional filter
that fails — LZF or Blosc output no smaller than the chunk — is skipped
and its filter-mask bit set, so the chunk is stored exactly as h5py
stores it; a mandatory filter that fails fails the edit. (Storing such
a chunk LZF-encoded at the raw size with a clear mask let a later
libhdf5 rewrite of it keep the stale mask, and h5py could no longer
read the dataset.)
- `resize`: grow a chunked dataset up to its maximum dimensions (h5py's
`Dataset.resize`).
- `set_attr`: add or replace an attribute in an object header, in free