The editor stored every chunk through the whole pipeline with filter mask 0.
For LZF that did not shrink a chunk, h5py instead stores it raw with the
filter's mask bit set. A chunk the editor stored LZF-encoded at exactly the
raw size was then rewritten raw by libhdf5 at the same size; libhdf5 does not
touch the index entry when the size is unchanged, so the stale mask 0 stayed
and h5py (and h5dump) could no longer read the dataset.
clawhdf5_format::filters::compress_chunk_masked runs the pipeline as
H5Z_pipeline does: an optional filter (H5Z_FLAG_OPTIONAL) that fails is
skipped and its bit set, a mandatory one fails the write, and LZF/Blosc
output no smaller than the input counts as failure, as in the reference
filters (their output buffer is the input's size). Deflate, LZ4, Zstd,
bitshuffle and bzip2 never fail on size in libhdf5 and are kept as before.
Test: edit_interop optional_filters_that_fail_are_skipped — the reviewer's
repro at every libver: the editor stores the chunk exactly as h5py does
(mask 1, size 5; shuffle+LZF+fletcher32 mask 2), h5py r+ rewrites and
extends the datasets, and h5py, h5dump and our reader read every value.
Fails on the previous editor (mask 0; h5dump cannot read /u8).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
New clawhdf5::FileEditor opens an HDF5 file (h5py-written at any libver,
HDF5 2.0 format included, or clawhdf5-written) under an exclusive flock
and changes only what an edit touches:
- write_selection/write_all/write_values: compact, contiguous (also
late-allocated) and chunked datasets, any selection. Chunks are decoded,
updated and re-encoded; a filtered chunk that no longer fits moves to the
end of the file unless it is the file's last structure, which grows in
place. New chunks go into v1 B-tree, Extensible Array (paged data blocks
included), Fixed Array and single-chunk indexes, created on first use.
- resize: grow chunked datasets up to maxshape.
- set_attr: add/replace compact attributes, in a NIL slot or a new
continuation chunk.
Each edit is planned in an in-memory image and refused whole
(Error::Unsupported) when any part is unsupported (v2 B-tree / implicit
new chunks, shrinking, vlen/reference data, dense or order-tracked
attributes, cache images, paged/persistent free space). Commit writes and
syncs new space before patching existing bytes. Layout v5 (HDF5 2.0)
array indexes use 8-byte filtered chunk sizes, as libhdf5 does.
Error gains Unsupported/InvalidArgument/Locked and is #[non_exhaustive];
the Python bindings map them. build_attr_message is public.
Tests (h5py, h5dump, h5rs check --data after every round; h5py r+
afterwards): appends crossing EA super/data blocks and B-tree splits, the
same B-tree node counts and EA statistics as libhdf5 for the same writes
(in order, reversed and shuffled; paged blocks), every layout and chunk
index overwritten under random selections, attributes to continuation
chunks, random operations against a model, refused edits leave the file
byte-identical, locking.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
