clawhdf5-format: the writer skips optional filters that fail, as libhdf5 does

FileBuilder stored every chunk of an LZF or Blosc dataset through the
filter with filter mask 0. libhdf5 counts LZF and Blosc output no smaller
than the chunk as a failure of the optional filter and stores the chunk raw
with the filter's mask bit set. For an LZF chunk whose stream was exactly
the chunk's size, the first libhdf5 rewrite stored raw data at the same
size and kept our stale mask 0 in the index, and h5py could no longer read
the dataset.

precompress_chunks now runs chunks through compress_chunk_masked (as
FileEditor does since f7e2ab1), sequentially and on the parallel path, and
build_chunked_data_from_precompressed records each chunk's real mask in
every index the writer builds: single chunk (layout field), Fixed Array and
Extensible Array filtered elements, and version-2 B-tree type 11 records
(create_datasets_parallel goes through the same path). The writer builds
no version-1 B-tree or implicit index. PrecompressedChunks::chunks gains
the mask. Files whose chunks all compress are byte-identical.

Latent only in the unreleased LZF/Blosc writer (added 2026-09-26); no
tagged release writes either filter.

Tests:
- plugin_filters_interop skipped_optional_filters_are_masked_as_libhdf5_masks_them:
  LZF, shuffle+LZF+fletcher32 and Blosc over random, compressible and
  alternating chunks in every index; masks equal an h5py-written twin's;
  h5py r+ rewrites and extends them; h5py, h5dump and our reader read
  every value. Before: 20 of 24 datasets had masks other than h5py's, and
  with that check disabled h5py failed to read the rewritten datasets
  ("filter returned failure during read").
- plugin_filters_interop files_whose_chunks_all_compress_are_unchanged:
  pins the pre-fix bytes of five all-compressing files.
- chunked_write skipped_lzf_chunks_are_masked_in_every_index (fails before:
  mask 0, want 2).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 14:45:51 -05:00
co-authored by Claude Opus 5.5
parent fe377266e1
commit 0e8522cfad
4 changed files with 613 additions and 12 deletions
+31
View File
@@ -1062,6 +1062,37 @@ and fails their objects (see below).
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
### Correctness
- **`FileBuilder` stored LZF and Blosc chunks with filter mask 0 even when
the filter had not shrunk them** (fixed 2026-09-26). Latent in the
unreleased LZF/Blosc writer only (added 2026-09-26, "Plugin filters"):
no tagged release writes LZF or Blosc, so v2.7.0 and earlier are
unaffected. libhdf5 treats LZF and Blosc output no smaller than the chunk
as a filter failure and, both being optional filters, stores such a chunk
unfiltered with the filter's mask bit set. clawhdf5 stored the filter's
output with a clear mask. For an LZF chunk whose stream was exactly the
chunk's size (h5py stores `[182, 0, 0, 0, 0]` in a 5-byte chunk raw),
the first libhdf5 rewrite of that chunk stored the new data raw at the
same size and, the size being unchanged, kept the stale mask 0: h5py
then failed to read the dataset ("filter returned failure during read").
The whole-file writer now runs chunks through the pipeline as libhdf5
does (`clawhdf5_format::filters::compress_chunk_masked`, as `FileEditor`
already did) and records each chunk's real mask in every chunk index it
builds (single chunk, Fixed Array, Extensible Array, version-2 B-tree;
it builds no version-1 B-tree or implicit index), in the sequential and
`parallel` paths and `create_datasets_parallel`. Files whose chunks all
compress are byte-identical to before. `PrecompressedChunks::chunks` is
now `(raw size, stored bytes, filter mask)` (**breaking** for code that
reads it). Files written before the fix read correctly; rewrite them
(with this build or `h5repack`) before modifying them with libhdf5.
Tests: `plugin_filters_interop`
`skipped_optional_filters_are_masked_as_libhdf5_masks_them` (LZF,
shuffle+LZF+fletcher32 and Blosc, random, compressible and alternating
chunks, every index: masks equal an h5py-written twin's; after h5py r+
rewrites and extends the datasets, h5py, h5dump and our reader read every
value — before the fix 20 of 24 datasets had other masks than h5py's,
and h5py could not read the rewritten `[x, 0, 0, 0, 0]` datasets) and
`files_whose_chunks_all_compress_are_unchanged`; `chunked_write`
`skipped_lzf_chunks_are_masked_in_every_index`.
- **Scale-offset data read wrong values in every release that decoded it
(v2.2.0 to v2.7.0), silently, on ordinary h5py files** (fixed
2026-09-26). Of 1480 scale-offset datasets h5py writes across every