fix(format): write and read the registered HDF5 LZ4 filter format

Filter 32004 chunks were framed as a 4-byte little-endian size plus one LZ4 block. That is not the registered HDF5 LZ4 format (H5Zlz4.c: 8-byte big-endian total size, 4-byte big-endian block size, then per block a 4-byte big-endian compressed length and the block, stored raw when the length equals the block size), so libhdf5 + hdf5plugin could not read our LZ4 datasets and we could not read theirs (h5ex_d_lz4.h5: "lz4: 0 is not a valid match offset"). Write the registered format (cd_values[0] is honoured as the block size, default 1 GiB like the plugin) and read it, multi-block and raw blocks included. Chunks in the old framing stay readable: an HDF5 chunk is under 4 GiB, so a registered chunk always starts with four zero bytes and is at least 12 bytes long, while an old one starts with four zero bytes only when empty (5 bytes). Tests: lz4_reads_registered_hdf5_format (chunk of the HDF Group's h5ex_d_lz4.h5, block size 3), lz4_writes_registered_hdf5_format, lz4_reads_legacy_clawhdf5_format, and hdf5plugin_reads_our_lz4 (ignored interop test; failed before with "filter returned failure during read"). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
2026-09-25 21:03:49 -05:00
parent 75bdb53342
commit aef8e766ae
4 changed files with 254 additions and 24 deletions
@@ -0,0 +1,10 @@
+# Filter conformance fixtures
+
+Files written by libhdf5 (and its registered filter plugins), used by the
+filter regression tests in `src/filters.rs` to compare our decoders against
+the values h5py/libhdf5 read from the same bytes. Chunk byte ranges quoted in
+the tests come from h5py's `DatasetID.get_chunk_info`.
+
+| File | Origin | Licence |
+|------|--------|---------|
+| `h5ex_d_lz4.h5` | HDF Group `HDF5Examples/C/H5FLT/tfiles/h5ex_d_lz4.h5` (hdf5 repository) | HDF5 licence (BSD-3-Clause style) |