fix(format): decode SZIP chunks the way libhdf5 does
SZIP-filtered datasets from libhdf5 came back as garbage or zeros with no error (ref_szip.h5, h5repack_szip.h5, noencoder.h5, le_data/be_data Szip_float_data_*), and 64-bit ones failed with "invalid bits per sample" (h5wasm compressed.h5). The decoder called aec_buffer_decode directly, but libhdf5 goes through szlib's SZ_BufftoBuffDecompress (H5Zszip.c), which libaec implements with reshaping (sz_compat.c). Differences, all fixed: - H5Zszip.c prefixes the stream with the 4-byte LE uncompressed size; it was fed to libaec as data. - 32- and 64-bit samples are coded as byte planes of 8-bit samples and must be de-interleaved. - The reference sample interval is ceil(pixels_per_scanline / pixels_per_block), not a fixed 128. - Scanlines that are not a whole number of blocks are padded and must be unpadded. - Byte order comes from the MSB option bit; LE data was decoded as MSB. Test: szip_decodes_libhdf5_chunks_exactly compares chunks from HDF Group test files (noencoder.h5, le_data.h5) and an h5py-written file (64-bit, 16-bit, padded scanlines, NN and EC) byte for byte with h5py's values; it failed before on the first case. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -8,3 +8,6 @@ the tests come from h5py's `DatasetID.get_chunk_info`.
|
||||
| File | Origin | Licence |
|
||||
|------|--------|---------|
|
||||
| `h5ex_d_lz4.h5` | HDF Group `HDF5Examples/C/H5FLT/tfiles/h5ex_d_lz4.h5` (hdf5 repository) | HDF5 licence (BSD-3-Clause style) |
|
||||
| `noencoder.h5` | HDF Group `test/testfiles/noencoder.h5` (hdf5 repository) | HDF5 licence (BSD-3-Clause style) |
|
||||
| `le_data.h5` | HDF Group `test/testfiles/le_data.h5` (hdf5 repository) | HDF5 licence (BSD-3-Clause style) |
|
||||
| `szip_h5py.h5` | Written for these tests with h5py 3 / libhdf5 2.0.0 (libaec szip): `f8` (8x10, chunks 4x10, `('nn', 8)`), `i8` (8x10, chunks 4x10, `('ec', 4)`), `u2` (70, chunks 35, `('nn', 8)`) | Same as this repository |
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user