docs: changelog and known issues for the remaining conformance errors

Conformance on tank, conformance/run.sh --no-fetch (2026-09-26): 597 of
697 ok, 6 our-errors (4 corrupt objects HDF5 2.0 reads through a bug, the
Blosc2 and ZFP filters), 2 mismatches (the known h5py big-endian VL bug).
Closes the known-issues entries for metadata cache images,
cve-2024-32624, cve-2020-10810/10812, and unfiltered chunks of the wrong
size; the N-Bit / 64-bit scale-offset entry is recorded as not our bug.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 10:40:18 -05:00
co-authored by Claude Opus 5.5
parent 67958b08d9
commit 378afa1584
2 changed files with 85 additions and 6 deletions
+54
View File
@@ -2,6 +2,60 @@
## Unreleased
### Remaining conformance errors (2026-09-26)
Conformance on tank, `conformance/run.sh --no-fetch`: 597 of 697 files
ok (575 before). Of the 6 our-errors left, 4 are corrupt data HDF5 2.0
reads only through a bug (listed in `CONFORMANCE.md`), 2 are the Blosc2 and
ZFP filters; the 2 mismatches are the known h5py big-endian VL bug.
- **Metadata cache images are read.** A file written with a metadata cache
image keeps its metadata cache entries in an image block the superblock
extension points at, and libhdf5 reads them in place of the file's own
bytes; in `h5clear_mdc_image.h5` the root group exists only there, and
every reader failed with `InvalidObjectHeaderVersion(0)`. `File`,
`MmapFile` and `LazyFile` (and `h5rs`) now apply the image at open
(`clawhdf5_format::superblock_ext`), with libhdf5's checks. A file whose
image libhdf5 cannot load opens in libhdf5 but nothing in it can be read;
`File::open` refuses it.
- **The superblock extension is decoded at open, as libhdf5 does:** a File
Space Info or Metadata Cache Image message libhdf5 cannot decode makes the
open fail (`cve-2020-10810`, `cve-2020-10812` were opened).
`FormatError::InvalidSuperblockExtension`, `InvalidCacheImage`.
- **Dataset storage libhdf5 refuses at open is refused at open**
(`FormatError::InvalidDatasetStorage`, `data_read::check_dataset_storage`):
an element count times element size that overflows (`cve-2024-32624`
`/Dset_OBJREF` opened and reported its shape), contiguous storage past the
end of the file, compact data of the wrong size. An empty contiguous
dataset at a defined address, which clawhdf5 up to v2.7.0 wrote, still
opens.
- **Wrong or missing data fixed:**
- a simple dataspace of rank 0 holds one element (it held 0;
`cve-2020-18494`), and contiguous storage larger than the dataset reads
(`cve-2024-32623`, `cve-2025-2309`; libhdf5 ignores the excess);
- scale-offset: the packed codes start at byte 21 whatever size the chunk
records for `minval`, a chunk with `minbits` 0 and a fill value is all
fill values, full-width `minbits` stores the elements as they are
(these decoded differently from libhdf5); E-scale is refused, as in
libhdf5; codes past the end of the chunk stay an error
(`cve-2025-2308`, where HDF5 2.0 reads past its buffer);
- shuffle uses its own parameter as the element size, as libhdf5 does
(`cve-2025-44905`);
- an unfiltered chunk the index records at other than the chunk's size is
refused (it read with zeros for the missing bytes; `cve-2025-44904`),
as is a chunk B-tree key with a non-zero element offset.
- **Refused as libhdf5 refuses them:** a v1 group with an empty link name
fails its listing (`FormatError::InvalidLinkName`; lookups still work,
`cve-2021-46244`); dataspaces with more than 32 dimensions, a rank on a
scalar or null dataspace, or a dimension over its maximum
(`FormatError::InvalidDataspace`).
- `ObjectHeader::object_class` classifies a header as libhdf5 does (a
dataset needs a datatype *and* a dataspace).
- Conformance harness: user-defined links were listed as objects by the
reference, unopenable objects were not deduplicated, nested array types
were hashed wrong (`tarray3.h5`), and the attributes of objects h5py
cannot open were compared; all fixed. `CONFORMANCE.md` lists the corrupt
objects HDF5 2.0 reads through a bug (`bad_nbit_parms_walk.h5` among
them: libhdf5's own test now requires that read to fail).
### Concurrent reads (2026-09-26)
- **Full reads of chunked datasets scale with threads again when rayon's
pool has one thread.** Each full read handed its chunks to rayon to