docs: changelog and known issues for the remaining conformance errors

Conformance on tank, conformance/run.sh --no-fetch (2026-09-26): 597 of
697 ok, 6 our-errors (4 corrupt objects HDF5 2.0 reads through a bug, the
Blosc2 and ZFP filters), 2 mismatches (the known h5py big-endian VL bug).
Closes the known-issues entries for metadata cache images,
cve-2024-32624, cve-2020-10810/10812, and unfiltered chunks of the wrong
size; the N-Bit / 64-bit scale-offset entry is recorded as not our bug.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 10:40:18 -05:00
co-authored by Claude Opus 5.5
parent 67958b08d9
commit 378afa1584
2 changed files with 85 additions and 6 deletions
+54
View File
@@ -2,6 +2,60 @@
## Unreleased ## Unreleased
### Remaining conformance errors (2026-09-26)
Conformance on tank, `conformance/run.sh --no-fetch`: 597 of 697 files
ok (575 before). Of the 6 our-errors left, 4 are corrupt data HDF5 2.0
reads only through a bug (listed in `CONFORMANCE.md`), 2 are the Blosc2 and
ZFP filters; the 2 mismatches are the known h5py big-endian VL bug.
- **Metadata cache images are read.** A file written with a metadata cache
image keeps its metadata cache entries in an image block the superblock
extension points at, and libhdf5 reads them in place of the file's own
bytes; in `h5clear_mdc_image.h5` the root group exists only there, and
every reader failed with `InvalidObjectHeaderVersion(0)`. `File`,
`MmapFile` and `LazyFile` (and `h5rs`) now apply the image at open
(`clawhdf5_format::superblock_ext`), with libhdf5's checks. A file whose
image libhdf5 cannot load opens in libhdf5 but nothing in it can be read;
`File::open` refuses it.
- **The superblock extension is decoded at open, as libhdf5 does:** a File
Space Info or Metadata Cache Image message libhdf5 cannot decode makes the
open fail (`cve-2020-10810`, `cve-2020-10812` were opened).
`FormatError::InvalidSuperblockExtension`, `InvalidCacheImage`.
- **Dataset storage libhdf5 refuses at open is refused at open**
(`FormatError::InvalidDatasetStorage`, `data_read::check_dataset_storage`):
an element count times element size that overflows (`cve-2024-32624`
`/Dset_OBJREF` opened and reported its shape), contiguous storage past the
end of the file, compact data of the wrong size. An empty contiguous
dataset at a defined address, which clawhdf5 up to v2.7.0 wrote, still
opens.
- **Wrong or missing data fixed:**
- a simple dataspace of rank 0 holds one element (it held 0;
`cve-2020-18494`), and contiguous storage larger than the dataset reads
(`cve-2024-32623`, `cve-2025-2309`; libhdf5 ignores the excess);
- scale-offset: the packed codes start at byte 21 whatever size the chunk
records for `minval`, a chunk with `minbits` 0 and a fill value is all
fill values, full-width `minbits` stores the elements as they are
(these decoded differently from libhdf5); E-scale is refused, as in
libhdf5; codes past the end of the chunk stay an error
(`cve-2025-2308`, where HDF5 2.0 reads past its buffer);
- shuffle uses its own parameter as the element size, as libhdf5 does
(`cve-2025-44905`);
- an unfiltered chunk the index records at other than the chunk's size is
refused (it read with zeros for the missing bytes; `cve-2025-44904`),
as is a chunk B-tree key with a non-zero element offset.
- **Refused as libhdf5 refuses them:** a v1 group with an empty link name
fails its listing (`FormatError::InvalidLinkName`; lookups still work,
`cve-2021-46244`); dataspaces with more than 32 dimensions, a rank on a
scalar or null dataspace, or a dimension over its maximum
(`FormatError::InvalidDataspace`).
- `ObjectHeader::object_class` classifies a header as libhdf5 does (a
dataset needs a datatype *and* a dataspace).
- Conformance harness: user-defined links were listed as objects by the
reference, unopenable objects were not deduplicated, nested array types
were hashed wrong (`tarray3.h5`), and the attributes of objects h5py
cannot open were compared; all fixed. `CONFORMANCE.md` lists the corrupt
objects HDF5 2.0 reads through a bug (`bad_nbit_parms_walk.h5` among
them: libhdf5's own test now requires that read to fail).
### Concurrent reads (2026-09-26) ### Concurrent reads (2026-09-26)
- **Full reads of chunked datasets scale with threads again when rayon's - **Full reads of chunked datasets scale with threads again when rayon's
pool has one thread.** Each full read handed its chunks to rayon to pool has one thread.** Each full read handed its chunks to rayon to
+31 -6
View File
@@ -202,9 +202,25 @@ fill-value item that did is fixed).
its datatype message stores (12 with 4-byte offsets), and the global its datatype message stores (12 with 4-byte offsets), and the global
heap is read with libhdf5's header padding heap is read with libhdf5's header padding
(`crates/clawhdf5/tests/vl_offset4_interop.rs`). (`crates/clawhdf5/tests/vl_offset4_interop.rs`).
- Metadata cache images are not supported. - Metadata cache images are not supported. **Fixed 2026-09-26:** the
image is applied at open, as libhdf5 loads it over the file's metadata
(`clawhdf5_format::superblock_ext`); `h5clear_mdc_image.h5` reads
(`crates/clawhdf5/tests/metadata_cache_image.rs`). A file whose image
libhdf5 cannot load (`cve-2025-6269-*`, `cve-2025-6516`) opens in
libhdf5 with nothing readable in it; `File::open` refuses it.
- x87 long double and binary128 are refused. - x87 long double and binary128 are refused.
- N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail. - N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail.
**Not our bug (checked 2026-09-26):** both are corrupt files HDF5 2.0
reads only by reading past a buffer. `cve-2025-2308`'s
`/Scale_offset_long_long_data_le` has scale-offset codes that run past
the end of the chunk (libhdf5's develop branch refuses it, "Buffer too
short"); `bad_nbit_parms_walk.h5` has an N-Bit parameter list one value
short (libhdf5's own `test_filter_bad_params` in `test/dsets.c` now
requires that read to fail). We refuse both; `CONFORMANCE.md` lists them
under *Known not-our-bug*. Scale-offset did decode three cases
differently from libhdf5 (codes after a `minval` of recorded size other
than 8, `minbits` 0 with a fill value, full-width `minbits`): fixed
2026-09-26.
- **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not - **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not
implemented. **Fixed 2026-09-26** for LZF (default-on `lzf` feature), implemented. **Fixed 2026-09-26** for LZF (default-on `lzf` feature),
bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or
@@ -219,7 +235,11 @@ fill-value item that did is fixed).
read with zeros for the missing bytes** (any filter; found reviewing the plugin read with zeros for the missing bytes** (any filter; found reviewing the plugin
filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt
chunk must never read as zeros. Unfiltered chunks are read at their stored chunk must never read as zeros. Unfiltered chunks are read at their stored
size and are not checked this way. size and are not checked this way. **Fixed 2026-09-26** for unfiltered
chunks too: in a dataset without filters a chunk the index records at
other than the chunk's size is refused, as libhdf5's develop branch
refuses it (`cve-2025-44904`, where HDF5 2.0 fills the rest from its
buffer).
- **Crash:** a hostile Blosc chunk (frame size below its header) panicked in - **Crash:** a hostile Blosc chunk (frame size below its header) panicked in
builds with overflow checks. **Fixed 2026-09-26**; the new decoders are builds with overflow checks. **Fixed 2026-09-26**; the new decoders are
fuzzed in the unit tests. fuzzed in the unit tests.
@@ -232,13 +252,18 @@ fill-value item that did is fixed).
refused. 17 of the 18 now refused. 17 of the 18 now
fail as in libhdf5 (conformance on tank, `conformance/run.sh --no-fetch`, fail as in libhdf5 (conformance on tank, `conformance/run.sh --no-fetch`,
2026-09-26: 571 of 697 ok). Still read where libhdf5 refuses: 2026-09-26: 571 of 697 ok). Still read where libhdf5 refuses:
- `cve-2024-32624.h5` `/Dset_OBJREF`: a dataspace whose storage size - ~~`cve-2024-32624.h5` `/Dset_OBJREF`: a dataspace whose storage size
overflows 64 bits. `File::dataset` and `shape()` succeed (libhdf5 overflows 64 bits. `File::dataset` and `shape()` succeed (libhdf5
refuses at open); reading the values fails. refuses at open); reading the values fails.~~ **Fixed 2026-09-26:**
- `cve-2020-10810.h5`, `cve-2020-10812.h5` (whole files libhdf5 cannot `File::dataset` (and `MmapFile`, `LazyFile`) refuse it at open
(`FormatError::InvalidDatasetStorage`), as they do contiguous storage
past the end of the file.
- ~~`cve-2020-10810.h5`, `cve-2020-10812.h5` (whole files libhdf5 cannot
open, not among the 18): libhdf5 decodes the superblock extension's File open, not among the 18): libhdf5 decodes the superblock extension's File
Space Info and metadata-cache-image messages at open and refuses these Space Info and metadata-cache-image messages at open and refuses these
files; we do not decode those messages at open. files; we do not decode those messages at open.~~ **Fixed 2026-09-26:**
the superblock extension is decoded at open with libhdf5's checks, and
both files are refused.
- Deliberately not refused, because clawhdf5 up to v2.7.0 wrote them: a - Deliberately not refused, because clawhdf5 up to v2.7.0 wrote them: a
float sign bit position outside the type, and a size-0 string type. float sign bit position outside the type, and a size-0 string type.
- Not refused because current libhdf5 reads it though HDF5 2.0.0 - Not refused because current libhdf5 reads it though HDF5 2.0.0