From 378afa1584164109fc5c5651bb28cc07dcb9ed1b Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 10:40:18 -0500 Subject: [PATCH] docs: changelog and known issues for the remaining conformance errors Conformance on tank, conformance/run.sh --no-fetch (2026-09-26): 597 of 697 ok, 6 our-errors (4 corrupt objects HDF5 2.0 reads through a bug, the Blosc2 and ZFP filters), 2 mismatches (the known h5py big-endian VL bug). Closes the known-issues entries for metadata cache images, cve-2024-32624, cve-2020-10810/10812, and unfiltered chunks of the wrong size; the N-Bit / 64-bit scale-offset entry is recorded as not our bug. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 54 ++++++++++++++++++++++++++++++++++++++++++++ docs/known-issues.md | 37 +++++++++++++++++++++++++----- 2 files changed, 85 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 29804fc..c13a8da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,60 @@ ## Unreleased +### Remaining conformance errors (2026-09-26) +Conformance on tank, `conformance/run.sh --no-fetch`: 597 of 697 files +ok (575 before). Of the 6 our-errors left, 4 are corrupt data HDF5 2.0 +reads only through a bug (listed in `CONFORMANCE.md`), 2 are the Blosc2 and +ZFP filters; the 2 mismatches are the known h5py big-endian VL bug. +- **Metadata cache images are read.** A file written with a metadata cache + image keeps its metadata cache entries in an image block the superblock + extension points at, and libhdf5 reads them in place of the file's own + bytes; in `h5clear_mdc_image.h5` the root group exists only there, and + every reader failed with `InvalidObjectHeaderVersion(0)`. `File`, + `MmapFile` and `LazyFile` (and `h5rs`) now apply the image at open + (`clawhdf5_format::superblock_ext`), with libhdf5's checks. A file whose + image libhdf5 cannot load opens in libhdf5 but nothing in it can be read; + `File::open` refuses it. +- **The superblock extension is decoded at open, as libhdf5 does:** a File + Space Info or Metadata Cache Image message libhdf5 cannot decode makes the + open fail (`cve-2020-10810`, `cve-2020-10812` were opened). + `FormatError::InvalidSuperblockExtension`, `InvalidCacheImage`. +- **Dataset storage libhdf5 refuses at open is refused at open** + (`FormatError::InvalidDatasetStorage`, `data_read::check_dataset_storage`): + an element count times element size that overflows (`cve-2024-32624` + `/Dset_OBJREF` opened and reported its shape), contiguous storage past the + end of the file, compact data of the wrong size. An empty contiguous + dataset at a defined address, which clawhdf5 up to v2.7.0 wrote, still + opens. +- **Wrong or missing data fixed:** + - a simple dataspace of rank 0 holds one element (it held 0; + `cve-2020-18494`), and contiguous storage larger than the dataset reads + (`cve-2024-32623`, `cve-2025-2309`; libhdf5 ignores the excess); + - scale-offset: the packed codes start at byte 21 whatever size the chunk + records for `minval`, a chunk with `minbits` 0 and a fill value is all + fill values, full-width `minbits` stores the elements as they are + (these decoded differently from libhdf5); E-scale is refused, as in + libhdf5; codes past the end of the chunk stay an error + (`cve-2025-2308`, where HDF5 2.0 reads past its buffer); + - shuffle uses its own parameter as the element size, as libhdf5 does + (`cve-2025-44905`); + - an unfiltered chunk the index records at other than the chunk's size is + refused (it read with zeros for the missing bytes; `cve-2025-44904`), + as is a chunk B-tree key with a non-zero element offset. +- **Refused as libhdf5 refuses them:** a v1 group with an empty link name + fails its listing (`FormatError::InvalidLinkName`; lookups still work, + `cve-2021-46244`); dataspaces with more than 32 dimensions, a rank on a + scalar or null dataspace, or a dimension over its maximum + (`FormatError::InvalidDataspace`). +- `ObjectHeader::object_class` classifies a header as libhdf5 does (a + dataset needs a datatype *and* a dataspace). +- Conformance harness: user-defined links were listed as objects by the + reference, unopenable objects were not deduplicated, nested array types + were hashed wrong (`tarray3.h5`), and the attributes of objects h5py + cannot open were compared; all fixed. `CONFORMANCE.md` lists the corrupt + objects HDF5 2.0 reads through a bug (`bad_nbit_parms_walk.h5` among + them: libhdf5's own test now requires that read to fail). + ### Concurrent reads (2026-09-26) - **Full reads of chunked datasets scale with threads again when rayon's pool has one thread.** Each full read handed its chunks to rayon to diff --git a/docs/known-issues.md b/docs/known-issues.md index 2657cdf..b7c471a 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -202,9 +202,25 @@ fill-value item that did is fixed). its datatype message stores (12 with 4-byte offsets), and the global heap is read with libhdf5's header padding (`crates/clawhdf5/tests/vl_offset4_interop.rs`). - - Metadata cache images are not supported. + - Metadata cache images are not supported. **Fixed 2026-09-26:** the + image is applied at open, as libhdf5 loads it over the file's metadata + (`clawhdf5_format::superblock_ext`); `h5clear_mdc_image.h5` reads + (`crates/clawhdf5/tests/metadata_cache_image.rs`). A file whose image + libhdf5 cannot load (`cve-2025-6269-*`, `cve-2025-6516`) opens in + libhdf5 with nothing readable in it; `File::open` refuses it. - x87 long double and binary128 are refused. - N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail. + **Not our bug (checked 2026-09-26):** both are corrupt files HDF5 2.0 + reads only by reading past a buffer. `cve-2025-2308`'s + `/Scale_offset_long_long_data_le` has scale-offset codes that run past + the end of the chunk (libhdf5's develop branch refuses it, "Buffer too + short"); `bad_nbit_parms_walk.h5` has an N-Bit parameter list one value + short (libhdf5's own `test_filter_bad_params` in `test/dsets.c` now + requires that read to fail). We refuse both; `CONFORMANCE.md` lists them + under *Known not-our-bug*. Scale-offset did decode three cases + differently from libhdf5 (codes after a `minval` of recorded size other + than 8, `minbits` 0 with a fill value, full-width `minbits`): fixed + 2026-09-26. - **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not implemented. **Fixed 2026-09-26** for LZF (default-on `lzf` feature), bitshuffle, bzip2 and Blosc 1 (`bitshuffle`, `bzip2`, `blosc`, or @@ -219,7 +235,11 @@ fill-value item that did is fixed). read with zeros for the missing bytes** (any filter; found reviewing the plugin filters). **Fixed 2026-09-26:** it is an error naming the chunk. A corrupt chunk must never read as zeros. Unfiltered chunks are read at their stored - size and are not checked this way. + size and are not checked this way. **Fixed 2026-09-26** for unfiltered + chunks too: in a dataset without filters a chunk the index records at + other than the chunk's size is refused, as libhdf5's develop branch + refuses it (`cve-2025-44904`, where HDF5 2.0 fills the rest from its + buffer). - **Crash:** a hostile Blosc chunk (frame size below its header) panicked in builds with overflow checks. **Fixed 2026-09-26**; the new decoders are fuzzed in the unit tests. @@ -232,13 +252,18 @@ fill-value item that did is fixed). refused. 17 of the 18 now fail as in libhdf5 (conformance on tank, `conformance/run.sh --no-fetch`, 2026-09-26: 571 of 697 ok). Still read where libhdf5 refuses: - - `cve-2024-32624.h5` `/Dset_OBJREF`: a dataspace whose storage size + - ~~`cve-2024-32624.h5` `/Dset_OBJREF`: a dataspace whose storage size overflows 64 bits. `File::dataset` and `shape()` succeed (libhdf5 - refuses at open); reading the values fails. - - `cve-2020-10810.h5`, `cve-2020-10812.h5` (whole files libhdf5 cannot + refuses at open); reading the values fails.~~ **Fixed 2026-09-26:** + `File::dataset` (and `MmapFile`, `LazyFile`) refuse it at open + (`FormatError::InvalidDatasetStorage`), as they do contiguous storage + past the end of the file. + - ~~`cve-2020-10810.h5`, `cve-2020-10812.h5` (whole files libhdf5 cannot open, not among the 18): libhdf5 decodes the superblock extension's File Space Info and metadata-cache-image messages at open and refuses these - files; we do not decode those messages at open. + files; we do not decode those messages at open.~~ **Fixed 2026-09-26:** + the superblock extension is decoded at open with libhdf5's checks, and + both files are refused. - Deliberately not refused, because clawhdf5 up to v2.7.0 wrote them: a float sign bit position outside the type, and a size-0 string type. - Not refused because current libhdf5 reads it though HDF5 2.0.0