diff --git a/CHANGELOG.md b/CHANGELOG.md index 1fc429d..93cf91e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -296,6 +296,57 @@ - CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake. ### Correctness +- **Corrupt files libhdf5 refuses are now refused instead of read.** On the + HDF Group's CVE reproducers, 18 objects that libhdf5 (HDF5 2.0, through + h5py) refuses to open were read by clawhdf5, some as wrong data (a chunk + dimension of 0 read as all fill values; chunks read at offsets off the + chunk grid). The + parser now makes libhdf5's checks, with libhdf5's error text: + - object headers (`FormatError::InvalidObjectHeader`): every message of a + v1 chunk is read and more than the prefix's count is refused (the rest + used to be dropped); v1 message sizes must be multiples of 8 and a v1 + chunk cannot end in a gap; a message running past its chunk is an error + (it used to end the chunk quietly); contradictory message flags; a + message of a class that cannot be shared flagged shareable; a + reference-count message in a v1 header; malformed continuation, + reference-count and modification-time messages; unknown v2 header + flags. + - datatypes (`FormatError::InvalidDatatype`): size 0; integer bits outside + the type; float exponent/mantissa outside the type, empty or + overlapping; a compound with no members, a member outside the compound, + a duplicate name or overlapping members; an enum whose size differs from + its base type's or with an empty name; array rank over 32 or a zero + dimension; an opaque tag length that is not a multiple of 8; in a + version-1 (unchecksummed) header, a numeric type that leaves more than + half its bits unused (`Datatype::parse_in_header`, + `Datatype::check_unused_bits`). A v1/v2 float's class bit 6 was read as + VAX byte order; libhdf5 ignores it before version 3, and so does this. + - chunked layouts (`FormatError::InvalidChunkDimensions`): a zero chunk + dimension, a chunk rank that does not match the dataspace, a chunk of + 4 GiB or more (0x80000000-sized chunks hung the reader), and v1 B-tree + chunk keys whose offsets are not multiples of the chunk dimensions, + including the keys that only bound a node + (`chunked_read::collect_chunk_info_checked`). + - truncated files (`FormatError::TruncatedFile`, `Superblock::data_end`): + a file shorter than the end of file its superblock records is refused + ("truncated file"), and nothing past that end is read. `File`, + `LazyFile` and `MmapFile` do this. + + Checks newer libhdf5 releases make but HDF5 2.0 does not (bit-field + offsets, the variable-length kind, array sizes) are left out, so files + h5py opens still open. Two libhdf5 checks are skipped on purpose because + clawhdf5 up to v2.7.0 wrote files that fail them without being wrong: + the sign bit of every float at position 63, and a size-0 string type for + an empty-string attribute (new fixtures written by v2.7.0 guard this). + Conformance: 569 -> 571 ok (h5stat_err_refcount.h5, + h5clear_fsm_persist_less.h5), and 17 of the 18 CVE objects now fail as in + libhdf5 (see `docs/known-issues.md` for the one left), as do 10 files + h5py refuses as truncated. Tests: + `header_validation_interop.rs` (h5py writes, the test damages a copy, both + libraries must refuse it), `legacy_writer_files.rs`, and unit tests next + to each check. **Breaking (format crate):** `FormatError` gained + `InvalidObjectHeader`, `InvalidDatatype`, `InvalidChunkDimensions` and + `TruncatedFile`; an exhaustive `match` on it needs the new arms. - `clawhdf5-format` VDS: variable-length and reference data from a source in another file is refused. Those elements are global-heap IDs and object addresses in the source file; copied into the virtual dataset they would diff --git a/docs/known-issues.md b/docs/known-issues.md index 4ea7518..07c888f 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -134,8 +134,30 @@ fill-value item that did is fixed). - N-Bit on 64-bit scale-offset data and some N-Bit parameter layouts fail. - **Filters:** blosc, blosc2, bitshuffle, bzip2, LZF and zfp are not implemented. -- **Header checks:** on 12 CVE datasets libhdf5 rejects a corrupt header and - we read data anyway. We need stricter header checks. +- ~~**Header checks:** on 12 CVE datasets libhdf5 rejects a corrupt header and + we read data anyway. We need stricter header checks.~~ **Fixed + 2026-09-26** (counted again: 18 objects on the CVE corpus that libhdf5 + refuses; some read as wrong data, e.g. a zero chunk dimension read as all + fill values): object headers, datatypes, chunk dimensions and chunk-index + offsets are checked as libhdf5 checks them, and truncated files are + refused. 17 of the 18 now + fail as in libhdf5 (conformance on tank, `conformance/run.sh --no-fetch`, + 2026-09-26: 571 of 697 ok). Still read where libhdf5 refuses: + - `cve-2024-32624.h5` `/Dset_OBJREF`: a dataspace whose storage size + overflows 64 bits. `File::dataset` and `shape()` succeed (libhdf5 + refuses at open); reading the values fails. + - `cve-2020-10810.h5`, `cve-2020-10812.h5` (whole files libhdf5 cannot + open, not among the 18): libhdf5 decodes the superblock extension's File + Space Info and metadata-cache-image messages at open and refuses these + files; we do not decode those messages at open. + - Deliberately not refused, because clawhdf5 up to v2.7.0 wrote them: a + float sign bit position outside the type, and a size-0 string type. + - Not refused because HDF5 2.0 (h5py 3.16) reads them though newer + libhdf5 refuses them: bit-field offset/precision outside the type, an + unknown variable-length kind, an array type whose stored size is not + its element count times its base size. + - (`cve-2024-32616` `/group1/dset3` and `cve-2025-2309`'s `Comp_OBJREF` + attribute are h5py/numpy type-mapping failures, not libhdf5 refusals.) - **Writer:** - Nested groups beyond one level: path-like names are now refused, not created.