docs: changelog and known issues for the header hardening
CHANGELOG (Correctness): the new header, datatype, chunk and truncation checks, what is left out on purpose (checks HDF5 2.0 lacks; the two v2.7.0 writer quirks), the conformance numbers and the new FormatError variants. known-issues: the "Header checks" audit gap is fixed, with the one CVE object and two CVE files libhdf5 still refuses and we read. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -296,6 +296,57 @@
|
||||
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
|
||||
|
||||
### Correctness
|
||||
- **Corrupt files libhdf5 refuses are now refused instead of read.** On the
|
||||
HDF Group's CVE reproducers, 18 objects that libhdf5 (HDF5 2.0, through
|
||||
h5py) refuses to open were read by clawhdf5, some as wrong data (a chunk
|
||||
dimension of 0 read as all fill values; chunks read at offsets off the
|
||||
chunk grid). The
|
||||
parser now makes libhdf5's checks, with libhdf5's error text:
|
||||
- object headers (`FormatError::InvalidObjectHeader`): every message of a
|
||||
v1 chunk is read and more than the prefix's count is refused (the rest
|
||||
used to be dropped); v1 message sizes must be multiples of 8 and a v1
|
||||
chunk cannot end in a gap; a message running past its chunk is an error
|
||||
(it used to end the chunk quietly); contradictory message flags; a
|
||||
message of a class that cannot be shared flagged shareable; a
|
||||
reference-count message in a v1 header; malformed continuation,
|
||||
reference-count and modification-time messages; unknown v2 header
|
||||
flags.
|
||||
- datatypes (`FormatError::InvalidDatatype`): size 0; integer bits outside
|
||||
the type; float exponent/mantissa outside the type, empty or
|
||||
overlapping; a compound with no members, a member outside the compound,
|
||||
a duplicate name or overlapping members; an enum whose size differs from
|
||||
its base type's or with an empty name; array rank over 32 or a zero
|
||||
dimension; an opaque tag length that is not a multiple of 8; in a
|
||||
version-1 (unchecksummed) header, a numeric type that leaves more than
|
||||
half its bits unused (`Datatype::parse_in_header`,
|
||||
`Datatype::check_unused_bits`). A v1/v2 float's class bit 6 was read as
|
||||
VAX byte order; libhdf5 ignores it before version 3, and so does this.
|
||||
- chunked layouts (`FormatError::InvalidChunkDimensions`): a zero chunk
|
||||
dimension, a chunk rank that does not match the dataspace, a chunk of
|
||||
4 GiB or more (0x80000000-sized chunks hung the reader), and v1 B-tree
|
||||
chunk keys whose offsets are not multiples of the chunk dimensions,
|
||||
including the keys that only bound a node
|
||||
(`chunked_read::collect_chunk_info_checked`).
|
||||
- truncated files (`FormatError::TruncatedFile`, `Superblock::data_end`):
|
||||
a file shorter than the end of file its superblock records is refused
|
||||
("truncated file"), and nothing past that end is read. `File`,
|
||||
`LazyFile` and `MmapFile` do this.
|
||||
|
||||
Checks newer libhdf5 releases make but HDF5 2.0 does not (bit-field
|
||||
offsets, the variable-length kind, array sizes) are left out, so files
|
||||
h5py opens still open. Two libhdf5 checks are skipped on purpose because
|
||||
clawhdf5 up to v2.7.0 wrote files that fail them without being wrong:
|
||||
the sign bit of every float at position 63, and a size-0 string type for
|
||||
an empty-string attribute (new fixtures written by v2.7.0 guard this).
|
||||
Conformance: 569 -> 571 ok (h5stat_err_refcount.h5,
|
||||
h5clear_fsm_persist_less.h5), and 17 of the 18 CVE objects now fail as in
|
||||
libhdf5 (see `docs/known-issues.md` for the one left), as do 10 files
|
||||
h5py refuses as truncated. Tests:
|
||||
`header_validation_interop.rs` (h5py writes, the test damages a copy, both
|
||||
libraries must refuse it), `legacy_writer_files.rs`, and unit tests next
|
||||
to each check. **Breaking (format crate):** `FormatError` gained
|
||||
`InvalidObjectHeader`, `InvalidDatatype`, `InvalidChunkDimensions` and
|
||||
`TruncatedFile`; an exhaustive `match` on it needs the new arms.
|
||||
- `clawhdf5-format` VDS: variable-length and reference data from a source in
|
||||
another file is refused. Those elements are global-heap IDs and object
|
||||
addresses in the source file; copied into the virtual dataset they would
|
||||
|
||||
Reference in New Issue
Block a user