conformance: fix three reference-probe artefacts

ref.py and compare.py reported 13 files as mismatches or our-errors that
were artefacts of the harness, not differences between the readers:

- User-defined links (tall.h5, tudlink.h5, twithub*.h5, tmany.h5, ...):
  h5py's `get(name, getlink=True)` reports a user-defined link as a
  HardLink, so ref.py listed it as an object. Read the link type from
  H5Lget_info instead.
- Objects h5py cannot open (cve-2019-8397/8398, cve-2021-46243,
  cve-2024-32618): the probe deduplicates by header address, ref.py by
  ObjectID, which an unopenable object does not have, so each extra hard
  link to it was listed again. Deduplicate those by link address.
- Nested array types (tarray3.h5): h5py expands them into trailing dims;
  hash_values stripped one level and numpy broadcast every element into a
  whole subarray. Strip every level.

compare.py no longer compares the attributes or links of an object h5py
could not open at all (cve-2018-17438/17439, cve-2019-9151): h5py read
none, so ours are neither extra nor errors against it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 10:13:11 -05:00
co-authored by Claude Opus 5.5
parent de2a53f613
commit 9e59499c56
2 changed files with 31 additions and 11 deletions
+24 -10
View File
@@ -111,8 +111,11 @@ def note_conversion(tid, dt, rec):
def hash_values(arr, dt, rec):
if dt.subdtype is not None:
# h5py expands an HDF5 array element type into trailing array dims
# h5py expands an HDF5 array element type into trailing array dims, a
# nested array type (an array of arrays) into all of them. Converting the
# expanded array back to the inner subarray type would broadcast every
# element into a whole subarray, so strip every level.
while dt.subdtype is not None:
dt = dt.subdtype[0]
arr = np.asarray(arr, dtype=dt)
if simple(dt):
@@ -173,9 +176,13 @@ def main(path):
return
objects = []
seen = set()
stack = [("/", None)]
# Objects h5py cannot open have no ObjectID to deduplicate by; they are
# deduplicated by the address their hard link points at instead, as the
# probe deduplicates every object by header address.
seen_unopenable = set()
stack = [("/", None, None)]
while stack:
p, obj = stack.pop()
p, obj, link_addr = stack.pop()
if len(objects) >= MAX_OBJECTS:
top["truncated"] = True
break
@@ -185,6 +192,10 @@ def main(path):
obj = f[p]
key = hash(obj.id) # h5py ObjectID hash = (fileno, object address/token)
except Exception as e: # noqa: BLE001
if link_addr is not None:
if link_addr in seen_unopenable:
continue
seen_unopenable.add(link_addr)
rec["kind"] = "unknown"
rec["error"] = err(e)
objects.append(rec)
@@ -232,15 +243,18 @@ def main(path):
base = "" if p == "/" else p
kids = []
for n in names:
# The link's own type: `obj.get(n, getlink=True)` reports
# a user-defined link (type 64-255) as a HardLink.
try:
link = obj.get(n, getlink=True)
info = obj.id.links.get_info(n.encode("utf-8", "surrogateescape"))
except Exception: # noqa: BLE001
link = None
if link is not None and not isinstance(link, h5py.HardLink):
info = None
if info is not None and info.type != h5py.h5l.TYPE_HARD:
continue
kids.append(f"{base}/{n}")
for k in reversed(kids):
stack.append((k, None))
addr = info.u if info is not None else None
kids.append((f"{base}/{n}", addr))
for k, addr in reversed(kids):
stack.append((k, None, addr))
except Exception as e: # noqa: BLE001
rec["list_error"] = err(e)
objects.append(rec)