Merge branch 'fix/p1-attrs-links' into fix/p1-read-gaps

# Conflicts:
#	crates/clawhdf5-format/src/attribute.rs
#	crates/clawhdf5-format/src/group_v1.rs
#	crates/clawhdf5/src/lazy.rs
#	crates/clawhdf5/src/mmap_file.rs
#	docs/known-issues.md
This commit is contained in:
osobh
2026-09-25 22:41:33 -05:00
17 changed files with 1568 additions and 378 deletions
+53
View File
@@ -370,6 +370,59 @@
flag; Fletcher32 ahead of deflate (NetCDF-4's order). Unknown-message flags
follow libhdf5 (`tbogus.h5`): "fail if unknown" is refused, "fail if unknown
and writing" is ignored by a reader.
- `clawhdf5-format` reader — dense groups and attributes (links or
attributes kept in a fractal heap indexed by a v2 B-tree):
- A link heap larger than the root indirect block's direct rows (512 KiB
with libhdf5's defaults: a few thousand long link names, or ~20 000 short
ones) could not be listed: child indirect blocks were given the wrong
number of rows, so every link stored in one was unreachable.
- v2 B-trees of depth 3 or more (a dense group of ~22 000+ links) were
misparsed: internal-node child pointers were read with widths from an
estimate instead of libhdf5's per-depth record capacities, and the
listing failed. The same B-tree code indexes dense attributes, shared
messages and chunks.
- Fractal-heap "huge" objects (larger than the heap's managed-object
limit, 4 KiB by default — e.g. an 8 KiB dense attribute or a link with a
very long name) and "tiny" objects are now read; the ID type was taken
from the wrong bits (6-7, the version, instead of 4-5), so a huge object
failed and took every attribute on its object down with it (NetCDF-4
files such as netcdf4-python's `issue671.nc`). Huge objects are found
directly from the ID or through the huge-object v2 B-tree, filtered or
not.
- Heaps with an I/O filter pipeline (a group created with a filter on its
creation property list compresses its link heap) are now read: the
header's pipeline was skipped with the wrong size, so its checksum was
looked for in the wrong place, and filtered direct blocks were read raw.
- A user-defined link (link class 65-255, e.g. 187 in libhdf5's
`tall.h5`/`tudlink.h5`) made its whole group unlistable. Such links
cannot be followed without the application that registered the class, so
they are now left out of `datasets()`/`groups()` and path lookup, as h5py
leaves out links it cannot open; reserved link types are still an error.
- `clawhdf5` — soft links are listed, as h5py lists them: `datasets()` and
`groups()` on `Group`/`MmapGroup`/`LazyGroup` include each soft link under
its own name as the kind of object it resolves to, and `dataset(name)` /
`group(name)` open through it. Relative targets resolve from the group
holding the link. Dangling or cyclic soft links, external links and
user-defined links are left out (h5py lists their names but cannot open
them). Previously soft links were missing from the listings, and in
old-style (symbol table) groups a soft link made the listing fail. New
`group_v2::resolve_group_children` / `resolve_path_from` and
`group_v1::v1_soft_links` in `clawhdf5-format`.
- `clawhdf5` — one unreadable attribute no longer fails `attrs()` for every
attribute on its object: it is left out of the map, and the new
`attrs_with_errors()` (on every group and dataset handle) returns the map
plus one error per attribute left out. Returned values are always complete.
An error in the attribute index itself (attribute info message, dense heap
header or B-tree) still fails the call. `clawhdf5-format` gains
`attribute::extract_attributes_tolerant`; `extract_attributes_full` stays
strict.
- `clawhdf5-format` reader — files with shared object header messages
(SOHM, `H5Pset_shared_mesg_index`): a datatype, dataspace, filter pipeline
or attribute stored in the file's SOHM heap failed with "invalid shared
message version: 2" — only shared fill values loaded the SOHM table — so
such files' datasets and attributes could not be read.
`shared_message::resolve_shared_message` now loads the table when a
reference needs it (36 cases of the audit's read matrix).
- `clawhdf5-format` writer — **files libhdf5 rejects or reads wrong:**
- Extensible Array (one unlimited dimension): chunks from index 244 on were
written but never indexed and read as 0, by libhdf5 and by us.