Merge branch 'fix/p1-attrs-links' into fix/p1-read-gaps
# Conflicts: # crates/clawhdf5-format/src/attribute.rs # crates/clawhdf5-format/src/group_v1.rs # crates/clawhdf5/src/lazy.rs # crates/clawhdf5/src/mmap_file.rs # docs/known-issues.md
This commit is contained in:
@@ -370,6 +370,59 @@
|
||||
flag; Fletcher32 ahead of deflate (NetCDF-4's order). Unknown-message flags
|
||||
follow libhdf5 (`tbogus.h5`): "fail if unknown" is refused, "fail if unknown
|
||||
and writing" is ignored by a reader.
|
||||
- `clawhdf5-format` reader — dense groups and attributes (links or
|
||||
attributes kept in a fractal heap indexed by a v2 B-tree):
|
||||
- A link heap larger than the root indirect block's direct rows (512 KiB
|
||||
with libhdf5's defaults: a few thousand long link names, or ~20 000 short
|
||||
ones) could not be listed: child indirect blocks were given the wrong
|
||||
number of rows, so every link stored in one was unreachable.
|
||||
- v2 B-trees of depth 3 or more (a dense group of ~22 000+ links) were
|
||||
misparsed: internal-node child pointers were read with widths from an
|
||||
estimate instead of libhdf5's per-depth record capacities, and the
|
||||
listing failed. The same B-tree code indexes dense attributes, shared
|
||||
messages and chunks.
|
||||
- Fractal-heap "huge" objects (larger than the heap's managed-object
|
||||
limit, 4 KiB by default — e.g. an 8 KiB dense attribute or a link with a
|
||||
very long name) and "tiny" objects are now read; the ID type was taken
|
||||
from the wrong bits (6-7, the version, instead of 4-5), so a huge object
|
||||
failed and took every attribute on its object down with it (NetCDF-4
|
||||
files such as netcdf4-python's `issue671.nc`). Huge objects are found
|
||||
directly from the ID or through the huge-object v2 B-tree, filtered or
|
||||
not.
|
||||
- Heaps with an I/O filter pipeline (a group created with a filter on its
|
||||
creation property list compresses its link heap) are now read: the
|
||||
header's pipeline was skipped with the wrong size, so its checksum was
|
||||
looked for in the wrong place, and filtered direct blocks were read raw.
|
||||
- A user-defined link (link class 65-255, e.g. 187 in libhdf5's
|
||||
`tall.h5`/`tudlink.h5`) made its whole group unlistable. Such links
|
||||
cannot be followed without the application that registered the class, so
|
||||
they are now left out of `datasets()`/`groups()` and path lookup, as h5py
|
||||
leaves out links it cannot open; reserved link types are still an error.
|
||||
- `clawhdf5` — soft links are listed, as h5py lists them: `datasets()` and
|
||||
`groups()` on `Group`/`MmapGroup`/`LazyGroup` include each soft link under
|
||||
its own name as the kind of object it resolves to, and `dataset(name)` /
|
||||
`group(name)` open through it. Relative targets resolve from the group
|
||||
holding the link. Dangling or cyclic soft links, external links and
|
||||
user-defined links are left out (h5py lists their names but cannot open
|
||||
them). Previously soft links were missing from the listings, and in
|
||||
old-style (symbol table) groups a soft link made the listing fail. New
|
||||
`group_v2::resolve_group_children` / `resolve_path_from` and
|
||||
`group_v1::v1_soft_links` in `clawhdf5-format`.
|
||||
- `clawhdf5` — one unreadable attribute no longer fails `attrs()` for every
|
||||
attribute on its object: it is left out of the map, and the new
|
||||
`attrs_with_errors()` (on every group and dataset handle) returns the map
|
||||
plus one error per attribute left out. Returned values are always complete.
|
||||
An error in the attribute index itself (attribute info message, dense heap
|
||||
header or B-tree) still fails the call. `clawhdf5-format` gains
|
||||
`attribute::extract_attributes_tolerant`; `extract_attributes_full` stays
|
||||
strict.
|
||||
- `clawhdf5-format` reader — files with shared object header messages
|
||||
(SOHM, `H5Pset_shared_mesg_index`): a datatype, dataspace, filter pipeline
|
||||
or attribute stored in the file's SOHM heap failed with "invalid shared
|
||||
message version: 2" — only shared fill values loaded the SOHM table — so
|
||||
such files' datasets and attributes could not be read.
|
||||
`shared_message::resolve_shared_message` now loads the table when a
|
||||
reference needs it (36 cases of the audit's read matrix).
|
||||
- `clawhdf5-format` writer — **files libhdf5 rejects or reads wrong:**
|
||||
- Extensible Array (one unlimited dimension): chunks from index 244 on were
|
||||
written but never indexed and read as 0, by libhdf5 and by us.
|
||||
|
||||
Reference in New Issue
Block a user