Merge branch 'feat/p2-writer-groups-links' into feat/p2-perf-coverage

# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-tools/tests/h5rs_interop.rs
This commit is contained in:
osobh
2026-09-26 09:10:50 -05:00
11 changed files with 2693 additions and 614 deletions
+91
View File
@@ -139,6 +139,97 @@
`crates/clawhdf5/tests/vl_data_interop.rs`). `read_vl_bytes` now also
treats address 0 as null whatever the length, as `VlResolver` does.
### Writer: groups and links (2026-09-26)
- **Nested groups, to any depth.** `FileWriter`/`FileBuilder` wrote the root
group plus one level, and refused path-like names. Now a name may be a path
(`create_dataset("a/b/x")`, `create_group("a/b")`, a leading `/` at the
root) and missing intermediate groups are created, as h5py does; groups
also nest through the new `GroupBuilder::create_group`/`add_group`. A group
added at a path that already holds a group is merged into it (h5py's
`require_group`); a name used twice otherwise, an empty or `"."`
component (`"a//b"`, `"a/"`) or an absolute path below the root is an
error. Datasets, attributes, dense attribute storage and dense link
storage work at every level.
- **Soft, hard and external links at any depth:** `add_soft_link(name,
target)` (h5py's `SoftLink`; the target may dangle),
`add_hard_link(name, target)` (h5py's `f[name] = f[target]`; the target
path is resolved when the file is written, may go through other hard
links, and a missing target, a soft link on the way or a cycle of
hard-link paths is an error) and `add_external_link`, on `FileWriter`,
`FileBuilder` and `GroupBuilder`. An object with several hard links gets
an Object Reference Count message, so libhdf5 can delete one of the links
without freeing the object.
- **Link creation order:** `track_order(true)` on a `GroupBuilder`, or on
`FileWriter`/`FileBuilder` for every group that does not set its own,
tracks and indexes link creation order (h5py's `track_order=True`): the
Link Info message carries the flags, each link its order, and a dense
group a creation-order B-tree (type 6). h5py then lists members in
insertion order. Attribute creation order is not tracked.
- A group holds at most 65 535 links (its link index is one B-tree leaf),
and in a group of more than 8 links (dense storage) each link message
must be at most 65 515 bytes (one fractal heap block; huge heap objects
are not written); more is an error. Measured at the limit: 65 535 links
with 100-byte names (a 7 MB heap) read in h5py, h5dump and clawhdf5, and
h5py can add to the group. `GroupBuilder`'s fields changed (they were
crate-private); `FinishedGroup` is unchanged for callers.
- Files that use one level of groups and no new link kinds are laid out as
before: byte-identical to the writer with the Group Info fix below
(compared on simple, mixed dense/chunked/compact/external-link and paged
files). Tests: h5py and clawhdf5 read the same
tree (every path, attribute and value) from a 5-level file; soft, hard,
external and cyclic hard links; 10 000, 20 000 and 65 535 links in one
group, with and without creation order; libhdf5 adding and deleting links
in our groups;
`h5rs check` passes and `h5rs dump` equals h5dump
(`crates/clawhdf5/tests/writer_groups_interop.rs`,
`crates/clawhdf5-tools/tests/h5rs_interop.rs`).
- **Big dense groups and attribute sets were unreadable.** The fractal heap
holding dense links or attributes wrote every doubling-table row as
direct blocks, but past the 512 KiB the root's direct blocks hold, rows
are child indirect blocks, and libhdf5 and `h5rs check` read them as
such: a group with 20 000 links of 20-byte names was written without
error and h5py could not list it ("incorrect metadata checksum"); 150
dense attributes of up to 56 KB could not be opened. This was in 2.7.0
too. The heap writer now writes child indirect blocks, nested as deep as
needed. Found on the way: an object bigger than the next block's space
was cut off (it now goes in the first block big enough), and h5py adding
a link to a heap over 64 KiB overwrote its first block (the header's
next-block offset was 0).
- **h5py crashed adding a link to a group of more than about 47 700
links** (35 000 with creation order tracked). The link index leaf's node size gave libhdf5 room for more than
65 535 records, which overflows the leaf's 2-byte count. The node is now
capped at 65 535 records. Dense attributes use the same index builder:
more than 65 535 on one object used to be written with the count modulo
65 536, and are now an error.
- **A dense link or attribute message over 65 515 bytes** (e.g. a soft link
with a long target in a group of more than 8 links) was written cut off,
and libhdf5 could not list the group ("object overruns end of direct
block"). It is now an error.
- **Chained hard links took exponential time to resolve.** A hard-link
target going through other hard links resolved them again on every path
through them: 26 links whose targets each named the previous one twice
took 46 s. Each hard link is now resolved once, and a cycle is reported
by the link's name.
- **A dataset attribute set twice read back as its first value**, as for
groups below (h5py listed the name twice). The later value now replaces
the earlier one; a hand-set attribute named like a provenance attribute
is replaced by the computed one.
- **A group attribute set twice read back as its first value.** Setting a
group (or root) attribute again wrote a second attribute message with the
same name, and h5py returned the first value. The later value now replaces
the earlier one, as `attrs[name] = v` does in h5py — also when a group is
merged from two builders.
- **Non-ASCII link names were marked ASCII.** A group or dataset name such as
`größe` was written with the ASCII character set flag (h5py reported
`cset` 0 for it); it is now flagged UTF-8, as h5py writes it.
- **libhdf5 could not add links to groups we wrote.** h5py in `"r+"` mode
failed with "Unable to create link (message type not found)" on every
group `FileWriter` wrote: libhdf5 reads a group's Group Info message before
inserting a link, and none was written. Every group now carries one
(version 0, default thresholds: 6 more bytes per group header, so files
are not byte-identical to earlier versions). Regression test:
`crates/clawhdf5/tests/writer_groups_interop.rs`.
### Plugin filters (2026-09-26)
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
written by h5py with `compression="lzf"`, or with hdf5plugin's