Merge branch 'feat/p2-writer-groups-links' into feat/p2-perf-coverage
# Conflicts: # CHANGELOG.md # crates/clawhdf5-tools/tests/h5rs_interop.rs
This commit is contained in:
@@ -139,6 +139,97 @@
|
||||
`crates/clawhdf5/tests/vl_data_interop.rs`). `read_vl_bytes` now also
|
||||
treats address 0 as null whatever the length, as `VlResolver` does.
|
||||
|
||||
### Writer: groups and links (2026-09-26)
|
||||
- **Nested groups, to any depth.** `FileWriter`/`FileBuilder` wrote the root
|
||||
group plus one level, and refused path-like names. Now a name may be a path
|
||||
(`create_dataset("a/b/x")`, `create_group("a/b")`, a leading `/` at the
|
||||
root) and missing intermediate groups are created, as h5py does; groups
|
||||
also nest through the new `GroupBuilder::create_group`/`add_group`. A group
|
||||
added at a path that already holds a group is merged into it (h5py's
|
||||
`require_group`); a name used twice otherwise, an empty or `"."`
|
||||
component (`"a//b"`, `"a/"`) or an absolute path below the root is an
|
||||
error. Datasets, attributes, dense attribute storage and dense link
|
||||
storage work at every level.
|
||||
- **Soft, hard and external links at any depth:** `add_soft_link(name,
|
||||
target)` (h5py's `SoftLink`; the target may dangle),
|
||||
`add_hard_link(name, target)` (h5py's `f[name] = f[target]`; the target
|
||||
path is resolved when the file is written, may go through other hard
|
||||
links, and a missing target, a soft link on the way or a cycle of
|
||||
hard-link paths is an error) and `add_external_link`, on `FileWriter`,
|
||||
`FileBuilder` and `GroupBuilder`. An object with several hard links gets
|
||||
an Object Reference Count message, so libhdf5 can delete one of the links
|
||||
without freeing the object.
|
||||
- **Link creation order:** `track_order(true)` on a `GroupBuilder`, or on
|
||||
`FileWriter`/`FileBuilder` for every group that does not set its own,
|
||||
tracks and indexes link creation order (h5py's `track_order=True`): the
|
||||
Link Info message carries the flags, each link its order, and a dense
|
||||
group a creation-order B-tree (type 6). h5py then lists members in
|
||||
insertion order. Attribute creation order is not tracked.
|
||||
- A group holds at most 65 535 links (its link index is one B-tree leaf),
|
||||
and in a group of more than 8 links (dense storage) each link message
|
||||
must be at most 65 515 bytes (one fractal heap block; huge heap objects
|
||||
are not written); more is an error. Measured at the limit: 65 535 links
|
||||
with 100-byte names (a 7 MB heap) read in h5py, h5dump and clawhdf5, and
|
||||
h5py can add to the group. `GroupBuilder`'s fields changed (they were
|
||||
crate-private); `FinishedGroup` is unchanged for callers.
|
||||
- Files that use one level of groups and no new link kinds are laid out as
|
||||
before: byte-identical to the writer with the Group Info fix below
|
||||
(compared on simple, mixed dense/chunked/compact/external-link and paged
|
||||
files). Tests: h5py and clawhdf5 read the same
|
||||
tree (every path, attribute and value) from a 5-level file; soft, hard,
|
||||
external and cyclic hard links; 10 000, 20 000 and 65 535 links in one
|
||||
group, with and without creation order; libhdf5 adding and deleting links
|
||||
in our groups;
|
||||
`h5rs check` passes and `h5rs dump` equals h5dump
|
||||
(`crates/clawhdf5/tests/writer_groups_interop.rs`,
|
||||
`crates/clawhdf5-tools/tests/h5rs_interop.rs`).
|
||||
- **Big dense groups and attribute sets were unreadable.** The fractal heap
|
||||
holding dense links or attributes wrote every doubling-table row as
|
||||
direct blocks, but past the 512 KiB the root's direct blocks hold, rows
|
||||
are child indirect blocks, and libhdf5 and `h5rs check` read them as
|
||||
such: a group with 20 000 links of 20-byte names was written without
|
||||
error and h5py could not list it ("incorrect metadata checksum"); 150
|
||||
dense attributes of up to 56 KB could not be opened. This was in 2.7.0
|
||||
too. The heap writer now writes child indirect blocks, nested as deep as
|
||||
needed. Found on the way: an object bigger than the next block's space
|
||||
was cut off (it now goes in the first block big enough), and h5py adding
|
||||
a link to a heap over 64 KiB overwrote its first block (the header's
|
||||
next-block offset was 0).
|
||||
- **h5py crashed adding a link to a group of more than about 47 700
|
||||
links** (35 000 with creation order tracked). The link index leaf's node size gave libhdf5 room for more than
|
||||
65 535 records, which overflows the leaf's 2-byte count. The node is now
|
||||
capped at 65 535 records. Dense attributes use the same index builder:
|
||||
more than 65 535 on one object used to be written with the count modulo
|
||||
65 536, and are now an error.
|
||||
- **A dense link or attribute message over 65 515 bytes** (e.g. a soft link
|
||||
with a long target in a group of more than 8 links) was written cut off,
|
||||
and libhdf5 could not list the group ("object overruns end of direct
|
||||
block"). It is now an error.
|
||||
- **Chained hard links took exponential time to resolve.** A hard-link
|
||||
target going through other hard links resolved them again on every path
|
||||
through them: 26 links whose targets each named the previous one twice
|
||||
took 46 s. Each hard link is now resolved once, and a cycle is reported
|
||||
by the link's name.
|
||||
- **A dataset attribute set twice read back as its first value**, as for
|
||||
groups below (h5py listed the name twice). The later value now replaces
|
||||
the earlier one; a hand-set attribute named like a provenance attribute
|
||||
is replaced by the computed one.
|
||||
- **A group attribute set twice read back as its first value.** Setting a
|
||||
group (or root) attribute again wrote a second attribute message with the
|
||||
same name, and h5py returned the first value. The later value now replaces
|
||||
the earlier one, as `attrs[name] = v` does in h5py — also when a group is
|
||||
merged from two builders.
|
||||
- **Non-ASCII link names were marked ASCII.** A group or dataset name such as
|
||||
`größe` was written with the ASCII character set flag (h5py reported
|
||||
`cset` 0 for it); it is now flagged UTF-8, as h5py writes it.
|
||||
- **libhdf5 could not add links to groups we wrote.** h5py in `"r+"` mode
|
||||
failed with "Unable to create link (message type not found)" on every
|
||||
group `FileWriter` wrote: libhdf5 reads a group's Group Info message before
|
||||
inserting a link, and none was written. Every group now carries one
|
||||
(version 0, default thresholds: 6 more bytes per group header, so files
|
||||
are not byte-identical to earlier versions). Regression test:
|
||||
`crates/clawhdf5/tests/writer_groups_interop.rs`.
|
||||
|
||||
### Plugin filters (2026-09-26)
|
||||
- **LZF, bitshuffle, bzip2 and Blosc read and write, in pure Rust.** Files
|
||||
written by h5py with `compression="lzf"`, or with hdf5plugin's
|
||||
|
||||
Reference in New Issue
Block a user