From 546fdb84fab84d5d65ef3baba6bf567f1dad26c7 Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 09:02:47 -0500 Subject: [PATCH] docs: dense storage fixes and the real limits of big groups The changelog, known issues and README said a group holds up to 65 535 links while a group of about 17 000 was already unreadable. Record the fixes (child indirect blocks, the next-block offset, the index leaf cap, refusing oversized dense messages, hard-link memoisation, dataset attribute overwrite) and the limits that remain true: 65 535 links or dense attributes per object, and 65 515 bytes per dense message. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 44 ++++++++++++++++++++++++++++++++++++++++---- README.md | 3 ++- docs/known-issues.md | 15 ++++++++++++--- 3 files changed, 54 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 080045d..d1f6b8e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,19 +28,55 @@ Link Info message carries the flags, each link its order, and a dense group a creation-order B-tree (type 6). h5py then lists members in insertion order. Attribute creation order is not tracked. -- A group holds at most 65 535 links (its link index is one B-tree leaf); - more is an error. `GroupBuilder`'s fields changed (they were +- A group holds at most 65 535 links (its link index is one B-tree leaf), + and in a group of more than 8 links (dense storage) each link message + must be at most 65 515 bytes (one fractal heap block; huge heap objects + are not written); more is an error. Measured at the limit: 65 535 links + with 100-byte names (a 7 MB heap) read in h5py, h5dump and clawhdf5, and + h5py can add to the group. `GroupBuilder`'s fields changed (they were crate-private); `FinishedGroup` is unchanged for callers. - Files that use one level of groups and no new link kinds are laid out as before: byte-identical to the writer with the Group Info fix below (compared on simple, mixed dense/chunked/compact/external-link and paged files). Tests: h5py and clawhdf5 read the same tree (every path, attribute and value) from a 5-level file; soft, hard, - external and cyclic hard links; 10 000 links in one group, with and - without creation order; libhdf5 adding and deleting links in our groups; + external and cyclic hard links; 10 000, 20 000 and 65 535 links in one + group, with and without creation order; libhdf5 adding and deleting links + in our groups; `h5rs check` passes and `h5rs dump` equals h5dump (`crates/clawhdf5/tests/writer_groups_interop.rs`, `crates/clawhdf5-tools/tests/h5rs_interop.rs`). +- **Big dense groups and attribute sets were unreadable.** The fractal heap + holding dense links or attributes wrote every doubling-table row as + direct blocks, but past the 512 KiB the root's direct blocks hold, rows + are child indirect blocks, and libhdf5 and `h5rs check` read them as + such: a group with 20 000 links of 20-byte names was written without + error and h5py could not list it ("incorrect metadata checksum"); 150 + dense attributes of up to 56 KB could not be opened. This was in 2.7.0 + too. The heap writer now writes child indirect blocks, nested as deep as + needed. Found on the way: an object bigger than the next block's space + was cut off (it now goes in the first block big enough), and h5py adding + a link to a heap over 64 KiB overwrote its first block (the header's + next-block offset was 0). +- **h5py crashed adding a link to a group of more than about 47 700 + links** (35 000 with creation order tracked). The link index leaf's node size gave libhdf5 room for more than + 65 535 records, which overflows the leaf's 2-byte count. The node is now + capped at 65 535 records. Dense attributes use the same index builder: + more than 65 535 on one object used to be written with the count modulo + 65 536, and are now an error. +- **A dense link or attribute message over 65 515 bytes** (e.g. a soft link + with a long target in a group of more than 8 links) was written cut off, + and libhdf5 could not list the group ("object overruns end of direct + block"). It is now an error. +- **Chained hard links took exponential time to resolve.** A hard-link + target going through other hard links resolved them again on every path + through them: 26 links whose targets each named the previous one twice + took 46 s. Each hard link is now resolved once, and a cycle is reported + by the link's name. +- **A dataset attribute set twice read back as its first value**, as for + groups below (h5py listed the name twice). The later value now replaces + the earlier one; a hand-set attribute named like a provenance attribute + is replaced by the computed one. - **A group attribute set twice read back as its first value.** Setting a group (or root) attribute again wrote a second attribute message with the same name, and h5py returned the first value. The later value now replaces diff --git a/README.md b/README.md index 3b3fd22..20294ec 100644 --- a/README.md +++ b/README.md @@ -429,7 +429,8 @@ b.add_external_link("raw", "raw.h5", "/data"); b.write("groups.h5")?; ``` -A group holds at most 65 535 links; more is an error. +A group holds at most 65 535 links; more is an error, as is a link over +65 515 bytes (a very long soft-link target) in a group of more than 8 links. ### Agent Memory diff --git a/docs/known-issues.md b/docs/known-issues.md index 76fc86e..cd40971 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -209,11 +209,20 @@ fill-value item that did is fixed). h5py, h5dump and `h5rs check --data` read them (`crates/clawhdf5/tests/writer_groups_interop.rs`, `crates/clawhdf5-tools/tests/h5rs_interop.rs`). Still missing: a group - with more than 65 535 links (its link index is one B-tree leaf) is an - error, and attribute creation order is not tracked. + with more than 65 535 links, or an object with more than 65 535 dense + attributes, is an error (the index is one B-tree leaf), and attribute + creation order is not tracked. + - ~~Dense link or attribute storage past 512 KiB of messages was written + unreadable (child indirect blocks of the fractal heap written as direct + blocks).~~ **Fixed 2026-09-26** (it affected 2.7.0 too): tested with + 20 000 and 65 535 links and with 8 MB of dense attributes, read by + h5py, h5dump, `h5rs check` and clawhdf5, and h5py can add links to + such groups. - ~~libhdf5 could not add a link to a group we wrote (no Group Info message).~~ **Fixed 2026-09-26.** - - Dense attribute storage for attributes over 64 KiB. + - Huge fractal heap objects: in dense storage (more than 8 attributes on + an object, or more than 8 links in a group) one attribute or link + message over 65 515 bytes is an error. - Output that HDF5 1.8 can read. - A B-tree v2 chunk index larger than one leaf, so datasets with several unlimited dimensions are limited to 65 535 chunks.