docs: dense storage fixes and the real limits of big groups

The changelog, known issues and README said a group holds up to 65 535
links while a group of about 17 000 was already unreadable. Record the
fixes (child indirect blocks, the next-block offset, the index leaf cap,
refusing oversized dense messages, hard-link memoisation, dataset
attribute overwrite) and the limits that remain true: 65 535 links or
dense attributes per object, and 65 515 bytes per dense message.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

CHANGELOG.md

@@ -28,19 +28,55 @@
   Link Info message carries the flags, each link its order, and a dense
   group a creation-order B-tree (type 6). h5py then lists members in
   insertion order. Attribute creation order is not tracked.
-- A group holds at most 65 535 links (its link index is one B-tree leaf);
+- A group holds at most 65 535 links (its link index is one B-tree leaf),
-  more is an error. `GroupBuilder`'s fields changed (they were
+  and in a group of more than 8 links (dense storage) each link message
+  must be at most 65 515 bytes (one fractal heap block; huge heap objects
+  are not written); more is an error. Measured at the limit: 65 535 links
+  with 100-byte names (a 7 MB heap) read in h5py, h5dump and clawhdf5, and
+  h5py can add to the group. `GroupBuilder`'s fields changed (they were
   crate-private); `FinishedGroup` is unchanged for callers.
 - Files that use one level of groups and no new link kinds are laid out as
   before: byte-identical to the writer with the Group Info fix below
   (compared on simple, mixed dense/chunked/compact/external-link and paged
   files). Tests: h5py and clawhdf5 read the same
   tree (every path, attribute and value) from a 5-level file; soft, hard,
-  external and cyclic hard links; 10 000 links in one group, with and
+  external and cyclic hard links; 10 000, 20 000 and 65 535 links in one
-  without creation order; libhdf5 adding and deleting links in our groups;
+  group, with and without creation order; libhdf5 adding and deleting links
+  in our groups;
   `h5rs check` passes and `h5rs dump` equals h5dump
   (`crates/clawhdf5/tests/writer_groups_interop.rs`,
   `crates/clawhdf5-tools/tests/h5rs_interop.rs`).
+- **Big dense groups and attribute sets were unreadable.** The fractal heap
+  holding dense links or attributes wrote every doubling-table row as
+  direct blocks, but past the 512 KiB the root's direct blocks hold, rows
+  are child indirect blocks, and libhdf5 and `h5rs check` read them as
+  such: a group with 20 000 links of 20-byte names was written without
+  error and h5py could not list it ("incorrect metadata checksum"); 150
+  dense attributes of up to 56 KB could not be opened. This was in 2.7.0
+  too. The heap writer now writes child indirect blocks, nested as deep as
+  needed. Found on the way: an object bigger than the next block's space
+  was cut off (it now goes in the first block big enough), and h5py adding
+  a link to a heap over 64 KiB overwrote its first block (the header's
+  next-block offset was 0).
+- **h5py crashed adding a link to a group of more than about 47 700
+  links** (35 000 with creation order tracked). The link index leaf's node size gave libhdf5 room for more than
+  65 535 records, which overflows the leaf's 2-byte count. The node is now
+  capped at 65 535 records. Dense attributes use the same index builder:
+  more than 65 535 on one object used to be written with the count modulo
+  65 536, and are now an error.
+- **A dense link or attribute message over 65 515 bytes** (e.g. a soft link
+  with a long target in a group of more than 8 links) was written cut off,
+  and libhdf5 could not list the group ("object overruns end of direct
+  block"). It is now an error.
+- **Chained hard links took exponential time to resolve.** A hard-link
+  target going through other hard links resolved them again on every path
+  through them: 26 links whose targets each named the previous one twice
+  took 46 s. Each hard link is now resolved once, and a cycle is reported
+  by the link's name.
+- **A dataset attribute set twice read back as its first value**, as for
+  groups below (h5py listed the name twice). The later value now replaces
+  the earlier one; a hand-set attribute named like a provenance attribute
+  is replaced by the computed one.
 - **A group attribute set twice read back as its first value.** Setting a
   group (or root) attribute again wrote a second attribute message with the
   same name, and h5py returned the first value. The later value now replaces

README.md

@@ -429,7 +429,8 @@ b.add_external_link("raw", "raw.h5", "/data");
 b.write("groups.h5")?;
 ```
 
-A group holds at most 65 535 links; more is an error.
+A group holds at most 65 535 links; more is an error, as is a link over
+65 515 bytes (a very long soft-link target) in a group of more than 8 links.
 
 ### Agent Memory
 

docs/known-issues.md

@@ -209,11 +209,20 @@ fill-value item that did is fixed).
     h5py, h5dump and `h5rs check --data` read them
     (`crates/clawhdf5/tests/writer_groups_interop.rs`,
     `crates/clawhdf5-tools/tests/h5rs_interop.rs`). Still missing: a group
-    with more than 65 535 links (its link index is one B-tree leaf) is an
+    with more than 65 535 links, or an object with more than 65 535 dense
-    error, and attribute creation order is not tracked.
+    attributes, is an error (the index is one B-tree leaf), and attribute
+    creation order is not tracked.
+  - ~~Dense link or attribute storage past 512 KiB of messages was written
+    unreadable (child indirect blocks of the fractal heap written as direct
+    blocks).~~ **Fixed 2026-09-26** (it affected 2.7.0 too): tested with
+    20 000 and 65 535 links and with 8 MB of dense attributes, read by
+    h5py, h5dump, `h5rs check` and clawhdf5, and h5py can add links to
+    such groups.
   - ~~libhdf5 could not add a link to a group we wrote (no Group Info
     message).~~ **Fixed 2026-09-26.**
-  - Dense attribute storage for attributes over 64 KiB.
+  - Huge fractal heap objects: in dense storage (more than 8 attributes on
+    an object, or more than 8 links in a group) one attribute or link
+    message over 65 515 bytes is an error.
   - Output that HDF5 1.8 can read.
   - A B-tree v2 chunk index larger than one leaf, so datasets with several
     unlimited dimensions are limited to 65 535 chunks.