docs: dense storage fixes and the real limits of big groups
The changelog, known issues and README said a group holds up to 65 535 links while a group of about 17 000 was already unreadable. Record the fixes (child indirect blocks, the next-block offset, the index leaf cap, refusing oversized dense messages, hard-link memoisation, dataset attribute overwrite) and the limits that remain true: 65 535 links or dense attributes per object, and 65 515 bytes per dense message. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+40
-4
@@ -28,19 +28,55 @@
|
||||
Link Info message carries the flags, each link its order, and a dense
|
||||
group a creation-order B-tree (type 6). h5py then lists members in
|
||||
insertion order. Attribute creation order is not tracked.
|
||||
- A group holds at most 65 535 links (its link index is one B-tree leaf);
|
||||
more is an error. `GroupBuilder`'s fields changed (they were
|
||||
- A group holds at most 65 535 links (its link index is one B-tree leaf),
|
||||
and in a group of more than 8 links (dense storage) each link message
|
||||
must be at most 65 515 bytes (one fractal heap block; huge heap objects
|
||||
are not written); more is an error. Measured at the limit: 65 535 links
|
||||
with 100-byte names (a 7 MB heap) read in h5py, h5dump and clawhdf5, and
|
||||
h5py can add to the group. `GroupBuilder`'s fields changed (they were
|
||||
crate-private); `FinishedGroup` is unchanged for callers.
|
||||
- Files that use one level of groups and no new link kinds are laid out as
|
||||
before: byte-identical to the writer with the Group Info fix below
|
||||
(compared on simple, mixed dense/chunked/compact/external-link and paged
|
||||
files). Tests: h5py and clawhdf5 read the same
|
||||
tree (every path, attribute and value) from a 5-level file; soft, hard,
|
||||
external and cyclic hard links; 10 000 links in one group, with and
|
||||
without creation order; libhdf5 adding and deleting links in our groups;
|
||||
external and cyclic hard links; 10 000, 20 000 and 65 535 links in one
|
||||
group, with and without creation order; libhdf5 adding and deleting links
|
||||
in our groups;
|
||||
`h5rs check` passes and `h5rs dump` equals h5dump
|
||||
(`crates/clawhdf5/tests/writer_groups_interop.rs`,
|
||||
`crates/clawhdf5-tools/tests/h5rs_interop.rs`).
|
||||
- **Big dense groups and attribute sets were unreadable.** The fractal heap
|
||||
holding dense links or attributes wrote every doubling-table row as
|
||||
direct blocks, but past the 512 KiB the root's direct blocks hold, rows
|
||||
are child indirect blocks, and libhdf5 and `h5rs check` read them as
|
||||
such: a group with 20 000 links of 20-byte names was written without
|
||||
error and h5py could not list it ("incorrect metadata checksum"); 150
|
||||
dense attributes of up to 56 KB could not be opened. This was in 2.7.0
|
||||
too. The heap writer now writes child indirect blocks, nested as deep as
|
||||
needed. Found on the way: an object bigger than the next block's space
|
||||
was cut off (it now goes in the first block big enough), and h5py adding
|
||||
a link to a heap over 64 KiB overwrote its first block (the header's
|
||||
next-block offset was 0).
|
||||
- **h5py crashed adding a link to a group of more than about 47 700
|
||||
links** (35 000 with creation order tracked). The link index leaf's node size gave libhdf5 room for more than
|
||||
65 535 records, which overflows the leaf's 2-byte count. The node is now
|
||||
capped at 65 535 records. Dense attributes use the same index builder:
|
||||
more than 65 535 on one object used to be written with the count modulo
|
||||
65 536, and are now an error.
|
||||
- **A dense link or attribute message over 65 515 bytes** (e.g. a soft link
|
||||
with a long target in a group of more than 8 links) was written cut off,
|
||||
and libhdf5 could not list the group ("object overruns end of direct
|
||||
block"). It is now an error.
|
||||
- **Chained hard links took exponential time to resolve.** A hard-link
|
||||
target going through other hard links resolved them again on every path
|
||||
through them: 26 links whose targets each named the previous one twice
|
||||
took 46 s. Each hard link is now resolved once, and a cycle is reported
|
||||
by the link's name.
|
||||
- **A dataset attribute set twice read back as its first value**, as for
|
||||
groups below (h5py listed the name twice). The later value now replaces
|
||||
the earlier one; a hand-set attribute named like a provenance attribute
|
||||
is replaced by the computed one.
|
||||
- **A group attribute set twice read back as its first value.** Setting a
|
||||
group (or root) attribute again wrote a second attribute message with the
|
||||
same name, and h5py returned the first value. The later value now replaces
|
||||
|
||||
Reference in New Issue
Block a user