edit: skip heap blocks too small for an attribute, as libhdf5 does

An attribute needing a heap block larger than the next one was refused
("skipping blocks too small for an object", "a first object too large for
the starting block"); once an object's move to dense storage was refused
it refused every new attribute, so 24% of set_attr calls in the review's
random workload failed.

Following H5HF__hdr_update_iter, H5HF__man_iblock_root_create/_double and
H5HF__hdr_skip_blocks, the smaller blocks are now skipped: the iterator
moves past them and they become an indirect free section with a first
row section (serialized, class 1, as H5HF__sect_indirect_serialize writes
it) and ghost normal rows, added as returned space so it merges with a
range skipped just before it (H5HF__sect_indirect_merge_row). Later
objects that best-fit a row section get a block created there
(H5HF__man_iblock_alloc_row / H5HF__sect_indirect_reduce_row: from the
start or end of the range, or from its middle, which splits it, with
libhdf5's span bookkeeping). Heaps with such sections, as libhdf5 writes
them, are now read too (they were refused at open).

dense_skipped_blocks_match_libhdf5 drives every path (merge, split, end,
last entry, row wrap) on earliest/v110/latest files against libhdf5
doing the same edits one session each; heaps, free sections and index
B-trees are equal after every phase. The refusal test now checks the
skip against libhdf5 and keeps a real refusal (last object in a block);
clawhdf5-written heaps get 1-4 KiB attributes too. The three tests fail
on the previous fheap.rs. Random workload refusals: 24% -> 2.2%, all the
documented last-object-in-a-block case.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:50:23 -05:00
co-authored by Claude Opus 5.5
parent c2ae7846c9
commit 8236b0e30a
5 changed files with 730 additions and 110 deletions
+13 -7
View File
@@ -74,13 +74,19 @@ reuse were added). `clawhdf5::FileEditor` refuses, with
an optional filter only when its own build lacks it, which none does for
these;
- attributes in dense storage when the heap cannot take them the way
libhdf5 would: an attribute that needs a heap block larger than the next
one (libhdf5 skips blocks and records them as free space — in practice an
attribute of roughly 1 to 4 KiB going into a young heap), a heap with I/O
filters or child indirect blocks (more than about 512 KiB of attributes),
free space the heap tracks outside direct blocks, replacing the last
attribute left in a heap block by one of another size (libhdf5 frees the
block), directly addressed huge objects; and shared attribute messages;
libhdf5 would: replacing the last attribute left in a heap block by one
of another size (libhdf5 frees the block), a heap with I/O filters or
child indirect blocks (more than about 512 KiB of attributes), free
space in child indirect blocks, directly addressed huge objects; and
shared attribute messages. Measured 2026-09-26 on tank with the review's
random-edit harness (120 runs of 150 random edits, `earliest`/`v110`/
`latest`, about 5600 `set_attr` calls of 8 bytes to 6 KiB): 2.2% of
`set_attr` calls are refused, every one the last-attribute-in-a-block
replacement; before blocks could be skipped (an attribute needing a heap
block larger than the next one — any attribute of about 1 KiB or more
once a heap has started, or at the move to dense storage), 24% were,
since an object whose move to dense storage was refused kept refusing
every new attribute;
- version-1 object headers asked for an attribute larger than a header
message (they have no dense storage);
- partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external