edit: skip heap blocks too small for an attribute, as libhdf5 does

An attribute needing a heap block larger than the next one was refused
("skipping blocks too small for an object", "a first object too large for
the starting block"); once an object's move to dense storage was refused
it refused every new attribute, so 24% of set_attr calls in the review's
random workload failed.

Following H5HF__hdr_update_iter, H5HF__man_iblock_root_create/_double and
H5HF__hdr_skip_blocks, the smaller blocks are now skipped: the iterator
moves past them and they become an indirect free section with a first
row section (serialized, class 1, as H5HF__sect_indirect_serialize writes
it) and ghost normal rows, added as returned space so it merges with a
range skipped just before it (H5HF__sect_indirect_merge_row). Later
objects that best-fit a row section get a block created there
(H5HF__man_iblock_alloc_row / H5HF__sect_indirect_reduce_row: from the
start or end of the range, or from its middle, which splits it, with
libhdf5's span bookkeeping). Heaps with such sections, as libhdf5 writes
them, are now read too (they were refused at open).

dense_skipped_blocks_match_libhdf5 drives every path (merge, split, end,
last entry, row wrap) on earliest/v110/latest files against libhdf5
doing the same edits one session each; heaps, free sections and index
B-trees are equal after every phase. The refusal test now checks the
skip against libhdf5 and keeps a real refusal (last object in a block);
clawhdf5-written heaps get 1-4 KiB attributes too. The three tests fail
on the previous fheap.rs. Random workload refusals: 24% -> 2.2%, all the
documented last-object-in-a-block case.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:50:23 -05:00
co-authored by Claude Opus 5.5
parent c2ae7846c9
commit 8236b0e30a
5 changed files with 730 additions and 110 deletions
+14 -4
View File
@@ -63,10 +63,20 @@
insertion. The heap is changed as `H5HF` changes it — best-fit free
sections from its free-space manager (kept as libhdf5 keeps `FSHD`/
`FSSE`), new direct blocks through the root indirect block (created,
doubled), huge objects through the huge-object B-tree (deleted with the
last huge object), removed objects' space merged back — with libhdf5's
statistics: after the same attribute workload the heap, its free space
and both index B-trees equal libhdf5's. Attributes are encoded as libhdf5
doubled), blocks too small for an attribute skipped as libhdf5 skips
them (`H5HF__hdr_skip_blocks`: an indirect free section with its row
sections, serialized as libhdf5 serializes them, merged with the range
skipped just before it, and later attributes given skipped blocks from
either end or the middle of a range, which splits it), huge objects
through the huge-object B-tree (deleted with the last huge object),
removed objects' space merged back — with libhdf5's statistics: after
the same attribute workload the heap, its free space and both index
B-trees equal libhdf5's (`dense_skipped_blocks_match_libhdf5` covers
every way of skipping, with libhdf5 doing one edit per session as the
editor does). In a random attribute workload (1-4 KiB attributes among
small ones) 24% of `set_attr` calls were refused before skipping was
implemented; 2.2% are now, all replacements of the last attribute in a
heap block. Attributes are encoded as libhdf5
encodes them for a file h5py opens `r+` (message version 1, 3 for
non-ASCII names; simple dataspaces with their maximum dimensions).
Still refused: see `docs/known-issues.md`.