edit: skip heap blocks too small for an attribute, as libhdf5 does
An attribute needing a heap block larger than the next one was refused
("skipping blocks too small for an object", "a first object too large for
the starting block"); once an object's move to dense storage was refused
it refused every new attribute, so 24% of set_attr calls in the review's
random workload failed.
Following H5HF__hdr_update_iter, H5HF__man_iblock_root_create/_double and
H5HF__hdr_skip_blocks, the smaller blocks are now skipped: the iterator
moves past them and they become an indirect free section with a first
row section (serialized, class 1, as H5HF__sect_indirect_serialize writes
it) and ghost normal rows, added as returned space so it merges with a
range skipped just before it (H5HF__sect_indirect_merge_row). Later
objects that best-fit a row section get a block created there
(H5HF__man_iblock_alloc_row / H5HF__sect_indirect_reduce_row: from the
start or end of the range, or from its middle, which splits it, with
libhdf5's span bookkeeping). Heaps with such sections, as libhdf5 writes
them, are now read too (they were refused at open).
dense_skipped_blocks_match_libhdf5 drives every path (merge, split, end,
last entry, row wrap) on earliest/v110/latest files against libhdf5
doing the same edits one session each; heaps, free sections and index
B-trees are equal after every phase. The refusal test now checks the
skip against libhdf5 and keeps a real refusal (last object in a block);
clawhdf5-written heaps get 1-4 KiB attributes too. The three tests fail
on the previous fheap.rs. Random workload refusals: 24% -> 2.2%, all the
documented last-object-in-a-block case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+14
-4
@@ -63,10 +63,20 @@
|
||||
insertion. The heap is changed as `H5HF` changes it — best-fit free
|
||||
sections from its free-space manager (kept as libhdf5 keeps `FSHD`/
|
||||
`FSSE`), new direct blocks through the root indirect block (created,
|
||||
doubled), huge objects through the huge-object B-tree (deleted with the
|
||||
last huge object), removed objects' space merged back — with libhdf5's
|
||||
statistics: after the same attribute workload the heap, its free space
|
||||
and both index B-trees equal libhdf5's. Attributes are encoded as libhdf5
|
||||
doubled), blocks too small for an attribute skipped as libhdf5 skips
|
||||
them (`H5HF__hdr_skip_blocks`: an indirect free section with its row
|
||||
sections, serialized as libhdf5 serializes them, merged with the range
|
||||
skipped just before it, and later attributes given skipped blocks from
|
||||
either end or the middle of a range, which splits it), huge objects
|
||||
through the huge-object B-tree (deleted with the last huge object),
|
||||
removed objects' space merged back — with libhdf5's statistics: after
|
||||
the same attribute workload the heap, its free space and both index
|
||||
B-trees equal libhdf5's (`dense_skipped_blocks_match_libhdf5` covers
|
||||
every way of skipping, with libhdf5 doing one edit per session as the
|
||||
editor does). In a random attribute workload (1-4 KiB attributes among
|
||||
small ones) 24% of `set_attr` calls were refused before skipping was
|
||||
implemented; 2.2% are now, all replacements of the last attribute in a
|
||||
heap block. Attributes are encoded as libhdf5
|
||||
encodes them for a file h5py opens `r+` (message version 1, 3 for
|
||||
non-ASCII names; simple dataspaces with their maximum dimensions).
|
||||
Still refused: see `docs/known-issues.md`.
|
||||
|
||||
Reference in New Issue
Block a user