edit: skip heap blocks too small for an attribute, as libhdf5 does
An attribute needing a heap block larger than the next one was refused
("skipping blocks too small for an object", "a first object too large for
the starting block"); once an object's move to dense storage was refused
it refused every new attribute, so 24% of set_attr calls in the review's
random workload failed.
Following H5HF__hdr_update_iter, H5HF__man_iblock_root_create/_double and
H5HF__hdr_skip_blocks, the smaller blocks are now skipped: the iterator
moves past them and they become an indirect free section with a first
row section (serialized, class 1, as H5HF__sect_indirect_serialize writes
it) and ghost normal rows, added as returned space so it merges with a
range skipped just before it (H5HF__sect_indirect_merge_row). Later
objects that best-fit a row section get a block created there
(H5HF__man_iblock_alloc_row / H5HF__sect_indirect_reduce_row: from the
start or end of the range, or from its middle, which splits it, with
libhdf5's span bookkeeping). Heaps with such sections, as libhdf5 writes
them, are now read too (they were refused at open).
dense_skipped_blocks_match_libhdf5 drives every path (merge, split, end,
last entry, row wrap) on earliest/v110/latest files against libhdf5
doing the same edits one session each; heaps, free sections and index
B-trees are equal after every phase. The refusal test now checks the
skip against libhdf5 and keeps a real refusal (last object in a block);
clawhdf5-written heaps get 1-4 KiB attributes too. The three tests fail
on the previous fheap.rs. Random workload refusals: 24% -> 2.2%, all the
documented last-object-in-a-block case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+13
-7
@@ -74,13 +74,19 @@ reuse were added). `clawhdf5::FileEditor` refuses, with
|
||||
an optional filter only when its own build lacks it, which none does for
|
||||
these;
|
||||
- attributes in dense storage when the heap cannot take them the way
|
||||
libhdf5 would: an attribute that needs a heap block larger than the next
|
||||
one (libhdf5 skips blocks and records them as free space — in practice an
|
||||
attribute of roughly 1 to 4 KiB going into a young heap), a heap with I/O
|
||||
filters or child indirect blocks (more than about 512 KiB of attributes),
|
||||
free space the heap tracks outside direct blocks, replacing the last
|
||||
attribute left in a heap block by one of another size (libhdf5 frees the
|
||||
block), directly addressed huge objects; and shared attribute messages;
|
||||
libhdf5 would: replacing the last attribute left in a heap block by one
|
||||
of another size (libhdf5 frees the block), a heap with I/O filters or
|
||||
child indirect blocks (more than about 512 KiB of attributes), free
|
||||
space in child indirect blocks, directly addressed huge objects; and
|
||||
shared attribute messages. Measured 2026-09-26 on tank with the review's
|
||||
random-edit harness (120 runs of 150 random edits, `earliest`/`v110`/
|
||||
`latest`, about 5600 `set_attr` calls of 8 bytes to 6 KiB): 2.2% of
|
||||
`set_attr` calls are refused, every one the last-attribute-in-a-block
|
||||
replacement; before blocks could be skipped (an attribute needing a heap
|
||||
block larger than the next one — any attribute of about 1 KiB or more
|
||||
once a heap has started, or at the move to dense storage), 24% were,
|
||||
since an object whose move to dense storage was refused kept refusing
|
||||
every new attribute;
|
||||
- version-1 object headers asked for an attribute larger than a header
|
||||
message (they have no dense storage);
|
||||
- partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
|
||||
|
||||
Reference in New Issue
Block a user