edit: shrink by visiting the chunks that exist

prune_plan stored one Vec<u64> for every chunk coordinate of the region a
shrink cuts off, existing or not, so a sparse dataset exhausted memory
(about 62 bytes per coordinate; (4, 2e7) with chunks (1, 1) took 2.5 GB,
larger extents never finished). It now places each existing chunk in
H5D__chunk_prune_by_extent's walk (its pass, then its coordinates) and
sorts, which gives the same chunks, order and actions in memory and time
proportional to the chunks that exist.

A unit test checks the plan against the full walk (kept as the test's
reference) for 3000 random extents and chunk subsets. The interop test
shrinks a (4, 10^12) dataset with chunks (1, 1) and 9 chunks (v1 and v2
B-tree): 0.56 s and 43 MB peak; the old code aborted on allocation under an
8 GB limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 18:50:23 -05:00
co-authored by Claude Opus 5.5
parent 159e588550
commit 930921e8cb
3 changed files with 245 additions and 68 deletions
+4 -1
View File
@@ -48,7 +48,10 @@
partial edge chunk outside the extent is overwritten with the fill value,
so it reads as fill after a later growth. Growth under early allocation
now allocates and fills the new chunks (`H5D__chunk_allocate`), which an
implicit index needs. Shrinking was `Error::Unsupported`.
implicit index needs. Shrinking was `Error::Unsupported`. Only the
chunks that exist are visited (placed in libhdf5's order), so shrinking
a sparse dataset costs memory and time in its chunks, not in the
coordinates cut off (a 2 x 10^12-coordinate shrink takes 0.6 s).
- **`FileEditor::set_attr` handles dense attribute storage and creation
order**: objects that track (and index) attribute creation order; the
move to dense storage when an object reaches its compact limit (or an