edit: shrink by visiting the chunks that exist
prune_plan stored one Vec<u64> for every chunk coordinate of the region a shrink cuts off, existing or not, so a sparse dataset exhausted memory (about 62 bytes per coordinate; (4, 2e7) with chunks (1, 1) took 2.5 GB, larger extents never finished). It now places each existing chunk in H5D__chunk_prune_by_extent's walk (its pass, then its coordinates) and sorts, which gives the same chunks, order and actions in memory and time proportional to the chunks that exist. A unit test checks the plan against the full walk (kept as the test's reference) for 3000 random extents and chunk subsets. The interop test shrinks a (4, 10^12) dataset with chunks (1, 1) and 9 chunks (v1 and v2 B-tree): 0.56 s and 43 MB peak; the old code aborted on allocation under an 8 GB limit. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+4
-1
@@ -48,7 +48,10 @@
|
||||
partial edge chunk outside the extent is overwritten with the fill value,
|
||||
so it reads as fill after a later growth. Growth under early allocation
|
||||
now allocates and fills the new chunks (`H5D__chunk_allocate`), which an
|
||||
implicit index needs. Shrinking was `Error::Unsupported`.
|
||||
implicit index needs. Shrinking was `Error::Unsupported`. Only the
|
||||
chunks that exist are visited (placed in libhdf5's order), so shrinking
|
||||
a sparse dataset costs memory and time in its chunks, not in the
|
||||
coordinates cut off (a 2 x 10^12-coordinate shrink takes 0.6 s).
|
||||
- **`FileEditor::set_attr` handles dense attribute storage and creation
|
||||
order**: objects that track (and index) attribute creation order; the
|
||||
move to dense storage when an object reaches its compact limit (or an
|
||||
|
||||
Reference in New Issue
Block a user