304aed581309fccce336088c7238389baae2a342
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 osobhandClaude Opus 5.5
|
1ffd013de9 |
edit: delete a heap's huge-object B-tree with its last huge object
libhdf5 deletes a fractal heap's huge-object B-tree when the heap is
closed with no huge object left (H5HF__huge_term) and starts huge IDs
over. The editor left the empty tree, and a read-only libhdf5 then
failed to list the object's attributes: closing the heap tried to delete
the tree ("no write intent on file"), and h5dump failed the same way.
Replacing an object's last huge attribute (one above the heap's 4 KiB
managed limit) now deletes the tree (header and nodes freed), resets the
next huge ID and the wrapped flag, as libhdf5 does; a later huge
attribute creates a new tree. Bt2::delete frees a whole tree.
Found by the extended random-operation test; regression:
last_huge_attribute_replaced (fails before: h5dump, h5py listing), which
also compares the heap with libhdf5's after the same replacement.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
|
||
|
osobhandClaude Opus 5.5
|
dc9cfba6bb |
edit: reuse space freed earlier in the editing session
A FileEditor now keeps the space its edits free — a filtered chunk that moved, chunks a shrink removed, B-tree nodes merged away, a heap's replaced root indirect block or free-space section info, huge objects replaced — and later edits allocate from it (best fit, lowest address among equals, zeroed) before growing the file. An edit never reuses what it frees itself: until it is committed the file still refers to that space. Reused blocks are written in the commit's first phase with the space past the old end of file (nothing on disk refers to them yet), before any existing byte changes, so the crash-safety ordering holds. Space still free when the editor is dropped is leaked, as libhdf5 leaks it without a persistent free-space manager (files that have one, or use paged aggregation, are still refused at open). FileEditor::reusable_bytes reports what is left to reuse. Tests: FreeList merging and best fit, the edit-local rule and the commit split (image unit tests); a shrink followed by regrowth writing the same data reuses every removed chunk and leaves the file size unchanged, while one editor per edit grows the file, h5py/h5dump/h5rs check read both and h5py continues (freed_space_is_reused_within_a_session). measure_append_waste (edit_interop, ignored), same workload, one editor, before -> after (bytes; libhdf5 in brackets), on tank 2026-09-26: gzip chunks 1024, 1000 appends of 100: 307210 -> 306780 (306058); gzip chunks 4096, 2000 appends of 10: 119684 -> 79829 (50292); unfiltered unchanged (no space is freed). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 5.5
|
773f427f16 |
edit: dense attributes, compact-to-dense transition, creation order
FileEditor::set_attr now handles every attribute storage libhdf5 uses for version-2 object headers: - objects that track (and index) attribute creation order: compact attributes carry their creation index in the message header, the Attribute Info message its maximum; - the move to dense storage when an object reaches its compact limit (or an attribute is too large for a header message), as H5O__attr_create does it: a new fractal heap, name index (v2 B-tree type 8) and, when creation order is indexed, creation-order index (type 9); the compact attributes moved over in header message order, their messages freed; - objects already in dense storage (h5py- or clawhdf5-written): new attributes inserted (H5A__dense_insert), an attribute replaced by one of the same encoded size rewritten in its heap object (H5A__dense_write), otherwise removed from both indexes and the heap and inserted anew. edit/fheap.rs follows H5HF: managed objects go to the best-fitting free section of the heap's free-space manager (FSHD/FSSE, kept as libhdf5 keeps it — sorted sections, counts, section info reallocated when its size changes, the manager deleted when empty); otherwise to a new direct block: the root direct block of an empty heap, else the block at the allocation iterator in the root indirect block (created from the root direct block, doubled as needed), with libhdf5's managed-space, allocated space, free space and iterator bookkeeping. Objects above the managed limit are huge objects in their own space, indexed by the huge-object B-tree (type 1). Removed objects return their space merged with adjacent free space. Refused before anything is written: heaps with I/O filters, child indirect blocks, an object larger than the next heap block (libhdf5 skips blocks and records them as free space), free sections other than those inside direct blocks, removing a direct block's last object (libhdf5 frees the block), directly addressed huge objects. Attributes are encoded as libhdf5 does when h5py opens a file r+ (low bound "earliest"): message version 1 (3 for non-ASCII names), simple dataspaces with their maximum dimensions. Header chunks are now visited in libhdf5's order (FIFO), which is also the order attributes move to dense storage in. Tests (edit_coverage_interop): 40 attributes on each of a plain group, a group tracking and indexing creation order, and a dataset (earliest, v110, latest), some above the 4 KiB managed limit, then same-size rewrites: the heap statistics, free-space sections and both index B-trees node for node equal libhdf5's doing the same through h5py; then replacements of other sizes, h5py adds/deletes/rewrites; h5py, h5dump, h5rs check and our reader agree throughout, h5py's attribute count included. clawhdf5-written dense storage (tracked and untracked) is extended the same way; refusals leave the file byte for byte as it was. edit_interop's attribute test now expects dense storage and tracked creation order to work. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> |
||
|
osobhandClaude Opus 5.5
|
e9c71e5d2e |
edit: version-2 B-tree chunk indexes, shrinking, early allocation
FileEditor can now: - add, move and resize chunks of datasets with two or more unlimited dimensions (version-2 B-tree chunk index, record types 10/11). The new edit/btree2.rs follows libhdf5's H5B2 code: H5B2_update (modify, or insert into a leaf with room, or fall back to H5B2__insert), the preemptive split/redistribute loop with its two retries, split1, split_root (depth growth, node geometry per depth), redistribute2/3, cumulative record counts and the pointer widths H5B2__hdr_init derives, a checksum per node. Removal (H5B2_remove: merge2/3, redistribution, root collapse, the internal-record swap with a leaf) is there too. A dataset without an index yet gets one from the layout message's node size and split/merge percentages. - shrink a chunked dataset along any dimension (resize to a smaller shape), as H5D__set_extent / H5D__chunk_prune_by_extent do: the same chunks visited in the same order; chunks wholly outside the new extent are removed from the index (version-1 B-tree: H5B_remove with its sibling key and link fix-ups and the empty-root case; version-2 B-tree; Fixed/Extensible Array elements reset to the fill element; an implicit index keeps its chunks, as libhdf5 does) and their space noted as free; the part of each partial edge chunk outside the extent is overwritten with the fill value, so elements that come back after a later growth read as fill. - under early allocation, allocate and fill the chunks a growth brings in (H5D__chunk_allocate), which an implicit index needs: libhdf5 refills them, and they may hold the data of chunks pruned earlier. Tests (crates/clawhdf5-tools/tests/edit_coverage_interop.rs): growth in both dimensions of v110/latest files, unfiltered and deflated, gives node-for-node the version-2 B-tree libhdf5 builds (h5py with its chunk cache off, so chunks enter the index in the editor's order), through a depth increase; random chunk order; 60 random shrink/grow/write steps on Extensible Array, version-2 B-tree, Fixed Array, 1-D and implicit datasets (earliest/v110/latest, with and without gzip+shuffle) give the values h5py gets doing the same and the same index shape (version-1 and version-2 B-tree node shapes, Extensible Array statistics); h5py r+ continues on every result; h5dump and h5rs check accept them. edit_interop's version-2 B-tree case now appends instead of expecting a refusal; shrinking is no longer an error in edit_tests. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> |