docs: editor coverage — version-2 B-trees, shrinking, dense attributes, reuse
CHANGELOG (Unreleased): the new FileEditor operations, space reuse, and the two reader fixes (implicit index grid, object-header continuation chains). known-issues: the editor's remaining refusals (skipped heap blocks, heaps with filters or child indirect blocks, freeing a heap block, implicit-index insertions, ...) and the append-waste sizes before and after reuse (measure_append_waste, tank 2026-09-26; file sizes are deterministic). range-reads design: status note on the reader changes. README and CLAUDE.md: what the editor covers and how to test it. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+68
-1
@@ -2,6 +2,72 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
|
||||
- **`FileEditor` adds, moves and resizes chunks of datasets with two or
|
||||
more unlimited dimensions** (version-2 B-tree chunk index, record types
|
||||
10/11), as libhdf5's `H5B2` code does: `H5B2_update`'s insert-or-modify,
|
||||
the preemptive split/redistribute loop, `split1`/`split_root` (depth
|
||||
growth), `redistribute2/3`, and removal with `merge2/3`, root collapse
|
||||
and the internal-record swap; node pointer widths and cumulative record
|
||||
counts per depth; a missing index is created from the layout message's
|
||||
parameters. After the same growth libhdf5's and the editor's trees are
|
||||
node for node the same (tested through a depth increase).
|
||||
- **`FileEditor::resize` shrinks** along any dimension (h5py's
|
||||
`Dataset.resize` to a smaller shape), as `H5D__chunk_prune_by_extent`
|
||||
does, visiting the same chunks in the same order: chunks wholly outside
|
||||
the new extent leave the index (version-1 B-tree removal with libhdf5's
|
||||
sibling key and link fix-ups and empty-root case, version-2 B-tree
|
||||
removal, Fixed/Extensible Array elements reset; an implicit index keeps
|
||||
its chunks, as in libhdf5) and their space is freed; the part of a
|
||||
partial edge chunk outside the extent is overwritten with the fill value,
|
||||
so it reads as fill after a later growth. Growth under early allocation
|
||||
now allocates and fills the new chunks (`H5D__chunk_allocate`), which an
|
||||
implicit index needs. Shrinking was `Error::Unsupported`.
|
||||
- **`FileEditor::set_attr` handles dense attribute storage and creation
|
||||
order**: objects that track (and index) attribute creation order; the
|
||||
move to dense storage when an object reaches its compact limit (or an
|
||||
attribute is too large for a header message), as `H5O__attr_create`
|
||||
does it (new fractal heap, name index, creation-order index when
|
||||
indexed, compact attributes moved over in header order); objects
|
||||
already in dense storage (h5py- or clawhdf5-written): insertion,
|
||||
same-size rewrites in place, other replacements by removal and
|
||||
insertion. The heap is changed as `H5HF` changes it — best-fit free
|
||||
sections from its free-space manager (kept as libhdf5 keeps `FSHD`/
|
||||
`FSSE`), new direct blocks through the root indirect block (created,
|
||||
doubled), huge objects through the huge-object B-tree (deleted with the
|
||||
last huge object), removed objects' space merged back — with libhdf5's
|
||||
statistics: after the same attribute workload the heap, its free space
|
||||
and both index B-trees equal libhdf5's. Attributes are encoded as libhdf5
|
||||
encodes them for a file h5py opens `r+` (message version 1, 3 for
|
||||
non-ASCII names; simple dataspaces with their maximum dimensions).
|
||||
Still refused: see `docs/known-issues.md`.
|
||||
- **Freed space is reused within an editing session.** A `FileEditor`
|
||||
reuses (best fit, zeroed) what its earlier edits freed — moved filtered
|
||||
chunks, pruned chunks, merged B-tree nodes, replaced heap blocks — never
|
||||
what the current edit frees, and writes reused blocks with the new space
|
||||
before any existing byte changes. `FileEditor::reusable_bytes`. The
|
||||
append workload of `measure_append_waste` leaks less (sizes in
|
||||
`docs/known-issues.md`).
|
||||
- **Reader: implicit chunk indexes below their maximum shape.** libhdf5
|
||||
places an implicit index's chunks by their position in the *maximum*
|
||||
chunk grid; the reader used the current grid and returned other chunks'
|
||||
values from the second chunk row on (h5py early allocation with a fixed
|
||||
`maxshape` larger than the shape).
|
||||
`chunked_read::generate_implicit_chunks_in_grid` takes the maximum.
|
||||
- **Reader: object headers with long continuation chains.** A version-1
|
||||
header whose continuation chunks chain more than 32 deep (a header that
|
||||
gains a chunk per attribute added when full, as libhdf5 and the editor
|
||||
grow it) was refused with `NestingDepthExceeded`; version-2 headers
|
||||
stopped at 256 chunks. Chunks are now followed without recursion, in the
|
||||
same order; a chunk address seen twice (a cycle) or more than 65 536
|
||||
chunks are refused.
|
||||
- Tests: `crates/clawhdf5-tools/tests/edit_coverage_interop.rs` (h5py
|
||||
`earliest`/`v110`/`latest` and clawhdf5-written files; structure
|
||||
comparisons with libhdf5 for version-2 B-trees, shrink on every index,
|
||||
and dense attribute heaps); the random-operation property test in
|
||||
`edit_interop.rs` now shrinks, grows two unlimited dimensions and moves
|
||||
attributes to dense storage (`CLAWHDF5_EDIT_SEED` for other seeds).
|
||||
|
||||
### Name lookups through the name index (2026-09-26)
|
||||
- **Finding one link or attribute by name reads the name index, not every
|
||||
entry.** In a dense group (links in a fractal heap) the v2 B-tree name
|
||||
@@ -237,7 +303,8 @@
|
||||
before any existing byte changes, then the metadata that links it in,
|
||||
then a second sync. There is no journal: a crash during the second
|
||||
phase can leave the file inconsistent (as with libhdf5 without SWMR).
|
||||
Freed space is not reused (see `docs/known-issues.md`).
|
||||
Freed space is not reused (see `docs/known-issues.md`; since reused
|
||||
within an editing session, above).
|
||||
- Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
|
||||
`v114` and `latest` files and clawhdf5 files; after every round h5py
|
||||
reads the expected values, h5dump and `h5rs check --data` accept the
|
||||
|
||||
Reference in New Issue
Block a user