docs: editor coverage — version-2 B-trees, shrinking, dense attributes, reuse

CHANGELOG (Unreleased): the new FileEditor operations, space reuse, and
the two reader fixes (implicit index grid, object-header continuation
chains). known-issues: the editor's remaining refusals (skipped heap
blocks, heaps with filters or child indirect blocks, freeing a heap
block, implicit-index insertions, ...) and the append-waste sizes before
and after reuse (measure_append_waste, tank 2026-09-26; file sizes are
deterministic). range-reads design: status note on the reader changes.
README and CLAUDE.md: what the editor covers and how to test it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 17:13:31 -05:00
co-authored by Claude Opus 5.5
parent 955fdb660d
commit 0aca0eb724
5 changed files with 124 additions and 32 deletions
+68 -1
View File
@@ -2,6 +2,72 @@
## Unreleased
### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
- **`FileEditor` adds, moves and resizes chunks of datasets with two or
more unlimited dimensions** (version-2 B-tree chunk index, record types
10/11), as libhdf5's `H5B2` code does: `H5B2_update`'s insert-or-modify,
the preemptive split/redistribute loop, `split1`/`split_root` (depth
growth), `redistribute2/3`, and removal with `merge2/3`, root collapse
and the internal-record swap; node pointer widths and cumulative record
counts per depth; a missing index is created from the layout message's
parameters. After the same growth libhdf5's and the editor's trees are
node for node the same (tested through a depth increase).
- **`FileEditor::resize` shrinks** along any dimension (h5py's
`Dataset.resize` to a smaller shape), as `H5D__chunk_prune_by_extent`
does, visiting the same chunks in the same order: chunks wholly outside
the new extent leave the index (version-1 B-tree removal with libhdf5's
sibling key and link fix-ups and empty-root case, version-2 B-tree
removal, Fixed/Extensible Array elements reset; an implicit index keeps
its chunks, as in libhdf5) and their space is freed; the part of a
partial edge chunk outside the extent is overwritten with the fill value,
so it reads as fill after a later growth. Growth under early allocation
now allocates and fills the new chunks (`H5D__chunk_allocate`), which an
implicit index needs. Shrinking was `Error::Unsupported`.
- **`FileEditor::set_attr` handles dense attribute storage and creation
order**: objects that track (and index) attribute creation order; the
move to dense storage when an object reaches its compact limit (or an
attribute is too large for a header message), as `H5O__attr_create`
does it (new fractal heap, name index, creation-order index when
indexed, compact attributes moved over in header order); objects
already in dense storage (h5py- or clawhdf5-written): insertion,
same-size rewrites in place, other replacements by removal and
insertion. The heap is changed as `H5HF` changes it — best-fit free
sections from its free-space manager (kept as libhdf5 keeps `FSHD`/
`FSSE`), new direct blocks through the root indirect block (created,
doubled), huge objects through the huge-object B-tree (deleted with the
last huge object), removed objects' space merged back — with libhdf5's
statistics: after the same attribute workload the heap, its free space
and both index B-trees equal libhdf5's. Attributes are encoded as libhdf5
encodes them for a file h5py opens `r+` (message version 1, 3 for
non-ASCII names; simple dataspaces with their maximum dimensions).
Still refused: see `docs/known-issues.md`.
- **Freed space is reused within an editing session.** A `FileEditor`
reuses (best fit, zeroed) what its earlier edits freed — moved filtered
chunks, pruned chunks, merged B-tree nodes, replaced heap blocks — never
what the current edit frees, and writes reused blocks with the new space
before any existing byte changes. `FileEditor::reusable_bytes`. The
append workload of `measure_append_waste` leaks less (sizes in
`docs/known-issues.md`).
- **Reader: implicit chunk indexes below their maximum shape.** libhdf5
places an implicit index's chunks by their position in the *maximum*
chunk grid; the reader used the current grid and returned other chunks'
values from the second chunk row on (h5py early allocation with a fixed
`maxshape` larger than the shape).
`chunked_read::generate_implicit_chunks_in_grid` takes the maximum.
- **Reader: object headers with long continuation chains.** A version-1
header whose continuation chunks chain more than 32 deep (a header that
gains a chunk per attribute added when full, as libhdf5 and the editor
grow it) was refused with `NestingDepthExceeded`; version-2 headers
stopped at 256 chunks. Chunks are now followed without recursion, in the
same order; a chunk address seen twice (a cycle) or more than 65 536
chunks are refused.
- Tests: `crates/clawhdf5-tools/tests/edit_coverage_interop.rs` (h5py
`earliest`/`v110`/`latest` and clawhdf5-written files; structure
comparisons with libhdf5 for version-2 B-trees, shrink on every index,
and dense attribute heaps); the random-operation property test in
`edit_interop.rs` now shrinks, grows two unlimited dimensions and moves
attributes to dense storage (`CLAWHDF5_EDIT_SEED` for other seeds).
### Name lookups through the name index (2026-09-26)
- **Finding one link or attribute by name reads the name index, not every
entry.** In a dense group (links in a fractal heap) the v2 B-tree name
@@ -237,7 +303,8 @@
before any existing byte changes, then the metadata that links it in,
then a second sync. There is no journal: a crash during the second
phase can leave the file inconsistent (as with libhdf5 without SWMR).
Freed space is not reused (see `docs/known-issues.md`).
Freed space is not reused (see `docs/known-issues.md`; since reused
within an editing session, above).
- Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
`v114` and `latest` files and clawhdf5 files; after every round h5py
reads the expected values, h5dump and `h5rs check --data` accept the