docs: editor coverage — version-2 B-trees, shrinking, dense attributes, reuse

CHANGELOG (Unreleased): the new FileEditor operations, space reuse, and
the two reader fixes (implicit index grid, object-header continuation
chains). known-issues: the editor's remaining refusals (skipped heap
blocks, heaps with filters or child indirect blocks, freeing a heap
block, implicit-index insertions, ...) and the append-waste sizes before
and after reuse (measure_append_waste, tank 2026-09-26; file sizes are
deterministic). range-reads design: status note on the reader changes.
README and CLAUDE.md: what the editor covers and how to test it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

CHANGELOG.md

@@ -2,6 +2,72 @@
 
 ## Unreleased
 
+### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
+- **`FileEditor` adds, moves and resizes chunks of datasets with two or
+  more unlimited dimensions** (version-2 B-tree chunk index, record types
+  10/11), as libhdf5's `H5B2` code does: `H5B2_update`'s insert-or-modify,
+  the preemptive split/redistribute loop, `split1`/`split_root` (depth
+  growth), `redistribute2/3`, and removal with `merge2/3`, root collapse
+  and the internal-record swap; node pointer widths and cumulative record
+  counts per depth; a missing index is created from the layout message's
+  parameters. After the same growth libhdf5's and the editor's trees are
+  node for node the same (tested through a depth increase).
+- **`FileEditor::resize` shrinks** along any dimension (h5py's
+  `Dataset.resize` to a smaller shape), as `H5D__chunk_prune_by_extent`
+  does, visiting the same chunks in the same order: chunks wholly outside
+  the new extent leave the index (version-1 B-tree removal with libhdf5's
+  sibling key and link fix-ups and empty-root case, version-2 B-tree
+  removal, Fixed/Extensible Array elements reset; an implicit index keeps
+  its chunks, as in libhdf5) and their space is freed; the part of a
+  partial edge chunk outside the extent is overwritten with the fill value,
+  so it reads as fill after a later growth. Growth under early allocation
+  now allocates and fills the new chunks (`H5D__chunk_allocate`), which an
+  implicit index needs. Shrinking was `Error::Unsupported`.
+- **`FileEditor::set_attr` handles dense attribute storage and creation
+  order**: objects that track (and index) attribute creation order; the
+  move to dense storage when an object reaches its compact limit (or an
+  attribute is too large for a header message), as `H5O__attr_create`
+  does it (new fractal heap, name index, creation-order index when
+  indexed, compact attributes moved over in header order); objects
+  already in dense storage (h5py- or clawhdf5-written): insertion,
+  same-size rewrites in place, other replacements by removal and
+  insertion. The heap is changed as `H5HF` changes it — best-fit free
+  sections from its free-space manager (kept as libhdf5 keeps `FSHD`/
+  `FSSE`), new direct blocks through the root indirect block (created,
+  doubled), huge objects through the huge-object B-tree (deleted with the
+  last huge object), removed objects' space merged back — with libhdf5's
+  statistics: after the same attribute workload the heap, its free space
+  and both index B-trees equal libhdf5's. Attributes are encoded as libhdf5
+  encodes them for a file h5py opens `r+` (message version 1, 3 for
+  non-ASCII names; simple dataspaces with their maximum dimensions).
+  Still refused: see `docs/known-issues.md`.
+- **Freed space is reused within an editing session.** A `FileEditor`
+  reuses (best fit, zeroed) what its earlier edits freed — moved filtered
+  chunks, pruned chunks, merged B-tree nodes, replaced heap blocks — never
+  what the current edit frees, and writes reused blocks with the new space
+  before any existing byte changes. `FileEditor::reusable_bytes`. The
+  append workload of `measure_append_waste` leaks less (sizes in
+  `docs/known-issues.md`).
+- **Reader: implicit chunk indexes below their maximum shape.** libhdf5
+  places an implicit index's chunks by their position in the *maximum*
+  chunk grid; the reader used the current grid and returned other chunks'
+  values from the second chunk row on (h5py early allocation with a fixed
+  `maxshape` larger than the shape).
+  `chunked_read::generate_implicit_chunks_in_grid` takes the maximum.
+- **Reader: object headers with long continuation chains.** A version-1
+  header whose continuation chunks chain more than 32 deep (a header that
+  gains a chunk per attribute added when full, as libhdf5 and the editor
+  grow it) was refused with `NestingDepthExceeded`; version-2 headers
+  stopped at 256 chunks. Chunks are now followed without recursion, in the
+  same order; a chunk address seen twice (a cycle) or more than 65 536
+  chunks are refused.
+- Tests: `crates/clawhdf5-tools/tests/edit_coverage_interop.rs` (h5py
+  `earliest`/`v110`/`latest` and clawhdf5-written files; structure
+  comparisons with libhdf5 for version-2 B-trees, shrink on every index,
+  and dense attribute heaps); the random-operation property test in
+  `edit_interop.rs` now shrinks, grows two unlimited dimensions and moves
+  attributes to dense storage (`CLAWHDF5_EDIT_SEED` for other seeds).
+
 ### Name lookups through the name index (2026-09-26)
 - **Finding one link or attribute by name reads the name index, not every
   entry.** In a dense group (links in a fractal heap) the v2 B-tree name
@@ -237,7 +303,8 @@
     before any existing byte changes, then the metadata that links it in,
     then a second sync. There is no journal: a crash during the second
     phase can leave the file inconsistent (as with libhdf5 without SWMR).
-    Freed space is not reused (see `docs/known-issues.md`).
+    Freed space is not reused (see `docs/known-issues.md`; since reused
+    within an editing session, above).
   - Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
     `v114` and `latest` files and clawhdf5 files; after every round h5py
     reads the expected values, h5dump and `h5rs check --data` accept the

CLAUDE.md

@@ -151,12 +151,18 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
   `MemorySource` for this bookkeeping is inferred from the caller-supplied
   `source_channel` string (a heuristic, not an authenticated trust boundary).
 - In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`)
-  overwrites values, grows chunked datasets and sets attributes in existing
-  files (h5py- or clawhdf5-written) without rewriting them; anything it
-  cannot do safely is `Error::Unsupported` before any write (limits in
+  overwrites values, grows and shrinks chunked datasets (every chunk index,
+  version-2 B-trees included) and sets attributes (compact and dense
+  storage) in existing files (h5py- or clawhdf5-written) without rewriting
+  them, changing indexes and heaps as libhdf5 does (index shapes and heap
+  bookkeeping are compared with libhdf5's in the tests); space an edit
+  frees is reused by later edits of the same editor. Anything it cannot do
+  safely is `Error::Unsupported` before any write (limits in
   `docs/known-issues.md`). Test changes with
-  `cargo test -p clawhdf5-tools --test edit_interop` (h5py, h5dump,
-  `h5rs check`).
+  `cargo test -p clawhdf5-tools --test edit_interop --test
+  edit_coverage_interop` (h5py, h5dump, `h5rs check`, structure comparisons
+  with libhdf5; libhdf5 sources for the algorithms are at
+  github.com/HDFGroup/hdf5, tag `hdf5_1_14_6`).
 - GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
 - Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
   no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.

README.md

@@ -444,9 +444,13 @@ ed.resize("x", &[1100])?;                   // h5py: ds.resize((1100,))
 let sel = Selection::Hyperslab { start: vec![1000], stride: vec![1], count: vec![100], block: vec![1] };
 ed.write_values("x", &sel, &[0.5f64; 100])?; // ds[1000:1100] = 0.5
 ed.set_attr("x", "units", &AttrValue::String("m/s".into()))?;
+ed.resize("x", &[900])?;                    // shrinking prunes chunks, like h5py
 ```
 
-Each call changes the file in place (no rewrite) and syncs it. What it
+Each call changes the file in place (no rewrite) and syncs it. Any chunk
+index (version-2 B-trees for several unlimited dimensions included) and
+attributes in compact or dense storage are handled as libhdf5 handles
+them; space an edit frees is reused by later edits of the same editor. What it
 cannot change safely is refused before anything is written; see
 [known issues](docs/known-issues.md) for the limits.
 

docs/design/range-reads.md

@@ -3,7 +3,13 @@
 Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
 change. Progress: M1, first part (the `Storage` trait and the metadata
 parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
-group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was
+group B-tree v2 lookups, dense groups and the facade are not converted yet.
+Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched
+converted code without changing the plan: object-header continuation chunks
+are followed without recursion (still one bounded `read_at` per chunk), and
+implicit chunk indexes are addressed over the maximum chunk grid (in
+`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps
+working on the whole file in memory; it is not part of this design. Every count below was
 taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
 next to it. No timing numbers appear here on purpose: the machine was shared
 with other build jobs when this was written.

docs/known-issues.md

@@ -26,40 +26,49 @@ libhdf5 modify them.
 
 ## In-place modification (`FileEditor`) limits
 
-**Status:** open (documented 2026-09-26). `clawhdf5::FileEditor` refuses,
-with `Error::Unsupported` and without writing anything:
-- new, moved or resized chunks in a **version-2 B-tree** chunk index (what
-  libhdf5 uses for two or more unlimited dimensions) — existing unfiltered
-  chunks, and filtered ones that re-encode to the same size and filter
-  mask, are
-  overwritten in place; `resize` works — and new chunks in an **implicit**
-  index (it has all of its chunks from the start);
-- **shrinking** a dataset;
+**Status:** open (documented 2026-09-26, updated the same day when
+version-2 B-tree chunk indexes, shrinking, dense attributes and space
+reuse were added). `clawhdf5::FileEditor` refuses, with
+`Error::Unsupported` and without writing anything:
+- new chunks in an **implicit** index (it has all of its chunks from the
+  start; they are written in place, and allocated/filled on growth under
+  early allocation as libhdf5 does);
 - variable-length and reference data;
 - chunks through a filter this build cannot encode (scale-offset, N-Bit,
   SZIP, or a plugin filter it lacks), even an optional one: libhdf5 skips
   an optional filter only when its own build lacks it, which none does for
   these;
-- attributes of an object in **dense storage**, past its compact limit (8
-  by default) or with tracked **creation order**;
+- attributes in dense storage when the heap cannot take them the way
+  libhdf5 would: an attribute that needs a heap block larger than the next
+  one (libhdf5 skips blocks and records them as free space — in practice an
+  attribute of roughly 1 to 4 KiB going into a young heap), a heap with I/O
+  filters or child indirect blocks (more than about 512 KiB of attributes),
+  free space the heap tracks outside direct blocks, replacing the last
+  attribute left in a heap block by one of another size (libhdf5 frees the
+  block), directly addressed huge objects; and shared attribute messages;
+- version-1 object headers asked for an attribute larger than a header
+  message (they have no dense storage);
 - partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
   raw data files, virtual datasets;
 - files with a metadata cache image, paged or persistent free-space
   management, a driver info block, or version-3 consistency flags set.
 
-**Space is never reused.** There is no free-space manager: the old bytes of
-a filtered chunk that grows and has to move, and of an attribute that is
-replaced by a larger one, are leaked (`h5repack` reclaims them). A chunk
-that is the last thing in the file grows in place instead, which covers the
-usual append. Measured 2026-09-26 on tank with
-`cargo test --release -p clawhdf5-tools --test edit_interop -- --ignored
---nocapture measure_append_waste` (file sizes are deterministic): 1000
-appends of 100 `f8` values to a 1-D dataset with 1024-element chunks give
-810 504 bytes unfiltered, as libhdf5's file, and 307 210 bytes with gzip
-(libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values with
-4096-element gzip chunks give 119 684 bytes against libhdf5's 50 292
-(`h5repack`: 49 930), because the chunk being appended to is followed by
-new index blocks and moves each time it grows.
+**Space is reused only within one editor.** Space an edit frees (a filtered
+chunk that moves, chunks a shrink removes, B-tree nodes merged away, a
+heap's replaced blocks) is reused by later edits of the same `FileEditor`;
+what is left when it is dropped is leaked, as libhdf5 leaks it without a
+persistent free-space manager (`h5repack` reclaims it). A chunk that is the
+last thing in the file grows in place, which covers the usual append.
+Measured 2026-09-26 on tank with `cargo test -p clawhdf5-tools --test
+edit_interop -- --ignored --nocapture measure_append_waste` (one editor for
+the whole workload; file sizes are deterministic): 1000 appends of 100 `f8`
+values to a 1-D dataset with 1024-element chunks give 810 504 bytes
+unfiltered, as libhdf5's file, and 306 780 bytes with gzip (307 210 before
+reuse; libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values
+with 4096-element gzip chunks give 79 829 bytes (119 684 before reuse)
+against libhdf5's 50 292 (`h5repack`: 49 930): the chunk being appended to
+is followed by new index blocks and moves each time it grows, and the
+space it leaves is too small for its next, larger version.
 
 **No journal.** A crash while an edit patches existing structures can leave
 the file inconsistent; see the `FileEditor` documentation.