docs: editor coverage — version-2 B-trees, shrinking, dense attributes, reuse

CHANGELOG (Unreleased): the new FileEditor operations, space reuse, and
the two reader fixes (implicit index grid, object-header continuation
chains). known-issues: the editor's remaining refusals (skipped heap
blocks, heaps with filters or child indirect blocks, freeing a heap
block, implicit-index insertions, ...) and the append-waste sizes before
and after reuse (measure_append_waste, tank 2026-09-26; file sizes are
deterministic). range-reads design: status note on the reader changes.
README and CLAUDE.md: what the editor covers and how to test it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 17:13:31 -05:00
co-authored by Claude Opus 5.5
parent 955fdb660d
commit 0aca0eb724
5 changed files with 124 additions and 32 deletions
+33 -24
View File
@@ -26,40 +26,49 @@ libhdf5 modify them.
## In-place modification (`FileEditor`) limits
**Status:** open (documented 2026-09-26). `clawhdf5::FileEditor` refuses,
with `Error::Unsupported` and without writing anything:
- new, moved or resized chunks in a **version-2 B-tree** chunk index (what
libhdf5 uses for two or more unlimited dimensions) — existing unfiltered
chunks, and filtered ones that re-encode to the same size and filter
mask, are
overwritten in place; `resize` works — and new chunks in an **implicit**
index (it has all of its chunks from the start);
- **shrinking** a dataset;
**Status:** open (documented 2026-09-26, updated the same day when
version-2 B-tree chunk indexes, shrinking, dense attributes and space
reuse were added). `clawhdf5::FileEditor` refuses, with
`Error::Unsupported` and without writing anything:
- new chunks in an **implicit** index (it has all of its chunks from the
start; they are written in place, and allocated/filled on growth under
early allocation as libhdf5 does);
- variable-length and reference data;
- chunks through a filter this build cannot encode (scale-offset, N-Bit,
SZIP, or a plugin filter it lacks), even an optional one: libhdf5 skips
an optional filter only when its own build lacks it, which none does for
these;
- attributes of an object in **dense storage**, past its compact limit (8
by default) or with tracked **creation order**;
- attributes in dense storage when the heap cannot take them the way
libhdf5 would: an attribute that needs a heap block larger than the next
one (libhdf5 skips blocks and records them as free space — in practice an
attribute of roughly 1 to 4 KiB going into a young heap), a heap with I/O
filters or child indirect blocks (more than about 512 KiB of attributes),
free space the heap tracks outside direct blocks, replacing the last
attribute left in a heap block by one of another size (libhdf5 frees the
block), directly addressed huge objects; and shared attribute messages;
- version-1 object headers asked for an attribute larger than a header
message (they have no dense storage);
- partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
raw data files, virtual datasets;
- files with a metadata cache image, paged or persistent free-space
management, a driver info block, or version-3 consistency flags set.
**Space is never reused.** There is no free-space manager: the old bytes of
a filtered chunk that grows and has to move, and of an attribute that is
replaced by a larger one, are leaked (`h5repack` reclaims them). A chunk
that is the last thing in the file grows in place instead, which covers the
usual append. Measured 2026-09-26 on tank with
`cargo test --release -p clawhdf5-tools --test edit_interop -- --ignored
--nocapture measure_append_waste` (file sizes are deterministic): 1000
appends of 100 `f8` values to a 1-D dataset with 1024-element chunks give
810 504 bytes unfiltered, as libhdf5's file, and 307 210 bytes with gzip
(libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values with
4096-element gzip chunks give 119 684 bytes against libhdf5's 50 292
(`h5repack`: 49 930), because the chunk being appended to is followed by
new index blocks and moves each time it grows.
**Space is reused only within one editor.** Space an edit frees (a filtered
chunk that moves, chunks a shrink removes, B-tree nodes merged away, a
heap's replaced blocks) is reused by later edits of the same `FileEditor`;
what is left when it is dropped is leaked, as libhdf5 leaks it without a
persistent free-space manager (`h5repack` reclaims it). A chunk that is the
last thing in the file grows in place, which covers the usual append.
Measured 2026-09-26 on tank with `cargo test -p clawhdf5-tools --test
edit_interop -- --ignored --nocapture measure_append_waste` (one editor for
the whole workload; file sizes are deterministic): 1000 appends of 100 `f8`
values to a 1-D dataset with 1024-element chunks give 810 504 bytes
unfiltered, as libhdf5's file, and 306 780 bytes with gzip (307 210 before
reuse; libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values
with 4096-element gzip chunks give 79 829 bytes (119 684 before reuse)
against libhdf5's 50 292 (`h5repack`: 49 930): the chunk being appended to
is followed by new index blocks and moves each time it grows, and the
space it leaves is too small for its next, larger version.
**No journal.** A crash while an edit patches existing structures can leave
the file inconsistent; see the `FileEditor` documentation.