Merge branch 'feat/p3-editor-coverage' into feat/p3-remote-editor

# Conflicts:
#	CHANGELOG.md
#	CLAUDE.md
#	docs/design/range-reads.md
This commit is contained in:
osobh
2026-09-26 19:17:46 -05:00
23 changed files with 7218 additions and 533 deletions
+113 -1
View File
@@ -191,6 +191,117 @@
- Conformance sweep (`conformance/run.sh --no-fetch`): 600 of 697 files
ok, `results.json` byte-identical to `8f59b2e`.
### Correctness: Fletcher-32 (2026-09-26)
- **Fletcher-32 checksums disagreed with libhdf5's on about one chunk in
32768** (fixed 2026-09-26). **Every release is affected, v2.1.0 through
v2.7.0**, both directions: `FileBuilder`/`FileWriter` (`with_fletcher32`)
and, before release, `FileEditor` wrote chunks that h5py and libhdf5
refuse ("filter returned failure during read"), and every reader
rejected valid libhdf5-written chunks with `Fletcher32Mismatch`. Our
checksum reduced its sums with `% 65535`; libhdf5's
`H5_checksum_fletcher32` uses the ones'-complement fold
`(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves
0, so the two differ whenever a sum is a non-zero multiple of 65535.
`clawhdf5_format::checksum::fletcher32` (new, public) is a port of
`H5_checksum_fletcher32` and the only implementation; the filter writes
and verifies with it, and, as libhdf5 does, also accepts a stored
checksum with the bytes of each 16-bit half swapped (libhdf5 1.6.2 and
earlier) and the `% 65535` form v2.7.0 and earlier wrote, so their files
stay readable. Tests: `crates/clawhdf5/tests/fletcher32_interop.rs` compares
it with libhdf5's own function (through ctypes) on every 1- and 2-byte
input and 40 000 random and fold-heavy ones, and has h5py read
fold-case chunks written by `FileBuilder` and `FileEditor` and us read
h5py's. Files written by earlier releases read with a fixed build; to
make one readable by libhdf5, rewrite its Fletcher-32 datasets with a
fixed build (see `docs/known-issues.md`). `clawhdf5_accel::checksum_fletcher32` is a
different, textbook Fletcher-32 (sums start at 0xffff) and is not used
for HDF5.
### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
- **`FileEditor` adds, moves and resizes chunks of datasets with two or
more unlimited dimensions** (version-2 B-tree chunk index, record types
10/11), as libhdf5's `H5B2` code does: `H5B2_update`'s insert-or-modify,
the preemptive split/redistribute loop, `split1`/`split_root` (depth
growth), `redistribute2/3`, and removal with `merge2/3`, root collapse
and the internal-record swap; node pointer widths and cumulative record
counts per depth; a missing index is created from the layout message's
parameters. After the same growth libhdf5's and the editor's trees are
node for node the same (tested through a depth increase).
- **`FileEditor::resize` shrinks** along any dimension (h5py's
`Dataset.resize` to a smaller shape), as `H5D__chunk_prune_by_extent`
does, visiting the same chunks in the same order: chunks wholly outside
the new extent leave the index (version-1 B-tree removal with libhdf5's
sibling key and link fix-ups and empty-root case, version-2 B-tree
removal, Fixed/Extensible Array elements reset; an implicit index keeps
its chunks, as in libhdf5) and their space is freed; the part of a
partial edge chunk outside the extent is overwritten with the fill value,
so it reads as fill after a later growth. Growth under early allocation
now allocates and fills the new chunks (`H5D__chunk_allocate`), which an
implicit index needs. Shrinking was `Error::Unsupported`. Only the
chunks that exist are visited (placed in libhdf5's order), so shrinking
a sparse dataset costs memory and time in its chunks, not in the
coordinates cut off (a 2 x 10^12-coordinate shrink takes 0.6 s).
- **`FileEditor::set_attr` handles dense attribute storage and creation
order**: objects that track (and index) attribute creation order; the
move to dense storage when an object reaches its compact limit (or an
attribute is too large for a header message), as `H5O__attr_create`
does it (new fractal heap, name index, creation-order index when
indexed, compact attributes moved over in header order); objects
already in dense storage (h5py- or clawhdf5-written): insertion,
same-size rewrites in place, other replacements by removal and
insertion. The heap is changed as `H5HF` changes it — best-fit free
sections from its free-space manager (kept as libhdf5 keeps `FSHD`/
`FSSE`), new direct blocks through the root indirect block (created,
doubled), blocks too small for an attribute skipped as libhdf5 skips
them (`H5HF__hdr_skip_blocks`: an indirect free section with its row
sections, serialized as libhdf5 serializes them, merged with the range
skipped just before it, and later attributes given skipped blocks from
either end or the middle of a range, which splits it), huge objects
through the huge-object B-tree (deleted with the last huge object),
removed objects' space merged back — with libhdf5's statistics: after
the same attribute workload the heap, its free space and both index
B-trees equal libhdf5's (`dense_skipped_blocks_match_libhdf5` covers
every way of skipping, with libhdf5 doing one edit per session as the
editor does). In a random attribute workload (1-4 KiB attributes among
small ones) 24% of `set_attr` calls were refused before skipping was
implemented; 2.2% are now, all replacements of the last attribute in a
heap block. Attributes are encoded as libhdf5
encodes them for a file h5py opens `r+` (message version 1, 3 for
non-ASCII names; simple dataspaces with their maximum dimensions).
Still refused: see `docs/known-issues.md`.
- **Freed space is reused within an editing session.** A `FileEditor`
reuses (best fit, zeroed) what its earlier edits freed — moved filtered
chunks, pruned chunks, merged B-tree nodes, replaced heap blocks — never
what the current edit frees, and writes reused blocks with the new space
before any existing byte changes. `FileEditor::reusable_bytes`. The
append workload of `measure_append_waste` leaks less (sizes in
`docs/known-issues.md`).
- **Reader: implicit chunk indexes below their maximum shape.** libhdf5
places an implicit index's chunks by their position in the *maximum*
chunk grid; the reader used the current grid and returned other chunks'
values from the second chunk row on (h5py early allocation with a fixed
`maxshape` larger than the shape).
`chunked_read::generate_implicit_chunks_in_grid` takes the maximum.
- **Reader: object headers with long continuation chains.** A version-1
header whose continuation chunks chain more than 32 deep (a header that
gains a chunk per attribute added when full, as libhdf5 and the editor
grow it) was refused with `NestingDepthExceeded`; version-2 headers
stopped at 256 chunks. Chunks are now read one at a time from a queue,
in the order their continuation messages are found (libhdf5's
`H5O_protect` order, which the editor already used; a version-1
chunk's messages used to be inserted at its continuation message), each
buffer released before the next is read; a chunk address seen twice (a
cycle), chunks adding up to more than the file (a crafted chain of
chunks nested in each other made storage with owned buffers read and
hold the square of the file's size), or more than 65 536 chunks are
refused, so a header's chunks read at most the file's size.
- Tests: `crates/clawhdf5-tools/tests/edit_coverage_interop.rs` (h5py
`earliest`/`v110`/`latest` and clawhdf5-written files; structure
comparisons with libhdf5 for version-2 B-trees, shrink on every index,
and dense attribute heaps); the random-operation property test in
`edit_interop.rs` now shrinks, grows two unlimited dimensions and moves
attributes to dense storage (`CLAWHDF5_EDIT_SEED` for other seeds).
### Name lookups through the name index (2026-09-26)
- **Finding one link or attribute by name reads the name index, not every
entry.** In a dense group (links in a fractal heap) the v2 B-tree name
@@ -427,7 +538,8 @@
before any existing byte changes, then the metadata that links it in,
then a second sync. There is no journal: a crash during the second
phase can leave the file inconsistent (as with libhdf5 without SWMR).
Freed space is not reused (see `docs/known-issues.md`).
Freed space is not reused (see `docs/known-issues.md`; since reused
within an editing session, above).
- Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
`v114` and `latest` files and clawhdf5 files; after every round h5py
reads the expected values, h5dump and `h5rs check --data` accept the