Merge branch 'feat/p3-editor-coverage' into feat/p3-remote-editor
# Conflicts: # CHANGELOG.md # CLAUDE.md # docs/design/range-reads.md
This commit is contained in:
+113
-1
@@ -191,6 +191,117 @@
|
||||
- Conformance sweep (`conformance/run.sh --no-fetch`): 600 of 697 files
|
||||
ok, `results.json` byte-identical to `8f59b2e`.
|
||||
|
||||
### Correctness: Fletcher-32 (2026-09-26)
|
||||
- **Fletcher-32 checksums disagreed with libhdf5's on about one chunk in
|
||||
32768** (fixed 2026-09-26). **Every release is affected, v2.1.0 through
|
||||
v2.7.0**, both directions: `FileBuilder`/`FileWriter` (`with_fletcher32`)
|
||||
and, before release, `FileEditor` wrote chunks that h5py and libhdf5
|
||||
refuse ("filter returned failure during read"), and every reader
|
||||
rejected valid libhdf5-written chunks with `Fletcher32Mismatch`. Our
|
||||
checksum reduced its sums with `% 65535`; libhdf5's
|
||||
`H5_checksum_fletcher32` uses the ones'-complement fold
|
||||
`(s & 0xffff) + (s >> 16)`, which leaves 0xffff where the modulo leaves
|
||||
0, so the two differ whenever a sum is a non-zero multiple of 65535.
|
||||
`clawhdf5_format::checksum::fletcher32` (new, public) is a port of
|
||||
`H5_checksum_fletcher32` and the only implementation; the filter writes
|
||||
and verifies with it, and, as libhdf5 does, also accepts a stored
|
||||
checksum with the bytes of each 16-bit half swapped (libhdf5 1.6.2 and
|
||||
earlier) and the `% 65535` form v2.7.0 and earlier wrote, so their files
|
||||
stay readable. Tests: `crates/clawhdf5/tests/fletcher32_interop.rs` compares
|
||||
it with libhdf5's own function (through ctypes) on every 1- and 2-byte
|
||||
input and 40 000 random and fold-heavy ones, and has h5py read
|
||||
fold-case chunks written by `FileBuilder` and `FileEditor` and us read
|
||||
h5py's. Files written by earlier releases read with a fixed build; to
|
||||
make one readable by libhdf5, rewrite its Fletcher-32 datasets with a
|
||||
fixed build (see `docs/known-issues.md`). `clawhdf5_accel::checksum_fletcher32` is a
|
||||
different, textbook Fletcher-32 (sums start at 0xffff) and is not used
|
||||
for HDF5.
|
||||
|
||||
### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26)
|
||||
- **`FileEditor` adds, moves and resizes chunks of datasets with two or
|
||||
more unlimited dimensions** (version-2 B-tree chunk index, record types
|
||||
10/11), as libhdf5's `H5B2` code does: `H5B2_update`'s insert-or-modify,
|
||||
the preemptive split/redistribute loop, `split1`/`split_root` (depth
|
||||
growth), `redistribute2/3`, and removal with `merge2/3`, root collapse
|
||||
and the internal-record swap; node pointer widths and cumulative record
|
||||
counts per depth; a missing index is created from the layout message's
|
||||
parameters. After the same growth libhdf5's and the editor's trees are
|
||||
node for node the same (tested through a depth increase).
|
||||
- **`FileEditor::resize` shrinks** along any dimension (h5py's
|
||||
`Dataset.resize` to a smaller shape), as `H5D__chunk_prune_by_extent`
|
||||
does, visiting the same chunks in the same order: chunks wholly outside
|
||||
the new extent leave the index (version-1 B-tree removal with libhdf5's
|
||||
sibling key and link fix-ups and empty-root case, version-2 B-tree
|
||||
removal, Fixed/Extensible Array elements reset; an implicit index keeps
|
||||
its chunks, as in libhdf5) and their space is freed; the part of a
|
||||
partial edge chunk outside the extent is overwritten with the fill value,
|
||||
so it reads as fill after a later growth. Growth under early allocation
|
||||
now allocates and fills the new chunks (`H5D__chunk_allocate`), which an
|
||||
implicit index needs. Shrinking was `Error::Unsupported`. Only the
|
||||
chunks that exist are visited (placed in libhdf5's order), so shrinking
|
||||
a sparse dataset costs memory and time in its chunks, not in the
|
||||
coordinates cut off (a 2 x 10^12-coordinate shrink takes 0.6 s).
|
||||
- **`FileEditor::set_attr` handles dense attribute storage and creation
|
||||
order**: objects that track (and index) attribute creation order; the
|
||||
move to dense storage when an object reaches its compact limit (or an
|
||||
attribute is too large for a header message), as `H5O__attr_create`
|
||||
does it (new fractal heap, name index, creation-order index when
|
||||
indexed, compact attributes moved over in header order); objects
|
||||
already in dense storage (h5py- or clawhdf5-written): insertion,
|
||||
same-size rewrites in place, other replacements by removal and
|
||||
insertion. The heap is changed as `H5HF` changes it — best-fit free
|
||||
sections from its free-space manager (kept as libhdf5 keeps `FSHD`/
|
||||
`FSSE`), new direct blocks through the root indirect block (created,
|
||||
doubled), blocks too small for an attribute skipped as libhdf5 skips
|
||||
them (`H5HF__hdr_skip_blocks`: an indirect free section with its row
|
||||
sections, serialized as libhdf5 serializes them, merged with the range
|
||||
skipped just before it, and later attributes given skipped blocks from
|
||||
either end or the middle of a range, which splits it), huge objects
|
||||
through the huge-object B-tree (deleted with the last huge object),
|
||||
removed objects' space merged back — with libhdf5's statistics: after
|
||||
the same attribute workload the heap, its free space and both index
|
||||
B-trees equal libhdf5's (`dense_skipped_blocks_match_libhdf5` covers
|
||||
every way of skipping, with libhdf5 doing one edit per session as the
|
||||
editor does). In a random attribute workload (1-4 KiB attributes among
|
||||
small ones) 24% of `set_attr` calls were refused before skipping was
|
||||
implemented; 2.2% are now, all replacements of the last attribute in a
|
||||
heap block. Attributes are encoded as libhdf5
|
||||
encodes them for a file h5py opens `r+` (message version 1, 3 for
|
||||
non-ASCII names; simple dataspaces with their maximum dimensions).
|
||||
Still refused: see `docs/known-issues.md`.
|
||||
- **Freed space is reused within an editing session.** A `FileEditor`
|
||||
reuses (best fit, zeroed) what its earlier edits freed — moved filtered
|
||||
chunks, pruned chunks, merged B-tree nodes, replaced heap blocks — never
|
||||
what the current edit frees, and writes reused blocks with the new space
|
||||
before any existing byte changes. `FileEditor::reusable_bytes`. The
|
||||
append workload of `measure_append_waste` leaks less (sizes in
|
||||
`docs/known-issues.md`).
|
||||
- **Reader: implicit chunk indexes below their maximum shape.** libhdf5
|
||||
places an implicit index's chunks by their position in the *maximum*
|
||||
chunk grid; the reader used the current grid and returned other chunks'
|
||||
values from the second chunk row on (h5py early allocation with a fixed
|
||||
`maxshape` larger than the shape).
|
||||
`chunked_read::generate_implicit_chunks_in_grid` takes the maximum.
|
||||
- **Reader: object headers with long continuation chains.** A version-1
|
||||
header whose continuation chunks chain more than 32 deep (a header that
|
||||
gains a chunk per attribute added when full, as libhdf5 and the editor
|
||||
grow it) was refused with `NestingDepthExceeded`; version-2 headers
|
||||
stopped at 256 chunks. Chunks are now read one at a time from a queue,
|
||||
in the order their continuation messages are found (libhdf5's
|
||||
`H5O_protect` order, which the editor already used; a version-1
|
||||
chunk's messages used to be inserted at its continuation message), each
|
||||
buffer released before the next is read; a chunk address seen twice (a
|
||||
cycle), chunks adding up to more than the file (a crafted chain of
|
||||
chunks nested in each other made storage with owned buffers read and
|
||||
hold the square of the file's size), or more than 65 536 chunks are
|
||||
refused, so a header's chunks read at most the file's size.
|
||||
- Tests: `crates/clawhdf5-tools/tests/edit_coverage_interop.rs` (h5py
|
||||
`earliest`/`v110`/`latest` and clawhdf5-written files; structure
|
||||
comparisons with libhdf5 for version-2 B-trees, shrink on every index,
|
||||
and dense attribute heaps); the random-operation property test in
|
||||
`edit_interop.rs` now shrinks, grows two unlimited dimensions and moves
|
||||
attributes to dense storage (`CLAWHDF5_EDIT_SEED` for other seeds).
|
||||
|
||||
### Name lookups through the name index (2026-09-26)
|
||||
- **Finding one link or attribute by name reads the name index, not every
|
||||
entry.** In a dense group (links in a fractal heap) the v2 B-tree name
|
||||
@@ -427,7 +538,8 @@
|
||||
before any existing byte changes, then the metadata that links it in,
|
||||
then a second sync. There is no journal: a crash during the second
|
||||
phase can leave the file inconsistent (as with libhdf5 without SWMR).
|
||||
Freed space is not reused (see `docs/known-issues.md`).
|
||||
Freed space is not reused (see `docs/known-issues.md`; since reused
|
||||
within an editing session, above).
|
||||
- Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
|
||||
`v114` and `latest` files and clawhdf5 files; after every round h5py
|
||||
reads the expected values, h5dump and `h5rs check --data` accept the
|
||||
|
||||
Reference in New Issue
Block a user