Merge branch 'feat/p3-editor-coverage' into feat/p3-remote-editor
# Conflicts: # CHANGELOG.md # CLAUDE.md # docs/design/range-reads.md
This commit is contained in:
+76
-24
@@ -7,6 +7,41 @@ deleting it.
|
||||
|
||||
---
|
||||
|
||||
## Fletcher-32 checksums disagreed with libhdf5 on about 1 chunk in 32768
|
||||
|
||||
**Status:** fixed 2026-09-26, after v2.7.0. **Every release (v2.1.0 to
|
||||
v2.7.0) is affected**, in both directions.
|
||||
|
||||
Our Fletcher-32 reduced its two running sums with `% 65535`; libhdf5's
|
||||
`H5_checksum_fletcher32` (H5checksum.c) folds them with
|
||||
`(s & 0xffff) + (s >> 16)`. Both are arithmetic mod 65535, but where a sum
|
||||
is a non-zero multiple of 65535 the fold leaves 0xffff and the modulo 0, so
|
||||
the checksums differ — for random data about one chunk in 32768 (each of
|
||||
the two sums hits it with probability about 1/65535). Found by the review
|
||||
of the editor work: a random-edit fuzzer with gzip + Fletcher-32 hit it on
|
||||
13 of about 100 seeds.
|
||||
|
||||
- Chunks we wrote (`FileBuilder`/`FileWriter` `with_fletcher32`, and the
|
||||
unreleased `FileEditor`) with such a sum are refused by h5py and libhdf5:
|
||||
"filter returned failure during read". h5py writing `[1, 0xfffe]` as
|
||||
big-endian `u2` stores checksum `0x0001ffff`; we computed `0x00010000`.
|
||||
- Chunks libhdf5 wrote with such a sum were refused by every reader here
|
||||
with `Fletcher32Mismatch`; the data itself was never wrong.
|
||||
|
||||
**Fix:** `clawhdf5_format::checksum::fletcher32`, a port of
|
||||
`H5_checksum_fletcher32`, used by the filter for writing and verifying. It
|
||||
also accepts a checksum whose 16-bit halves are byte-swapped, as libhdf5
|
||||
does for files from 1.6.2 and earlier, and the `% 65535` form clawhdf5
|
||||
v2.7.0 and earlier wrote (the two differ only in a half that is 0xffff).
|
||||
**Test:**
|
||||
`crates/clawhdf5/tests/fletcher32_interop.rs` (libhdf5's own function
|
||||
through ctypes on every 1- and 2-byte input plus 40 000 random and
|
||||
fold-heavy inputs; h5py reads fold-case chunks from `FileBuilder` and
|
||||
`FileEditor`; we read h5py's). **Existing data:** a Fletcher-32 dataset
|
||||
written by v2.7.0 or earlier may hold chunks libhdf5 cannot read; a fixed
|
||||
build reads them. Rewrite such datasets with a fixed build (read, then
|
||||
write them again) before handing the file to libhdf5 or h5py.
|
||||
|
||||
## LZF/Blosc chunks written with a stale filter mask
|
||||
|
||||
**Status:** fixed 2026-09-26, before any release (the LZF and Blosc writers
|
||||
@@ -26,40 +61,57 @@ libhdf5 modify them.
|
||||
|
||||
## In-place modification (`FileEditor`) limits
|
||||
|
||||
**Status:** open (documented 2026-09-26). `clawhdf5::FileEditor` refuses,
|
||||
with `Error::Unsupported` and without writing anything:
|
||||
- new, moved or resized chunks in a **version-2 B-tree** chunk index (what
|
||||
libhdf5 uses for two or more unlimited dimensions) — existing unfiltered
|
||||
chunks, and filtered ones that re-encode to the same size and filter
|
||||
mask, are
|
||||
overwritten in place; `resize` works — and new chunks in an **implicit**
|
||||
index (it has all of its chunks from the start);
|
||||
- **shrinking** a dataset;
|
||||
**Status:** open (documented 2026-09-26, updated the same day when
|
||||
version-2 B-tree chunk indexes, shrinking, dense attributes and space
|
||||
reuse were added). `clawhdf5::FileEditor` refuses, with
|
||||
`Error::Unsupported` and without writing anything:
|
||||
- new chunks in an **implicit** index (it has all of its chunks from the
|
||||
start; they are written in place, and allocated/filled on growth under
|
||||
early allocation as libhdf5 does);
|
||||
- variable-length and reference data;
|
||||
- chunks through a filter this build cannot encode (scale-offset, N-Bit,
|
||||
SZIP, or a plugin filter it lacks), even an optional one: libhdf5 skips
|
||||
an optional filter only when its own build lacks it, which none does for
|
||||
these;
|
||||
- attributes of an object in **dense storage**, past its compact limit (8
|
||||
by default) or with tracked **creation order**;
|
||||
- attributes in dense storage when the heap cannot take them the way
|
||||
libhdf5 would: replacing the last attribute left in a heap block by one
|
||||
of another size (libhdf5 frees the block), a heap with I/O filters or
|
||||
child indirect blocks (more than about 512 KiB of attributes), free
|
||||
space in child indirect blocks, directly addressed huge objects; and
|
||||
shared attribute messages. Measured 2026-09-26 on tank with the review's
|
||||
random-edit harness (120 runs of 150 random edits, `earliest`/`v110`/
|
||||
`latest`, about 5600 `set_attr` calls of 8 bytes to 6 KiB): 2.2% of
|
||||
`set_attr` calls are refused, every one the last-attribute-in-a-block
|
||||
replacement; before blocks could be skipped (an attribute needing a heap
|
||||
block larger than the next one — any attribute of about 1 KiB or more
|
||||
once a heap has started, or at the move to dense storage), 24% were,
|
||||
since an object whose move to dense storage was refused kept refusing
|
||||
every new attribute;
|
||||
- version-1 object headers asked for an attribute larger than a header
|
||||
message (they have no dense storage);
|
||||
- partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
|
||||
raw data files, virtual datasets;
|
||||
- files with a metadata cache image, paged or persistent free-space
|
||||
management, a driver info block, or version-3 consistency flags set.
|
||||
|
||||
**Space is never reused.** There is no free-space manager: the old bytes of
|
||||
a filtered chunk that grows and has to move, and of an attribute that is
|
||||
replaced by a larger one, are leaked (`h5repack` reclaims them). A chunk
|
||||
that is the last thing in the file grows in place instead, which covers the
|
||||
usual append. Measured 2026-09-26 on tank with
|
||||
`cargo test --release -p clawhdf5-tools --test edit_interop -- --ignored
|
||||
--nocapture measure_append_waste` (file sizes are deterministic): 1000
|
||||
appends of 100 `f8` values to a 1-D dataset with 1024-element chunks give
|
||||
810 504 bytes unfiltered, as libhdf5's file, and 307 210 bytes with gzip
|
||||
(libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values with
|
||||
4096-element gzip chunks give 119 684 bytes against libhdf5's 50 292
|
||||
(`h5repack`: 49 930), because the chunk being appended to is followed by
|
||||
new index blocks and moves each time it grows.
|
||||
**Space is reused only within one editor.** Space an edit frees (a filtered
|
||||
chunk that moves, chunks a shrink removes, B-tree nodes merged away, a
|
||||
heap's replaced blocks) is reused by later edits of the same `FileEditor`;
|
||||
what is left when it is dropped is leaked, as libhdf5 leaks it without a
|
||||
persistent free-space manager (`h5repack` reclaims it). A chunk that is the
|
||||
last thing in the file grows in place, which covers the usual append.
|
||||
Measured 2026-09-26 on tank with `cargo test -p clawhdf5-tools --test
|
||||
edit_interop -- --ignored --nocapture measure_append_waste` (one editor for
|
||||
the whole workload; file sizes are deterministic): 1000 appends of 100 `f8`
|
||||
values to a 1-D dataset with 1024-element chunks give 810 504 bytes
|
||||
unfiltered, as libhdf5's file (`h5repack` of either: 810 360), and 306 780
|
||||
bytes with gzip (307 210 before reuse; libhdf5's file: 306 058; `h5repack`
|
||||
of the editor's file: 306 104, of libhdf5's: 305 954); 2000 appends of 10
|
||||
values with 4096-element gzip chunks give 79 829 bytes (119 684 before
|
||||
reuse) against libhdf5's 50 292 (`h5repack` of the editor's file: 49 930,
|
||||
of libhdf5's: 50 188): the chunk being appended to
|
||||
is followed by new index blocks and moves each time it grows, and the
|
||||
space it leaves is too small for its next, larger version.
|
||||
|
||||
**No journal.** A crash while an edit patches existing structures can leave
|
||||
the file inconsistent; see the `FileEditor` documentation.
|
||||
|
||||
Reference in New Issue
Block a user