Merge branch 'feat/p3-in-place-modify' into feat/p3-range-zfp-edit

# Conflicts:
#	CHANGELOG.md
#	crates/clawhdf5-py/src/lib.rs
#	crates/clawhdf5/src/error.rs
This commit is contained in:
osobh
2026-09-26 14:52:55 -05:00
21 changed files with 5919 additions and 19 deletions
+91
View File
@@ -173,6 +173,66 @@
- Conformance on tank (2026-09-26, `conformance/run.sh --no-fetch`): 600 of
697 files ok (599 before); `h5ex_d_zfp.h5` now reads.
### In-place modification (2026-09-26)
- **`clawhdf5::FileEditor` modifies an existing file where it lies.**
`FileBuilder` builds whole files in memory; the editor opens a file
written by libhdf5 (any `libver`, including HDF5 2.0's own format) or by
clawhdf5 and changes only what an edit touches, recomputing the checksum
of every structure it changes. It takes an exclusive `flock` on the file
(the lock libhdf5 takes), so a second editor gets `Error::Locked`.
- `write_selection` / `write_all` / `write_values`: overwrite values of a
compact, contiguous (also never-written, late-allocated) or chunked
dataset, in its own datatype, under any selection. Chunks are decoded,
updated and re-encoded through the dataset's filters; a chunk that no
longer fits moves to the end of the file. New chunks are added to
version-1 B-tree (every chunked dataset of h5py's default `libver`),
Extensible Array, Fixed Array and single-chunk indexes — creating the
index, its data blocks, super blocks and pages, and splitting B-tree
nodes, as libhdf5 does: after the same sequence of writes the B-tree has
the same number of nodes per level and the Extensible Array header the
same block statistics as libhdf5's (tested). Filters run as libhdf5's
`H5Z_pipeline` runs them (new
`clawhdf5_format::filters::compress_chunk_masked`): an optional filter
that fails — LZF or Blosc output no smaller than the chunk — is skipped
and its filter-mask bit set, so the chunk is stored exactly as h5py
stores it; a mandatory filter that fails fails the edit. (Storing such
a chunk LZF-encoded at the raw size with a clear mask let a later
libhdf5 rewrite of it keep the stale mask, and h5py could no longer
read the dataset.)
- `resize`: grow a chunked dataset up to its maximum dimensions (h5py's
`Dataset.resize`).
- `set_attr`: add or replace an attribute in an object header, in free
space or in a new continuation chunk at the end of the file. A
version-2 header (h5py `libver='v110'` and later) without an Attribute
Info message gets one, as libhdf5's `H5O__attr_create` adds it: libhdf5
counts such a header's attributes through that message, and without it
h5py reported `len(obj.attrs) == 0` while listing them.
- Each edit is planned in memory and refused as a whole
(`Error::Unsupported`, file untouched) when any part is not supported:
new chunks in a version-2 B-tree index (two or more unlimited
dimensions) or an implicit index, shrinking, variable-length and
reference data, chunks through a filter this build cannot encode
(scale-offset, N-Bit, SZIP), attributes in dense storage, past an
object's compact limit or with tracked creation order, files with a
metadata cache
image, paged or persistent free space, or marked open by another
writer. New error variants `Error::Unsupported`,
`Error::InvalidArgument`, `Error::Locked`, and `clawhdf5::Error` is now
`#[non_exhaustive]` — a breaking change for code that matches it
exhaustively (the Python bindings map the new variants to
`NotImplementedError`, `ValueError` and `OSError`).
- Durability: the new space (chunks, index blocks) is written and synced
before any existing byte changes, then the metadata that links it in,
then a second sync. There is no journal: a crash during the second
phase can leave the file inconsistent (as with libhdf5 without SWMR).
Freed space is not reused (see `docs/known-issues.md`).
- Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
`v114` and `latest` files and clawhdf5 files; after every round h5py
reads the expected values, h5dump and `h5rs check --data` accept the
file, and h5py `r+` modifies it further; random operations against a
model) and `crates/clawhdf5/tests/edit_tests.rs`.
- `clawhdf5_format::type_builders::build_attr_message` is public.
### Chunked full reads (2026-09-26)
- **Chunks are decoded straight into the output, into reused buffers.** A
full read of a chunked dataset faulted in about three times its size in
@@ -1173,6 +1233,37 @@ and fails their objects (see below).
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
### Correctness
- **`FileBuilder` stored LZF and Blosc chunks with filter mask 0 even when
the filter had not shrunk them** (fixed 2026-09-26). Latent in the
unreleased LZF/Blosc writer only (added 2026-09-26, "Plugin filters"):
no tagged release writes LZF or Blosc, so v2.7.0 and earlier are
unaffected. libhdf5 treats LZF and Blosc output no smaller than the chunk
as a filter failure and, both being optional filters, stores such a chunk
unfiltered with the filter's mask bit set. clawhdf5 stored the filter's
output with a clear mask. For an LZF chunk whose stream was exactly the
chunk's size (h5py stores `[182, 0, 0, 0, 0]` in a 5-byte chunk raw),
the first libhdf5 rewrite of that chunk stored the new data raw at the
same size and, the size being unchanged, kept the stale mask 0: h5py
then failed to read the dataset ("filter returned failure during read").
The whole-file writer now runs chunks through the pipeline as libhdf5
does (`clawhdf5_format::filters::compress_chunk_masked`, as `FileEditor`
already did) and records each chunk's real mask in every chunk index it
builds (single chunk, Fixed Array, Extensible Array, version-2 B-tree;
it builds no version-1 B-tree or implicit index), in the sequential and
`parallel` paths and `create_datasets_parallel`. Files whose chunks all
compress are byte-identical to before. `PrecompressedChunks::chunks` is
now `(raw size, stored bytes, filter mask)` (**breaking** for code that
reads it). Files written before the fix read correctly; rewrite them
(with this build or `h5repack`) before modifying them with libhdf5.
Tests: `plugin_filters_interop`
`skipped_optional_filters_are_masked_as_libhdf5_masks_them` (LZF,
shuffle+LZF+fletcher32 and Blosc, random, compressible and alternating
chunks, every index: masks equal an h5py-written twin's; after h5py r+
rewrites and extends the datasets, h5py, h5dump and our reader read every
value — before the fix 20 of 24 datasets had other masks than h5py's,
and h5py could not read the rewritten `[x, 0, 0, 0, 0]` datasets) and
`files_whose_chunks_all_compress_are_unchanged`; `chunked_write`
`skipped_lzf_chunks_are_masked_in_every_index`.
- **Scale-offset data read wrong values in every release that decoded it
(v2.2.0 to v2.7.0), silently, on ordinary h5py files** (fixed
2026-09-26). Of 1480 scale-offset datasets h5py writes across every