Merge branch 'feat/p3-in-place-modify' into feat/p3-range-zfp-edit
# Conflicts: # CHANGELOG.md # crates/clawhdf5-py/src/lib.rs # crates/clawhdf5/src/error.rs
This commit is contained in:
@@ -173,6 +173,66 @@
|
||||
- Conformance on tank (2026-09-26, `conformance/run.sh --no-fetch`): 600 of
|
||||
697 files ok (599 before); `h5ex_d_zfp.h5` now reads.
|
||||
|
||||
### In-place modification (2026-09-26)
|
||||
- **`clawhdf5::FileEditor` modifies an existing file where it lies.**
|
||||
`FileBuilder` builds whole files in memory; the editor opens a file
|
||||
written by libhdf5 (any `libver`, including HDF5 2.0's own format) or by
|
||||
clawhdf5 and changes only what an edit touches, recomputing the checksum
|
||||
of every structure it changes. It takes an exclusive `flock` on the file
|
||||
(the lock libhdf5 takes), so a second editor gets `Error::Locked`.
|
||||
- `write_selection` / `write_all` / `write_values`: overwrite values of a
|
||||
compact, contiguous (also never-written, late-allocated) or chunked
|
||||
dataset, in its own datatype, under any selection. Chunks are decoded,
|
||||
updated and re-encoded through the dataset's filters; a chunk that no
|
||||
longer fits moves to the end of the file. New chunks are added to
|
||||
version-1 B-tree (every chunked dataset of h5py's default `libver`),
|
||||
Extensible Array, Fixed Array and single-chunk indexes — creating the
|
||||
index, its data blocks, super blocks and pages, and splitting B-tree
|
||||
nodes, as libhdf5 does: after the same sequence of writes the B-tree has
|
||||
the same number of nodes per level and the Extensible Array header the
|
||||
same block statistics as libhdf5's (tested). Filters run as libhdf5's
|
||||
`H5Z_pipeline` runs them (new
|
||||
`clawhdf5_format::filters::compress_chunk_masked`): an optional filter
|
||||
that fails — LZF or Blosc output no smaller than the chunk — is skipped
|
||||
and its filter-mask bit set, so the chunk is stored exactly as h5py
|
||||
stores it; a mandatory filter that fails fails the edit. (Storing such
|
||||
a chunk LZF-encoded at the raw size with a clear mask let a later
|
||||
libhdf5 rewrite of it keep the stale mask, and h5py could no longer
|
||||
read the dataset.)
|
||||
- `resize`: grow a chunked dataset up to its maximum dimensions (h5py's
|
||||
`Dataset.resize`).
|
||||
- `set_attr`: add or replace an attribute in an object header, in free
|
||||
space or in a new continuation chunk at the end of the file. A
|
||||
version-2 header (h5py `libver='v110'` and later) without an Attribute
|
||||
Info message gets one, as libhdf5's `H5O__attr_create` adds it: libhdf5
|
||||
counts such a header's attributes through that message, and without it
|
||||
h5py reported `len(obj.attrs) == 0` while listing them.
|
||||
- Each edit is planned in memory and refused as a whole
|
||||
(`Error::Unsupported`, file untouched) when any part is not supported:
|
||||
new chunks in a version-2 B-tree index (two or more unlimited
|
||||
dimensions) or an implicit index, shrinking, variable-length and
|
||||
reference data, chunks through a filter this build cannot encode
|
||||
(scale-offset, N-Bit, SZIP), attributes in dense storage, past an
|
||||
object's compact limit or with tracked creation order, files with a
|
||||
metadata cache
|
||||
image, paged or persistent free space, or marked open by another
|
||||
writer. New error variants `Error::Unsupported`,
|
||||
`Error::InvalidArgument`, `Error::Locked`, and `clawhdf5::Error` is now
|
||||
`#[non_exhaustive]` — a breaking change for code that matches it
|
||||
exhaustively (the Python bindings map the new variants to
|
||||
`NotImplementedError`, `ValueError` and `OSError`).
|
||||
- Durability: the new space (chunks, index blocks) is written and synced
|
||||
before any existing byte changes, then the metadata that links it in,
|
||||
then a second sync. There is no journal: a crash during the second
|
||||
phase can leave the file inconsistent (as with libhdf5 without SWMR).
|
||||
Freed space is not reused (see `docs/known-issues.md`).
|
||||
- Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
|
||||
`v114` and `latest` files and clawhdf5 files; after every round h5py
|
||||
reads the expected values, h5dump and `h5rs check --data` accept the
|
||||
file, and h5py `r+` modifies it further; random operations against a
|
||||
model) and `crates/clawhdf5/tests/edit_tests.rs`.
|
||||
- `clawhdf5_format::type_builders::build_attr_message` is public.
|
||||
|
||||
### Chunked full reads (2026-09-26)
|
||||
- **Chunks are decoded straight into the output, into reused buffers.** A
|
||||
full read of a chunked dataset faulted in about three times its size in
|
||||
@@ -1173,6 +1233,37 @@ and fails their objects (see below).
|
||||
- CI keeps zlib-ng building and tested; the arm64 job no longer needs cmake.
|
||||
|
||||
### Correctness
|
||||
- **`FileBuilder` stored LZF and Blosc chunks with filter mask 0 even when
|
||||
the filter had not shrunk them** (fixed 2026-09-26). Latent in the
|
||||
unreleased LZF/Blosc writer only (added 2026-09-26, "Plugin filters"):
|
||||
no tagged release writes LZF or Blosc, so v2.7.0 and earlier are
|
||||
unaffected. libhdf5 treats LZF and Blosc output no smaller than the chunk
|
||||
as a filter failure and, both being optional filters, stores such a chunk
|
||||
unfiltered with the filter's mask bit set. clawhdf5 stored the filter's
|
||||
output with a clear mask. For an LZF chunk whose stream was exactly the
|
||||
chunk's size (h5py stores `[182, 0, 0, 0, 0]` in a 5-byte chunk raw),
|
||||
the first libhdf5 rewrite of that chunk stored the new data raw at the
|
||||
same size and, the size being unchanged, kept the stale mask 0: h5py
|
||||
then failed to read the dataset ("filter returned failure during read").
|
||||
The whole-file writer now runs chunks through the pipeline as libhdf5
|
||||
does (`clawhdf5_format::filters::compress_chunk_masked`, as `FileEditor`
|
||||
already did) and records each chunk's real mask in every chunk index it
|
||||
builds (single chunk, Fixed Array, Extensible Array, version-2 B-tree;
|
||||
it builds no version-1 B-tree or implicit index), in the sequential and
|
||||
`parallel` paths and `create_datasets_parallel`. Files whose chunks all
|
||||
compress are byte-identical to before. `PrecompressedChunks::chunks` is
|
||||
now `(raw size, stored bytes, filter mask)` (**breaking** for code that
|
||||
reads it). Files written before the fix read correctly; rewrite them
|
||||
(with this build or `h5repack`) before modifying them with libhdf5.
|
||||
Tests: `plugin_filters_interop`
|
||||
`skipped_optional_filters_are_masked_as_libhdf5_masks_them` (LZF,
|
||||
shuffle+LZF+fletcher32 and Blosc, random, compressible and alternating
|
||||
chunks, every index: masks equal an h5py-written twin's; after h5py r+
|
||||
rewrites and extends the datasets, h5py, h5dump and our reader read every
|
||||
value — before the fix 20 of 24 datasets had other masks than h5py's,
|
||||
and h5py could not read the rewritten `[x, 0, 0, 0, 0]` datasets) and
|
||||
`files_whose_chunks_all_compress_are_unchanged`; `chunked_write`
|
||||
`skipped_lzf_chunks_are_masked_in_every_index`.
|
||||
- **Scale-offset data read wrong values in every release that decoded it
|
||||
(v2.2.0 to v2.7.0), silently, on ordinary h5py files** (fixed
|
||||
2026-09-26). Of 1480 scale-offset datasets h5py writes across every
|
||||
|
||||
Reference in New Issue
Block a user