edit: record the maximum before resizing a chunked dataset that has none

FileEditor::resize (on main since PR #18, a4c2ace) scrambled the values of
a chunked dataset whose dataspace records no maximum dimensions when it
shrank it: clawhdf5's writer stores such a dataspace for every chunked
dataset created without a maxshape, the maximum is then the current
dimensions, and the Fixed Array index linearises chunks by the maximum, so
patching only the current dimensions moved every chunk after the first
row. h5py, h5dump and our reader all read the wrong values; the dataset
could not grow back either.

libhdf5 never writes such a dataspace (H5S_set_extent_simple records the
maximum, equal to the dimensions when none is given); reading one,
H5S_extent_get_dims reports the current dimensions as the maximum and
H5S_set_extent checks against none, so its own H5Dset_extent scrambles
such a file the same way. The editor now records the maximum libhdf5 would
have written (the dimensions the index was built with) before changing the
current ones, moving the grown dataspace message in the header when it
must. The writer records the maximum of every chunked dataset too, so
h5py can resize what clawhdf5 writes (the pinned file hashes of three
no-maxshape cases in plugin_filters_interop change by 8 bytes a dimension).

Tests: edit_resize_interop.rs (a 2.7.0-written fixture, new FileBuilder
files and h5py files through shrinks, zero extents and growth, against a
model with our reader and h5py; h5py resizing a FileBuilder file), and in
test_edit.py resizes checked against a numpy model, independently of h5py.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 07:40:01 -05:00
co-authored by Claude Opus 5.5
parent 92fb0830e0
commit 1f7651644b
8 changed files with 482 additions and 7 deletions
+32
View File
@@ -2,6 +2,38 @@
## Unreleased
### Correctness: resizing chunked datasets with no recorded maximum (2026-09-27)
- **`FileEditor::resize` scrambled the values of a chunked dataset whose
dataspace records no maximum dimensions when it shrank it** (fixed
2026-09-27). The editor shipped on main in PR #18 (a4c2ace) and reached
Python as `Dataset.resize` in `'r+'` files; no release has it. clawhdf5's
writer stores such a dataspace for every chunked dataset created without
a `maxshape`, with a Fixed Array (or Single Chunk) chunk index. The
editor patched only the current dimensions, and with no maximum
recorded the maximum is the current dimensions — which is also what the
Fixed Array linearises chunks by — so a shrink moved every chunk after
the first row: h5py, h5dump and our reader all read wrong values
without complaint (20x20, chunks 6x6, resized to 15x15: row 6 read
`0 0 0 0 0 0 120 ...`). After a shrink the dataset could not grow back
either (`3 exceeds the maximum 0`). libhdf5 itself never writes such a
dataspace (`H5S_set_extent_simple` records the maximum, equal to the
dimensions when none is given); reading one, `H5S_extent_get_dims`
reports the current dimensions as the maximum and `H5S_set_extent`
checks against no maximum at all, so libhdf5's own `H5Dset_extent`
scrambles such a file the same way (and lets it grow past its Fixed
Array). The editor now records the maximum libhdf5 would have written
— the dimensions before the first resize, the ones the index was built
with — then changes the current ones (the dataspace message grows by one
length per dimension and moves in the header when it has to). The
dataset then shrinks, grows back to that extent and refuses more, as
one libhdf5 wrote would. The writer (`FileBuilder`) now records the
maximum of every chunked dataset too, as libhdf5 does, so h5py can
resize what it writes (8 more bytes per dimension). Tests:
`crates/clawhdf5/tests/edit_resize_interop.rs` (a 2.7.0-written fixture,
new `FileBuilder` files and h5py files through shrinks, zero extents
and growth, checked against a model with our reader and h5py; h5py
resizing a `FileBuilder` file) and `test_edit.py`'s numpy-model checks.
### Python bindings: in-place editing (2026-09-27)
- **`clawhdf5.File(path, 'r+')`** (and `'a'` on an existing file) opens a
file for editing through `clawhdf5::FileEditor`, holding its exclusive