edit: record the maximum before resizing a chunked dataset that has none
FileEditor::resize (on main since PR #18, a4c2ace) scrambled the values of
a chunked dataset whose dataspace records no maximum dimensions when it
shrank it: clawhdf5's writer stores such a dataspace for every chunked
dataset created without a maxshape, the maximum is then the current
dimensions, and the Fixed Array index linearises chunks by the maximum, so
patching only the current dimensions moved every chunk after the first
row. h5py, h5dump and our reader all read the wrong values; the dataset
could not grow back either.
libhdf5 never writes such a dataspace (H5S_set_extent_simple records the
maximum, equal to the dimensions when none is given); reading one,
H5S_extent_get_dims reports the current dimensions as the maximum and
H5S_set_extent checks against none, so its own H5Dset_extent scrambles
such a file the same way. The editor now records the maximum libhdf5 would
have written (the dimensions the index was built with) before changing the
current ones, moving the grown dataspace message in the header when it
must. The writer records the maximum of every chunked dataset too, so
h5py can resize what clawhdf5 writes (the pinned file hashes of three
no-maxshape cases in plugin_filters_interop change by 8 bytes a dimension).
Tests: edit_resize_interop.rs (a 2.7.0-written fixture, new FileBuilder
files and h5py files through shrinks, zero extents and growth, against a
model with our reader and h5py; h5py resizing a FileBuilder file), and in
test_edit.py resizes checked against a numpy model, independently of h5py.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,38 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Correctness: resizing chunked datasets with no recorded maximum (2026-09-27)
|
||||
- **`FileEditor::resize` scrambled the values of a chunked dataset whose
|
||||
dataspace records no maximum dimensions when it shrank it** (fixed
|
||||
2026-09-27). The editor shipped on main in PR #18 (a4c2ace) and reached
|
||||
Python as `Dataset.resize` in `'r+'` files; no release has it. clawhdf5's
|
||||
writer stores such a dataspace for every chunked dataset created without
|
||||
a `maxshape`, with a Fixed Array (or Single Chunk) chunk index. The
|
||||
editor patched only the current dimensions, and with no maximum
|
||||
recorded the maximum is the current dimensions — which is also what the
|
||||
Fixed Array linearises chunks by — so a shrink moved every chunk after
|
||||
the first row: h5py, h5dump and our reader all read wrong values
|
||||
without complaint (20x20, chunks 6x6, resized to 15x15: row 6 read
|
||||
`0 0 0 0 0 0 120 ...`). After a shrink the dataset could not grow back
|
||||
either (`3 exceeds the maximum 0`). libhdf5 itself never writes such a
|
||||
dataspace (`H5S_set_extent_simple` records the maximum, equal to the
|
||||
dimensions when none is given); reading one, `H5S_extent_get_dims`
|
||||
reports the current dimensions as the maximum and `H5S_set_extent`
|
||||
checks against no maximum at all, so libhdf5's own `H5Dset_extent`
|
||||
scrambles such a file the same way (and lets it grow past its Fixed
|
||||
Array). The editor now records the maximum libhdf5 would have written
|
||||
— the dimensions before the first resize, the ones the index was built
|
||||
with — then changes the current ones (the dataspace message grows by one
|
||||
length per dimension and moves in the header when it has to). The
|
||||
dataset then shrinks, grows back to that extent and refuses more, as
|
||||
one libhdf5 wrote would. The writer (`FileBuilder`) now records the
|
||||
maximum of every chunked dataset too, as libhdf5 does, so h5py can
|
||||
resize what it writes (8 more bytes per dimension). Tests:
|
||||
`crates/clawhdf5/tests/edit_resize_interop.rs` (a 2.7.0-written fixture,
|
||||
new `FileBuilder` files and h5py files through shrinks, zero extents
|
||||
and growth, checked against a model with our reader and h5py; h5py
|
||||
resizing a `FileBuilder` file) and `test_edit.py`'s numpy-model checks.
|
||||
|
||||
### Python bindings: in-place editing (2026-09-27)
|
||||
- **`clawhdf5.File(path, 'r+')`** (and `'a'` on an existing file) opens a
|
||||
file for editing through `clawhdf5::FileEditor`, holding its exclusive
|
||||
|
||||
Reference in New Issue
Block a user