Merge branch 'feat/p3-in-place-modify' into feat/p3-range-zfp-edit
# Conflicts: # CHANGELOG.md # crates/clawhdf5-py/src/lib.rs # crates/clawhdf5/src/error.rs
This commit is contained in:
@@ -7,6 +7,63 @@ deleting it.
|
||||
|
||||
---
|
||||
|
||||
## LZF/Blosc chunks written with a stale filter mask
|
||||
|
||||
**Status:** fixed 2026-09-26, before any release (the LZF and Blosc writers
|
||||
were added the same day; v2.7.0 and earlier write neither).
|
||||
|
||||
`FileBuilder` stored every chunk of an LZF or Blosc dataset through the
|
||||
filter with filter mask 0. libhdf5 counts LZF and Blosc output no smaller
|
||||
than the chunk as a failure of the (optional) filter and stores the chunk
|
||||
raw with the filter's mask bit set. When a chunk's LZF stream was exactly
|
||||
the chunk's size, the first libhdf5 rewrite of it stored raw data at the
|
||||
same size and left our mask 0 in the index, so h5py could no longer read
|
||||
the dataset. `FileEditor` had the same bug, fixed earlier the same day.
|
||||
Both now use `clawhdf5_format::filters::compress_chunk_masked`, and every
|
||||
chunk index the writer builds records the real mask (see `CHANGELOG.md`).
|
||||
Files written before the fix read correctly; rewrite them before letting
|
||||
libhdf5 modify them.
|
||||
|
||||
## In-place modification (`FileEditor`) limits
|
||||
|
||||
**Status:** open (documented 2026-09-26). `clawhdf5::FileEditor` refuses,
|
||||
with `Error::Unsupported` and without writing anything:
|
||||
- new, moved or resized chunks in a **version-2 B-tree** chunk index (what
|
||||
libhdf5 uses for two or more unlimited dimensions) — existing unfiltered
|
||||
chunks, and filtered ones that re-encode to the same size and filter
|
||||
mask, are
|
||||
overwritten in place; `resize` works — and new chunks in an **implicit**
|
||||
index (it has all of its chunks from the start);
|
||||
- **shrinking** a dataset;
|
||||
- variable-length and reference data;
|
||||
- chunks through a filter this build cannot encode (scale-offset, N-Bit,
|
||||
SZIP, or a plugin filter it lacks), even an optional one: libhdf5 skips
|
||||
an optional filter only when its own build lacks it, which none does for
|
||||
these;
|
||||
- attributes of an object in **dense storage**, past its compact limit (8
|
||||
by default) or with tracked **creation order**;
|
||||
- partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
|
||||
raw data files, virtual datasets;
|
||||
- files with a metadata cache image, paged or persistent free-space
|
||||
management, a driver info block, or version-3 consistency flags set.
|
||||
|
||||
**Space is never reused.** There is no free-space manager: the old bytes of
|
||||
a filtered chunk that grows and has to move, and of an attribute that is
|
||||
replaced by a larger one, are leaked (`h5repack` reclaims them). A chunk
|
||||
that is the last thing in the file grows in place instead, which covers the
|
||||
usual append. Measured 2026-09-26 on tank with
|
||||
`cargo test --release -p clawhdf5-tools --test edit_interop -- --ignored
|
||||
--nocapture measure_append_waste` (file sizes are deterministic): 1000
|
||||
appends of 100 `f8` values to a 1-D dataset with 1024-element chunks give
|
||||
810 504 bytes unfiltered, as libhdf5's file, and 307 210 bytes with gzip
|
||||
(libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values with
|
||||
4096-element gzip chunks give 119 684 bytes against libhdf5's 50 292
|
||||
(`h5repack`: 49 930), because the chunk being appended to is followed by
|
||||
new index blocks and moves each time it grows.
|
||||
|
||||
**No journal.** A crash while an edit patches existing structures can leave
|
||||
the file inconsistent; see the `FileEditor` documentation.
|
||||
|
||||
## Selection reads that decode more than the selection
|
||||
|
||||
**Status:** open (documented 2026-09-26). `Dataset::read_selection` (and so
|
||||
|
||||
Reference in New Issue
Block a user