docs: FileEditor — changelog, limits and leaked space, README example
known-issues records what the editor refuses, that freed space is never reused (append-workload file sizes measured 2026-09-26 on tank with the ignored measure_append_waste test; sizes are deterministic), and that there is no journal. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,52 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### In-place modification (2026-09-26)
|
||||
- **`clawhdf5::FileEditor` modifies an existing file where it lies.**
|
||||
`FileBuilder` builds whole files in memory; the editor opens a file
|
||||
written by libhdf5 (any `libver`, including HDF5 2.0's own format) or by
|
||||
clawhdf5 and changes only what an edit touches, recomputing the checksum
|
||||
of every structure it changes. It takes an exclusive `flock` on the file
|
||||
(the lock libhdf5 takes), so a second editor gets `Error::Locked`.
|
||||
- `write_selection` / `write_all` / `write_values`: overwrite values of a
|
||||
compact, contiguous (also never-written, late-allocated) or chunked
|
||||
dataset, in its own datatype, under any selection. Chunks are decoded,
|
||||
updated and re-encoded through the dataset's filters; a chunk that no
|
||||
longer fits moves to the end of the file. New chunks are added to
|
||||
version-1 B-tree (every chunked dataset of h5py's default `libver`),
|
||||
Extensible Array, Fixed Array and single-chunk indexes — creating the
|
||||
index, its data blocks, super blocks and pages, and splitting B-tree
|
||||
nodes, as libhdf5 does: after the same sequence of writes the B-tree has
|
||||
the same number of nodes per level and the Extensible Array header the
|
||||
same block statistics as libhdf5's (tested).
|
||||
- `resize`: grow a chunked dataset up to its maximum dimensions (h5py's
|
||||
`Dataset.resize`).
|
||||
- `set_attr`: add or replace an attribute in an object header, in free
|
||||
space or in a new continuation chunk at the end of the file.
|
||||
- Each edit is planned in memory and refused as a whole
|
||||
(`Error::Unsupported`, file untouched) when any part is not supported:
|
||||
new chunks in a version-2 B-tree index (two or more unlimited
|
||||
dimensions) or an implicit index, shrinking, variable-length and
|
||||
reference data, attributes in dense storage, past an object's compact
|
||||
limit or with tracked creation order, files with a metadata cache
|
||||
image, paged or persistent free space, or marked open by another
|
||||
writer. New error variants `Error::Unsupported`,
|
||||
`Error::InvalidArgument`, `Error::Locked`, and `clawhdf5::Error` is now
|
||||
`#[non_exhaustive]` — a breaking change for code that matches it
|
||||
exhaustively (the Python bindings map the new variants to
|
||||
`NotImplementedError`, `ValueError` and `OSError`).
|
||||
- Durability: the new space (chunks, index blocks) is written and synced
|
||||
before any existing byte changes, then the metadata that links it in,
|
||||
then a second sync. There is no journal: a crash during the second
|
||||
phase can leave the file inconsistent (as with libhdf5 without SWMR).
|
||||
Freed space is not reused (see `docs/known-issues.md`).
|
||||
- Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
|
||||
`v114` and `latest` files and clawhdf5 files; after every round h5py
|
||||
reads the expected values, h5dump and `h5rs check --data` accept the
|
||||
file, and h5py `r+` modifies it further; random operations against a
|
||||
model) and `crates/clawhdf5/tests/edit_tests.rs`.
|
||||
- `clawhdf5_format::type_builders::build_attr_message` is public.
|
||||
|
||||
### Chunked full reads (2026-09-26)
|
||||
- **Chunks are decoded straight into the output, into reused buffers.** A
|
||||
full read of a chunked dataset faulted in about three times its size in
|
||||
|
||||
Reference in New Issue
Block a user