docs: FileEditor — changelog, limits and leaked space, README example

known-issues records what the editor refuses, that freed space is never
reused (append-workload file sizes measured 2026-09-26 on tank with the
ignored measure_append_waste test; sizes are deterministic), and that there
is no journal.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-26 13:35:06 -05:00
co-authored by Claude Opus 5.5
parent 3c89a31df0
commit 677dc5ec7c
4 changed files with 105 additions and 0 deletions
+35
View File
@@ -7,6 +7,41 @@ deleting it.
---
## In-place modification (`FileEditor`) limits
**Status:** open (documented 2026-09-26). `clawhdf5::FileEditor` refuses,
with `Error::Unsupported` and without writing anything:
- new, moved or resized chunks in a **version-2 B-tree** chunk index (what
libhdf5 uses for two or more unlimited dimensions) — existing unfiltered
chunks, and filtered ones that re-encode to the same size, are
overwritten in place; `resize` works — and new chunks in an **implicit**
index (it has all of its chunks from the start);
- **shrinking** a dataset;
- variable-length and reference data;
- attributes of an object in **dense storage**, past its compact limit (8
by default) or with tracked **creation order**;
- partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
raw data files, virtual datasets;
- files with a metadata cache image, paged or persistent free-space
management, a driver info block, or version-3 consistency flags set.
**Space is never reused.** There is no free-space manager: the old bytes of
a filtered chunk that grows and has to move, and of an attribute that is
replaced by a larger one, are leaked (`h5repack` reclaims them). A chunk
that is the last thing in the file grows in place instead, which covers the
usual append. Measured 2026-09-26 on tank with
`cargo test --release -p clawhdf5-tools --test edit_interop -- --ignored
--nocapture measure_append_waste` (file sizes are deterministic): 1000
appends of 100 `f8` values to a 1-D dataset with 1024-element chunks give
810 504 bytes unfiltered, as libhdf5's file, and 307 210 bytes with gzip
(libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values with
4096-element gzip chunks give 119 684 bytes against libhdf5's 50 292
(`h5repack`: 49 930), because the chunk being appended to is followed by
new index blocks and moves each time it grows.
**No journal.** A crash while an edit patches existing structures can leave
the file inconsistent; see the `FileEditor` documentation.
## Selection reads that decode more than the selection
**Status:** open (documented 2026-09-26). `Dataset::read_selection` (and so