docs: FileEditor — changelog, limits and leaked space, README example
known-issues records what the editor refuses, that freed space is never reused (append-workload file sizes measured 2026-09-26 on tank with the ignored measure_append_waste test; sizes are deterministic), and that there is no journal. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,52 @@
|
|||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
|
### In-place modification (2026-09-26)
|
||||||
|
- **`clawhdf5::FileEditor` modifies an existing file where it lies.**
|
||||||
|
`FileBuilder` builds whole files in memory; the editor opens a file
|
||||||
|
written by libhdf5 (any `libver`, including HDF5 2.0's own format) or by
|
||||||
|
clawhdf5 and changes only what an edit touches, recomputing the checksum
|
||||||
|
of every structure it changes. It takes an exclusive `flock` on the file
|
||||||
|
(the lock libhdf5 takes), so a second editor gets `Error::Locked`.
|
||||||
|
- `write_selection` / `write_all` / `write_values`: overwrite values of a
|
||||||
|
compact, contiguous (also never-written, late-allocated) or chunked
|
||||||
|
dataset, in its own datatype, under any selection. Chunks are decoded,
|
||||||
|
updated and re-encoded through the dataset's filters; a chunk that no
|
||||||
|
longer fits moves to the end of the file. New chunks are added to
|
||||||
|
version-1 B-tree (every chunked dataset of h5py's default `libver`),
|
||||||
|
Extensible Array, Fixed Array and single-chunk indexes — creating the
|
||||||
|
index, its data blocks, super blocks and pages, and splitting B-tree
|
||||||
|
nodes, as libhdf5 does: after the same sequence of writes the B-tree has
|
||||||
|
the same number of nodes per level and the Extensible Array header the
|
||||||
|
same block statistics as libhdf5's (tested).
|
||||||
|
- `resize`: grow a chunked dataset up to its maximum dimensions (h5py's
|
||||||
|
`Dataset.resize`).
|
||||||
|
- `set_attr`: add or replace an attribute in an object header, in free
|
||||||
|
space or in a new continuation chunk at the end of the file.
|
||||||
|
- Each edit is planned in memory and refused as a whole
|
||||||
|
(`Error::Unsupported`, file untouched) when any part is not supported:
|
||||||
|
new chunks in a version-2 B-tree index (two or more unlimited
|
||||||
|
dimensions) or an implicit index, shrinking, variable-length and
|
||||||
|
reference data, attributes in dense storage, past an object's compact
|
||||||
|
limit or with tracked creation order, files with a metadata cache
|
||||||
|
image, paged or persistent free space, or marked open by another
|
||||||
|
writer. New error variants `Error::Unsupported`,
|
||||||
|
`Error::InvalidArgument`, `Error::Locked`, and `clawhdf5::Error` is now
|
||||||
|
`#[non_exhaustive]` — a breaking change for code that matches it
|
||||||
|
exhaustively (the Python bindings map the new variants to
|
||||||
|
`NotImplementedError`, `ValueError` and `OSError`).
|
||||||
|
- Durability: the new space (chunks, index blocks) is written and synced
|
||||||
|
before any existing byte changes, then the metadata that links it in,
|
||||||
|
then a second sync. There is no journal: a crash during the second
|
||||||
|
phase can leave the file inconsistent (as with libhdf5 without SWMR).
|
||||||
|
Freed space is not reused (see `docs/known-issues.md`).
|
||||||
|
- Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`,
|
||||||
|
`v114` and `latest` files and clawhdf5 files; after every round h5py
|
||||||
|
reads the expected values, h5dump and `h5rs check --data` accept the
|
||||||
|
file, and h5py `r+` modifies it further; random operations against a
|
||||||
|
model) and `crates/clawhdf5/tests/edit_tests.rs`.
|
||||||
|
- `clawhdf5_format::type_builders::build_attr_message` is public.
|
||||||
|
|
||||||
### Chunked full reads (2026-09-26)
|
### Chunked full reads (2026-09-26)
|
||||||
- **Chunks are decoded straight into the output, into reused buffers.** A
|
- **Chunks are decoded straight into the output, into reused buffers.** A
|
||||||
full read of a chunked dataset faulted in about three times its size in
|
full read of a chunked dataset faulted in about three times its size in
|
||||||
|
|||||||
@@ -150,6 +150,13 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F
|
|||||||
Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
|
Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
|
||||||
`MemorySource` for this bookkeeping is inferred from the caller-supplied
|
`MemorySource` for this bookkeeping is inferred from the caller-supplied
|
||||||
`source_channel` string (a heuristic, not an authenticated trust boundary).
|
`source_channel` string (a heuristic, not an authenticated trust boundary).
|
||||||
|
- In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`)
|
||||||
|
overwrites values, grows chunked datasets and sets attributes in existing
|
||||||
|
files (h5py- or clawhdf5-written) without rewriting them; anything it
|
||||||
|
cannot do safely is `Error::Unsupported` before any write (limits in
|
||||||
|
`docs/known-issues.md`). Test changes with
|
||||||
|
`cargo test -p clawhdf5-tools --test edit_interop` (h5py, h5dump,
|
||||||
|
`h5rs check`).
|
||||||
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
|
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
|
||||||
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
|
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
|
||||||
no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.
|
no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.
|
||||||
|
|||||||
@@ -433,6 +433,23 @@ b.write("groups.h5")?;
|
|||||||
A group holds at most 65 535 links; more is an error, as is a link over
|
A group holds at most 65 535 links; more is an error, as is a link over
|
||||||
65 515 bytes (a very long soft-link target) in a group of more than 8 links.
|
65 515 bytes (a very long soft-link target) in a group of more than 8 links.
|
||||||
|
|
||||||
|
### Modifying an existing file
|
||||||
|
|
||||||
|
```rust
|
||||||
|
use clawhdf5::{AttrValue, FileEditor, Selection};
|
||||||
|
|
||||||
|
// A file from h5py or clawhdf5, dataset "x" chunked with maxshape=(None,).
|
||||||
|
let mut ed = FileEditor::open("data.h5")?; // exclusive lock, like libhdf5
|
||||||
|
ed.resize("x", &[1100])?; // h5py: ds.resize((1100,))
|
||||||
|
let sel = Selection::Hyperslab { start: vec![1000], stride: vec![1], count: vec![100], block: vec![1] };
|
||||||
|
ed.write_values("x", &sel, &[0.5f64; 100])?; // ds[1000:1100] = 0.5
|
||||||
|
ed.set_attr("x", "units", &AttrValue::String("m/s".into()))?;
|
||||||
|
```
|
||||||
|
|
||||||
|
Each call changes the file in place (no rewrite) and syncs it. What it
|
||||||
|
cannot change safely is refused before anything is written; see
|
||||||
|
[known issues](docs/known-issues.md) for the limits.
|
||||||
|
|
||||||
### Python
|
### Python
|
||||||
|
|
||||||
`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
|
`crates/clawhdf5-py` is a Python package (PyO3 + numpy) that reads HDF5 with
|
||||||
|
|||||||
@@ -7,6 +7,41 @@ deleting it.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## In-place modification (`FileEditor`) limits
|
||||||
|
|
||||||
|
**Status:** open (documented 2026-09-26). `clawhdf5::FileEditor` refuses,
|
||||||
|
with `Error::Unsupported` and without writing anything:
|
||||||
|
- new, moved or resized chunks in a **version-2 B-tree** chunk index (what
|
||||||
|
libhdf5 uses for two or more unlimited dimensions) — existing unfiltered
|
||||||
|
chunks, and filtered ones that re-encode to the same size, are
|
||||||
|
overwritten in place; `resize` works — and new chunks in an **implicit**
|
||||||
|
index (it has all of its chunks from the start);
|
||||||
|
- **shrinking** a dataset;
|
||||||
|
- variable-length and reference data;
|
||||||
|
- attributes of an object in **dense storage**, past its compact limit (8
|
||||||
|
by default) or with tracked **creation order**;
|
||||||
|
- partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external
|
||||||
|
raw data files, virtual datasets;
|
||||||
|
- files with a metadata cache image, paged or persistent free-space
|
||||||
|
management, a driver info block, or version-3 consistency flags set.
|
||||||
|
|
||||||
|
**Space is never reused.** There is no free-space manager: the old bytes of
|
||||||
|
a filtered chunk that grows and has to move, and of an attribute that is
|
||||||
|
replaced by a larger one, are leaked (`h5repack` reclaims them). A chunk
|
||||||
|
that is the last thing in the file grows in place instead, which covers the
|
||||||
|
usual append. Measured 2026-09-26 on tank with
|
||||||
|
`cargo test --release -p clawhdf5-tools --test edit_interop -- --ignored
|
||||||
|
--nocapture measure_append_waste` (file sizes are deterministic): 1000
|
||||||
|
appends of 100 `f8` values to a 1-D dataset with 1024-element chunks give
|
||||||
|
810 504 bytes unfiltered, as libhdf5's file, and 307 210 bytes with gzip
|
||||||
|
(libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values with
|
||||||
|
4096-element gzip chunks give 119 684 bytes against libhdf5's 50 292
|
||||||
|
(`h5repack`: 49 930), because the chunk being appended to is followed by
|
||||||
|
new index blocks and moves each time it grows.
|
||||||
|
|
||||||
|
**No journal.** A crash while an edit patches existing structures can leave
|
||||||
|
the file inconsistent; see the `FileEditor` documentation.
|
||||||
|
|
||||||
## Selection reads that decode more than the selection
|
## Selection reads that decode more than the selection
|
||||||
|
|
||||||
**Status:** open (documented 2026-09-26). `Dataset::read_selection` (and so
|
**Status:** open (documented 2026-09-26). `Dataset::read_selection` (and so
|
||||||
|
|||||||
Reference in New Issue
Block a user