From 0aca0eb724c2bb5ba2595501fadcefc106d88547 Mon Sep 17 00:00:00 2001 From: osobh Date: Sat, 26 Sep 2026 17:13:31 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20editor=20coverage=20=E2=80=94=20version?= =?UTF-8?q?-2=20B-trees,=20shrinking,=20dense=20attributes,=20reuse?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CHANGELOG (Unreleased): the new FileEditor operations, space reuse, and the two reader fixes (implicit index grid, object-header continuation chains). known-issues: the editor's remaining refusals (skipped heap blocks, heaps with filters or child indirect blocks, freeing a heap block, implicit-index insertions, ...) and the append-waste sizes before and after reuse (measure_append_waste, tank 2026-09-26; file sizes are deterministic). range-reads design: status note on the reader changes. README and CLAUDE.md: what the editor covers and how to test it. Co-Authored-By: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 69 +++++++++++++++++++++++++++++++++++++- CLAUDE.md | 16 ++++++--- README.md | 6 +++- docs/design/range-reads.md | 8 ++++- docs/known-issues.md | 57 ++++++++++++++++++------------- 5 files changed, 124 insertions(+), 32 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fa0a30f..551b079 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,72 @@ ## Unreleased +### In-place editing: version-2 B-tree indexes, shrinking, dense attributes (2026-09-26) +- **`FileEditor` adds, moves and resizes chunks of datasets with two or + more unlimited dimensions** (version-2 B-tree chunk index, record types + 10/11), as libhdf5's `H5B2` code does: `H5B2_update`'s insert-or-modify, + the preemptive split/redistribute loop, `split1`/`split_root` (depth + growth), `redistribute2/3`, and removal with `merge2/3`, root collapse + and the internal-record swap; node pointer widths and cumulative record + counts per depth; a missing index is created from the layout message's + parameters. After the same growth libhdf5's and the editor's trees are + node for node the same (tested through a depth increase). +- **`FileEditor::resize` shrinks** along any dimension (h5py's + `Dataset.resize` to a smaller shape), as `H5D__chunk_prune_by_extent` + does, visiting the same chunks in the same order: chunks wholly outside + the new extent leave the index (version-1 B-tree removal with libhdf5's + sibling key and link fix-ups and empty-root case, version-2 B-tree + removal, Fixed/Extensible Array elements reset; an implicit index keeps + its chunks, as in libhdf5) and their space is freed; the part of a + partial edge chunk outside the extent is overwritten with the fill value, + so it reads as fill after a later growth. Growth under early allocation + now allocates and fills the new chunks (`H5D__chunk_allocate`), which an + implicit index needs. Shrinking was `Error::Unsupported`. +- **`FileEditor::set_attr` handles dense attribute storage and creation + order**: objects that track (and index) attribute creation order; the + move to dense storage when an object reaches its compact limit (or an + attribute is too large for a header message), as `H5O__attr_create` + does it (new fractal heap, name index, creation-order index when + indexed, compact attributes moved over in header order); objects + already in dense storage (h5py- or clawhdf5-written): insertion, + same-size rewrites in place, other replacements by removal and + insertion. The heap is changed as `H5HF` changes it — best-fit free + sections from its free-space manager (kept as libhdf5 keeps `FSHD`/ + `FSSE`), new direct blocks through the root indirect block (created, + doubled), huge objects through the huge-object B-tree (deleted with the + last huge object), removed objects' space merged back — with libhdf5's + statistics: after the same attribute workload the heap, its free space + and both index B-trees equal libhdf5's. Attributes are encoded as libhdf5 + encodes them for a file h5py opens `r+` (message version 1, 3 for + non-ASCII names; simple dataspaces with their maximum dimensions). + Still refused: see `docs/known-issues.md`. +- **Freed space is reused within an editing session.** A `FileEditor` + reuses (best fit, zeroed) what its earlier edits freed — moved filtered + chunks, pruned chunks, merged B-tree nodes, replaced heap blocks — never + what the current edit frees, and writes reused blocks with the new space + before any existing byte changes. `FileEditor::reusable_bytes`. The + append workload of `measure_append_waste` leaks less (sizes in + `docs/known-issues.md`). +- **Reader: implicit chunk indexes below their maximum shape.** libhdf5 + places an implicit index's chunks by their position in the *maximum* + chunk grid; the reader used the current grid and returned other chunks' + values from the second chunk row on (h5py early allocation with a fixed + `maxshape` larger than the shape). + `chunked_read::generate_implicit_chunks_in_grid` takes the maximum. +- **Reader: object headers with long continuation chains.** A version-1 + header whose continuation chunks chain more than 32 deep (a header that + gains a chunk per attribute added when full, as libhdf5 and the editor + grow it) was refused with `NestingDepthExceeded`; version-2 headers + stopped at 256 chunks. Chunks are now followed without recursion, in the + same order; a chunk address seen twice (a cycle) or more than 65 536 + chunks are refused. +- Tests: `crates/clawhdf5-tools/tests/edit_coverage_interop.rs` (h5py + `earliest`/`v110`/`latest` and clawhdf5-written files; structure + comparisons with libhdf5 for version-2 B-trees, shrink on every index, + and dense attribute heaps); the random-operation property test in + `edit_interop.rs` now shrinks, grows two unlimited dimensions and moves + attributes to dense storage (`CLAWHDF5_EDIT_SEED` for other seeds). + ### Name lookups through the name index (2026-09-26) - **Finding one link or attribute by name reads the name index, not every entry.** In a dense group (links in a fractal heap) the v2 B-tree name @@ -237,7 +303,8 @@ before any existing byte changes, then the metadata that links it in, then a second sync. There is no journal: a crash during the second phase can leave the file inconsistent (as with libhdf5 without SWMR). - Freed space is not reused (see `docs/known-issues.md`). + Freed space is not reused (see `docs/known-issues.md`; since reused + within an editing session, above). - Tests: `crates/clawhdf5-tools/tests/edit_interop.rs` (h5py `earliest`, `v114` and `latest` files and clawhdf5 files; after every round h5py reads the expected values, h5dump and `h5rs check --data` accept the diff --git a/CLAUDE.md b/CLAUDE.md index f493066..14463ac 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -151,12 +151,18 @@ Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal F `MemorySource` for this bookkeeping is inferred from the caller-supplied `source_channel` string (a heuristic, not an authenticated trust boundary). - In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`) - overwrites values, grows chunked datasets and sets attributes in existing - files (h5py- or clawhdf5-written) without rewriting them; anything it - cannot do safely is `Error::Unsupported` before any write (limits in + overwrites values, grows and shrinks chunked datasets (every chunk index, + version-2 B-trees included) and sets attributes (compact and dense + storage) in existing files (h5py- or clawhdf5-written) without rewriting + them, changing indexes and heaps as libhdf5 does (index shapes and heap + bookkeeping are compared with libhdf5's in the tests); space an edit + frees is reused by later edits of the same editor. Anything it cannot do + safely is `Error::Unsupported` before any write (limits in `docs/known-issues.md`). Test changes with - `cargo test -p clawhdf5-tools --test edit_interop` (h5py, h5dump, - `h5rs check`). + `cargo test -p clawhdf5-tools --test edit_interop --test + edit_coverage_interop` (h5py, h5dump, `h5rs check`, structure comparisons + with libhdf5; libhdf5 sources for the algorithms are at + github.com/HDFGroup/hdf5, tag `hdf5_1_14_6`). - GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only - Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory; no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page. diff --git a/README.md b/README.md index 88ad61b..a87a43b 100644 --- a/README.md +++ b/README.md @@ -444,9 +444,13 @@ ed.resize("x", &[1100])?; // h5py: ds.resize((1100,)) let sel = Selection::Hyperslab { start: vec![1000], stride: vec![1], count: vec![100], block: vec![1] }; ed.write_values("x", &sel, &[0.5f64; 100])?; // ds[1000:1100] = 0.5 ed.set_attr("x", "units", &AttrValue::String("m/s".into()))?; +ed.resize("x", &[900])?; // shrinking prunes chunks, like h5py ``` -Each call changes the file in place (no rewrite) and syncs it. What it +Each call changes the file in place (no rewrite) and syncs it. Any chunk +index (version-2 B-trees for several unlimited dimensions included) and +attributes in compact or dense storage are handled as libhdf5 handles +them; space an edit frees is reused by later edits of the same editor. What it cannot change safely is refused before anything is written; see [known issues](docs/known-issues.md) for the limits. diff --git a/docs/design/range-reads.md b/docs/design/range-reads.md index 1c7fa55..7490c7a 100644 --- a/docs/design/range-reads.md +++ b/docs/design/range-reads.md @@ -3,7 +3,13 @@ Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural change. Progress: M1, first part (the `Storage` trait and the metadata parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done; -group B-tree v2 lookups, dense groups and the facade are not converted yet. Every count below was +group B-tree v2 lookups, dense groups and the facade are not converted yet. +Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched +converted code without changing the plan: object-header continuation chunks +are followed without recursion (still one bounded `read_at` per chunk), and +implicit chunk indexes are addressed over the maximum chunk grid (in +`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps +working on the whole file in memory; it is not part of this design. Every count below was taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given next to it. No timing numbers appear here on purpose: the machine was shared with other build jobs when this was written. diff --git a/docs/known-issues.md b/docs/known-issues.md index dcb365f..1b05553 100644 --- a/docs/known-issues.md +++ b/docs/known-issues.md @@ -26,40 +26,49 @@ libhdf5 modify them. ## In-place modification (`FileEditor`) limits -**Status:** open (documented 2026-09-26). `clawhdf5::FileEditor` refuses, -with `Error::Unsupported` and without writing anything: -- new, moved or resized chunks in a **version-2 B-tree** chunk index (what - libhdf5 uses for two or more unlimited dimensions) — existing unfiltered - chunks, and filtered ones that re-encode to the same size and filter - mask, are - overwritten in place; `resize` works — and new chunks in an **implicit** - index (it has all of its chunks from the start); -- **shrinking** a dataset; +**Status:** open (documented 2026-09-26, updated the same day when +version-2 B-tree chunk indexes, shrinking, dense attributes and space +reuse were added). `clawhdf5::FileEditor` refuses, with +`Error::Unsupported` and without writing anything: +- new chunks in an **implicit** index (it has all of its chunks from the + start; they are written in place, and allocated/filled on growth under + early allocation as libhdf5 does); - variable-length and reference data; - chunks through a filter this build cannot encode (scale-offset, N-Bit, SZIP, or a plugin filter it lacks), even an optional one: libhdf5 skips an optional filter only when its own build lacks it, which none does for these; -- attributes of an object in **dense storage**, past its compact limit (8 - by default) or with tracked **creation order**; +- attributes in dense storage when the heap cannot take them the way + libhdf5 would: an attribute that needs a heap block larger than the next + one (libhdf5 skips blocks and records them as free space — in practice an + attribute of roughly 1 to 4 KiB going into a young heap), a heap with I/O + filters or child indirect blocks (more than about 512 KiB of attributes), + free space the heap tracks outside direct blocks, replacing the last + attribute left in a heap block by one of another size (libhdf5 frees the + block), directly addressed huge objects; and shared attribute messages; +- version-1 object headers asked for an attribute larger than a header + message (they have no dense storage); - partial edge chunks stored unfiltered (`H5Pset_chunk_opts`), external raw data files, virtual datasets; - files with a metadata cache image, paged or persistent free-space management, a driver info block, or version-3 consistency flags set. -**Space is never reused.** There is no free-space manager: the old bytes of -a filtered chunk that grows and has to move, and of an attribute that is -replaced by a larger one, are leaked (`h5repack` reclaims them). A chunk -that is the last thing in the file grows in place instead, which covers the -usual append. Measured 2026-09-26 on tank with -`cargo test --release -p clawhdf5-tools --test edit_interop -- --ignored ---nocapture measure_append_waste` (file sizes are deterministic): 1000 -appends of 100 `f8` values to a 1-D dataset with 1024-element chunks give -810 504 bytes unfiltered, as libhdf5's file, and 307 210 bytes with gzip -(libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values with -4096-element gzip chunks give 119 684 bytes against libhdf5's 50 292 -(`h5repack`: 49 930), because the chunk being appended to is followed by -new index blocks and moves each time it grows. +**Space is reused only within one editor.** Space an edit frees (a filtered +chunk that moves, chunks a shrink removes, B-tree nodes merged away, a +heap's replaced blocks) is reused by later edits of the same `FileEditor`; +what is left when it is dropped is leaked, as libhdf5 leaks it without a +persistent free-space manager (`h5repack` reclaims it). A chunk that is the +last thing in the file grows in place, which covers the usual append. +Measured 2026-09-26 on tank with `cargo test -p clawhdf5-tools --test +edit_interop -- --ignored --nocapture measure_append_waste` (one editor for +the whole workload; file sizes are deterministic): 1000 appends of 100 `f8` +values to a 1-D dataset with 1024-element chunks give 810 504 bytes +unfiltered, as libhdf5's file, and 306 780 bytes with gzip (307 210 before +reuse; libhdf5: 306 058; `h5repack`: 306 104); 2000 appends of 10 values +with 4096-element gzip chunks give 79 829 bytes (119 684 before reuse) +against libhdf5's 50 292 (`h5repack`: 49 930): the chunk being appended to +is followed by new index blocks and moves each time it grows, and the +space it leaves is too small for its next, larger version. **No journal.** A crash while an edit patches existing structures can leave the file inconsistent; see the `FileEditor` documentation.