From ac0020594bf961ffa95fa412019353b9662bd5be Mon Sep 17 00:00:00 2001 From: osobh Date: Mon, 28 Sep 2026 11:05:38 -0500 Subject: [PATCH] docs: design status notes reflect what is merged range-reads.md opens with a table of milestones M0-M5 and the PRs that merged them (#17-#21), replacing a header left garbled by earlier merges, and each milestone's status names its PR. swmr.md says the reader is merged (PR #19) and the writer does not exist. openclaw.md links the Node package's known-issues entry. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/range-reads.md | 76 ++++++++++++++++++++------------------ docs/design/swmr.md | 13 +++++-- docs/openclaw.md | 4 +- 3 files changed, 52 insertions(+), 41 deletions(-) diff --git a/docs/design/range-reads.md b/docs/design/range-reads.md index 9dea022..18ebec4 100644 --- a/docs/design/range-reads.md +++ b/docs/design/range-reads.md @@ -1,40 +1,33 @@ # Design: range reads (reading HDF5 without holding the whole file) -Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural -change. Progress: M0 and M1 are done, and so is M2 (branch -`feat/p3-m2-raw-data`): every read path of the format crate works through -`Storage`, v2 B-trees, dense groups and raw data included, and -`File::open_storage` gives the facade's read API over any `Storage` (see -`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch -`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S), -object stores) and URLs in `h5rs` (see the M3 status below). M5 (SWMR) is -done on branch `feat/p3-m5-swmr-reader`, with its own design in -[`swmr.md`](swmr.md) (see the M5 status below). M4 (wasm) is -next. Every count in §1–§2 was +Status (updated 2026-09-28): **implemented and merged.** Proposed +2026-09-26 as the plan for Phase 3's largest architectural change; every +milestone below is on `main`: -object stores) and URLs in `h5rs` (see the M3 status below); the Python -bindings followed on branch `feat/p3-python-remote-edit` (2026-09-27), -which completes M3. M4 (wasm) is next. Every count in §1–§2 was +| Milestone | What | Merged | +|---|---|---| +| M0 | indexed name lookups, checked address conversion | PR #17 (`8f59b2e`) | +| M1 | metadata parsers over the `Storage` trait | PR #17 (`8f59b2e`) | +| M2 | raw data over `Storage`, `File::open_storage` | PR #18 (`a4c2ace`) | +| M3 | `clawhdf5-remote` (block cache, HTTP(S), object stores), URLs in `h5rs`; Python `clawhdf5.File(url)` | PR #18 (`a4c2ace`); Python in PR #19 (`7a8fae0`) | +| M4 | wasm `openUrl` through the restartable `NeedBytes` mode | PR #19 (`7a8fae0`); fewer round trips in PR #21 (`9b5803f`) | +| M5 | SWMR reader (`File::open_swmr`), design in [`swmr.md`](swmr.md) | PR #19 (`7a8fae0`) | -change. Progress: M1, first part (the `Storage` trait and the metadata -parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done; -group B-tree v2 lookups, dense groups and the facade are not converted yet. -Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched +Each milestone's own *Status* note in §4 records what was built and how it +differs from the plan. What is still missing is tracked in +[`docs/known-issues.md`](../known-issues.md) ("Range reads", "Remote +files" and "`clawhdf5-wasm`" limits); the main gaps are a paged file's page +size as the block size, remote SWMR, and a SWMR writer. -object stores) and URLs in `h5rs` (see the M3 status below). M4 is done on -branch `feat/p3-m4-wasm-lazy` (2026-09-27): `openUrl` in the browser -reader, through the restartable `NeedBytes` mode (see the M4 status -below). M5 (SWMR) is not started. Also on 2026-09-26 (branch `feat/p3-editor-coverage`) two reader fixes touched converted code without changing the plan: object-header continuation chunks are followed without recursion (still one bounded `read_at` per chunk), and implicit chunk indexes are addressed over the maximum chunk grid (in -`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps -working on the whole file in memory; it is not part of this design. Every -count in §1–§2 was taken on `tank` on 2026-09-26 at commit `de2a53f`, and -every count in a milestone's status on the date it gives, with the -commands given next to it. No timing numbers appear here on purpose: the machine was shared -with other build jobs when this was written. +`chunked_read`, an M2 module). The in-place editor (`FileEditor`) is not part +of this design. Every count in §1–§2 was taken on `tank` on 2026-09-26 at +commit `de2a53f`, and every count in a milestone's status on the date it +gives, with the commands given next to it. No timing numbers appear here on +purpose: the machine was shared with other build jobs when this was written. ## The problem @@ -398,7 +391,8 @@ Rejected. It is how one would retrofit a C library that cannot change; we can. Adopt **(a)**, with a block cache as a required part of every non-local backend, **(c)** as a cache policy, and the wasm path through the restartable `NeedBytes` mode. Every milestone keeps `main` green: `cargo test ---workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap +--workspace`, clippy, the conformance gate unchanged (575/697 when this was written; 602/697 +after PR #21), and the mmap fast path within benchmark noise. **M0 — prerequisites (≈1 week).** @@ -414,7 +408,8 @@ fast path within benchmark noise. n children decodes its links O(n) times. Look names up through the index (above) and let a listing hand out its entries, so the cache has less to absorb. -- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` — link and +- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` (merged in + PR #17) — link and attribute names through the name indexes (`group_v2::resolve_child`, `attribute::find_attribute_in_file`; creation-order lookups by name do not exist in the API, so the creation-order index is still only listed), @@ -438,6 +433,11 @@ fast path within benchmark noise. (`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so callers and the other crates don't move yet. - Replace the 5 open-ended slices and 38 `len()` checks with bounded reads. +- *Status 2026-09-26:* done on branch `feat/p3-storage-trait` (merged in + PR #17) for the `Storage` trait and the metadata parsers listed in + `CHANGELOG.md` under "Range reads, milestone M1"; group B-tree v2 lookups, + dense groups and the facade were converted in M2. Both error enums are + `#[non_exhaustive]`; storage failures are `FormatError::Storage`. **M2 — raw data over the trait (1–2 weeks).** - `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`, @@ -449,7 +449,8 @@ fast path within benchmark noise. on other backends (they already return `Option`/`Result`). - Facade: `File::open_storage(Box)`; `File::open` keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`. -- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data`. As planned, +- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data` (merged in PR + #18). As planned, with these choices: - `File::open_storage` takes an `Arc` (the file handle is shared by its datasets and may be sent across threads). @@ -487,7 +488,8 @@ fast path within benchmark noise. §2; the page size for paged files; the first block prefetched on open) and a request counter exposed for tests and users. - Python bindings: `clawhdf5.File("s3://…")` / `https://` through it. -- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the +- *Status 2026-09-26:* done on branch `feat/p3-m3-remote` (merged in PR + #18), except the Python bindings (done 2026-09-27, below), with these choices: - A new crate, `clawhdf5-remote`, instead of a `remote` feature of `clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io` @@ -526,7 +528,7 @@ fast path within benchmark noise. requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole 6.4 MB: 35 001 object headers spread over the file). - *Status 2026-09-27, Python bindings:* done on branch - `feat/p3-python-remote-edit`. `clawhdf5.File(url)` and + `feat/p3-python-remote-edit` (merged in PR #19). `clawhdf5.File(url)` and `File.open_url(url, **options)` (cache and HTTP options) go through `clawhdf5_remote::storage_for_url`; the default wheel is plain HTTP (no C), `https`/`s3`/`gcs`/`azure` are build features. The bindings' own @@ -543,7 +545,8 @@ fast path within benchmark noise. Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back to a whole download when the server does not answer 206. - `examples/wasm-viewer`: open by URL. -- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy`, as planned, +- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy` (merged in PR + #19), as planned, with these choices: - **NeedBytes, not a Worker.** `clawhdf5_wasm::lazy::LazyStorage` is a `Storage` over the blocks fetched so far. A call (open, list, read) @@ -607,7 +610,7 @@ fast path within benchmark noise. missing blocks: listing 3000 datasets went from 185 passes to 6. The 32-bit risk below is covered by a Node test that reads data at 3 GiB from a mock server and is refused a 4 GiB file. - - Fewer round trips (2026-09-27, later): the walks descend into every + - Fewer round trips (2026-09-27, later; merged in PR #21): the walks descend into every child after a failure (not only read the siblings), and parsers call `Storage::hint` for what they read next (node bodies, object header chunks, a dense group's heap blocks, a listing's child @@ -621,7 +624,8 @@ fast path within benchmark noise. **M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow; add `File::refresh()` that re-reads the superblock/EOF and invalidates cached blocks past the old end. Needs libhdf5 SWMR semantics research first. -- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader`; design and +- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader` (merged in + PR #19); design and libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's `H5Drefresh`), not per file — a SWMR writer only grows datasets, and the diff --git a/docs/design/swmr.md b/docs/design/swmr.md index dba542a..4b89845 100644 --- a/docs/design/swmr.md +++ b/docs/design/swmr.md @@ -1,7 +1,8 @@ # Design: reading files a SWMR writer is still appending to (range-read M5) -Status: design 2026-09-27, implemented on branch `feat/p3-m5-swmr-reader` -(see "Status" at the end). This is milestone M5 of +Status: design 2026-09-27; the reader is implemented and merged (branch +`feat/p3-m5-swmr-reader`, PR #19, `7a8fae0`; see "Status" at the end). +clawhdf5 has no SWMR writer. This is milestone M5 of [`range-reads.md`](range-reads.md): "`Storage::len()` may grow; add a refresh". It covers the reader only; clawhdf5 does not write SWMR files. @@ -165,7 +166,7 @@ writer cannot add them), and `MmapFile`/`LazyFile`. ## Status Implemented 2026-09-27 on branch `feat/p3-m5-swmr-reader` as designed -above (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the +above, merged to `main` in PR #19 (`7a8fae0`) (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the same day (h5py 3.16 / HDF5 2.0, `cargo test -p clawhdf5 --test swmr_interop`, and once with `CLAWHDF5_SWMR_STEPS=20000` in a release build): no read returned a value the writer had not written at that @@ -175,4 +176,8 @@ variant of the test with the chunk cache left on in live mode fails it (stale chunk index / edge chunk), which is why live files do not use it. Also found: `File::open` of such a file had been failing since the -end-of-file check of 2026-09-26 (item 1; `docs/known-issues.md`). +end-of-file check of 2026-09-26 (item 1; fixed before any release, see +[`docs/known-issues.md`](../known-issues.md#files-a-swmr-writer-had-open-could-not-be-read-past-a-stale-end-of-file)). + +Not done (tracked in `docs/known-issues.md`, "Range reads" limits): SWMR +writing, remote SWMR, and live reading through `MmapFile`/`LazyFile`. diff --git a/docs/openclaw.md b/docs/openclaw.md index 6e76a08..e6ac2f0 100644 --- a/docs/openclaw.md +++ b/docs/openclaw.md @@ -71,4 +71,6 @@ Building blocks, usable as a library today, but not an OpenClaw plugin: and export rewrites every heading as `##`. - `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a TypeScript wrapper. **Not published, not built or tested in CI, and known to - be broken**; see `docs/known-issues.md`. + be broken**; see + [`docs/known-issues.md`](known-issues.md#the-nodejs-package-packagesclawhdf5-node-does-not-work) + (re-checked 2026-09-28: unchanged).