docs: design status notes reflect what is merged

range-reads.md opens with a table of milestones M0-M5 and the PRs that
merged them (#17-#21), replacing a header left garbled by earlier
merges, and each milestone's status names its PR. swmr.md says the
reader is merged (PR #19) and the writer does not exist. openclaw.md
links the Node package's known-issues entry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-28 11:05:38 -05:00
co-authored by Claude Opus 5.5
parent 06d8e2ee45
commit ac0020594b
3 changed files with 52 additions and 41 deletions
+40 -36
View File
@@ -1,40 +1,33 @@
# Design: range reads (reading HDF5 without holding the whole file)
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
change. Progress: M0 and M1 are done, and so is M2 (branch
`feat/p3-m2-raw-data`): every read path of the format crate works through
`Storage`, v2 B-trees, dense groups and raw data included, and
`File::open_storage` gives the facade's read API over any `Storage` (see
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
object stores) and URLs in `h5rs` (see the M3 status below). M5 (SWMR) is
done on branch `feat/p3-m5-swmr-reader`, with its own design in
[`swmr.md`](swmr.md) (see the M5 status below). M4 (wasm) is
next. Every count in §1–§2 was
Status (updated 2026-09-28): **implemented and merged.** Proposed
2026-09-26 as the plan for Phase 3's largest architectural change; every
milestone below is on `main`:
object stores) and URLs in `h5rs` (see the M3 status below); the Python
bindings followed on branch `feat/p3-python-remote-edit` (2026-09-27),
which completes M3. M4 (wasm) is next. Every count in §1–§2 was
| Milestone | What | Merged |
|---|---|---|
| M0 | indexed name lookups, checked address conversion | PR #17 (`8f59b2e`) |
| M1 | metadata parsers over the `Storage` trait | PR #17 (`8f59b2e`) |
| M2 | raw data over `Storage`, `File::open_storage` | PR #18 (`a4c2ace`) |
| M3 | `clawhdf5-remote` (block cache, HTTP(S), object stores), URLs in `h5rs`; Python `clawhdf5.File(url)` | PR #18 (`a4c2ace`); Python in PR #19 (`7a8fae0`) |
| M4 | wasm `openUrl` through the restartable `NeedBytes` mode | PR #19 (`7a8fae0`); fewer round trips in PR #21 (`9b5803f`) |
| M5 | SWMR reader (`File::open_swmr`), design in [`swmr.md`](swmr.md) | PR #19 (`7a8fae0`) |
change. Progress: M1, first part (the `Storage` trait and the metadata
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
group B-tree v2 lookups, dense groups and the facade are not converted yet.
Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched
Each milestone's own *Status* note in §4 records what was built and how it
differs from the plan. What is still missing is tracked in
[`docs/known-issues.md`](../known-issues.md) ("Range reads", "Remote
files" and "`clawhdf5-wasm`" limits); the main gaps are a paged file's page
size as the block size, remote SWMR, and a SWMR writer.
object stores) and URLs in `h5rs` (see the M3 status below). M4 is done on
branch `feat/p3-m4-wasm-lazy` (2026-09-27): `openUrl` in the browser
reader, through the restartable `NeedBytes` mode (see the M4 status
below). M5 (SWMR) is not started.
Also on 2026-09-26 (branch `feat/p3-editor-coverage`) two reader fixes touched
converted code without changing the plan: object-header continuation chunks
are followed without recursion (still one bounded `read_at` per chunk), and
implicit chunk indexes are addressed over the maximum chunk grid (in
`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps
working on the whole file in memory; it is not part of this design. Every
count in §1–§2 was taken on `tank` on 2026-09-26 at commit `de2a53f`, and
every count in a milestone's status on the date it gives, with the
commands given next to it. No timing numbers appear here on purpose: the machine was shared
with other build jobs when this was written.
`chunked_read`, an M2 module). The in-place editor (`FileEditor`) is not part
of this design. Every count in §1–§2 was taken on `tank` on 2026-09-26 at
commit `de2a53f`, and every count in a milestone's status on the date it
gives, with the commands given next to it. No timing numbers appear here on
purpose: the machine was shared with other build jobs when this was written.
## The problem
@@ -398,7 +391,8 @@ Rejected. It is how one would retrofit a C library that cannot change; we can.
Adopt **(a)**, with a block cache as a required part of every non-local
backend, **(c)** as a cache policy, and the wasm path through the restartable
`NeedBytes` mode. Every milestone keeps `main` green: `cargo test
--workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap
--workspace`, clippy, the conformance gate unchanged (575/697 when this was written; 602/697
after PR #21), and the mmap
fast path within benchmark noise.
**M0 — prerequisites (≈1 week).**
@@ -414,7 +408,8 @@ fast path within benchmark noise.
n children decodes its links O(n) times. Look names up through the index
(above) and let a listing hand out its entries, so the cache has less to
absorb.
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` — link and
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` (merged in
PR #17) — link and
attribute names through the name indexes (`group_v2::resolve_child`,
`attribute::find_attribute_in_file`; creation-order lookups by name do
not exist in the API, so the creation-order index is still only listed),
@@ -438,6 +433,11 @@ fast path within benchmark noise.
(`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so
callers and the other crates don't move yet.
- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.
- *Status 2026-09-26:* done on branch `feat/p3-storage-trait` (merged in
PR #17) for the `Storage` trait and the metadata parsers listed in
`CHANGELOG.md` under "Range reads, milestone M1"; group B-tree v2 lookups,
dense groups and the facade were converted in M2. Both error enums are
`#[non_exhaustive]`; storage failures are `FormatError::Storage`.
**M2 — raw data over the trait (1–2 weeks).**
- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`,
@@ -449,7 +449,8 @@ fast path within benchmark noise.
on other backends (they already return `Option`/`Result`).
- Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data`. As planned,
- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data` (merged in PR
#18). As planned,
with these choices:
- `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the
file handle is shared by its datasets and may be sent across threads).
@@ -487,7 +488,8 @@ fast path within benchmark noise.
§2; the page size for paged files; the first block prefetched on open) and
a request counter exposed for tests and users.
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote` (merged in PR
#18), except the
Python bindings (done 2026-09-27, below), with these choices:
- A new crate, `clawhdf5-remote`, instead of a `remote` feature of
`clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
@@ -526,7 +528,7 @@ fast path within benchmark noise.
requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
6.4 MB: 35 001 object headers spread over the file).
- *Status 2026-09-27, Python bindings:* done on branch
`feat/p3-python-remote-edit`. `clawhdf5.File(url)` and
`feat/p3-python-remote-edit` (merged in PR #19). `clawhdf5.File(url)` and
`File.open_url(url, **options)` (cache and HTTP options) go through
`clawhdf5_remote::storage_for_url`; the default wheel is plain HTTP (no
C), `https`/`s3`/`gcs`/`azure` are build features. The bindings' own
@@ -543,7 +545,8 @@ fast path within benchmark noise.
Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
to a whole download when the server does not answer 206.
- `examples/wasm-viewer`: open by URL.
- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy`, as planned,
- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy` (merged in PR
#19), as planned,
with these choices:
- **NeedBytes, not a Worker.** `clawhdf5_wasm::lazy::LazyStorage` is a
`Storage` over the blocks fetched so far. A call (open, list, read)
@@ -607,7 +610,7 @@ fast path within benchmark noise.
missing blocks: listing 3000 datasets went from 185 passes to 6.
The 32-bit risk below is covered by a Node test that reads data at
3 GiB from a mock server and is refused a 4 GiB file.
- Fewer round trips (2026-09-27, later): the walks descend into every
- Fewer round trips (2026-09-27, later; merged in PR #21): the walks descend into every
child after a failure (not only read the siblings), and parsers
call `Storage::hint` for what they read next (node bodies, object
header chunks, a dense group's heap blocks, a listing's child
@@ -621,7 +624,8 @@ fast path within benchmark noise.
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
blocks past the old end. Needs libhdf5 SWMR semantics research first.
- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader`; design and
- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader` (merged in
PR #19); design and
libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch
above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's
`H5Drefresh`), not per file — a SWMR writer only grows datasets, and the
+9 -4
View File
@@ -1,7 +1,8 @@
# Design: reading files a SWMR writer is still appending to (range-read M5)
Status: design 2026-09-27, implemented on branch `feat/p3-m5-swmr-reader`
(see "Status" at the end). This is milestone M5 of
Status: design 2026-09-27; the reader is implemented and merged (branch
`feat/p3-m5-swmr-reader`, PR #19, `7a8fae0`; see "Status" at the end).
clawhdf5 has no SWMR writer. This is milestone M5 of
[`range-reads.md`](range-reads.md): "`Storage::len()` may grow; add a
refresh". It covers the reader only; clawhdf5 does not write SWMR files.
@@ -165,7 +166,7 @@ writer cannot add them), and `MmapFile`/`LazyFile`.
## Status
Implemented 2026-09-27 on branch `feat/p3-m5-swmr-reader` as designed
above (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the
above, merged to `main` in PR #19 (`7a8fae0`) (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the
same day (h5py 3.16 / HDF5 2.0, `cargo test -p clawhdf5 --test
swmr_interop`, and once with `CLAWHDF5_SWMR_STEPS=20000` in a release
build): no read returned a value the writer had not written at that
@@ -175,4 +176,8 @@ variant of the test with the chunk cache left on in live mode fails it
(stale chunk index / edge chunk), which is why live files do not use it.
Also found: `File::open` of such a file had been failing since the
end-of-file check of 2026-09-26 (item 1; `docs/known-issues.md`).
end-of-file check of 2026-09-26 (item 1; fixed before any release, see
[`docs/known-issues.md`](../known-issues.md#files-a-swmr-writer-had-open-could-not-be-read-past-a-stale-end-of-file)).
Not done (tracked in `docs/known-issues.md`, "Range reads" limits): SWMR
writing, remote SWMR, and live reading through `MmapFile`/`LazyFile`.
+3 -1
View File
@@ -71,4 +71,6 @@ Building blocks, usable as a library today, but not an OpenClaw plugin:
and export rewrites every heading as `##`.
- `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a
TypeScript wrapper. **Not published, not built or tested in CI, and known to
be broken**; see `docs/known-issues.md`.
be broken**; see
[`docs/known-issues.md`](known-issues.md#the-nodejs-package-packagesclawhdf5-node-does-not-work)
(re-checked 2026-09-28: unchanged).