docs: design status notes reflect what is merged

range-reads.md opens with a table of milestones M0-M5 and the PRs that
merged them (#17-#21), replacing a header left garbled by earlier
merges, and each milestone's status names its PR. swmr.md says the
reader is merged (PR #19) and the writer does not exist. openclaw.md
links the Node package's known-issues entry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-28 11:05:38 -05:00
co-authored by Claude Opus 5.5
parent 06d8e2ee45
commit ac0020594b
3 changed files with 52 additions and 41 deletions
+40 -36
View File
@@ -1,40 +1,33 @@
# Design: range reads (reading HDF5 without holding the whole file) # Design: range reads (reading HDF5 without holding the whole file)
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural Status (updated 2026-09-28): **implemented and merged.** Proposed
change. Progress: M0 and M1 are done, and so is M2 (branch 2026-09-26 as the plan for Phase 3's largest architectural change; every
`feat/p3-m2-raw-data`): every read path of the format crate works through milestone below is on `main`:
`Storage`, v2 B-trees, dense groups and raw data included, and
`File::open_storage` gives the facade's read API over any `Storage` (see
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
object stores) and URLs in `h5rs` (see the M3 status below). M5 (SWMR) is
done on branch `feat/p3-m5-swmr-reader`, with its own design in
[`swmr.md`](swmr.md) (see the M5 status below). M4 (wasm) is
next. Every count in §1–§2 was
object stores) and URLs in `h5rs` (see the M3 status below); the Python | Milestone | What | Merged |
bindings followed on branch `feat/p3-python-remote-edit` (2026-09-27), |---|---|---|
which completes M3. M4 (wasm) is next. Every count in §1–§2 was | M0 | indexed name lookups, checked address conversion | PR #17 (`8f59b2e`) |
| M1 | metadata parsers over the `Storage` trait | PR #17 (`8f59b2e`) |
| M2 | raw data over `Storage`, `File::open_storage` | PR #18 (`a4c2ace`) |
| M3 | `clawhdf5-remote` (block cache, HTTP(S), object stores), URLs in `h5rs`; Python `clawhdf5.File(url)` | PR #18 (`a4c2ace`); Python in PR #19 (`7a8fae0`) |
| M4 | wasm `openUrl` through the restartable `NeedBytes` mode | PR #19 (`7a8fae0`); fewer round trips in PR #21 (`9b5803f`) |
| M5 | SWMR reader (`File::open_swmr`), design in [`swmr.md`](swmr.md) | PR #19 (`7a8fae0`) |
change. Progress: M1, first part (the `Storage` trait and the metadata Each milestone's own *Status* note in §4 records what was built and how it
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done; differs from the plan. What is still missing is tracked in
group B-tree v2 lookups, dense groups and the facade are not converted yet. [`docs/known-issues.md`](../known-issues.md) ("Range reads", "Remote
Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched files" and "`clawhdf5-wasm`" limits); the main gaps are a paged file's page
size as the block size, remote SWMR, and a SWMR writer.
object stores) and URLs in `h5rs` (see the M3 status below). M4 is done on
branch `feat/p3-m4-wasm-lazy` (2026-09-27): `openUrl` in the browser
reader, through the restartable `NeedBytes` mode (see the M4 status
below). M5 (SWMR) is not started.
Also on 2026-09-26 (branch `feat/p3-editor-coverage`) two reader fixes touched Also on 2026-09-26 (branch `feat/p3-editor-coverage`) two reader fixes touched
converted code without changing the plan: object-header continuation chunks converted code without changing the plan: object-header continuation chunks
are followed without recursion (still one bounded `read_at` per chunk), and are followed without recursion (still one bounded `read_at` per chunk), and
implicit chunk indexes are addressed over the maximum chunk grid (in implicit chunk indexes are addressed over the maximum chunk grid (in
`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps `chunked_read`, an M2 module). The in-place editor (`FileEditor`) is not part
working on the whole file in memory; it is not part of this design. Every of this design. Every count in §1–§2 was taken on `tank` on 2026-09-26 at
count in §1–§2 was taken on `tank` on 2026-09-26 at commit `de2a53f`, and commit `de2a53f`, and every count in a milestone's status on the date it
every count in a milestone's status on the date it gives, with the gives, with the commands given next to it. No timing numbers appear here on
commands given next to it. No timing numbers appear here on purpose: the machine was shared purpose: the machine was shared with other build jobs when this was written.
with other build jobs when this was written.
## The problem ## The problem
@@ -398,7 +391,8 @@ Rejected. It is how one would retrofit a C library that cannot change; we can.
Adopt **(a)**, with a block cache as a required part of every non-local Adopt **(a)**, with a block cache as a required part of every non-local
backend, **(c)** as a cache policy, and the wasm path through the restartable backend, **(c)** as a cache policy, and the wasm path through the restartable
`NeedBytes` mode. Every milestone keeps `main` green: `cargo test `NeedBytes` mode. Every milestone keeps `main` green: `cargo test
--workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap --workspace`, clippy, the conformance gate unchanged (575/697 when this was written; 602/697
after PR #21), and the mmap
fast path within benchmark noise. fast path within benchmark noise.
**M0 — prerequisites (≈1 week).** **M0 — prerequisites (≈1 week).**
@@ -414,7 +408,8 @@ fast path within benchmark noise.
n children decodes its links O(n) times. Look names up through the index n children decodes its links O(n) times. Look names up through the index
(above) and let a listing hand out its entries, so the cache has less to (above) and let a listing hand out its entries, so the cache has less to
absorb. absorb.
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` — link and - *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` (merged in
PR #17) — link and
attribute names through the name indexes (`group_v2::resolve_child`, attribute names through the name indexes (`group_v2::resolve_child`,
`attribute::find_attribute_in_file`; creation-order lookups by name do `attribute::find_attribute_in_file`; creation-order lookups by name do
not exist in the API, so the creation-order index is still only listed), not exist in the API, so the creation-order index is still only listed),
@@ -438,6 +433,11 @@ fast path within benchmark noise.
(`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so (`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so
callers and the other crates don't move yet. callers and the other crates don't move yet.
- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads. - Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.
- *Status 2026-09-26:* done on branch `feat/p3-storage-trait` (merged in
PR #17) for the `Storage` trait and the metadata parsers listed in
`CHANGELOG.md` under "Range reads, milestone M1"; group B-tree v2 lookups,
dense groups and the facade were converted in M2. Both error enums are
`#[non_exhaustive]`; storage failures are `FormatError::Storage`.
**M2 — raw data over the trait (1–2 weeks).** **M2 — raw data over the trait (1–2 weeks).**
- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`, - `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`,
@@ -449,7 +449,8 @@ fast path within benchmark noise.
on other backends (they already return `Option`/`Result`). on other backends (they already return `Option`/`Result`).
- Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open` - Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`. keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data`. As planned, - *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data` (merged in PR
#18). As planned,
with these choices: with these choices:
- `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the - `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the
file handle is shared by its datasets and may be sent across threads). file handle is shared by its datasets and may be sent across threads).
@@ -487,7 +488,8 @@ fast path within benchmark noise.
§2; the page size for paged files; the first block prefetched on open) and §2; the page size for paged files; the first block prefetched on open) and
a request counter exposed for tests and users. a request counter exposed for tests and users.
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it. - Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the - *Status 2026-09-26:* done on branch `feat/p3-m3-remote` (merged in PR
#18), except the
Python bindings (done 2026-09-27, below), with these choices: Python bindings (done 2026-09-27, below), with these choices:
- A new crate, `clawhdf5-remote`, instead of a `remote` feature of - A new crate, `clawhdf5-remote`, instead of a `remote` feature of
`clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io` `clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
@@ -526,7 +528,7 @@ fast path within benchmark noise.
requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
6.4 MB: 35 001 object headers spread over the file). 6.4 MB: 35 001 object headers spread over the file).
- *Status 2026-09-27, Python bindings:* done on branch - *Status 2026-09-27, Python bindings:* done on branch
`feat/p3-python-remote-edit`. `clawhdf5.File(url)` and `feat/p3-python-remote-edit` (merged in PR #19). `clawhdf5.File(url)` and
`File.open_url(url, **options)` (cache and HTTP options) go through `File.open_url(url, **options)` (cache and HTTP options) go through
`clawhdf5_remote::storage_for_url`; the default wheel is plain HTTP (no `clawhdf5_remote::storage_for_url`; the default wheel is plain HTTP (no
C), `https`/`s3`/`gcs`/`azure` are build features. The bindings' own C), `https`/`s3`/`gcs`/`azure` are build features. The bindings' own
@@ -543,7 +545,8 @@ fast path within benchmark noise.
Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
to a whole download when the server does not answer 206. to a whole download when the server does not answer 206.
- `examples/wasm-viewer`: open by URL. - `examples/wasm-viewer`: open by URL.
- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy`, as planned, - *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy` (merged in PR
#19), as planned,
with these choices: with these choices:
- **NeedBytes, not a Worker.** `clawhdf5_wasm::lazy::LazyStorage` is a - **NeedBytes, not a Worker.** `clawhdf5_wasm::lazy::LazyStorage` is a
`Storage` over the blocks fetched so far. A call (open, list, read) `Storage` over the blocks fetched so far. A call (open, list, read)
@@ -607,7 +610,7 @@ fast path within benchmark noise.
missing blocks: listing 3000 datasets went from 185 passes to 6. missing blocks: listing 3000 datasets went from 185 passes to 6.
The 32-bit risk below is covered by a Node test that reads data at The 32-bit risk below is covered by a Node test that reads data at
3 GiB from a mock server and is refused a 4 GiB file. 3 GiB from a mock server and is refused a 4 GiB file.
- Fewer round trips (2026-09-27, later): the walks descend into every - Fewer round trips (2026-09-27, later; merged in PR #21): the walks descend into every
child after a failure (not only read the siblings), and parsers child after a failure (not only read the siblings), and parsers
call `Storage::hint` for what they read next (node bodies, object call `Storage::hint` for what they read next (node bodies, object
header chunks, a dense group's heap blocks, a listing's child header chunks, a dense group's heap blocks, a listing's child
@@ -621,7 +624,8 @@ fast path within benchmark noise.
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow; **M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
blocks past the old end. Needs libhdf5 SWMR semantics research first. blocks past the old end. Needs libhdf5 SWMR semantics research first.
- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader`; design and - *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader` (merged in
PR #19); design and
libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch
above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's
`H5Drefresh`), not per file — a SWMR writer only grows datasets, and the `H5Drefresh`), not per file — a SWMR writer only grows datasets, and the
+9 -4
View File
@@ -1,7 +1,8 @@
# Design: reading files a SWMR writer is still appending to (range-read M5) # Design: reading files a SWMR writer is still appending to (range-read M5)
Status: design 2026-09-27, implemented on branch `feat/p3-m5-swmr-reader` Status: design 2026-09-27; the reader is implemented and merged (branch
(see "Status" at the end). This is milestone M5 of `feat/p3-m5-swmr-reader`, PR #19, `7a8fae0`; see "Status" at the end).
clawhdf5 has no SWMR writer. This is milestone M5 of
[`range-reads.md`](range-reads.md): "`Storage::len()` may grow; add a [`range-reads.md`](range-reads.md): "`Storage::len()` may grow; add a
refresh". It covers the reader only; clawhdf5 does not write SWMR files. refresh". It covers the reader only; clawhdf5 does not write SWMR files.
@@ -165,7 +166,7 @@ writer cannot add them), and `MmapFile`/`LazyFile`.
## Status ## Status
Implemented 2026-09-27 on branch `feat/p3-m5-swmr-reader` as designed Implemented 2026-09-27 on branch `feat/p3-m5-swmr-reader` as designed
above (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the above, merged to `main` in PR #19 (`7a8fae0`) (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the
same day (h5py 3.16 / HDF5 2.0, `cargo test -p clawhdf5 --test same day (h5py 3.16 / HDF5 2.0, `cargo test -p clawhdf5 --test
swmr_interop`, and once with `CLAWHDF5_SWMR_STEPS=20000` in a release swmr_interop`, and once with `CLAWHDF5_SWMR_STEPS=20000` in a release
build): no read returned a value the writer had not written at that build): no read returned a value the writer had not written at that
@@ -175,4 +176,8 @@ variant of the test with the chunk cache left on in live mode fails it
(stale chunk index / edge chunk), which is why live files do not use it. (stale chunk index / edge chunk), which is why live files do not use it.
Also found: `File::open` of such a file had been failing since the Also found: `File::open` of such a file had been failing since the
end-of-file check of 2026-09-26 (item 1; `docs/known-issues.md`). end-of-file check of 2026-09-26 (item 1; fixed before any release, see
[`docs/known-issues.md`](../known-issues.md#files-a-swmr-writer-had-open-could-not-be-read-past-a-stale-end-of-file)).
Not done (tracked in `docs/known-issues.md`, "Range reads" limits): SWMR
writing, remote SWMR, and live reading through `MmapFile`/`LazyFile`.
+3 -1
View File
@@ -71,4 +71,6 @@ Building blocks, usable as a library today, but not an OpenClaw plugin:
and export rewrites every heading as `##`. and export rewrites every heading as `##`.
- `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a - `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a
TypeScript wrapper. **Not published, not built or tested in CI, and known to TypeScript wrapper. **Not published, not built or tested in CI, and known to
be broken**; see `docs/known-issues.md`. be broken**; see
[`docs/known-issues.md`](known-issues.md#the-nodejs-package-packagesclawhdf5-node-does-not-work)
(re-checked 2026-09-28: unchanged).