docs: design status notes reflect what is merged
range-reads.md opens with a table of milestones M0-M5 and the PRs that merged them (#17-#21), replacing a header left garbled by earlier merges, and each milestone's status names its PR. swmr.md says the reader is merged (PR #19) and the writer does not exist. openclaw.md links the Node package's known-issues entry. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+40
-36
@@ -1,40 +1,33 @@
|
||||
# Design: range reads (reading HDF5 without holding the whole file)
|
||||
|
||||
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
|
||||
change. Progress: M0 and M1 are done, and so is M2 (branch
|
||||
`feat/p3-m2-raw-data`): every read path of the format crate works through
|
||||
`Storage`, v2 B-trees, dense groups and raw data included, and
|
||||
`File::open_storage` gives the facade's read API over any `Storage` (see
|
||||
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
|
||||
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
|
||||
object stores) and URLs in `h5rs` (see the M3 status below). M5 (SWMR) is
|
||||
done on branch `feat/p3-m5-swmr-reader`, with its own design in
|
||||
[`swmr.md`](swmr.md) (see the M5 status below). M4 (wasm) is
|
||||
next. Every count in §1–§2 was
|
||||
Status (updated 2026-09-28): **implemented and merged.** Proposed
|
||||
2026-09-26 as the plan for Phase 3's largest architectural change; every
|
||||
milestone below is on `main`:
|
||||
|
||||
object stores) and URLs in `h5rs` (see the M3 status below); the Python
|
||||
bindings followed on branch `feat/p3-python-remote-edit` (2026-09-27),
|
||||
which completes M3. M4 (wasm) is next. Every count in §1–§2 was
|
||||
| Milestone | What | Merged |
|
||||
|---|---|---|
|
||||
| M0 | indexed name lookups, checked address conversion | PR #17 (`8f59b2e`) |
|
||||
| M1 | metadata parsers over the `Storage` trait | PR #17 (`8f59b2e`) |
|
||||
| M2 | raw data over `Storage`, `File::open_storage` | PR #18 (`a4c2ace`) |
|
||||
| M3 | `clawhdf5-remote` (block cache, HTTP(S), object stores), URLs in `h5rs`; Python `clawhdf5.File(url)` | PR #18 (`a4c2ace`); Python in PR #19 (`7a8fae0`) |
|
||||
| M4 | wasm `openUrl` through the restartable `NeedBytes` mode | PR #19 (`7a8fae0`); fewer round trips in PR #21 (`9b5803f`) |
|
||||
| M5 | SWMR reader (`File::open_swmr`), design in [`swmr.md`](swmr.md) | PR #19 (`7a8fae0`) |
|
||||
|
||||
change. Progress: M1, first part (the `Storage` trait and the metadata
|
||||
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
|
||||
group B-tree v2 lookups, dense groups and the facade are not converted yet.
|
||||
Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched
|
||||
Each milestone's own *Status* note in §4 records what was built and how it
|
||||
differs from the plan. What is still missing is tracked in
|
||||
[`docs/known-issues.md`](../known-issues.md) ("Range reads", "Remote
|
||||
files" and "`clawhdf5-wasm`" limits); the main gaps are a paged file's page
|
||||
size as the block size, remote SWMR, and a SWMR writer.
|
||||
|
||||
object stores) and URLs in `h5rs` (see the M3 status below). M4 is done on
|
||||
branch `feat/p3-m4-wasm-lazy` (2026-09-27): `openUrl` in the browser
|
||||
reader, through the restartable `NeedBytes` mode (see the M4 status
|
||||
below). M5 (SWMR) is not started.
|
||||
Also on 2026-09-26 (branch `feat/p3-editor-coverage`) two reader fixes touched
|
||||
converted code without changing the plan: object-header continuation chunks
|
||||
are followed without recursion (still one bounded `read_at` per chunk), and
|
||||
implicit chunk indexes are addressed over the maximum chunk grid (in
|
||||
`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps
|
||||
working on the whole file in memory; it is not part of this design. Every
|
||||
count in §1–§2 was taken on `tank` on 2026-09-26 at commit `de2a53f`, and
|
||||
every count in a milestone's status on the date it gives, with the
|
||||
commands given next to it. No timing numbers appear here on purpose: the machine was shared
|
||||
with other build jobs when this was written.
|
||||
`chunked_read`, an M2 module). The in-place editor (`FileEditor`) is not part
|
||||
of this design. Every count in §1–§2 was taken on `tank` on 2026-09-26 at
|
||||
commit `de2a53f`, and every count in a milestone's status on the date it
|
||||
gives, with the commands given next to it. No timing numbers appear here on
|
||||
purpose: the machine was shared with other build jobs when this was written.
|
||||
|
||||
## The problem
|
||||
|
||||
@@ -398,7 +391,8 @@ Rejected. It is how one would retrofit a C library that cannot change; we can.
|
||||
Adopt **(a)**, with a block cache as a required part of every non-local
|
||||
backend, **(c)** as a cache policy, and the wasm path through the restartable
|
||||
`NeedBytes` mode. Every milestone keeps `main` green: `cargo test
|
||||
--workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap
|
||||
--workspace`, clippy, the conformance gate unchanged (575/697 when this was written; 602/697
|
||||
after PR #21), and the mmap
|
||||
fast path within benchmark noise.
|
||||
|
||||
**M0 — prerequisites (≈1 week).**
|
||||
@@ -414,7 +408,8 @@ fast path within benchmark noise.
|
||||
n children decodes its links O(n) times. Look names up through the index
|
||||
(above) and let a listing hand out its entries, so the cache has less to
|
||||
absorb.
|
||||
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` — link and
|
||||
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` (merged in
|
||||
PR #17) — link and
|
||||
attribute names through the name indexes (`group_v2::resolve_child`,
|
||||
`attribute::find_attribute_in_file`; creation-order lookups by name do
|
||||
not exist in the API, so the creation-order index is still only listed),
|
||||
@@ -438,6 +433,11 @@ fast path within benchmark noise.
|
||||
(`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so
|
||||
callers and the other crates don't move yet.
|
||||
- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.
|
||||
- *Status 2026-09-26:* done on branch `feat/p3-storage-trait` (merged in
|
||||
PR #17) for the `Storage` trait and the metadata parsers listed in
|
||||
`CHANGELOG.md` under "Range reads, milestone M1"; group B-tree v2 lookups,
|
||||
dense groups and the facade were converted in M2. Both error enums are
|
||||
`#[non_exhaustive]`; storage failures are `FormatError::Storage`.
|
||||
|
||||
**M2 — raw data over the trait (1–2 weeks).**
|
||||
- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`,
|
||||
@@ -449,7 +449,8 @@ fast path within benchmark noise.
|
||||
on other backends (they already return `Option`/`Result`).
|
||||
- Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
|
||||
keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
|
||||
- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data`. As planned,
|
||||
- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data` (merged in PR
|
||||
#18). As planned,
|
||||
with these choices:
|
||||
- `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the
|
||||
file handle is shared by its datasets and may be sent across threads).
|
||||
@@ -487,7 +488,8 @@ fast path within benchmark noise.
|
||||
§2; the page size for paged files; the first block prefetched on open) and
|
||||
a request counter exposed for tests and users.
|
||||
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
|
||||
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the
|
||||
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote` (merged in PR
|
||||
#18), except the
|
||||
Python bindings (done 2026-09-27, below), with these choices:
|
||||
- A new crate, `clawhdf5-remote`, instead of a `remote` feature of
|
||||
`clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
|
||||
@@ -526,7 +528,7 @@ fast path within benchmark noise.
|
||||
requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
|
||||
6.4 MB: 35 001 object headers spread over the file).
|
||||
- *Status 2026-09-27, Python bindings:* done on branch
|
||||
`feat/p3-python-remote-edit`. `clawhdf5.File(url)` and
|
||||
`feat/p3-python-remote-edit` (merged in PR #19). `clawhdf5.File(url)` and
|
||||
`File.open_url(url, **options)` (cache and HTTP options) go through
|
||||
`clawhdf5_remote::storage_for_url`; the default wheel is plain HTTP (no
|
||||
C), `https`/`s3`/`gcs`/`azure` are build features. The bindings' own
|
||||
@@ -543,7 +545,8 @@ fast path within benchmark noise.
|
||||
Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
|
||||
to a whole download when the server does not answer 206.
|
||||
- `examples/wasm-viewer`: open by URL.
|
||||
- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy`, as planned,
|
||||
- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy` (merged in PR
|
||||
#19), as planned,
|
||||
with these choices:
|
||||
- **NeedBytes, not a Worker.** `clawhdf5_wasm::lazy::LazyStorage` is a
|
||||
`Storage` over the blocks fetched so far. A call (open, list, read)
|
||||
@@ -607,7 +610,7 @@ fast path within benchmark noise.
|
||||
missing blocks: listing 3000 datasets went from 185 passes to 6.
|
||||
The 32-bit risk below is covered by a Node test that reads data at
|
||||
3 GiB from a mock server and is refused a 4 GiB file.
|
||||
- Fewer round trips (2026-09-27, later): the walks descend into every
|
||||
- Fewer round trips (2026-09-27, later; merged in PR #21): the walks descend into every
|
||||
child after a failure (not only read the siblings), and parsers
|
||||
call `Storage::hint` for what they read next (node bodies, object
|
||||
header chunks, a dense group's heap blocks, a listing's child
|
||||
@@ -621,7 +624,8 @@ fast path within benchmark noise.
|
||||
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
|
||||
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
|
||||
blocks past the old end. Needs libhdf5 SWMR semantics research first.
|
||||
- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader`; design and
|
||||
- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader` (merged in
|
||||
PR #19); design and
|
||||
libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch
|
||||
above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's
|
||||
`H5Drefresh`), not per file — a SWMR writer only grows datasets, and the
|
||||
|
||||
Reference in New Issue
Block a user