docs: design status notes reflect what is merged
range-reads.md opens with a table of milestones M0-M5 and the PRs that merged them (#17-#21), replacing a header left garbled by earlier merges, and each milestone's status names its PR. swmr.md says the reader is merged (PR #19) and the writer does not exist. openclaw.md links the Node package's known-issues entry. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+40
-36
@@ -1,40 +1,33 @@
|
|||||||
# Design: range reads (reading HDF5 without holding the whole file)
|
# Design: range reads (reading HDF5 without holding the whole file)
|
||||||
|
|
||||||
Status: proposal, 2026-09-26; the plan for Phase 3's largest architectural
|
Status (updated 2026-09-28): **implemented and merged.** Proposed
|
||||||
change. Progress: M0 and M1 are done, and so is M2 (branch
|
2026-09-26 as the plan for Phase 3's largest architectural change; every
|
||||||
`feat/p3-m2-raw-data`): every read path of the format crate works through
|
milestone below is on `main`:
|
||||||
`Storage`, v2 B-trees, dense groups and raw data included, and
|
|
||||||
`File::open_storage` gives the facade's read API over any `Storage` (see
|
|
||||||
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
|
|
||||||
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
|
|
||||||
object stores) and URLs in `h5rs` (see the M3 status below). M5 (SWMR) is
|
|
||||||
done on branch `feat/p3-m5-swmr-reader`, with its own design in
|
|
||||||
[`swmr.md`](swmr.md) (see the M5 status below). M4 (wasm) is
|
|
||||||
next. Every count in §1–§2 was
|
|
||||||
|
|
||||||
object stores) and URLs in `h5rs` (see the M3 status below); the Python
|
| Milestone | What | Merged |
|
||||||
bindings followed on branch `feat/p3-python-remote-edit` (2026-09-27),
|
|---|---|---|
|
||||||
which completes M3. M4 (wasm) is next. Every count in §1–§2 was
|
| M0 | indexed name lookups, checked address conversion | PR #17 (`8f59b2e`) |
|
||||||
|
| M1 | metadata parsers over the `Storage` trait | PR #17 (`8f59b2e`) |
|
||||||
|
| M2 | raw data over `Storage`, `File::open_storage` | PR #18 (`a4c2ace`) |
|
||||||
|
| M3 | `clawhdf5-remote` (block cache, HTTP(S), object stores), URLs in `h5rs`; Python `clawhdf5.File(url)` | PR #18 (`a4c2ace`); Python in PR #19 (`7a8fae0`) |
|
||||||
|
| M4 | wasm `openUrl` through the restartable `NeedBytes` mode | PR #19 (`7a8fae0`); fewer round trips in PR #21 (`9b5803f`) |
|
||||||
|
| M5 | SWMR reader (`File::open_swmr`), design in [`swmr.md`](swmr.md) | PR #19 (`7a8fae0`) |
|
||||||
|
|
||||||
change. Progress: M1, first part (the `Storage` trait and the metadata
|
Each milestone's own *Status* note in §4 records what was built and how it
|
||||||
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
|
differs from the plan. What is still missing is tracked in
|
||||||
group B-tree v2 lookups, dense groups and the facade are not converted yet.
|
[`docs/known-issues.md`](../known-issues.md) ("Range reads", "Remote
|
||||||
Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched
|
files" and "`clawhdf5-wasm`" limits); the main gaps are a paged file's page
|
||||||
|
size as the block size, remote SWMR, and a SWMR writer.
|
||||||
|
|
||||||
object stores) and URLs in `h5rs` (see the M3 status below). M4 is done on
|
|
||||||
branch `feat/p3-m4-wasm-lazy` (2026-09-27): `openUrl` in the browser
|
|
||||||
reader, through the restartable `NeedBytes` mode (see the M4 status
|
|
||||||
below). M5 (SWMR) is not started.
|
|
||||||
Also on 2026-09-26 (branch `feat/p3-editor-coverage`) two reader fixes touched
|
Also on 2026-09-26 (branch `feat/p3-editor-coverage`) two reader fixes touched
|
||||||
converted code without changing the plan: object-header continuation chunks
|
converted code without changing the plan: object-header continuation chunks
|
||||||
are followed without recursion (still one bounded `read_at` per chunk), and
|
are followed without recursion (still one bounded `read_at` per chunk), and
|
||||||
implicit chunk indexes are addressed over the maximum chunk grid (in
|
implicit chunk indexes are addressed over the maximum chunk grid (in
|
||||||
`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps
|
`chunked_read`, an M2 module). The in-place editor (`FileEditor`) is not part
|
||||||
working on the whole file in memory; it is not part of this design. Every
|
of this design. Every count in §1–§2 was taken on `tank` on 2026-09-26 at
|
||||||
count in §1–§2 was taken on `tank` on 2026-09-26 at commit `de2a53f`, and
|
commit `de2a53f`, and every count in a milestone's status on the date it
|
||||||
every count in a milestone's status on the date it gives, with the
|
gives, with the commands given next to it. No timing numbers appear here on
|
||||||
commands given next to it. No timing numbers appear here on purpose: the machine was shared
|
purpose: the machine was shared with other build jobs when this was written.
|
||||||
with other build jobs when this was written.
|
|
||||||
|
|
||||||
## The problem
|
## The problem
|
||||||
|
|
||||||
@@ -398,7 +391,8 @@ Rejected. It is how one would retrofit a C library that cannot change; we can.
|
|||||||
Adopt **(a)**, with a block cache as a required part of every non-local
|
Adopt **(a)**, with a block cache as a required part of every non-local
|
||||||
backend, **(c)** as a cache policy, and the wasm path through the restartable
|
backend, **(c)** as a cache policy, and the wasm path through the restartable
|
||||||
`NeedBytes` mode. Every milestone keeps `main` green: `cargo test
|
`NeedBytes` mode. Every milestone keeps `main` green: `cargo test
|
||||||
--workspace`, clippy, the conformance gate at 575/697 unchanged, and the mmap
|
--workspace`, clippy, the conformance gate unchanged (575/697 when this was written; 602/697
|
||||||
|
after PR #21), and the mmap
|
||||||
fast path within benchmark noise.
|
fast path within benchmark noise.
|
||||||
|
|
||||||
**M0 — prerequisites (≈1 week).**
|
**M0 — prerequisites (≈1 week).**
|
||||||
@@ -414,7 +408,8 @@ fast path within benchmark noise.
|
|||||||
n children decodes its links O(n) times. Look names up through the index
|
n children decodes its links O(n) times. Look names up through the index
|
||||||
(above) and let a listing hand out its entries, so the cache has less to
|
(above) and let a listing hand out its entries, so the cache has less to
|
||||||
absorb.
|
absorb.
|
||||||
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` — link and
|
- *Status 2026-09-26:* done on branch `perf/p3-indexed-lookups` (merged in
|
||||||
|
PR #17) — link and
|
||||||
attribute names through the name indexes (`group_v2::resolve_child`,
|
attribute names through the name indexes (`group_v2::resolve_child`,
|
||||||
`attribute::find_attribute_in_file`; creation-order lookups by name do
|
`attribute::find_attribute_in_file`; creation-order lookups by name do
|
||||||
not exist in the API, so the creation-order index is still only listed),
|
not exist in the API, so the creation-order index is still only listed),
|
||||||
@@ -438,6 +433,11 @@ fast path within benchmark noise.
|
|||||||
(`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so
|
(`fn parse(data: &[u8], ..) { parse_in(data, ..) }`, generic core), so
|
||||||
callers and the other crates don't move yet.
|
callers and the other crates don't move yet.
|
||||||
- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.
|
- Replace the 5 open-ended slices and 38 `len()` checks with bounded reads.
|
||||||
|
- *Status 2026-09-26:* done on branch `feat/p3-storage-trait` (merged in
|
||||||
|
PR #17) for the `Storage` trait and the metadata parsers listed in
|
||||||
|
`CHANGELOG.md` under "Range reads, milestone M1"; group B-tree v2 lookups,
|
||||||
|
dense groups and the facade were converted in M2. Both error enums are
|
||||||
|
`#[non_exhaustive]`; storage failures are `FormatError::Storage`.
|
||||||
|
|
||||||
**M2 — raw data over the trait (1–2 weeks).**
|
**M2 — raw data over the trait (1–2 weeks).**
|
||||||
- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`,
|
- `data_read`, `chunked_read`, `parallel_read`, `partial_read`, `vds`,
|
||||||
@@ -449,7 +449,8 @@ fast path within benchmark noise.
|
|||||||
on other backends (they already return `Option`/`Result`).
|
on other backends (they already return `Option`/`Result`).
|
||||||
- Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
|
- Facade: `File::open_storage(Box<dyn Storage + Send + Sync>)`; `File::open`
|
||||||
keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
|
keeps mmap and `from_bytes` keeps `Vec`, both through `impl Storage for [u8]`.
|
||||||
- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data`. As planned,
|
- *Status 2026-09-26:* done on branch `feat/p3-m2-raw-data` (merged in PR
|
||||||
|
#18). As planned,
|
||||||
with these choices:
|
with these choices:
|
||||||
- `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the
|
- `File::open_storage` takes an `Arc<dyn Storage + Send + Sync>` (the
|
||||||
file handle is shared by its datasets and may be sent across threads).
|
file handle is shared by its datasets and may be sent across threads).
|
||||||
@@ -487,7 +488,8 @@ fast path within benchmark noise.
|
|||||||
§2; the page size for paged files; the first block prefetched on open) and
|
§2; the page size for paged files; the first block prefetched on open) and
|
||||||
a request counter exposed for tests and users.
|
a request counter exposed for tests and users.
|
||||||
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
|
- Python bindings: `clawhdf5.File("s3://…")` / `https://` through it.
|
||||||
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote`, except the
|
- *Status 2026-09-26:* done on branch `feat/p3-m3-remote` (merged in PR
|
||||||
|
#18), except the
|
||||||
Python bindings (done 2026-09-27, below), with these choices:
|
Python bindings (done 2026-09-27, below), with these choices:
|
||||||
- A new crate, `clawhdf5-remote`, instead of a `remote` feature of
|
- A new crate, `clawhdf5-remote`, instead of a `remote` feature of
|
||||||
`clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
|
`clawhdf5-io`: `open_url` returns a `clawhdf5::File`, and `clawhdf5-io`
|
||||||
@@ -526,7 +528,7 @@ fast path within benchmark noise.
|
|||||||
requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
|
requests (§2 predicted 2 blocks of 1 MiB), B in 1, C in 7 (its whole
|
||||||
6.4 MB: 35 001 object headers spread over the file).
|
6.4 MB: 35 001 object headers spread over the file).
|
||||||
- *Status 2026-09-27, Python bindings:* done on branch
|
- *Status 2026-09-27, Python bindings:* done on branch
|
||||||
`feat/p3-python-remote-edit`. `clawhdf5.File(url)` and
|
`feat/p3-python-remote-edit` (merged in PR #19). `clawhdf5.File(url)` and
|
||||||
`File.open_url(url, **options)` (cache and HTTP options) go through
|
`File.open_url(url, **options)` (cache and HTTP options) go through
|
||||||
`clawhdf5_remote::storage_for_url`; the default wheel is plain HTTP (no
|
`clawhdf5_remote::storage_for_url`; the default wheel is plain HTTP (no
|
||||||
C), `https`/`s3`/`gcs`/`azure` are build features. The bindings' own
|
C), `https`/`s3`/`gcs`/`azure` are build features. The bindings' own
|
||||||
@@ -543,7 +545,8 @@ fast path within benchmark noise.
|
|||||||
Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
|
Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
|
||||||
to a whole download when the server does not answer 206.
|
to a whole download when the server does not answer 206.
|
||||||
- `examples/wasm-viewer`: open by URL.
|
- `examples/wasm-viewer`: open by URL.
|
||||||
- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy`, as planned,
|
- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy` (merged in PR
|
||||||
|
#19), as planned,
|
||||||
with these choices:
|
with these choices:
|
||||||
- **NeedBytes, not a Worker.** `clawhdf5_wasm::lazy::LazyStorage` is a
|
- **NeedBytes, not a Worker.** `clawhdf5_wasm::lazy::LazyStorage` is a
|
||||||
`Storage` over the blocks fetched so far. A call (open, list, read)
|
`Storage` over the blocks fetched so far. A call (open, list, read)
|
||||||
@@ -607,7 +610,7 @@ fast path within benchmark noise.
|
|||||||
missing blocks: listing 3000 datasets went from 185 passes to 6.
|
missing blocks: listing 3000 datasets went from 185 passes to 6.
|
||||||
The 32-bit risk below is covered by a Node test that reads data at
|
The 32-bit risk below is covered by a Node test that reads data at
|
||||||
3 GiB from a mock server and is refused a 4 GiB file.
|
3 GiB from a mock server and is refused a 4 GiB file.
|
||||||
- Fewer round trips (2026-09-27, later): the walks descend into every
|
- Fewer round trips (2026-09-27, later; merged in PR #21): the walks descend into every
|
||||||
child after a failure (not only read the siblings), and parsers
|
child after a failure (not only read the siblings), and parsers
|
||||||
call `Storage::hint` for what they read next (node bodies, object
|
call `Storage::hint` for what they read next (node bodies, object
|
||||||
header chunks, a dense group's heap blocks, a listing's child
|
header chunks, a dense group's heap blocks, a listing's child
|
||||||
@@ -621,7 +624,8 @@ fast path within benchmark noise.
|
|||||||
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
|
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
|
||||||
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
|
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
|
||||||
blocks past the old end. Needs libhdf5 SWMR semantics research first.
|
blocks past the old end. Needs libhdf5 SWMR semantics research first.
|
||||||
- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader`; design and
|
- *Status 2026-09-27:* done on branch `feat/p3-m5-swmr-reader` (merged in
|
||||||
|
PR #19); design and
|
||||||
libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch
|
libhdf5 research in [`swmr.md`](swmr.md). Differences from the sketch
|
||||||
above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's
|
above: the refresh is per dataset (`Dataset::refresh`, as libhdf5's
|
||||||
`H5Drefresh`), not per file — a SWMR writer only grows datasets, and the
|
`H5Drefresh`), not per file — a SWMR writer only grows datasets, and the
|
||||||
|
|||||||
+9
-4
@@ -1,7 +1,8 @@
|
|||||||
# Design: reading files a SWMR writer is still appending to (range-read M5)
|
# Design: reading files a SWMR writer is still appending to (range-read M5)
|
||||||
|
|
||||||
Status: design 2026-09-27, implemented on branch `feat/p3-m5-swmr-reader`
|
Status: design 2026-09-27; the reader is implemented and merged (branch
|
||||||
(see "Status" at the end). This is milestone M5 of
|
`feat/p3-m5-swmr-reader`, PR #19, `7a8fae0`; see "Status" at the end).
|
||||||
|
clawhdf5 has no SWMR writer. This is milestone M5 of
|
||||||
[`range-reads.md`](range-reads.md): "`Storage::len()` may grow; add a
|
[`range-reads.md`](range-reads.md): "`Storage::len()` may grow; add a
|
||||||
refresh". It covers the reader only; clawhdf5 does not write SWMR files.
|
refresh". It covers the reader only; clawhdf5 does not write SWMR files.
|
||||||
|
|
||||||
@@ -165,7 +166,7 @@ writer cannot add them), and `MmapFile`/`LazyFile`.
|
|||||||
## Status
|
## Status
|
||||||
|
|
||||||
Implemented 2026-09-27 on branch `feat/p3-m5-swmr-reader` as designed
|
Implemented 2026-09-27 on branch `feat/p3-m5-swmr-reader` as designed
|
||||||
above (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the
|
above, merged to `main` in PR #19 (`7a8fae0`) (`CHANGELOG.md`, "Range reads, milestone M5"). Observed on tank the
|
||||||
same day (h5py 3.16 / HDF5 2.0, `cargo test -p clawhdf5 --test
|
same day (h5py 3.16 / HDF5 2.0, `cargo test -p clawhdf5 --test
|
||||||
swmr_interop`, and once with `CLAWHDF5_SWMR_STEPS=20000` in a release
|
swmr_interop`, and once with `CLAWHDF5_SWMR_STEPS=20000` in a release
|
||||||
build): no read returned a value the writer had not written at that
|
build): no read returned a value the writer had not written at that
|
||||||
@@ -175,4 +176,8 @@ variant of the test with the chunk cache left on in live mode fails it
|
|||||||
(stale chunk index / edge chunk), which is why live files do not use it.
|
(stale chunk index / edge chunk), which is why live files do not use it.
|
||||||
|
|
||||||
Also found: `File::open` of such a file had been failing since the
|
Also found: `File::open` of such a file had been failing since the
|
||||||
end-of-file check of 2026-09-26 (item 1; `docs/known-issues.md`).
|
end-of-file check of 2026-09-26 (item 1; fixed before any release, see
|
||||||
|
[`docs/known-issues.md`](../known-issues.md#files-a-swmr-writer-had-open-could-not-be-read-past-a-stale-end-of-file)).
|
||||||
|
|
||||||
|
Not done (tracked in `docs/known-issues.md`, "Range reads" limits): SWMR
|
||||||
|
writing, remote SWMR, and live reading through `MmapFile`/`LazyFile`.
|
||||||
|
|||||||
+3
-1
@@ -71,4 +71,6 @@ Building blocks, usable as a library today, but not an OpenClaw plugin:
|
|||||||
and export rewrites every heading as `##`.
|
and export rewrites every heading as `##`.
|
||||||
- `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a
|
- `crates/clawhdf5-napi` and `packages/clawhdf5-node` — Node bindings and a
|
||||||
TypeScript wrapper. **Not published, not built or tested in CI, and known to
|
TypeScript wrapper. **Not published, not built or tested in CI, and known to
|
||||||
be broken**; see `docs/known-issues.md`.
|
be broken**; see
|
||||||
|
[`docs/known-issues.md`](known-issues.md#the-nodejs-package-packagesclawhdf5-node-does-not-work)
|
||||||
|
(re-checked 2026-09-28: unchanged).
|
||||||
|
|||||||
Reference in New Issue
Block a user