docs: range-read M4 (openUrl in the browser): changelog, limits, design status
CHANGELOG (Unreleased), known-issues (the browser can open URLs; the limits of openUrl: round trips per wave of misses, a call holds what it reads, CORS and validator visibility, the download fallback), the M4 status in docs/design/range-reads.md (why NeedBytes rather than a Worker, why not clawhdf5-remote's BlockCache, what was tested; the status paragraph at the top lost a garbled duplicate), the viewer's README (API, options, how it works, tests; the size table is marked as predating openUrl), CLAUDE.md and the README crate list. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+61
-10
@@ -7,20 +7,19 @@ change. Progress: M0 and M1 are done, and so is M2 (branch
|
||||
`File::open_storage` gives the facade's read API over any `Storage` (see
|
||||
`CHANGELOG.md`, "Range reads, milestone M2"). M3 is done on branch
|
||||
`feat/p3-m3-remote`: the `clawhdf5-remote` crate (block cache, HTTP(S),
|
||||
object stores) and URLs in `h5rs` (see the M3 status below). M4 (wasm) is
|
||||
next. Every count in §1–§2 was
|
||||
|
||||
change. Progress: M1, first part (the `Storage` trait and the metadata
|
||||
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
|
||||
group B-tree v2 lookups, dense groups and the facade are not converted yet.
|
||||
Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched
|
||||
object stores) and URLs in `h5rs` (see the M3 status below). M4 is done on
|
||||
branch `feat/p3-m4-wasm-lazy` (2026-09-27): `openUrl` in the browser
|
||||
reader, through the restartable `NeedBytes` mode (see the M4 status
|
||||
below). M5 (SWMR) is not started.
|
||||
Also on 2026-09-26 (branch `feat/p3-editor-coverage`) two reader fixes touched
|
||||
converted code without changing the plan: object-header continuation chunks
|
||||
are followed without recursion (still one bounded `read_at` per chunk), and
|
||||
implicit chunk indexes are addressed over the maximum chunk grid (in
|
||||
`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps
|
||||
working on the whole file in memory; it is not part of this design. Every count below was
|
||||
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
|
||||
next to it. No timing numbers appear here on purpose: the machine was shared
|
||||
working on the whole file in memory; it is not part of this design. Every
|
||||
count in §1–§2 was taken on `tank` on 2026-09-26 at commit `de2a53f`, and
|
||||
every count in a milestone's status on the date it gives, with the
|
||||
commands given next to it. No timing numbers appear here on purpose: the machine was shared
|
||||
with other build jobs when this was written.
|
||||
|
||||
## The problem
|
||||
@@ -519,6 +518,58 @@ fast path within benchmark noise.
|
||||
Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
|
||||
to a whole download when the server does not answer 206.
|
||||
- `examples/wasm-viewer`: open by URL.
|
||||
- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy`, as planned,
|
||||
with these choices:
|
||||
- **NeedBytes, not a Worker.** `clawhdf5_wasm::lazy::LazyStorage` is a
|
||||
`Storage` over the blocks fetched so far. A call (open, list, read)
|
||||
runs as a pass; a read that misses records its blocks and fails with
|
||||
a storage error. The error itself is not the signal: parsers catch
|
||||
errors and carry on (a listing leaves out a link it cannot resolve),
|
||||
so *any* pass that recorded a miss is thrown away, whatever it
|
||||
returned, and re-run once the missing ranges have been fetched
|
||||
(`attempt` → `Step::Need(ranges)` → `supply` → again). A Worker with
|
||||
synchronous XHR would have kept the parsers' single pass, but it
|
||||
needs the page to run the reader off the main thread (a second
|
||||
module, messages for every call and every typed array copied back),
|
||||
since synchronous XHR can return binary data only there; the
|
||||
restartable loop costs a
|
||||
re-parse per wave of misses instead, which is CPU, not network.
|
||||
- **Progress:** no block is evicted while a call is in flight
|
||||
(`LazyStorage::operation`), so each pass that does not finish asks for
|
||||
at least one new block, and a call ends after at most one pass per
|
||||
block it reads. The budget (`cacheSize`, 64 MiB) is applied between
|
||||
calls, raw-data blocks (`read_ranges`, or reads longer than a block)
|
||||
evicted before metadata. The price: a call holds everything it reads
|
||||
until it finishes.
|
||||
- **Not `clawhdf5-remote`'s `BlockCache`.** It fetches through its
|
||||
backend by blocking, evicts during a read, and does not keep reads
|
||||
that miss more than half its budget; a restartable pass needs every
|
||||
block it has read to be there when it is re-run. The block
|
||||
arithmetic and coalescing (runs of consecutive blocks, a one-block
|
||||
hole filled, at most 8 MiB per request) follow it; the cache is
|
||||
about 450 lines (with its documentation) in the wasm crate, with no
|
||||
in-flight tracking or HTTP.
|
||||
- **The HTTP is JavaScript** (`crates/clawhdf5-wasm/js/remote.js`,
|
||||
shipped as a wasm-bindgen snippet): `fetch` with `Range`, six
|
||||
requests at a time, every answer checked (206, `Content-Range` when
|
||||
visible, body length, the ETag/Last-Modified and length of the
|
||||
first answer). A `200` to the first request is kept as the whole
|
||||
file up to `maxDownload` (512 MiB), or refused with
|
||||
`fallback: "error"`. The first block comes with the request that
|
||||
learns the length, as in M3.
|
||||
- Tested (tank, 2026-09-27): natively, `cargo test -p clawhdf5-wasm
|
||||
--test lazy` compares what the viewer shows of each file (kinds,
|
||||
listings, attributes, info, whole reads, hyperslabs) read lazily at
|
||||
512 B to 1 MiB blocks with the in-memory reader and the facade's
|
||||
range-storage path, for built files, the h5py/netCDF4 fixture and,
|
||||
with `CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus`, 656 corpus
|
||||
files. The built package, under Node and headless Chromium
|
||||
(`WASM_BIG_MB=200 bash examples/wasm-viewer/test/run.sh`, against
|
||||
`test/serve.py`, which counts requests): every fixture check over
|
||||
HTTP; in a 200 MB h5py file, listing, two small reads, a group's
|
||||
attributes and a 10-value window of the 25-million-value dataset
|
||||
took 5 requests, 6 MiB; with the corpus variable, 622 files agree
|
||||
with `open(bytes)`.
|
||||
|
||||
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
|
||||
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
|
||||
|
||||
Reference in New Issue
Block a user