Merge branch 'feat/p3-m4-wasm-lazy' into feat/p3-wasm-swmr-python
# Conflicts: # CHANGELOG.md # docs/design/range-reads.md # docs/known-issues.md
This commit is contained in:
@@ -20,13 +20,20 @@ change. Progress: M1, first part (the `Storage` trait and the metadata
|
||||
parsers listed in `CHANGELOG.md` under "Range reads, milestone M1") is done;
|
||||
group B-tree v2 lookups, dense groups and the facade are not converted yet.
|
||||
Later the same day (branch `feat/p3-editor-coverage`) two reader fixes touched
|
||||
|
||||
object stores) and URLs in `h5rs` (see the M3 status below). M4 is done on
|
||||
branch `feat/p3-m4-wasm-lazy` (2026-09-27): `openUrl` in the browser
|
||||
reader, through the restartable `NeedBytes` mode (see the M4 status
|
||||
below). M5 (SWMR) is not started.
|
||||
Also on 2026-09-26 (branch `feat/p3-editor-coverage`) two reader fixes touched
|
||||
converted code without changing the plan: object-header continuation chunks
|
||||
are followed without recursion (still one bounded `read_at` per chunk), and
|
||||
implicit chunk indexes are addressed over the maximum chunk grid (in
|
||||
`chunked_read`, an M2 module). The in-place editor (`FileEditor`) keeps
|
||||
working on the whole file in memory; it is not part of this design. Every count below was
|
||||
taken on `tank` on 2026-09-26 at commit `de2a53f`, with the commands given
|
||||
next to it. No timing numbers appear here on purpose: the machine was shared
|
||||
working on the whole file in memory; it is not part of this design. Every
|
||||
count in §1–§2 was taken on `tank` on 2026-09-26 at commit `de2a53f`, and
|
||||
every count in a milestone's status on the date it gives, with the
|
||||
commands given next to it. No timing numbers appear here on purpose: the machine was shared
|
||||
with other build jobs when this was written.
|
||||
|
||||
## The problem
|
||||
@@ -536,6 +543,70 @@ fast path within benchmark noise.
|
||||
Worker, no synchronous XHR — the thing h5wasm's lazy files need). Falls back
|
||||
to a whole download when the server does not answer 206.
|
||||
- `examples/wasm-viewer`: open by URL.
|
||||
- *Status 2026-09-27:* done on branch `feat/p3-m4-wasm-lazy`, as planned,
|
||||
with these choices:
|
||||
- **NeedBytes, not a Worker.** `clawhdf5_wasm::lazy::LazyStorage` is a
|
||||
`Storage` over the blocks fetched so far. A call (open, list, read)
|
||||
runs as a pass; a read that misses records its blocks and fails with
|
||||
a storage error. The error itself is not the signal: parsers catch
|
||||
errors and carry on (a listing leaves out a link it cannot resolve),
|
||||
so *any* pass that recorded a miss is thrown away, whatever it
|
||||
returned, and re-run once the missing ranges have been fetched
|
||||
(`attempt` → `Step::Need(ranges)` → `supply` → again). A Worker with
|
||||
synchronous XHR would have kept the parsers' single pass, but it
|
||||
needs the page to run the reader off the main thread (a second
|
||||
module, messages for every call and every typed array copied back),
|
||||
since synchronous XHR can return binary data only there; the
|
||||
restartable loop costs a
|
||||
re-parse per wave of misses instead, which is CPU, not network.
|
||||
- **Progress:** no block is evicted while a call is in flight
|
||||
(`LazyStorage::operation`), so each pass that does not finish asks for
|
||||
at least one new block, and a call ends after at most one pass per
|
||||
block it reads. The budget (`cacheSize`, 64 MiB) is applied between
|
||||
calls, raw-data blocks (`read_ranges`, or reads longer than a block)
|
||||
evicted before metadata. The price: a call holds everything it reads
|
||||
until it finishes.
|
||||
- **Not `clawhdf5-remote`'s `BlockCache`.** It fetches through its
|
||||
backend by blocking, evicts during a read, and does not keep reads
|
||||
that miss more than half its budget; a restartable pass needs every
|
||||
block it has read to be there when it is re-run. The block
|
||||
arithmetic and coalescing (runs of consecutive blocks, a one-block
|
||||
hole filled, at most 8 MiB per request) follow it; the cache is
|
||||
about 450 lines (with its documentation) in the wasm crate, with no
|
||||
in-flight tracking or HTTP. (A hole already cached is not filled:
|
||||
it would be fetched again.)
|
||||
- **The HTTP is JavaScript** (`crates/clawhdf5-wasm/js/remote.js`,
|
||||
shipped as a wasm-bindgen snippet): `fetch` with `Range`, six
|
||||
requests at a time, every answer checked (206, `Content-Range` when
|
||||
visible, body length, the ETag/Last-Modified and length of the
|
||||
first answer). A `200` to the first request is kept as the whole
|
||||
file up to `maxDownload` (512 MiB), or refused with
|
||||
`fallback: "error"`. The first block comes with the request that
|
||||
learns the length, as in M3.
|
||||
- Tested (tank, 2026-09-27): natively, `cargo test -p clawhdf5-wasm
|
||||
--test lazy` compares what the viewer shows of each file (kinds,
|
||||
listings, attributes, info, whole reads, hyperslabs) read lazily at
|
||||
512 B to 1 MiB blocks with the in-memory reader and the facade's
|
||||
range-storage path, for built files, the h5py/netCDF4 fixture and,
|
||||
with `CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus`, 656 corpus
|
||||
files. The built package, under Node and headless Chromium
|
||||
(`WASM_BIG_MB=200 bash examples/wasm-viewer/test/run.sh`, against
|
||||
`test/serve.py`, which counts requests): every fixture check over
|
||||
HTTP; in a 200 MB h5py file, listing, two small reads, a group's
|
||||
attributes and a 10-value window of the 25-million-value dataset
|
||||
took 5 requests, 6 MiB; with the corpus variable, 622 files agree
|
||||
with `open(bytes)`.
|
||||
- After review (2026-09-27): a call may fetch at most `maxFetch`
|
||||
(512 MiB, at most 1 GiB) and a longer single read is refused before
|
||||
fetching; `read()` refuses a decode over 1 GiB; a file of 4 GiB or
|
||||
more is refused at open on wasm32 (offsets become `usize`); every
|
||||
response body is cut off at the length asked for. Parsers that walk
|
||||
siblings (B-tree v1/v2 collectors, symbol table nodes, dense links,
|
||||
the wasm listing's child headers) read the siblings after the first
|
||||
failure before returning it, so a pass asks for a whole level's
|
||||
missing blocks: listing 3000 datasets went from 185 passes to 6.
|
||||
The 32-bit risk below is covered by a Node test that reads data at
|
||||
3 GiB from a mock server and is refused a 4 GiB file.
|
||||
|
||||
**M5 — SWMR and growth (later, separate design).** `Storage::len()` may grow;
|
||||
add `File::refresh()` that re-reads the superblock/EOF and invalidates cached
|
||||
|
||||
Reference in New Issue
Block a user