docs: range-read M4 (openUrl in the browser): changelog, limits, design status

CHANGELOG (Unreleased), known-issues (the browser can open URLs; the
limits of openUrl: round trips per wave of misses, a call holds what it
reads, CORS and validator visibility, the download fallback), the M4
status in docs/design/range-reads.md (why NeedBytes rather than a
Worker, why not clawhdf5-remote's BlockCache, what was tested; the
status paragraph at the top lost a garbled duplicate), the viewer's
README (API, options, how it works, tests; the size table is marked as
predating openUrl), CLAUDE.md and the README crate list.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 06:47:29 -05:00
co-authored by Claude Opus 5.5
parent b8c85f7627
commit f825a89e23
7 changed files with 245 additions and 49 deletions
+49
View File
@@ -2,6 +2,55 @@
## Unreleased
### Range reads, milestone M4: remote files in the browser (2026-09-27)
- **`openUrl(url, opts)`** in `clawhdf5-wasm` opens an HDF5/NetCDF-4 file
on a web server without downloading it: the returned `RemoteFile` has
`H5File`'s methods (`kind`, `list`, `info`, `attrs`, `attrErrors`,
`read`, `readHyperslab`), each returning a promise, and `stats()`
(requests, bytes fetched, file size). Only the byte ranges a call needs
are fetched, with `fetch` and `Range` headers, in 1 MiB blocks
(`blockSize`) kept in a 64 MiB cache (`cacheSize`). `open(bytes)` is
unchanged.
- **How, on the browser's main thread:** the design's restartable
"NeedBytes" mode (`clawhdf5_wasm::lazy::LazyStorage`). A call runs as a
pass over the blocks fetched so far; a read that misses records the
missing blocks and fails, the pass's result is dropped (even if a parser
caught the error and carried on), the blocks are fetched and the pass is
re-run. No Web Worker and no synchronous XHR (what h5wasm's lazy files
need). No block is evicted while a call is in flight, so every call ends;
the cache is trimmed between calls, raw-data blocks before metadata.
`clawhdf5-remote`'s `BlockCache` is not reused: it fetches by blocking,
and it evicts, and does not keep large reads, during a read, where a
restartable pass needs every block it read to stay until it finishes.
- **Every answer is checked** (`js/remote.js`): a `206` with exactly the
bytes asked for (`Content-Range`, when the page can see it, and the body
length), and the same `ETag`/`Last-Modified` and length as at open, or
the call fails: never data from another file or offset. A server that
answers `200` to a `Range` request is downloaded whole (up to
`maxDownload`, 512 MiB) unless `fallback: "error"`. Other options:
`headers`, `credentials`, `parallel` (6 requests at a time), `fetch`.
- **The viewer** (`examples/wasm-viewer`) has a URL box, opens
`?file=<url>` lazily, and shows the requests and bytes fetched.
- Counted on tank, 2026-09-27 (`WASM_BIG_MB=200 bash
examples/wasm-viewer/test/run.sh`, requests as `test/serve.py` counted
them): in a 200 MB h5py file, listing the root, reading two small
datasets, a group's attributes, the large dataset's shape and a
10-value window of it (25 million values) took 5 requests and 6 MiB.
With
`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus`, the 622 corpus files
up to 16 MiB that open read over HTTP as from bytes: the same listings,
attributes and values (datasets up to 2^20 values), and an error
wherever bytes give one. Natively, `cargo test -p clawhdf5-wasm --test
lazy` with the same variable compares 656 files (up to 64 MiB, error
messages included, against the facade's range-storage path).
- `Reader::open_storage` (the wasm crate's core over any `Storage`), and
variable-length strings resolve through the file's storage rather than
`File::as_bytes`.
- The package is larger: the facade's `Storage` read path is now
reachable from JavaScript (it was compiled out before), and the promise
glue and `remote.js` add JavaScript. Not measured for the docs yet (the
build machine was shared); the viewer README's size table predates M4.
### Range reads, milestone M3: remote files (2026-09-26)
- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
a `clawhdf5::File` (through `File::open_storage`) that reads the file by