docs: range-read M4 (openUrl in the browser): changelog, limits, design status
CHANGELOG (Unreleased), known-issues (the browser can open URLs; the limits of openUrl: round trips per wave of misses, a call holds what it reads, CORS and validator visibility, the download fallback), the M4 status in docs/design/range-reads.md (why NeedBytes rather than a Worker, why not clawhdf5-remote's BlockCache, what was tested; the status paragraph at the top lost a garbled duplicate), the viewer's README (API, options, how it works, tests; the size table is marked as predating openUrl), CLAUDE.md and the README crate list. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -2,6 +2,55 @@
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Range reads, milestone M4: remote files in the browser (2026-09-27)
|
||||
- **`openUrl(url, opts)`** in `clawhdf5-wasm` opens an HDF5/NetCDF-4 file
|
||||
on a web server without downloading it: the returned `RemoteFile` has
|
||||
`H5File`'s methods (`kind`, `list`, `info`, `attrs`, `attrErrors`,
|
||||
`read`, `readHyperslab`), each returning a promise, and `stats()`
|
||||
(requests, bytes fetched, file size). Only the byte ranges a call needs
|
||||
are fetched, with `fetch` and `Range` headers, in 1 MiB blocks
|
||||
(`blockSize`) kept in a 64 MiB cache (`cacheSize`). `open(bytes)` is
|
||||
unchanged.
|
||||
- **How, on the browser's main thread:** the design's restartable
|
||||
"NeedBytes" mode (`clawhdf5_wasm::lazy::LazyStorage`). A call runs as a
|
||||
pass over the blocks fetched so far; a read that misses records the
|
||||
missing blocks and fails, the pass's result is dropped (even if a parser
|
||||
caught the error and carried on), the blocks are fetched and the pass is
|
||||
re-run. No Web Worker and no synchronous XHR (what h5wasm's lazy files
|
||||
need). No block is evicted while a call is in flight, so every call ends;
|
||||
the cache is trimmed between calls, raw-data blocks before metadata.
|
||||
`clawhdf5-remote`'s `BlockCache` is not reused: it fetches by blocking,
|
||||
and it evicts, and does not keep large reads, during a read, where a
|
||||
restartable pass needs every block it read to stay until it finishes.
|
||||
- **Every answer is checked** (`js/remote.js`): a `206` with exactly the
|
||||
bytes asked for (`Content-Range`, when the page can see it, and the body
|
||||
length), and the same `ETag`/`Last-Modified` and length as at open, or
|
||||
the call fails: never data from another file or offset. A server that
|
||||
answers `200` to a `Range` request is downloaded whole (up to
|
||||
`maxDownload`, 512 MiB) unless `fallback: "error"`. Other options:
|
||||
`headers`, `credentials`, `parallel` (6 requests at a time), `fetch`.
|
||||
- **The viewer** (`examples/wasm-viewer`) has a URL box, opens
|
||||
`?file=<url>` lazily, and shows the requests and bytes fetched.
|
||||
- Counted on tank, 2026-09-27 (`WASM_BIG_MB=200 bash
|
||||
examples/wasm-viewer/test/run.sh`, requests as `test/serve.py` counted
|
||||
them): in a 200 MB h5py file, listing the root, reading two small
|
||||
datasets, a group's attributes, the large dataset's shape and a
|
||||
10-value window of it (25 million values) took 5 requests and 6 MiB.
|
||||
With
|
||||
`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus`, the 622 corpus files
|
||||
up to 16 MiB that open read over HTTP as from bytes: the same listings,
|
||||
attributes and values (datasets up to 2^20 values), and an error
|
||||
wherever bytes give one. Natively, `cargo test -p clawhdf5-wasm --test
|
||||
lazy` with the same variable compares 656 files (up to 64 MiB, error
|
||||
messages included, against the facade's range-storage path).
|
||||
- `Reader::open_storage` (the wasm crate's core over any `Storage`), and
|
||||
variable-length strings resolve through the file's storage rather than
|
||||
`File::as_bytes`.
|
||||
- The package is larger: the facade's `Storage` read path is now
|
||||
reachable from JavaScript (it was compiled out before), and the promise
|
||||
glue and `remote.js` add JavaScript. Not measured for the docs yet (the
|
||||
build machine was shared); the viewer README's size table predates M4.
|
||||
|
||||
### Range reads, milestone M3: remote files (2026-09-26)
|
||||
- **New crate `clawhdf5-remote`.** `open_url("http://host/file.h5")` gives
|
||||
a `clawhdf5::File` (through `File::open_storage`) that reads the file by
|
||||
|
||||
Reference in New Issue
Block a user