docs: openUrl hardening after review (limits, listing passes, CORS tests)
CHANGELOG (M4 section), known-issues (wasm limits: maxFetch, the 1 GiB decode limit, the 4 GiB file limit on wasm32, bodies cut off at their length, listing passes, the cross-origin tests, and a pre-existing nondeterministic error choice on cve-2025-2310.h5 that can fail the native corpus comparison), the viewer README (options, how listing costs, tests) and the M4 status in docs/design/range-reads.md. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -54,11 +54,14 @@ r.free();
|
||||
|
||||
`openUrl(url, opts)` options, all optional: `blockSize` (bytes per block
|
||||
fetched, 512 B to 64 MiB, default 1 MiB), `cacheSize` (bytes of blocks kept
|
||||
between calls, default 64 MiB), `fallback` (`"download"`, the default,
|
||||
reads the whole file when the server ignores `Range`, up to `maxDownload`
|
||||
bytes, default 512 MiB; `"error"` refuses such a server), `headers` and
|
||||
`credentials` (passed to every request), `parallel` (requests in flight,
|
||||
default 6), `fetch` (a `fetch`-compatible function to use).
|
||||
between calls, default 64 MiB), `maxFetch` (bytes one call may fetch, and
|
||||
so the longest single read, up to 1 GiB, default 512 MiB), `fallback`
|
||||
(`"download"`, the default, reads the whole file when the server ignores
|
||||
`Range`, up to `maxDownload` bytes, default 512 MiB, at most 1 GiB;
|
||||
`"error"` refuses such a server), `headers` (a `Headers`, `[name, value]`
|
||||
pairs or an object) and `credentials` (passed to every request),
|
||||
`parallel` (requests in flight, 1 to 1024, default 6; when one fails the
|
||||
others are aborted), `fetch` (a `fetch`-compatible function to use).
|
||||
|
||||
How it works: the reader is synchronous and a page cannot block on the
|
||||
network, so each call runs as a *pass* over the blocks fetched so far. A
|
||||
@@ -66,7 +69,9 @@ pass that needs a block not yet fetched is abandoned, the missing blocks
|
||||
are fetched (in parallel, adjacent blocks in one request), and the pass is
|
||||
run again, until one completes (`docs/design/range-reads.md`, M4). Opening
|
||||
costs one request (the first block, which also gives the file's size);
|
||||
listing a group whose metadata is in blocks already fetched costs none;
|
||||
listing a group whose metadata is in blocks already fetched costs none,
|
||||
and otherwise a round trip per level of the group's index plus one for
|
||||
its children's headers, all fetched together;
|
||||
reading a chunked dataset costs a round trip for its chunk index (a few
|
||||
for a deep one) and one batch of requests for its chunks. Every answer is
|
||||
checked — a `206` with exactly the bytes asked for, from the same file
|
||||
@@ -86,9 +91,12 @@ else throws an `Error` naming the type.
|
||||
`Content-Range` (`Access-Control-Expose-Headers: Content-Range`) or answer
|
||||
`HEAD` with `Content-Length`; if the page cannot see `ETag` or
|
||||
`Last-Modified` either, a file replaced on the server is detected only by
|
||||
a change of length. A call keeps what it reads until it finishes, so a
|
||||
whole read of a large dataset needs its stored bytes in memory; read
|
||||
windows of large datasets. More in `docs/known-issues.md`.
|
||||
a change of length. A call keeps what it reads until it finishes, and
|
||||
may fetch at most `maxFetch`; `read()` of a dataset that would take more
|
||||
than 1 GiB to decode is refused (read windows of large datasets with
|
||||
`readHyperslab`). Files of 4 GiB or more are refused at open (wasm32).
|
||||
Every response body is cut off past the length asked for. More in
|
||||
`docs/known-issues.md`.
|
||||
- Compound, reference, opaque and variable-length-sequence datasets are
|
||||
refused with an error. Attributes of those types are listed with
|
||||
`value: null` and their `dtype`.
|
||||
@@ -104,7 +112,9 @@ else throws an `Error` naming the type.
|
||||
`fixture.nc` (netCDF4) with `test/make_fixture.py`, and `big.h5`, a 200 MB
|
||||
h5py file (`WASM_BIG_MB` sets its size, 0 leaves it out; it goes under
|
||||
`TMPDIR`), serves them with `test/serve.py` (range requests, a request
|
||||
counter, and `/norange/...` for a server without range support), then:
|
||||
counter, `/norange/...` for a server without range support,
|
||||
`/noexpose/...` and `/unexposed/...` for one that does not expose its
|
||||
headers to CORS), then:
|
||||
|
||||
- runs `test/test.mjs` under Node: every dataset (whole and a strided
|
||||
hyperslab), listing and attribute is compared with what libhdf5 reads
|
||||
@@ -114,14 +124,21 @@ counter, and `/norange/...` for a server without range support), then:
|
||||
`big.h5` (listing it and reading small things of it must take at most 8
|
||||
requests and under 5% of the file), the download fallback and its
|
||||
limit, and the errors: HTTP status, a file that changes, a server that
|
||||
sends the wrong bytes or stops honouring `Range`.
|
||||
sends the wrong bytes or stops honouring `Range`. Also the cross-origin
|
||||
path with no exposed headers (`/noexpose/`: length from `HEAD`), the
|
||||
size limits on `make_fixture.py`'s `limits.h5`, `hostile_vl.h5` (a heap
|
||||
collection claiming 2 GiB) and `far.h5` (data at 3 GiB, served by a
|
||||
mock), bodies longer than asked for, `headers` forms, `parallel`, and
|
||||
sibling requests aborted after a failure.
|
||||
`CLAWHDF5_WASM_CORPUS=DIR` also compares every HDF5 file under `DIR` (up
|
||||
to 16 MiB) read by URL with the same file read from bytes;
|
||||
- runs `test/browser.sh`: loads the page in headless Chromium with
|
||||
`?file=fix/fixture.h5&path=...` for eight objects and checks the rendered
|
||||
tree, types, shapes, attribute and value cells, the request counter, and
|
||||
the error shown for an unsupported type; then a server without range
|
||||
support, and `big.h5` (a small dataset and a window of the large one,
|
||||
the error shown for an unsupported type; then the file from another
|
||||
origin (localhost), with CORS exposing `Content-Range` and exposing
|
||||
nothing (`/unexposed/`, where the server must see a `HEAD`), a server
|
||||
without range support, and `big.h5` (a small dataset and a window of the large one,
|
||||
with a single-digit percentage of the file fetched). Skipped when no
|
||||
Chromium is found (`CHROME` names one; a Playwright download under
|
||||
`~/.cache/ms-playwright` is picked up). Drag-and-drop, the file picker
|
||||
|
||||
Reference in New Issue
Block a user