quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

wasm: openUrl reads remote files by HTTP range requests (range-read M4)

Browse Source 
openUrl(url, opts) returns a RemoteFile with the methods of H5File
(kind, list, info, attrs, attrErrors, read, readHyperslab), each a
promise, and stats(). It runs every call through the restartable
LazyStorage: a pass that misses reports the byte ranges, js/remote.js
fetches them with fetch() and Range headers (six at a time), and the
pass is re-run. This keeps the main thread free without a Worker or
synchronous XHR (h5wasm's lazy files need both), as the design doc
recommends; the cost is re-running a pass per wave of misses.

Every answer is checked: a 206 with exactly the bytes asked for, and
the same ETag/Last-Modified and length as at open, else an error (never
data). A server that ignores Range (200) is downloaded whole, up to
maxDownload (512 MiB), unless fallback: "error". Options: blockSize,
cacheSize, headers, credentials, parallel, fetch.

test/serve.py is a range-capable static server with request counting
(and /norange/ for a server without range support). test.mjs repeats
every fixture check on files opened by URL (1 MiB and 512 B blocks),
checks the request budget on a 200 MB h5py file (list, three small
reads and a window of the big dataset: 5 requests, 6 MiB), the
download fallback, and HTTP errors, changed files and wrong answers;
with CLAWHDF5_WASM_CORPUS every corpus file is compared with open(bytes).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-27 06:44:45 -05:00
co-authored by Claude Opus 5.5
parent 076feb089a
commit 5107583b97
8 changed files with 1209 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/wasm-viewer/viewer-lib.js
+21
View File
@@ -75,3 +75,24 @@ export function toRows(data, rows, cols, per = 1) {
export function perElement(elementShape) {
return elementShape.reduce((a, b) => a * b, 1);
}
/** A byte count for people: `1.5 KiB`, `200 MiB`. */
export function formatBytes(n) {
const units = ["B", "KiB", "MiB", "GiB", "TiB"];
let i = 0;
let v = n;
while (v >= 1024 && i < units.length - 1) {
v /= 1024;
i++;
}
const s = i === 0 ? String(v) : v < 10 ? String(Number(v.toFixed(1))) : String(Math.round(v));
return `${s} ${units[i]}`;
}
/** What reading a remote file has cost, from `RemoteFile.stats()`. */
export function formatStats({ lazy, requests, bytesFetched, size }) {
if (!lazy) return `downloaded whole (${formatBytes(size)}): the server does not support range requests`;
const pct = size > 0 ? (100 * bytesFetched) / size : 0;
return `${requests} request${requests === 1 ? "" : "s"}, ${formatBytes(bytesFetched)} of ` +
`${formatBytes(size)} fetched (${pct.toFixed(1)}%)`;
}