feat(wasm): examples/wasm-viewer, an HDF5/NetCDF-4 viewer page
Drop a file (or pass ?file=<url>&path=<object>), browse the tree lazily, see a dataset's type, shape, max shape and attributes, and page through its values as 50x12 hyperslab windows (leading dims of 3-D+ data held at chosen indices). build.sh produces pkg/ (not committed) with wasm-bindgen --target web and checks the CLI matches the crate version. test/run.sh builds it and runs test.mjs under Node against the h5py/ netCDF4 fixture (250 checks: every dataset whole and as a strided hyperslab, listings, attributes, error paths, the page's DOM-free helpers), then browser.sh renders the page in headless Chromium for eight objects and checks the DOM. The fixture gains LZ4 (read) and Zstd (refused: links C) datasets and a compound attribute (value null plus its type). ci-test.sh runs it when node and wasm-bindgen exist; the CI container has neither, so CI relies on the native h5py_interop test. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -0,0 +1,2 @@
|
||||
# Generated by build.sh.
|
||||
/pkg/
|
||||
@@ -0,0 +1,97 @@
|
||||
# HDF5 viewer in the browser
|
||||
|
||||
A single page that opens an HDF5 or NetCDF-4 file entirely in the browser
|
||||
with `clawhdf5-wasm` (clawhdf5's reader compiled to WebAssembly): drop a
|
||||
file, browse its groups, and look at a dataset's type, shape, attributes
|
||||
and values (a 50 x 12 window at a time, read as a hyperslab, with the
|
||||
leading dimensions of a 3-D+ dataset held at chosen indices). The file never
|
||||
leaves the page.
|
||||
|
||||
## Build and open
|
||||
|
||||
```bash
|
||||
rustup target add wasm32-unknown-unknown
|
||||
cargo install wasm-bindgen-cli --version 0.2.129 # must equal the crate version; build.sh checks
|
||||
bash examples/wasm-viewer/build.sh # writes examples/wasm-viewer/pkg/ (not committed)
|
||||
python3 -m http.server -d examples/wasm-viewer 8000 # wasm cannot load from file://
|
||||
```
|
||||
|
||||
Then open <http://localhost:8000/>. `?file=<url>&path=<object>` opens a
|
||||
file from a URL (same origin, or one serving CORS headers) and selects an
|
||||
object in it, e.g. `?file=data/run1.h5&path=/results/energy`.
|
||||
|
||||
## JavaScript API
|
||||
|
||||
```js
|
||||
import init, { open } from "./pkg/clawhdf5_wasm.js";
|
||||
await init();
|
||||
const f = open(new Uint8Array(await blob.arrayBuffer()));
|
||||
f.list("/"); // [{ name, kind: "group" | "dataset" }], groups first
|
||||
f.info("/grid"); // { shape, maxshape, dtype, elementShape }
|
||||
f.attrs("/grid"); // [{ name, value, dtype }]
|
||||
f.read("/grid"); // { shape, dtype, data }
|
||||
f.readHyperslab("/grid", [0, 0], [10, 5], [2, 1]); // start, count, stride?, block?
|
||||
f.free();
|
||||
```
|
||||
|
||||
`data` is the typed array of the stored width (`Float64Array`,
|
||||
`Float32Array` also for `f16`, `Int8Array` ... `BigInt64Array`,
|
||||
`BigUint64Array`), or an array of strings for fixed- and variable-length
|
||||
strings and enumerations (h5py booleans read as `"TRUE"`/`"FALSE"`). Array
|
||||
datatypes are flattened, their dimensions appended to `shape`. Anything
|
||||
else throws an `Error` naming the type.
|
||||
|
||||
## Limits
|
||||
|
||||
- Read-only, and the whole file is held in memory (no range requests).
|
||||
- Compound, reference, opaque and variable-length-sequence datasets are
|
||||
refused with an error. Attributes of those types are listed with
|
||||
`value: null` and their `dtype`.
|
||||
- No Zstd or SZIP filters (they link C): such a dataset fails with
|
||||
`unsupported filter`. Deflate, shuffle, Fletcher-32, LZ4, N-Bit and
|
||||
scale-offset are read (within the limits in `docs/known-issues.md`).
|
||||
- Virtual datasets whose sources are in other files, and external links,
|
||||
cannot be followed: there is no file system.
|
||||
|
||||
## Tests
|
||||
|
||||
`test/run.sh` builds the package, writes `fixture.h5` (h5py) and
|
||||
`fixture.nc` (netCDF4) with `test/make_fixture.py`, then:
|
||||
|
||||
- runs `test/test.mjs` under Node: every dataset (whole and a strided
|
||||
hyperslab), listing and attribute is compared with what libhdf5 reads
|
||||
back, error paths are checked, and so are the page's DOM-free helpers
|
||||
(`viewer-lib.js`);
|
||||
- runs `test/browser.sh`: loads the page in headless Chromium with
|
||||
`?file=fixture.h5&path=...` for eight objects and checks the rendered tree,
|
||||
types, shapes, attribute and value cells, and the error shown for an
|
||||
unsupported type. Skipped when no Chromium is found (`CHROME` names one;
|
||||
a Playwright download under `~/.cache/ms-playwright` is picked up).
|
||||
Drag-and-drop and the file picker are not driven by it; they share
|
||||
`load()` with the `?file=` path.
|
||||
|
||||
The same expectations are checked natively, without Node, by
|
||||
`crates/clawhdf5-wasm/tests/h5py_interop.rs`, which is what CI runs (the CI
|
||||
container has no Node or browser).
|
||||
|
||||
## Size
|
||||
|
||||
Measured 2026-09-26 on tank (rustc 1.98.1, wasm-bindgen 0.2.129, gzip 1.14,
|
||||
`gzip -9 -n`), after `bash examples/wasm-viewer/build.sh`:
|
||||
|
||||
| | raw | gzip -9 |
|
||||
|---|---:|---:|
|
||||
| `pkg/clawhdf5_wasm_bg.wasm` (profile `wasm-release`, opt-level `s`) | 627,501 B | 191,639 B |
|
||||
| `pkg/clawhdf5_wasm.js` (wasm-bindgen glue) | 21,826 B | 4,487 B |
|
||||
| same wasm at opt-level `z` | 693,068 B | 192,550 B |
|
||||
| same wasm at opt-level `3` | 544,035 B | 198,803 B |
|
||||
| h5wasm 0.10.3: wasm embedded in `dist/esm/hdf5_util.js` | 3,544,184 B | 907,096 B |
|
||||
| h5wasm 0.10.3: `dist/esm/hdf5_util.js` as shipped | 4,150,134 B | 986,699 B |
|
||||
|
||||
h5wasm figures: `npm pack [email protected]` (npm reports
|
||||
`dist.unpackedSize` 14,731,385 B for the whole package), wasm extracted from
|
||||
the `binaryDecode` literal in `hdf5_util.js`. h5wasm is the whole of libhdf5
|
||||
(writing, every datatype, plugins), so this compares download size, not
|
||||
equal functionality. No `wasm-opt` pass was applied (binaryen is not
|
||||
installed on tank). opt-level `s` is used because it is the smallest
|
||||
compressed.
|
||||
Executable
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the clawhdf5-wasm package the viewer loads, into examples/wasm-viewer/pkg/.
|
||||
#
|
||||
# Needs the wasm32-unknown-unknown target and the wasm-bindgen CLI at the
|
||||
# exact version cargo resolves for the wasm-bindgen crate:
|
||||
# rustup target add wasm32-unknown-unknown
|
||||
# cargo install wasm-bindgen-cli --version <that version>
|
||||
#
|
||||
# Then serve this directory over HTTP (browsers do not load wasm modules from
|
||||
# file://) and open it:
|
||||
# python3 -m http.server -d examples/wasm-viewer 8000
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
ROOT="$(cd "$HERE/../.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# The resolved crate version (Cargo.lock is not committed, so ask cargo).
|
||||
want=$(cargo pkgid wasm-bindgen | sed 's/.*[@#]//')
|
||||
if ! command -v wasm-bindgen >/dev/null; then
|
||||
echo "wasm-bindgen CLI not found: cargo install wasm-bindgen-cli --version $want" >&2
|
||||
exit 1
|
||||
fi
|
||||
have=$(wasm-bindgen --version | awk '{print $2}')
|
||||
if [ "$want" != "$have" ]; then
|
||||
echo "wasm-bindgen CLI is $have but the crate is $want:" >&2
|
||||
echo " cargo install wasm-bindgen-cli --version $want" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cargo build -p clawhdf5-wasm --target wasm32-unknown-unknown --profile wasm-release
|
||||
|
||||
target_dir=$(cargo metadata --format-version 1 --no-deps \
|
||||
| sed -n 's/.*"target_directory":"\([^"]*\)".*/\1/p')
|
||||
wasm="$target_dir/wasm32-unknown-unknown/wasm-release/clawhdf5_wasm.wasm"
|
||||
|
||||
rm -rf "$HERE/pkg"
|
||||
wasm-bindgen --target web --out-dir "$HERE/pkg" "$wasm"
|
||||
echo "built $HERE/pkg ($(wc -c < "$HERE/pkg/clawhdf5_wasm_bg.wasm") bytes of wasm)"
|
||||
@@ -0,0 +1,278 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>HDF5 Viewer</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #fbfbfa; --panel: #ffffff; --ink: #1d1d1b; --muted: #6b6b66;
|
||||
--line: #e2e1dc; --accent: #2f5d8a; --accent-soft: #e8eff6; --bad: #a3312a;
|
||||
--mono: ui-monospace, "SF Mono", Menlo, Consolas, monospace;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--bg: #161615; --panel: #1f1f1d; --ink: #ecebe6; --muted: #9a9993;
|
||||
--line: #34332f; --accent: #8db8e0; --accent-soft: #23303c; --bad: #e0857c;
|
||||
}
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; background: var(--bg); color: var(--ink);
|
||||
font: 14px/1.45 system-ui, -apple-system, "Segoe UI", sans-serif; }
|
||||
header { display: flex; align-items: center; gap: 12px; padding: 12px 16px;
|
||||
border-bottom: 1px solid var(--line); background: var(--panel); flex-wrap: wrap; }
|
||||
header h1 { font-size: 15px; margin: 0; font-weight: 600; }
|
||||
header .file { color: var(--muted); font-family: var(--mono); font-size: 13px; }
|
||||
header label.button { margin-left: auto; }
|
||||
.button { border: 1px solid var(--line); background: var(--panel); color: var(--ink);
|
||||
padding: 5px 10px; border-radius: 6px; cursor: pointer; font: inherit; }
|
||||
.button:hover { border-color: var(--accent); }
|
||||
main { display: grid; grid-template-columns: minmax(200px, 300px) 1fr; min-height: calc(100vh - 50px); }
|
||||
@media (max-width: 700px) { main { grid-template-columns: 1fr; } nav { border-right: 0; border-bottom: 1px solid var(--line); } }
|
||||
nav { border-right: 1px solid var(--line); padding: 8px; overflow: auto; background: var(--panel); }
|
||||
section { padding: 16px; overflow: auto; min-width: 0; }
|
||||
ul.tree { list-style: none; margin: 0; padding-left: 14px; }
|
||||
nav > ul.tree { padding-left: 0; }
|
||||
.node { display: flex; gap: 6px; align-items: baseline; padding: 2px 6px; border-radius: 4px;
|
||||
cursor: pointer; white-space: nowrap; font-family: var(--mono); font-size: 13px; }
|
||||
.node:hover { background: var(--accent-soft); }
|
||||
.node.selected { background: var(--accent-soft); color: var(--accent); }
|
||||
.node .icon { width: 1em; color: var(--muted); flex: none; text-align: center; }
|
||||
.drop { border: 2px dashed var(--line); border-radius: 10px; padding: 48px 16px; text-align: center;
|
||||
color: var(--muted); max-width: 560px; margin: 48px auto; }
|
||||
body.dragging .drop, body.dragging nav { border-color: var(--accent); }
|
||||
h2 { font-size: 16px; margin: 0 0 8px; font-family: var(--mono); word-break: break-all; }
|
||||
h3 { font-size: 12px; text-transform: uppercase; letter-spacing: .06em; color: var(--muted); margin: 20px 0 6px; }
|
||||
dl.meta { display: grid; grid-template-columns: max-content 1fr; gap: 2px 16px; margin: 0; }
|
||||
dl.meta dt { color: var(--muted); }
|
||||
dl.meta dd { margin: 0; font-family: var(--mono); word-break: break-word; }
|
||||
table { border-collapse: collapse; font-family: var(--mono); font-size: 12.5px; }
|
||||
th, td { border: 1px solid var(--line); padding: 3px 8px; text-align: right; white-space: nowrap; }
|
||||
th { background: var(--bg); color: var(--muted); font-weight: 500; }
|
||||
table.attrs td { text-align: left; white-space: normal; word-break: break-word; }
|
||||
.scroll { overflow: auto; max-width: 100%; }
|
||||
.controls { display: flex; flex-wrap: wrap; gap: 8px 14px; align-items: center; margin: 6px 0 10px; }
|
||||
.controls input { width: 6em; font: inherit; padding: 2px 4px; background: var(--panel); color: var(--ink);
|
||||
border: 1px solid var(--line); border-radius: 4px; }
|
||||
.error { color: var(--bad); font-family: var(--mono); white-space: pre-wrap; }
|
||||
.muted { color: var(--muted); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>HDF5 Viewer</h1>
|
||||
<span class="file" id="filename">no file</span>
|
||||
<label class="button">Open file…<input type="file" id="picker" accept=".h5,.hdf5,.he5,.nc,.nc4,.cdf" hidden></label>
|
||||
</header>
|
||||
<main>
|
||||
<nav id="tree"></nav>
|
||||
<section id="detail">
|
||||
<div class="drop">
|
||||
<p><strong>Drop an HDF5 or NetCDF-4 file here</strong>, or use “Open file…”.</p>
|
||||
<p class="muted">The file is read in this page by clawhdf5 compiled to WebAssembly; it is not uploaded anywhere.</p>
|
||||
<p class="muted" id="version"></p>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
<script type="module">
|
||||
import init, { open, version } from "./pkg/clawhdf5_wasm.js";
|
||||
import { joinPath, formatValue, viewWindow, toRows, perElement } from "./viewer-lib.js";
|
||||
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const el = (tag, props = {}, ...kids) => {
|
||||
const e = Object.assign(document.createElement(tag), props);
|
||||
e.append(...kids);
|
||||
return e;
|
||||
};
|
||||
|
||||
let file = null;
|
||||
let selectedNode = null;
|
||||
|
||||
await init();
|
||||
$("version").textContent = `clawhdf5 ${version()}`;
|
||||
|
||||
async function load(blob) {
|
||||
const bytes = new Uint8Array(await blob.arrayBuffer());
|
||||
if (file) file.free();
|
||||
file = null;
|
||||
$("filename").textContent = blob.name;
|
||||
$("tree").replaceChildren();
|
||||
try {
|
||||
file = open(bytes);
|
||||
} catch (e) {
|
||||
$("detail").replaceChildren(el("p", { className: "error", textContent: `Cannot open ${blob.name}: ${e.message}` }));
|
||||
return;
|
||||
}
|
||||
const root = el("ul", { className: "tree" });
|
||||
root.append(treeNode("/", "/", "group"));
|
||||
$("tree").append(root);
|
||||
root.querySelector(".node").click();
|
||||
}
|
||||
|
||||
function treeNode(path, name, kind) {
|
||||
const li = el("li");
|
||||
const icon = el("span", { className: "icon", textContent: kind === "group" ? "▸" : "·" });
|
||||
const row = el("div", { className: "node", title: path }, icon, el("span", { textContent: name }));
|
||||
row.dataset.path = path;
|
||||
li.append(row);
|
||||
let children = null;
|
||||
row.addEventListener("click", () => {
|
||||
if (selectedNode) selectedNode.classList.remove("selected");
|
||||
row.classList.add("selected");
|
||||
selectedNode = row;
|
||||
if (kind === "group") {
|
||||
if (children) {
|
||||
children.hidden = !children.hidden;
|
||||
} else {
|
||||
children = el("ul", { className: "tree" });
|
||||
try {
|
||||
for (const c of file.list(path)) children.append(treeNode(joinPath(path, c.name), c.name, c.kind));
|
||||
} catch (e) {
|
||||
children.append(el("li", { className: "error", textContent: e.message }));
|
||||
}
|
||||
li.append(children);
|
||||
}
|
||||
icon.textContent = children.hidden ? "▸" : "▾";
|
||||
}
|
||||
show(path, kind);
|
||||
});
|
||||
return li;
|
||||
}
|
||||
|
||||
function attrsTable(path) {
|
||||
let attrs, errors;
|
||||
try {
|
||||
attrs = file.attrs(path);
|
||||
errors = file.attrErrors(path);
|
||||
} catch (e) {
|
||||
return el("p", { className: "error", textContent: e.message });
|
||||
}
|
||||
if (!attrs.length && !errors.length) return el("p", { className: "muted", textContent: "none" });
|
||||
const t = el("table", { className: "attrs" }, el("tr", {}, el("th", { textContent: "name" }), el("th", { textContent: "value" })));
|
||||
for (const a of attrs) {
|
||||
const v = a.value === null ? el("span", { className: "muted", textContent: `(${a.dtype})` }) : formatValue(a.value);
|
||||
t.append(el("tr", {}, el("td", { textContent: a.name }), el("td", {}, v)));
|
||||
}
|
||||
for (const msg of errors) t.append(el("tr", {}, el("td", { className: "error", colSpan: 2, textContent: msg })));
|
||||
return el("div", { className: "scroll" }, t);
|
||||
}
|
||||
|
||||
function show(path, kind) {
|
||||
const out = [el("h2", { textContent: path })];
|
||||
if (kind === "dataset") {
|
||||
let info;
|
||||
try {
|
||||
info = file.info(path);
|
||||
} catch (e) {
|
||||
$("detail").replaceChildren(...out, el("p", { className: "error", textContent: e.message }));
|
||||
return;
|
||||
}
|
||||
const max = info.maxshape === null ? "—" : `(${info.maxshape.map((d) => d ?? "∞").join(", ")})`;
|
||||
out.push(el("dl", { className: "meta" },
|
||||
el("dt", { textContent: "type" }), el("dd", { textContent: info.dtype }),
|
||||
el("dt", { textContent: "shape" }), el("dd", { textContent: `(${info.shape.join(", ")})` }),
|
||||
el("dt", { textContent: "max shape" }), el("dd", { textContent: max })));
|
||||
out.push(el("h3", { textContent: "Attributes" }), attrsTable(path));
|
||||
out.push(el("h3", { textContent: "Values" }), valuesView(path, info));
|
||||
} else {
|
||||
out.push(el("h3", { textContent: "Attributes" }), attrsTable(path));
|
||||
}
|
||||
$("detail").replaceChildren(...out);
|
||||
}
|
||||
|
||||
function valuesView(path, info) {
|
||||
const shape = info.shape;
|
||||
const per = perElement(info.elementShape);
|
||||
const state = { row: 0, col: 0, rows: 50, cols: 12, fixed: shape.slice(0, Math.max(0, shape.length - 2)).map(() => 0) };
|
||||
const box = el("div");
|
||||
const controls = el("div", { className: "controls" });
|
||||
const body = el("div", { className: "scroll" });
|
||||
box.append(controls, body);
|
||||
|
||||
const num = (label, key, idx) => {
|
||||
const input = el("input", { type: "number", min: 0, value: idx === undefined ? state[key] : state.fixed[idx] });
|
||||
input.addEventListener("change", () => {
|
||||
const v = Math.max(0, Math.floor(Number(input.value) || 0));
|
||||
if (idx === undefined) state[key] = v; else state.fixed[idx] = v;
|
||||
render();
|
||||
});
|
||||
return el("label", {}, `${label} `, input);
|
||||
};
|
||||
if (shape.length >= 1) controls.append(num("row", "row"));
|
||||
if (shape.length >= 2) controls.append(num("column", "col"));
|
||||
state.fixed.forEach((_, i) => controls.append(num(`dim ${i}`, "fixed", i)));
|
||||
|
||||
function render() {
|
||||
const w = viewWindow(shape, state);
|
||||
let res;
|
||||
try {
|
||||
res = w === null ? file.read(path) : file.readHyperslab(path, w.start, w.count);
|
||||
} catch (e) {
|
||||
body.replaceChildren(el("p", { className: "error", textContent: e.message }));
|
||||
return;
|
||||
}
|
||||
if (w === null) {
|
||||
body.replaceChildren(el("pre", { textContent: toRows(res.data, 1, 1, per)[0][0] }));
|
||||
return;
|
||||
}
|
||||
const t = el("table");
|
||||
const head = el("tr", {}, el("th"));
|
||||
for (let c = 0; c < w.cols; c++) head.append(el("th", { textContent: String(w.col + c) }));
|
||||
t.append(head);
|
||||
toRows(res.data, w.rows, w.cols, per).forEach((cells, r) => {
|
||||
const tr = el("tr", {}, el("th", { textContent: String(w.row + r) }));
|
||||
for (const c of cells) tr.append(el("td", { textContent: c }));
|
||||
t.append(tr);
|
||||
});
|
||||
const total = shape.reduce((a, b) => a * b, 1);
|
||||
const note = el("p", { className: "muted", textContent: `showing rows ${w.row}–${w.row + w.rows - 1}` +
|
||||
(shape.length >= 2 ? `, columns ${w.col}–${w.col + w.cols - 1}` : "") + ` of ${total} values` });
|
||||
body.replaceChildren(t, note);
|
||||
}
|
||||
render();
|
||||
return box;
|
||||
}
|
||||
|
||||
// Expand the tree down to `path` and select it.
|
||||
function reveal(path) {
|
||||
const rowFor = (p) => [...document.querySelectorAll(".node")].find((n) => n.dataset.path === p);
|
||||
let cur = "/";
|
||||
for (const part of path.split("/").filter(Boolean)) {
|
||||
const row = rowFor(cur);
|
||||
if (!row) return;
|
||||
const kids = row.parentElement.querySelector(":scope > ul");
|
||||
if (!kids || kids.hidden) row.click();
|
||||
cur = joinPath(cur, part);
|
||||
}
|
||||
const target = rowFor(cur);
|
||||
if (target && target !== selectedNode) target.click();
|
||||
}
|
||||
|
||||
// ?file=<url>&path=<object> opens a file from a URL (same origin, or one
|
||||
// that allows CORS) and selects an object in it.
|
||||
const params = new URLSearchParams(location.search);
|
||||
if (params.get("file")) {
|
||||
const url = params.get("file");
|
||||
try {
|
||||
const resp = await fetch(url);
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const blob = await resp.blob();
|
||||
await load(new File([blob], url.split("/").pop()));
|
||||
if (file && params.get("path")) reveal(params.get("path"));
|
||||
} catch (e) {
|
||||
$("detail").replaceChildren(el("p", { className: "error", textContent: `Cannot fetch ${url}: ${e.message}` }));
|
||||
}
|
||||
}
|
||||
|
||||
$("picker").addEventListener("change", (e) => e.target.files[0] && load(e.target.files[0]));
|
||||
document.addEventListener("dragover", (e) => { e.preventDefault(); document.body.classList.add("dragging"); });
|
||||
document.addEventListener("dragleave", () => document.body.classList.remove("dragging"));
|
||||
document.addEventListener("drop", (e) => {
|
||||
e.preventDefault();
|
||||
document.body.classList.remove("dragging");
|
||||
const f = e.dataTransfer.files[0];
|
||||
if (f) load(f);
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env bash
|
||||
# Load the viewer page in headless Chromium and check what it renders.
|
||||
#
|
||||
# browser.sh FIXTURE_DIR
|
||||
#
|
||||
# FIXTURE_DIR holds fixture.h5 from make_fixture.py; ../pkg must be built.
|
||||
# The page is opened with ?file=fixture.h5&path=<object>, which fetches the
|
||||
# file, builds the tree down to <object> and shows it; the rendered DOM is
|
||||
# dumped and checked for the values libhdf5 reads.
|
||||
#
|
||||
# Browser: $CHROME, else chromium/google-chrome on PATH, else a Playwright
|
||||
# download under ~/.cache/ms-playwright. Exit 3 when none is found.
|
||||
# BROWSER_DEBUG=/some/prefix saves each rendered page as prefix.<path>.html.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
FIX="$(cd "$1" && pwd)"
|
||||
PY="${CLAWHDF5_PYTHON:-python3}"
|
||||
|
||||
chrome="${CHROME:-}"
|
||||
if [ -z "$chrome" ]; then
|
||||
for c in chromium chromium-browser google-chrome chrome-headless-shell; do
|
||||
if command -v "$c" >/dev/null; then chrome="$(command -v "$c")"; break; fi
|
||||
done
|
||||
fi
|
||||
if [ -z "$chrome" ]; then
|
||||
chrome="$(ls -d "$HOME"/.cache/ms-playwright/chromium_headless_shell-*/chrome-headless-shell-linux64/chrome-headless-shell 2>/dev/null | tail -1 || true)"
|
||||
fi
|
||||
if [ -z "$chrome" ] || [ ! -x "$chrome" ]; then
|
||||
echo "no Chromium found (set CHROME)" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
root="$(mktemp -d)"
|
||||
server=""
|
||||
cleanup() {
|
||||
[ -n "$server" ] && kill "$server" 2>/dev/null || true
|
||||
rm -rf "$root"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
ln -s "$HERE/../index.html" "$HERE/../viewer-lib.js" "$HERE/../pkg" "$FIX/fixture.h5" "$root/"
|
||||
|
||||
port=$("$PY" -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])')
|
||||
"$PY" -m http.server --bind 127.0.0.1 --directory "$root" "$port" >/dev/null 2>&1 &
|
||||
server=$!
|
||||
for _ in $(seq 50); do
|
||||
"$PY" -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:$port/index.html')" 2>/dev/null && break
|
||||
sleep 0.1
|
||||
done
|
||||
|
||||
fails=0
|
||||
# A fresh profile per page: a second instance on the same profile fails.
|
||||
render() {
|
||||
local profile
|
||||
profile="$(mktemp -d "$root/profile.XXXXXX")"
|
||||
"$chrome" --headless --no-sandbox --disable-gpu --user-data-dir="$profile" \
|
||||
--virtual-time-budget=20000 \
|
||||
--dump-dom "http://127.0.0.1:$port/index.html?file=fixture.h5&path=$1" 2>/dev/null
|
||||
}
|
||||
# expect PATH TEXT...: every TEXT appears in the page rendered for PATH.
|
||||
expect() {
|
||||
local path="$1" dom
|
||||
shift
|
||||
dom="$(render "$path")"
|
||||
[ -n "${BROWSER_DEBUG:-}" ] && printf "%s\n" "$dom" > "$BROWSER_DEBUG.$(echo "$path" | tr / _).html"
|
||||
for text in "$@"; do
|
||||
if ! grep -qF -- "$text" <<<"$dom"; then
|
||||
echo "FAIL: page for $path lacks: $text" >&2
|
||||
fails=$((fails + 1))
|
||||
fi
|
||||
done
|
||||
echo "rendered $path"
|
||||
}
|
||||
|
||||
# Tree (root expanded; the group row carries its path) and root attributes.
|
||||
expect "/" 'data-path="/sensors"' 'data-path="/grid"' '<td>title</td><td>"wasm fixture"</td>' \
|
||||
'<td>big</td><td>9223372036854775813</td>' '(compound{x: f64, n: i32})'
|
||||
# A chunked, deflated 2-D dataset: type, shape, the first window of values.
|
||||
expect "/grid" '<dd>f64</dd>' '<dd>(6, 10)</dd>' '<th>9</th>' '<td>0.25</td>' '<td>14.75</td>' \
|
||||
'showing rows 0–5, columns 0–9 of 60 values'
|
||||
# Nested path revealed through the tree; big-endian float32.
|
||||
expect "/sensors/temp" 'data-path="/sensors/temp"' '<dd>f32</dd>' '<td>21.5</td>' '<td>22.25</td>'
|
||||
# 64-bit integers stay exact; strings; array datatype cells.
|
||||
expect "/u64" '<td>18446744073709551615</td>'
|
||||
expect "/vlen_str" '<td>"двa"</td>' '<dd>vlen string</dd>'
|
||||
expect "/pairs" '<td>[2, 3]</td>' '<dd>array[2]<i32></dd>'
|
||||
# 3-D: leading dimension held at 0, window over the last two.
|
||||
expect "/cube" '<dd>(2, 5, 6)</dd>' '<td>29</td>' 'dim 0'
|
||||
# Unsupported type: an error, not values.
|
||||
expect "/table" 'class="error"' 'reading compound{x: f64, n: i32} datasets is not supported'
|
||||
|
||||
if [ "$fails" -gt 0 ]; then
|
||||
echo "browser: $fails checks failed" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "browser: all checks passed ($chrome)"
|
||||
@@ -22,6 +22,11 @@ import h5py
|
||||
import netCDF4
|
||||
import numpy as np
|
||||
|
||||
try: # registers the LZ4/Zstd filters with libhdf5; optional
|
||||
import hdf5plugin
|
||||
except ImportError:
|
||||
hdf5plugin = None
|
||||
|
||||
# netCDF4 1.7 trips numpy 2.5's shape-setting deprecation on assignment.
|
||||
warnings.filterwarnings("ignore", category=DeprecationWarning)
|
||||
|
||||
@@ -37,6 +42,8 @@ with h5py.File(h5, "w") as f:
|
||||
f.attrs["scale"] = np.array([0.5, 2.0])
|
||||
f.attrs["big"] = np.uint64(2**63 + 5)
|
||||
f.attrs.create("vlen_note", "héllo", dtype=h5py.string_dtype())
|
||||
# No plain JavaScript form: listed with value null and its type.
|
||||
f.attrs["origin"] = np.array((1.5, 2), dtype=[("x", "<f8"), ("n", "<i4")])
|
||||
f.create_dataset(
|
||||
"grid", data=np.arange(60, dtype="<f8").reshape(6, 10) / 4,
|
||||
chunks=(4, 3), compression="gzip", shuffle=True,
|
||||
@@ -62,6 +69,12 @@ with h5py.File(h5, "w") as f:
|
||||
"cube", data=np.arange(2 * 5 * 6, dtype="<i4").reshape(2, 5, 6),
|
||||
chunks=(1, 2, 3), compression="gzip",
|
||||
)
|
||||
if hdf5plugin is not None:
|
||||
# LZ4 is built into clawhdf5-wasm; Zstd links C and is not.
|
||||
f.create_dataset("lz4", data=np.arange(40, dtype="<i4"), chunks=(10,),
|
||||
**hdf5plugin.LZ4())
|
||||
f.create_dataset("zstd", data=np.arange(40, dtype="<i4"), chunks=(10,),
|
||||
**hdf5plugin.Zstd())
|
||||
comp = np.zeros(2, dtype=[("x", "<f8"), ("n", "<i4")])
|
||||
f.create_dataset("table", data=comp)
|
||||
g = f.create_group("sensors")
|
||||
@@ -123,6 +136,8 @@ def entry(ds, slab=None):
|
||||
|
||||
def attr(v):
|
||||
v = np.asarray(v) if not isinstance(v, (str, bytes)) else v
|
||||
if isinstance(v, np.ndarray) and v.dtype.names:
|
||||
return {"raw": "compound"}
|
||||
if isinstance(v, bytes):
|
||||
return {"string": v.decode()}
|
||||
if isinstance(v, str):
|
||||
@@ -160,6 +175,8 @@ def describe(path):
|
||||
walk(key.rstrip("/") + "/" + n, o)
|
||||
elif obj.dtype.names:
|
||||
expected["errors"][key] = "compound"
|
||||
elif key == "/zstd":
|
||||
expected["errors"][key] = "unsupported filter: 32015"
|
||||
else:
|
||||
expected["datasets"][key] = entry(obj, slab_for(obj))
|
||||
|
||||
|
||||
Executable
+37
@@ -0,0 +1,37 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the wasm package (../build.sh), test it under Node against files
|
||||
# written by h5py and netCDF4 (make_fixture.py), then load the viewer page
|
||||
# in headless Chromium if one is found (browser.sh).
|
||||
#
|
||||
# Needs node, the wasm-bindgen CLI (see ../build.sh) and a Python with h5py,
|
||||
# netCDF4 and numpy: CLAWHDF5_PYTHON names it (default python3). Without that
|
||||
# Python the test is skipped, unless CLAWHDF5_REQUIRE_INTEROP=1.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
PY="${CLAWHDF5_PYTHON:-python3}"
|
||||
|
||||
command -v node >/dev/null || { echo "node not found" >&2; exit 1; }
|
||||
if ! "$PY" -c "import h5py, netCDF4, numpy" >/dev/null 2>&1; then
|
||||
if [ "${CLAWHDF5_REQUIRE_INTEROP:-0}" = "1" ]; then
|
||||
echo "CLAWHDF5_REQUIRE_INTEROP=1 but $PY lacks h5py/netCDF4/numpy" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "SKIP: $PY lacks h5py/netCDF4/numpy"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
bash "$HERE/../build.sh"
|
||||
fix="$(mktemp -d)"
|
||||
trap 'rm -rf "$fix"' EXIT
|
||||
"$PY" "$HERE/make_fixture.py" "$fix"
|
||||
node "$HERE/test.mjs" "$HERE/../pkg" "$fix"
|
||||
|
||||
# The page itself, in headless Chromium when one is available.
|
||||
status=0
|
||||
bash "$HERE/browser.sh" "$fix" || status=$?
|
||||
if [ "$status" = 3 ]; then
|
||||
echo "SKIP: viewer page in a browser (no Chromium; set CHROME)"
|
||||
elif [ "$status" != 0 ]; then
|
||||
exit "$status"
|
||||
fi
|
||||
@@ -0,0 +1,137 @@
|
||||
// Node test of the built wasm package (the exact pkg/ the viewer page loads)
|
||||
// and the viewer's DOM-free helpers. Run by test/run.sh:
|
||||
// node test.mjs PKG_DIR FIXTURE_DIR
|
||||
// FIXTURE_DIR holds fixture.h5, fixture.nc and expected.json from
|
||||
// make_fixture.py (values as libhdf5 reads them back).
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
|
||||
const [pkgDir, fixDir] = process.argv.slice(2);
|
||||
const pkg = await import(pathToFileURL(join(pkgDir, "clawhdf5_wasm.js")));
|
||||
pkg.initSync({ module: readFileSync(join(pkgDir, "clawhdf5_wasm_bg.wasm")) });
|
||||
const lib = await import(pathToFileURL(join(import.meta.dirname, "..", "viewer-lib.js")));
|
||||
|
||||
let checks = 0;
|
||||
const eq = (a, b, msg) => { assert.deepEqual(a, b, msg); checks++; };
|
||||
|
||||
const ARRAY_TYPES = {
|
||||
f32: Float32Array, f64: Float64Array, i8: Int8Array, i16: Int16Array, i32: Int32Array,
|
||||
i64: BigInt64Array, u8: Uint8Array, u16: Uint16Array, u32: Uint32Array, u64: BigUint64Array,
|
||||
strings: Array,
|
||||
};
|
||||
|
||||
function values(kind, data) {
|
||||
const arr = Array.from(data);
|
||||
if (kind === "f32" || kind === "f64" || kind === "strings") return arr;
|
||||
return arr.map(String);
|
||||
}
|
||||
|
||||
function checkAttr(ctx, a, want) {
|
||||
const v = a.value;
|
||||
if ("raw" in want) {
|
||||
eq(v, null, ctx);
|
||||
assert.ok(a.dtype.includes(want.raw), `${ctx}: ${a.dtype}`);
|
||||
return;
|
||||
}
|
||||
eq(a.dtype, null, `${ctx} dtype`);
|
||||
if ("string" in want) return eq(v, want.string, ctx);
|
||||
if ("strings" in want) return eq(v, want.strings, ctx);
|
||||
if ("int" in want) {
|
||||
if (want.scalar) {
|
||||
assert.ok(typeof v === "number" || typeof v === "bigint", ctx);
|
||||
return eq([String(v)], want.int, ctx);
|
||||
}
|
||||
assert.ok(v instanceof BigInt64Array || v instanceof BigUint64Array, ctx);
|
||||
return eq(Array.from(v, String), want.int, ctx);
|
||||
}
|
||||
if ("float" in want) {
|
||||
if (want.scalar) return eq([v], want.float, ctx);
|
||||
assert.ok(v instanceof Float64Array, ctx);
|
||||
return eq(Array.from(v), want.float, ctx);
|
||||
}
|
||||
assert.fail(`${ctx}: unknown expectation ${JSON.stringify(want)}`);
|
||||
}
|
||||
|
||||
const expected = JSON.parse(readFileSync(join(fixDir, "expected.json"), "utf8"));
|
||||
for (const [name, exp] of Object.entries(expected)) {
|
||||
const file = pkg.open(new Uint8Array(readFileSync(join(fixDir, name))));
|
||||
|
||||
for (const [path, want] of Object.entries(exp.lists)) {
|
||||
eq(file.kind(path), "group", `${name}:${path} kind`);
|
||||
const list = file.list(path);
|
||||
for (const [kind, key] of [["group", "groups"], ["dataset", "datasets"]]) {
|
||||
eq(list.filter((c) => c.kind === kind).map((c) => c.name).sort(), want[key], `${name}:${path} ${key}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [path, want] of Object.entries(exp.datasets)) {
|
||||
const ctx = `${name}:${path}`;
|
||||
eq(file.kind(path), "dataset", `${ctx} kind`);
|
||||
const info = file.info(path);
|
||||
eq([...info.shape, ...info.elementShape], want.shape, `${ctx} info shape`);
|
||||
const r = file.read(path);
|
||||
eq(r.shape, want.shape, `${ctx} shape`);
|
||||
eq(r.dtype, info.dtype, `${ctx} dtype`);
|
||||
assert.ok(r.data instanceof ARRAY_TYPES[want.kind], `${ctx}: ${r.data.constructor.name} for ${want.kind}`);
|
||||
eq(values(want.kind, r.data), want.values, ctx);
|
||||
if (want.slab) {
|
||||
const s = want.slab;
|
||||
const part = file.readHyperslab(path, s.start, s.count, s.stride);
|
||||
eq(part.shape, s.shape, `${ctx} slab shape`);
|
||||
eq(values(want.kind, part.data), s.values, `${ctx} slab`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [path, what] of Object.entries(exp.errors)) {
|
||||
assert.throws(() => file.read(path), (e) => e instanceof Error && e.message.includes(what), `${name}:${path}`);
|
||||
checks++;
|
||||
}
|
||||
|
||||
for (const [path, want] of Object.entries(exp.attrs)) {
|
||||
const attrs = file.attrs(path);
|
||||
eq(file.attrErrors(path), [], `${name}:${path} attr errors`);
|
||||
const seen = attrs.filter((a) => !a.name.startsWith("_") && !exp.skip_attrs.includes(a.name));
|
||||
eq(seen.map((a) => a.name).sort(), Object.keys(want).sort(), `${name}:${path} attr names`);
|
||||
for (const a of seen) checkAttr(`${name}:${path}@${a.name}`, a, want[a.name]);
|
||||
}
|
||||
file.free();
|
||||
}
|
||||
|
||||
// Errors reach JavaScript as thrown Errors, never as data.
|
||||
const h5 = pkg.open(new Uint8Array(readFileSync(join(fixDir, "fixture.h5"))));
|
||||
const throwsMsg = (fn, re) => { assert.throws(fn, (e) => e instanceof Error && re.test(e.message)); checks++; };
|
||||
throwsMsg(() => pkg.open(new Uint8Array(64)), /./);
|
||||
throwsMsg(() => h5.read("/nope"), /./);
|
||||
throwsMsg(() => h5.list("/grid"), /not a group/);
|
||||
throwsMsg(() => h5.readHyperslab("/grid", [0], [1]), /dimensions/);
|
||||
throwsMsg(() => h5.readHyperslab("/grid", [5, 0], [2, 1]), /exceeds/);
|
||||
throwsMsg(() => h5.readHyperslab("/grid", [-1, 0], [1, 1]), /non-negative integers/);
|
||||
throwsMsg(() => h5.readHyperslab("/grid", [0.5, 0], [1, 1]), /non-negative integers/);
|
||||
// Info for a dataset with an unlimited dimension (netCDF "time").
|
||||
const nc = pkg.open(new Uint8Array(readFileSync(join(fixDir, "fixture.nc"))));
|
||||
eq(nc.info("/time").maxshape, [null], "unlimited dimension is null");
|
||||
// Big integers stay exact.
|
||||
eq(h5.read("/u64").data[0], 18446744073709551615n, "u64 max");
|
||||
eq(typeof pkg.version(), "string", "version");
|
||||
|
||||
// Viewer helpers.
|
||||
eq(lib.joinPath("/", "a"), "/a", "joinPath root");
|
||||
eq(lib.joinPath("/a", "b"), "/a/b", "joinPath nested");
|
||||
eq(lib.viewWindow([], {}), null, "scalar window");
|
||||
eq(lib.viewWindow([7], { row: 5, rows: 50 }), { start: [5], count: [2], rows: 2, cols: 1, row: 5, col: 0 }, "1-D window");
|
||||
const w = lib.viewWindow([2, 5, 6], { row: 1, col: 4, rows: 3, cols: 5, fixed: [1] });
|
||||
eq(w, { start: [1, 1, 4], count: [1, 3, 2], rows: 3, cols: 2, row: 1, col: 4 }, "3-D window");
|
||||
// The window the page would request reads the same values as a direct slab.
|
||||
const cube = h5.readHyperslab("/cube", w.start, w.count);
|
||||
eq(lib.toRows(cube.data, w.rows, w.cols), [["40", "41"], ["46", "47"], ["52", "53"]], "cube window cells");
|
||||
const pairs = h5.readHyperslab("/pairs", [1], [2]);
|
||||
eq(lib.toRows(pairs.data, 2, 1, lib.perElement([2])), [["[2, 3]"], ["[4, 5]"]], "array-type cells");
|
||||
eq(lib.formatValue(0.1 + 0.2), "0.3", "float formatting");
|
||||
eq(lib.formatValue(2n ** 64n - 1n), "18446744073709551615", "bigint formatting");
|
||||
eq(lib.formatValue("x"), '"x"', "string formatting");
|
||||
h5.free();
|
||||
nc.free();
|
||||
|
||||
console.log(`wasm package: ${checks} checks passed`);
|
||||
@@ -0,0 +1,77 @@
|
||||
// DOM-free helpers for the viewer, so Node can test them (test/test.mjs).
|
||||
|
||||
/** Child path of `name` in the group at `parent`. */
|
||||
export function joinPath(parent, name) {
|
||||
return parent === "/" ? `/${name}` : `${parent}/${name}`;
|
||||
}
|
||||
|
||||
/** One value as display text. */
|
||||
export function formatValue(v) {
|
||||
if (v === null || v === undefined) return "";
|
||||
if (typeof v === "bigint") return v.toString();
|
||||
if (typeof v === "number") {
|
||||
if (Number.isInteger(v)) return String(v);
|
||||
return String(Number(v.toPrecision(7)));
|
||||
}
|
||||
if (typeof v === "string") return JSON.stringify(v);
|
||||
if (ArrayBuffer.isView(v) || Array.isArray(v)) {
|
||||
const items = Array.from(v.slice(0, 16), formatValue);
|
||||
if (v.length > 16) items.push(`… (${v.length} values)`);
|
||||
return `[${items.join(", ")}]`;
|
||||
}
|
||||
return String(v);
|
||||
}
|
||||
|
||||
/**
|
||||
* The window of a dataset to show: a hyperslab over its `shape` with the
|
||||
* last dimension as columns, the one before as rows, and any leading
|
||||
* dimensions held at `fixed` indices. `row`/`col` are the window's top-left
|
||||
* corner. Returns null for a scalar (read it whole).
|
||||
*/
|
||||
export function viewWindow(shape, { row = 0, col = 0, rows = 50, cols = 12, fixed = [] } = {}) {
|
||||
const rank = shape.length;
|
||||
if (rank === 0) return null;
|
||||
const clamp = (x, n) => Math.max(0, Math.min(x, Math.max(0, n - 1)));
|
||||
if (rank === 1) {
|
||||
const r0 = clamp(row, shape[0]);
|
||||
const n = Math.max(0, Math.min(rows, shape[0] - r0));
|
||||
return { start: [r0], count: [n], rows: n, cols: 1, row: r0, col: 0 };
|
||||
}
|
||||
const lead = shape.slice(0, rank - 2).map((n, i) => clamp(fixed[i] ?? 0, n));
|
||||
const nr = shape[rank - 2];
|
||||
const nc = shape[rank - 1];
|
||||
const r0 = clamp(row, nr);
|
||||
const c0 = clamp(col, nc);
|
||||
const r = Math.max(0, Math.min(rows, nr - r0));
|
||||
const c = Math.max(0, Math.min(cols, nc - c0));
|
||||
return {
|
||||
start: [...lead, r0, c0],
|
||||
count: [...lead.map(() => 1), r, c],
|
||||
rows: r,
|
||||
cols: c,
|
||||
row: r0,
|
||||
col: c0,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Split row-major `data` into `rows` x `cols` cells of display text; each
|
||||
* cell holds `per` consecutive values (the elements of an array datatype).
|
||||
*/
|
||||
export function toRows(data, rows, cols, per = 1) {
|
||||
const out = [];
|
||||
for (let r = 0; r < rows; r++) {
|
||||
const row = [];
|
||||
for (let c = 0; c < cols; c++) {
|
||||
const i = (r * cols + c) * per;
|
||||
row.push(per === 1 ? formatValue(data[i]) : formatValue(data.slice(i, i + per)));
|
||||
}
|
||||
out.push(row);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Number of values an array datatype packs into each element. */
|
||||
export function perElement(elementShape) {
|
||||
return elementShape.reduce((a, b) => a * b, 1);
|
||||
}
|
||||
Reference in New Issue
Block a user