h5rs tools, browser reader, libhdf5 header checks, plugin filters, concurrency benchmark #14
@@ -22,6 +22,9 @@ jobs:
|
||||
run: rustup component add rustfmt clippy
|
||||
- name: Install thumbv7em-none-eabihf target
|
||||
run: rustup target add thumbv7em-none-eabihf
|
||||
- name: Install wasm32-unknown-unknown target
|
||||
# ci-test.sh builds the reader and clawhdf5-wasm for the browser.
|
||||
run: rustup target add wasm32-unknown-unknown
|
||||
- name: Install Python interop dependencies
|
||||
# The interop suites used to skip silently when python3/h5py were
|
||||
# missing, so they never ran in CI. Install them and make a missing
|
||||
|
||||
@@ -397,6 +397,32 @@
|
||||
infinity; batches are all or nothing. CLI: `create --float16`. See
|
||||
`BENCHMARKS.md`, "float16 embedding storage".
|
||||
|
||||
### Browser (WebAssembly)
|
||||
- **New crate `clawhdf5-wasm`:** the reader compiled to
|
||||
`wasm32-unknown-unknown` with a wasm-bindgen JavaScript API —
|
||||
`open(bytes)`, `list`, `info`, `attrs`, `read`, `readHyperslab` — returning
|
||||
typed arrays of the stored width (`BigInt64Array` for 64-bit integers),
|
||||
string arrays for strings and enums, and a thrown `Error` for types with no
|
||||
typed-array form (compound, reference, opaque, VL sequences) or filters the
|
||||
build lacks (Zstd, SZIP). Read-only; the file is held in memory.
|
||||
- **`examples/wasm-viewer/`:** a drop-a-file HDF5/NetCDF-4 viewer page (tree,
|
||||
type/shape/attributes, values paged as hyperslabs; `?file=&path=` opens a
|
||||
URL). `build.sh` produces the package; `test/run.sh` checks it under Node
|
||||
(251 checks against values h5py/libhdf5 read back from an h5py- and a
|
||||
netCDF4-written file) and renders the page in headless Chromium. Size,
|
||||
measured 2026-09-26 on tank (`gzip -9 -n`): 627,501 B of wasm, 191,639 B
|
||||
gzipped, plus 21,826 B (4,487 B) of JS glue; h5wasm 0.10.3's embedded wasm
|
||||
is 3,544,184 B (907,096 B) — full libhdf5, so not equal functionality. See
|
||||
`examples/wasm-viewer/README.md`.
|
||||
- The facade's read path already built for `wasm32-unknown-unknown` (nothing
|
||||
needed gating); `ci-test.sh` now builds it (`--no-default-features`) and
|
||||
lints `clawhdf5-wasm` for that target, and CI installs the target. The Node
|
||||
and browser tests run in `ci-test.sh` only where `node` and `wasm-bindgen`
|
||||
exist (not the CI container); CI checks the same expectations natively
|
||||
(`clawhdf5-wasm`'s `h5py_interop` test).
|
||||
- `Dataset::raw_datatype()` (facade) returns the full stored datatype, for
|
||||
decoding `read_selection` bytes with `clawhdf5_format::data_read`.
|
||||
|
||||
### Build
|
||||
- **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
|
||||
`clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
|
||||
|
||||
@@ -5,7 +5,7 @@ Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persist
|
||||
|
||||
## Architecture
|
||||
|
||||
Cargo workspace with 17 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
|
||||
Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
|
||||
|
||||
| Crate | Role |
|
||||
|-------|------|
|
||||
@@ -25,6 +25,7 @@ Cargo workspace with 17 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
| `clawhdf5-tools` | `h5rs`: pure-Rust HDF5 tools — `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` (structural + checksum validator) |
|
||||
| `clawhdf5-napi` | Node.js native addon bindings |
|
||||
| `clawhdf5-py` | PyO3 Python bindings |
|
||||
| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
|
||||
| `clawhdf5-bench` | Benchmark suite |
|
||||
|
||||
## Key Features
|
||||
@@ -150,6 +151,14 @@ Cargo workspace with 17 crates under `crates/` (plus `libaec-sys`, an internal F
|
||||
`MemorySource` for this bookkeeping is inferred from the caller-supplied
|
||||
`source_channel` string (a heuristic, not an authenticated trust boundary).
|
||||
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
|
||||
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
|
||||
no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.
|
||||
`examples/wasm-viewer/test/run.sh` builds the package (needs the
|
||||
`wasm-bindgen` CLI at the crate's exact version) and tests it under Node
|
||||
and headless Chromium (a Playwright download in `~/.cache/ms-playwright`
|
||||
on tank); the CI container has neither, so CI runs the native
|
||||
`clawhdf5-wasm` `h5py_interop` test on the same fixture. Size numbers are
|
||||
in the example's README.
|
||||
- Python and Node.js bindings for cross-language use
|
||||
- NetCDF-4 compatibility for scientific data interop
|
||||
|
||||
|
||||
+10
@@ -17,6 +17,7 @@ members = [
|
||||
"crates/clawhdf5-napi",
|
||||
"crates/clawhdf5-bench",
|
||||
"crates/clawhdf5-tools",
|
||||
"crates/clawhdf5-wasm",
|
||||
"crates/libaec-sys",
|
||||
]
|
||||
resolver = "2"
|
||||
@@ -35,3 +36,12 @@ tempfile = "3"
|
||||
criterion = { version = "0.5", features = ["html_reports"] }
|
||||
half = "2.7"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
|
||||
# The browser build of clawhdf5-wasm (examples/wasm-viewer/build.sh): size
|
||||
# over speed, whole-program optimisation. Native profiles are unaffected.
|
||||
[profile.wasm-release]
|
||||
inherits = "release"
|
||||
opt-level = "s"
|
||||
lto = true
|
||||
codegen-units = 1
|
||||
panic = "abort"
|
||||
|
||||
@@ -587,7 +587,7 @@ let exported = backend.export_markdown("MEMORY.md")?;
|
||||
## Crate Map
|
||||
|
||||
```
|
||||
clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
and benches; plus libaec-sys, an internal FFI bindings
|
||||
crate for the optional szip feature)
|
||||
│
|
||||
@@ -610,7 +610,8 @@ clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
|
||||
│
|
||||
├── Bindings
|
||||
│ ├── clawhdf5-py — Python (PyO3)
|
||||
│ └── clawhdf5-napi — Node.js (napi-rs)
|
||||
│ ├── clawhdf5-napi — Node.js (napi-rs)
|
||||
│ └── clawhdf5-wasm — Browser (WebAssembly, wasm-bindgen; read-only)
|
||||
│
|
||||
└── Tooling
|
||||
└── clawhdf5-bench — Benchmark suite
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
[package]
|
||||
name = "clawhdf5-wasm"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
rust-version.workspace = true
|
||||
description = "Read HDF5 and NetCDF-4 files in the browser: clawhdf5's reader compiled to WebAssembly"
|
||||
license.workspace = true
|
||||
repository.workspace = true
|
||||
keywords = ["hdf5", "netcdf", "wasm", "browser"]
|
||||
categories = ["wasm", "parser-implementations", "science"]
|
||||
# Distributed as the wasm-bindgen package built by examples/wasm-viewer/build.sh.
|
||||
publish = false
|
||||
|
||||
[lib]
|
||||
# cdylib for wasm-bindgen; rlib so the pure-Rust core is tested natively.
|
||||
crate-type = ["cdylib", "rlib"]
|
||||
|
||||
[dependencies]
|
||||
# No mmap (there is no file system) and no `parallel` (no threads on
|
||||
# wasm32-unknown-unknown). lz4 is pure Rust; zstd and szip link C and are
|
||||
# left out, so such datasets fail with a clear filter error.
|
||||
clawhdf5 = { path = "../clawhdf5", version = "2.7.0", default-features = false, features = ["lz4"] }
|
||||
clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
|
||||
# Must match the wasm-bindgen CLI exactly; build.sh checks.
|
||||
wasm-bindgen = "0.2.129"
|
||||
js-sys = "0.3.106"
|
||||
|
||||
[dev-dependencies]
|
||||
serde_json = "1"
|
||||
tempfile = { workspace = true }
|
||||
@@ -0,0 +1,603 @@
|
||||
//! The reader behind the JavaScript API, in plain Rust so it is tested
|
||||
//! natively. The `wasm_bindgen` layer in `lib.rs` only converts these types
|
||||
//! to JavaScript values.
|
||||
//!
|
||||
//! Every read either returns the dataset's values or an error: a datatype
|
||||
//! with no typed-array mapping (compound, reference, opaque, ...) is refused
|
||||
//! with a message naming it, never returned as reinterpreted bytes.
|
||||
|
||||
use clawhdf5::{AttrValue, File, Selection};
|
||||
use clawhdf5_format::data_read;
|
||||
use clawhdf5_format::datatype::{Datatype, DatatypeByteOrder};
|
||||
|
||||
/// Errors are reported to JavaScript as messages.
|
||||
pub type Result<T> = std::result::Result<T, String>;
|
||||
|
||||
fn err(e: impl std::fmt::Display) -> String {
|
||||
e.to_string()
|
||||
}
|
||||
|
||||
/// What a path names.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum Kind {
|
||||
Group,
|
||||
Dataset,
|
||||
}
|
||||
|
||||
impl Kind {
|
||||
pub fn as_str(self) -> &'static str {
|
||||
match self {
|
||||
Kind::Group => "group",
|
||||
Kind::Dataset => "dataset",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One entry of a group listing.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Child {
|
||||
pub name: String,
|
||||
pub kind: Kind,
|
||||
}
|
||||
|
||||
/// A dataset's metadata.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct DatasetInfo {
|
||||
/// Dataspace dimensions (empty for a scalar).
|
||||
pub shape: Vec<u64>,
|
||||
/// Maximum dimensions, `None` per unlimited dimension; `None` overall
|
||||
/// when the dataspace records none.
|
||||
pub maxshape: Option<Vec<Option<u64>>>,
|
||||
/// Human-readable datatype, e.g. `f64`, `i16 (big-endian)`, `string[8]`.
|
||||
pub dtype: String,
|
||||
/// Dimensions of an array datatype's elements, appended to the shape of
|
||||
/// what [`Reader::read`] returns (empty otherwise).
|
||||
pub element_shape: Vec<u64>,
|
||||
}
|
||||
|
||||
/// Decoded values, one variant per JavaScript typed array.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub enum Data {
|
||||
F32(Vec<f32>),
|
||||
F64(Vec<f64>),
|
||||
I8(Vec<i8>),
|
||||
I16(Vec<i16>),
|
||||
I32(Vec<i32>),
|
||||
I64(Vec<i64>),
|
||||
U8(Vec<u8>),
|
||||
U16(Vec<u16>),
|
||||
U32(Vec<u32>),
|
||||
U64(Vec<u64>),
|
||||
/// Fixed- and variable-length strings, and enumeration member names.
|
||||
Strings(Vec<String>),
|
||||
}
|
||||
|
||||
impl Data {
|
||||
pub fn len(&self) -> usize {
|
||||
match self {
|
||||
Data::F32(v) => v.len(),
|
||||
Data::F64(v) => v.len(),
|
||||
Data::I8(v) => v.len(),
|
||||
Data::I16(v) => v.len(),
|
||||
Data::I32(v) => v.len(),
|
||||
Data::I64(v) => v.len(),
|
||||
Data::U8(v) => v.len(),
|
||||
Data::U16(v) => v.len(),
|
||||
Data::U32(v) => v.len(),
|
||||
Data::U64(v) => v.len(),
|
||||
Data::Strings(v) => v.len(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.len() == 0
|
||||
}
|
||||
}
|
||||
|
||||
/// Values in row-major order with their shape.
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct Array {
|
||||
pub shape: Vec<u64>,
|
||||
pub data: Data,
|
||||
}
|
||||
|
||||
/// A regular hyperslab, as in `H5Sselect_hyperslab`. `stride` and `block`
|
||||
/// default to 1 in every dimension.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Hyperslab {
|
||||
pub start: Vec<u64>,
|
||||
pub count: Vec<u64>,
|
||||
pub stride: Option<Vec<u64>>,
|
||||
pub block: Option<Vec<u64>>,
|
||||
}
|
||||
|
||||
/// An attribute: its value, or why it has none.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Attr {
|
||||
pub name: String,
|
||||
pub value: AttrValue,
|
||||
}
|
||||
|
||||
/// An open file, held in memory.
|
||||
pub struct Reader {
|
||||
file: File,
|
||||
}
|
||||
|
||||
impl Reader {
|
||||
/// Parse a file from its bytes (the browser hands over the whole file).
|
||||
pub fn open(bytes: Vec<u8>) -> Result<Self> {
|
||||
Ok(Self {
|
||||
file: File::from_bytes(bytes).map_err(err)?,
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether `path` names a group or a dataset.
|
||||
pub fn kind(&self, path: &str) -> Result<Kind> {
|
||||
match self.file.dataset(path) {
|
||||
Ok(_) => Ok(Kind::Dataset),
|
||||
Err(clawhdf5::Error::NotADataset(_)) => Ok(Kind::Group),
|
||||
Err(e) => Err(err(e)),
|
||||
}
|
||||
}
|
||||
|
||||
/// The groups, then the datasets, in the group at `path` (`/` is the
|
||||
/// root). Soft links are listed as their targets; external and dangling
|
||||
/// links, and named datatypes, are left out.
|
||||
pub fn list(&self, path: &str) -> Result<Vec<Child>> {
|
||||
if self.kind(path)? != Kind::Group {
|
||||
return Err(format!("not a group: {path}"));
|
||||
}
|
||||
let group = self.file.group(path).map_err(err)?;
|
||||
let mut out: Vec<Child> = group
|
||||
.groups()
|
||||
.map_err(err)?
|
||||
.into_iter()
|
||||
.map(|name| Child {
|
||||
name,
|
||||
kind: Kind::Group,
|
||||
})
|
||||
.collect();
|
||||
out.extend(
|
||||
group
|
||||
.datasets()
|
||||
.map_err(err)?
|
||||
.into_iter()
|
||||
.map(|name| Child {
|
||||
name,
|
||||
kind: Kind::Dataset,
|
||||
}),
|
||||
);
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
/// Shape, max shape and datatype of the dataset at `path`.
|
||||
pub fn info(&self, path: &str) -> Result<DatasetInfo> {
|
||||
let ds = self.file.dataset(path).map_err(err)?;
|
||||
let dt = ds.raw_datatype().map_err(err)?;
|
||||
let maxshape = ds.max_dimensions().map_err(err)?.map(|dims| {
|
||||
dims.into_iter()
|
||||
.map(|d| (d != u64::MAX).then_some(d))
|
||||
.collect()
|
||||
});
|
||||
Ok(DatasetInfo {
|
||||
shape: ds.shape().map_err(err)?,
|
||||
maxshape,
|
||||
dtype: describe(&dt),
|
||||
element_shape: element_shape(&dt),
|
||||
})
|
||||
}
|
||||
|
||||
/// The attributes of the group or dataset at `path`, sorted by name, and
|
||||
/// one message per attribute that could not be read at all. An attribute
|
||||
/// whose type has no plain JavaScript form is returned as
|
||||
/// [`AttrValue::Raw`].
|
||||
pub fn attrs(&self, path: &str) -> Result<(Vec<Attr>, Vec<String>)> {
|
||||
let (map, errors) = match self.kind(path)? {
|
||||
Kind::Dataset => self
|
||||
.file
|
||||
.dataset(path)
|
||||
.and_then(|d| d.attrs_with_errors())
|
||||
.map_err(err)?,
|
||||
Kind::Group => self
|
||||
.file
|
||||
.group(path)
|
||||
.and_then(|g| g.attrs_with_errors())
|
||||
.map_err(err)?,
|
||||
};
|
||||
let mut attrs: Vec<Attr> = map
|
||||
.into_iter()
|
||||
.map(|(name, value)| Attr { name, value })
|
||||
.collect();
|
||||
attrs.sort_by(|a, b| a.name.cmp(&b.name));
|
||||
Ok((attrs, errors.into_iter().map(err).collect()))
|
||||
}
|
||||
|
||||
/// Read the dataset at `path`, whole or a hyperslab of it.
|
||||
pub fn read(&self, path: &str, slab: Option<&Hyperslab>) -> Result<Array> {
|
||||
let ds = self.file.dataset(path).map_err(err)?;
|
||||
let dt = ds.raw_datatype().map_err(err)?;
|
||||
let shape = ds.shape().map_err(err)?;
|
||||
let (selection, mut out_shape) = match slab {
|
||||
None => (Selection::All, shape.clone()),
|
||||
Some(h) => hyperslab_selection(h, &shape)?,
|
||||
};
|
||||
let raw = ds.read_selection(&selection).map_err(err)?;
|
||||
let data = self.decode(&raw, &dt)?;
|
||||
out_shape.extend(element_shape(&dt));
|
||||
let expected = out_shape
|
||||
.iter()
|
||||
.try_fold(1u64, |acc, &d| acc.checked_mul(d))
|
||||
.ok_or("selection size overflows")?;
|
||||
if data.len() as u64 != expected {
|
||||
return Err(format!(
|
||||
"read {} values for shape {out_shape:?} ({expected} expected)",
|
||||
data.len()
|
||||
));
|
||||
}
|
||||
Ok(Array {
|
||||
shape: out_shape,
|
||||
data,
|
||||
})
|
||||
}
|
||||
|
||||
fn decode(&self, raw: &[u8], dt: &Datatype) -> Result<Data> {
|
||||
let base = array_base(dt);
|
||||
let is_array = !std::ptr::eq(base, dt);
|
||||
Ok(match base {
|
||||
Datatype::FloatingPoint { size, .. } if *size <= 4 => {
|
||||
Data::F32(data_read::read_as_f32(raw, dt).map_err(err)?)
|
||||
}
|
||||
Datatype::FloatingPoint { .. } => {
|
||||
Data::F64(data_read::read_as_f64(raw, dt).map_err(err)?)
|
||||
}
|
||||
Datatype::FixedPoint { size, signed, .. } => {
|
||||
let signed_ints = || data_read::read_as_i64(raw, dt).map_err(err);
|
||||
let unsigned_ints = || data_read::read_as_u64(raw, dt).map_err(err);
|
||||
match (size, signed) {
|
||||
(1, true) => Data::I8(narrow(signed_ints()?)?),
|
||||
(2, true) => Data::I16(narrow(signed_ints()?)?),
|
||||
(4, true) => Data::I32(narrow(signed_ints()?)?),
|
||||
(_, true) => Data::I64(signed_ints()?),
|
||||
(1, false) => Data::U8(narrow(unsigned_ints()?)?),
|
||||
(2, false) => Data::U16(narrow(unsigned_ints()?)?),
|
||||
(4, false) => Data::U32(narrow(unsigned_ints()?)?),
|
||||
(_, false) => Data::U64(unsigned_ints()?),
|
||||
}
|
||||
}
|
||||
Datatype::String { .. } if !is_array => {
|
||||
Data::Strings(data_read::read_as_strings(raw, dt).map_err(err)?)
|
||||
}
|
||||
Datatype::VariableLength {
|
||||
is_string: true, ..
|
||||
} if !is_array => {
|
||||
let size = dt.type_size() as usize;
|
||||
if size == 0 || !raw.len().is_multiple_of(size) {
|
||||
return Err(format!(
|
||||
"{} bytes is not a whole number of {size}-byte string references",
|
||||
raw.len()
|
||||
));
|
||||
}
|
||||
let sb = self.file.superblock();
|
||||
Data::Strings(
|
||||
clawhdf5_format::vl_data::read_vl_strings(
|
||||
self.file.as_bytes(),
|
||||
raw,
|
||||
(raw.len() / size) as u64,
|
||||
sb.offset_size,
|
||||
sb.length_size,
|
||||
)
|
||||
.map_err(err)?,
|
||||
)
|
||||
}
|
||||
Datatype::Enumeration { .. } if !is_array => {
|
||||
Data::Strings(data_read::read_enum_names(raw, dt).map_err(err)?)
|
||||
}
|
||||
_ => {
|
||||
return Err(format!(
|
||||
"reading {} datasets is not supported",
|
||||
describe(dt)
|
||||
));
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Narrow integers read at 64 bits to the dataset's own width. The source is
|
||||
/// that width, so this cannot fail on correct input; it is checked anyway.
|
||||
fn narrow<S: Copy + std::fmt::Display, T: TryFrom<S>>(v: Vec<S>) -> Result<Vec<T>> {
|
||||
v.into_iter()
|
||||
.map(|x| T::try_from(x).map_err(|_| format!("value {x} out of range")))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Innermost element type of (possibly nested) array datatypes.
|
||||
fn array_base(dt: &Datatype) -> &Datatype {
|
||||
match dt {
|
||||
Datatype::Array { base_type, .. } => array_base(base_type),
|
||||
_ => dt,
|
||||
}
|
||||
}
|
||||
|
||||
fn element_shape(dt: &Datatype) -> Vec<u64> {
|
||||
match dt {
|
||||
Datatype::Array {
|
||||
base_type,
|
||||
dimensions,
|
||||
} => {
|
||||
let mut dims: Vec<u64> = dimensions.iter().map(|&d| u64::from(d)).collect();
|
||||
dims.extend(element_shape(base_type));
|
||||
dims
|
||||
}
|
||||
_ => Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
fn hyperslab_selection(h: &Hyperslab, shape: &[u64]) -> Result<(Selection, Vec<u64>)> {
|
||||
let rank = shape.len();
|
||||
let ones = vec![1u64; rank];
|
||||
let stride = h.stride.clone().unwrap_or_else(|| ones.clone());
|
||||
let block = h.block.clone().unwrap_or(ones);
|
||||
for (what, v) in [
|
||||
("start", &h.start),
|
||||
("count", &h.count),
|
||||
("stride", &stride),
|
||||
("block", &block),
|
||||
] {
|
||||
if v.len() != rank {
|
||||
return Err(format!(
|
||||
"hyperslab {what} has {} dimensions, the dataset has {rank}",
|
||||
v.len()
|
||||
));
|
||||
}
|
||||
}
|
||||
let mut out = Vec::with_capacity(rank);
|
||||
for d in 0..rank {
|
||||
if stride[d] == 0 || block[d] == 0 {
|
||||
return Err(format!("hyperslab stride and block must be >= 1 (dim {d})"));
|
||||
}
|
||||
if h.count[d] > 1 && block[d] > stride[d] {
|
||||
return Err(format!(
|
||||
"hyperslab blocks overlap in dim {d}: block {} > stride {}",
|
||||
block[d], stride[d]
|
||||
));
|
||||
}
|
||||
// Last element selected: start + (count-1)*stride + block - 1.
|
||||
if h.count[d] > 0 {
|
||||
let last = (h.count[d] - 1)
|
||||
.checked_mul(stride[d])
|
||||
.and_then(|x| x.checked_add(h.start[d]))
|
||||
.and_then(|x| x.checked_add(block[d] - 1));
|
||||
match last {
|
||||
Some(l) if l < shape[d] => {}
|
||||
_ => {
|
||||
return Err(format!(
|
||||
"hyperslab exceeds dimension {d} (extent {})",
|
||||
shape[d]
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
out.push(
|
||||
h.count[d]
|
||||
.checked_mul(block[d])
|
||||
.ok_or("selection size overflows")?,
|
||||
);
|
||||
}
|
||||
Ok((
|
||||
Selection::Hyperslab {
|
||||
start: h.start.clone(),
|
||||
stride,
|
||||
count: h.count.clone(),
|
||||
block,
|
||||
},
|
||||
out,
|
||||
))
|
||||
}
|
||||
|
||||
/// A short, human-readable datatype name.
|
||||
pub fn describe(dt: &Datatype) -> String {
|
||||
fn endian(order: &DatatypeByteOrder) -> &'static str {
|
||||
match order {
|
||||
DatatypeByteOrder::BigEndian => " (big-endian)",
|
||||
DatatypeByteOrder::Vax => " (VAX)",
|
||||
_ => "",
|
||||
}
|
||||
}
|
||||
match dt {
|
||||
Datatype::FixedPoint {
|
||||
size,
|
||||
signed,
|
||||
byte_order,
|
||||
..
|
||||
} => format!(
|
||||
"{}{}{}",
|
||||
if *signed { "i" } else { "u" },
|
||||
size * 8,
|
||||
endian(byte_order)
|
||||
),
|
||||
Datatype::FloatingPoint {
|
||||
size, byte_order, ..
|
||||
} => format!("f{}{}", size * 8, endian(byte_order)),
|
||||
Datatype::Time { size, .. } => format!("time{}", size * 8),
|
||||
Datatype::String { size, .. } => format!("string[{size}]"),
|
||||
Datatype::BitField { size, .. } => format!("bitfield{}", size * 8),
|
||||
Datatype::Opaque { size, .. } => format!("opaque[{size}]"),
|
||||
Datatype::Compound { members, .. } => {
|
||||
let fields: Vec<String> = members
|
||||
.iter()
|
||||
.map(|m| format!("{}: {}", m.name, describe(&m.datatype)))
|
||||
.collect();
|
||||
format!("compound{{{}}}", fields.join(", "))
|
||||
}
|
||||
Datatype::Reference { .. } => "reference".to_string(),
|
||||
Datatype::Enumeration {
|
||||
base_type, members, ..
|
||||
} => {
|
||||
let names: Vec<&str> = members.iter().map(|m| m.name.as_str()).collect();
|
||||
format!("enum<{}>{{{}}}", describe(base_type), names.join(", "))
|
||||
}
|
||||
Datatype::VariableLength {
|
||||
is_string: true, ..
|
||||
} => "vlen string".to_string(),
|
||||
Datatype::VariableLength { base_type, .. } => {
|
||||
format!("vlen<{}>", describe(base_type))
|
||||
}
|
||||
Datatype::Array {
|
||||
base_type,
|
||||
dimensions,
|
||||
} => format!("array{dimensions:?}<{}>", describe(base_type)),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use clawhdf5::FileBuilder;
|
||||
|
||||
fn sample() -> Reader {
|
||||
let mut b = FileBuilder::new();
|
||||
b.create_dataset("grid")
|
||||
.with_f64_data(&(0..12).map(f64::from).collect::<Vec<_>>())
|
||||
.with_shape(&[3, 4])
|
||||
.with_chunks(&[2, 2])
|
||||
.with_deflate(4);
|
||||
b.create_dataset("bytes").with_u8_data(&[1, 2, 250]);
|
||||
let mut g = b.create_group("sensors");
|
||||
g.create_dataset("temp").with_f32_data(&[1.5, -2.25]);
|
||||
g.set_attr("location", AttrValue::String("lab".into()));
|
||||
b.add_group(g.finish());
|
||||
b.set_attr("version", AttrValue::I64(3));
|
||||
b.set_attr("scale", AttrValue::F64Array(vec![0.5, 2.0]));
|
||||
Reader::open(b.finish().unwrap()).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lists_groups_then_datasets() {
|
||||
let r = sample();
|
||||
let names: Vec<(String, Kind)> = r
|
||||
.list("/")
|
||||
.unwrap()
|
||||
.into_iter()
|
||||
.map(|c| (c.name, c.kind))
|
||||
.collect();
|
||||
assert_eq!(names[0], ("sensors".to_string(), Kind::Group));
|
||||
let mut ds: Vec<&str> = names[1..].iter().map(|(n, _)| n.as_str()).collect();
|
||||
ds.sort();
|
||||
assert_eq!(ds, ["bytes", "grid"]);
|
||||
assert_eq!(
|
||||
r.list("sensors").unwrap(),
|
||||
vec![Child {
|
||||
name: "temp".into(),
|
||||
kind: Kind::Dataset
|
||||
}]
|
||||
);
|
||||
assert!(r.list("grid").unwrap_err().contains("not a group"));
|
||||
assert!(r.list("missing").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn info_reports_shape_and_dtype() {
|
||||
let r = sample();
|
||||
let i = r.info("grid").unwrap();
|
||||
assert_eq!(i.shape, vec![3, 4]);
|
||||
assert_eq!(i.dtype, "f64");
|
||||
assert!(i.element_shape.is_empty());
|
||||
assert_eq!(r.info("sensors/temp").unwrap().dtype, "f32");
|
||||
assert_eq!(r.kind("/sensors").unwrap(), Kind::Group);
|
||||
assert_eq!(r.kind("/sensors/temp").unwrap(), Kind::Dataset);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reads_whole_and_hyperslab() {
|
||||
let r = sample();
|
||||
let all = r.read("grid", None).unwrap();
|
||||
assert_eq!(all.shape, vec![3, 4]);
|
||||
assert_eq!(all.data, Data::F64((0..12).map(f64::from).collect()));
|
||||
|
||||
let slab = Hyperslab {
|
||||
start: vec![1, 0],
|
||||
count: vec![2, 2],
|
||||
stride: Some(vec![1, 2]),
|
||||
block: None,
|
||||
};
|
||||
let part = r.read("grid", Some(&slab)).unwrap();
|
||||
assert_eq!(part.shape, vec![2, 2]);
|
||||
assert_eq!(part.data, Data::F64(vec![4.0, 6.0, 8.0, 10.0]));
|
||||
|
||||
assert_eq!(
|
||||
r.read("bytes", None).unwrap().data,
|
||||
Data::U8(vec![1, 2, 250])
|
||||
);
|
||||
assert_eq!(
|
||||
r.read("sensors/temp", None).unwrap().data,
|
||||
Data::F32(vec![1.5, -2.25])
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bad_hyperslabs_are_refused() {
|
||||
let r = sample();
|
||||
let mk = |start: Vec<u64>, count: Vec<u64>| Hyperslab {
|
||||
start,
|
||||
count,
|
||||
stride: None,
|
||||
block: None,
|
||||
};
|
||||
assert!(
|
||||
r.read("grid", Some(&mk(vec![0], vec![1])))
|
||||
.unwrap_err()
|
||||
.contains("dimensions")
|
||||
);
|
||||
assert!(
|
||||
r.read("grid", Some(&mk(vec![2, 0], vec![2, 1])))
|
||||
.unwrap_err()
|
||||
.contains("exceeds")
|
||||
);
|
||||
let overlap = Hyperslab {
|
||||
start: vec![0, 0],
|
||||
count: vec![2, 1],
|
||||
stride: Some(vec![1, 1]),
|
||||
block: Some(vec![2, 1]),
|
||||
};
|
||||
assert!(
|
||||
r.read("grid", Some(&overlap))
|
||||
.unwrap_err()
|
||||
.contains("overlap")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn attrs_are_sorted() {
|
||||
let r = sample();
|
||||
let (attrs, errors) = r.attrs("/").unwrap();
|
||||
assert!(errors.is_empty());
|
||||
let names: Vec<&str> = attrs.iter().map(|a| a.name.as_str()).collect();
|
||||
assert_eq!(names, ["scale", "version"]);
|
||||
let (g, _) = r.attrs("sensors").unwrap();
|
||||
assert!(matches!(&g[0].value, AttrValue::String(s) if s == "lab"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compound_is_refused_not_reinterpreted() {
|
||||
use clawhdf5::CompoundTypeBuilder;
|
||||
let ct = CompoundTypeBuilder::new()
|
||||
.f64_field("x")
|
||||
.i32_field("n")
|
||||
.build();
|
||||
let mut rec = Vec::new();
|
||||
rec.extend_from_slice(&1.0f64.to_le_bytes());
|
||||
rec.extend_from_slice(&7i32.to_le_bytes());
|
||||
let mut b = FileBuilder::new();
|
||||
b.create_dataset("table").with_compound_data(ct, rec, 1);
|
||||
let r = Reader::open(b.finish().unwrap()).unwrap();
|
||||
let e = r.read("table", None).unwrap_err();
|
||||
assert!(e.contains("compound{x: f64, n: i32}"), "{e}");
|
||||
assert!(e.contains("not supported"), "{e}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn garbage_is_an_error() {
|
||||
assert!(Reader::open(vec![0u8; 64]).is_err());
|
||||
assert!(Reader::open(Vec::new()).is_err());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,244 @@
|
||||
//! clawhdf5's HDF5 reader for JavaScript, via `wasm-bindgen`.
|
||||
//!
|
||||
//! ```js
|
||||
//! import init, { open } from "./pkg/clawhdf5_wasm.js";
|
||||
//! await init();
|
||||
//! const file = open(new Uint8Array(await blob.arrayBuffer()));
|
||||
//! file.list("/"); // [{ name, kind: "group" | "dataset" }]
|
||||
//! file.info("/x"); // { shape, maxshape, dtype, elementShape }
|
||||
//! file.attrs("/x"); // [{ name, value, dtype }]
|
||||
//! file.read("/x"); // { shape, dtype, data: Float64Array | ... | string[] }
|
||||
//! file.readHyperslab("/x", [0, 0], [10, 10]); // stride, block optional
|
||||
//! file.free();
|
||||
//! ```
|
||||
//!
|
||||
//! Numeric data comes back in the typed array of the stored width
|
||||
//! (`Int16Array` for `i16`, `BigInt64Array` for `i64`, `Float32Array` for
|
||||
//! `f32` and `f16`, ...); strings and enumeration names as arrays of strings.
|
||||
//! Anything else is a thrown `Error` naming the datatype. Only the reader is
|
||||
//! exposed: nothing here writes files.
|
||||
//!
|
||||
//! The logic lives in [`core`], which is plain Rust and tested natively.
|
||||
|
||||
pub mod core;
|
||||
|
||||
use clawhdf5::AttrValue;
|
||||
use js_sys::{Array, Object, Reflect};
|
||||
use wasm_bindgen::prelude::*;
|
||||
|
||||
use crate::core::{Data, Hyperslab, Reader};
|
||||
|
||||
/// JavaScript numbers are exact up to 2^53.
|
||||
const MAX_SAFE_INTEGER: f64 = 9_007_199_254_740_991.0;
|
||||
|
||||
fn js_err(msg: String) -> JsError {
|
||||
JsError::new(&msg)
|
||||
}
|
||||
|
||||
fn set(obj: &Object, key: &str, value: impl Into<JsValue>) {
|
||||
// Defining a property on a fresh plain object cannot fail.
|
||||
Reflect::set(obj, &JsValue::from_str(key), &value.into()).unwrap_throw();
|
||||
}
|
||||
|
||||
fn shape_to_js(shape: &[u64]) -> Array {
|
||||
shape.iter().map(|&d| JsValue::from_f64(d as f64)).collect()
|
||||
}
|
||||
|
||||
fn indices_from_js(what: &str, v: &[f64]) -> Result<Vec<u64>, JsError> {
|
||||
v.iter()
|
||||
.map(|&x| {
|
||||
if x.is_finite() && x >= 0.0 && x.fract() == 0.0 && x <= MAX_SAFE_INTEGER {
|
||||
Ok(x as u64)
|
||||
} else {
|
||||
Err(js_err(format!(
|
||||
"{what} must hold non-negative integers, got {x}"
|
||||
)))
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn data_to_js(data: Data) -> JsValue {
|
||||
match data {
|
||||
Data::F32(v) => js_sys::Float32Array::from(&v[..]).into(),
|
||||
Data::F64(v) => js_sys::Float64Array::from(&v[..]).into(),
|
||||
Data::I8(v) => js_sys::Int8Array::from(&v[..]).into(),
|
||||
Data::I16(v) => js_sys::Int16Array::from(&v[..]).into(),
|
||||
Data::I32(v) => js_sys::Int32Array::from(&v[..]).into(),
|
||||
Data::I64(v) => js_sys::BigInt64Array::from(&v[..]).into(),
|
||||
Data::U8(v) => js_sys::Uint8Array::from(&v[..]).into(),
|
||||
Data::U16(v) => js_sys::Uint16Array::from(&v[..]).into(),
|
||||
Data::U32(v) => js_sys::Uint32Array::from(&v[..]).into(),
|
||||
Data::U64(v) => js_sys::BigUint64Array::from(&v[..]).into(),
|
||||
Data::Strings(v) => v
|
||||
.into_iter()
|
||||
.map(|s| JsValue::from_str(&s))
|
||||
.collect::<Array>()
|
||||
.into(),
|
||||
}
|
||||
}
|
||||
|
||||
/// A scalar integer as a `number` when exact, else a `bigint`.
|
||||
fn int_to_js(x: i128) -> JsValue {
|
||||
if (x as f64).abs() <= MAX_SAFE_INTEGER {
|
||||
JsValue::from_f64(x as f64)
|
||||
} else if let Ok(v) = i64::try_from(x) {
|
||||
JsValue::from(v)
|
||||
} else {
|
||||
JsValue::from(x as u64)
|
||||
}
|
||||
}
|
||||
|
||||
/// An attribute value, and the datatype of one that has no JavaScript form.
|
||||
fn attr_to_js(value: AttrValue) -> (JsValue, Option<String>) {
|
||||
match value {
|
||||
AttrValue::F64(x) => (JsValue::from_f64(x), None),
|
||||
AttrValue::F64Array(v) => (js_sys::Float64Array::from(&v[..]).into(), None),
|
||||
AttrValue::I64(x) => (int_to_js(i128::from(x)), None),
|
||||
AttrValue::I64Array(v) => (js_sys::BigInt64Array::from(&v[..]).into(), None),
|
||||
AttrValue::U64(x) => (int_to_js(i128::from(x)), None),
|
||||
AttrValue::U64Array(v) => (js_sys::BigUint64Array::from(&v[..]).into(), None),
|
||||
AttrValue::String(s) => (JsValue::from_str(&s), None),
|
||||
AttrValue::StringArray(v) => (
|
||||
v.iter()
|
||||
.map(|s| JsValue::from_str(s))
|
||||
.collect::<Array>()
|
||||
.into(),
|
||||
None,
|
||||
),
|
||||
AttrValue::Raw { datatype, .. } => (JsValue::NULL, Some(core::describe(&datatype))),
|
||||
}
|
||||
}
|
||||
|
||||
/// An open HDF5 (or NetCDF-4) file.
|
||||
#[wasm_bindgen]
|
||||
pub struct H5File {
|
||||
inner: Reader,
|
||||
}
|
||||
|
||||
/// Open a file from its bytes. Throws if they are not an HDF5 file.
|
||||
#[wasm_bindgen]
|
||||
pub fn open(bytes: Vec<u8>) -> Result<H5File, JsError> {
|
||||
H5File::new(bytes)
|
||||
}
|
||||
|
||||
/// The clawhdf5 version this module was built from.
|
||||
#[wasm_bindgen]
|
||||
pub fn version() -> String {
|
||||
env!("CARGO_PKG_VERSION").to_string()
|
||||
}
|
||||
|
||||
#[wasm_bindgen]
|
||||
impl H5File {
|
||||
/// Same as [`open`].
|
||||
#[wasm_bindgen(constructor)]
|
||||
pub fn new(bytes: Vec<u8>) -> Result<H5File, JsError> {
|
||||
Ok(H5File {
|
||||
inner: Reader::open(bytes).map_err(js_err)?,
|
||||
})
|
||||
}
|
||||
|
||||
/// `"group"` or `"dataset"`.
|
||||
pub fn kind(&self, path: &str) -> Result<String, JsError> {
|
||||
Ok(self.inner.kind(path).map_err(js_err)?.as_str().to_string())
|
||||
}
|
||||
|
||||
/// The group's members: `[{ name, kind }]`, groups first.
|
||||
pub fn list(&self, path: &str) -> Result<Array, JsError> {
|
||||
Ok(self
|
||||
.inner
|
||||
.list(path)
|
||||
.map_err(js_err)?
|
||||
.into_iter()
|
||||
.map(|c| {
|
||||
let o = Object::new();
|
||||
set(&o, "name", c.name);
|
||||
set(&o, "kind", c.kind.as_str());
|
||||
JsValue::from(o)
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// `{ shape, maxshape, dtype, elementShape }`. `maxshape` is `null`
|
||||
/// when not recorded, with `null` for each unlimited dimension.
|
||||
pub fn info(&self, path: &str) -> Result<Object, JsError> {
|
||||
let i = self.inner.info(path).map_err(js_err)?;
|
||||
let o = Object::new();
|
||||
set(&o, "shape", shape_to_js(&i.shape));
|
||||
let max: JsValue = match i.maxshape {
|
||||
None => JsValue::NULL,
|
||||
Some(dims) => dims
|
||||
.into_iter()
|
||||
.map(|d| d.map_or(JsValue::NULL, |d| JsValue::from_f64(d as f64)))
|
||||
.collect::<Array>()
|
||||
.into(),
|
||||
};
|
||||
set(&o, "maxshape", max);
|
||||
set(&o, "dtype", i.dtype);
|
||||
set(&o, "elementShape", shape_to_js(&i.element_shape));
|
||||
Ok(o)
|
||||
}
|
||||
|
||||
/// `[{ name, value, dtype }]`, sorted by name. Scalars are `number`
|
||||
/// (`bigint` beyond 2^53) or `string`; arrays are typed arrays or
|
||||
/// `string[]`. An attribute with no JavaScript form has `value: null`
|
||||
/// and its `dtype`; one that could not be read at all is reported by
|
||||
/// [`attrErrors`](Self::attr_errors).
|
||||
pub fn attrs(&self, path: &str) -> Result<Array, JsError> {
|
||||
let (attrs, _) = self.inner.attrs(path).map_err(js_err)?;
|
||||
Ok(attrs
|
||||
.into_iter()
|
||||
.map(|a| {
|
||||
let o = Object::new();
|
||||
set(&o, "name", a.name);
|
||||
let (value, dtype) = attr_to_js(a.value);
|
||||
set(&o, "value", value);
|
||||
set(&o, "dtype", dtype.map_or(JsValue::NULL, JsValue::from));
|
||||
JsValue::from(o)
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// Messages for attributes that could not be read.
|
||||
#[wasm_bindgen(js_name = attrErrors)]
|
||||
pub fn attr_errors(&self, path: &str) -> Result<Array, JsError> {
|
||||
let (_, errors) = self.inner.attrs(path).map_err(js_err)?;
|
||||
Ok(errors.into_iter().map(JsValue::from).collect())
|
||||
}
|
||||
|
||||
/// The whole dataset: `{ shape, dtype, data }`, `data` in row-major
|
||||
/// order.
|
||||
pub fn read(&self, path: &str) -> Result<Object, JsError> {
|
||||
self.read_impl(path, None)
|
||||
}
|
||||
|
||||
/// A regular hyperslab (`H5Sselect_hyperslab`): `stride` and `block`
|
||||
/// default to 1. The result's shape is `count * block` per dimension.
|
||||
#[wasm_bindgen(js_name = readHyperslab)]
|
||||
pub fn read_hyperslab(
|
||||
&self,
|
||||
path: &str,
|
||||
start: Vec<f64>,
|
||||
count: Vec<f64>,
|
||||
stride: Option<Vec<f64>>,
|
||||
block: Option<Vec<f64>>,
|
||||
) -> Result<Object, JsError> {
|
||||
let slab = Hyperslab {
|
||||
start: indices_from_js("start", &start)?,
|
||||
count: indices_from_js("count", &count)?,
|
||||
stride: stride.map(|s| indices_from_js("stride", &s)).transpose()?,
|
||||
block: block.map(|b| indices_from_js("block", &b)).transpose()?,
|
||||
};
|
||||
self.read_impl(path, Some(&slab))
|
||||
}
|
||||
|
||||
fn read_impl(&self, path: &str, slab: Option<&Hyperslab>) -> Result<Object, JsError> {
|
||||
let dtype = self.inner.info(path).map_err(js_err)?.dtype;
|
||||
let a = self.inner.read(path, slab).map_err(js_err)?;
|
||||
let o = Object::new();
|
||||
set(&o, "shape", shape_to_js(&a.shape));
|
||||
set(&o, "dtype", dtype);
|
||||
set(&o, "data", data_to_js(a.data));
|
||||
Ok(o)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,265 @@
|
||||
//! The wasm reader's core against files written by h5py and netCDF4, with
|
||||
//! the values libhdf5 reads back as the reference. The generator,
|
||||
//! `examples/wasm-viewer/test/make_fixture.py`, is shared with the Node test
|
||||
//! of the built wasm package, so both compare against the same expectations.
|
||||
//!
|
||||
//! Skipped when python3 with h5py/netCDF4 is missing, unless
|
||||
//! `CLAWHDF5_REQUIRE_INTEROP=1`. `CLAWHDF5_PYTHON` names the interpreter.
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::process::Command;
|
||||
|
||||
use clawhdf5::AttrValue;
|
||||
use clawhdf5_wasm::core::{Data, Hyperslab, Kind, Reader};
|
||||
use serde_json::Value;
|
||||
|
||||
fn python() -> String {
|
||||
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
|
||||
}
|
||||
|
||||
fn interop_required() -> bool {
|
||||
std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
|
||||
}
|
||||
|
||||
fn python_available() -> bool {
|
||||
Command::new(python())
|
||||
.args(["-c", "import h5py, netCDF4, numpy"])
|
||||
.output()
|
||||
.map(|o| o.status.success())
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
fn generator() -> PathBuf {
|
||||
Path::new(env!("CARGO_MANIFEST_DIR")).join("../../examples/wasm-viewer/test/make_fixture.py")
|
||||
}
|
||||
|
||||
/// Values as comparable strings: integers exactly, floats by their f64
|
||||
/// value (an f32 widens exactly), strings as themselves.
|
||||
fn data_strings(d: &Data) -> (&'static str, Vec<String>) {
|
||||
fn s<T: ToString>(v: &[T]) -> Vec<String> {
|
||||
v.iter().map(ToString::to_string).collect()
|
||||
}
|
||||
fn f<T: Copy + Into<f64>>(v: &[T]) -> Vec<String> {
|
||||
v.iter().map(|&x| format!("{:?}", x.into())).collect()
|
||||
}
|
||||
match d {
|
||||
Data::F32(v) => ("f32", f(v)),
|
||||
Data::F64(v) => ("f64", f(v)),
|
||||
Data::I8(v) => ("i8", s(v)),
|
||||
Data::I16(v) => ("i16", s(v)),
|
||||
Data::I32(v) => ("i32", s(v)),
|
||||
Data::I64(v) => ("i64", s(v)),
|
||||
Data::U8(v) => ("u8", s(v)),
|
||||
Data::U16(v) => ("u16", s(v)),
|
||||
Data::U32(v) => ("u32", s(v)),
|
||||
Data::U64(v) => ("u64", s(v)),
|
||||
Data::Strings(v) => ("strings", v.clone()),
|
||||
}
|
||||
}
|
||||
|
||||
fn expected_strings(kind: &str, values: &Value) -> Vec<String> {
|
||||
values
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|v| match (kind, v) {
|
||||
("f32" | "f64", Value::Number(n)) => format!("{:?}", n.as_f64().unwrap()),
|
||||
(_, Value::String(s)) => s.clone(),
|
||||
other => panic!("unexpected expected value {other:?}"),
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn shape(v: &Value) -> Vec<u64> {
|
||||
v.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|x| x.as_u64().unwrap())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn check_attr(file: &str, path: &str, name: &str, got: &AttrValue, want: &Value) {
|
||||
let ctx = format!("{file}:{path}@{name}");
|
||||
let ints = |v: &Value| -> Vec<String> {
|
||||
v.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|x| x.as_str().unwrap().to_string())
|
||||
.collect()
|
||||
};
|
||||
let scalar = want["scalar"].as_bool().unwrap_or(false);
|
||||
match got {
|
||||
AttrValue::String(s) => assert_eq!(want["string"].as_str(), Some(s.as_str()), "{ctx}"),
|
||||
AttrValue::StringArray(v) => {
|
||||
let w: Vec<&str> = want["strings"]
|
||||
.as_array()
|
||||
.unwrap_or_else(|| panic!("{ctx}: got {v:?}"))
|
||||
.iter()
|
||||
.map(|x| x.as_str().unwrap())
|
||||
.collect();
|
||||
assert_eq!(v, &w, "{ctx}");
|
||||
}
|
||||
AttrValue::I64(x) => {
|
||||
assert!(scalar, "{ctx}");
|
||||
assert_eq!(vec![x.to_string()], ints(&want["int"]), "{ctx}");
|
||||
}
|
||||
AttrValue::U64(x) => {
|
||||
assert!(scalar, "{ctx}");
|
||||
assert_eq!(vec![x.to_string()], ints(&want["int"]), "{ctx}");
|
||||
}
|
||||
AttrValue::I64Array(v) => assert_eq!(
|
||||
v.iter().map(ToString::to_string).collect::<Vec<_>>(),
|
||||
ints(&want["int"]),
|
||||
"{ctx}"
|
||||
),
|
||||
AttrValue::U64Array(v) => assert_eq!(
|
||||
v.iter().map(ToString::to_string).collect::<Vec<_>>(),
|
||||
ints(&want["int"]),
|
||||
"{ctx}"
|
||||
),
|
||||
AttrValue::F64(x) => {
|
||||
assert!(scalar, "{ctx}");
|
||||
assert_eq!(Some(*x), want["float"][0].as_f64(), "{ctx}");
|
||||
}
|
||||
AttrValue::F64Array(v) => {
|
||||
let w: Vec<f64> = want["float"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|x| x.as_f64().unwrap())
|
||||
.collect();
|
||||
assert_eq!(v, &w, "{ctx}");
|
||||
}
|
||||
AttrValue::Raw { datatype, .. } => {
|
||||
let want = want["raw"]
|
||||
.as_str()
|
||||
.unwrap_or_else(|| panic!("{ctx}: undecoded {datatype:?}"));
|
||||
let d = clawhdf5_wasm::core::describe(datatype);
|
||||
assert!(d.contains(want), "{ctx}: {d}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn check_file(dir: &Path, file: &str, exp: &Value) {
|
||||
let r = Reader::open(std::fs::read(dir.join(file)).unwrap()).unwrap();
|
||||
|
||||
for (path, want) in exp["lists"].as_object().unwrap() {
|
||||
let list = r
|
||||
.list(path)
|
||||
.unwrap_or_else(|e| panic!("{file}:{path}: {e}"));
|
||||
for (kind, key) in [(Kind::Group, "groups"), (Kind::Dataset, "datasets")] {
|
||||
let mut got: Vec<&str> = list
|
||||
.iter()
|
||||
.filter(|c| c.kind == kind)
|
||||
.map(|c| c.name.as_str())
|
||||
.collect();
|
||||
got.sort();
|
||||
let w: Vec<&str> = want[key]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|x| x.as_str().unwrap())
|
||||
.collect();
|
||||
assert_eq!(got, w, "{file}:{path} {key}");
|
||||
}
|
||||
}
|
||||
|
||||
for (path, want) in exp["datasets"].as_object().unwrap() {
|
||||
let kind = want["kind"].as_str().unwrap();
|
||||
let a = match (r.read(path, None), want["unavailable"].as_str()) {
|
||||
(Ok(a), _) => a,
|
||||
// A filter this build may lack (zstd: the wasm build has it
|
||||
// off, a workspace build may unify it on) must fail clearly.
|
||||
(Err(e), Some(why)) => {
|
||||
assert!(e.contains(why), "{file}:{path}: {e}");
|
||||
continue;
|
||||
}
|
||||
(Err(e), None) => panic!("{file}:{path}: {e}"),
|
||||
};
|
||||
assert_eq!(a.shape, shape(&want["shape"]), "{file}:{path} shape");
|
||||
let (got_kind, got) = data_strings(&a.data);
|
||||
assert_eq!(got_kind, kind, "{file}:{path} kind");
|
||||
assert_eq!(
|
||||
got,
|
||||
expected_strings(kind, &want["values"]),
|
||||
"{file}:{path}"
|
||||
);
|
||||
|
||||
if let Some(slab) = want.get("slab") {
|
||||
let h = Hyperslab {
|
||||
start: shape(&slab["start"]),
|
||||
count: shape(&slab["count"]),
|
||||
stride: Some(shape(&slab["stride"])),
|
||||
block: None,
|
||||
};
|
||||
let a = r
|
||||
.read(path, Some(&h))
|
||||
.unwrap_or_else(|e| panic!("{file}:{path} {h:?}: {e}"));
|
||||
assert_eq!(a.shape, shape(&slab["shape"]), "{file}:{path} slab shape");
|
||||
assert_eq!(
|
||||
data_strings(&a.data).1,
|
||||
expected_strings(kind, &slab["values"]),
|
||||
"{file}:{path} slab"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (path, what) in exp["errors"].as_object().unwrap() {
|
||||
let e = r.read(path, None).expect_err(path);
|
||||
assert!(e.contains(what.as_str().unwrap()), "{file}:{path}: {e}");
|
||||
}
|
||||
|
||||
let skip: Vec<&str> = exp["skip_attrs"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.map(|x| x.as_str().unwrap())
|
||||
.collect();
|
||||
for (path, want) in exp["attrs"].as_object().unwrap() {
|
||||
let (attrs, errors) = r
|
||||
.attrs(path)
|
||||
.unwrap_or_else(|e| panic!("{file}:{path}: {e}"));
|
||||
assert!(errors.is_empty(), "{file}:{path}: {errors:?}");
|
||||
let want = want.as_object().unwrap();
|
||||
let mut compared = 0;
|
||||
for a in &attrs {
|
||||
if a.name.starts_with('_') || skip.contains(&a.name.as_str()) {
|
||||
continue;
|
||||
}
|
||||
let w = want
|
||||
.get(&a.name)
|
||||
.unwrap_or_else(|| panic!("{file}:{path}: unexpected attribute {}", a.name));
|
||||
check_attr(file, path, &a.name, &a.value, w);
|
||||
compared += 1;
|
||||
}
|
||||
assert_eq!(compared, want.len(), "{file}:{path}: attributes missing");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reads_what_h5py_and_netcdf4_wrote() {
|
||||
if !python_available() {
|
||||
assert!(
|
||||
!interop_required(),
|
||||
"CLAWHDF5_REQUIRE_INTEROP=1 but python with h5py, netCDF4 and numpy is not available"
|
||||
);
|
||||
eprintln!("SKIP: python with h5py/netCDF4 not available");
|
||||
return;
|
||||
}
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let out = Command::new(python())
|
||||
.arg(generator())
|
||||
.arg(dir.path())
|
||||
.output()
|
||||
.expect("run python");
|
||||
assert!(
|
||||
out.status.success(),
|
||||
"fixture generator failed:\n{}",
|
||||
String::from_utf8_lossy(&out.stderr)
|
||||
);
|
||||
let exp: Value =
|
||||
serde_json::from_slice(&std::fs::read(dir.path().join("expected.json")).unwrap()).unwrap();
|
||||
for file in ["fixture.h5", "fixture.nc"] {
|
||||
check_file(dir.path(), file, &exp[file]);
|
||||
}
|
||||
}
|
||||
@@ -144,6 +144,35 @@ mod tests {
|
||||
assert_eq!(ds.dtype().unwrap(), DType::I32);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dataset_raw_datatype() {
|
||||
use clawhdf5_format::datatype::Datatype;
|
||||
let bytes = make_simple_file();
|
||||
let file = File::from_bytes(bytes).unwrap();
|
||||
let dt = file.dataset("counts").unwrap().raw_datatype().unwrap();
|
||||
assert!(
|
||||
matches!(
|
||||
dt,
|
||||
Datatype::FixedPoint {
|
||||
size: 4,
|
||||
signed: true,
|
||||
..
|
||||
}
|
||||
),
|
||||
"{dt:?}"
|
||||
);
|
||||
// The full type pairs with read_selection's bytes.
|
||||
let raw = file
|
||||
.dataset("counts")
|
||||
.unwrap()
|
||||
.read_selection(&Selection::All)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
clawhdf5_format::data_read::read_as_i64(&raw, &dt).unwrap(),
|
||||
vec![10, 20, 30]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn root_group_datasets() {
|
||||
let bytes = make_simple_file();
|
||||
|
||||
@@ -424,6 +424,15 @@ impl<'f> Dataset<'f> {
|
||||
Ok(classify_datatype(&dt))
|
||||
}
|
||||
|
||||
/// Returns the dataset's full datatype as stored in the file (byte order,
|
||||
/// string padding, compound layout, ...), with a committed datatype
|
||||
/// resolved. Use it with the `clawhdf5_format::data_read` converters on
|
||||
/// the bytes [`read_selection`](Self::read_selection) returns, for types
|
||||
/// the typed `read_*` methods do not cover.
|
||||
pub fn raw_datatype(&self) -> Result<Datatype, Error> {
|
||||
self.datatype()
|
||||
}
|
||||
|
||||
/// Read all data as `f64` values.
|
||||
pub fn read_f64(&self) -> Result<Vec<f64>, Error> {
|
||||
let dt = self.datatype()?;
|
||||
|
||||
@@ -473,6 +473,23 @@ the same agent-store interop test.
|
||||
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
||||
which is what libhdf5 itself writes.
|
||||
|
||||
## `clawhdf5-wasm` (browser) limits
|
||||
|
||||
**Status:** open (by design for now; added 2026-09-26).
|
||||
|
||||
- The whole file is held in memory: `open()` takes its bytes. There are no
|
||||
HTTP range reads, so a multi-GB file does not fit a browser tab.
|
||||
- Compound, reference, opaque, bitfield, time and VL-sequence datasets are
|
||||
refused with an error naming the type; attributes of those types come back
|
||||
as `value: null` with their `dtype`.
|
||||
- No Zstd or SZIP (both link C): such datasets fail with
|
||||
`unsupported filter: 32015` / `: 4`. pcodec is not enabled either.
|
||||
- External links and virtual-dataset sources in other files cannot be
|
||||
followed (no file system).
|
||||
- Variable-length string datasets are read by decoding `read_selection`'s
|
||||
bytes with `clawhdf5_format::vl_data` in the wasm crate; `File` itself still
|
||||
cannot (see the audit gaps above).
|
||||
|
||||
## The Node.js package (`packages/clawhdf5-node`) does not work
|
||||
|
||||
**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
# Generated by build.sh.
|
||||
/pkg/
|
||||
@@ -0,0 +1,97 @@
|
||||
# HDF5 viewer in the browser
|
||||
|
||||
A single page that opens an HDF5 or NetCDF-4 file entirely in the browser
|
||||
with `clawhdf5-wasm` (clawhdf5's reader compiled to WebAssembly): drop a
|
||||
file, browse its groups, and look at a dataset's type, shape, attributes
|
||||
and values (a 50 x 12 window at a time, read as a hyperslab, with the
|
||||
leading dimensions of a 3-D+ dataset held at chosen indices). The file never
|
||||
leaves the page.
|
||||
|
||||
## Build and open
|
||||
|
||||
```bash
|
||||
rustup target add wasm32-unknown-unknown
|
||||
cargo install wasm-bindgen-cli --version 0.2.129 # must equal the crate version; build.sh checks
|
||||
bash examples/wasm-viewer/build.sh # writes examples/wasm-viewer/pkg/ (not committed)
|
||||
python3 -m http.server -d examples/wasm-viewer 8000 # wasm cannot load from file://
|
||||
```
|
||||
|
||||
Then open <http://localhost:8000/>. `?file=<url>&path=<object>` opens a
|
||||
file from a URL (same origin, or one serving CORS headers) and selects an
|
||||
object in it, e.g. `?file=data/run1.h5&path=/results/energy`.
|
||||
|
||||
## JavaScript API
|
||||
|
||||
```js
|
||||
import init, { open } from "./pkg/clawhdf5_wasm.js";
|
||||
await init();
|
||||
const f = open(new Uint8Array(await blob.arrayBuffer()));
|
||||
f.list("/"); // [{ name, kind: "group" | "dataset" }], groups first
|
||||
f.info("/grid"); // { shape, maxshape, dtype, elementShape }
|
||||
f.attrs("/grid"); // [{ name, value, dtype }]
|
||||
f.read("/grid"); // { shape, dtype, data }
|
||||
f.readHyperslab("/grid", [0, 0], [10, 5], [2, 1]); // start, count, stride?, block?
|
||||
f.free();
|
||||
```
|
||||
|
||||
`data` is the typed array of the stored width (`Float64Array`,
|
||||
`Float32Array` also for `f16`, `Int8Array` ... `BigInt64Array`,
|
||||
`BigUint64Array`), or an array of strings for fixed- and variable-length
|
||||
strings and enumerations (h5py booleans read as `"TRUE"`/`"FALSE"`). Array
|
||||
datatypes are flattened, their dimensions appended to `shape`. Anything
|
||||
else throws an `Error` naming the type.
|
||||
|
||||
## Limits
|
||||
|
||||
- Read-only, and the whole file is held in memory (no range requests).
|
||||
- Compound, reference, opaque and variable-length-sequence datasets are
|
||||
refused with an error. Attributes of those types are listed with
|
||||
`value: null` and their `dtype`.
|
||||
- No Zstd or SZIP filters (they link C): such a dataset fails with
|
||||
`unsupported filter`. Deflate, shuffle, Fletcher-32, LZ4, N-Bit and
|
||||
scale-offset are read (within the limits in `docs/known-issues.md`).
|
||||
- Virtual datasets whose sources are in other files, and external links,
|
||||
cannot be followed: there is no file system.
|
||||
|
||||
## Tests
|
||||
|
||||
`test/run.sh` builds the package, writes `fixture.h5` (h5py) and
|
||||
`fixture.nc` (netCDF4) with `test/make_fixture.py`, then:
|
||||
|
||||
- runs `test/test.mjs` under Node: every dataset (whole and a strided
|
||||
hyperslab), listing and attribute is compared with what libhdf5 reads
|
||||
back, error paths are checked, and so are the page's DOM-free helpers
|
||||
(`viewer-lib.js`);
|
||||
- runs `test/browser.sh`: loads the page in headless Chromium with
|
||||
`?file=fixture.h5&path=...` for eight objects and checks the rendered tree,
|
||||
types, shapes, attribute and value cells, and the error shown for an
|
||||
unsupported type. Skipped when no Chromium is found (`CHROME` names one;
|
||||
a Playwright download under `~/.cache/ms-playwright` is picked up).
|
||||
Drag-and-drop and the file picker are not driven by it; they share
|
||||
`load()` with the `?file=` path.
|
||||
|
||||
The same expectations are checked natively, without Node, by
|
||||
`crates/clawhdf5-wasm/tests/h5py_interop.rs`, which is what CI runs (the CI
|
||||
container has no Node or browser).
|
||||
|
||||
## Size
|
||||
|
||||
Measured 2026-09-26 on tank (rustc 1.98.1, wasm-bindgen 0.2.129, gzip 1.14,
|
||||
`gzip -9 -n`), after `bash examples/wasm-viewer/build.sh`:
|
||||
|
||||
| | raw | gzip -9 |
|
||||
|---|---:|---:|
|
||||
| `pkg/clawhdf5_wasm_bg.wasm` (profile `wasm-release`, opt-level `s`) | 627,501 B | 191,639 B |
|
||||
| `pkg/clawhdf5_wasm.js` (wasm-bindgen glue) | 21,826 B | 4,487 B |
|
||||
| same wasm at opt-level `z` | 693,068 B | 192,550 B |
|
||||
| same wasm at opt-level `3` | 544,035 B | 198,803 B |
|
||||
| h5wasm 0.10.3: wasm embedded in `dist/esm/hdf5_util.js` | 3,544,184 B | 907,096 B |
|
||||
| h5wasm 0.10.3: `dist/esm/hdf5_util.js` as shipped | 4,150,134 B | 986,699 B |
|
||||
|
||||
h5wasm figures: `npm pack [email protected]` (npm reports
|
||||
`dist.unpackedSize` 14,731,385 B for the whole package), wasm extracted from
|
||||
the `binaryDecode` literal in `hdf5_util.js`. h5wasm is the whole of libhdf5
|
||||
(writing, every datatype, plugins), so this compares download size, not
|
||||
equal functionality. No `wasm-opt` pass was applied (binaryen is not
|
||||
installed on tank). opt-level `s` is used because it is the smallest
|
||||
compressed.
|
||||
Executable
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the clawhdf5-wasm package the viewer loads, into examples/wasm-viewer/pkg/.
|
||||
#
|
||||
# Needs the wasm32-unknown-unknown target and the wasm-bindgen CLI at the
|
||||
# exact version cargo resolves for the wasm-bindgen crate:
|
||||
# rustup target add wasm32-unknown-unknown
|
||||
# cargo install wasm-bindgen-cli --version <that version>
|
||||
#
|
||||
# Then serve this directory over HTTP (browsers do not load wasm modules from
|
||||
# file://) and open it:
|
||||
# python3 -m http.server -d examples/wasm-viewer 8000
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
ROOT="$(cd "$HERE/../.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
# The resolved crate version (Cargo.lock is not committed, so ask cargo).
|
||||
want=$(cargo pkgid wasm-bindgen | sed 's/.*[@#]//')
|
||||
if ! command -v wasm-bindgen >/dev/null; then
|
||||
echo "wasm-bindgen CLI not found: cargo install wasm-bindgen-cli --version $want" >&2
|
||||
exit 1
|
||||
fi
|
||||
have=$(wasm-bindgen --version | awk '{print $2}')
|
||||
if [ "$want" != "$have" ]; then
|
||||
echo "wasm-bindgen CLI is $have but the crate is $want:" >&2
|
||||
echo " cargo install wasm-bindgen-cli --version $want" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cargo build -p clawhdf5-wasm --target wasm32-unknown-unknown --profile wasm-release
|
||||
|
||||
target_dir=$(cargo metadata --format-version 1 --no-deps \
|
||||
| sed -n 's/.*"target_directory":"\([^"]*\)".*/\1/p')
|
||||
wasm="$target_dir/wasm32-unknown-unknown/wasm-release/clawhdf5_wasm.wasm"
|
||||
|
||||
rm -rf "$HERE/pkg"
|
||||
wasm-bindgen --target web --out-dir "$HERE/pkg" "$wasm"
|
||||
echo "built $HERE/pkg ($(wc -c < "$HERE/pkg/clawhdf5_wasm_bg.wasm") bytes of wasm)"
|
||||
@@ -0,0 +1,278 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>HDF5 Viewer</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #fbfbfa; --panel: #ffffff; --ink: #1d1d1b; --muted: #6b6b66;
|
||||
--line: #e2e1dc; --accent: #2f5d8a; --accent-soft: #e8eff6; --bad: #a3312a;
|
||||
--mono: ui-monospace, "SF Mono", Menlo, Consolas, monospace;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--bg: #161615; --panel: #1f1f1d; --ink: #ecebe6; --muted: #9a9993;
|
||||
--line: #34332f; --accent: #8db8e0; --accent-soft: #23303c; --bad: #e0857c;
|
||||
}
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; background: var(--bg); color: var(--ink);
|
||||
font: 14px/1.45 system-ui, -apple-system, "Segoe UI", sans-serif; }
|
||||
header { display: flex; align-items: center; gap: 12px; padding: 12px 16px;
|
||||
border-bottom: 1px solid var(--line); background: var(--panel); flex-wrap: wrap; }
|
||||
header h1 { font-size: 15px; margin: 0; font-weight: 600; }
|
||||
header .file { color: var(--muted); font-family: var(--mono); font-size: 13px; }
|
||||
header label.button { margin-left: auto; }
|
||||
.button { border: 1px solid var(--line); background: var(--panel); color: var(--ink);
|
||||
padding: 5px 10px; border-radius: 6px; cursor: pointer; font: inherit; }
|
||||
.button:hover { border-color: var(--accent); }
|
||||
main { display: grid; grid-template-columns: minmax(200px, 300px) 1fr; min-height: calc(100vh - 50px); }
|
||||
@media (max-width: 700px) { main { grid-template-columns: 1fr; } nav { border-right: 0; border-bottom: 1px solid var(--line); } }
|
||||
nav { border-right: 1px solid var(--line); padding: 8px; overflow: auto; background: var(--panel); }
|
||||
section { padding: 16px; overflow: auto; min-width: 0; }
|
||||
ul.tree { list-style: none; margin: 0; padding-left: 14px; }
|
||||
nav > ul.tree { padding-left: 0; }
|
||||
.node { display: flex; gap: 6px; align-items: baseline; padding: 2px 6px; border-radius: 4px;
|
||||
cursor: pointer; white-space: nowrap; font-family: var(--mono); font-size: 13px; }
|
||||
.node:hover { background: var(--accent-soft); }
|
||||
.node.selected { background: var(--accent-soft); color: var(--accent); }
|
||||
.node .icon { width: 1em; color: var(--muted); flex: none; text-align: center; }
|
||||
.drop { border: 2px dashed var(--line); border-radius: 10px; padding: 48px 16px; text-align: center;
|
||||
color: var(--muted); max-width: 560px; margin: 48px auto; }
|
||||
body.dragging .drop, body.dragging nav { border-color: var(--accent); }
|
||||
h2 { font-size: 16px; margin: 0 0 8px; font-family: var(--mono); word-break: break-all; }
|
||||
h3 { font-size: 12px; text-transform: uppercase; letter-spacing: .06em; color: var(--muted); margin: 20px 0 6px; }
|
||||
dl.meta { display: grid; grid-template-columns: max-content 1fr; gap: 2px 16px; margin: 0; }
|
||||
dl.meta dt { color: var(--muted); }
|
||||
dl.meta dd { margin: 0; font-family: var(--mono); word-break: break-word; }
|
||||
table { border-collapse: collapse; font-family: var(--mono); font-size: 12.5px; }
|
||||
th, td { border: 1px solid var(--line); padding: 3px 8px; text-align: right; white-space: nowrap; }
|
||||
th { background: var(--bg); color: var(--muted); font-weight: 500; }
|
||||
table.attrs td { text-align: left; white-space: normal; word-break: break-word; }
|
||||
.scroll { overflow: auto; max-width: 100%; }
|
||||
.controls { display: flex; flex-wrap: wrap; gap: 8px 14px; align-items: center; margin: 6px 0 10px; }
|
||||
.controls input { width: 6em; font: inherit; padding: 2px 4px; background: var(--panel); color: var(--ink);
|
||||
border: 1px solid var(--line); border-radius: 4px; }
|
||||
.error { color: var(--bad); font-family: var(--mono); white-space: pre-wrap; }
|
||||
.muted { color: var(--muted); }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<h1>HDF5 Viewer</h1>
|
||||
<span class="file" id="filename">no file</span>
|
||||
<label class="button">Open file…<input type="file" id="picker" accept=".h5,.hdf5,.he5,.nc,.nc4,.cdf" hidden></label>
|
||||
</header>
|
||||
<main>
|
||||
<nav id="tree"></nav>
|
||||
<section id="detail">
|
||||
<div class="drop">
|
||||
<p><strong>Drop an HDF5 or NetCDF-4 file here</strong>, or use “Open file…”.</p>
|
||||
<p class="muted">The file is read in this page by clawhdf5 compiled to WebAssembly; it is not uploaded anywhere.</p>
|
||||
<p class="muted" id="version"></p>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
<script type="module">
|
||||
import init, { open, version } from "./pkg/clawhdf5_wasm.js";
|
||||
import { joinPath, formatValue, viewWindow, toRows, perElement } from "./viewer-lib.js";
|
||||
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const el = (tag, props = {}, ...kids) => {
|
||||
const e = Object.assign(document.createElement(tag), props);
|
||||
e.append(...kids);
|
||||
return e;
|
||||
};
|
||||
|
||||
let file = null;
|
||||
let selectedNode = null;
|
||||
|
||||
await init();
|
||||
$("version").textContent = `clawhdf5 ${version()}`;
|
||||
|
||||
async function load(blob) {
|
||||
const bytes = new Uint8Array(await blob.arrayBuffer());
|
||||
if (file) file.free();
|
||||
file = null;
|
||||
$("filename").textContent = blob.name;
|
||||
$("tree").replaceChildren();
|
||||
try {
|
||||
file = open(bytes);
|
||||
} catch (e) {
|
||||
$("detail").replaceChildren(el("p", { className: "error", textContent: `Cannot open ${blob.name}: ${e.message}` }));
|
||||
return;
|
||||
}
|
||||
const root = el("ul", { className: "tree" });
|
||||
root.append(treeNode("/", "/", "group"));
|
||||
$("tree").append(root);
|
||||
root.querySelector(".node").click();
|
||||
}
|
||||
|
||||
function treeNode(path, name, kind) {
|
||||
const li = el("li");
|
||||
const icon = el("span", { className: "icon", textContent: kind === "group" ? "▸" : "·" });
|
||||
const row = el("div", { className: "node", title: path }, icon, el("span", { textContent: name }));
|
||||
row.dataset.path = path;
|
||||
li.append(row);
|
||||
let children = null;
|
||||
row.addEventListener("click", () => {
|
||||
if (selectedNode) selectedNode.classList.remove("selected");
|
||||
row.classList.add("selected");
|
||||
selectedNode = row;
|
||||
if (kind === "group") {
|
||||
if (children) {
|
||||
children.hidden = !children.hidden;
|
||||
} else {
|
||||
children = el("ul", { className: "tree" });
|
||||
try {
|
||||
for (const c of file.list(path)) children.append(treeNode(joinPath(path, c.name), c.name, c.kind));
|
||||
} catch (e) {
|
||||
children.append(el("li", { className: "error", textContent: e.message }));
|
||||
}
|
||||
li.append(children);
|
||||
}
|
||||
icon.textContent = children.hidden ? "▸" : "▾";
|
||||
}
|
||||
show(path, kind);
|
||||
});
|
||||
return li;
|
||||
}
|
||||
|
||||
function attrsTable(path) {
|
||||
let attrs, errors;
|
||||
try {
|
||||
attrs = file.attrs(path);
|
||||
errors = file.attrErrors(path);
|
||||
} catch (e) {
|
||||
return el("p", { className: "error", textContent: e.message });
|
||||
}
|
||||
if (!attrs.length && !errors.length) return el("p", { className: "muted", textContent: "none" });
|
||||
const t = el("table", { className: "attrs" }, el("tr", {}, el("th", { textContent: "name" }), el("th", { textContent: "value" })));
|
||||
for (const a of attrs) {
|
||||
const v = a.value === null ? el("span", { className: "muted", textContent: `(${a.dtype})` }) : formatValue(a.value);
|
||||
t.append(el("tr", {}, el("td", { textContent: a.name }), el("td", {}, v)));
|
||||
}
|
||||
for (const msg of errors) t.append(el("tr", {}, el("td", { className: "error", colSpan: 2, textContent: msg })));
|
||||
return el("div", { className: "scroll" }, t);
|
||||
}
|
||||
|
||||
function show(path, kind) {
|
||||
const out = [el("h2", { textContent: path })];
|
||||
if (kind === "dataset") {
|
||||
let info;
|
||||
try {
|
||||
info = file.info(path);
|
||||
} catch (e) {
|
||||
$("detail").replaceChildren(...out, el("p", { className: "error", textContent: e.message }));
|
||||
return;
|
||||
}
|
||||
const max = info.maxshape === null ? "—" : `(${info.maxshape.map((d) => d ?? "∞").join(", ")})`;
|
||||
out.push(el("dl", { className: "meta" },
|
||||
el("dt", { textContent: "type" }), el("dd", { textContent: info.dtype }),
|
||||
el("dt", { textContent: "shape" }), el("dd", { textContent: `(${info.shape.join(", ")})` }),
|
||||
el("dt", { textContent: "max shape" }), el("dd", { textContent: max })));
|
||||
out.push(el("h3", { textContent: "Attributes" }), attrsTable(path));
|
||||
out.push(el("h3", { textContent: "Values" }), valuesView(path, info));
|
||||
} else {
|
||||
out.push(el("h3", { textContent: "Attributes" }), attrsTable(path));
|
||||
}
|
||||
$("detail").replaceChildren(...out);
|
||||
}
|
||||
|
||||
function valuesView(path, info) {
|
||||
const shape = info.shape;
|
||||
const per = perElement(info.elementShape);
|
||||
const state = { row: 0, col: 0, rows: 50, cols: 12, fixed: shape.slice(0, Math.max(0, shape.length - 2)).map(() => 0) };
|
||||
const box = el("div");
|
||||
const controls = el("div", { className: "controls" });
|
||||
const body = el("div", { className: "scroll" });
|
||||
box.append(controls, body);
|
||||
|
||||
const num = (label, key, idx) => {
|
||||
const input = el("input", { type: "number", min: 0, value: idx === undefined ? state[key] : state.fixed[idx] });
|
||||
input.addEventListener("change", () => {
|
||||
const v = Math.max(0, Math.floor(Number(input.value) || 0));
|
||||
if (idx === undefined) state[key] = v; else state.fixed[idx] = v;
|
||||
render();
|
||||
});
|
||||
return el("label", {}, `${label} `, input);
|
||||
};
|
||||
if (shape.length >= 1) controls.append(num("row", "row"));
|
||||
if (shape.length >= 2) controls.append(num("column", "col"));
|
||||
state.fixed.forEach((_, i) => controls.append(num(`dim ${i}`, "fixed", i)));
|
||||
|
||||
function render() {
|
||||
const w = viewWindow(shape, state);
|
||||
let res;
|
||||
try {
|
||||
res = w === null ? file.read(path) : file.readHyperslab(path, w.start, w.count);
|
||||
} catch (e) {
|
||||
body.replaceChildren(el("p", { className: "error", textContent: e.message }));
|
||||
return;
|
||||
}
|
||||
if (w === null) {
|
||||
body.replaceChildren(el("pre", { textContent: toRows(res.data, 1, 1, per)[0][0] }));
|
||||
return;
|
||||
}
|
||||
const t = el("table");
|
||||
const head = el("tr", {}, el("th"));
|
||||
for (let c = 0; c < w.cols; c++) head.append(el("th", { textContent: String(w.col + c) }));
|
||||
t.append(head);
|
||||
toRows(res.data, w.rows, w.cols, per).forEach((cells, r) => {
|
||||
const tr = el("tr", {}, el("th", { textContent: String(w.row + r) }));
|
||||
for (const c of cells) tr.append(el("td", { textContent: c }));
|
||||
t.append(tr);
|
||||
});
|
||||
const total = shape.reduce((a, b) => a * b, 1);
|
||||
const note = el("p", { className: "muted", textContent: `showing rows ${w.row}–${w.row + w.rows - 1}` +
|
||||
(shape.length >= 2 ? `, columns ${w.col}–${w.col + w.cols - 1}` : "") + ` of ${total} values` });
|
||||
body.replaceChildren(t, note);
|
||||
}
|
||||
render();
|
||||
return box;
|
||||
}
|
||||
|
||||
// Expand the tree down to `path` and select it.
|
||||
function reveal(path) {
|
||||
const rowFor = (p) => [...document.querySelectorAll(".node")].find((n) => n.dataset.path === p);
|
||||
let cur = "/";
|
||||
for (const part of path.split("/").filter(Boolean)) {
|
||||
const row = rowFor(cur);
|
||||
if (!row) return;
|
||||
const kids = row.parentElement.querySelector(":scope > ul");
|
||||
if (!kids || kids.hidden) row.click();
|
||||
cur = joinPath(cur, part);
|
||||
}
|
||||
const target = rowFor(cur);
|
||||
if (target && target !== selectedNode) target.click();
|
||||
}
|
||||
|
||||
// ?file=<url>&path=<object> opens a file from a URL (same origin, or one
|
||||
// that allows CORS) and selects an object in it.
|
||||
const params = new URLSearchParams(location.search);
|
||||
if (params.get("file")) {
|
||||
const url = params.get("file");
|
||||
try {
|
||||
const resp = await fetch(url);
|
||||
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
|
||||
const blob = await resp.blob();
|
||||
await load(new File([blob], url.split("/").pop()));
|
||||
if (file && params.get("path")) reveal(params.get("path"));
|
||||
} catch (e) {
|
||||
$("detail").replaceChildren(el("p", { className: "error", textContent: `Cannot fetch ${url}: ${e.message}` }));
|
||||
}
|
||||
}
|
||||
|
||||
$("picker").addEventListener("change", (e) => e.target.files[0] && load(e.target.files[0]));
|
||||
document.addEventListener("dragover", (e) => { e.preventDefault(); document.body.classList.add("dragging"); });
|
||||
document.addEventListener("dragleave", () => document.body.classList.remove("dragging"));
|
||||
document.addEventListener("drop", (e) => {
|
||||
e.preventDefault();
|
||||
document.body.classList.remove("dragging");
|
||||
const f = e.dataTransfer.files[0];
|
||||
if (f) load(f);
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env bash
|
||||
# Load the viewer page in headless Chromium and check what it renders.
|
||||
#
|
||||
# browser.sh FIXTURE_DIR
|
||||
#
|
||||
# FIXTURE_DIR holds fixture.h5 from make_fixture.py; ../pkg must be built.
|
||||
# The page is opened with ?file=fixture.h5&path=<object>, which fetches the
|
||||
# file, builds the tree down to <object> and shows it; the rendered DOM is
|
||||
# dumped and checked for the values libhdf5 reads.
|
||||
#
|
||||
# Browser: $CHROME, else chromium/google-chrome on PATH, else a Playwright
|
||||
# download under ~/.cache/ms-playwright. Exit 3 when none is found.
|
||||
# BROWSER_DEBUG=/some/prefix saves each rendered page as prefix.<path>.html.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
FIX="$(cd "$1" && pwd)"
|
||||
PY="${CLAWHDF5_PYTHON:-python3}"
|
||||
|
||||
chrome="${CHROME:-}"
|
||||
if [ -z "$chrome" ]; then
|
||||
for c in chromium chromium-browser google-chrome chrome-headless-shell; do
|
||||
if command -v "$c" >/dev/null; then chrome="$(command -v "$c")"; break; fi
|
||||
done
|
||||
fi
|
||||
if [ -z "$chrome" ]; then
|
||||
chrome="$(ls -d "$HOME"/.cache/ms-playwright/chromium_headless_shell-*/chrome-headless-shell-linux64/chrome-headless-shell 2>/dev/null | tail -1 || true)"
|
||||
fi
|
||||
if [ -z "$chrome" ] || [ ! -x "$chrome" ]; then
|
||||
echo "no Chromium found (set CHROME)" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
root="$(mktemp -d)"
|
||||
server=""
|
||||
cleanup() {
|
||||
[ -n "$server" ] && kill "$server" 2>/dev/null || true
|
||||
rm -rf "$root"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
ln -s "$HERE/../index.html" "$HERE/../viewer-lib.js" "$HERE/../pkg" "$FIX/fixture.h5" "$root/"
|
||||
|
||||
port=$("$PY" -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])')
|
||||
"$PY" -m http.server --bind 127.0.0.1 --directory "$root" "$port" >/dev/null 2>&1 &
|
||||
server=$!
|
||||
for _ in $(seq 50); do
|
||||
"$PY" -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:$port/index.html')" 2>/dev/null && break
|
||||
sleep 0.1
|
||||
done
|
||||
|
||||
fails=0
|
||||
# A fresh profile per page: a second instance on the same profile fails.
|
||||
render() {
|
||||
local profile
|
||||
profile="$(mktemp -d "$root/profile.XXXXXX")"
|
||||
"$chrome" --headless --no-sandbox --disable-gpu --user-data-dir="$profile" \
|
||||
--virtual-time-budget=20000 \
|
||||
--dump-dom "http://127.0.0.1:$port/index.html?file=fixture.h5&path=$1" 2>/dev/null
|
||||
}
|
||||
# expect PATH TEXT...: every TEXT appears in the page rendered for PATH.
|
||||
expect() {
|
||||
local path="$1" dom
|
||||
shift
|
||||
dom="$(render "$path")"
|
||||
[ -n "${BROWSER_DEBUG:-}" ] && printf "%s\n" "$dom" > "$BROWSER_DEBUG.$(echo "$path" | tr / _).html"
|
||||
for text in "$@"; do
|
||||
if ! grep -qF -- "$text" <<<"$dom"; then
|
||||
echo "FAIL: page for $path lacks: $text" >&2
|
||||
fails=$((fails + 1))
|
||||
fi
|
||||
done
|
||||
echo "rendered $path"
|
||||
}
|
||||
|
||||
# Tree (root expanded; the group row carries its path) and root attributes.
|
||||
expect "/" 'data-path="/sensors"' 'data-path="/grid"' '<td>title</td><td>"wasm fixture"</td>' \
|
||||
'<td>big</td><td>9223372036854775813</td>' '(compound{x: f64, n: i32})'
|
||||
# A chunked, deflated 2-D dataset: type, shape, the first window of values.
|
||||
expect "/grid" '<dd>f64</dd>' '<dd>(6, 10)</dd>' '<th>9</th>' '<td>0.25</td>' '<td>14.75</td>' \
|
||||
'showing rows 0–5, columns 0–9 of 60 values'
|
||||
# Nested path revealed through the tree; big-endian float32.
|
||||
expect "/sensors/temp" 'data-path="/sensors/temp"' '<dd>f32</dd>' '<td>21.5</td>' '<td>22.25</td>'
|
||||
# 64-bit integers stay exact; strings; array datatype cells.
|
||||
expect "/u64" '<td>18446744073709551615</td>'
|
||||
expect "/vlen_str" '<td>"двa"</td>' '<dd>vlen string</dd>'
|
||||
expect "/pairs" '<td>[2, 3]</td>' '<dd>array[2]<i32></dd>'
|
||||
# 3-D: leading dimension held at 0, window over the last two.
|
||||
expect "/cube" '<dd>(2, 5, 6)</dd>' '<td>29</td>' 'dim 0'
|
||||
# Unsupported type: an error, not values.
|
||||
expect "/table" 'class="error"' 'reading compound{x: f64, n: i32} datasets is not supported'
|
||||
|
||||
if [ "$fails" -gt 0 ]; then
|
||||
echo "browser: $fails checks failed" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "browser: all checks passed ($chrome)"
|
||||
@@ -0,0 +1,203 @@
|
||||
"""Write HDF5 and NetCDF-4 test files with h5py/netCDF4, and what libhdf5
|
||||
reads back from them, for the clawhdf5-wasm tests.
|
||||
|
||||
python make_fixture.py OUT_DIR
|
||||
|
||||
writes OUT_DIR/fixture.h5, OUT_DIR/fixture.nc and OUT_DIR/expected.json.
|
||||
Both the Rust test (crates/clawhdf5-wasm/tests/h5py_interop.rs, native) and
|
||||
the Node test (test.mjs, the built wasm package) compare against the same
|
||||
expected.json, so the two check the same values.
|
||||
|
||||
Every expected value comes from h5py reading the file back (numpy slicing
|
||||
for hyperslabs), never from the arrays that were written. Integers are
|
||||
encoded as strings so JSON.parse keeps 64-bit values exact.
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
import warnings
|
||||
from pathlib import Path
|
||||
|
||||
import h5py
|
||||
import netCDF4
|
||||
import numpy as np
|
||||
|
||||
try: # registers the LZ4/Zstd filters with libhdf5; optional
|
||||
import hdf5plugin
|
||||
except ImportError:
|
||||
hdf5plugin = None
|
||||
|
||||
# netCDF4 1.7 trips numpy 2.5's shape-setting deprecation on assignment.
|
||||
warnings.filterwarnings("ignore", category=DeprecationWarning)
|
||||
|
||||
out = Path(sys.argv[1])
|
||||
out.mkdir(parents=True, exist_ok=True)
|
||||
h5 = out / "fixture.h5"
|
||||
nc = out / "fixture.nc"
|
||||
|
||||
rng = np.random.default_rng(7)
|
||||
with h5py.File(h5, "w") as f:
|
||||
f.attrs["title"] = "wasm fixture"
|
||||
f.attrs["version"] = np.int64(3)
|
||||
f.attrs["scale"] = np.array([0.5, 2.0])
|
||||
f.attrs["big"] = np.uint64(2**63 + 5)
|
||||
f.attrs.create("vlen_note", "héllo", dtype=h5py.string_dtype())
|
||||
# No plain JavaScript form: listed with value null and its type.
|
||||
f.attrs["origin"] = np.array((1.5, 2), dtype=[("x", "<f8"), ("n", "<i4")])
|
||||
f.create_dataset(
|
||||
"grid", data=np.arange(60, dtype="<f8").reshape(6, 10) / 4,
|
||||
chunks=(4, 3), compression="gzip", shuffle=True,
|
||||
)
|
||||
f.create_dataset("f32_be", data=rng.standard_normal(7).astype(">f4"))
|
||||
f.create_dataset("f16", data=np.array([0.5, -1.25, 65504], dtype="<f2"))
|
||||
f.create_dataset("i8", data=np.array([-128, -1, 0, 127], dtype="i1"))
|
||||
f.create_dataset("i16_be", data=np.array([-32768, 5, 32767], dtype=">i2"))
|
||||
f.create_dataset("u16", data=np.array([0, 40000, 65535], dtype="<u2"))
|
||||
f.create_dataset("u32", data=np.array([0, 4_000_000_000], dtype="<u4"))
|
||||
f.create_dataset("i64", data=np.array([-(2**63), 2**53 + 1, 7], dtype="<i8"))
|
||||
f.create_dataset("u64", data=np.array([2**64 - 1, 1], dtype="<u8"))
|
||||
f.create_dataset("scalar", data=np.float64(2.5))
|
||||
f.create_dataset("fixed_str", data=np.array([b"alpha", b"be", b""], dtype="S5"))
|
||||
f.create_dataset(
|
||||
"vlen_str", data=["one", "двa", ""], dtype=h5py.string_dtype()
|
||||
)
|
||||
f.create_dataset("flags", data=np.array([True, False, True]))
|
||||
# An array datatype: four elements, each an i4[2].
|
||||
pairs = f.create_dataset("pairs", shape=(4,), dtype=np.dtype(("<i4", (2,))))
|
||||
pairs[...] = np.arange(8, dtype="<i4").reshape(4, 2)
|
||||
f.create_dataset(
|
||||
"cube", data=np.arange(2 * 5 * 6, dtype="<i4").reshape(2, 5, 6),
|
||||
chunks=(1, 2, 3), compression="gzip",
|
||||
)
|
||||
if hdf5plugin is not None:
|
||||
# LZ4 is built into clawhdf5-wasm; Zstd links C and is not.
|
||||
f.create_dataset("lz4", data=np.arange(40, dtype="<i4"), chunks=(10,),
|
||||
**hdf5plugin.LZ4())
|
||||
f.create_dataset("zstd", data=np.arange(40, dtype="<i4"), chunks=(10,),
|
||||
**hdf5plugin.Zstd())
|
||||
comp = np.zeros(2, dtype=[("x", "<f8"), ("n", "<i4")])
|
||||
f.create_dataset("table", data=comp)
|
||||
g = f.create_group("sensors")
|
||||
g.attrs["location"] = "lab"
|
||||
g.create_dataset("temp", data=np.array([21.5, 22.0, 22.25], dtype="<f4"))
|
||||
g.create_group("empty")
|
||||
f["alias"] = h5py.SoftLink("/sensors/temp")
|
||||
|
||||
with netCDF4.Dataset(nc, "w") as d:
|
||||
d.title = "nc fixture"
|
||||
d.createDimension("time", None)
|
||||
d.createDimension("x", 4)
|
||||
t = d.createVariable("time", "f8", ("time",))
|
||||
t.units = "days since 2000-01-01"
|
||||
v = d.createVariable("temp", "f4", ("time", "x"), zlib=True)
|
||||
t[:] = np.arange(3)
|
||||
v[:] = np.arange(12, dtype="f4").reshape(3, 4) + 0.5
|
||||
|
||||
|
||||
def kind(dt):
|
||||
"""The typed-array kind clawhdf5-wasm returns for a numpy dtype."""
|
||||
if dt.kind == "b" or h5py.check_enum_dtype(dt) is not None:
|
||||
return "strings"
|
||||
if h5py.check_string_dtype(dt) is not None or dt.kind == "S":
|
||||
return "strings"
|
||||
if dt.subdtype is not None:
|
||||
return kind(dt.subdtype[0])
|
||||
if dt.kind == "f":
|
||||
return "f64" if dt.itemsize == 8 else "f32"
|
||||
if dt.kind in "iu":
|
||||
return f"{dt.kind}{dt.itemsize * 8}"
|
||||
raise ValueError(dt)
|
||||
|
||||
|
||||
def flat(a, k):
|
||||
a = np.asarray(a)
|
||||
if k == "strings":
|
||||
if a.dtype.kind == "b":
|
||||
return ["TRUE" if x else "FALSE" for x in a.ravel()]
|
||||
return [x.decode() if isinstance(x, bytes) else str(x) for x in a.ravel()]
|
||||
if k.startswith(("i", "u")):
|
||||
return [str(int(x)) for x in a.ravel()]
|
||||
return [float(x) for x in a.ravel()]
|
||||
|
||||
|
||||
def entry(ds, slab=None):
|
||||
k = kind(ds.dtype)
|
||||
data = ds[()]
|
||||
e = {"kind": k, "shape": list(np.shape(data)), "values": flat(data, k)}
|
||||
if slab:
|
||||
start, count, stride = slab
|
||||
idx = tuple(slice(s, s + (c - 1) * st + 1, st)
|
||||
for s, c, st in zip(start, count, stride))
|
||||
part = ds[idx]
|
||||
e["slab"] = {"start": start, "count": count, "stride": stride,
|
||||
"shape": list(part.shape), "values": flat(part, k)}
|
||||
return e
|
||||
|
||||
|
||||
def attr(v):
|
||||
v = np.asarray(v) if not isinstance(v, (str, bytes)) else v
|
||||
if isinstance(v, np.ndarray) and v.dtype.names:
|
||||
return {"raw": "compound"}
|
||||
if isinstance(v, bytes):
|
||||
return {"string": v.decode()}
|
||||
if isinstance(v, str):
|
||||
return {"string": v}
|
||||
if v.dtype.kind in "iu":
|
||||
return {"int": [str(int(x)) for x in v.ravel()], "scalar": v.ndim == 0}
|
||||
if v.dtype.kind == "f":
|
||||
return {"float": [float(x) for x in v.ravel()], "scalar": v.ndim == 0}
|
||||
if v.dtype.kind in "OSU":
|
||||
items = [x.decode() if isinstance(x, bytes) else str(x) for x in v.ravel()]
|
||||
return {"string": items[0]} if v.ndim == 0 else {"strings": items}
|
||||
raise ValueError(v.dtype)
|
||||
|
||||
|
||||
# Attributes the reader returns but the comparison leaves out: netCDF-4's
|
||||
# internal ones (a leading underscore), and dimension-scale bookkeeping.
|
||||
SKIP_ATTRS = ["DIMENSION_LIST", "REFERENCE_LIST", "CLASS", "NAME"]
|
||||
|
||||
|
||||
def describe(path):
|
||||
expected = {"datasets": {}, "errors": {}, "attrs": {}, "lists": {},
|
||||
"skip_attrs": SKIP_ATTRS}
|
||||
with h5py.File(path, "r") as f:
|
||||
def walk(key, obj):
|
||||
expected["attrs"][key] = {
|
||||
k: attr(obj.attrs[k]) for k in obj.attrs
|
||||
if not k.startswith("_") and k not in SKIP_ATTRS}
|
||||
if isinstance(obj, h5py.Group):
|
||||
members = {n: obj.get(n) for n in obj}
|
||||
groups = [n for n, o in members.items() if isinstance(o, h5py.Group)]
|
||||
sets = [n for n, o in members.items() if isinstance(o, h5py.Dataset)]
|
||||
expected["lists"][key] = {"groups": sorted(groups),
|
||||
"datasets": sorted(sets)}
|
||||
for n, o in members.items():
|
||||
walk(key.rstrip("/") + "/" + n, o)
|
||||
elif obj.dtype.names:
|
||||
expected["errors"][key] = "compound"
|
||||
else:
|
||||
expected["datasets"][key] = entry(obj, slab_for(obj))
|
||||
if key == "/zstd":
|
||||
# Refused by the wasm build (no zstd); a native build
|
||||
# that unifies in clawhdf5-format/zstd reads it.
|
||||
expected["datasets"][key]["unavailable"] = "unsupported filter: 32015"
|
||||
|
||||
walk("/", f)
|
||||
return expected
|
||||
|
||||
|
||||
def slab_for(obj):
|
||||
"""A strided hyperslab inside the dataset's extent, or None."""
|
||||
if obj.ndim == 0 or obj.shape[0] < 2 or 0 in obj.shape:
|
||||
return None
|
||||
start = [1] + [0] * (obj.ndim - 1)
|
||||
count = [max(1, (obj.shape[0] - 1) // 2)] + [
|
||||
max(1, (n + 1) // 2) for n in obj.shape[1:]]
|
||||
stride = [2 if obj.shape[0] > 2 else 1] + [2] * (obj.ndim - 1)
|
||||
count = [min(c, (n - s - 1) // st + 1)
|
||||
for c, s, st, n in zip(count, start, stride, obj.shape)]
|
||||
return (start, count, stride)
|
||||
|
||||
|
||||
json.dump({"fixture.h5": describe(h5), "fixture.nc": describe(nc)},
|
||||
open(out / "expected.json", "w"), indent=1, ensure_ascii=False)
|
||||
Executable
+37
@@ -0,0 +1,37 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the wasm package (../build.sh), test it under Node against files
|
||||
# written by h5py and netCDF4 (make_fixture.py), then load the viewer page
|
||||
# in headless Chromium if one is found (browser.sh).
|
||||
#
|
||||
# Needs node, the wasm-bindgen CLI (see ../build.sh) and a Python with h5py,
|
||||
# netCDF4 and numpy: CLAWHDF5_PYTHON names it (default python3). Without that
|
||||
# Python the test is skipped, unless CLAWHDF5_REQUIRE_INTEROP=1.
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
PY="${CLAWHDF5_PYTHON:-python3}"
|
||||
|
||||
command -v node >/dev/null || { echo "node not found" >&2; exit 1; }
|
||||
if ! "$PY" -c "import h5py, netCDF4, numpy" >/dev/null 2>&1; then
|
||||
if [ "${CLAWHDF5_REQUIRE_INTEROP:-0}" = "1" ]; then
|
||||
echo "CLAWHDF5_REQUIRE_INTEROP=1 but $PY lacks h5py/netCDF4/numpy" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "SKIP: $PY lacks h5py/netCDF4/numpy"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
bash "$HERE/../build.sh"
|
||||
fix="$(mktemp -d)"
|
||||
trap 'rm -rf "$fix"' EXIT
|
||||
"$PY" "$HERE/make_fixture.py" "$fix"
|
||||
node "$HERE/test.mjs" "$HERE/../pkg" "$fix"
|
||||
|
||||
# The page itself, in headless Chromium when one is available.
|
||||
status=0
|
||||
bash "$HERE/browser.sh" "$fix" || status=$?
|
||||
if [ "$status" = 3 ]; then
|
||||
echo "SKIP: viewer page in a browser (no Chromium; set CHROME)"
|
||||
elif [ "$status" != 0 ]; then
|
||||
exit "$status"
|
||||
fi
|
||||
@@ -0,0 +1,143 @@
|
||||
// Node test of the built wasm package (the exact pkg/ the viewer page loads)
|
||||
// and the viewer's DOM-free helpers. Run by test/run.sh:
|
||||
// node test.mjs PKG_DIR FIXTURE_DIR
|
||||
// FIXTURE_DIR holds fixture.h5, fixture.nc and expected.json from
|
||||
// make_fixture.py (values as libhdf5 reads them back).
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
|
||||
const [pkgDir, fixDir] = process.argv.slice(2);
|
||||
const pkg = await import(pathToFileURL(join(pkgDir, "clawhdf5_wasm.js")));
|
||||
pkg.initSync({ module: readFileSync(join(pkgDir, "clawhdf5_wasm_bg.wasm")) });
|
||||
const lib = await import(pathToFileURL(join(import.meta.dirname, "..", "viewer-lib.js")));
|
||||
|
||||
let checks = 0;
|
||||
const eq = (a, b, msg) => { assert.deepEqual(a, b, msg); checks++; };
|
||||
|
||||
const ARRAY_TYPES = {
|
||||
f32: Float32Array, f64: Float64Array, i8: Int8Array, i16: Int16Array, i32: Int32Array,
|
||||
i64: BigInt64Array, u8: Uint8Array, u16: Uint16Array, u32: Uint32Array, u64: BigUint64Array,
|
||||
strings: Array,
|
||||
};
|
||||
|
||||
function values(kind, data) {
|
||||
const arr = Array.from(data);
|
||||
if (kind === "f32" || kind === "f64" || kind === "strings") return arr;
|
||||
return arr.map(String);
|
||||
}
|
||||
|
||||
function checkAttr(ctx, a, want) {
|
||||
const v = a.value;
|
||||
if ("raw" in want) {
|
||||
eq(v, null, ctx);
|
||||
assert.ok(a.dtype.includes(want.raw), `${ctx}: ${a.dtype}`);
|
||||
return;
|
||||
}
|
||||
eq(a.dtype, null, `${ctx} dtype`);
|
||||
if ("string" in want) return eq(v, want.string, ctx);
|
||||
if ("strings" in want) return eq(v, want.strings, ctx);
|
||||
if ("int" in want) {
|
||||
if (want.scalar) {
|
||||
assert.ok(typeof v === "number" || typeof v === "bigint", ctx);
|
||||
return eq([String(v)], want.int, ctx);
|
||||
}
|
||||
assert.ok(v instanceof BigInt64Array || v instanceof BigUint64Array, ctx);
|
||||
return eq(Array.from(v, String), want.int, ctx);
|
||||
}
|
||||
if ("float" in want) {
|
||||
if (want.scalar) return eq([v], want.float, ctx);
|
||||
assert.ok(v instanceof Float64Array, ctx);
|
||||
return eq(Array.from(v), want.float, ctx);
|
||||
}
|
||||
assert.fail(`${ctx}: unknown expectation ${JSON.stringify(want)}`);
|
||||
}
|
||||
|
||||
const expected = JSON.parse(readFileSync(join(fixDir, "expected.json"), "utf8"));
|
||||
for (const [name, exp] of Object.entries(expected)) {
|
||||
const file = pkg.open(new Uint8Array(readFileSync(join(fixDir, name))));
|
||||
|
||||
for (const [path, want] of Object.entries(exp.lists)) {
|
||||
eq(file.kind(path), "group", `${name}:${path} kind`);
|
||||
const list = file.list(path);
|
||||
for (const [kind, key] of [["group", "groups"], ["dataset", "datasets"]]) {
|
||||
eq(list.filter((c) => c.kind === kind).map((c) => c.name).sort(), want[key], `${name}:${path} ${key}`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [path, want] of Object.entries(exp.datasets)) {
|
||||
const ctx = `${name}:${path}`;
|
||||
eq(file.kind(path), "dataset", `${ctx} kind`);
|
||||
if (want.unavailable) {
|
||||
// The wasm build has no zstd (it links C): a clear error, no data.
|
||||
assert.throws(() => file.read(path), (e) => e.message.includes(want.unavailable), ctx);
|
||||
checks++;
|
||||
continue;
|
||||
}
|
||||
const info = file.info(path);
|
||||
eq([...info.shape, ...info.elementShape], want.shape, `${ctx} info shape`);
|
||||
const r = file.read(path);
|
||||
eq(r.shape, want.shape, `${ctx} shape`);
|
||||
eq(r.dtype, info.dtype, `${ctx} dtype`);
|
||||
assert.ok(r.data instanceof ARRAY_TYPES[want.kind], `${ctx}: ${r.data.constructor.name} for ${want.kind}`);
|
||||
eq(values(want.kind, r.data), want.values, ctx);
|
||||
if (want.slab) {
|
||||
const s = want.slab;
|
||||
const part = file.readHyperslab(path, s.start, s.count, s.stride);
|
||||
eq(part.shape, s.shape, `${ctx} slab shape`);
|
||||
eq(values(want.kind, part.data), s.values, `${ctx} slab`);
|
||||
}
|
||||
}
|
||||
|
||||
for (const [path, what] of Object.entries(exp.errors)) {
|
||||
assert.throws(() => file.read(path), (e) => e instanceof Error && e.message.includes(what), `${name}:${path}`);
|
||||
checks++;
|
||||
}
|
||||
|
||||
for (const [path, want] of Object.entries(exp.attrs)) {
|
||||
const attrs = file.attrs(path);
|
||||
eq(file.attrErrors(path), [], `${name}:${path} attr errors`);
|
||||
const seen = attrs.filter((a) => !a.name.startsWith("_") && !exp.skip_attrs.includes(a.name));
|
||||
eq(seen.map((a) => a.name).sort(), Object.keys(want).sort(), `${name}:${path} attr names`);
|
||||
for (const a of seen) checkAttr(`${name}:${path}@${a.name}`, a, want[a.name]);
|
||||
}
|
||||
file.free();
|
||||
}
|
||||
|
||||
// Errors reach JavaScript as thrown Errors, never as data.
|
||||
const h5 = pkg.open(new Uint8Array(readFileSync(join(fixDir, "fixture.h5"))));
|
||||
const throwsMsg = (fn, re) => { assert.throws(fn, (e) => e instanceof Error && re.test(e.message)); checks++; };
|
||||
throwsMsg(() => pkg.open(new Uint8Array(64)), /./);
|
||||
throwsMsg(() => h5.read("/nope"), /./);
|
||||
throwsMsg(() => h5.list("/grid"), /not a group/);
|
||||
throwsMsg(() => h5.readHyperslab("/grid", [0], [1]), /dimensions/);
|
||||
throwsMsg(() => h5.readHyperslab("/grid", [5, 0], [2, 1]), /exceeds/);
|
||||
throwsMsg(() => h5.readHyperslab("/grid", [-1, 0], [1, 1]), /non-negative integers/);
|
||||
throwsMsg(() => h5.readHyperslab("/grid", [0.5, 0], [1, 1]), /non-negative integers/);
|
||||
// Info for a dataset with an unlimited dimension (netCDF "time").
|
||||
const nc = pkg.open(new Uint8Array(readFileSync(join(fixDir, "fixture.nc"))));
|
||||
eq(nc.info("/time").maxshape, [null], "unlimited dimension is null");
|
||||
// Big integers stay exact.
|
||||
eq(h5.read("/u64").data[0], 18446744073709551615n, "u64 max");
|
||||
eq(typeof pkg.version(), "string", "version");
|
||||
|
||||
// Viewer helpers.
|
||||
eq(lib.joinPath("/", "a"), "/a", "joinPath root");
|
||||
eq(lib.joinPath("/a", "b"), "/a/b", "joinPath nested");
|
||||
eq(lib.viewWindow([], {}), null, "scalar window");
|
||||
eq(lib.viewWindow([7], { row: 5, rows: 50 }), { start: [5], count: [2], rows: 2, cols: 1, row: 5, col: 0 }, "1-D window");
|
||||
const w = lib.viewWindow([2, 5, 6], { row: 1, col: 4, rows: 3, cols: 5, fixed: [1] });
|
||||
eq(w, { start: [1, 1, 4], count: [1, 3, 2], rows: 3, cols: 2, row: 1, col: 4 }, "3-D window");
|
||||
// The window the page would request reads the same values as a direct slab.
|
||||
const cube = h5.readHyperslab("/cube", w.start, w.count);
|
||||
eq(lib.toRows(cube.data, w.rows, w.cols), [["40", "41"], ["46", "47"], ["52", "53"]], "cube window cells");
|
||||
const pairs = h5.readHyperslab("/pairs", [1], [2]);
|
||||
eq(lib.toRows(pairs.data, 2, 1, lib.perElement([2])), [["[2, 3]"], ["[4, 5]"]], "array-type cells");
|
||||
eq(lib.formatValue(0.1 + 0.2), "0.3", "float formatting");
|
||||
eq(lib.formatValue(2n ** 64n - 1n), "18446744073709551615", "bigint formatting");
|
||||
eq(lib.formatValue("x"), '"x"', "string formatting");
|
||||
h5.free();
|
||||
nc.free();
|
||||
|
||||
console.log(`wasm package: ${checks} checks passed`);
|
||||
@@ -0,0 +1,77 @@
|
||||
// DOM-free helpers for the viewer, so Node can test them (test/test.mjs).
|
||||
|
||||
/** Child path of `name` in the group at `parent`. */
|
||||
export function joinPath(parent, name) {
|
||||
return parent === "/" ? `/${name}` : `${parent}/${name}`;
|
||||
}
|
||||
|
||||
/** One value as display text. */
|
||||
export function formatValue(v) {
|
||||
if (v === null || v === undefined) return "";
|
||||
if (typeof v === "bigint") return v.toString();
|
||||
if (typeof v === "number") {
|
||||
if (Number.isInteger(v)) return String(v);
|
||||
return String(Number(v.toPrecision(7)));
|
||||
}
|
||||
if (typeof v === "string") return JSON.stringify(v);
|
||||
if (ArrayBuffer.isView(v) || Array.isArray(v)) {
|
||||
const items = Array.from(v.slice(0, 16), formatValue);
|
||||
if (v.length > 16) items.push(`… (${v.length} values)`);
|
||||
return `[${items.join(", ")}]`;
|
||||
}
|
||||
return String(v);
|
||||
}
|
||||
|
||||
/**
|
||||
* The window of a dataset to show: a hyperslab over its `shape` with the
|
||||
* last dimension as columns, the one before as rows, and any leading
|
||||
* dimensions held at `fixed` indices. `row`/`col` are the window's top-left
|
||||
* corner. Returns null for a scalar (read it whole).
|
||||
*/
|
||||
export function viewWindow(shape, { row = 0, col = 0, rows = 50, cols = 12, fixed = [] } = {}) {
|
||||
const rank = shape.length;
|
||||
if (rank === 0) return null;
|
||||
const clamp = (x, n) => Math.max(0, Math.min(x, Math.max(0, n - 1)));
|
||||
if (rank === 1) {
|
||||
const r0 = clamp(row, shape[0]);
|
||||
const n = Math.max(0, Math.min(rows, shape[0] - r0));
|
||||
return { start: [r0], count: [n], rows: n, cols: 1, row: r0, col: 0 };
|
||||
}
|
||||
const lead = shape.slice(0, rank - 2).map((n, i) => clamp(fixed[i] ?? 0, n));
|
||||
const nr = shape[rank - 2];
|
||||
const nc = shape[rank - 1];
|
||||
const r0 = clamp(row, nr);
|
||||
const c0 = clamp(col, nc);
|
||||
const r = Math.max(0, Math.min(rows, nr - r0));
|
||||
const c = Math.max(0, Math.min(cols, nc - c0));
|
||||
return {
|
||||
start: [...lead, r0, c0],
|
||||
count: [...lead.map(() => 1), r, c],
|
||||
rows: r,
|
||||
cols: c,
|
||||
row: r0,
|
||||
col: c0,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Split row-major `data` into `rows` x `cols` cells of display text; each
|
||||
* cell holds `per` consecutive values (the elements of an array datatype).
|
||||
*/
|
||||
export function toRows(data, rows, cols, per = 1) {
|
||||
const out = [];
|
||||
for (let r = 0; r < rows; r++) {
|
||||
const row = [];
|
||||
for (let c = 0; c < cols; c++) {
|
||||
const i = (r * cols + c) * per;
|
||||
row.push(per === 1 ? formatValue(data[i]) : formatValue(data.slice(i, i + per)));
|
||||
}
|
||||
out.push(row);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Number of values an array datatype packs into each element. */
|
||||
export function perElement(elementShape) {
|
||||
return elementShape.reduce((a, b) => a * b, 1);
|
||||
}
|
||||
+32
-2
@@ -108,14 +108,17 @@ run_step "cargo clippy (fast-deflate / zlib-ng)" cargo clippy \
|
||||
# that: fail if a crate that compiles C (a *-sys crate, cc or cmake) enters the
|
||||
# default dependency tree of any of them. clawhdf5-migrate (bundled SQLite),
|
||||
# clawhdf5-napi (Node) and clawhdf5-gpu (graphics drivers) are exempt.
|
||||
# js-sys (clawhdf5-wasm's bindings to JavaScript) builds no C.
|
||||
no_c_in_default_build() {
|
||||
local crate found=0
|
||||
for crate in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \
|
||||
clawhdf5-agent clawhdf5-ann clawhdf5-accel clawhdf5-netcdf4 clawhdf5-cli \
|
||||
clawhdf5-tools; do
|
||||
clawhdf5-tools \
|
||||
clawhdf5-wasm; do
|
||||
local c_deps
|
||||
c_deps=$(cargo tree -q -p "$crate" -e normal,build --prefix none \
|
||||
| grep -E '^([a-z0-9_-]+-sys|cc|cmake) v' | sort -u)
|
||||
| grep -E '^([a-z0-9_-]+-sys|cc|cmake) v' \
|
||||
| grep -v '^js-sys v' | sort -u)
|
||||
if [ -n "$c_deps" ]; then
|
||||
echo "$crate pulls in C by default:"
|
||||
echo "$c_deps" | sed 's/^/ /'
|
||||
@@ -126,6 +129,33 @@ no_c_in_default_build() {
|
||||
}
|
||||
run_step "no C in the default build (core crates)" no_c_in_default_build
|
||||
|
||||
# The reader in the browser: the facade's read path must build for
|
||||
# wasm32-unknown-unknown (no mmap, no threads, no file system), and the
|
||||
# wasm-bindgen crate must build and lint there. Needs
|
||||
# `rustup target add wasm32-unknown-unknown`.
|
||||
run_step "wasm32 build (clawhdf5, no default features)" cargo build \
|
||||
-p clawhdf5 \
|
||||
--target wasm32-unknown-unknown \
|
||||
--no-default-features
|
||||
run_step "wasm32 clippy (clawhdf5-wasm)" cargo clippy \
|
||||
-p clawhdf5-wasm \
|
||||
--target wasm32-unknown-unknown \
|
||||
--all-targets \
|
||||
-- -D warnings
|
||||
|
||||
# The built wasm package, run under Node against h5py/netCDF4-written files,
|
||||
# and the viewer page in headless Chromium when one is found.
|
||||
# Needs node and the wasm-bindgen CLI, which the CI container does not have;
|
||||
# the same expectations are checked natively by clawhdf5-wasm's h5py_interop
|
||||
# test in the cargo test step.
|
||||
if command -v node >/dev/null && command -v wasm-bindgen >/dev/null; then
|
||||
run_step "wasm package under Node (+ browser)" bash "$SCRIPT_DIR/../examples/wasm-viewer/test/run.sh"
|
||||
else
|
||||
echo ""
|
||||
echo "==> [wasm package under Node] SKIPPED: needs node and wasm-bindgen"
|
||||
STEPS+=("SKIP: wasm package under Node")
|
||||
fi
|
||||
|
||||
# The workspace declares a minimum Rust version (rust-version in Cargo.toml);
|
||||
# check that it really builds there, so the README badge and the manifests
|
||||
# cannot drift from the truth. Separate target dir: a different toolchain
|
||||
|
||||
Reference in New Issue
Block a user