h5rs tools, browser reader, libhdf5 header checks, plugin filters, concurrency benchmark #14

Merged
osobh merged 60 commits from feat/p1-proof into main 2026-09-26 13:14:39 +00:00
22 changed files with 2253 additions and 5 deletions
Showing only changes of commit f7c362cef5 - Show all commits
+3
View File
@@ -22,6 +22,9 @@ jobs:
run: rustup component add rustfmt clippy
- name: Install thumbv7em-none-eabihf target
run: rustup target add thumbv7em-none-eabihf
- name: Install wasm32-unknown-unknown target
# ci-test.sh builds the reader and clawhdf5-wasm for the browser.
run: rustup target add wasm32-unknown-unknown
- name: Install Python interop dependencies
# The interop suites used to skip silently when python3/h5py were
# missing, so they never ran in CI. Install them and make a missing
+26
View File
@@ -397,6 +397,32 @@
infinity; batches are all or nothing. CLI: `create --float16`. See
`BENCHMARKS.md`, "float16 embedding storage".
### Browser (WebAssembly)
- **New crate `clawhdf5-wasm`:** the reader compiled to
`wasm32-unknown-unknown` with a wasm-bindgen JavaScript API —
`open(bytes)`, `list`, `info`, `attrs`, `read`, `readHyperslab` — returning
typed arrays of the stored width (`BigInt64Array` for 64-bit integers),
string arrays for strings and enums, and a thrown `Error` for types with no
typed-array form (compound, reference, opaque, VL sequences) or filters the
build lacks (Zstd, SZIP). Read-only; the file is held in memory.
- **`examples/wasm-viewer/`:** a drop-a-file HDF5/NetCDF-4 viewer page (tree,
type/shape/attributes, values paged as hyperslabs; `?file=&path=` opens a
URL). `build.sh` produces the package; `test/run.sh` checks it under Node
(251 checks against values h5py/libhdf5 read back from an h5py- and a
netCDF4-written file) and renders the page in headless Chromium. Size,
measured 2026-09-26 on tank (`gzip -9 -n`): 627,501 B of wasm, 191,639 B
gzipped, plus 21,826 B (4,487 B) of JS glue; h5wasm 0.10.3's embedded wasm
is 3,544,184 B (907,096 B) — full libhdf5, so not equal functionality. See
`examples/wasm-viewer/README.md`.
- The facade's read path already built for `wasm32-unknown-unknown` (nothing
needed gating); `ci-test.sh` now builds it (`--no-default-features`) and
lints `clawhdf5-wasm` for that target, and CI installs the target. The Node
and browser tests run in `ci-test.sh` only where `node` and `wasm-bindgen`
exist (not the CI container); CI checks the same expectations natively
(`clawhdf5-wasm`'s `h5py_interop` test).
- `Dataset::raw_datatype()` (facade) returns the full stored datatype, for
decoding `read_selection` bytes with `clawhdf5_format::data_read`.
### Build
- **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
`clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
+10 -1
View File
@@ -5,7 +5,7 @@ Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persist
## Architecture
Cargo workspace with 17 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
Cargo workspace with 18 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
| Crate | Role |
|-------|------|
@@ -25,6 +25,7 @@ Cargo workspace with 17 crates under `crates/` (plus `libaec-sys`, an internal F
| `clawhdf5-tools` | `h5rs`: pure-Rust HDF5 tools — `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` (structural + checksum validator) |
| `clawhdf5-napi` | Node.js native addon bindings |
| `clawhdf5-py` | PyO3 Python bindings |
| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
| `clawhdf5-bench` | Benchmark suite |
## Key Features
@@ -150,6 +151,14 @@ Cargo workspace with 17 crates under `crates/` (plus `libaec-sys`, an internal F
`MemorySource` for this bookkeeping is inferred from the caller-supplied
`source_channel` string (a heuristic, not an authenticated trust boundary).
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.
`examples/wasm-viewer/test/run.sh` builds the package (needs the
`wasm-bindgen` CLI at the crate's exact version) and tests it under Node
and headless Chromium (a Playwright download in `~/.cache/ms-playwright`
on tank); the CI container has neither, so CI runs the native
`clawhdf5-wasm` `h5py_interop` test on the same fixture. Size numbers are
in the example's README.
- Python and Node.js bindings for cross-language use
- NetCDF-4 compatibility for scientific data interop
+10
View File
@@ -17,6 +17,7 @@ members = [
"crates/clawhdf5-napi",
"crates/clawhdf5-bench",
"crates/clawhdf5-tools",
"crates/clawhdf5-wasm",
"crates/libaec-sys",
]
resolver = "2"
@@ -35,3 +36,12 @@ tempfile = "3"
criterion = { version = "0.5", features = ["html_reports"] }
half = "2.7"
serde = { version = "1", features = ["derive"] }
# The browser build of clawhdf5-wasm (examples/wasm-viewer/build.sh): size
# over speed, whole-program optimisation. Native profiles are unaffected.
[profile.wasm-release]
inherits = "release"
opt-level = "s"
lto = true
codegen-units = 1
panic = "abort"
+3 -2
View File
@@ -587,7 +587,7 @@ let exported = backend.export_markdown("MEMORY.md")?;
## Crate Map
```
clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
clawhdf5 workspace (17 crates, ~86K lines of Rust in src/, ~104K with tests
and benches; plus libaec-sys, an internal FFI bindings
crate for the optional szip feature)
│
@@ -610,7 +610,8 @@ clawhdf5 workspace (16 crates, ~86K lines of Rust in src/, ~104K with tests
│
├── Bindings
│ ├── clawhdf5-py — Python (PyO3)
│ └── clawhdf5-napi — Node.js (napi-rs)
│ ├── clawhdf5-napi — Node.js (napi-rs)
│ └── clawhdf5-wasm — Browser (WebAssembly, wasm-bindgen; read-only)
│
└── Tooling
└── clawhdf5-bench — Benchmark suite
+30
View File
@@ -0,0 +1,30 @@
[package]
name = "clawhdf5-wasm"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
description = "Read HDF5 and NetCDF-4 files in the browser: clawhdf5's reader compiled to WebAssembly"
license.workspace = true
repository.workspace = true
keywords = ["hdf5", "netcdf", "wasm", "browser"]
categories = ["wasm", "parser-implementations", "science"]
# Distributed as the wasm-bindgen package built by examples/wasm-viewer/build.sh.
publish = false
[lib]
# cdylib for wasm-bindgen; rlib so the pure-Rust core is tested natively.
crate-type = ["cdylib", "rlib"]
[dependencies]
# No mmap (there is no file system) and no `parallel` (no threads on
# wasm32-unknown-unknown). lz4 is pure Rust; zstd and szip link C and are
# left out, so such datasets fail with a clear filter error.
clawhdf5 = { path = "../clawhdf5", version = "2.7.0", default-features = false, features = ["lz4"] }
clawhdf5-format = { path = "../clawhdf5-format", version = "2.7.0" }
# Must match the wasm-bindgen CLI exactly; build.sh checks.
wasm-bindgen = "0.2.129"
js-sys = "0.3.106"
[dev-dependencies]
serde_json = "1"
tempfile = { workspace = true }
+603
View File
@@ -0,0 +1,603 @@
//! The reader behind the JavaScript API, in plain Rust so it is tested
//! natively. The `wasm_bindgen` layer in `lib.rs` only converts these types
//! to JavaScript values.
//!
//! Every read either returns the dataset's values or an error: a datatype
//! with no typed-array mapping (compound, reference, opaque, ...) is refused
//! with a message naming it, never returned as reinterpreted bytes.
use clawhdf5::{AttrValue, File, Selection};
use clawhdf5_format::data_read;
use clawhdf5_format::datatype::{Datatype, DatatypeByteOrder};
/// Errors are reported to JavaScript as messages.
pub type Result<T> = std::result::Result<T, String>;
fn err(e: impl std::fmt::Display) -> String {
e.to_string()
}
/// What a path names.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Kind {
Group,
Dataset,
}
impl Kind {
pub fn as_str(self) -> &'static str {
match self {
Kind::Group => "group",
Kind::Dataset => "dataset",
}
}
}
/// One entry of a group listing.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Child {
pub name: String,
pub kind: Kind,
}
/// A dataset's metadata.
#[derive(Debug, Clone, PartialEq)]
pub struct DatasetInfo {
/// Dataspace dimensions (empty for a scalar).
pub shape: Vec<u64>,
/// Maximum dimensions, `None` per unlimited dimension; `None` overall
/// when the dataspace records none.
pub maxshape: Option<Vec<Option<u64>>>,
/// Human-readable datatype, e.g. `f64`, `i16 (big-endian)`, `string[8]`.
pub dtype: String,
/// Dimensions of an array datatype's elements, appended to the shape of
/// what [`Reader::read`] returns (empty otherwise).
pub element_shape: Vec<u64>,
}
/// Decoded values, one variant per JavaScript typed array.
#[derive(Debug, Clone, PartialEq)]
pub enum Data {
F32(Vec<f32>),
F64(Vec<f64>),
I8(Vec<i8>),
I16(Vec<i16>),
I32(Vec<i32>),
I64(Vec<i64>),
U8(Vec<u8>),
U16(Vec<u16>),
U32(Vec<u32>),
U64(Vec<u64>),
/// Fixed- and variable-length strings, and enumeration member names.
Strings(Vec<String>),
}
impl Data {
pub fn len(&self) -> usize {
match self {
Data::F32(v) => v.len(),
Data::F64(v) => v.len(),
Data::I8(v) => v.len(),
Data::I16(v) => v.len(),
Data::I32(v) => v.len(),
Data::I64(v) => v.len(),
Data::U8(v) => v.len(),
Data::U16(v) => v.len(),
Data::U32(v) => v.len(),
Data::U64(v) => v.len(),
Data::Strings(v) => v.len(),
}
}
pub fn is_empty(&self) -> bool {
self.len() == 0
}
}
/// Values in row-major order with their shape.
#[derive(Debug, Clone, PartialEq)]
pub struct Array {
pub shape: Vec<u64>,
pub data: Data,
}
/// A regular hyperslab, as in `H5Sselect_hyperslab`. `stride` and `block`
/// default to 1 in every dimension.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Hyperslab {
pub start: Vec<u64>,
pub count: Vec<u64>,
pub stride: Option<Vec<u64>>,
pub block: Option<Vec<u64>>,
}
/// An attribute: its value, or why it has none.
#[derive(Debug, Clone)]
pub struct Attr {
pub name: String,
pub value: AttrValue,
}
/// An open file, held in memory.
pub struct Reader {
file: File,
}
impl Reader {
/// Parse a file from its bytes (the browser hands over the whole file).
pub fn open(bytes: Vec<u8>) -> Result<Self> {
Ok(Self {
file: File::from_bytes(bytes).map_err(err)?,
})
}
/// Whether `path` names a group or a dataset.
pub fn kind(&self, path: &str) -> Result<Kind> {
match self.file.dataset(path) {
Ok(_) => Ok(Kind::Dataset),
Err(clawhdf5::Error::NotADataset(_)) => Ok(Kind::Group),
Err(e) => Err(err(e)),
}
}
/// The groups, then the datasets, in the group at `path` (`/` is the
/// root). Soft links are listed as their targets; external and dangling
/// links, and named datatypes, are left out.
pub fn list(&self, path: &str) -> Result<Vec<Child>> {
if self.kind(path)? != Kind::Group {
return Err(format!("not a group: {path}"));
}
let group = self.file.group(path).map_err(err)?;
let mut out: Vec<Child> = group
.groups()
.map_err(err)?
.into_iter()
.map(|name| Child {
name,
kind: Kind::Group,
})
.collect();
out.extend(
group
.datasets()
.map_err(err)?
.into_iter()
.map(|name| Child {
name,
kind: Kind::Dataset,
}),
);
Ok(out)
}
/// Shape, max shape and datatype of the dataset at `path`.
pub fn info(&self, path: &str) -> Result<DatasetInfo> {
let ds = self.file.dataset(path).map_err(err)?;
let dt = ds.raw_datatype().map_err(err)?;
let maxshape = ds.max_dimensions().map_err(err)?.map(|dims| {
dims.into_iter()
.map(|d| (d != u64::MAX).then_some(d))
.collect()
});
Ok(DatasetInfo {
shape: ds.shape().map_err(err)?,
maxshape,
dtype: describe(&dt),
element_shape: element_shape(&dt),
})
}
/// The attributes of the group or dataset at `path`, sorted by name, and
/// one message per attribute that could not be read at all. An attribute
/// whose type has no plain JavaScript form is returned as
/// [`AttrValue::Raw`].
pub fn attrs(&self, path: &str) -> Result<(Vec<Attr>, Vec<String>)> {
let (map, errors) = match self.kind(path)? {
Kind::Dataset => self
.file
.dataset(path)
.and_then(|d| d.attrs_with_errors())
.map_err(err)?,
Kind::Group => self
.file
.group(path)
.and_then(|g| g.attrs_with_errors())
.map_err(err)?,
};
let mut attrs: Vec<Attr> = map
.into_iter()
.map(|(name, value)| Attr { name, value })
.collect();
attrs.sort_by(|a, b| a.name.cmp(&b.name));
Ok((attrs, errors.into_iter().map(err).collect()))
}
/// Read the dataset at `path`, whole or a hyperslab of it.
pub fn read(&self, path: &str, slab: Option<&Hyperslab>) -> Result<Array> {
let ds = self.file.dataset(path).map_err(err)?;
let dt = ds.raw_datatype().map_err(err)?;
let shape = ds.shape().map_err(err)?;
let (selection, mut out_shape) = match slab {
None => (Selection::All, shape.clone()),
Some(h) => hyperslab_selection(h, &shape)?,
};
let raw = ds.read_selection(&selection).map_err(err)?;
let data = self.decode(&raw, &dt)?;
out_shape.extend(element_shape(&dt));
let expected = out_shape
.iter()
.try_fold(1u64, |acc, &d| acc.checked_mul(d))
.ok_or("selection size overflows")?;
if data.len() as u64 != expected {
return Err(format!(
"read {} values for shape {out_shape:?} ({expected} expected)",
data.len()
));
}
Ok(Array {
shape: out_shape,
data,
})
}
fn decode(&self, raw: &[u8], dt: &Datatype) -> Result<Data> {
let base = array_base(dt);
let is_array = !std::ptr::eq(base, dt);
Ok(match base {
Datatype::FloatingPoint { size, .. } if *size <= 4 => {
Data::F32(data_read::read_as_f32(raw, dt).map_err(err)?)
}
Datatype::FloatingPoint { .. } => {
Data::F64(data_read::read_as_f64(raw, dt).map_err(err)?)
}
Datatype::FixedPoint { size, signed, .. } => {
let signed_ints = || data_read::read_as_i64(raw, dt).map_err(err);
let unsigned_ints = || data_read::read_as_u64(raw, dt).map_err(err);
match (size, signed) {
(1, true) => Data::I8(narrow(signed_ints()?)?),
(2, true) => Data::I16(narrow(signed_ints()?)?),
(4, true) => Data::I32(narrow(signed_ints()?)?),
(_, true) => Data::I64(signed_ints()?),
(1, false) => Data::U8(narrow(unsigned_ints()?)?),
(2, false) => Data::U16(narrow(unsigned_ints()?)?),
(4, false) => Data::U32(narrow(unsigned_ints()?)?),
(_, false) => Data::U64(unsigned_ints()?),
}
}
Datatype::String { .. } if !is_array => {
Data::Strings(data_read::read_as_strings(raw, dt).map_err(err)?)
}
Datatype::VariableLength {
is_string: true, ..
} if !is_array => {
let size = dt.type_size() as usize;
if size == 0 || !raw.len().is_multiple_of(size) {
return Err(format!(
"{} bytes is not a whole number of {size}-byte string references",
raw.len()
));
}
let sb = self.file.superblock();
Data::Strings(
clawhdf5_format::vl_data::read_vl_strings(
self.file.as_bytes(),
raw,
(raw.len() / size) as u64,
sb.offset_size,
sb.length_size,
)
.map_err(err)?,
)
}
Datatype::Enumeration { .. } if !is_array => {
Data::Strings(data_read::read_enum_names(raw, dt).map_err(err)?)
}
_ => {
return Err(format!(
"reading {} datasets is not supported",
describe(dt)
));
}
})
}
}
/// Narrow integers read at 64 bits to the dataset's own width. The source is
/// that width, so this cannot fail on correct input; it is checked anyway.
fn narrow<S: Copy + std::fmt::Display, T: TryFrom<S>>(v: Vec<S>) -> Result<Vec<T>> {
v.into_iter()
.map(|x| T::try_from(x).map_err(|_| format!("value {x} out of range")))
.collect()
}
/// Innermost element type of (possibly nested) array datatypes.
fn array_base(dt: &Datatype) -> &Datatype {
match dt {
Datatype::Array { base_type, .. } => array_base(base_type),
_ => dt,
}
}
fn element_shape(dt: &Datatype) -> Vec<u64> {
match dt {
Datatype::Array {
base_type,
dimensions,
} => {
let mut dims: Vec<u64> = dimensions.iter().map(|&d| u64::from(d)).collect();
dims.extend(element_shape(base_type));
dims
}
_ => Vec::new(),
}
}
fn hyperslab_selection(h: &Hyperslab, shape: &[u64]) -> Result<(Selection, Vec<u64>)> {
let rank = shape.len();
let ones = vec![1u64; rank];
let stride = h.stride.clone().unwrap_or_else(|| ones.clone());
let block = h.block.clone().unwrap_or(ones);
for (what, v) in [
("start", &h.start),
("count", &h.count),
("stride", &stride),
("block", &block),
] {
if v.len() != rank {
return Err(format!(
"hyperslab {what} has {} dimensions, the dataset has {rank}",
v.len()
));
}
}
let mut out = Vec::with_capacity(rank);
for d in 0..rank {
if stride[d] == 0 || block[d] == 0 {
return Err(format!("hyperslab stride and block must be >= 1 (dim {d})"));
}
if h.count[d] > 1 && block[d] > stride[d] {
return Err(format!(
"hyperslab blocks overlap in dim {d}: block {} > stride {}",
block[d], stride[d]
));
}
// Last element selected: start + (count-1)*stride + block - 1.
if h.count[d] > 0 {
let last = (h.count[d] - 1)
.checked_mul(stride[d])
.and_then(|x| x.checked_add(h.start[d]))
.and_then(|x| x.checked_add(block[d] - 1));
match last {
Some(l) if l < shape[d] => {}
_ => {
return Err(format!(
"hyperslab exceeds dimension {d} (extent {})",
shape[d]
));
}
}
}
out.push(
h.count[d]
.checked_mul(block[d])
.ok_or("selection size overflows")?,
);
}
Ok((
Selection::Hyperslab {
start: h.start.clone(),
stride,
count: h.count.clone(),
block,
},
out,
))
}
/// A short, human-readable datatype name.
pub fn describe(dt: &Datatype) -> String {
fn endian(order: &DatatypeByteOrder) -> &'static str {
match order {
DatatypeByteOrder::BigEndian => " (big-endian)",
DatatypeByteOrder::Vax => " (VAX)",
_ => "",
}
}
match dt {
Datatype::FixedPoint {
size,
signed,
byte_order,
..
} => format!(
"{}{}{}",
if *signed { "i" } else { "u" },
size * 8,
endian(byte_order)
),
Datatype::FloatingPoint {
size, byte_order, ..
} => format!("f{}{}", size * 8, endian(byte_order)),
Datatype::Time { size, .. } => format!("time{}", size * 8),
Datatype::String { size, .. } => format!("string[{size}]"),
Datatype::BitField { size, .. } => format!("bitfield{}", size * 8),
Datatype::Opaque { size, .. } => format!("opaque[{size}]"),
Datatype::Compound { members, .. } => {
let fields: Vec<String> = members
.iter()
.map(|m| format!("{}: {}", m.name, describe(&m.datatype)))
.collect();
format!("compound{{{}}}", fields.join(", "))
}
Datatype::Reference { .. } => "reference".to_string(),
Datatype::Enumeration {
base_type, members, ..
} => {
let names: Vec<&str> = members.iter().map(|m| m.name.as_str()).collect();
format!("enum<{}>{{{}}}", describe(base_type), names.join(", "))
}
Datatype::VariableLength {
is_string: true, ..
} => "vlen string".to_string(),
Datatype::VariableLength { base_type, .. } => {
format!("vlen<{}>", describe(base_type))
}
Datatype::Array {
base_type,
dimensions,
} => format!("array{dimensions:?}<{}>", describe(base_type)),
}
}
#[cfg(test)]
mod tests {
use super::*;
use clawhdf5::FileBuilder;
fn sample() -> Reader {
let mut b = FileBuilder::new();
b.create_dataset("grid")
.with_f64_data(&(0..12).map(f64::from).collect::<Vec<_>>())
.with_shape(&[3, 4])
.with_chunks(&[2, 2])
.with_deflate(4);
b.create_dataset("bytes").with_u8_data(&[1, 2, 250]);
let mut g = b.create_group("sensors");
g.create_dataset("temp").with_f32_data(&[1.5, -2.25]);
g.set_attr("location", AttrValue::String("lab".into()));
b.add_group(g.finish());
b.set_attr("version", AttrValue::I64(3));
b.set_attr("scale", AttrValue::F64Array(vec![0.5, 2.0]));
Reader::open(b.finish().unwrap()).unwrap()
}
#[test]
fn lists_groups_then_datasets() {
let r = sample();
let names: Vec<(String, Kind)> = r
.list("/")
.unwrap()
.into_iter()
.map(|c| (c.name, c.kind))
.collect();
assert_eq!(names[0], ("sensors".to_string(), Kind::Group));
let mut ds: Vec<&str> = names[1..].iter().map(|(n, _)| n.as_str()).collect();
ds.sort();
assert_eq!(ds, ["bytes", "grid"]);
assert_eq!(
r.list("sensors").unwrap(),
vec![Child {
name: "temp".into(),
kind: Kind::Dataset
}]
);
assert!(r.list("grid").unwrap_err().contains("not a group"));
assert!(r.list("missing").is_err());
}
#[test]
fn info_reports_shape_and_dtype() {
let r = sample();
let i = r.info("grid").unwrap();
assert_eq!(i.shape, vec![3, 4]);
assert_eq!(i.dtype, "f64");
assert!(i.element_shape.is_empty());
assert_eq!(r.info("sensors/temp").unwrap().dtype, "f32");
assert_eq!(r.kind("/sensors").unwrap(), Kind::Group);
assert_eq!(r.kind("/sensors/temp").unwrap(), Kind::Dataset);
}
#[test]
fn reads_whole_and_hyperslab() {
let r = sample();
let all = r.read("grid", None).unwrap();
assert_eq!(all.shape, vec![3, 4]);
assert_eq!(all.data, Data::F64((0..12).map(f64::from).collect()));
let slab = Hyperslab {
start: vec![1, 0],
count: vec![2, 2],
stride: Some(vec![1, 2]),
block: None,
};
let part = r.read("grid", Some(&slab)).unwrap();
assert_eq!(part.shape, vec![2, 2]);
assert_eq!(part.data, Data::F64(vec![4.0, 6.0, 8.0, 10.0]));
assert_eq!(
r.read("bytes", None).unwrap().data,
Data::U8(vec![1, 2, 250])
);
assert_eq!(
r.read("sensors/temp", None).unwrap().data,
Data::F32(vec![1.5, -2.25])
);
}
#[test]
fn bad_hyperslabs_are_refused() {
let r = sample();
let mk = |start: Vec<u64>, count: Vec<u64>| Hyperslab {
start,
count,
stride: None,
block: None,
};
assert!(
r.read("grid", Some(&mk(vec![0], vec![1])))
.unwrap_err()
.contains("dimensions")
);
assert!(
r.read("grid", Some(&mk(vec![2, 0], vec![2, 1])))
.unwrap_err()
.contains("exceeds")
);
let overlap = Hyperslab {
start: vec![0, 0],
count: vec![2, 1],
stride: Some(vec![1, 1]),
block: Some(vec![2, 1]),
};
assert!(
r.read("grid", Some(&overlap))
.unwrap_err()
.contains("overlap")
);
}
#[test]
fn attrs_are_sorted() {
let r = sample();
let (attrs, errors) = r.attrs("/").unwrap();
assert!(errors.is_empty());
let names: Vec<&str> = attrs.iter().map(|a| a.name.as_str()).collect();
assert_eq!(names, ["scale", "version"]);
let (g, _) = r.attrs("sensors").unwrap();
assert!(matches!(&g[0].value, AttrValue::String(s) if s == "lab"));
}
#[test]
fn compound_is_refused_not_reinterpreted() {
use clawhdf5::CompoundTypeBuilder;
let ct = CompoundTypeBuilder::new()
.f64_field("x")
.i32_field("n")
.build();
let mut rec = Vec::new();
rec.extend_from_slice(&1.0f64.to_le_bytes());
rec.extend_from_slice(&7i32.to_le_bytes());
let mut b = FileBuilder::new();
b.create_dataset("table").with_compound_data(ct, rec, 1);
let r = Reader::open(b.finish().unwrap()).unwrap();
let e = r.read("table", None).unwrap_err();
assert!(e.contains("compound{x: f64, n: i32}"), "{e}");
assert!(e.contains("not supported"), "{e}");
}
#[test]
fn garbage_is_an_error() {
assert!(Reader::open(vec![0u8; 64]).is_err());
assert!(Reader::open(Vec::new()).is_err());
}
}
+244
View File
@@ -0,0 +1,244 @@
//! clawhdf5's HDF5 reader for JavaScript, via `wasm-bindgen`.
//!
//! ```js
//! import init, { open } from "./pkg/clawhdf5_wasm.js";
//! await init();
//! const file = open(new Uint8Array(await blob.arrayBuffer()));
//! file.list("/"); // [{ name, kind: "group" | "dataset" }]
//! file.info("/x"); // { shape, maxshape, dtype, elementShape }
//! file.attrs("/x"); // [{ name, value, dtype }]
//! file.read("/x"); // { shape, dtype, data: Float64Array | ... | string[] }
//! file.readHyperslab("/x", [0, 0], [10, 10]); // stride, block optional
//! file.free();
//! ```
//!
//! Numeric data comes back in the typed array of the stored width
//! (`Int16Array` for `i16`, `BigInt64Array` for `i64`, `Float32Array` for
//! `f32` and `f16`, ...); strings and enumeration names as arrays of strings.
//! Anything else is a thrown `Error` naming the datatype. Only the reader is
//! exposed: nothing here writes files.
//!
//! The logic lives in [`core`], which is plain Rust and tested natively.
pub mod core;
use clawhdf5::AttrValue;
use js_sys::{Array, Object, Reflect};
use wasm_bindgen::prelude::*;
use crate::core::{Data, Hyperslab, Reader};
/// JavaScript numbers are exact up to 2^53.
const MAX_SAFE_INTEGER: f64 = 9_007_199_254_740_991.0;
fn js_err(msg: String) -> JsError {
JsError::new(&msg)
}
fn set(obj: &Object, key: &str, value: impl Into<JsValue>) {
// Defining a property on a fresh plain object cannot fail.
Reflect::set(obj, &JsValue::from_str(key), &value.into()).unwrap_throw();
}
fn shape_to_js(shape: &[u64]) -> Array {
shape.iter().map(|&d| JsValue::from_f64(d as f64)).collect()
}
fn indices_from_js(what: &str, v: &[f64]) -> Result<Vec<u64>, JsError> {
v.iter()
.map(|&x| {
if x.is_finite() && x >= 0.0 && x.fract() == 0.0 && x <= MAX_SAFE_INTEGER {
Ok(x as u64)
} else {
Err(js_err(format!(
"{what} must hold non-negative integers, got {x}"
)))
}
})
.collect()
}
fn data_to_js(data: Data) -> JsValue {
match data {
Data::F32(v) => js_sys::Float32Array::from(&v[..]).into(),
Data::F64(v) => js_sys::Float64Array::from(&v[..]).into(),
Data::I8(v) => js_sys::Int8Array::from(&v[..]).into(),
Data::I16(v) => js_sys::Int16Array::from(&v[..]).into(),
Data::I32(v) => js_sys::Int32Array::from(&v[..]).into(),
Data::I64(v) => js_sys::BigInt64Array::from(&v[..]).into(),
Data::U8(v) => js_sys::Uint8Array::from(&v[..]).into(),
Data::U16(v) => js_sys::Uint16Array::from(&v[..]).into(),
Data::U32(v) => js_sys::Uint32Array::from(&v[..]).into(),
Data::U64(v) => js_sys::BigUint64Array::from(&v[..]).into(),
Data::Strings(v) => v
.into_iter()
.map(|s| JsValue::from_str(&s))
.collect::<Array>()
.into(),
}
}
/// A scalar integer as a `number` when exact, else a `bigint`.
fn int_to_js(x: i128) -> JsValue {
if (x as f64).abs() <= MAX_SAFE_INTEGER {
JsValue::from_f64(x as f64)
} else if let Ok(v) = i64::try_from(x) {
JsValue::from(v)
} else {
JsValue::from(x as u64)
}
}
/// An attribute value, and the datatype of one that has no JavaScript form.
fn attr_to_js(value: AttrValue) -> (JsValue, Option<String>) {
match value {
AttrValue::F64(x) => (JsValue::from_f64(x), None),
AttrValue::F64Array(v) => (js_sys::Float64Array::from(&v[..]).into(), None),
AttrValue::I64(x) => (int_to_js(i128::from(x)), None),
AttrValue::I64Array(v) => (js_sys::BigInt64Array::from(&v[..]).into(), None),
AttrValue::U64(x) => (int_to_js(i128::from(x)), None),
AttrValue::U64Array(v) => (js_sys::BigUint64Array::from(&v[..]).into(), None),
AttrValue::String(s) => (JsValue::from_str(&s), None),
AttrValue::StringArray(v) => (
v.iter()
.map(|s| JsValue::from_str(s))
.collect::<Array>()
.into(),
None,
),
AttrValue::Raw { datatype, .. } => (JsValue::NULL, Some(core::describe(&datatype))),
}
}
/// An open HDF5 (or NetCDF-4) file.
#[wasm_bindgen]
pub struct H5File {
inner: Reader,
}
/// Open a file from its bytes. Throws if they are not an HDF5 file.
#[wasm_bindgen]
pub fn open(bytes: Vec<u8>) -> Result<H5File, JsError> {
H5File::new(bytes)
}
/// The clawhdf5 version this module was built from.
#[wasm_bindgen]
pub fn version() -> String {
env!("CARGO_PKG_VERSION").to_string()
}
#[wasm_bindgen]
impl H5File {
/// Same as [`open`].
#[wasm_bindgen(constructor)]
pub fn new(bytes: Vec<u8>) -> Result<H5File, JsError> {
Ok(H5File {
inner: Reader::open(bytes).map_err(js_err)?,
})
}
/// `"group"` or `"dataset"`.
pub fn kind(&self, path: &str) -> Result<String, JsError> {
Ok(self.inner.kind(path).map_err(js_err)?.as_str().to_string())
}
/// The group's members: `[{ name, kind }]`, groups first.
pub fn list(&self, path: &str) -> Result<Array, JsError> {
Ok(self
.inner
.list(path)
.map_err(js_err)?
.into_iter()
.map(|c| {
let o = Object::new();
set(&o, "name", c.name);
set(&o, "kind", c.kind.as_str());
JsValue::from(o)
})
.collect())
}
/// `{ shape, maxshape, dtype, elementShape }`. `maxshape` is `null`
/// when not recorded, with `null` for each unlimited dimension.
pub fn info(&self, path: &str) -> Result<Object, JsError> {
let i = self.inner.info(path).map_err(js_err)?;
let o = Object::new();
set(&o, "shape", shape_to_js(&i.shape));
let max: JsValue = match i.maxshape {
None => JsValue::NULL,
Some(dims) => dims
.into_iter()
.map(|d| d.map_or(JsValue::NULL, |d| JsValue::from_f64(d as f64)))
.collect::<Array>()
.into(),
};
set(&o, "maxshape", max);
set(&o, "dtype", i.dtype);
set(&o, "elementShape", shape_to_js(&i.element_shape));
Ok(o)
}
/// `[{ name, value, dtype }]`, sorted by name. Scalars are `number`
/// (`bigint` beyond 2^53) or `string`; arrays are typed arrays or
/// `string[]`. An attribute with no JavaScript form has `value: null`
/// and its `dtype`; one that could not be read at all is reported by
/// [`attrErrors`](Self::attr_errors).
pub fn attrs(&self, path: &str) -> Result<Array, JsError> {
let (attrs, _) = self.inner.attrs(path).map_err(js_err)?;
Ok(attrs
.into_iter()
.map(|a| {
let o = Object::new();
set(&o, "name", a.name);
let (value, dtype) = attr_to_js(a.value);
set(&o, "value", value);
set(&o, "dtype", dtype.map_or(JsValue::NULL, JsValue::from));
JsValue::from(o)
})
.collect())
}
/// Messages for attributes that could not be read.
#[wasm_bindgen(js_name = attrErrors)]
pub fn attr_errors(&self, path: &str) -> Result<Array, JsError> {
let (_, errors) = self.inner.attrs(path).map_err(js_err)?;
Ok(errors.into_iter().map(JsValue::from).collect())
}
/// The whole dataset: `{ shape, dtype, data }`, `data` in row-major
/// order.
pub fn read(&self, path: &str) -> Result<Object, JsError> {
self.read_impl(path, None)
}
/// A regular hyperslab (`H5Sselect_hyperslab`): `stride` and `block`
/// default to 1. The result's shape is `count * block` per dimension.
#[wasm_bindgen(js_name = readHyperslab)]
pub fn read_hyperslab(
&self,
path: &str,
start: Vec<f64>,
count: Vec<f64>,
stride: Option<Vec<f64>>,
block: Option<Vec<f64>>,
) -> Result<Object, JsError> {
let slab = Hyperslab {
start: indices_from_js("start", &start)?,
count: indices_from_js("count", &count)?,
stride: stride.map(|s| indices_from_js("stride", &s)).transpose()?,
block: block.map(|b| indices_from_js("block", &b)).transpose()?,
};
self.read_impl(path, Some(&slab))
}
fn read_impl(&self, path: &str, slab: Option<&Hyperslab>) -> Result<Object, JsError> {
let dtype = self.inner.info(path).map_err(js_err)?.dtype;
let a = self.inner.read(path, slab).map_err(js_err)?;
let o = Object::new();
set(&o, "shape", shape_to_js(&a.shape));
set(&o, "dtype", dtype);
set(&o, "data", data_to_js(a.data));
Ok(o)
}
}
+265
View File
@@ -0,0 +1,265 @@
//! The wasm reader's core against files written by h5py and netCDF4, with
//! the values libhdf5 reads back as the reference. The generator,
//! `examples/wasm-viewer/test/make_fixture.py`, is shared with the Node test
//! of the built wasm package, so both compare against the same expectations.
//!
//! Skipped when python3 with h5py/netCDF4 is missing, unless
//! `CLAWHDF5_REQUIRE_INTEROP=1`. `CLAWHDF5_PYTHON` names the interpreter.
use std::path::{Path, PathBuf};
use std::process::Command;
use clawhdf5::AttrValue;
use clawhdf5_wasm::core::{Data, Hyperslab, Kind, Reader};
use serde_json::Value;
fn python() -> String {
std::env::var("CLAWHDF5_PYTHON").unwrap_or_else(|_| "python3".to_string())
}
fn interop_required() -> bool {
std::env::var("CLAWHDF5_REQUIRE_INTEROP").is_ok_and(|v| v == "1")
}
fn python_available() -> bool {
Command::new(python())
.args(["-c", "import h5py, netCDF4, numpy"])
.output()
.map(|o| o.status.success())
.unwrap_or(false)
}
fn generator() -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR")).join("../../examples/wasm-viewer/test/make_fixture.py")
}
/// Values as comparable strings: integers exactly, floats by their f64
/// value (an f32 widens exactly), strings as themselves.
fn data_strings(d: &Data) -> (&'static str, Vec<String>) {
fn s<T: ToString>(v: &[T]) -> Vec<String> {
v.iter().map(ToString::to_string).collect()
}
fn f<T: Copy + Into<f64>>(v: &[T]) -> Vec<String> {
v.iter().map(|&x| format!("{:?}", x.into())).collect()
}
match d {
Data::F32(v) => ("f32", f(v)),
Data::F64(v) => ("f64", f(v)),
Data::I8(v) => ("i8", s(v)),
Data::I16(v) => ("i16", s(v)),
Data::I32(v) => ("i32", s(v)),
Data::I64(v) => ("i64", s(v)),
Data::U8(v) => ("u8", s(v)),
Data::U16(v) => ("u16", s(v)),
Data::U32(v) => ("u32", s(v)),
Data::U64(v) => ("u64", s(v)),
Data::Strings(v) => ("strings", v.clone()),
}
}
fn expected_strings(kind: &str, values: &Value) -> Vec<String> {
values
.as_array()
.unwrap()
.iter()
.map(|v| match (kind, v) {
("f32" | "f64", Value::Number(n)) => format!("{:?}", n.as_f64().unwrap()),
(_, Value::String(s)) => s.clone(),
other => panic!("unexpected expected value {other:?}"),
})
.collect()
}
fn shape(v: &Value) -> Vec<u64> {
v.as_array()
.unwrap()
.iter()
.map(|x| x.as_u64().unwrap())
.collect()
}
fn check_attr(file: &str, path: &str, name: &str, got: &AttrValue, want: &Value) {
let ctx = format!("{file}:{path}@{name}");
let ints = |v: &Value| -> Vec<String> {
v.as_array()
.unwrap()
.iter()
.map(|x| x.as_str().unwrap().to_string())
.collect()
};
let scalar = want["scalar"].as_bool().unwrap_or(false);
match got {
AttrValue::String(s) => assert_eq!(want["string"].as_str(), Some(s.as_str()), "{ctx}"),
AttrValue::StringArray(v) => {
let w: Vec<&str> = want["strings"]
.as_array()
.unwrap_or_else(|| panic!("{ctx}: got {v:?}"))
.iter()
.map(|x| x.as_str().unwrap())
.collect();
assert_eq!(v, &w, "{ctx}");
}
AttrValue::I64(x) => {
assert!(scalar, "{ctx}");
assert_eq!(vec![x.to_string()], ints(&want["int"]), "{ctx}");
}
AttrValue::U64(x) => {
assert!(scalar, "{ctx}");
assert_eq!(vec![x.to_string()], ints(&want["int"]), "{ctx}");
}
AttrValue::I64Array(v) => assert_eq!(
v.iter().map(ToString::to_string).collect::<Vec<_>>(),
ints(&want["int"]),
"{ctx}"
),
AttrValue::U64Array(v) => assert_eq!(
v.iter().map(ToString::to_string).collect::<Vec<_>>(),
ints(&want["int"]),
"{ctx}"
),
AttrValue::F64(x) => {
assert!(scalar, "{ctx}");
assert_eq!(Some(*x), want["float"][0].as_f64(), "{ctx}");
}
AttrValue::F64Array(v) => {
let w: Vec<f64> = want["float"]
.as_array()
.unwrap()
.iter()
.map(|x| x.as_f64().unwrap())
.collect();
assert_eq!(v, &w, "{ctx}");
}
AttrValue::Raw { datatype, .. } => {
let want = want["raw"]
.as_str()
.unwrap_or_else(|| panic!("{ctx}: undecoded {datatype:?}"));
let d = clawhdf5_wasm::core::describe(datatype);
assert!(d.contains(want), "{ctx}: {d}");
}
}
}
fn check_file(dir: &Path, file: &str, exp: &Value) {
let r = Reader::open(std::fs::read(dir.join(file)).unwrap()).unwrap();
for (path, want) in exp["lists"].as_object().unwrap() {
let list = r
.list(path)
.unwrap_or_else(|e| panic!("{file}:{path}: {e}"));
for (kind, key) in [(Kind::Group, "groups"), (Kind::Dataset, "datasets")] {
let mut got: Vec<&str> = list
.iter()
.filter(|c| c.kind == kind)
.map(|c| c.name.as_str())
.collect();
got.sort();
let w: Vec<&str> = want[key]
.as_array()
.unwrap()
.iter()
.map(|x| x.as_str().unwrap())
.collect();
assert_eq!(got, w, "{file}:{path} {key}");
}
}
for (path, want) in exp["datasets"].as_object().unwrap() {
let kind = want["kind"].as_str().unwrap();
let a = match (r.read(path, None), want["unavailable"].as_str()) {
(Ok(a), _) => a,
// A filter this build may lack (zstd: the wasm build has it
// off, a workspace build may unify it on) must fail clearly.
(Err(e), Some(why)) => {
assert!(e.contains(why), "{file}:{path}: {e}");
continue;
}
(Err(e), None) => panic!("{file}:{path}: {e}"),
};
assert_eq!(a.shape, shape(&want["shape"]), "{file}:{path} shape");
let (got_kind, got) = data_strings(&a.data);
assert_eq!(got_kind, kind, "{file}:{path} kind");
assert_eq!(
got,
expected_strings(kind, &want["values"]),
"{file}:{path}"
);
if let Some(slab) = want.get("slab") {
let h = Hyperslab {
start: shape(&slab["start"]),
count: shape(&slab["count"]),
stride: Some(shape(&slab["stride"])),
block: None,
};
let a = r
.read(path, Some(&h))
.unwrap_or_else(|e| panic!("{file}:{path} {h:?}: {e}"));
assert_eq!(a.shape, shape(&slab["shape"]), "{file}:{path} slab shape");
assert_eq!(
data_strings(&a.data).1,
expected_strings(kind, &slab["values"]),
"{file}:{path} slab"
);
}
}
for (path, what) in exp["errors"].as_object().unwrap() {
let e = r.read(path, None).expect_err(path);
assert!(e.contains(what.as_str().unwrap()), "{file}:{path}: {e}");
}
let skip: Vec<&str> = exp["skip_attrs"]
.as_array()
.unwrap()
.iter()
.map(|x| x.as_str().unwrap())
.collect();
for (path, want) in exp["attrs"].as_object().unwrap() {
let (attrs, errors) = r
.attrs(path)
.unwrap_or_else(|e| panic!("{file}:{path}: {e}"));
assert!(errors.is_empty(), "{file}:{path}: {errors:?}");
let want = want.as_object().unwrap();
let mut compared = 0;
for a in &attrs {
if a.name.starts_with('_') || skip.contains(&a.name.as_str()) {
continue;
}
let w = want
.get(&a.name)
.unwrap_or_else(|| panic!("{file}:{path}: unexpected attribute {}", a.name));
check_attr(file, path, &a.name, &a.value, w);
compared += 1;
}
assert_eq!(compared, want.len(), "{file}:{path}: attributes missing");
}
}
#[test]
fn reads_what_h5py_and_netcdf4_wrote() {
if !python_available() {
assert!(
!interop_required(),
"CLAWHDF5_REQUIRE_INTEROP=1 but python with h5py, netCDF4 and numpy is not available"
);
eprintln!("SKIP: python with h5py/netCDF4 not available");
return;
}
let dir = tempfile::tempdir().unwrap();
let out = Command::new(python())
.arg(generator())
.arg(dir.path())
.output()
.expect("run python");
assert!(
out.status.success(),
"fixture generator failed:\n{}",
String::from_utf8_lossy(&out.stderr)
);
let exp: Value =
serde_json::from_slice(&std::fs::read(dir.path().join("expected.json")).unwrap()).unwrap();
for file in ["fixture.h5", "fixture.nc"] {
check_file(dir.path(), file, &exp[file]);
}
}
+29
View File
@@ -144,6 +144,35 @@ mod tests {
assert_eq!(ds.dtype().unwrap(), DType::I32);
}
#[test]
fn dataset_raw_datatype() {
use clawhdf5_format::datatype::Datatype;
let bytes = make_simple_file();
let file = File::from_bytes(bytes).unwrap();
let dt = file.dataset("counts").unwrap().raw_datatype().unwrap();
assert!(
matches!(
dt,
Datatype::FixedPoint {
size: 4,
signed: true,
..
}
),
"{dt:?}"
);
// The full type pairs with read_selection's bytes.
let raw = file
.dataset("counts")
.unwrap()
.read_selection(&Selection::All)
.unwrap();
assert_eq!(
clawhdf5_format::data_read::read_as_i64(&raw, &dt).unwrap(),
vec![10, 20, 30]
);
}
#[test]
fn root_group_datasets() {
let bytes = make_simple_file();
+9
View File
@@ -424,6 +424,15 @@ impl<'f> Dataset<'f> {
Ok(classify_datatype(&dt))
}
/// Returns the dataset's full datatype as stored in the file (byte order,
/// string padding, compound layout, ...), with a committed datatype
/// resolved. Use it with the `clawhdf5_format::data_read` converters on
/// the bytes [`read_selection`](Self::read_selection) returns, for types
/// the typed `read_*` methods do not cover.
pub fn raw_datatype(&self) -> Result<Datatype, Error> {
self.datatype()
}
/// Read all data as `f64` values.
pub fn read_f64(&self) -> Result<Vec<f64>, Error> {
let dt = self.datatype()?;
+17
View File
@@ -473,6 +473,23 @@ the same agent-store interop test.
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
which is what libhdf5 itself writes.
## `clawhdf5-wasm` (browser) limits
**Status:** open (by design for now; added 2026-09-26).
- The whole file is held in memory: `open()` takes its bytes. There are no
HTTP range reads, so a multi-GB file does not fit a browser tab.
- Compound, reference, opaque, bitfield, time and VL-sequence datasets are
refused with an error naming the type; attributes of those types come back
as `value: null` with their `dtype`.
- No Zstd or SZIP (both link C): such datasets fail with
`unsupported filter: 32015` / `: 4`. pcodec is not enabled either.
- External links and virtual-dataset sources in other files cannot be
followed (no file system).
- Variable-length string datasets are read by decoding `read_selection`'s
bytes with `clawhdf5_format::vl_data` in the wasm crate; `File` itself still
cannot (see the audit gaps above).
## The Node.js package (`packages/clawhdf5-node`) does not work
**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
+2
View File
@@ -0,0 +1,2 @@
# Generated by build.sh.
/pkg/
+97
View File
@@ -0,0 +1,97 @@
# HDF5 viewer in the browser
A single page that opens an HDF5 or NetCDF-4 file entirely in the browser
with `clawhdf5-wasm` (clawhdf5's reader compiled to WebAssembly): drop a
file, browse its groups, and look at a dataset's type, shape, attributes
and values (a 50 x 12 window at a time, read as a hyperslab, with the
leading dimensions of a 3-D+ dataset held at chosen indices). The file never
leaves the page.
## Build and open
```bash
rustup target add wasm32-unknown-unknown
cargo install wasm-bindgen-cli --version 0.2.129 # must equal the crate version; build.sh checks
bash examples/wasm-viewer/build.sh # writes examples/wasm-viewer/pkg/ (not committed)
python3 -m http.server -d examples/wasm-viewer 8000 # wasm cannot load from file://
```
Then open <http://localhost:8000/>. `?file=<url>&path=<object>` opens a
file from a URL (same origin, or one serving CORS headers) and selects an
object in it, e.g. `?file=data/run1.h5&path=/results/energy`.
## JavaScript API
```js
import init, { open } from "./pkg/clawhdf5_wasm.js";
await init();
const f = open(new Uint8Array(await blob.arrayBuffer()));
f.list("/"); // [{ name, kind: "group" | "dataset" }], groups first
f.info("/grid"); // { shape, maxshape, dtype, elementShape }
f.attrs("/grid"); // [{ name, value, dtype }]
f.read("/grid"); // { shape, dtype, data }
f.readHyperslab("/grid", [0, 0], [10, 5], [2, 1]); // start, count, stride?, block?
f.free();
```
`data` is the typed array of the stored width (`Float64Array`,
`Float32Array` also for `f16`, `Int8Array` ... `BigInt64Array`,
`BigUint64Array`), or an array of strings for fixed- and variable-length
strings and enumerations (h5py booleans read as `"TRUE"`/`"FALSE"`). Array
datatypes are flattened, their dimensions appended to `shape`. Anything
else throws an `Error` naming the type.
## Limits
- Read-only, and the whole file is held in memory (no range requests).
- Compound, reference, opaque and variable-length-sequence datasets are
refused with an error. Attributes of those types are listed with
`value: null` and their `dtype`.
- No Zstd or SZIP filters (they link C): such a dataset fails with
`unsupported filter`. Deflate, shuffle, Fletcher-32, LZ4, N-Bit and
scale-offset are read (within the limits in `docs/known-issues.md`).
- Virtual datasets whose sources are in other files, and external links,
cannot be followed: there is no file system.
## Tests
`test/run.sh` builds the package, writes `fixture.h5` (h5py) and
`fixture.nc` (netCDF4) with `test/make_fixture.py`, then:
- runs `test/test.mjs` under Node: every dataset (whole and a strided
hyperslab), listing and attribute is compared with what libhdf5 reads
back, error paths are checked, and so are the page's DOM-free helpers
(`viewer-lib.js`);
- runs `test/browser.sh`: loads the page in headless Chromium with
`?file=fixture.h5&path=...` for eight objects and checks the rendered tree,
types, shapes, attribute and value cells, and the error shown for an
unsupported type. Skipped when no Chromium is found (`CHROME` names one;
a Playwright download under `~/.cache/ms-playwright` is picked up).
Drag-and-drop and the file picker are not driven by it; they share
`load()` with the `?file=` path.
The same expectations are checked natively, without Node, by
`crates/clawhdf5-wasm/tests/h5py_interop.rs`, which is what CI runs (the CI
container has no Node or browser).
## Size
Measured 2026-09-26 on tank (rustc 1.98.1, wasm-bindgen 0.2.129, gzip 1.14,
`gzip -9 -n`), after `bash examples/wasm-viewer/build.sh`:
| | raw | gzip -9 |
|---|---:|---:|
| `pkg/clawhdf5_wasm_bg.wasm` (profile `wasm-release`, opt-level `s`) | 627,501 B | 191,639 B |
| `pkg/clawhdf5_wasm.js` (wasm-bindgen glue) | 21,826 B | 4,487 B |
| same wasm at opt-level `z` | 693,068 B | 192,550 B |
| same wasm at opt-level `3` | 544,035 B | 198,803 B |
| h5wasm 0.10.3: wasm embedded in `dist/esm/hdf5_util.js` | 3,544,184 B | 907,096 B |
| h5wasm 0.10.3: `dist/esm/hdf5_util.js` as shipped | 4,150,134 B | 986,699 B |
h5wasm figures: `npm pack [email protected]` (npm reports
`dist.unpackedSize` 14,731,385 B for the whole package), wasm extracted from
the `binaryDecode` literal in `hdf5_util.js`. h5wasm is the whole of libhdf5
(writing, every datatype, plugins), so this compares download size, not
equal functionality. No `wasm-opt` pass was applied (binaryen is not
installed on tank). opt-level `s` is used because it is the smallest
compressed.
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env bash
# Build the clawhdf5-wasm package the viewer loads, into examples/wasm-viewer/pkg/.
#
# Needs the wasm32-unknown-unknown target and the wasm-bindgen CLI at the
# exact version cargo resolves for the wasm-bindgen crate:
# rustup target add wasm32-unknown-unknown
# cargo install wasm-bindgen-cli --version <that version>
#
# Then serve this directory over HTTP (browsers do not load wasm modules from
# file://) and open it:
# python3 -m http.server -d examples/wasm-viewer 8000
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
ROOT="$(cd "$HERE/../.." && pwd)"
cd "$ROOT"
# The resolved crate version (Cargo.lock is not committed, so ask cargo).
want=$(cargo pkgid wasm-bindgen | sed 's/.*[@#]//')
if ! command -v wasm-bindgen >/dev/null; then
echo "wasm-bindgen CLI not found: cargo install wasm-bindgen-cli --version $want" >&2
exit 1
fi
have=$(wasm-bindgen --version | awk '{print $2}')
if [ "$want" != "$have" ]; then
echo "wasm-bindgen CLI is $have but the crate is $want:" >&2
echo " cargo install wasm-bindgen-cli --version $want" >&2
exit 1
fi
cargo build -p clawhdf5-wasm --target wasm32-unknown-unknown --profile wasm-release
target_dir=$(cargo metadata --format-version 1 --no-deps \
| sed -n 's/.*"target_directory":"\([^"]*\)".*/\1/p')
wasm="$target_dir/wasm32-unknown-unknown/wasm-release/clawhdf5_wasm.wasm"
rm -rf "$HERE/pkg"
wasm-bindgen --target web --out-dir "$HERE/pkg" "$wasm"
echo "built $HERE/pkg ($(wc -c < "$HERE/pkg/clawhdf5_wasm_bg.wasm") bytes of wasm)"
+278
View File
@@ -0,0 +1,278 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>HDF5 Viewer</title>
<style>
:root {
--bg: #fbfbfa; --panel: #ffffff; --ink: #1d1d1b; --muted: #6b6b66;
--line: #e2e1dc; --accent: #2f5d8a; --accent-soft: #e8eff6; --bad: #a3312a;
--mono: ui-monospace, "SF Mono", Menlo, Consolas, monospace;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #161615; --panel: #1f1f1d; --ink: #ecebe6; --muted: #9a9993;
--line: #34332f; --accent: #8db8e0; --accent-soft: #23303c; --bad: #e0857c;
}
}
* { box-sizing: border-box; }
body { margin: 0; background: var(--bg); color: var(--ink);
font: 14px/1.45 system-ui, -apple-system, "Segoe UI", sans-serif; }
header { display: flex; align-items: center; gap: 12px; padding: 12px 16px;
border-bottom: 1px solid var(--line); background: var(--panel); flex-wrap: wrap; }
header h1 { font-size: 15px; margin: 0; font-weight: 600; }
header .file { color: var(--muted); font-family: var(--mono); font-size: 13px; }
header label.button { margin-left: auto; }
.button { border: 1px solid var(--line); background: var(--panel); color: var(--ink);
padding: 5px 10px; border-radius: 6px; cursor: pointer; font: inherit; }
.button:hover { border-color: var(--accent); }
main { display: grid; grid-template-columns: minmax(200px, 300px) 1fr; min-height: calc(100vh - 50px); }
@media (max-width: 700px) { main { grid-template-columns: 1fr; } nav { border-right: 0; border-bottom: 1px solid var(--line); } }
nav { border-right: 1px solid var(--line); padding: 8px; overflow: auto; background: var(--panel); }
section { padding: 16px; overflow: auto; min-width: 0; }
ul.tree { list-style: none; margin: 0; padding-left: 14px; }
nav > ul.tree { padding-left: 0; }
.node { display: flex; gap: 6px; align-items: baseline; padding: 2px 6px; border-radius: 4px;
cursor: pointer; white-space: nowrap; font-family: var(--mono); font-size: 13px; }
.node:hover { background: var(--accent-soft); }
.node.selected { background: var(--accent-soft); color: var(--accent); }
.node .icon { width: 1em; color: var(--muted); flex: none; text-align: center; }
.drop { border: 2px dashed var(--line); border-radius: 10px; padding: 48px 16px; text-align: center;
color: var(--muted); max-width: 560px; margin: 48px auto; }
body.dragging .drop, body.dragging nav { border-color: var(--accent); }
h2 { font-size: 16px; margin: 0 0 8px; font-family: var(--mono); word-break: break-all; }
h3 { font-size: 12px; text-transform: uppercase; letter-spacing: .06em; color: var(--muted); margin: 20px 0 6px; }
dl.meta { display: grid; grid-template-columns: max-content 1fr; gap: 2px 16px; margin: 0; }
dl.meta dt { color: var(--muted); }
dl.meta dd { margin: 0; font-family: var(--mono); word-break: break-word; }
table { border-collapse: collapse; font-family: var(--mono); font-size: 12.5px; }
th, td { border: 1px solid var(--line); padding: 3px 8px; text-align: right; white-space: nowrap; }
th { background: var(--bg); color: var(--muted); font-weight: 500; }
table.attrs td { text-align: left; white-space: normal; word-break: break-word; }
.scroll { overflow: auto; max-width: 100%; }
.controls { display: flex; flex-wrap: wrap; gap: 8px 14px; align-items: center; margin: 6px 0 10px; }
.controls input { width: 6em; font: inherit; padding: 2px 4px; background: var(--panel); color: var(--ink);
border: 1px solid var(--line); border-radius: 4px; }
.error { color: var(--bad); font-family: var(--mono); white-space: pre-wrap; }
.muted { color: var(--muted); }
</style>
</head>
<body>
<header>
<h1>HDF5 Viewer</h1>
<span class="file" id="filename">no file</span>
<label class="button">Open file…<input type="file" id="picker" accept=".h5,.hdf5,.he5,.nc,.nc4,.cdf" hidden></label>
</header>
<main>
<nav id="tree"></nav>
<section id="detail">
<div class="drop">
<p><strong>Drop an HDF5 or NetCDF-4 file here</strong>, or use “Open file…”.</p>
<p class="muted">The file is read in this page by clawhdf5 compiled to WebAssembly; it is not uploaded anywhere.</p>
<p class="muted" id="version"></p>
</div>
</section>
</main>
<script type="module">
import init, { open, version } from "./pkg/clawhdf5_wasm.js";
import { joinPath, formatValue, viewWindow, toRows, perElement } from "./viewer-lib.js";
const $ = (id) => document.getElementById(id);
const el = (tag, props = {}, ...kids) => {
const e = Object.assign(document.createElement(tag), props);
e.append(...kids);
return e;
};
let file = null;
let selectedNode = null;
await init();
$("version").textContent = `clawhdf5 ${version()}`;
async function load(blob) {
const bytes = new Uint8Array(await blob.arrayBuffer());
if (file) file.free();
file = null;
$("filename").textContent = blob.name;
$("tree").replaceChildren();
try {
file = open(bytes);
} catch (e) {
$("detail").replaceChildren(el("p", { className: "error", textContent: `Cannot open ${blob.name}: ${e.message}` }));
return;
}
const root = el("ul", { className: "tree" });
root.append(treeNode("/", "/", "group"));
$("tree").append(root);
root.querySelector(".node").click();
}
function treeNode(path, name, kind) {
const li = el("li");
const icon = el("span", { className: "icon", textContent: kind === "group" ? "▸" : "·" });
const row = el("div", { className: "node", title: path }, icon, el("span", { textContent: name }));
row.dataset.path = path;
li.append(row);
let children = null;
row.addEventListener("click", () => {
if (selectedNode) selectedNode.classList.remove("selected");
row.classList.add("selected");
selectedNode = row;
if (kind === "group") {
if (children) {
children.hidden = !children.hidden;
} else {
children = el("ul", { className: "tree" });
try {
for (const c of file.list(path)) children.append(treeNode(joinPath(path, c.name), c.name, c.kind));
} catch (e) {
children.append(el("li", { className: "error", textContent: e.message }));
}
li.append(children);
}
icon.textContent = children.hidden ? "▸" : "▾";
}
show(path, kind);
});
return li;
}
function attrsTable(path) {
let attrs, errors;
try {
attrs = file.attrs(path);
errors = file.attrErrors(path);
} catch (e) {
return el("p", { className: "error", textContent: e.message });
}
if (!attrs.length && !errors.length) return el("p", { className: "muted", textContent: "none" });
const t = el("table", { className: "attrs" }, el("tr", {}, el("th", { textContent: "name" }), el("th", { textContent: "value" })));
for (const a of attrs) {
const v = a.value === null ? el("span", { className: "muted", textContent: `(${a.dtype})` }) : formatValue(a.value);
t.append(el("tr", {}, el("td", { textContent: a.name }), el("td", {}, v)));
}
for (const msg of errors) t.append(el("tr", {}, el("td", { className: "error", colSpan: 2, textContent: msg })));
return el("div", { className: "scroll" }, t);
}
function show(path, kind) {
const out = [el("h2", { textContent: path })];
if (kind === "dataset") {
let info;
try {
info = file.info(path);
} catch (e) {
$("detail").replaceChildren(...out, el("p", { className: "error", textContent: e.message }));
return;
}
const max = info.maxshape === null ? "—" : `(${info.maxshape.map((d) => d ?? "∞").join(", ")})`;
out.push(el("dl", { className: "meta" },
el("dt", { textContent: "type" }), el("dd", { textContent: info.dtype }),
el("dt", { textContent: "shape" }), el("dd", { textContent: `(${info.shape.join(", ")})` }),
el("dt", { textContent: "max shape" }), el("dd", { textContent: max })));
out.push(el("h3", { textContent: "Attributes" }), attrsTable(path));
out.push(el("h3", { textContent: "Values" }), valuesView(path, info));
} else {
out.push(el("h3", { textContent: "Attributes" }), attrsTable(path));
}
$("detail").replaceChildren(...out);
}
function valuesView(path, info) {
const shape = info.shape;
const per = perElement(info.elementShape);
const state = { row: 0, col: 0, rows: 50, cols: 12, fixed: shape.slice(0, Math.max(0, shape.length - 2)).map(() => 0) };
const box = el("div");
const controls = el("div", { className: "controls" });
const body = el("div", { className: "scroll" });
box.append(controls, body);
const num = (label, key, idx) => {
const input = el("input", { type: "number", min: 0, value: idx === undefined ? state[key] : state.fixed[idx] });
input.addEventListener("change", () => {
const v = Math.max(0, Math.floor(Number(input.value) || 0));
if (idx === undefined) state[key] = v; else state.fixed[idx] = v;
render();
});
return el("label", {}, `${label} `, input);
};
if (shape.length >= 1) controls.append(num("row", "row"));
if (shape.length >= 2) controls.append(num("column", "col"));
state.fixed.forEach((_, i) => controls.append(num(`dim ${i}`, "fixed", i)));
function render() {
const w = viewWindow(shape, state);
let res;
try {
res = w === null ? file.read(path) : file.readHyperslab(path, w.start, w.count);
} catch (e) {
body.replaceChildren(el("p", { className: "error", textContent: e.message }));
return;
}
if (w === null) {
body.replaceChildren(el("pre", { textContent: toRows(res.data, 1, 1, per)[0][0] }));
return;
}
const t = el("table");
const head = el("tr", {}, el("th"));
for (let c = 0; c < w.cols; c++) head.append(el("th", { textContent: String(w.col + c) }));
t.append(head);
toRows(res.data, w.rows, w.cols, per).forEach((cells, r) => {
const tr = el("tr", {}, el("th", { textContent: String(w.row + r) }));
for (const c of cells) tr.append(el("td", { textContent: c }));
t.append(tr);
});
const total = shape.reduce((a, b) => a * b, 1);
const note = el("p", { className: "muted", textContent: `showing rows ${w.row}–${w.row + w.rows - 1}` +
(shape.length >= 2 ? `, columns ${w.col}–${w.col + w.cols - 1}` : "") + ` of ${total} values` });
body.replaceChildren(t, note);
}
render();
return box;
}
// Expand the tree down to `path` and select it.
function reveal(path) {
const rowFor = (p) => [...document.querySelectorAll(".node")].find((n) => n.dataset.path === p);
let cur = "/";
for (const part of path.split("/").filter(Boolean)) {
const row = rowFor(cur);
if (!row) return;
const kids = row.parentElement.querySelector(":scope > ul");
if (!kids || kids.hidden) row.click();
cur = joinPath(cur, part);
}
const target = rowFor(cur);
if (target && target !== selectedNode) target.click();
}
// ?file=<url>&path=<object> opens a file from a URL (same origin, or one
// that allows CORS) and selects an object in it.
const params = new URLSearchParams(location.search);
if (params.get("file")) {
const url = params.get("file");
try {
const resp = await fetch(url);
if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
const blob = await resp.blob();
await load(new File([blob], url.split("/").pop()));
if (file && params.get("path")) reveal(params.get("path"));
} catch (e) {
$("detail").replaceChildren(el("p", { className: "error", textContent: `Cannot fetch ${url}: ${e.message}` }));
}
}
$("picker").addEventListener("change", (e) => e.target.files[0] && load(e.target.files[0]));
document.addEventListener("dragover", (e) => { e.preventDefault(); document.body.classList.add("dragging"); });
document.addEventListener("dragleave", () => document.body.classList.remove("dragging"));
document.addEventListener("drop", (e) => {
e.preventDefault();
document.body.classList.remove("dragging");
const f = e.dataTransfer.files[0];
if (f) load(f);
});
</script>
</body>
</html>
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env bash
# Load the viewer page in headless Chromium and check what it renders.
#
# browser.sh FIXTURE_DIR
#
# FIXTURE_DIR holds fixture.h5 from make_fixture.py; ../pkg must be built.
# The page is opened with ?file=fixture.h5&path=<object>, which fetches the
# file, builds the tree down to <object> and shows it; the rendered DOM is
# dumped and checked for the values libhdf5 reads.
#
# Browser: $CHROME, else chromium/google-chrome on PATH, else a Playwright
# download under ~/.cache/ms-playwright. Exit 3 when none is found.
# BROWSER_DEBUG=/some/prefix saves each rendered page as prefix.<path>.html.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
FIX="$(cd "$1" && pwd)"
PY="${CLAWHDF5_PYTHON:-python3}"
chrome="${CHROME:-}"
if [ -z "$chrome" ]; then
for c in chromium chromium-browser google-chrome chrome-headless-shell; do
if command -v "$c" >/dev/null; then chrome="$(command -v "$c")"; break; fi
done
fi
if [ -z "$chrome" ]; then
chrome="$(ls -d "$HOME"/.cache/ms-playwright/chromium_headless_shell-*/chrome-headless-shell-linux64/chrome-headless-shell 2>/dev/null | tail -1 || true)"
fi
if [ -z "$chrome" ] || [ ! -x "$chrome" ]; then
echo "no Chromium found (set CHROME)" >&2
exit 3
fi
root="$(mktemp -d)"
server=""
cleanup() {
[ -n "$server" ] && kill "$server" 2>/dev/null || true
rm -rf "$root"
}
trap cleanup EXIT
ln -s "$HERE/../index.html" "$HERE/../viewer-lib.js" "$HERE/../pkg" "$FIX/fixture.h5" "$root/"
port=$("$PY" -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])')
"$PY" -m http.server --bind 127.0.0.1 --directory "$root" "$port" >/dev/null 2>&1 &
server=$!
for _ in $(seq 50); do
"$PY" -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:$port/index.html')" 2>/dev/null && break
sleep 0.1
done
fails=0
# A fresh profile per page: a second instance on the same profile fails.
render() {
local profile
profile="$(mktemp -d "$root/profile.XXXXXX")"
"$chrome" --headless --no-sandbox --disable-gpu --user-data-dir="$profile" \
--virtual-time-budget=20000 \
--dump-dom "http://127.0.0.1:$port/index.html?file=fixture.h5&path=$1" 2>/dev/null
}
# expect PATH TEXT...: every TEXT appears in the page rendered for PATH.
expect() {
local path="$1" dom
shift
dom="$(render "$path")"
[ -n "${BROWSER_DEBUG:-}" ] && printf "%s\n" "$dom" > "$BROWSER_DEBUG.$(echo "$path" | tr / _).html"
for text in "$@"; do
if ! grep -qF -- "$text" <<<"$dom"; then
echo "FAIL: page for $path lacks: $text" >&2
fails=$((fails + 1))
fi
done
echo "rendered $path"
}
# Tree (root expanded; the group row carries its path) and root attributes.
expect "/" 'data-path="/sensors"' 'data-path="/grid"' '<td>title</td><td>"wasm fixture"</td>' \
'<td>big</td><td>9223372036854775813</td>' '(compound{x: f64, n: i32})'
# A chunked, deflated 2-D dataset: type, shape, the first window of values.
expect "/grid" '<dd>f64</dd>' '<dd>(6, 10)</dd>' '<th>9</th>' '<td>0.25</td>' '<td>14.75</td>' \
'showing rows 0–5, columns 0–9 of 60 values'
# Nested path revealed through the tree; big-endian float32.
expect "/sensors/temp" 'data-path="/sensors/temp"' '<dd>f32</dd>' '<td>21.5</td>' '<td>22.25</td>'
# 64-bit integers stay exact; strings; array datatype cells.
expect "/u64" '<td>18446744073709551615</td>'
expect "/vlen_str" '<td>"двa"</td>' '<dd>vlen string</dd>'
expect "/pairs" '<td>[2, 3]</td>' '<dd>array[2]&lt;i32&gt;</dd>'
# 3-D: leading dimension held at 0, window over the last two.
expect "/cube" '<dd>(2, 5, 6)</dd>' '<td>29</td>' 'dim 0'
# Unsupported type: an error, not values.
expect "/table" 'class="error"' 'reading compound{x: f64, n: i32} datasets is not supported'
if [ "$fails" -gt 0 ]; then
echo "browser: $fails checks failed" >&2
exit 1
fi
echo "browser: all checks passed ($chrome)"
+203
View File
@@ -0,0 +1,203 @@
"""Write HDF5 and NetCDF-4 test files with h5py/netCDF4, and what libhdf5
reads back from them, for the clawhdf5-wasm tests.
python make_fixture.py OUT_DIR
writes OUT_DIR/fixture.h5, OUT_DIR/fixture.nc and OUT_DIR/expected.json.
Both the Rust test (crates/clawhdf5-wasm/tests/h5py_interop.rs, native) and
the Node test (test.mjs, the built wasm package) compare against the same
expected.json, so the two check the same values.
Every expected value comes from h5py reading the file back (numpy slicing
for hyperslabs), never from the arrays that were written. Integers are
encoded as strings so JSON.parse keeps 64-bit values exact.
"""
import json
import sys
import warnings
from pathlib import Path
import h5py
import netCDF4
import numpy as np
try: # registers the LZ4/Zstd filters with libhdf5; optional
import hdf5plugin
except ImportError:
hdf5plugin = None
# netCDF4 1.7 trips numpy 2.5's shape-setting deprecation on assignment.
warnings.filterwarnings("ignore", category=DeprecationWarning)
out = Path(sys.argv[1])
out.mkdir(parents=True, exist_ok=True)
h5 = out / "fixture.h5"
nc = out / "fixture.nc"
rng = np.random.default_rng(7)
with h5py.File(h5, "w") as f:
f.attrs["title"] = "wasm fixture"
f.attrs["version"] = np.int64(3)
f.attrs["scale"] = np.array([0.5, 2.0])
f.attrs["big"] = np.uint64(2**63 + 5)
f.attrs.create("vlen_note", "héllo", dtype=h5py.string_dtype())
# No plain JavaScript form: listed with value null and its type.
f.attrs["origin"] = np.array((1.5, 2), dtype=[("x", "<f8"), ("n", "<i4")])
f.create_dataset(
"grid", data=np.arange(60, dtype="<f8").reshape(6, 10) / 4,
chunks=(4, 3), compression="gzip", shuffle=True,
)
f.create_dataset("f32_be", data=rng.standard_normal(7).astype(">f4"))
f.create_dataset("f16", data=np.array([0.5, -1.25, 65504], dtype="<f2"))
f.create_dataset("i8", data=np.array([-128, -1, 0, 127], dtype="i1"))
f.create_dataset("i16_be", data=np.array([-32768, 5, 32767], dtype=">i2"))
f.create_dataset("u16", data=np.array([0, 40000, 65535], dtype="<u2"))
f.create_dataset("u32", data=np.array([0, 4_000_000_000], dtype="<u4"))
f.create_dataset("i64", data=np.array([-(2**63), 2**53 + 1, 7], dtype="<i8"))
f.create_dataset("u64", data=np.array([2**64 - 1, 1], dtype="<u8"))
f.create_dataset("scalar", data=np.float64(2.5))
f.create_dataset("fixed_str", data=np.array([b"alpha", b"be", b""], dtype="S5"))
f.create_dataset(
"vlen_str", data=["one", "двa", ""], dtype=h5py.string_dtype()
)
f.create_dataset("flags", data=np.array([True, False, True]))
# An array datatype: four elements, each an i4[2].
pairs = f.create_dataset("pairs", shape=(4,), dtype=np.dtype(("<i4", (2,))))
pairs[...] = np.arange(8, dtype="<i4").reshape(4, 2)
f.create_dataset(
"cube", data=np.arange(2 * 5 * 6, dtype="<i4").reshape(2, 5, 6),
chunks=(1, 2, 3), compression="gzip",
)
if hdf5plugin is not None:
# LZ4 is built into clawhdf5-wasm; Zstd links C and is not.
f.create_dataset("lz4", data=np.arange(40, dtype="<i4"), chunks=(10,),
**hdf5plugin.LZ4())
f.create_dataset("zstd", data=np.arange(40, dtype="<i4"), chunks=(10,),
**hdf5plugin.Zstd())
comp = np.zeros(2, dtype=[("x", "<f8"), ("n", "<i4")])
f.create_dataset("table", data=comp)
g = f.create_group("sensors")
g.attrs["location"] = "lab"
g.create_dataset("temp", data=np.array([21.5, 22.0, 22.25], dtype="<f4"))
g.create_group("empty")
f["alias"] = h5py.SoftLink("/sensors/temp")
with netCDF4.Dataset(nc, "w") as d:
d.title = "nc fixture"
d.createDimension("time", None)
d.createDimension("x", 4)
t = d.createVariable("time", "f8", ("time",))
t.units = "days since 2000-01-01"
v = d.createVariable("temp", "f4", ("time", "x"), zlib=True)
t[:] = np.arange(3)
v[:] = np.arange(12, dtype="f4").reshape(3, 4) + 0.5
def kind(dt):
"""The typed-array kind clawhdf5-wasm returns for a numpy dtype."""
if dt.kind == "b" or h5py.check_enum_dtype(dt) is not None:
return "strings"
if h5py.check_string_dtype(dt) is not None or dt.kind == "S":
return "strings"
if dt.subdtype is not None:
return kind(dt.subdtype[0])
if dt.kind == "f":
return "f64" if dt.itemsize == 8 else "f32"
if dt.kind in "iu":
return f"{dt.kind}{dt.itemsize * 8}"
raise ValueError(dt)
def flat(a, k):
a = np.asarray(a)
if k == "strings":
if a.dtype.kind == "b":
return ["TRUE" if x else "FALSE" for x in a.ravel()]
return [x.decode() if isinstance(x, bytes) else str(x) for x in a.ravel()]
if k.startswith(("i", "u")):
return [str(int(x)) for x in a.ravel()]
return [float(x) for x in a.ravel()]
def entry(ds, slab=None):
k = kind(ds.dtype)
data = ds[()]
e = {"kind": k, "shape": list(np.shape(data)), "values": flat(data, k)}
if slab:
start, count, stride = slab
idx = tuple(slice(s, s + (c - 1) * st + 1, st)
for s, c, st in zip(start, count, stride))
part = ds[idx]
e["slab"] = {"start": start, "count": count, "stride": stride,
"shape": list(part.shape), "values": flat(part, k)}
return e
def attr(v):
v = np.asarray(v) if not isinstance(v, (str, bytes)) else v
if isinstance(v, np.ndarray) and v.dtype.names:
return {"raw": "compound"}
if isinstance(v, bytes):
return {"string": v.decode()}
if isinstance(v, str):
return {"string": v}
if v.dtype.kind in "iu":
return {"int": [str(int(x)) for x in v.ravel()], "scalar": v.ndim == 0}
if v.dtype.kind == "f":
return {"float": [float(x) for x in v.ravel()], "scalar": v.ndim == 0}
if v.dtype.kind in "OSU":
items = [x.decode() if isinstance(x, bytes) else str(x) for x in v.ravel()]
return {"string": items[0]} if v.ndim == 0 else {"strings": items}
raise ValueError(v.dtype)
# Attributes the reader returns but the comparison leaves out: netCDF-4's
# internal ones (a leading underscore), and dimension-scale bookkeeping.
SKIP_ATTRS = ["DIMENSION_LIST", "REFERENCE_LIST", "CLASS", "NAME"]
def describe(path):
expected = {"datasets": {}, "errors": {}, "attrs": {}, "lists": {},
"skip_attrs": SKIP_ATTRS}
with h5py.File(path, "r") as f:
def walk(key, obj):
expected["attrs"][key] = {
k: attr(obj.attrs[k]) for k in obj.attrs
if not k.startswith("_") and k not in SKIP_ATTRS}
if isinstance(obj, h5py.Group):
members = {n: obj.get(n) for n in obj}
groups = [n for n, o in members.items() if isinstance(o, h5py.Group)]
sets = [n for n, o in members.items() if isinstance(o, h5py.Dataset)]
expected["lists"][key] = {"groups": sorted(groups),
"datasets": sorted(sets)}
for n, o in members.items():
walk(key.rstrip("/") + "/" + n, o)
elif obj.dtype.names:
expected["errors"][key] = "compound"
else:
expected["datasets"][key] = entry(obj, slab_for(obj))
if key == "/zstd":
# Refused by the wasm build (no zstd); a native build
# that unifies in clawhdf5-format/zstd reads it.
expected["datasets"][key]["unavailable"] = "unsupported filter: 32015"
walk("/", f)
return expected
def slab_for(obj):
"""A strided hyperslab inside the dataset's extent, or None."""
if obj.ndim == 0 or obj.shape[0] < 2 or 0 in obj.shape:
return None
start = [1] + [0] * (obj.ndim - 1)
count = [max(1, (obj.shape[0] - 1) // 2)] + [
max(1, (n + 1) // 2) for n in obj.shape[1:]]
stride = [2 if obj.shape[0] > 2 else 1] + [2] * (obj.ndim - 1)
count = [min(c, (n - s - 1) // st + 1)
for c, s, st, n in zip(count, start, stride, obj.shape)]
return (start, count, stride)
json.dump({"fixture.h5": describe(h5), "fixture.nc": describe(nc)},
open(out / "expected.json", "w"), indent=1, ensure_ascii=False)
+37
View File
@@ -0,0 +1,37 @@
#!/usr/bin/env bash
# Build the wasm package (../build.sh), test it under Node against files
# written by h5py and netCDF4 (make_fixture.py), then load the viewer page
# in headless Chromium if one is found (browser.sh).
#
# Needs node, the wasm-bindgen CLI (see ../build.sh) and a Python with h5py,
# netCDF4 and numpy: CLAWHDF5_PYTHON names it (default python3). Without that
# Python the test is skipped, unless CLAWHDF5_REQUIRE_INTEROP=1.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
PY="${CLAWHDF5_PYTHON:-python3}"
command -v node >/dev/null || { echo "node not found" >&2; exit 1; }
if ! "$PY" -c "import h5py, netCDF4, numpy" >/dev/null 2>&1; then
if [ "${CLAWHDF5_REQUIRE_INTEROP:-0}" = "1" ]; then
echo "CLAWHDF5_REQUIRE_INTEROP=1 but $PY lacks h5py/netCDF4/numpy" >&2
exit 1
fi
echo "SKIP: $PY lacks h5py/netCDF4/numpy"
exit 0
fi
bash "$HERE/../build.sh"
fix="$(mktemp -d)"
trap 'rm -rf "$fix"' EXIT
"$PY" "$HERE/make_fixture.py" "$fix"
node "$HERE/test.mjs" "$HERE/../pkg" "$fix"
# The page itself, in headless Chromium when one is available.
status=0
bash "$HERE/browser.sh" "$fix" || status=$?
if [ "$status" = 3 ]; then
echo "SKIP: viewer page in a browser (no Chromium; set CHROME)"
elif [ "$status" != 0 ]; then
exit "$status"
fi
+143
View File
@@ -0,0 +1,143 @@
// Node test of the built wasm package (the exact pkg/ the viewer page loads)
// and the viewer's DOM-free helpers. Run by test/run.sh:
// node test.mjs PKG_DIR FIXTURE_DIR
// FIXTURE_DIR holds fixture.h5, fixture.nc and expected.json from
// make_fixture.py (values as libhdf5 reads them back).
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
const [pkgDir, fixDir] = process.argv.slice(2);
const pkg = await import(pathToFileURL(join(pkgDir, "clawhdf5_wasm.js")));
pkg.initSync({ module: readFileSync(join(pkgDir, "clawhdf5_wasm_bg.wasm")) });
const lib = await import(pathToFileURL(join(import.meta.dirname, "..", "viewer-lib.js")));
let checks = 0;
const eq = (a, b, msg) => { assert.deepEqual(a, b, msg); checks++; };
const ARRAY_TYPES = {
f32: Float32Array, f64: Float64Array, i8: Int8Array, i16: Int16Array, i32: Int32Array,
i64: BigInt64Array, u8: Uint8Array, u16: Uint16Array, u32: Uint32Array, u64: BigUint64Array,
strings: Array,
};
function values(kind, data) {
const arr = Array.from(data);
if (kind === "f32" || kind === "f64" || kind === "strings") return arr;
return arr.map(String);
}
function checkAttr(ctx, a, want) {
const v = a.value;
if ("raw" in want) {
eq(v, null, ctx);
assert.ok(a.dtype.includes(want.raw), `${ctx}: ${a.dtype}`);
return;
}
eq(a.dtype, null, `${ctx} dtype`);
if ("string" in want) return eq(v, want.string, ctx);
if ("strings" in want) return eq(v, want.strings, ctx);
if ("int" in want) {
if (want.scalar) {
assert.ok(typeof v === "number" || typeof v === "bigint", ctx);
return eq([String(v)], want.int, ctx);
}
assert.ok(v instanceof BigInt64Array || v instanceof BigUint64Array, ctx);
return eq(Array.from(v, String), want.int, ctx);
}
if ("float" in want) {
if (want.scalar) return eq([v], want.float, ctx);
assert.ok(v instanceof Float64Array, ctx);
return eq(Array.from(v), want.float, ctx);
}
assert.fail(`${ctx}: unknown expectation ${JSON.stringify(want)}`);
}
const expected = JSON.parse(readFileSync(join(fixDir, "expected.json"), "utf8"));
for (const [name, exp] of Object.entries(expected)) {
const file = pkg.open(new Uint8Array(readFileSync(join(fixDir, name))));
for (const [path, want] of Object.entries(exp.lists)) {
eq(file.kind(path), "group", `${name}:${path} kind`);
const list = file.list(path);
for (const [kind, key] of [["group", "groups"], ["dataset", "datasets"]]) {
eq(list.filter((c) => c.kind === kind).map((c) => c.name).sort(), want[key], `${name}:${path} ${key}`);
}
}
for (const [path, want] of Object.entries(exp.datasets)) {
const ctx = `${name}:${path}`;
eq(file.kind(path), "dataset", `${ctx} kind`);
if (want.unavailable) {
// The wasm build has no zstd (it links C): a clear error, no data.
assert.throws(() => file.read(path), (e) => e.message.includes(want.unavailable), ctx);
checks++;
continue;
}
const info = file.info(path);
eq([...info.shape, ...info.elementShape], want.shape, `${ctx} info shape`);
const r = file.read(path);
eq(r.shape, want.shape, `${ctx} shape`);
eq(r.dtype, info.dtype, `${ctx} dtype`);
assert.ok(r.data instanceof ARRAY_TYPES[want.kind], `${ctx}: ${r.data.constructor.name} for ${want.kind}`);
eq(values(want.kind, r.data), want.values, ctx);
if (want.slab) {
const s = want.slab;
const part = file.readHyperslab(path, s.start, s.count, s.stride);
eq(part.shape, s.shape, `${ctx} slab shape`);
eq(values(want.kind, part.data), s.values, `${ctx} slab`);
}
}
for (const [path, what] of Object.entries(exp.errors)) {
assert.throws(() => file.read(path), (e) => e instanceof Error && e.message.includes(what), `${name}:${path}`);
checks++;
}
for (const [path, want] of Object.entries(exp.attrs)) {
const attrs = file.attrs(path);
eq(file.attrErrors(path), [], `${name}:${path} attr errors`);
const seen = attrs.filter((a) => !a.name.startsWith("_") && !exp.skip_attrs.includes(a.name));
eq(seen.map((a) => a.name).sort(), Object.keys(want).sort(), `${name}:${path} attr names`);
for (const a of seen) checkAttr(`${name}:${path}@${a.name}`, a, want[a.name]);
}
file.free();
}
// Errors reach JavaScript as thrown Errors, never as data.
const h5 = pkg.open(new Uint8Array(readFileSync(join(fixDir, "fixture.h5"))));
const throwsMsg = (fn, re) => { assert.throws(fn, (e) => e instanceof Error && re.test(e.message)); checks++; };
throwsMsg(() => pkg.open(new Uint8Array(64)), /./);
throwsMsg(() => h5.read("/nope"), /./);
throwsMsg(() => h5.list("/grid"), /not a group/);
throwsMsg(() => h5.readHyperslab("/grid", [0], [1]), /dimensions/);
throwsMsg(() => h5.readHyperslab("/grid", [5, 0], [2, 1]), /exceeds/);
throwsMsg(() => h5.readHyperslab("/grid", [-1, 0], [1, 1]), /non-negative integers/);
throwsMsg(() => h5.readHyperslab("/grid", [0.5, 0], [1, 1]), /non-negative integers/);
// Info for a dataset with an unlimited dimension (netCDF "time").
const nc = pkg.open(new Uint8Array(readFileSync(join(fixDir, "fixture.nc"))));
eq(nc.info("/time").maxshape, [null], "unlimited dimension is null");
// Big integers stay exact.
eq(h5.read("/u64").data[0], 18446744073709551615n, "u64 max");
eq(typeof pkg.version(), "string", "version");
// Viewer helpers.
eq(lib.joinPath("/", "a"), "/a", "joinPath root");
eq(lib.joinPath("/a", "b"), "/a/b", "joinPath nested");
eq(lib.viewWindow([], {}), null, "scalar window");
eq(lib.viewWindow([7], { row: 5, rows: 50 }), { start: [5], count: [2], rows: 2, cols: 1, row: 5, col: 0 }, "1-D window");
const w = lib.viewWindow([2, 5, 6], { row: 1, col: 4, rows: 3, cols: 5, fixed: [1] });
eq(w, { start: [1, 1, 4], count: [1, 3, 2], rows: 3, cols: 2, row: 1, col: 4 }, "3-D window");
// The window the page would request reads the same values as a direct slab.
const cube = h5.readHyperslab("/cube", w.start, w.count);
eq(lib.toRows(cube.data, w.rows, w.cols), [["40", "41"], ["46", "47"], ["52", "53"]], "cube window cells");
const pairs = h5.readHyperslab("/pairs", [1], [2]);
eq(lib.toRows(pairs.data, 2, 1, lib.perElement([2])), [["[2, 3]"], ["[4, 5]"]], "array-type cells");
eq(lib.formatValue(0.1 + 0.2), "0.3", "float formatting");
eq(lib.formatValue(2n ** 64n - 1n), "18446744073709551615", "bigint formatting");
eq(lib.formatValue("x"), '"x"', "string formatting");
h5.free();
nc.free();
console.log(`wasm package: ${checks} checks passed`);
+77
View File
@@ -0,0 +1,77 @@
// DOM-free helpers for the viewer, so Node can test them (test/test.mjs).
/** Child path of `name` in the group at `parent`. */
export function joinPath(parent, name) {
return parent === "/" ? `/${name}` : `${parent}/${name}`;
}
/** One value as display text. */
export function formatValue(v) {
if (v === null || v === undefined) return "";
if (typeof v === "bigint") return v.toString();
if (typeof v === "number") {
if (Number.isInteger(v)) return String(v);
return String(Number(v.toPrecision(7)));
}
if (typeof v === "string") return JSON.stringify(v);
if (ArrayBuffer.isView(v) || Array.isArray(v)) {
const items = Array.from(v.slice(0, 16), formatValue);
if (v.length > 16) items.push(`… (${v.length} values)`);
return `[${items.join(", ")}]`;
}
return String(v);
}
/**
* The window of a dataset to show: a hyperslab over its `shape` with the
* last dimension as columns, the one before as rows, and any leading
* dimensions held at `fixed` indices. `row`/`col` are the window's top-left
* corner. Returns null for a scalar (read it whole).
*/
export function viewWindow(shape, { row = 0, col = 0, rows = 50, cols = 12, fixed = [] } = {}) {
const rank = shape.length;
if (rank === 0) return null;
const clamp = (x, n) => Math.max(0, Math.min(x, Math.max(0, n - 1)));
if (rank === 1) {
const r0 = clamp(row, shape[0]);
const n = Math.max(0, Math.min(rows, shape[0] - r0));
return { start: [r0], count: [n], rows: n, cols: 1, row: r0, col: 0 };
}
const lead = shape.slice(0, rank - 2).map((n, i) => clamp(fixed[i] ?? 0, n));
const nr = shape[rank - 2];
const nc = shape[rank - 1];
const r0 = clamp(row, nr);
const c0 = clamp(col, nc);
const r = Math.max(0, Math.min(rows, nr - r0));
const c = Math.max(0, Math.min(cols, nc - c0));
return {
start: [...lead, r0, c0],
count: [...lead.map(() => 1), r, c],
rows: r,
cols: c,
row: r0,
col: c0,
};
}
/**
* Split row-major `data` into `rows` x `cols` cells of display text; each
* cell holds `per` consecutive values (the elements of an array datatype).
*/
export function toRows(data, rows, cols, per = 1) {
const out = [];
for (let r = 0; r < rows; r++) {
const row = [];
for (let c = 0; c < cols; c++) {
const i = (r * cols + c) * per;
row.push(per === 1 ? formatValue(data[i]) : formatValue(data.slice(i, i + per)));
}
out.push(row);
}
return out;
}
/** Number of values an array datatype packs into each element. */
export function perElement(elementShape) {
return elementShape.reduce((a, b) => a * b, 1);
}
+32 -2
View File
@@ -108,14 +108,17 @@ run_step "cargo clippy (fast-deflate / zlib-ng)" cargo clippy \
# that: fail if a crate that compiles C (a *-sys crate, cc or cmake) enters the
# default dependency tree of any of them. clawhdf5-migrate (bundled SQLite),
# clawhdf5-napi (Node) and clawhdf5-gpu (graphics drivers) are exempt.
# js-sys (clawhdf5-wasm's bindings to JavaScript) builds no C.
no_c_in_default_build() {
local crate found=0
for crate in clawhdf5-format clawhdf5-io clawhdf5-filters clawhdf5 \
clawhdf5-agent clawhdf5-ann clawhdf5-accel clawhdf5-netcdf4 clawhdf5-cli \
clawhdf5-tools; do
clawhdf5-tools \
clawhdf5-wasm; do
local c_deps
c_deps=$(cargo tree -q -p "$crate" -e normal,build --prefix none \
| grep -E '^([a-z0-9_-]+-sys|cc|cmake) v' | sort -u)
| grep -E '^([a-z0-9_-]+-sys|cc|cmake) v' \
| grep -v '^js-sys v' | sort -u)
if [ -n "$c_deps" ]; then
echo "$crate pulls in C by default:"
echo "$c_deps" | sed 's/^/ /'
@@ -126,6 +129,33 @@ no_c_in_default_build() {
}
run_step "no C in the default build (core crates)" no_c_in_default_build
# The reader in the browser: the facade's read path must build for
# wasm32-unknown-unknown (no mmap, no threads, no file system), and the
# wasm-bindgen crate must build and lint there. Needs
# `rustup target add wasm32-unknown-unknown`.
run_step "wasm32 build (clawhdf5, no default features)" cargo build \
-p clawhdf5 \
--target wasm32-unknown-unknown \
--no-default-features
run_step "wasm32 clippy (clawhdf5-wasm)" cargo clippy \
-p clawhdf5-wasm \
--target wasm32-unknown-unknown \
--all-targets \
-- -D warnings
# The built wasm package, run under Node against h5py/netCDF4-written files,
# and the viewer page in headless Chromium when one is found.
# Needs node and the wasm-bindgen CLI, which the CI container does not have;
# the same expectations are checked natively by clawhdf5-wasm's h5py_interop
# test in the cargo test step.
if command -v node >/dev/null && command -v wasm-bindgen >/dev/null; then
run_step "wasm package under Node (+ browser)" bash "$SCRIPT_DIR/../examples/wasm-viewer/test/run.sh"
else
echo ""
echo "==> [wasm package under Node] SKIPPED: needs node and wasm-bindgen"
STEPS+=("SKIP: wasm package under Node")
fi
# The workspace declares a minimum Rust version (rust-version in Cargo.toml);
# check that it really builds there, so the README badge and the manifests
# cannot drift from the truth. Separate target dir: a different toolchain