h5rs tools, browser reader, libhdf5 header checks, plugin filters, concurrency benchmark #14
@@ -280,6 +280,32 @@
|
|||||||
infinity; batches are all or nothing. CLI: `create --float16`. See
|
infinity; batches are all or nothing. CLI: `create --float16`. See
|
||||||
`BENCHMARKS.md`, "float16 embedding storage".
|
`BENCHMARKS.md`, "float16 embedding storage".
|
||||||
|
|
||||||
|
### Browser (WebAssembly)
|
||||||
|
- **New crate `clawhdf5-wasm`:** the reader compiled to
|
||||||
|
`wasm32-unknown-unknown` with a wasm-bindgen JavaScript API —
|
||||||
|
`open(bytes)`, `list`, `info`, `attrs`, `read`, `readHyperslab` — returning
|
||||||
|
typed arrays of the stored width (`BigInt64Array` for 64-bit integers),
|
||||||
|
string arrays for strings and enums, and a thrown `Error` for types with no
|
||||||
|
typed-array form (compound, reference, opaque, VL sequences) or filters the
|
||||||
|
build lacks (Zstd, SZIP). Read-only; the file is held in memory.
|
||||||
|
- **`examples/wasm-viewer/`:** a drop-a-file HDF5/NetCDF-4 viewer page (tree,
|
||||||
|
type/shape/attributes, values paged as hyperslabs; `?file=&path=` opens a
|
||||||
|
URL). `build.sh` produces the package; `test/run.sh` checks it under Node
|
||||||
|
(251 checks against values h5py/libhdf5 read back from an h5py- and a
|
||||||
|
netCDF4-written file) and renders the page in headless Chromium. Size,
|
||||||
|
measured 2026-09-26 on tank (`gzip -9 -n`): 627,501 B of wasm, 191,639 B
|
||||||
|
gzipped, plus 21,826 B (4,487 B) of JS glue; h5wasm 0.10.3's embedded wasm
|
||||||
|
is 3,544,184 B (907,096 B) — full libhdf5, so not equal functionality. See
|
||||||
|
`examples/wasm-viewer/README.md`.
|
||||||
|
- The facade's read path already built for `wasm32-unknown-unknown` (nothing
|
||||||
|
needed gating); `ci-test.sh` now builds it (`--no-default-features`) and
|
||||||
|
lints `clawhdf5-wasm` for that target, and CI installs the target. The Node
|
||||||
|
and browser tests run in `ci-test.sh` only where `node` and `wasm-bindgen`
|
||||||
|
exist (not the CI container); CI checks the same expectations natively
|
||||||
|
(`clawhdf5-wasm`'s `h5py_interop` test).
|
||||||
|
- `Dataset::raw_datatype()` (facade) returns the full stored datatype, for
|
||||||
|
decoding `read_selection` bytes with `clawhdf5_format::data_read`.
|
||||||
|
|
||||||
### Build
|
### Build
|
||||||
- **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
|
- **Pure-Rust default.** `clawhdf5-format`, `clawhdf5-filters` and the
|
||||||
`clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
|
`clawhdf5` facade default to the `zlib-rs` deflate backend; `fast-deflate`
|
||||||
|
|||||||
@@ -150,6 +150,14 @@ Cargo workspace with 17 crates under `crates/` (plus `libaec-sys`, an internal F
|
|||||||
`MemorySource` for this bookkeeping is inferred from the caller-supplied
|
`MemorySource` for this bookkeeping is inferred from the caller-supplied
|
||||||
`source_channel` string (a heuristic, not an authenticated trust boundary).
|
`source_channel` string (a heuristic, not an authenticated trust boundary).
|
||||||
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
|
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
|
||||||
|
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only, file held in memory;
|
||||||
|
no Zstd/SZIP since they link C) and the `examples/wasm-viewer/` page.
|
||||||
|
`examples/wasm-viewer/test/run.sh` builds the package (needs the
|
||||||
|
`wasm-bindgen` CLI at the crate's exact version) and tests it under Node
|
||||||
|
and headless Chromium (a Playwright download in `~/.cache/ms-playwright`
|
||||||
|
on tank); the CI container has neither, so CI runs the native
|
||||||
|
`clawhdf5-wasm` `h5py_interop` test on the same fixture. Size numbers are
|
||||||
|
in the example's README.
|
||||||
- Python and Node.js bindings for cross-language use
|
- Python and Node.js bindings for cross-language use
|
||||||
- NetCDF-4 compatibility for scientific data interop
|
- NetCDF-4 compatibility for scientific data interop
|
||||||
|
|
||||||
|
|||||||
@@ -422,6 +422,23 @@ the same agent-store interop test.
|
|||||||
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
**Fix:** an empty contiguous dataset gets the undefined address (all `0xff`),
|
||||||
which is what libhdf5 itself writes.
|
which is what libhdf5 itself writes.
|
||||||
|
|
||||||
|
## `clawhdf5-wasm` (browser) limits
|
||||||
|
|
||||||
|
**Status:** open (by design for now; added 2026-09-26).
|
||||||
|
|
||||||
|
- The whole file is held in memory: `open()` takes its bytes. There are no
|
||||||
|
HTTP range reads, so a multi-GB file does not fit a browser tab.
|
||||||
|
- Compound, reference, opaque, bitfield, time and VL-sequence datasets are
|
||||||
|
refused with an error naming the type; attributes of those types come back
|
||||||
|
as `value: null` with their `dtype`.
|
||||||
|
- No Zstd or SZIP (both link C): such datasets fail with
|
||||||
|
`unsupported filter: 32015` / `: 4`. pcodec is not enabled either.
|
||||||
|
- External links and virtual-dataset sources in other files cannot be
|
||||||
|
followed (no file system).
|
||||||
|
- Variable-length string datasets are read by decoding `read_selection`'s
|
||||||
|
bytes with `clawhdf5_format::vl_data` in the wasm crate; `File` itself still
|
||||||
|
cannot (see the audit gaps above).
|
||||||
|
|
||||||
## The Node.js package (`packages/clawhdf5-node`) does not work
|
## The Node.js package (`packages/clawhdf5-node`) does not work
|
||||||
|
|
||||||
**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
|
**Status:** open (found 2026-09-25). Unpublished; not built or tested in CI.
|
||||||
|
|||||||
Reference in New Issue
Block a user