osobhandClaude Opus 5.5
b55b24b7ba
Every crate under crates/ now has a README (android, bench, cli, napi and wasm had none), each saying what the crate is, its main types and functions (names checked against the code), its cargo features with defaults and which ones build C (checked with `cargo tree`), and links to the top-level docs. Corrections to the old stubs: - clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs clawhdf5-format as a dependency. - clawhdf5-filters: deflate backends only, and no library crate depends on it; the filter pipeline and every other codec are in -format. - clawhdf5-gpu: vector distance compute, not I/O; not used by HDF5Memory::search. - clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO. - clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and search(q, k, ef). - clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm backends are reported but run the scalar kernels. - clawhdf5-gpu: the old example called l2_distances, which does not exist (l2_search). - clawhdf5-agent: it described a "vector store" with "GPU acceleration"; it now covers HDF5Memory, search options, WAL, signing, the graph. - crates.io/docs.rs badges removed and `cargo install <crate>` replaced: nothing is published; depend on git. - fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh. - tools: the FileEditor interop tests that live in this crate. - remote, py: license, other front ends, limits, File.mode/flush/chunks. The Rust examples of the facade, format, filters, accel, ann, derive and agent READMEs were compiled and run as tests (netcdf4, gpu and remote compiled only) in a scratch crate; the CLI example was run. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5-wasm
clawhdf5's HDF5 and NetCDF-4 reader compiled to WebAssembly with wasm-bindgen, for the browser (and Node). Read-only. Two ways in:
open(bytes)— a file already in memory (a dropped file, a fetched blob);openUrl(url, opts)— a file on a web server, read by HTTP range requests as each call needs its bytes, without downloading it (range-read milestone M4,docs/design/range-reads.md).
Both give list, info, attrs, read and readHyperslab; the remote
file's methods return promises, and stats() counts requests and bytes.
The JavaScript API, options, limits, package size and tests are documented
with the demo page, examples/wasm-viewer/README.md.
Layout
src/core.rs— the reader over anyclawhdf5_format::storage::Storage(Reader::open_storage), plain Rust and tested natively.src/lazy.rs— the restartable "NeedBytes" cache behindopenUrl: a call runs as a pass over the blocks fetched so far; a pass that misses is abandoned, the missing (and hinted) blocks are fetched, and the pass is run again. No block is evicted while a call runs.js/remote.js— the HTTP side:fetchwithRange, checking every answer (a206with exactly the bytes asked for, same ETag/Last-Modified and length) so a call fails rather than return another file's bytes.src/lib.rs— the wasm-bindgen exports.
Build and test
rustup target add wasm32-unknown-unknown
cargo install wasm-bindgen-cli --version 0.2.129 # must equal the crate's wasm-bindgen
bash examples/wasm-viewer/build.sh # -> examples/wasm-viewer/pkg/
cargo test -p clawhdf5-wasm # native: h5py_interop, lazy, vl_strings
bash examples/wasm-viewer/test/run.sh # Node + headless Chromium (not in CI)
CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus cargo test -p clawhdf5-wasm --test lazy compares every corpus file read lazily with the
same file read from bytes.
Built without mmap and parallel and without the Zstd and SZIP filters
(they link C): such datasets fail with unsupported filter. No C is
compiled; publish = false (it is distributed as the package
build.sh makes).
License
MIT