# Where clawhdf5 fits Situations clawhdf5 was built for, what it gives you in each, and — at the end — when to use something else. Code for each is in [QUICKSTART.md](QUICKSTART.md); limits are in [known-issues.md](known-issues.md). --- ## HDF5 data ### Reading HDF5 where libhdf5 is a burden You ship a Rust service, a CLI, a static binary, a WebAssembly page or a cross-compiled ARM build, and linking libhdf5 (and its C toolchain, threadsafe-build and version questions) is the hard part. - The default build compiles no C at all, including deflate (pure-Rust zlib-rs); `scripts/ci-test.sh` fails if a C-building crate enters the core crates' default dependency tree. - Reads are checked against h5py object by object on 697 public files; 602 are identical and none mismatches ([CONFORMANCE.md](../CONFORMANCE.md)). - The common plugin filters (LZF, bitshuffle, bzip2, Blosc, Blosc2, ZFP) are pure Rust too, so files written with hdf5plugin read without installing plugins. ### Many threads reading one file A service answers requests from one large HDF5 file, and h5py threads do not scale (libhdf5 serialises API calls; h5py users fall back to process pools). - A `clawhdf5::File` is `Send + Sync` with no library-wide lock: open it once and share it. - Full reads of deflate data from 16 threads through one `File` ran at 1.58x the throughput of 16 h5py processes on tank on 2026-09-26 ([BENCHMARKS.md](../BENCHMARKS.md#results-after-in-place-chunk-decoding-2026-09-26-tank-c5334b1)). - The Python bindings release the GIL for every read, so Python threads get the same. ### Data on a web server or in object storage The file is on HTTP, S3, GCS or Azure, and you need a few datasets from it, not the whole download. - `clawhdf5_remote::open_url` (Rust), `clawhdf5.File(url)` (Python) and `h5rs` with `--features remote` read by range requests through a block cache, with the file pinned by ETag/Last-Modified so a changed file is an error rather than mixed data. - In the browser, `clawhdf5-wasm`'s `openUrl` does the same from the page's main thread; the [viewer](../examples/wasm-viewer/README.md) is a working example. Opening one dataset of a 3000-dataset, 198 MB h5py file took 5 requests and 5.2 MB at 1 MiB blocks (tank, 2026-09-27, CHANGELOG). - Design and measured request counts: [design/range-reads.md](design/range-reads.md). ### Files you did not write and do not trust User uploads, files from instruments or old archives, fuzzed inputs. - On the HDF Group's CVE corpus clawhdf5 has no panic, crash, hang or runaway allocation, where h5dump 1.14.6 crashes on 2 files and h5py on 1 ([CONFORMANCE.md](../CONFORMANCE.md#cve-corpus-clawhdf5-vs-h5dump-vs-h5py)). - `h5rs check --data file.h5` validates the structures and checksums and decodes every dataset; it uses the library's parsers, so it accepts what they accept, not everything libhdf5 would reject. ### Watching a running experiment An acquisition process writes with libhdf5 in SWMR mode and a dashboard or monitor follows it. - `File::open_swmr` + `Dataset::refresh()` follow the writer as h5py's SWMR reader does, retrying reads that race a flush and never returning torn data. Tested live against an h5py writer. - clawhdf5 does not write SWMR files; the writer stays libhdf5. ### Patching files in place Fix a calibration constant, append to a time series, grow a dataset: files too large to rewrite, or written by someone else. - `FileEditor` (Rust) and `clawhdf5.File(path, 'r+')` (Python) overwrite values, resize chunked datasets and set attributes without rewriting the file, changing indexes and heaps as libhdf5 does; everything is checked against h5py and h5dump in the tests. - Anything it cannot do safely is refused before a byte is written. --- ## Agent memory ### A personal assistant that remembers An assistant accumulates preferences, decisions and context over months. - `clawhdf5-agent` keeps records, sessions and a knowledge graph in one `.h5` file with a write-ahead log: back it up or move it with the agent. - Hybrid search (HNSW + BM25) reaches 81.4% turn-level Hit@5 on the full LongMemEval haystack — retrieval recall, not QA accuracy ([BENCHMARKS.md](../BENCHMARKS.md#longmemeval-results)). - The consolidation engine (Working → Episodic → Semantic) and the knowledge graph are library components you drive; see [agent-memory.md](agent-memory.md#library-components). ### Several agents, kept apart A coding agent, a research agent and a scheduler should not read each other's memories. - One store per agent; each store has a single writer (an exclusive lock), and other processes can open it read-only. - `SearchOptions::with_sources` restricts a search to chosen source channels. - The write-anomaly detector flags injection patterns and write bursts (alerts, never blocks); its source classification is a heuristic on the `source_channel` string, not an authenticated boundary. - There is no built-in way to share a graph between stores; export and import it yourself. ### On a small device A Raspberry Pi or another ARM board, no server, no network. - Pure Rust, no database server, one file. - The int8 index uses NEON `SDOT` on cores with the dot-product extension (plain NEON elsewhere); on a Raspberry Pi 5 it was 1.18x the `f32` index's QPS at equal recall ([BENCHMARKS.md](../BENCHMARKS.md#on-arm-raspberry-pi-5-cortex-a76)). CI builds and tests the aarch64 code on an ARM runner. - WAL appends are not fsynced: on power loss, saves since the last checkpoint can be lost, while checkpoints themselves are made durable as a unit. Checkpoint (`flush_wal`) as often as you need. - `clawhdf5-android` has JNI bindings for the store. ### Tamper-evident memory You need to know whether a store was edited outside your agent. - With a signing key, every checkpoint stores an Ed25519-signed manifest (SHA-256 per record in a Merkle tree, plus settings, sessions and graph); `HDF5Memory::verify` names the records that changed. Saves still in the WAL are not covered until the next checkpoint. ### `.brain` files (ClawBrainHub) [ClawBrainHub](https://clawbrainhub.com) packages agents as `.brain` files, which are HDF5 files its `cbh-core` crate reads and writes through clawhdf5's facade (`File`, `FileBuilder`, `AttrValue`, `Selection`). It is the one verified consumer of clawhdf5. --- ## When to use something else - **Parallel writes from MPI ranks**: `clawhdf5-io`'s `mpi-io` gathers writes to rank 0 and reads on one rank then broadcasts; it is not collective I/O. Use libhdf5 with MPI-IO. - **Writing SWMR files**, **creating or deleting objects in an existing file**, **writing variable-length data**, **writing Blosc2 or ZFP**: not supported. - **Files that must open in HDF5 1.8**: clawhdf5's output is not tested there. - **Node.js**: the package does not work ([known-issues.md](known-issues.md#the-nodejs-package-packagesclawhdf5-node-does-not-work)). - **An OpenClaw or ZeroClaw memory backend**: clawhdf5 is neither ([openclaw.md](openclaw.md)). ## Choosing features | Situation | Crate / features | |---|---| | Read and write HDF5 | `clawhdf5` (defaults: `mmap`, `provenance`, `lzf`) | | Plugin-filtered files (hdf5plugin) | `clawhdf5`, `features = ["plugin-filters"]` | | Zstd, LZ4 | `zstd` (links libzstd), `lz4` | | SZIP | `clawhdf5-format`'s `szip` (libaec, C) | | zlib-ng instead of zlib-rs | `fast-deflate` (needs cmake) | | Remote files | `clawhdf5-remote` (`http` default; `https`, `s3`, `gcs`, `azure`) | | Agent memory | `clawhdf5-agent` (defaults: `float16`, `hnsw`, `parallel`) | | BLAS for the agent's brute-force paths | `fast-math`, `openblas`, or `accelerate` (macOS) | | GPU distance computation | `gpu` (wgpu) | | Async wrapper | `async` (Tokio) |