osobhandClaude Opus 5.5
b0b4018919
QUICKSTART used APIs that do not exist (file.dataset_names(), File::attr, AttrValue::Str, memory.search(&q, 5), MemoryConfig::new with a &str, consolidation without timestamps), `clawhdf5 = "2.0"` from crates.io, and "3-45x faster than libhdf5". It now covers HDF5 in Rust (write, read, strings, in-place append, remote, SWMR), Python (read, r+, w, URLs), NetCDF-4, h5rs, agent memory and the CLI, every snippet compiled and run (Python against a wheel built from the tree). USE_CASES dropped claims with no source (the agent crate adds ~2MB, IVF-PQ under 1.2 ms on modest hardware, an OpenClaw scenario, a .brain layout and `clawhub publish` commands) and now covers the HDF5 cases (no-C builds, threads, remote data, untrusted files, SWMR, in-place edits), the agent cases with measured numbers, and when to use something else. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
7.7 KiB
7.7 KiB
Where clawhdf5 fits
Situations clawhdf5 was built for, what it gives you in each, and — at the end — when to use something else. Code for each is in QUICKSTART.md; limits are in known-issues.md.
HDF5 data
Reading HDF5 where libhdf5 is a burden
You ship a Rust service, a CLI, a static binary, a WebAssembly page or a cross-compiled ARM build, and linking libhdf5 (and its C toolchain, threadsafe-build and version questions) is the hard part.
- The default build compiles no C at all, including deflate (pure-Rust
zlib-rs);
scripts/ci-test.shfails if a C-building crate enters the core crates' default dependency tree. - Reads are checked against h5py object by object on 697 public files; 602 are identical and none mismatches (CONFORMANCE.md).
- The common plugin filters (LZF, bitshuffle, bzip2, Blosc, Blosc2, ZFP) are pure Rust too, so files written with hdf5plugin read without installing plugins.
Many threads reading one file
A service answers requests from one large HDF5 file, and h5py threads do not scale (libhdf5 serialises API calls; h5py users fall back to process pools).
- A
clawhdf5::FileisSend + Syncwith no library-wide lock: open it once and share it. - Full reads of deflate data from 16 threads through one
Fileran at 1.58x the throughput of 16 h5py processes on tank on 2026-09-26 (BENCHMARKS.md). - The Python bindings release the GIL for every read, so Python threads get the same.
Data on a web server or in object storage
The file is on HTTP, S3, GCS or Azure, and you need a few datasets from it, not the whole download.
clawhdf5_remote::open_url(Rust),clawhdf5.File(url)(Python) andh5rswith--features remoteread by range requests through a block cache, with the file pinned by ETag/Last-Modified so a changed file is an error rather than mixed data.- In the browser,
clawhdf5-wasm'sopenUrldoes the same from the page's main thread; the viewer is a working example. Opening one dataset of a 3000-dataset, 198 MB h5py file took 5 requests and 5.2 MB at 1 MiB blocks (tank, 2026-09-27, CHANGELOG). - Design and measured request counts: design/range-reads.md.
Files you did not write and do not trust
User uploads, files from instruments or old archives, fuzzed inputs.
- On the HDF Group's CVE corpus clawhdf5 has no panic, crash, hang or runaway allocation, where h5dump 1.14.6 crashes on 2 files and h5py on 1 (CONFORMANCE.md).
h5rs check --data file.h5validates the structures and checksums and decodes every dataset; it uses the library's parsers, so it accepts what they accept, not everything libhdf5 would reject.
Watching a running experiment
An acquisition process writes with libhdf5 in SWMR mode and a dashboard or monitor follows it.
File::open_swmr+Dataset::refresh()follow the writer as h5py's SWMR reader does, retrying reads that race a flush and never returning torn data. Tested live against an h5py writer.- clawhdf5 does not write SWMR files; the writer stays libhdf5.
Patching files in place
Fix a calibration constant, append to a time series, grow a dataset: files too large to rewrite, or written by someone else.
FileEditor(Rust) andclawhdf5.File(path, 'r+')(Python) overwrite values, resize chunked datasets and set attributes without rewriting the file, changing indexes and heaps as libhdf5 does; everything is checked against h5py and h5dump in the tests.- Anything it cannot do safely is refused before a byte is written.
Agent memory
A personal assistant that remembers
An assistant accumulates preferences, decisions and context over months.
clawhdf5-agentkeeps records, sessions and a knowledge graph in one.h5file with a write-ahead log: back it up or move it with the agent.- Hybrid search (HNSW + BM25) reaches 81.4% turn-level Hit@5 on the full LongMemEval haystack — retrieval recall, not QA accuracy (BENCHMARKS.md).
- The consolidation engine (Working → Episodic → Semantic) and the knowledge graph are library components you drive; see agent-memory.md.
Several agents, kept apart
A coding agent, a research agent and a scheduler should not read each other's memories.
- One store per agent; each store has a single writer (an exclusive lock), and other processes can open it read-only.
SearchOptions::with_sourcesrestricts a search to chosen source channels.- The write-anomaly detector flags injection patterns and write bursts
(alerts, never blocks); its source classification is a heuristic on the
source_channelstring, not an authenticated boundary. - There is no built-in way to share a graph between stores; export and import it yourself.
On a small device
A Raspberry Pi or another ARM board, no server, no network.
- Pure Rust, no database server, one file.
- The int8 index uses NEON
SDOTon cores with the dot-product extension (plain NEON elsewhere); on a Raspberry Pi 5 it was 1.18x thef32index's QPS at equal recall (BENCHMARKS.md). CI builds and tests the aarch64 code on an ARM runner. - WAL appends are not fsynced: on power loss, saves since the last
checkpoint can be lost, while checkpoints themselves are made durable as
a unit. Checkpoint (
flush_wal) as often as you need. clawhdf5-androidhas JNI bindings for the store.
Tamper-evident memory
You need to know whether a store was edited outside your agent.
- With a signing key, every checkpoint stores an Ed25519-signed manifest
(SHA-256 per record in a Merkle tree, plus settings, sessions and graph);
HDF5Memory::verifynames the records that changed. Saves still in the WAL are not covered until the next checkpoint.
.brain files (ClawBrainHub)
ClawBrainHub packages agents as .brain files,
which are HDF5 files its cbh-core crate reads and writes through
clawhdf5's facade (File, FileBuilder, AttrValue, Selection). It is
the one verified consumer of clawhdf5.
When to use something else
- Parallel writes from MPI ranks:
clawhdf5-io'smpi-iogathers writes to rank 0 and reads on one rank then broadcasts; it is not collective I/O. Use libhdf5 with MPI-IO. - Writing SWMR files, creating or deleting objects in an existing file, writing variable-length data, writing Blosc2 or ZFP: not supported.
- Files that must open in HDF5 1.8: clawhdf5's output is not tested there.
- Node.js: the package does not work (known-issues.md).
- An OpenClaw or ZeroClaw memory backend: clawhdf5 is neither (openclaw.md).
Choosing features
| Situation | Crate / features |
|---|---|
| Read and write HDF5 | clawhdf5 (defaults: mmap, provenance, lzf) |
| Plugin-filtered files (hdf5plugin) | clawhdf5, features = ["plugin-filters"] |
| Zstd, LZ4 | zstd (links libzstd), lz4 |
| SZIP | clawhdf5-format's szip (libaec, C) |
| zlib-ng instead of zlib-rs | fast-deflate (needs cmake) |
| Remote files | clawhdf5-remote (http default; https, s3, gcs, azure) |
| Agent memory | clawhdf5-agent (defaults: float16, hnsw, parallel) |
| BLAS for the agent's brute-force paths | fast-math, openblas, or accelerate (macOS) |
| GPU distance computation | gpu (wgpu) |
| Async wrapper | async (Tokio) |