quantumclaw/clawhdf5
3
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: crate READMEs describe each crate as it is today

Browse Source 
Every crate under crates/ now has a README (android, bench, cli, napi and
wasm had none), each saying what the crate is, its main types and
functions (names checked against the code), its cargo features with
defaults and which ones build C (checked with `cargo tree`), and links to
the top-level docs.

Corrections to the old stubs:
- clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs
  clawhdf5-format as a dependency.
- clawhdf5-filters: deflate backends only, and no library crate depends
  on it; the filter pipeline and every other codec are in -format.
- clawhdf5-gpu: vector distance compute, not I/O; not used by
  HDF5Memory::search.
- clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO.
- clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and
  search(q, k, ef).
- clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm
  backends are reported but run the scalar kernels.
- clawhdf5-gpu: the old example called l2_distances, which does not
  exist (l2_search).
- clawhdf5-agent: it described a "vector store" with "GPU acceleration";
  it now covers HDF5Memory, search options, WAL, signing, the graph.
- crates.io/docs.rs badges removed and `cargo install <crate>` replaced:
  nothing is published; depend on git.
- fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh.
- tools: the FileEditor interop tests that live in this crate.
- remote, py: license, other front ends, limits, File.mode/flush/chunks.

The Rust examples of the facade, format, filters, accel, ann, derive and
agent READMEs were compiled and run as tests (netcdf4, gpu and remote
compiled only) in a scratch crate; the CLI example was run.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-28 11:13:30 -05:00
co-authored by Claude Opus 5.5
parent 9b5803f587
commit b55b24b7ba
20 changed files with 901 additions and 142 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/clawhdf5-accel/README.md
+51 -14
View File
@@ -1,24 +1,61 @@
# clawhdf5-accel
[![crates.io](https://img.shields.io/crates/v/clawhdf5-accel.svg)](https://crates.io/crates/clawhdf5-accel)
[![docs.rs](https://docs.rs/clawhdf5-accel/badge.svg)](https://docs.rs/clawhdf5-accel)
CPU SIMD kernels for vector search: dot products, cosine similarity, L2
distance, norms and int8 dot products, dispatched at run time to the best
backend the CPU has, with a portable scalar fallback for every operation.
[`clawhdf5-ann`](../clawhdf5-ann/README.md) and
[`clawhdf5-agent`](../clawhdf5-agent/README.md) use it in their distance
loops; it has nothing to do with HDF5 file I/O.
SIMD-accelerated operations for clawhdf5.
Not on crates.io yet; depend on it from git:
```toml
[dependencies]
clawhdf5-accel = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## API
```rust
use clawhdf5_accel::{cosine_similarity, detect_backend, dot_i8, dot_product, l2_distance};
let a = [1.0f32, 2.0, 3.0, 4.0];
let b = [4.0f32, 3.0, 2.0, 1.0];
assert_eq!(dot_product(&a, &b), 20.0);
let _cos = cosine_similarity(&a, &b);
let _l2 = l2_distance(&a, &b);
assert_eq!(dot_i8(&[1, -2, 3], &[4, 5, -6]), -24);
println!("{:?}", detect_backend()); // e.g. Avx2 on x86-64, Neon on aarch64
```
Also `vector_norm`, `batch_norms`, `batch_cosine`, `batch_cosine_prenorm`,
`f16_to_f32_batch`, `checksum_fletcher32` and `align_to_cache_line`.
## Backends
`detect_backend()` picks once per process: `Avx512` (with the `avx512`
feature), `Avx2` (AVX2 + FMA), `Neon` (every aarch64 CPU), or `Scalar`.
`Sse4` and `WasmSimd128` are reported when detected but run the scalar
kernels.
`dot_i8`, used by the agent's quantised (int8) HNSW index, runs on
AVX2 and on NEON — with the `SDOT` instruction (through inline assembly,
since the intrinsic is unstable) on cores that have dotprod, such as the
Raspberry Pi 5, and plain NEON on older ones. At equal recall the int8
index answers 1.63x the queries per second of the f32 one on x86-64
(AVX2) and 1.18x on a Raspberry Pi 5 ([`BENCHMARKS.md`](../../BENCHMARKS.md)).
The aarch64 code is compiled out on x86, so only the `test-arm64` CI job
builds and tests it.
## Features
- AVX2 and NEON SIMD acceleration
- AVX-512 support (`avx512` feature)
- Float16 conversion (`float16` feature)
- CRC32 checksum acceleration
| Feature | Default | What | Builds C |
|---|---|---|---|
| `avx512` | no | AVX-512F kernels | no |
| `float16` | no | `f16_to_f32_batch` through the `half` crate (a software conversion otherwise) | no |
## Usage
```rust
use clawhdf5_accel::checksum::crc32_simd;
let crc = crc32_simd(&data);
```
The half-precision conversion used for stored embeddings is
`clawhdf5_format::float16`, not this crate's.
## License
crates/clawhdf5-agent/README.md
+112 -16
View File
@@ -1,28 +1,124 @@
# clawhdf5-agent
[![crates.io](https://img.shields.io/crates/v/clawhdf5-agent.svg)](https://crates.io/crates/clawhdf5-agent)
[![docs.rs](https://img.shields.io/docsrs/clawhdf5-agent)](https://docs.rs/clawhdf5-agent)
Persistent memory for AI agents in a single HDF5 file: text chunks with
embeddings and metadata, hybrid search (HNSW vector search + BM25 keyword
search, fused), sessions, a knowledge graph, a write-ahead log for crash
safety, and optionally Ed25519-signed checkpoints. Stores open in h5py like
any other HDF5 file. Built on [`clawhdf5`](../clawhdf5/README.md),
[`clawhdf5-ann`](../clawhdf5-ann/README.md) and
[`clawhdf5-accel`](../clawhdf5-accel/README.md).
HDF5-backed persistent memory store for on-device AI agents.
It is a library: no agent framework integrates it (OpenClaw and ZeroClaw
integration claims were withdrawn on 2026-09-25; see
[`docs/openclaw.md`](../../docs/openclaw.md)). The command-line front end
is [`clawhdf5-cli`](../clawhdf5-cli/README.md).
Built on [clawhdf5](https://crates.io/crates/clawhdf5), clawhdf5-agent provides a vector-searchable memory backend optimized for edge AI workloads. Store embeddings, text chunks, and metadata in a single HDF5 file with SIMD-accelerated similarity search.
## Features
- Persistent vector store in HDF5 format
- Cosine similarity and L2 distance search
- SIMD-accelerated via clawhdf5-accel (AVX2, NEON)
- Optional GPU acceleration via clawhdf5-gpu
- Memory-mapped access for large stores
- f16 storage support for compact embeddings
## Usage
Not on crates.io yet; depend on it from git:
```toml
[dependencies]
clawhdf5-agent = "2.1.0"
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## Usage
```rust,no_run
use std::path::PathBuf;
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions};
let config = MemoryConfig::new(PathBuf::from("agent.h5"), "my-agent", 384);
let mut mem = HDF5Memory::create(config)?;
mem.save(MemoryEntry {
chunk: "The deploy key rotates every Monday.".into(),
embedding: vec![0.01; 384], // from your embedding model
source_channel: "chat".into(),
timestamp: 1_790_000_000.0,
session_id: "s1".into(),
tags: "ops".into(),
})?;
let query = vec![0.01f32; 384];
let hits = mem.search(&query, "deploy key", &SearchOptions::new(5).with_sources(["chat"]));
for h in &hits {
println!("{:.3} {}", h.score, h.chunk);
}
mem.flush_wal()?; // checkpoint now; otherwise one is made after 500 WAL entries (wal_max_entries)
# Ok::<(), clawhdf5_agent::MemoryError>(())
```
## What is in it
- **`HDF5Memory`** — `create`, `open` (single writer: an exclusive lock on
`<store>.h5.lock`, a second opener gets `MemoryError::Locked`),
`open_read_only` (no lock, never writes). Through the `AgentMemory`
trait: `save`, `save_batch`, `delete`, `compact`, `count`, `snapshot`,
sessions; also `save_or_update`, `delete_batch`, `flush_wal`.
- **Search** — `search(query_embedding, text, &SearchOptions)`: optional
source-channel filter applied before ranking, vector + BM25 fusion
(weighted or RRF), Hebbian activation scaling, optional re-ranking
(`reranker::ReRankConfig`) and confidence rejection
(`confidence::ConfidenceConfig`). `hybrid_search` and
`hybrid_search_with` are thin wrappers. The vector stage uses the HNSW
index (`hnsw` feature); its graph is saved to `<store>.h5.ann` at each
checkpoint and reloaded on open (rebuilt if stale or damaged).
- **Storage settings** (`MemoryConfig`, persisted with the store):
`float16` embeddings (on by default for new stores; 48% smaller file at
100K records, same retrieval on LongMemEval), `quantized_index` (int8
copy of the vectors in the index, on by default; re-scored against the
exact embeddings), `compression` (off by default), HNSW `m`/`ef`
parameters, WAL settings (`wal_enabled`, on by default; `wal_max_entries`,
500: the WAL is checkpointed into the `.h5` once it holds more).
- **WAL** (`wal`) — every write is appended to `<store>.h5.wal` with a
chained CRC32 per entry, so a corrupted, reordered or spliced entry stops
replay. Recovers from a process crash at any point, including between a
checkpoint and the WAL truncate. WAL appends are not fsynced: saves since
the last checkpoint can be lost on power failure. An unreadable WAL is
quarantined to `<store>.h5.wal.corrupt-<ts>`.
- **Signed checkpoints** (`signing`) — `set_signing_key` signs a manifest
(SHA-256 Merkle tree over records, plus settings, sessions and graph) at
every checkpoint; `HDF5Memory::verify(path, &public_key)` checks it and
locates edits. WAL entries after the checkpoint are not covered.
- **Knowledge graph** (`knowledge`, `entity_extract`) — `add_entity`,
`add_entity_alias`, `add_relation`, `extract_and_store_entities`,
traversal and spreading activation.
- **Also:** sessions (`session`), temporal index (`temporal`),
consolidation tiers (`consolidation`), an in-memory TTL tier
(`ephemeral`), multi-modal embeddings (`multimodal`), `AGENTS.md`
generation (`agents_md`), query expansion, and a session-scoped
provenance ledger and write-anomaly detector on every save
(`take_anomaly_alerts`; alerts never block a save, and the source is
inferred from `source_channel`, not authenticated).
- `openclaw::ClawhdfBackend` is `search` with re-ranking and confidence
on, plus Markdown import/export. The module name is historical: it is not
an OpenClaw plugin.
## Features
| Feature | Default | What | Builds C |
|---|---|---|---|
| `hnsw` | yes | HNSW vector index (`clawhdf5-ann`); without it the vector stage is an exact linear cosine scan | no |
| `parallel` | yes | build the HNSW index on a rayon pool (same graph either way) | no |
| `float16` | yes | f16 helpers in `vector_search` (`half`). Stores' `MemoryConfig::float16` works without it. | no |
| `fast-math` | no | `matrixmultiply` batch distances in `strategy` | no |
| `accelerate` | no | Apple Accelerate BLAS in `strategy` (macOS) | links a system framework |
| `openblas` | no | OpenBLAS in `strategy` | yes (`openblas-src`) |
| `gpu` | no | `gpu_search` through [`clawhdf5-gpu`](../clawhdf5-gpu/README.md) (wgpu), used by `strategy`, not by `HDF5Memory::search` | no, but needs GPU drivers |
| `zstd` | no | Zstd instead of deflate when `MemoryConfig::compression` is on | yes (libzstd) |
| `async` | no | `async_memory` wrapper on tokio | no |
`--no-default-features --features float16` forces the exact linear scan.
## Measurements and limits
- Search recall and latency, file size, LongMemEval and MemoryArena
retrieval numbers: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with
the `clawhdf5-bench` binaries (`search_harness`, `longmemeval_bench`,
`footprint_bench`, ...).
- Known issues and their history: [`docs/known-issues.md`](../../docs/known-issues.md).
- Migrating a SQLite memory database:
[`clawhdf5-migrate`](../clawhdf5-migrate/README.md).
## License
MIT
crates/clawhdf5-android/README.md
+44
View File
@@ -0,0 +1,44 @@
# clawhdf5-android
A C ABI over [`clawhdf5-agent`](../clawhdf5-agent/README.md) for Android
apps: a `cdylib` exporting `extern "C"` functions (`edgehdf5_*`, a name
kept from the project's earlier "edgehdf5" days) that manage an
`HDF5Memory` through an opaque handle.
The functions are plain C symbols, not JNI-mangled `Java_...` entry points:
a Kotlin/Java app calls them through a thin JNI shim or JNA of its own. No
such shim, Gradle project or AAR is in this repository, and the crate is
not built for an Android target in CI (only its host-side unit tests run
with the workspace).
## Functions
| Function | What |
|---|---|
| `edgehdf5_create(path, agent_id, embedding_dim)` / `edgehdf5_open(path)` | a handle, or null on failure |
| `edgehdf5_close(handle)` | drop the store; what is not yet checkpointed stays in its WAL, as with any `HDF5Memory` |
| `edgehdf5_save(handle, ...)` | save one entry; the embedding length is checked against the store's dimension before the pointer is read |
| `edgehdf5_delete`, `edgehdf5_count`, `edgehdf5_count_active` | |
| `edgehdf5_hybrid_search(handle, query, len, text, vector_weight, keyword_weight, max_results, out_indices, out_scores, out_chunks)` | results into caller-provided arrays; returns the number written |
| `edgehdf5_add_session`, `edgehdf5_get_session_summary` | sessions |
| `edgehdf5_add_entity`, `edgehdf5_add_relation` | knowledge graph |
| `edgehdf5_free_string` | free a string this library returned |
Every function is `unsafe`: the caller guarantees valid, NUL-terminated
strings and correctly sized buffers (see each function's `# Safety`
section), and serialises access to a handle; separate handles are
independent.
## Build
```bash
cargo build --release -p clawhdf5-android # host build; for a device, add --target aarch64-linux-android with the NDK's linker configured
```
It depends on `clawhdf5-agent` with **default features off**, so there is
no HNSW index (the vector stage is an exact linear scan) and no rayon
pool. No C is compiled.
## License
MIT
crates/clawhdf5-ann/README.md
+55 -10
View File
@@ -1,25 +1,70 @@
# clawhdf5-ann
[![crates.io](https://img.shields.io/crates/v/clawhdf5-ann.svg)](https://crates.io/crates/clawhdf5-ann)
[![docs.rs](https://docs.rs/clawhdf5-ann/badge.svg)](https://docs.rs/clawhdf5-ann)
An HNSW (Hierarchical Navigable Small World) approximate nearest-neighbour
index in pure Rust, with cosine or L2 distance, optional int8 storage of
the vectors, deletions, and persistence as an HDF5 file. It is the vector
stage of [`clawhdf5-agent`](../clawhdf5-agent/README.md)'s search (the
agent's `hnsw` feature, on by default); distances run on
[`clawhdf5-accel`](../clawhdf5-accel/README.md)'s SIMD kernels.
HNSW approximate nearest neighbor index stored as HDF5.
Neighbours are chosen with the HNSW paper's diversity heuristic, not plain
closest-M (which capped recall on clustered data at 0.31 recall@10 at 100K
vectors).
## Features
Not on crates.io yet; depend on it from git:
- Build and query HNSW indexes persisted in HDF5 format
- Pure Rust, no C dependencies
- Efficient similarity search for high-dimensional vectors
```toml
[dependencies]
clawhdf5-ann = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## Usage
```rust
use clawhdf5_ann::HnswIndex;
use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};
let index = HnswIndex::from_hdf5("vectors.h5").unwrap();
let neighbors = index.search(&query, 10);
let vectors: Vec<Vec<f32>> = (0..500)
.map(|i| (0..16).map(|j| ((i * 31 + j * 7) % 97) as f32 / 97.0).collect())
.collect();
// m = 16 connections per node, ef_construction = 200
let mut index = HnswIndex::build_with(&vectors, 16, 200, DistanceMetric::Cosine, Storage::Int8);
let hits = index.search(&vectors[42], 10, 64); // (id, distance), closest first; ef >= k
assert!(hits[0].1 < 1e-3); // vector 42 itself (or an identical one)
let id = index.insert(vec![0.5; 16]);
index.mark_deleted(id);
// Persist as HDF5 (a self-contained file: graph and vectors) and load it back
let bytes = index.to_hdf5_bytes().unwrap();
let loaded = HnswIndex::load_from_hdf5(&bytes).unwrap();
assert_eq!(loaded.len(), index.len());
```
- `HnswIndex::build` (L2), `build_with_metric`, `build_with` (metric and
storage); `new`/`new_with` plus `insert` for an index built
incrementally.
- `Storage::Int8` keeps each vector as `i8`, a quarter of the memory; it
applies to `Cosine` only (an L2 index keeps `Float32`). Distances are then
approximate, so a caller that needs exact ranking re-scores the
candidates, as the agent does.
- `mark_deleted`, `is_deleted`, `deleted_count`, `active_len`, `compact`
(returns the old-to-new id map).
- `save_to_hdf5(&mut writer)` / `to_hdf5_bytes` / `load_from_hdf5` store
the whole index; `graph_to_bytes` / `from_graph_bytes` store only the
graph (with a CRC32) for a caller that keeps the vectors elsewhere — the
agent's `<store>.h5.ann` sidecar.
## Features
| Feature | Default | What | Builds C |
|---|---|---|---|
| `parallel` | no | build the graph on a rayon pool; the graph is identical with or without it | no |
Recall and speed against exact search, for the index alone and in the
agent: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with
`cargo run --release -p clawhdf5-bench --bin search_harness`.
## License
MIT
crates/clawhdf5-bench/README.md
+49
View File
@@ -0,0 +1,49 @@
# clawhdf5-bench
The measurement harnesses behind [`BENCHMARKS.md`](../../BENCHMARKS.md):
HDF5 read and write speed (against libhdf5 and h5py where noted) and the
agent store's search, footprint and retrieval quality. Not meant for
publishing; nothing else in the workspace depends on it. Run everything with
`--release`, and quote numbers with the machine, date and command, as
`BENCHMARKS.md` does.
## Binaries
| Binary | Measures |
|---|---|
| `read_harness` | full reads vs hyperslab selections of a chunked 2-D dataset (compressed and not) and a contiguous one: does a selection cost scale with the selection or the dataset? (`-- --large` for 512 MB) |
| `concurrent_read` | decoded read throughput vs threads on one open `File`; `scripts/concurrent_read_h5py.py` runs the same workload with h5py (threads and processes) and `scripts/compare_concurrent_read.py` tabulates both |
| `search_harness` | HNSW recall@10 vs exact search, QPS and latency per `ef`, and end-to-end `HDF5Memory` ingest/checkpoint/open/search at 1K–100K (`--full`); studies: `--float16-study`, `--options-study`, `--signing-study`, `--ann-only --uniform` |
| `longmemeval_bench` | LongMemEval retrieval recall (turn and session Hit@k, MRR) — **retrieval, not QA accuracy**. Oracle or full `longmemeval_s` haystack; `--features embeddings` (or `embeddings-cuda`) embeds with MiniLM, otherwise the vector stage is inert and the run is BM25-only |
| `memory_arena` | a deterministic multi-session retrieval benchmark (BM25-only) |
| `footprint_bench` | file size and bytes per record at 100–100K records, float16 or `--f32`, WAL on/off, compressed or not |
| `consolidation_efficiency` | retrieval before and after consolidation on signal + noise records |
| `ephemeral_perf` | the in-memory ephemeral tier's set/get latency |
| `mpi_io_bench` | `clawhdf5-io`'s `MpiVol` (root-read + broadcast, not collective I/O); needs `--features mpi-io` and `mpirun` |
```bash
cargo run --release -p clawhdf5-bench --bin search_harness -- --full
cargo run --release -p clawhdf5-bench --bin read_harness
```
## Criterion benches and example
- `cargo bench -p clawhdf5-bench` runs `h5bench_write`, `h5bench_read` and
`h5bench_meta` (h5bench-style sequential, chunked, strided and metadata
workloads). `--features libhdf5-compare` adds the same workloads through
libhdf5 (the `hdf5-metno` crate; needs a system libhdf5 1.14).
- `examples/worldmodel_sampling.rs`: shuffled per-frame reads of a
`(N, H, W, C)` `uint8` dataset, clawhdf5 against h5py on the same file.
## Features
| Feature | What | Builds C |
|---|---|---|
| `libhdf5-compare` | libhdf5 variants of the Criterion benches | links the system libhdf5 |
| `mpi-io` | `mpi_io_bench` | yes (`mpi-sys`; needs an MPI installation) |
| `embeddings` | MiniLM embeddings for `longmemeval_bench` (candle) | yes (a `cc` build dependency in the candle/tokenizers tree) |
| `embeddings-cuda` | the same on a CUDA GPU (minutes instead of hours on the full haystack) | yes (CUDA) |
## License
MIT
crates/clawhdf5-cli/README.md
+49
View File
@@ -0,0 +1,49 @@
# clawhdf5-cli
The `clawhdf5` command: create, fill, search and inspect a
[`clawhdf5-agent`](../clawhdf5-agent/README.md) memory store from the
shell. Output is JSON. (For general HDF5 files use `h5rs` from
[`clawhdf5-tools`](../clawhdf5-tools/README.md).)
```bash
cargo install --path crates/clawhdf5-cli # installs `clawhdf5`; not on crates.io yet
# or: cargo run -p clawhdf5-cli -- --help
```
No C is compiled.
## Commands
The store is `--path FILE` (or `CLAWHDF5_PATH`) before the subcommand.
| Command | What |
|---|---|
| `create [--agent-id ID] [--dim N] [--wal] [--f32] [--f32-index]` | a new store (dimension 384 by default); float16 embeddings and an int8 index copy unless `--f32` / `--f32-index`. The WAL is off unless `--wal` (the library's default is on), so each save is checkpointed at once |
| `save [--json '{...}']` | save one entry, from `--json` or stdin: `{"chunk", "embedding", "source_channel", "timestamp", "session_id", "tags"}` |
| `search --embedding '[...]' [--query TEXT] [-k N] [--vector-weight W] [--keyword-weight W]` | hybrid search (defaults 5 results, weights 0.7 / 0.3) |
| `recall INDEX` | one entry by index |
| `stats` | counts and configuration |
| `flush-wal` | checkpoint the WAL into the `.h5` |
| `agents-md [--output FILE]` | generate an `AGENTS.md` from the store |
| `export` | every entry as JSON lines |
| `snapshot DEST` | a copy of the store's `.h5` file |
| `keygen --out FILE` | a new Ed25519 signing key (64 hex characters, created owner-only on Unix) |
| `verify --public-key HEX_OR_FILE` | check a signed store; exit status 2 if it does not verify |
`recall`, `stats`, `agents-md` and `export` open the store read-only
(no lock, nothing written), so they work while another process has it
open. `save`, `search` (which records activation boosts) and `flush-wal`
open it for writing and take the store's lock. With
`--signing-key FILE` (or `CLAWHDF5_SIGNING_KEY`) every checkpoint a command
makes is signed; a signed store refuses to checkpoint without the key.
```bash
clawhdf5 --path mem.h5 create --agent-id demo --dim 3
echo '{"chunk":"hello","embedding":[0.1,0.2,0.3],"source_channel":"cli","timestamp":0,"session_id":"s1","tags":""}' \
| clawhdf5 --path mem.h5 save
clawhdf5 --path mem.h5 search --embedding '[0.1,0.2,0.3]' --query hello -k 3
```
## License
MIT
crates/clawhdf5-derive/README.md
+34 -12
View File
@@ -1,28 +1,50 @@
# clawhdf5-derive
[![crates.io](https://img.shields.io/crates/v/clawhdf5-derive.svg)](https://crates.io/crates/clawhdf5-derive)
[![docs.rs](https://docs.rs/clawhdf5-derive/badge.svg)](https://docs.rs/clawhdf5-derive)
`#[derive(H5Type)]`: maps a Rust struct with named fields to an HDF5
compound datatype. The derive generates three inherent methods:
Derive macros for clawhdf5 HDF5 traits.
- `hdf5_datatype() -> clawhdf5_format::datatype::Datatype` — the
`Datatype::Compound` (members in field order, packed, little-endian);
- `to_bytes(&self) -> Vec<u8>` — one element in that layout;
- `from_bytes(&[u8]) -> Self` — the reverse (panics if the slice is shorter
than the compound).
## Features
Supported field types: `f32`, `f64`, `i8`–`i64`, `u8`–`u64`, `bool`
(stored as `u8`) and fixed-size arrays `[T; N]` of those numeric types.
Tuple structs, enums and nested structs are refused at compile time.
- `#[derive(HDF5Type)]` for automatic HDF5 datatype mapping
- Struct-to-compound-type derivation
The generated code names `clawhdf5_format`, so the crate using the derive
must depend on [`clawhdf5-format`](../clawhdf5-format/README.md) too. Not
on crates.io yet:
## Usage
```toml
[dependencies]
clawhdf5-derive = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
clawhdf5-format = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## Example
```rust
use clawhdf5_derive::HDF5Type;
use clawhdf5_derive::H5Type;
use clawhdf5_format::datatype::Datatype;
#[derive(HDF5Type)]
#[derive(H5Type, Debug, PartialEq)]
struct Point {
x: f64,
y: f64,
z: f64,
id: u32,
pos: [f64; 3],
valid: bool,
}
let p = Point { id: 7, pos: [1.0, 2.0, 3.0], valid: true };
let bytes = p.to_bytes();
assert_eq!(bytes.len(), 4 + 24 + 1);
assert_eq!(Point::from_bytes(&bytes), p);
assert!(matches!(Point::hdf5_datatype(), Datatype::Compound { size: 29, .. }));
```
Tests: `crates/clawhdf5-format/tests/derive_tests.rs`.
## License
MIT
crates/clawhdf5-filters/README.md
+39 -10
View File
@@ -1,27 +1,56 @@
# clawhdf5-filters
[![crates.io](https://img.shields.io/crates/v/clawhdf5-filters.svg)](https://crates.io/crates/clawhdf5-filters)
[![docs.rs](https://docs.rs/clawhdf5-filters/badge.svg)](https://docs.rs/clawhdf5-filters)
Standalone deflate (zlib) compression and decompression with a choice of
backend: pure-Rust zlib-rs (default), zlib-ng, Apple's Compression
framework, or miniz_oxide.
Filter and compression pipeline for clawhdf5.
This crate holds **deflate backends only**. The HDF5 filter pipeline, the
filter registry and every other codec (shuffle, Fletcher-32, N-Bit,
scale-offset, LZ4, Zstd, SZIP, pcodec, LZF, bitshuffle, bzip2, Blosc,
Blosc2, ZFP) live in [`clawhdf5-format`](../clawhdf5-format/README.md),
which calls flate2 itself and selects its deflate backend with its own
features. No library crate of the workspace depends on this one (the
`clawhdf5` facade uses it only in tests).
## Features
Not on crates.io yet; depend on it from git:
- DEFLATE compression/decompression
- Pure-Rust deflate via zlib-rs (default, `zlib-rs` feature)
- zlib-ng instead, if you want it (`fast-deflate` feature; C, needs cmake)
- Apple Compression framework support (`apple-compression` feature)
```toml
[dependencies]
clawhdf5-filters = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## Usage
## API
```rust
use clawhdf5_filters::{deflate_compress, deflate_decompress};
use clawhdf5_filters::{deflate_backend, deflate_compress, deflate_decompress};
let data: Vec<u8> = (0..10_000u32).map(|i| (i % 251) as u8).collect();
let compressed = deflate_compress(&data, 6).unwrap();
// The second argument bounds the output: the expected decompressed size.
let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
assert_eq!(decompressed, data);
println!("backend: {}", deflate_backend()); // "zlib-rs" by default
```
Also `deflate_compress_miniz`/`deflate_decompress_miniz` (always
miniz_oxide) and `fast_deflate::{compress, decompress, active_backend}`.
## Features
Backend priority: `apple-compression` (macOS only) > zlib-ng > zlib-rs >
miniz_oxide (with none enabled).
| Feature | Default | Backend | Builds C |
|---|---|---|---|
| `zlib-rs` | yes | zlib-rs through flate2, with `runtime_detection` (needed for its SIMD) | no |
| `fast-deflate` | no | zlib-ng through flate2 | yes (cmake) |
| `system-zlib` | no | the system zlib through flate2 | yes (`libz-sys`) |
| `apple-compression` | no | Apple Compression framework, macOS only (ignored elsewhere) | no (links a system framework) |
zlib-rs matches zlib-ng on HDF5 reads and writes and produces
byte-identical output: see "Deflate backend" in
[`BENCHMARKS.md`](../../BENCHMARKS.md).
## License
MIT
crates/clawhdf5-format/README.md
+95 -16
View File
@@ -1,27 +1,106 @@
# clawhdf5-format
[![crates.io](https://img.shields.io/crates/v/clawhdf5-format.svg)](https://crates.io/crates/clawhdf5-format)
[![docs.rs](https://docs.rs/clawhdf5-format/badge.svg)](https://docs.rs/clawhdf5-format)
The HDF5 file format in pure Rust: parsers and writers for every on-disk
structure, the filter pipeline and its codecs, and the shared type
definitions the other crates use. Most users want the
[`clawhdf5`](../clawhdf5/README.md) facade, which wraps this crate in an
h5py-like API; use this one directly for low-level access or in `no_std`
code.
Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
Not on crates.io yet; depend on it from git:
```toml
[dependencies]
clawhdf5-format = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## What is in it
- **Parsing:** superblock v0–v3 (`superblock`, with the superblock
extension and metadata cache images, `superblock_ext`), object headers v1
and v2 (`object_header`), every header message the readers use
(`datatype`, `dataspace`, `data_layout` v1–v4 including virtual datasets,
`fill_value`, `attribute`, `link_message`, `shared_message`, ...), groups
old and new (`group_v1` symbol tables with local heaps, `group_v2` with
fractal heaps and v2 B-trees), and every chunk index (v1 B-tree, single
chunk, implicit, fixed array, extensible array, v2 B-tree).
- **Reading data:** `data_read` (contiguous, compact, chunked),
`partial_read` and `selection` (hyperslabs and points), `vl_data`
(variable-length strings and sequences through the global heap),
`chunk_cache`.
- **Storage:** the `storage::Storage` trait (`read_at`, `read_ranges`,
`len`, `hint`) that every read path goes through, so a file can be read
from memory, a file handle or a remote backend
([`clawhdf5-remote`](../clawhdf5-remote/README.md)).
- **Writing:** `file_writer::FileWriter` and the builders in
`type_builders` (datasets, groups, attributes, compound and enum types,
links, virtual datasets, creation-order tracking); chunk indexes and
dense-storage B-trees of any size (`chunked_write`, `btree_v2_write`,
`ea_writer`). Output is read by h5py and h5dump.
- **Filters:** `filter_pipeline` and `filter_registry` (look up by ID; other
IDs can be registered at run time with `register_filter`). Built in:
deflate, shuffle, Fletcher-32, N-Bit, scale-offset; behind features LZ4,
Zstd, SZIP (decode), pcodec, and the plugin filters LZF, bitshuffle,
bzip2, Blosc 1 (read and write), Blosc2 and ZFP (read only).
- **Shared pieces:** `float16` (the one IEEE half-precision conversion the
workspace uses), `provenance` (SHA-256 dataset hashes), `checksum`
(Jenkins lookup3 for v2+ structures).
## Example
```rust
use clawhdf5_format::file_writer::{AttrValue, FileWriter};
use clawhdf5_format::{group_v2, object_header, signature, superblock};
// Write a file to memory
let mut fw = FileWriter::new();
fw.create_dataset("data")
.with_f64_data(&[1.0, 2.0, 3.0])
.with_shape(&[3])
.set_attr("unit", AttrValue::String("m/s".into()));
let bytes = fw.finish().unwrap();
// Parse it back: superblock -> path -> object header
let (_user_block, file) = signature::split_user_block(&bytes).unwrap();
let sb = superblock::Superblock::parse(file, 0).unwrap();
let addr = group_v2::resolve_path_any(file, &sb, "data").unwrap();
let hdr = object_header::ObjectHeader::parse(file, addr as usize, sb.offset_size, sb.length_size)
.unwrap();
assert!(!hdr.messages.is_empty());
```
## Features
- Zero-copy superblock, object header, and B-tree parsing
- Chunked dataset read/write with filter pipelines
- `no_std` support (disable `std` feature)
- Optional parallel reads via Rayon
- SHA-256 provenance tracking
| Feature | Default | What | Builds C |
|---|---|---|---|
| `std` | yes | standard library; without it the crate is `no_std` + `alloc` (CI builds it for `thumbv7em-none-eabihf`) | no |
| `checksum` | yes | verify Jenkins lookup3 checksums | no |
| `deflate` | yes | deflate through flate2 | no |
| `zlib-rs` | yes | flate2's pure-Rust zlib-rs backend, with `runtime_detection` (without it zlib-rs loses SIMD and inflates 3.5x slower) | no |
| `system-zlib-decompress` | yes | no effect (nothing reads it; kept so existing feature lists still build) | no |
| `provenance` | yes | SHA-256 provenance hashes | no |
| `lzf` | yes | LZF (32000) | no |
| `parallel` | no | rayon-parallel chunk decoding | no |
| `fast-checksum` | no | hardware CRC32 through `crc32fast` | no |
| `lz4` | no | LZ4 (32004) | no |
| `pcodec` | no | pcodec | no |
| `bitshuffle`, `bzip2`, `blosc` | no | 32008, 307, 32001, read and write | no |
| `blosc2`, `zfp` | no | 32026, 32013, read only | no |
| `plugin-filters` | no | all six plugin filters above | no |
| `lookup-stats` | no | counters for name-lookup benchmarks | no |
| `zstd` | no | Zstandard (32015) | yes (libzstd) |
| `szip` | no | SZIP (4) decoding | links the system libaec (`libaec-dev`) |
| `fast-deflate` | no | zlib-ng | yes (cmake) |
| `system-zlib` | no | the system zlib | yes (`libz-sys`) |
| `blake3_hash` | no | `provenance::blake3_hash` | yes (`cc`) |
## Usage
## Robustness
```rust
use clawhdf5_format::Superblock;
let data = std::fs::read("data.h5").unwrap();
let sb = Superblock::from_bytes(&data).unwrap();
println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
```
Every parser is meant to return an error, never panic, on hostile input:
nine cargo-fuzz targets live in [`fuzz/`](fuzz/README.md), the conformance
sweep includes the HDF Group's CVE corpus
([`CONFORMANCE.md`](../../CONFORMANCE.md)), and header checks follow
libhdf5's. Open gaps are in [`docs/known-issues.md`](../../docs/known-issues.md).
## License
crates/clawhdf5-format/fuzz/README.md
+12 -4
View File
@@ -51,10 +51,18 @@ done
## CI
These targets are **not** run in CI (`.gitea/workflows/ci.yml`) — cargo-fuzz
requires nightly and each meaningful run takes minutes, which doesn't fit a
per-PR gate. Run them manually on a schedule (e.g. before a release, or after
touching parser code) instead.
These targets are **not** run by the CI workflows (`.gitea/workflows/ci.yml`)
— cargo-fuzz requires nightly and each meaningful run takes minutes, which
doesn't fit a per-PR gate. Run them by hand before a release or after
touching parser code. `scripts/ci-test.sh` has an opt-in smoke run: with
`CLAWHDF5_FUZZ_SECONDS=N` it runs every target of this crate and of
`crates/clawhdf5-agent/fuzz` (the WAL parser) for N seconds each.
Other robustness checks that do run: the nightly conformance sweep reads
the HDF Group's CVE reproducers and fails on any panic, hang, crash or
out-of-memory ([`conformance/README.md`](../../../conformance/README.md)),
and `scripts/h5rs-fuzz.sh` runs every `h5rs` subcommand over them, optionally
on byte-flipped copies.
## Reproducing Crashes
crates/clawhdf5-gpu/README.md
+47 -10
View File
@@ -1,25 +1,62 @@
# clawhdf5-gpu
[![crates.io](https://img.shields.io/crates/v/clawhdf5-gpu.svg)](https://crates.io/crates/clawhdf5-gpu)
[![docs.rs](https://docs.rs/clawhdf5-gpu/badge.svg)](https://docs.rs/clawhdf5-gpu)
GPU vector distance computation through [wgpu](https://wgpu.rs) and
hand-written WGSL compute shaders: upload a set of vectors once, then run
cosine or L2 top-k searches, dot products, distance matrices and norms
against them on Vulkan, Metal, DirectX 12 or OpenGL.
GPU-accelerated vector operations for clawhdf5 using wgpu compute shaders.
This crate does **not** read or write HDF5: dataset I/O in clawhdf5 is
CPU-only. It is a vector-search accelerator used optionally by
[`clawhdf5-agent`](../clawhdf5-agent/README.md) (its `gpu` feature exposes
`gpu_search::GpuSearchBackend` and a GPU arm of `strategy::search_with_metrics`;
`HDF5Memory::search` itself uses the HNSW index on the CPU).
## Features
Not on crates.io yet; depend on it from git:
- GPU-accelerated distance computations (L2, cosine)
- wgpu-based compute shaders for cross-platform GPU support
- Float16 support via `half` crate
```toml
[dependencies]
clawhdf5-gpu = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## Usage
```rust
```rust,no_run
use clawhdf5_gpu::GpuAccelerator;
let accel = GpuAccelerator::new().unwrap();
let distances = accel.l2_distances(&query, &vectors).unwrap();
// Fall back to a CPU path when there is no usable GPU.
let mut gpu = match GpuAccelerator::new() {
Ok(g) => g,
Err(_) => return,
};
let dim = 128;
let vectors = vec![0.5f32; 1000 * dim]; // 1000 vectors, row-major
gpu.upload_vectors(&vectors, dim).unwrap();
let norms = gpu.compute_norms_gpu(&vectors, dim).unwrap();
gpu.upload_norms(&norms).unwrap();
let query = vec![1.0f32; dim];
let top10 = gpu.cosine_search(&query, 10).unwrap(); // (index, similarity), best first
let near10 = gpu.l2_search(&query, 10).unwrap(); // (index, distance), nearest first
```
`GpuAccelerator` also has `is_available`, `device_info`,
`batch_cosine_search`, `batch_dot_product`, `distance_matrix`,
`compute_norms`, and `f16_to_f32_batch`/`f32_to_f16_batch`. Vector sets
larger than the device's largest storage buffer binding are split into
chunks and the results merged. A GPU→CPU readback waits at most 30 s and
then fails with `GpuError::BufferMap` instead of hanging.
## Features
| Feature | Default | What |
|---|---|---|
| `gpu-wgpu` | yes | the wgpu implementation. Without it `GpuAccelerator::new()` returns `GpuError::NotCompiled` and `is_available()` is `false`. |
No C is compiled, but wgpu talks to the system's graphics drivers at run
time; the crate is exempt from CI's "no C in the default build" check for
that reason.
## License
MIT
crates/clawhdf5-io/README.md
+45 -15
View File
@@ -1,24 +1,54 @@
# clawhdf5-io
[![crates.io](https://img.shields.io/crates/v/clawhdf5-io.svg)](https://crates.io/crates/clawhdf5-io)
[![docs.rs](https://docs.rs/clawhdf5-io/badge.svg)](https://docs.rs/clawhdf5-io)
I/O building blocks under [`clawhdf5`](../clawhdf5/README.md): the
`HDF5Read`/`HDF5ReadWrite` traits with in-memory, borrowed, file and
memory-mapped readers, plus several experimental modules (async reads, an
HSDS client, a VOL-style trait, sub-filing, prefetch, and an MPI connector).
The facade uses it for memory-mapped reads (`MmapReader`, and the private
copy-on-write mapping that applies a metadata cache image).
I/O abstraction layer for clawhdf5.
Remote files are **not** read through this crate: HTTP(S) and object
stores go through `clawhdf5_format::storage::Storage` and
[`clawhdf5-remote`](../clawhdf5-remote/README.md).
Not on crates.io yet; depend on it from git:
```toml
[dependencies]
clawhdf5-io = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["mmap"] }
```
## Main items
| Item | What |
|---|---|
| `HDF5Read`, `HDF5ReadWrite` | byte-level read/write traits; `MemoryReader`, `BorrowedReader`, `FileReader`, `FileWriter` implement them |
| `MmapReader`, `MmapReadWrite` (`mmap`) | memory-mapped files through `memmap2`; `HDF5Read::private_copy` gives a copy-on-write view |
| `prefetch::PrefetchReader`, `sweep::SweepDetector` | read-ahead (`madvise(MADV_WILLNEED)` on mappings) and chunk-sweep prediction |
| `ParallelConfig` | lane partitioning for parallel chunk decoding |
| `vol::VirtualObjectLayer`, `vol::NativeVol` | a backend-agnostic object-layer trait (modelled on libhdf5's VOL) |
| `async_read` (`async`) | tokio-based `AsyncHDF5Read` and `AsyncHDF5File` |
| `hsds::HsdsClient` (`hsds`) | a REST client for an HSDS server |
| `subfiling` | splitting one logical file across several physical files |
| `mpi_vol::MpiVol` (`mpi-io`) | an MPI connector: see below |
### MPI (`mpi-io`)
`MpiVol` is **not collective MPI-IO**. Reads are root-read + broadcast
(rank 0 reads the file with `std::fs::read`, parses the dataset and
broadcasts the bytes); writes gather every rank's shard to rank 0, which
writes the merged dataset. It does not call `MPI_File_read_at_all` or any
other MPI-IO routine. Collective I/O is on the [roadmap](../../ROADMAP.md).
`clawhdf5-bench`'s `mpi_io_bench` binary exercises it.
## Features
- Memory-mapped file access (`mmap` feature)
- Async I/O via Tokio (`async` feature)
- HSDS remote access (`hsds` feature)
- Prefetching and sweep optimizations
## Usage
```rust
use clawhdf5_io::MmapReader;
let reader = MmapReader::open("data.h5").unwrap();
```
| Feature | Default | What | Builds C |
|---|---|---|---|
| `mmap` | no (the `clawhdf5` facade turns it on) | `MmapReader`, `MmapReadWrite` | no |
| `async` | no | `async_read` (tokio) | no |
| `hsds` | no | `hsds` (reqwest, and `async`) | yes: reqwest's default TLS is native-tls (OpenSSL on Linux) |
| `mpi-io` | no | a real `MpiVol` (without it `MpiVol::new_world` returns an error) | yes: `mpi-sys` needs an MPI installation and libclang |
## License
crates/clawhdf5-migrate/README.md
+7 -5
View File
@@ -1,11 +1,8 @@
# clawhdf5-migrate
[![crates.io](https://img.shields.io/crates/v/clawhdf5-migrate.svg)](https://crates.io/crates/clawhdf5-migrate)
[![docs.rs](https://img.shields.io/docsrs/clawhdf5-migrate)](https://docs.rs/clawhdf5-migrate)
CLI tool to migrate a SQLite agent-memory database in the `memory_chunks` / `sessions` / `entities` / `relations` layout (table and
column names are configurable) to a
[clawhdf5-agent](https://crates.io/crates/clawhdf5-agent) store. This is **not**
[clawhdf5-agent](../clawhdf5-agent/README.md) store. This is **not**
ZeroClaw's schema — ZeroClaw keeps memories in a single `memories` table and
does not use clawhdf5.
@@ -15,10 +12,15 @@ the knowledge graph (entities and relations) are carried over.
## Installation
Not on crates.io yet; install from a checkout:
```bash
cargo install clawhdf5-migrate
cargo install --path crates/clawhdf5-migrate
```
It builds C: `rusqlite` is built with its `bundled` feature, which compiles
SQLite (so no system libsqlite is needed, but a C compiler is).
## Usage
```bash
crates/clawhdf5-napi/README.md
+38
View File
@@ -0,0 +1,38 @@
# clawhdf5-napi
> **Status: does not work end to end.** The TypeScript package built on
> this crate (`packages/clawhdf5-node`) has never run successfully, is
> unpublished, and is not built or tested in CI. See "The Node.js package
> does not work" in [`docs/known-issues.md`](../../docs/known-issues.md).
> Fix it and add CI, or remove it, before depending on it.
A Node.js native addon ([napi-rs](https://napi.rs), N-API 9) exposing
[`clawhdf5-agent`](../clawhdf5-agent/README.md) as a `ClawhdfMemory`
class. It wraps `clawhdf5_agent::openclaw::ClawhdfBackend` (the agent's
`search` with re-ranking and confidence on) and the consolidation engine.
It was written for an OpenClaw integration that is not being pursued
([`docs/openclaw.md`](../../docs/openclaw.md)).
## What the addon exposes
`ClawhdfMemory.create(path, dim)`, `.open(path)`, `.openOrCreate(path,
dim)`, and on an instance: `search`, `get`, `write`, `ingestMarkdown`,
`exportMarkdown`, `save`, `saveBatch`, `stats`, `compact`, `tickSession`,
`flushWal`, `walPendingCount`, `runConsolidation`, and the ephemeral tier
(`enableEphemeral`, `ephemeralSet`/`Get`/`Delete`, `ephemeralStats`,
`promoteEphemeral`). napi-rs converts names and `#[napi(object)]` fields to
camelCase.
## Build
```bash
cargo build --release -p clawhdf5-napi # the Rust cdylib
# the .node package: npm install -g @napi-rs/cli; cd packages/clawhdf5-node; napi build --platform --release
```
It links against Node's N-API through `napi-sys` (a `-sys` crate), so it is
exempt from CI's "no C in the default build" check.
## License
MIT
crates/clawhdf5-netcdf4/README.md
+39 -11
View File
@@ -1,25 +1,53 @@
# clawhdf5-netcdf4
[![crates.io](https://img.shields.io/crates/v/clawhdf5-netcdf4.svg)](https://crates.io/crates/clawhdf5-netcdf4)
[![docs.rs](https://docs.rs/clawhdf5-netcdf4/badge.svg)](https://docs.rs/clawhdf5-netcdf4)
Read NetCDF-4 files in pure Rust. NetCDF-4 files are HDF5 files with
conventions for dimensions, coordinate variables and attributes; this crate
reads them through the [`clawhdf5`](../clawhdf5/README.md) facade, with no
libnetcdf or libhdf5. Read-only: NetCDF-3 (classic) files are not HDF5 and
are not supported.
NetCDF-4 read support built on clawhdf5 — pure Rust, no C dependencies.
Not on crates.io yet; depend on it from git:
## Features
- Read NetCDF-4 / HDF5-backed `.nc` files
- Dimension, variable, and CF convention support
- Climate and scientific data access
```toml
[dependencies]
clawhdf5-netcdf4 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## Usage
```rust
```rust,no_run
use clawhdf5_netcdf4::NetCDF4File;
let nc = NetCDF4File::open("climate.nc").unwrap();
let temp = nc.variable("temperature").unwrap();
let nc = NetCDF4File::open("climate.nc")?;
for dim in nc.dimensions()? {
println!("{}: {} (unlimited = {})", dim.name, dim.size, dim.is_unlimited);
}
let mut temp = nc.variable("temperature")?;
let dims: Vec<&str> = temp.dimensions().iter().map(|d| d.name.as_str()).collect();
println!("{:?} over {:?}", temp.shape()?, dims);
let cf = temp.cf_attributes()?;
println!("units: {:?}", cf.units);
// scale_factor/add_offset applied; _FillValue and missing_value become NaN
let values: Vec<f64> = temp.read_f64()?;
# Ok::<(), clawhdf5_netcdf4::Error>(())
```
## API
| Item | What |
|---|---|
| `NetCDF4File` | `open`, `from_bytes`, `dimensions`, `variables`, `variable`, `global_attrs`, `group`, `group_names`, `nc_properties`, and `hdf5_file` for the underlying `clawhdf5::File` |
| `NetCDF4Group` | the same for a sub-group (`dimensions`, `variables`, `attrs`, nested `group`) |
| `Variable` | `name`, `shape`, `dimensions`, `nc_type`, `is_coordinate`, `attrs`, `cf_attributes`; `read_f64` (CF scale/offset and fill applied), `read_raw_f32`/`_f64`/`_i32`/`_i64`/`_u64`, `read_string`, `read_raw` |
| `Dimension` | `name`, `size`, `is_unlimited` |
| `CfAttributes` | CF convention attributes: `units`, `long_name`, `standard_name`, `fill_value` (`_FillValue`), `missing_value`, `scale_factor`, `add_offset`, `valid_range`, `calendar`, `axis` |
| `NcType` | the NetCDF type of a variable |
No cargo features. Tests compare against files written by netCDF4-python
(`tests/interop_tests.rs`; the CI job requires them with
`CLAWHDF5_REQUIRE_INTEROP=1`). What the HDF5 reader underneath cannot
read is listed in [`docs/known-issues.md`](../../docs/known-issues.md).
## License
MIT
crates/clawhdf5-py/README.md
+2 -3
View File
@@ -1,8 +1,5 @@
# clawhdf5-py
[![crates.io](https://img.shields.io/crates/v/clawhdf5-py.svg)](https://crates.io/crates/clawhdf5-py)
[![docs.rs](https://docs.rs/clawhdf5-py/badge.svg)](https://docs.rs/clawhdf5-py)
Python bindings for clawhdf5 — a pure-Rust HDF5 library. The package is
`clawhdf5` (`import clawhdf5`); it needs numpy and no libhdf5.
@@ -61,6 +58,8 @@ with clawhdf5.File("data.h5", "r") as f:
`clawhdf5.InternalError`, a `RuntimeError`.
- Attributes return what h5py returns; `clawhdf5.Empty` stands for a null
dataspace (h5py's `Empty`).
- Also as in h5py: `File.mode` (`'r'`, or `'r+'` for a writable file), `File.flush()` (a no-op:
edits are already synced), `Dataset.chunks`.
## Remote files
crates/clawhdf5-remote/README.md
+31
View File
@@ -114,3 +114,34 @@ let s = storage.stats(); // requests, bytes_fetched, hits, misses, cached_bytes,
directory with range support (the server the tests use), and
`cargo run -p clawhdf5-remote --example read_url -- URL [DATASET]` lists a
file and prints what it cost.
## Other front ends
- `h5rs` (built with `--features remote`, or `remote-https`) takes URLs as
FILE arguments: [`clawhdf5-tools`](../clawhdf5-tools/README.md).
- Python: `clawhdf5.File("http://…")` and `File.open_url(url, ...)` go
through this crate: [`clawhdf5-py`](../clawhdf5-py/README.md).
- The browser does **not** use this crate (its cache fetches by blocking);
`clawhdf5-wasm`'s `openUrl` has its own restartable cache:
[`examples/wasm-viewer`](../../examples/wasm-viewer/README.md).
## Limits
Files a SWMR writer is still appending to cannot be followed remotely
(the file is pinned at open, so growth is `RemoteError::FileChanged`); the
block size is fixed rather than taken from a paged file's page size; the
cloud backends are built and unit-tested but have not been run against a
real bucket. The full list is under "Remote files (`clawhdf5-remote`)
limits" in [`docs/known-issues.md`](../../docs/known-issues.md); the design
is milestone M3 of [`docs/design/range-reads.md`](../../docs/design/range-reads.md).
Not on crates.io yet; depend on it from git:
```toml
[dependencies]
clawhdf5-remote = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## License
MIT
crates/clawhdf5-tools/README.md
+12 -1
View File
@@ -306,4 +306,15 @@ CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 cargo test -p clawhd
The interop tests write their files with h5py and compare with h5ls, h5stat,
h5dump and h5diff; each skips when what it needs is missing unless
`CLAWHDF5_REQUIRE_INTEROP=1`.
`CLAWHDF5_REQUIRE_INTEROP=1`. `tests/remote.rs` runs every subcommand on
URLs against a local range server.
This crate also holds the interop tests of the library's in-place editor
(`clawhdf5::FileEditor`), since they use `h5rs check` and h5dump on every
edited file and compare index and heap structures with what libhdf5 makes
of the same edits:
```bash
CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 \
cargo test -p clawhdf5-tools --test edit_interop --test edit_coverage_interop
```
crates/clawhdf5-wasm/README.md
+51
View File
@@ -0,0 +1,51 @@
# clawhdf5-wasm
clawhdf5's HDF5 and NetCDF-4 reader compiled to WebAssembly with
wasm-bindgen, for the browser (and Node). Read-only. Two ways in:
- `open(bytes)` — a file already in memory (a dropped file, a fetched
blob);
- `openUrl(url, opts)` — a file on a web server, read by HTTP range
requests as each call needs its bytes, without downloading it
(range-read milestone M4, [`docs/design/range-reads.md`](../../docs/design/range-reads.md)).
Both give `list`, `info`, `attrs`, `read` and `readHyperslab`; the remote
file's methods return promises, and `stats()` counts requests and bytes.
The JavaScript API, options, limits, package size and tests are documented
with the demo page, [`examples/wasm-viewer/README.md`](../../examples/wasm-viewer/README.md).
## Layout
- `src/core.rs` — the reader over any `clawhdf5_format::storage::Storage`
(`Reader::open_storage`), plain Rust and tested natively.
- `src/lazy.rs` — the restartable "NeedBytes" cache behind `openUrl`: a
call runs as a pass over the blocks fetched so far; a pass that misses is
abandoned, the missing (and hinted) blocks are fetched, and the pass is
run again. No block is evicted while a call runs.
- `js/remote.js` — the HTTP side: `fetch` with `Range`, checking every
answer (a `206` with exactly the bytes asked for, same ETag/Last-Modified
and length) so a call fails rather than return another file's bytes.
- `src/lib.rs` — the wasm-bindgen exports.
## Build and test
```bash
rustup target add wasm32-unknown-unknown
cargo install wasm-bindgen-cli --version 0.2.129 # must equal the crate's wasm-bindgen
bash examples/wasm-viewer/build.sh # -> examples/wasm-viewer/pkg/
cargo test -p clawhdf5-wasm # native: h5py_interop, lazy, vl_strings
bash examples/wasm-viewer/test/run.sh # Node + headless Chromium (not in CI)
```
`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus cargo test -p
clawhdf5-wasm --test lazy` compares every corpus file read lazily with the
same file read from bytes.
Built without `mmap` and `parallel` and without the Zstd and SZIP filters
(they link C): such datasets fail with `unsupported filter`. No C is
compiled; `publish = false` (it is distributed as the package
`build.sh` makes).
## License
MIT
crates/clawhdf5/README.md
+89 -15
View File
@@ -1,27 +1,101 @@
# clawhdf5
[![crates.io](https://img.shields.io/crates/v/clawhdf5.svg)](https://crates.io/crates/clawhdf5)
[![docs.rs](https://docs.rs/clawhdf5/badge.svg)](https://docs.rs/clawhdf5)
The main crate: a pure-Rust HDF5 reader, writer and in-place editor, with no
libhdf5 and, by default, no C code. It wraps
[`clawhdf5-format`](../clawhdf5-format/README.md) (the binary format) and
[`clawhdf5-io`](../clawhdf5-io/README.md) (memory-mapped reads) in an
h5py-like API.
Pure-Rust HDF5 reader/writer — no C dependencies.
Not on crates.io yet; depend on it from git:
```toml
[dependencies]
clawhdf5 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```
## Main types
| Type | What it does |
|---|---|
| `File` | Opens a file (`open`, `open_buffered`, `from_bytes`, `open_storage` for any `Storage`), walks groups (`root`, `group`, `dataset`), lists `datasets`/`groups`/`attrs`. `File` is `Send + Sync`: several threads can read one open file. |
| `Dataset` | `shape`, `dtype`, `max_dimensions`, `attrs`; reads `read_f64`/`read_f32`/`read_i32`/`read_i64`/`read_u64`, strings (`read_string`, `read_string_bytes`), variable-length data (`read_vlen`), hyperslabs and point selections (`read_selection`, `read_f64_selection`, ...), zero-copy views of contiguous data (`read_f64_zerocopy`, ...), `verify_provenance`. |
| `FileBuilder` | Writes a new file: datasets of every numeric type, strings, compounds (`CompoundTypeBuilder`), enums, chunked and compressed layouts (deflate, shuffle, Fletcher-32, LZF, and with features LZ4, Zstd, bitshuffle, bzip2, Blosc, pcodec), nested groups, soft/hard/external links, virtual datasets, attribute creation order. Files open in h5py and h5dump. |
| `FileEditor` | Changes an existing file in place without rewriting it: `write_values`/`write_selection`/`write_all`, `resize` of chunked datasets (every chunk index), `set_attr` (compact and dense storage). Anything it cannot do safely is `Error::Unsupported` before any write. |
| `MmapFile`, `LazyFile` | Alternative readers: memory-mapped, and one that reads lazily and caches. |
| `File::open_swmr` | Reads a file a libhdf5 SWMR writer is still appending to (`Dataset::refresh`, bounded retries), as h5py's `swmr=True` reader does. |
## Examples
```rust,no_run
use clawhdf5::{AttrValue, File, FileBuilder, FileEditor, Selection};
// Write
let mut b = FileBuilder::new();
b.create_dataset("sensors/temperature")
.with_f64_data(&[20.5, 21.0, 21.5, 22.0])
.with_shape(&[4])
.with_maxshape(&[u64::MAX]) // unlimited, so it can grow
.with_chunks(&[2])
.with_deflate(4);
b.set_attr("version", AttrValue::I64(1));
b.write("data.h5")?;
// Read
let file = File::open("data.h5")?;
let ds = file.dataset("sensors/temperature")?;
assert_eq!(ds.shape()?, vec![4]);
let values = ds.read_f64()?;
// Edit in place: grow the dataset and fill the new tail
let mut ed = FileEditor::open("data.h5")?;
ed.resize("sensors/temperature", &[6])?;
let tail = Selection::Hyperslab {
start: vec![4],
stride: vec![1],
count: vec![2],
block: vec![1],
};
ed.write_values("sensors/temperature", &tail, &[22.5f64, 23.0])?;
# Ok::<(), clawhdf5::Error>(())
```
Remote files (HTTP range requests, S3/GCS/Azure) are read through
`File::open_storage`; [`clawhdf5-remote`](../clawhdf5-remote/README.md)
provides the storage and its block cache.
## Features
- Read and write HDF5 files entirely in Rust
- Memory-mapped I/O for large files (`mmap` feature, enabled by default)
- Parallel chunk reads via Rayon (`parallel` feature)
- Lazy dataset access for minimal memory usage
- h5py-compatible file output
| Feature | Default | What | Builds C |
|---|---|---|---|
| `mmap` | yes | memory-mapped reads (`File::open` maps the file; `MmapFile`) | no |
| `provenance` | yes | SHA-256 `_provenance_sha256` attributes (`DatasetBuilder::with_provenance`, `Dataset::verify_provenance`) | no |
| `lzf` | yes | LZF filter (32000), h5py's `compression="lzf"` | no |
| `parallel` | no | chunk decoding on a rayon pool | no |
| `lz4` | no | LZ4 filter (32004) | no |
| `pcodec` | no | pcodec filter | no |
| `bitshuffle`, `bzip2`, `blosc` | no | plugin filters 32008, 307, 32001 (read and write) | no (bzip2 uses the pure-Rust `libbz2-rs-sys`) |
| `blosc2`, `zfp` | no | plugin filters 32026 and 32013, **read only** | no |
| `plugin-filters` | no | `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2`, `zfp` | no |
| `zstd` | no | Zstandard filter (32015) | yes (libzstd) |
| `fast-deflate` | no | zlib-ng instead of the pure-Rust zlib-rs | yes (cmake) |
| `blake3_hash` | no | `provenance::blake3_hash` helpers | yes (`cc`, for blake3's SIMD code) |
| `apple-compression` | no | currently has no effect in this crate (it is not forwarded) | — |
## Usage
SZIP decoding is a `clawhdf5-format` feature (`szip`, links the system
libaec); the facade does not forward it.
```rust
use clawhdf5::File;
## Limits and further reading
let file = File::open("data.h5").unwrap();
let dataset = file.dataset("/group/data").unwrap();
let values: Vec<f64> = dataset.read_1d().unwrap();
```
- What is known not to work, and what was wrong in earlier releases:
[`docs/known-issues.md`](../../docs/known-issues.md) (editor limits, range
reads, external links and external raw data, which are explicit errors).
- Read coverage against libhdf5/h5py on eight public corpora:
[`CONFORMANCE.md`](../../CONFORMANCE.md).
- Read and write speed against libhdf5 and h5py:
[`BENCHMARKS.md`](../../BENCHMARKS.md).
- Range reads and SWMR design: [`docs/design/range-reads.md`](../../docs/design/range-reads.md),
[`docs/design/swmr.md`](../../docs/design/swmr.md).
- Changes: [`CHANGELOG.md`](../../CHANGELOG.md).
## License