diff --git a/crates/clawhdf5-accel/README.md b/crates/clawhdf5-accel/README.md index d7c0b2f..26bacfc 100644 --- a/crates/clawhdf5-accel/README.md +++ b/crates/clawhdf5-accel/README.md @@ -1,24 +1,61 @@ # clawhdf5-accel -[![crates.io](https://img.shields.io/crates/v/clawhdf5-accel.svg)](https://crates.io/crates/clawhdf5-accel) -[![docs.rs](https://docs.rs/clawhdf5-accel/badge.svg)](https://docs.rs/clawhdf5-accel) +CPU SIMD kernels for vector search: dot products, cosine similarity, L2 +distance, norms and int8 dot products, dispatched at run time to the best +backend the CPU has, with a portable scalar fallback for every operation. +[`clawhdf5-ann`](../clawhdf5-ann/README.md) and +[`clawhdf5-agent`](../clawhdf5-agent/README.md) use it in their distance +loops; it has nothing to do with HDF5 file I/O. -SIMD-accelerated operations for clawhdf5. +Not on crates.io yet; depend on it from git: + +```toml +[dependencies] +clawhdf5-accel = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +``` + +## API + +```rust +use clawhdf5_accel::{cosine_similarity, detect_backend, dot_i8, dot_product, l2_distance}; + +let a = [1.0f32, 2.0, 3.0, 4.0]; +let b = [4.0f32, 3.0, 2.0, 1.0]; +assert_eq!(dot_product(&a, &b), 20.0); +let _cos = cosine_similarity(&a, &b); +let _l2 = l2_distance(&a, &b); +assert_eq!(dot_i8(&[1, -2, 3], &[4, 5, -6]), -24); +println!("{:?}", detect_backend()); // e.g. Avx2 on x86-64, Neon on aarch64 +``` + +Also `vector_norm`, `batch_norms`, `batch_cosine`, `batch_cosine_prenorm`, +`f16_to_f32_batch`, `checksum_fletcher32` and `align_to_cache_line`. + +## Backends + +`detect_backend()` picks once per process: `Avx512` (with the `avx512` +feature), `Avx2` (AVX2 + FMA), `Neon` (every aarch64 CPU), or `Scalar`. +`Sse4` and `WasmSimd128` are reported when detected but run the scalar +kernels. +`dot_i8`, used by the agent's quantised (int8) HNSW index, runs on +AVX2 and on NEON — with the `SDOT` instruction (through inline assembly, +since the intrinsic is unstable) on cores that have dotprod, such as the +Raspberry Pi 5, and plain NEON on older ones. At equal recall the int8 +index answers 1.63x the queries per second of the f32 one on x86-64 +(AVX2) and 1.18x on a Raspberry Pi 5 ([`BENCHMARKS.md`](../../BENCHMARKS.md)). + +The aarch64 code is compiled out on x86, so only the `test-arm64` CI job +builds and tests it. ## Features -- AVX2 and NEON SIMD acceleration -- AVX-512 support (`avx512` feature) -- Float16 conversion (`float16` feature) -- CRC32 checksum acceleration +| Feature | Default | What | Builds C | +|---|---|---|---| +| `avx512` | no | AVX-512F kernels | no | +| `float16` | no | `f16_to_f32_batch` through the `half` crate (a software conversion otherwise) | no | -## Usage - -```rust -use clawhdf5_accel::checksum::crc32_simd; - -let crc = crc32_simd(&data); -``` +The half-precision conversion used for stored embeddings is +`clawhdf5_format::float16`, not this crate's. ## License diff --git a/crates/clawhdf5-agent/README.md b/crates/clawhdf5-agent/README.md index d0e5925..8eeafe7 100644 --- a/crates/clawhdf5-agent/README.md +++ b/crates/clawhdf5-agent/README.md @@ -1,28 +1,124 @@ # clawhdf5-agent -[![crates.io](https://img.shields.io/crates/v/clawhdf5-agent.svg)](https://crates.io/crates/clawhdf5-agent) -[![docs.rs](https://img.shields.io/docsrs/clawhdf5-agent)](https://docs.rs/clawhdf5-agent) +Persistent memory for AI agents in a single HDF5 file: text chunks with +embeddings and metadata, hybrid search (HNSW vector search + BM25 keyword +search, fused), sessions, a knowledge graph, a write-ahead log for crash +safety, and optionally Ed25519-signed checkpoints. Stores open in h5py like +any other HDF5 file. Built on [`clawhdf5`](../clawhdf5/README.md), +[`clawhdf5-ann`](../clawhdf5-ann/README.md) and +[`clawhdf5-accel`](../clawhdf5-accel/README.md). -HDF5-backed persistent memory store for on-device AI agents. +It is a library: no agent framework integrates it (OpenClaw and ZeroClaw +integration claims were withdrawn on 2026-09-25; see +[`docs/openclaw.md`](../../docs/openclaw.md)). The command-line front end +is [`clawhdf5-cli`](../clawhdf5-cli/README.md). -Built on [clawhdf5](https://crates.io/crates/clawhdf5), clawhdf5-agent provides a vector-searchable memory backend optimized for edge AI workloads. Store embeddings, text chunks, and metadata in a single HDF5 file with SIMD-accelerated similarity search. - -## Features - -- Persistent vector store in HDF5 format -- Cosine similarity and L2 distance search -- SIMD-accelerated via clawhdf5-accel (AVX2, NEON) -- Optional GPU acceleration via clawhdf5-gpu -- Memory-mapped access for large stores -- f16 storage support for compact embeddings - -## Usage +Not on crates.io yet; depend on it from git: ```toml [dependencies] -clawhdf5-agent = "2.1.0" +clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } ``` +## Usage + +```rust,no_run +use std::path::PathBuf; +use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions}; + +let config = MemoryConfig::new(PathBuf::from("agent.h5"), "my-agent", 384); +let mut mem = HDF5Memory::create(config)?; + +mem.save(MemoryEntry { + chunk: "The deploy key rotates every Monday.".into(), + embedding: vec![0.01; 384], // from your embedding model + source_channel: "chat".into(), + timestamp: 1_790_000_000.0, + session_id: "s1".into(), + tags: "ops".into(), +})?; + +let query = vec![0.01f32; 384]; +let hits = mem.search(&query, "deploy key", &SearchOptions::new(5).with_sources(["chat"])); +for h in &hits { + println!("{:.3} {}", h.score, h.chunk); +} +mem.flush_wal()?; // checkpoint now; otherwise one is made after 500 WAL entries (wal_max_entries) +# Ok::<(), clawhdf5_agent::MemoryError>(()) +``` + +## What is in it + +- **`HDF5Memory`** — `create`, `open` (single writer: an exclusive lock on + `.h5.lock`, a second opener gets `MemoryError::Locked`), + `open_read_only` (no lock, never writes). Through the `AgentMemory` + trait: `save`, `save_batch`, `delete`, `compact`, `count`, `snapshot`, + sessions; also `save_or_update`, `delete_batch`, `flush_wal`. +- **Search** — `search(query_embedding, text, &SearchOptions)`: optional + source-channel filter applied before ranking, vector + BM25 fusion + (weighted or RRF), Hebbian activation scaling, optional re-ranking + (`reranker::ReRankConfig`) and confidence rejection + (`confidence::ConfidenceConfig`). `hybrid_search` and + `hybrid_search_with` are thin wrappers. The vector stage uses the HNSW + index (`hnsw` feature); its graph is saved to `.h5.ann` at each + checkpoint and reloaded on open (rebuilt if stale or damaged). +- **Storage settings** (`MemoryConfig`, persisted with the store): + `float16` embeddings (on by default for new stores; 48% smaller file at + 100K records, same retrieval on LongMemEval), `quantized_index` (int8 + copy of the vectors in the index, on by default; re-scored against the + exact embeddings), `compression` (off by default), HNSW `m`/`ef` + parameters, WAL settings (`wal_enabled`, on by default; `wal_max_entries`, + 500: the WAL is checkpointed into the `.h5` once it holds more). +- **WAL** (`wal`) — every write is appended to `.h5.wal` with a + chained CRC32 per entry, so a corrupted, reordered or spliced entry stops + replay. Recovers from a process crash at any point, including between a + checkpoint and the WAL truncate. WAL appends are not fsynced: saves since + the last checkpoint can be lost on power failure. An unreadable WAL is + quarantined to `.h5.wal.corrupt-`. +- **Signed checkpoints** (`signing`) — `set_signing_key` signs a manifest + (SHA-256 Merkle tree over records, plus settings, sessions and graph) at + every checkpoint; `HDF5Memory::verify(path, &public_key)` checks it and + locates edits. WAL entries after the checkpoint are not covered. +- **Knowledge graph** (`knowledge`, `entity_extract`) — `add_entity`, + `add_entity_alias`, `add_relation`, `extract_and_store_entities`, + traversal and spreading activation. +- **Also:** sessions (`session`), temporal index (`temporal`), + consolidation tiers (`consolidation`), an in-memory TTL tier + (`ephemeral`), multi-modal embeddings (`multimodal`), `AGENTS.md` + generation (`agents_md`), query expansion, and a session-scoped + provenance ledger and write-anomaly detector on every save + (`take_anomaly_alerts`; alerts never block a save, and the source is + inferred from `source_channel`, not authenticated). +- `openclaw::ClawhdfBackend` is `search` with re-ranking and confidence + on, plus Markdown import/export. The module name is historical: it is not + an OpenClaw plugin. + +## Features + +| Feature | Default | What | Builds C | +|---|---|---|---| +| `hnsw` | yes | HNSW vector index (`clawhdf5-ann`); without it the vector stage is an exact linear cosine scan | no | +| `parallel` | yes | build the HNSW index on a rayon pool (same graph either way) | no | +| `float16` | yes | f16 helpers in `vector_search` (`half`). Stores' `MemoryConfig::float16` works without it. | no | +| `fast-math` | no | `matrixmultiply` batch distances in `strategy` | no | +| `accelerate` | no | Apple Accelerate BLAS in `strategy` (macOS) | links a system framework | +| `openblas` | no | OpenBLAS in `strategy` | yes (`openblas-src`) | +| `gpu` | no | `gpu_search` through [`clawhdf5-gpu`](../clawhdf5-gpu/README.md) (wgpu), used by `strategy`, not by `HDF5Memory::search` | no, but needs GPU drivers | +| `zstd` | no | Zstd instead of deflate when `MemoryConfig::compression` is on | yes (libzstd) | +| `async` | no | `async_memory` wrapper on tokio | no | + +`--no-default-features --features float16` forces the exact linear scan. + +## Measurements and limits + +- Search recall and latency, file size, LongMemEval and MemoryArena + retrieval numbers: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with + the `clawhdf5-bench` binaries (`search_harness`, `longmemeval_bench`, + `footprint_bench`, ...). +- Known issues and their history: [`docs/known-issues.md`](../../docs/known-issues.md). +- Migrating a SQLite memory database: + [`clawhdf5-migrate`](../clawhdf5-migrate/README.md). + ## License MIT diff --git a/crates/clawhdf5-android/README.md b/crates/clawhdf5-android/README.md new file mode 100644 index 0000000..838a088 --- /dev/null +++ b/crates/clawhdf5-android/README.md @@ -0,0 +1,44 @@ +# clawhdf5-android + +A C ABI over [`clawhdf5-agent`](../clawhdf5-agent/README.md) for Android +apps: a `cdylib` exporting `extern "C"` functions (`edgehdf5_*`, a name +kept from the project's earlier "edgehdf5" days) that manage an +`HDF5Memory` through an opaque handle. + +The functions are plain C symbols, not JNI-mangled `Java_...` entry points: +a Kotlin/Java app calls them through a thin JNI shim or JNA of its own. No +such shim, Gradle project or AAR is in this repository, and the crate is +not built for an Android target in CI (only its host-side unit tests run +with the workspace). + +## Functions + +| Function | What | +|---|---| +| `edgehdf5_create(path, agent_id, embedding_dim)` / `edgehdf5_open(path)` | a handle, or null on failure | +| `edgehdf5_close(handle)` | drop the store; what is not yet checkpointed stays in its WAL, as with any `HDF5Memory` | +| `edgehdf5_save(handle, ...)` | save one entry; the embedding length is checked against the store's dimension before the pointer is read | +| `edgehdf5_delete`, `edgehdf5_count`, `edgehdf5_count_active` | | +| `edgehdf5_hybrid_search(handle, query, len, text, vector_weight, keyword_weight, max_results, out_indices, out_scores, out_chunks)` | results into caller-provided arrays; returns the number written | +| `edgehdf5_add_session`, `edgehdf5_get_session_summary` | sessions | +| `edgehdf5_add_entity`, `edgehdf5_add_relation` | knowledge graph | +| `edgehdf5_free_string` | free a string this library returned | + +Every function is `unsafe`: the caller guarantees valid, NUL-terminated +strings and correctly sized buffers (see each function's `# Safety` +section), and serialises access to a handle; separate handles are +independent. + +## Build + +```bash +cargo build --release -p clawhdf5-android # host build; for a device, add --target aarch64-linux-android with the NDK's linker configured +``` + +It depends on `clawhdf5-agent` with **default features off**, so there is +no HNSW index (the vector stage is an exact linear scan) and no rayon +pool. No C is compiled. + +## License + +MIT diff --git a/crates/clawhdf5-ann/README.md b/crates/clawhdf5-ann/README.md index dd7a0af..233379f 100644 --- a/crates/clawhdf5-ann/README.md +++ b/crates/clawhdf5-ann/README.md @@ -1,25 +1,70 @@ # clawhdf5-ann -[![crates.io](https://img.shields.io/crates/v/clawhdf5-ann.svg)](https://crates.io/crates/clawhdf5-ann) -[![docs.rs](https://docs.rs/clawhdf5-ann/badge.svg)](https://docs.rs/clawhdf5-ann) +An HNSW (Hierarchical Navigable Small World) approximate nearest-neighbour +index in pure Rust, with cosine or L2 distance, optional int8 storage of +the vectors, deletions, and persistence as an HDF5 file. It is the vector +stage of [`clawhdf5-agent`](../clawhdf5-agent/README.md)'s search (the +agent's `hnsw` feature, on by default); distances run on +[`clawhdf5-accel`](../clawhdf5-accel/README.md)'s SIMD kernels. -HNSW approximate nearest neighbor index stored as HDF5. +Neighbours are chosen with the HNSW paper's diversity heuristic, not plain +closest-M (which capped recall on clustered data at 0.31 recall@10 at 100K +vectors). -## Features +Not on crates.io yet; depend on it from git: -- Build and query HNSW indexes persisted in HDF5 format -- Pure Rust, no C dependencies -- Efficient similarity search for high-dimensional vectors +```toml +[dependencies] +clawhdf5-ann = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +``` ## Usage ```rust -use clawhdf5_ann::HnswIndex; +use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage}; -let index = HnswIndex::from_hdf5("vectors.h5").unwrap(); -let neighbors = index.search(&query, 10); +let vectors: Vec> = (0..500) + .map(|i| (0..16).map(|j| ((i * 31 + j * 7) % 97) as f32 / 97.0).collect()) + .collect(); + +// m = 16 connections per node, ef_construction = 200 +let mut index = HnswIndex::build_with(&vectors, 16, 200, DistanceMetric::Cosine, Storage::Int8); +let hits = index.search(&vectors[42], 10, 64); // (id, distance), closest first; ef >= k +assert!(hits[0].1 < 1e-3); // vector 42 itself (or an identical one) + +let id = index.insert(vec![0.5; 16]); +index.mark_deleted(id); + +// Persist as HDF5 (a self-contained file: graph and vectors) and load it back +let bytes = index.to_hdf5_bytes().unwrap(); +let loaded = HnswIndex::load_from_hdf5(&bytes).unwrap(); +assert_eq!(loaded.len(), index.len()); ``` +- `HnswIndex::build` (L2), `build_with_metric`, `build_with` (metric and + storage); `new`/`new_with` plus `insert` for an index built + incrementally. +- `Storage::Int8` keeps each vector as `i8`, a quarter of the memory; it + applies to `Cosine` only (an L2 index keeps `Float32`). Distances are then + approximate, so a caller that needs exact ranking re-scores the + candidates, as the agent does. +- `mark_deleted`, `is_deleted`, `deleted_count`, `active_len`, `compact` + (returns the old-to-new id map). +- `save_to_hdf5(&mut writer)` / `to_hdf5_bytes` / `load_from_hdf5` store + the whole index; `graph_to_bytes` / `from_graph_bytes` store only the + graph (with a CRC32) for a caller that keeps the vectors elsewhere — the + agent's `.h5.ann` sidecar. + +## Features + +| Feature | Default | What | Builds C | +|---|---|---|---| +| `parallel` | no | build the graph on a rayon pool; the graph is identical with or without it | no | + +Recall and speed against exact search, for the index alone and in the +agent: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with +`cargo run --release -p clawhdf5-bench --bin search_harness`. + ## License MIT diff --git a/crates/clawhdf5-bench/README.md b/crates/clawhdf5-bench/README.md new file mode 100644 index 0000000..afa214f --- /dev/null +++ b/crates/clawhdf5-bench/README.md @@ -0,0 +1,49 @@ +# clawhdf5-bench + +The measurement harnesses behind [`BENCHMARKS.md`](../../BENCHMARKS.md): +HDF5 read and write speed (against libhdf5 and h5py where noted) and the +agent store's search, footprint and retrieval quality. Not meant for +publishing; nothing else in the workspace depends on it. Run everything with +`--release`, and quote numbers with the machine, date and command, as +`BENCHMARKS.md` does. + +## Binaries + +| Binary | Measures | +|---|---| +| `read_harness` | full reads vs hyperslab selections of a chunked 2-D dataset (compressed and not) and a contiguous one: does a selection cost scale with the selection or the dataset? (`-- --large` for 512 MB) | +| `concurrent_read` | decoded read throughput vs threads on one open `File`; `scripts/concurrent_read_h5py.py` runs the same workload with h5py (threads and processes) and `scripts/compare_concurrent_read.py` tabulates both | +| `search_harness` | HNSW recall@10 vs exact search, QPS and latency per `ef`, and end-to-end `HDF5Memory` ingest/checkpoint/open/search at 1K–100K (`--full`); studies: `--float16-study`, `--options-study`, `--signing-study`, `--ann-only --uniform` | +| `longmemeval_bench` | LongMemEval retrieval recall (turn and session Hit@k, MRR) — **retrieval, not QA accuracy**. Oracle or full `longmemeval_s` haystack; `--features embeddings` (or `embeddings-cuda`) embeds with MiniLM, otherwise the vector stage is inert and the run is BM25-only | +| `memory_arena` | a deterministic multi-session retrieval benchmark (BM25-only) | +| `footprint_bench` | file size and bytes per record at 100–100K records, float16 or `--f32`, WAL on/off, compressed or not | +| `consolidation_efficiency` | retrieval before and after consolidation on signal + noise records | +| `ephemeral_perf` | the in-memory ephemeral tier's set/get latency | +| `mpi_io_bench` | `clawhdf5-io`'s `MpiVol` (root-read + broadcast, not collective I/O); needs `--features mpi-io` and `mpirun` | + +```bash +cargo run --release -p clawhdf5-bench --bin search_harness -- --full +cargo run --release -p clawhdf5-bench --bin read_harness +``` + +## Criterion benches and example + +- `cargo bench -p clawhdf5-bench` runs `h5bench_write`, `h5bench_read` and + `h5bench_meta` (h5bench-style sequential, chunked, strided and metadata + workloads). `--features libhdf5-compare` adds the same workloads through + libhdf5 (the `hdf5-metno` crate; needs a system libhdf5 1.14). +- `examples/worldmodel_sampling.rs`: shuffled per-frame reads of a + `(N, H, W, C)` `uint8` dataset, clawhdf5 against h5py on the same file. + +## Features + +| Feature | What | Builds C | +|---|---|---| +| `libhdf5-compare` | libhdf5 variants of the Criterion benches | links the system libhdf5 | +| `mpi-io` | `mpi_io_bench` | yes (`mpi-sys`; needs an MPI installation) | +| `embeddings` | MiniLM embeddings for `longmemeval_bench` (candle) | yes (a `cc` build dependency in the candle/tokenizers tree) | +| `embeddings-cuda` | the same on a CUDA GPU (minutes instead of hours on the full haystack) | yes (CUDA) | + +## License + +MIT diff --git a/crates/clawhdf5-cli/README.md b/crates/clawhdf5-cli/README.md new file mode 100644 index 0000000..4495e39 --- /dev/null +++ b/crates/clawhdf5-cli/README.md @@ -0,0 +1,49 @@ +# clawhdf5-cli + +The `clawhdf5` command: create, fill, search and inspect a +[`clawhdf5-agent`](../clawhdf5-agent/README.md) memory store from the +shell. Output is JSON. (For general HDF5 files use `h5rs` from +[`clawhdf5-tools`](../clawhdf5-tools/README.md).) + +```bash +cargo install --path crates/clawhdf5-cli # installs `clawhdf5`; not on crates.io yet +# or: cargo run -p clawhdf5-cli -- --help +``` + +No C is compiled. + +## Commands + +The store is `--path FILE` (or `CLAWHDF5_PATH`) before the subcommand. + +| Command | What | +|---|---| +| `create [--agent-id ID] [--dim N] [--wal] [--f32] [--f32-index]` | a new store (dimension 384 by default); float16 embeddings and an int8 index copy unless `--f32` / `--f32-index`. The WAL is off unless `--wal` (the library's default is on), so each save is checkpointed at once | +| `save [--json '{...}']` | save one entry, from `--json` or stdin: `{"chunk", "embedding", "source_channel", "timestamp", "session_id", "tags"}` | +| `search --embedding '[...]' [--query TEXT] [-k N] [--vector-weight W] [--keyword-weight W]` | hybrid search (defaults 5 results, weights 0.7 / 0.3) | +| `recall INDEX` | one entry by index | +| `stats` | counts and configuration | +| `flush-wal` | checkpoint the WAL into the `.h5` | +| `agents-md [--output FILE]` | generate an `AGENTS.md` from the store | +| `export` | every entry as JSON lines | +| `snapshot DEST` | a copy of the store's `.h5` file | +| `keygen --out FILE` | a new Ed25519 signing key (64 hex characters, created owner-only on Unix) | +| `verify --public-key HEX_OR_FILE` | check a signed store; exit status 2 if it does not verify | + +`recall`, `stats`, `agents-md` and `export` open the store read-only +(no lock, nothing written), so they work while another process has it +open. `save`, `search` (which records activation boosts) and `flush-wal` +open it for writing and take the store's lock. With +`--signing-key FILE` (or `CLAWHDF5_SIGNING_KEY`) every checkpoint a command +makes is signed; a signed store refuses to checkpoint without the key. + +```bash +clawhdf5 --path mem.h5 create --agent-id demo --dim 3 +echo '{"chunk":"hello","embedding":[0.1,0.2,0.3],"source_channel":"cli","timestamp":0,"session_id":"s1","tags":""}' \ + | clawhdf5 --path mem.h5 save +clawhdf5 --path mem.h5 search --embedding '[0.1,0.2,0.3]' --query hello -k 3 +``` + +## License + +MIT diff --git a/crates/clawhdf5-derive/README.md b/crates/clawhdf5-derive/README.md index 8c0160c..87ed241 100644 --- a/crates/clawhdf5-derive/README.md +++ b/crates/clawhdf5-derive/README.md @@ -1,28 +1,50 @@ # clawhdf5-derive -[![crates.io](https://img.shields.io/crates/v/clawhdf5-derive.svg)](https://crates.io/crates/clawhdf5-derive) -[![docs.rs](https://docs.rs/clawhdf5-derive/badge.svg)](https://docs.rs/clawhdf5-derive) +`#[derive(H5Type)]`: maps a Rust struct with named fields to an HDF5 +compound datatype. The derive generates three inherent methods: -Derive macros for clawhdf5 HDF5 traits. +- `hdf5_datatype() -> clawhdf5_format::datatype::Datatype` — the + `Datatype::Compound` (members in field order, packed, little-endian); +- `to_bytes(&self) -> Vec` — one element in that layout; +- `from_bytes(&[u8]) -> Self` — the reverse (panics if the slice is shorter + than the compound). -## Features +Supported field types: `f32`, `f64`, `i8`–`i64`, `u8`–`u64`, `bool` +(stored as `u8`) and fixed-size arrays `[T; N]` of those numeric types. +Tuple structs, enums and nested structs are refused at compile time. -- `#[derive(HDF5Type)]` for automatic HDF5 datatype mapping -- Struct-to-compound-type derivation +The generated code names `clawhdf5_format`, so the crate using the derive +must depend on [`clawhdf5-format`](../clawhdf5-format/README.md) too. Not +on crates.io yet: -## Usage +```toml +[dependencies] +clawhdf5-derive = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +clawhdf5-format = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +``` + +## Example ```rust -use clawhdf5_derive::HDF5Type; +use clawhdf5_derive::H5Type; +use clawhdf5_format::datatype::Datatype; -#[derive(HDF5Type)] +#[derive(H5Type, Debug, PartialEq)] struct Point { - x: f64, - y: f64, - z: f64, + id: u32, + pos: [f64; 3], + valid: bool, } + +let p = Point { id: 7, pos: [1.0, 2.0, 3.0], valid: true }; +let bytes = p.to_bytes(); +assert_eq!(bytes.len(), 4 + 24 + 1); +assert_eq!(Point::from_bytes(&bytes), p); +assert!(matches!(Point::hdf5_datatype(), Datatype::Compound { size: 29, .. })); ``` +Tests: `crates/clawhdf5-format/tests/derive_tests.rs`. + ## License MIT diff --git a/crates/clawhdf5-filters/README.md b/crates/clawhdf5-filters/README.md index 45fcf30..78421e5 100644 --- a/crates/clawhdf5-filters/README.md +++ b/crates/clawhdf5-filters/README.md @@ -1,27 +1,56 @@ # clawhdf5-filters -[![crates.io](https://img.shields.io/crates/v/clawhdf5-filters.svg)](https://crates.io/crates/clawhdf5-filters) -[![docs.rs](https://docs.rs/clawhdf5-filters/badge.svg)](https://docs.rs/clawhdf5-filters) +Standalone deflate (zlib) compression and decompression with a choice of +backend: pure-Rust zlib-rs (default), zlib-ng, Apple's Compression +framework, or miniz_oxide. -Filter and compression pipeline for clawhdf5. +This crate holds **deflate backends only**. The HDF5 filter pipeline, the +filter registry and every other codec (shuffle, Fletcher-32, N-Bit, +scale-offset, LZ4, Zstd, SZIP, pcodec, LZF, bitshuffle, bzip2, Blosc, +Blosc2, ZFP) live in [`clawhdf5-format`](../clawhdf5-format/README.md), +which calls flate2 itself and selects its deflate backend with its own +features. No library crate of the workspace depends on this one (the +`clawhdf5` facade uses it only in tests). -## Features +Not on crates.io yet; depend on it from git: -- DEFLATE compression/decompression -- Pure-Rust deflate via zlib-rs (default, `zlib-rs` feature) -- zlib-ng instead, if you want it (`fast-deflate` feature; C, needs cmake) -- Apple Compression framework support (`apple-compression` feature) +```toml +[dependencies] +clawhdf5-filters = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +``` -## Usage +## API ```rust -use clawhdf5_filters::{deflate_compress, deflate_decompress}; +use clawhdf5_filters::{deflate_backend, deflate_compress, deflate_decompress}; +let data: Vec = (0..10_000u32).map(|i| (i % 251) as u8).collect(); let compressed = deflate_compress(&data, 6).unwrap(); // The second argument bounds the output: the expected decompressed size. let decompressed = deflate_decompress(&compressed, data.len()).unwrap(); +assert_eq!(decompressed, data); +println!("backend: {}", deflate_backend()); // "zlib-rs" by default ``` +Also `deflate_compress_miniz`/`deflate_decompress_miniz` (always +miniz_oxide) and `fast_deflate::{compress, decompress, active_backend}`. + +## Features + +Backend priority: `apple-compression` (macOS only) > zlib-ng > zlib-rs > +miniz_oxide (with none enabled). + +| Feature | Default | Backend | Builds C | +|---|---|---|---| +| `zlib-rs` | yes | zlib-rs through flate2, with `runtime_detection` (needed for its SIMD) | no | +| `fast-deflate` | no | zlib-ng through flate2 | yes (cmake) | +| `system-zlib` | no | the system zlib through flate2 | yes (`libz-sys`) | +| `apple-compression` | no | Apple Compression framework, macOS only (ignored elsewhere) | no (links a system framework) | + +zlib-rs matches zlib-ng on HDF5 reads and writes and produces +byte-identical output: see "Deflate backend" in +[`BENCHMARKS.md`](../../BENCHMARKS.md). + ## License MIT diff --git a/crates/clawhdf5-format/README.md b/crates/clawhdf5-format/README.md index eb3a056..3910da6 100644 --- a/crates/clawhdf5-format/README.md +++ b/crates/clawhdf5-format/README.md @@ -1,27 +1,106 @@ # clawhdf5-format -[![crates.io](https://img.shields.io/crates/v/clawhdf5-format.svg)](https://crates.io/crates/clawhdf5-format) -[![docs.rs](https://docs.rs/clawhdf5-format/badge.svg)](https://docs.rs/clawhdf5-format) +The HDF5 file format in pure Rust: parsers and writers for every on-disk +structure, the filter pipeline and its codecs, and the shared type +definitions the other crates use. Most users want the +[`clawhdf5`](../clawhdf5/README.md) facade, which wraps this crate in an +h5py-like API; use this one directly for low-level access or in `no_std` +code. -Pure-Rust HDF5 binary format parsing and writing — no C dependencies. +Not on crates.io yet; depend on it from git: + +```toml +[dependencies] +clawhdf5-format = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +``` + +## What is in it + +- **Parsing:** superblock v0–v3 (`superblock`, with the superblock + extension and metadata cache images, `superblock_ext`), object headers v1 + and v2 (`object_header`), every header message the readers use + (`datatype`, `dataspace`, `data_layout` v1–v4 including virtual datasets, + `fill_value`, `attribute`, `link_message`, `shared_message`, ...), groups + old and new (`group_v1` symbol tables with local heaps, `group_v2` with + fractal heaps and v2 B-trees), and every chunk index (v1 B-tree, single + chunk, implicit, fixed array, extensible array, v2 B-tree). +- **Reading data:** `data_read` (contiguous, compact, chunked), + `partial_read` and `selection` (hyperslabs and points), `vl_data` + (variable-length strings and sequences through the global heap), + `chunk_cache`. +- **Storage:** the `storage::Storage` trait (`read_at`, `read_ranges`, + `len`, `hint`) that every read path goes through, so a file can be read + from memory, a file handle or a remote backend + ([`clawhdf5-remote`](../clawhdf5-remote/README.md)). +- **Writing:** `file_writer::FileWriter` and the builders in + `type_builders` (datasets, groups, attributes, compound and enum types, + links, virtual datasets, creation-order tracking); chunk indexes and + dense-storage B-trees of any size (`chunked_write`, `btree_v2_write`, + `ea_writer`). Output is read by h5py and h5dump. +- **Filters:** `filter_pipeline` and `filter_registry` (look up by ID; other + IDs can be registered at run time with `register_filter`). Built in: + deflate, shuffle, Fletcher-32, N-Bit, scale-offset; behind features LZ4, + Zstd, SZIP (decode), pcodec, and the plugin filters LZF, bitshuffle, + bzip2, Blosc 1 (read and write), Blosc2 and ZFP (read only). +- **Shared pieces:** `float16` (the one IEEE half-precision conversion the + workspace uses), `provenance` (SHA-256 dataset hashes), `checksum` + (Jenkins lookup3 for v2+ structures). + +## Example + +```rust +use clawhdf5_format::file_writer::{AttrValue, FileWriter}; +use clawhdf5_format::{group_v2, object_header, signature, superblock}; + +// Write a file to memory +let mut fw = FileWriter::new(); +fw.create_dataset("data") + .with_f64_data(&[1.0, 2.0, 3.0]) + .with_shape(&[3]) + .set_attr("unit", AttrValue::String("m/s".into())); +let bytes = fw.finish().unwrap(); + +// Parse it back: superblock -> path -> object header +let (_user_block, file) = signature::split_user_block(&bytes).unwrap(); +let sb = superblock::Superblock::parse(file, 0).unwrap(); +let addr = group_v2::resolve_path_any(file, &sb, "data").unwrap(); +let hdr = object_header::ObjectHeader::parse(file, addr as usize, sb.offset_size, sb.length_size) + .unwrap(); +assert!(!hdr.messages.is_empty()); +``` ## Features -- Zero-copy superblock, object header, and B-tree parsing -- Chunked dataset read/write with filter pipelines -- `no_std` support (disable `std` feature) -- Optional parallel reads via Rayon -- SHA-256 provenance tracking +| Feature | Default | What | Builds C | +|---|---|---|---| +| `std` | yes | standard library; without it the crate is `no_std` + `alloc` (CI builds it for `thumbv7em-none-eabihf`) | no | +| `checksum` | yes | verify Jenkins lookup3 checksums | no | +| `deflate` | yes | deflate through flate2 | no | +| `zlib-rs` | yes | flate2's pure-Rust zlib-rs backend, with `runtime_detection` (without it zlib-rs loses SIMD and inflates 3.5x slower) | no | +| `system-zlib-decompress` | yes | no effect (nothing reads it; kept so existing feature lists still build) | no | +| `provenance` | yes | SHA-256 provenance hashes | no | +| `lzf` | yes | LZF (32000) | no | +| `parallel` | no | rayon-parallel chunk decoding | no | +| `fast-checksum` | no | hardware CRC32 through `crc32fast` | no | +| `lz4` | no | LZ4 (32004) | no | +| `pcodec` | no | pcodec | no | +| `bitshuffle`, `bzip2`, `blosc` | no | 32008, 307, 32001, read and write | no | +| `blosc2`, `zfp` | no | 32026, 32013, read only | no | +| `plugin-filters` | no | all six plugin filters above | no | +| `lookup-stats` | no | counters for name-lookup benchmarks | no | +| `zstd` | no | Zstandard (32015) | yes (libzstd) | +| `szip` | no | SZIP (4) decoding | links the system libaec (`libaec-dev`) | +| `fast-deflate` | no | zlib-ng | yes (cmake) | +| `system-zlib` | no | the system zlib | yes (`libz-sys`) | +| `blake3_hash` | no | `provenance::blake3_hash` | yes (`cc`) | -## Usage +## Robustness -```rust -use clawhdf5_format::Superblock; - -let data = std::fs::read("data.h5").unwrap(); -let sb = Superblock::from_bytes(&data).unwrap(); -println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor()); -``` +Every parser is meant to return an error, never panic, on hostile input: +nine cargo-fuzz targets live in [`fuzz/`](fuzz/README.md), the conformance +sweep includes the HDF Group's CVE corpus +([`CONFORMANCE.md`](../../CONFORMANCE.md)), and header checks follow +libhdf5's. Open gaps are in [`docs/known-issues.md`](../../docs/known-issues.md). ## License diff --git a/crates/clawhdf5-format/fuzz/README.md b/crates/clawhdf5-format/fuzz/README.md index 1343bf6..c80e82d 100644 --- a/crates/clawhdf5-format/fuzz/README.md +++ b/crates/clawhdf5-format/fuzz/README.md @@ -51,10 +51,18 @@ done ## CI -These targets are **not** run in CI (`.gitea/workflows/ci.yml`) — cargo-fuzz -requires nightly and each meaningful run takes minutes, which doesn't fit a -per-PR gate. Run them manually on a schedule (e.g. before a release, or after -touching parser code) instead. +These targets are **not** run by the CI workflows (`.gitea/workflows/ci.yml`) +— cargo-fuzz requires nightly and each meaningful run takes minutes, which +doesn't fit a per-PR gate. Run them by hand before a release or after +touching parser code. `scripts/ci-test.sh` has an opt-in smoke run: with +`CLAWHDF5_FUZZ_SECONDS=N` it runs every target of this crate and of +`crates/clawhdf5-agent/fuzz` (the WAL parser) for N seconds each. + +Other robustness checks that do run: the nightly conformance sweep reads +the HDF Group's CVE reproducers and fails on any panic, hang, crash or +out-of-memory ([`conformance/README.md`](../../../conformance/README.md)), +and `scripts/h5rs-fuzz.sh` runs every `h5rs` subcommand over them, optionally +on byte-flipped copies. ## Reproducing Crashes diff --git a/crates/clawhdf5-gpu/README.md b/crates/clawhdf5-gpu/README.md index e758e1a..cc6710b 100644 --- a/crates/clawhdf5-gpu/README.md +++ b/crates/clawhdf5-gpu/README.md @@ -1,25 +1,62 @@ # clawhdf5-gpu -[![crates.io](https://img.shields.io/crates/v/clawhdf5-gpu.svg)](https://crates.io/crates/clawhdf5-gpu) -[![docs.rs](https://docs.rs/clawhdf5-gpu/badge.svg)](https://docs.rs/clawhdf5-gpu) +GPU vector distance computation through [wgpu](https://wgpu.rs) and +hand-written WGSL compute shaders: upload a set of vectors once, then run +cosine or L2 top-k searches, dot products, distance matrices and norms +against them on Vulkan, Metal, DirectX 12 or OpenGL. -GPU-accelerated vector operations for clawhdf5 using wgpu compute shaders. +This crate does **not** read or write HDF5: dataset I/O in clawhdf5 is +CPU-only. It is a vector-search accelerator used optionally by +[`clawhdf5-agent`](../clawhdf5-agent/README.md) (its `gpu` feature exposes +`gpu_search::GpuSearchBackend` and a GPU arm of `strategy::search_with_metrics`; +`HDF5Memory::search` itself uses the HNSW index on the CPU). -## Features +Not on crates.io yet; depend on it from git: -- GPU-accelerated distance computations (L2, cosine) -- wgpu-based compute shaders for cross-platform GPU support -- Float16 support via `half` crate +```toml +[dependencies] +clawhdf5-gpu = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +``` ## Usage -```rust +```rust,no_run use clawhdf5_gpu::GpuAccelerator; -let accel = GpuAccelerator::new().unwrap(); -let distances = accel.l2_distances(&query, &vectors).unwrap(); +// Fall back to a CPU path when there is no usable GPU. +let mut gpu = match GpuAccelerator::new() { + Ok(g) => g, + Err(_) => return, +}; + +let dim = 128; +let vectors = vec![0.5f32; 1000 * dim]; // 1000 vectors, row-major +gpu.upload_vectors(&vectors, dim).unwrap(); +let norms = gpu.compute_norms_gpu(&vectors, dim).unwrap(); +gpu.upload_norms(&norms).unwrap(); + +let query = vec![1.0f32; dim]; +let top10 = gpu.cosine_search(&query, 10).unwrap(); // (index, similarity), best first +let near10 = gpu.l2_search(&query, 10).unwrap(); // (index, distance), nearest first ``` +`GpuAccelerator` also has `is_available`, `device_info`, +`batch_cosine_search`, `batch_dot_product`, `distance_matrix`, +`compute_norms`, and `f16_to_f32_batch`/`f32_to_f16_batch`. Vector sets +larger than the device's largest storage buffer binding are split into +chunks and the results merged. A GPU→CPU readback waits at most 30 s and +then fails with `GpuError::BufferMap` instead of hanging. + +## Features + +| Feature | Default | What | +|---|---|---| +| `gpu-wgpu` | yes | the wgpu implementation. Without it `GpuAccelerator::new()` returns `GpuError::NotCompiled` and `is_available()` is `false`. | + +No C is compiled, but wgpu talks to the system's graphics drivers at run +time; the crate is exempt from CI's "no C in the default build" check for +that reason. + ## License MIT diff --git a/crates/clawhdf5-io/README.md b/crates/clawhdf5-io/README.md index fb34684..a372453 100644 --- a/crates/clawhdf5-io/README.md +++ b/crates/clawhdf5-io/README.md @@ -1,24 +1,54 @@ # clawhdf5-io -[![crates.io](https://img.shields.io/crates/v/clawhdf5-io.svg)](https://crates.io/crates/clawhdf5-io) -[![docs.rs](https://docs.rs/clawhdf5-io/badge.svg)](https://docs.rs/clawhdf5-io) +I/O building blocks under [`clawhdf5`](../clawhdf5/README.md): the +`HDF5Read`/`HDF5ReadWrite` traits with in-memory, borrowed, file and +memory-mapped readers, plus several experimental modules (async reads, an +HSDS client, a VOL-style trait, sub-filing, prefetch, and an MPI connector). +The facade uses it for memory-mapped reads (`MmapReader`, and the private +copy-on-write mapping that applies a metadata cache image). -I/O abstraction layer for clawhdf5. +Remote files are **not** read through this crate: HTTP(S) and object +stores go through `clawhdf5_format::storage::Storage` and +[`clawhdf5-remote`](../clawhdf5-remote/README.md). + +Not on crates.io yet; depend on it from git: + +```toml +[dependencies] +clawhdf5-io = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["mmap"] } +``` + +## Main items + +| Item | What | +|---|---| +| `HDF5Read`, `HDF5ReadWrite` | byte-level read/write traits; `MemoryReader`, `BorrowedReader`, `FileReader`, `FileWriter` implement them | +| `MmapReader`, `MmapReadWrite` (`mmap`) | memory-mapped files through `memmap2`; `HDF5Read::private_copy` gives a copy-on-write view | +| `prefetch::PrefetchReader`, `sweep::SweepDetector` | read-ahead (`madvise(MADV_WILLNEED)` on mappings) and chunk-sweep prediction | +| `ParallelConfig` | lane partitioning for parallel chunk decoding | +| `vol::VirtualObjectLayer`, `vol::NativeVol` | a backend-agnostic object-layer trait (modelled on libhdf5's VOL) | +| `async_read` (`async`) | tokio-based `AsyncHDF5Read` and `AsyncHDF5File` | +| `hsds::HsdsClient` (`hsds`) | a REST client for an HSDS server | +| `subfiling` | splitting one logical file across several physical files | +| `mpi_vol::MpiVol` (`mpi-io`) | an MPI connector: see below | + +### MPI (`mpi-io`) + +`MpiVol` is **not collective MPI-IO**. Reads are root-read + broadcast +(rank 0 reads the file with `std::fs::read`, parses the dataset and +broadcasts the bytes); writes gather every rank's shard to rank 0, which +writes the merged dataset. It does not call `MPI_File_read_at_all` or any +other MPI-IO routine. Collective I/O is on the [roadmap](../../ROADMAP.md). +`clawhdf5-bench`'s `mpi_io_bench` binary exercises it. ## Features -- Memory-mapped file access (`mmap` feature) -- Async I/O via Tokio (`async` feature) -- HSDS remote access (`hsds` feature) -- Prefetching and sweep optimizations - -## Usage - -```rust -use clawhdf5_io::MmapReader; - -let reader = MmapReader::open("data.h5").unwrap(); -``` +| Feature | Default | What | Builds C | +|---|---|---|---| +| `mmap` | no (the `clawhdf5` facade turns it on) | `MmapReader`, `MmapReadWrite` | no | +| `async` | no | `async_read` (tokio) | no | +| `hsds` | no | `hsds` (reqwest, and `async`) | yes: reqwest's default TLS is native-tls (OpenSSL on Linux) | +| `mpi-io` | no | a real `MpiVol` (without it `MpiVol::new_world` returns an error) | yes: `mpi-sys` needs an MPI installation and libclang | ## License diff --git a/crates/clawhdf5-migrate/README.md b/crates/clawhdf5-migrate/README.md index 092bc94..9e00f38 100644 --- a/crates/clawhdf5-migrate/README.md +++ b/crates/clawhdf5-migrate/README.md @@ -1,11 +1,8 @@ # clawhdf5-migrate -[![crates.io](https://img.shields.io/crates/v/clawhdf5-migrate.svg)](https://crates.io/crates/clawhdf5-migrate) -[![docs.rs](https://img.shields.io/docsrs/clawhdf5-migrate)](https://docs.rs/clawhdf5-migrate) - CLI tool to migrate a SQLite agent-memory database in the `memory_chunks` / `sessions` / `entities` / `relations` layout (table and column names are configurable) to a -[clawhdf5-agent](https://crates.io/crates/clawhdf5-agent) store. This is **not** +[clawhdf5-agent](../clawhdf5-agent/README.md) store. This is **not** ZeroClaw's schema — ZeroClaw keeps memories in a single `memories` table and does not use clawhdf5. @@ -15,10 +12,15 @@ the knowledge graph (entities and relations) are carried over. ## Installation +Not on crates.io yet; install from a checkout: + ```bash -cargo install clawhdf5-migrate +cargo install --path crates/clawhdf5-migrate ``` +It builds C: `rusqlite` is built with its `bundled` feature, which compiles +SQLite (so no system libsqlite is needed, but a C compiler is). + ## Usage ```bash diff --git a/crates/clawhdf5-napi/README.md b/crates/clawhdf5-napi/README.md new file mode 100644 index 0000000..f102069 --- /dev/null +++ b/crates/clawhdf5-napi/README.md @@ -0,0 +1,38 @@ +# clawhdf5-napi + +> **Status: does not work end to end.** The TypeScript package built on +> this crate (`packages/clawhdf5-node`) has never run successfully, is +> unpublished, and is not built or tested in CI. See "The Node.js package +> does not work" in [`docs/known-issues.md`](../../docs/known-issues.md). +> Fix it and add CI, or remove it, before depending on it. + +A Node.js native addon ([napi-rs](https://napi.rs), N-API 9) exposing +[`clawhdf5-agent`](../clawhdf5-agent/README.md) as a `ClawhdfMemory` +class. It wraps `clawhdf5_agent::openclaw::ClawhdfBackend` (the agent's +`search` with re-ranking and confidence on) and the consolidation engine. +It was written for an OpenClaw integration that is not being pursued +([`docs/openclaw.md`](../../docs/openclaw.md)). + +## What the addon exposes + +`ClawhdfMemory.create(path, dim)`, `.open(path)`, `.openOrCreate(path, +dim)`, and on an instance: `search`, `get`, `write`, `ingestMarkdown`, +`exportMarkdown`, `save`, `saveBatch`, `stats`, `compact`, `tickSession`, +`flushWal`, `walPendingCount`, `runConsolidation`, and the ephemeral tier +(`enableEphemeral`, `ephemeralSet`/`Get`/`Delete`, `ephemeralStats`, +`promoteEphemeral`). napi-rs converts names and `#[napi(object)]` fields to +camelCase. + +## Build + +```bash +cargo build --release -p clawhdf5-napi # the Rust cdylib +# the .node package: npm install -g @napi-rs/cli; cd packages/clawhdf5-node; napi build --platform --release +``` + +It links against Node's N-API through `napi-sys` (a `-sys` crate), so it is +exempt from CI's "no C in the default build" check. + +## License + +MIT diff --git a/crates/clawhdf5-netcdf4/README.md b/crates/clawhdf5-netcdf4/README.md index ddc3452..9e82b97 100644 --- a/crates/clawhdf5-netcdf4/README.md +++ b/crates/clawhdf5-netcdf4/README.md @@ -1,25 +1,53 @@ # clawhdf5-netcdf4 -[![crates.io](https://img.shields.io/crates/v/clawhdf5-netcdf4.svg)](https://crates.io/crates/clawhdf5-netcdf4) -[![docs.rs](https://docs.rs/clawhdf5-netcdf4/badge.svg)](https://docs.rs/clawhdf5-netcdf4) +Read NetCDF-4 files in pure Rust. NetCDF-4 files are HDF5 files with +conventions for dimensions, coordinate variables and attributes; this crate +reads them through the [`clawhdf5`](../clawhdf5/README.md) facade, with no +libnetcdf or libhdf5. Read-only: NetCDF-3 (classic) files are not HDF5 and +are not supported. -NetCDF-4 read support built on clawhdf5 — pure Rust, no C dependencies. +Not on crates.io yet; depend on it from git: -## Features - -- Read NetCDF-4 / HDF5-backed `.nc` files -- Dimension, variable, and CF convention support -- Climate and scientific data access +```toml +[dependencies] +clawhdf5-netcdf4 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +``` ## Usage -```rust +```rust,no_run use clawhdf5_netcdf4::NetCDF4File; -let nc = NetCDF4File::open("climate.nc").unwrap(); -let temp = nc.variable("temperature").unwrap(); +let nc = NetCDF4File::open("climate.nc")?; +for dim in nc.dimensions()? { + println!("{}: {} (unlimited = {})", dim.name, dim.size, dim.is_unlimited); +} +let mut temp = nc.variable("temperature")?; +let dims: Vec<&str> = temp.dimensions().iter().map(|d| d.name.as_str()).collect(); +println!("{:?} over {:?}", temp.shape()?, dims); +let cf = temp.cf_attributes()?; +println!("units: {:?}", cf.units); +// scale_factor/add_offset applied; _FillValue and missing_value become NaN +let values: Vec = temp.read_f64()?; +# Ok::<(), clawhdf5_netcdf4::Error>(()) ``` +## API + +| Item | What | +|---|---| +| `NetCDF4File` | `open`, `from_bytes`, `dimensions`, `variables`, `variable`, `global_attrs`, `group`, `group_names`, `nc_properties`, and `hdf5_file` for the underlying `clawhdf5::File` | +| `NetCDF4Group` | the same for a sub-group (`dimensions`, `variables`, `attrs`, nested `group`) | +| `Variable` | `name`, `shape`, `dimensions`, `nc_type`, `is_coordinate`, `attrs`, `cf_attributes`; `read_f64` (CF scale/offset and fill applied), `read_raw_f32`/`_f64`/`_i32`/`_i64`/`_u64`, `read_string`, `read_raw` | +| `Dimension` | `name`, `size`, `is_unlimited` | +| `CfAttributes` | CF convention attributes: `units`, `long_name`, `standard_name`, `fill_value` (`_FillValue`), `missing_value`, `scale_factor`, `add_offset`, `valid_range`, `calendar`, `axis` | +| `NcType` | the NetCDF type of a variable | + +No cargo features. Tests compare against files written by netCDF4-python +(`tests/interop_tests.rs`; the CI job requires them with +`CLAWHDF5_REQUIRE_INTEROP=1`). What the HDF5 reader underneath cannot +read is listed in [`docs/known-issues.md`](../../docs/known-issues.md). + ## License MIT diff --git a/crates/clawhdf5-py/README.md b/crates/clawhdf5-py/README.md index 06eebec..73f5053 100644 --- a/crates/clawhdf5-py/README.md +++ b/crates/clawhdf5-py/README.md @@ -1,8 +1,5 @@ # clawhdf5-py -[![crates.io](https://img.shields.io/crates/v/clawhdf5-py.svg)](https://crates.io/crates/clawhdf5-py) -[![docs.rs](https://docs.rs/clawhdf5-py/badge.svg)](https://docs.rs/clawhdf5-py) - Python bindings for clawhdf5 — a pure-Rust HDF5 library. The package is `clawhdf5` (`import clawhdf5`); it needs numpy and no libhdf5. @@ -61,6 +58,8 @@ with clawhdf5.File("data.h5", "r") as f: `clawhdf5.InternalError`, a `RuntimeError`. - Attributes return what h5py returns; `clawhdf5.Empty` stands for a null dataspace (h5py's `Empty`). +- Also as in h5py: `File.mode` (`'r'`, or `'r+'` for a writable file), `File.flush()` (a no-op: + edits are already synced), `Dataset.chunks`. ## Remote files diff --git a/crates/clawhdf5-remote/README.md b/crates/clawhdf5-remote/README.md index cd663b4..876f2a2 100644 --- a/crates/clawhdf5-remote/README.md +++ b/crates/clawhdf5-remote/README.md @@ -114,3 +114,34 @@ let s = storage.stats(); // requests, bytes_fetched, hits, misses, cached_bytes, directory with range support (the server the tests use), and `cargo run -p clawhdf5-remote --example read_url -- URL [DATASET]` lists a file and prints what it cost. + +## Other front ends + +- `h5rs` (built with `--features remote`, or `remote-https`) takes URLs as + FILE arguments: [`clawhdf5-tools`](../clawhdf5-tools/README.md). +- Python: `clawhdf5.File("http://…")` and `File.open_url(url, ...)` go + through this crate: [`clawhdf5-py`](../clawhdf5-py/README.md). +- The browser does **not** use this crate (its cache fetches by blocking); + `clawhdf5-wasm`'s `openUrl` has its own restartable cache: + [`examples/wasm-viewer`](../../examples/wasm-viewer/README.md). + +## Limits + +Files a SWMR writer is still appending to cannot be followed remotely +(the file is pinned at open, so growth is `RemoteError::FileChanged`); the +block size is fixed rather than taken from a paged file's page size; the +cloud backends are built and unit-tested but have not been run against a +real bucket. The full list is under "Remote files (`clawhdf5-remote`) +limits" in [`docs/known-issues.md`](../../docs/known-issues.md); the design +is milestone M3 of [`docs/design/range-reads.md`](../../docs/design/range-reads.md). + +Not on crates.io yet; depend on it from git: + +```toml +[dependencies] +clawhdf5-remote = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +``` + +## License + +MIT diff --git a/crates/clawhdf5-tools/README.md b/crates/clawhdf5-tools/README.md index 24ecb9c..5c1cd99 100644 --- a/crates/clawhdf5-tools/README.md +++ b/crates/clawhdf5-tools/README.md @@ -306,4 +306,15 @@ CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 cargo test -p clawhd The interop tests write their files with h5py and compare with h5ls, h5stat, h5dump and h5diff; each skips when what it needs is missing unless -`CLAWHDF5_REQUIRE_INTEROP=1`. +`CLAWHDF5_REQUIRE_INTEROP=1`. `tests/remote.rs` runs every subcommand on +URLs against a local range server. + +This crate also holds the interop tests of the library's in-place editor +(`clawhdf5::FileEditor`), since they use `h5rs check` and h5dump on every +edited file and compare index and heap structures with what libhdf5 makes +of the same edits: + +```bash +CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 \ + cargo test -p clawhdf5-tools --test edit_interop --test edit_coverage_interop +``` diff --git a/crates/clawhdf5-wasm/README.md b/crates/clawhdf5-wasm/README.md new file mode 100644 index 0000000..80c27a2 --- /dev/null +++ b/crates/clawhdf5-wasm/README.md @@ -0,0 +1,51 @@ +# clawhdf5-wasm + +clawhdf5's HDF5 and NetCDF-4 reader compiled to WebAssembly with +wasm-bindgen, for the browser (and Node). Read-only. Two ways in: + +- `open(bytes)` — a file already in memory (a dropped file, a fetched + blob); +- `openUrl(url, opts)` — a file on a web server, read by HTTP range + requests as each call needs its bytes, without downloading it + (range-read milestone M4, [`docs/design/range-reads.md`](../../docs/design/range-reads.md)). + +Both give `list`, `info`, `attrs`, `read` and `readHyperslab`; the remote +file's methods return promises, and `stats()` counts requests and bytes. +The JavaScript API, options, limits, package size and tests are documented +with the demo page, [`examples/wasm-viewer/README.md`](../../examples/wasm-viewer/README.md). + +## Layout + +- `src/core.rs` — the reader over any `clawhdf5_format::storage::Storage` + (`Reader::open_storage`), plain Rust and tested natively. +- `src/lazy.rs` — the restartable "NeedBytes" cache behind `openUrl`: a + call runs as a pass over the blocks fetched so far; a pass that misses is + abandoned, the missing (and hinted) blocks are fetched, and the pass is + run again. No block is evicted while a call runs. +- `js/remote.js` — the HTTP side: `fetch` with `Range`, checking every + answer (a `206` with exactly the bytes asked for, same ETag/Last-Modified + and length) so a call fails rather than return another file's bytes. +- `src/lib.rs` — the wasm-bindgen exports. + +## Build and test + +```bash +rustup target add wasm32-unknown-unknown +cargo install wasm-bindgen-cli --version 0.2.129 # must equal the crate's wasm-bindgen +bash examples/wasm-viewer/build.sh # -> examples/wasm-viewer/pkg/ +cargo test -p clawhdf5-wasm # native: h5py_interop, lazy, vl_strings +bash examples/wasm-viewer/test/run.sh # Node + headless Chromium (not in CI) +``` + +`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus cargo test -p +clawhdf5-wasm --test lazy` compares every corpus file read lazily with the +same file read from bytes. + +Built without `mmap` and `parallel` and without the Zstd and SZIP filters +(they link C): such datasets fail with `unsupported filter`. No C is +compiled; `publish = false` (it is distributed as the package +`build.sh` makes). + +## License + +MIT diff --git a/crates/clawhdf5/README.md b/crates/clawhdf5/README.md index 707715b..6ba17e5 100644 --- a/crates/clawhdf5/README.md +++ b/crates/clawhdf5/README.md @@ -1,27 +1,101 @@ # clawhdf5 -[![crates.io](https://img.shields.io/crates/v/clawhdf5.svg)](https://crates.io/crates/clawhdf5) -[![docs.rs](https://docs.rs/clawhdf5/badge.svg)](https://docs.rs/clawhdf5) +The main crate: a pure-Rust HDF5 reader, writer and in-place editor, with no +libhdf5 and, by default, no C code. It wraps +[`clawhdf5-format`](../clawhdf5-format/README.md) (the binary format) and +[`clawhdf5-io`](../clawhdf5-io/README.md) (memory-mapped reads) in an +h5py-like API. -Pure-Rust HDF5 reader/writer — no C dependencies. +Not on crates.io yet; depend on it from git: + +```toml +[dependencies] +clawhdf5 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" } +``` + +## Main types + +| Type | What it does | +|---|---| +| `File` | Opens a file (`open`, `open_buffered`, `from_bytes`, `open_storage` for any `Storage`), walks groups (`root`, `group`, `dataset`), lists `datasets`/`groups`/`attrs`. `File` is `Send + Sync`: several threads can read one open file. | +| `Dataset` | `shape`, `dtype`, `max_dimensions`, `attrs`; reads `read_f64`/`read_f32`/`read_i32`/`read_i64`/`read_u64`, strings (`read_string`, `read_string_bytes`), variable-length data (`read_vlen`), hyperslabs and point selections (`read_selection`, `read_f64_selection`, ...), zero-copy views of contiguous data (`read_f64_zerocopy`, ...), `verify_provenance`. | +| `FileBuilder` | Writes a new file: datasets of every numeric type, strings, compounds (`CompoundTypeBuilder`), enums, chunked and compressed layouts (deflate, shuffle, Fletcher-32, LZF, and with features LZ4, Zstd, bitshuffle, bzip2, Blosc, pcodec), nested groups, soft/hard/external links, virtual datasets, attribute creation order. Files open in h5py and h5dump. | +| `FileEditor` | Changes an existing file in place without rewriting it: `write_values`/`write_selection`/`write_all`, `resize` of chunked datasets (every chunk index), `set_attr` (compact and dense storage). Anything it cannot do safely is `Error::Unsupported` before any write. | +| `MmapFile`, `LazyFile` | Alternative readers: memory-mapped, and one that reads lazily and caches. | +| `File::open_swmr` | Reads a file a libhdf5 SWMR writer is still appending to (`Dataset::refresh`, bounded retries), as h5py's `swmr=True` reader does. | + +## Examples + +```rust,no_run +use clawhdf5::{AttrValue, File, FileBuilder, FileEditor, Selection}; + +// Write +let mut b = FileBuilder::new(); +b.create_dataset("sensors/temperature") + .with_f64_data(&[20.5, 21.0, 21.5, 22.0]) + .with_shape(&[4]) + .with_maxshape(&[u64::MAX]) // unlimited, so it can grow + .with_chunks(&[2]) + .with_deflate(4); +b.set_attr("version", AttrValue::I64(1)); +b.write("data.h5")?; + +// Read +let file = File::open("data.h5")?; +let ds = file.dataset("sensors/temperature")?; +assert_eq!(ds.shape()?, vec![4]); +let values = ds.read_f64()?; + +// Edit in place: grow the dataset and fill the new tail +let mut ed = FileEditor::open("data.h5")?; +ed.resize("sensors/temperature", &[6])?; +let tail = Selection::Hyperslab { + start: vec![4], + stride: vec![1], + count: vec![2], + block: vec![1], +}; +ed.write_values("sensors/temperature", &tail, &[22.5f64, 23.0])?; +# Ok::<(), clawhdf5::Error>(()) +``` + +Remote files (HTTP range requests, S3/GCS/Azure) are read through +`File::open_storage`; [`clawhdf5-remote`](../clawhdf5-remote/README.md) +provides the storage and its block cache. ## Features -- Read and write HDF5 files entirely in Rust -- Memory-mapped I/O for large files (`mmap` feature, enabled by default) -- Parallel chunk reads via Rayon (`parallel` feature) -- Lazy dataset access for minimal memory usage -- h5py-compatible file output +| Feature | Default | What | Builds C | +|---|---|---|---| +| `mmap` | yes | memory-mapped reads (`File::open` maps the file; `MmapFile`) | no | +| `provenance` | yes | SHA-256 `_provenance_sha256` attributes (`DatasetBuilder::with_provenance`, `Dataset::verify_provenance`) | no | +| `lzf` | yes | LZF filter (32000), h5py's `compression="lzf"` | no | +| `parallel` | no | chunk decoding on a rayon pool | no | +| `lz4` | no | LZ4 filter (32004) | no | +| `pcodec` | no | pcodec filter | no | +| `bitshuffle`, `bzip2`, `blosc` | no | plugin filters 32008, 307, 32001 (read and write) | no (bzip2 uses the pure-Rust `libbz2-rs-sys`) | +| `blosc2`, `zfp` | no | plugin filters 32026 and 32013, **read only** | no | +| `plugin-filters` | no | `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2`, `zfp` | no | +| `zstd` | no | Zstandard filter (32015) | yes (libzstd) | +| `fast-deflate` | no | zlib-ng instead of the pure-Rust zlib-rs | yes (cmake) | +| `blake3_hash` | no | `provenance::blake3_hash` helpers | yes (`cc`, for blake3's SIMD code) | +| `apple-compression` | no | currently has no effect in this crate (it is not forwarded) | — | -## Usage +SZIP decoding is a `clawhdf5-format` feature (`szip`, links the system +libaec); the facade does not forward it. -```rust -use clawhdf5::File; +## Limits and further reading -let file = File::open("data.h5").unwrap(); -let dataset = file.dataset("/group/data").unwrap(); -let values: Vec = dataset.read_1d().unwrap(); -``` +- What is known not to work, and what was wrong in earlier releases: + [`docs/known-issues.md`](../../docs/known-issues.md) (editor limits, range + reads, external links and external raw data, which are explicit errors). +- Read coverage against libhdf5/h5py on eight public corpora: + [`CONFORMANCE.md`](../../CONFORMANCE.md). +- Read and write speed against libhdf5 and h5py: + [`BENCHMARKS.md`](../../BENCHMARKS.md). +- Range reads and SWMR design: [`docs/design/range-reads.md`](../../docs/design/range-reads.md), + [`docs/design/swmr.md`](../../docs/design/swmr.md). +- Changes: [`CHANGELOG.md`](../../CHANGELOG.md). ## License