2026-09-28 17:24:26 +00:00
20 changed files with 901 additions and 142 deletions
@@ -1,24 +1,61 @@
 # clawhdf5-accel
-[![crates.io](https://img.shields.io/crates/v/clawhdf5-accel.svg)](https://crates.io/crates/clawhdf5-accel)
+CPU SIMD kernels for vector search: dot products, cosine similarity, L2
-[![docs.rs](https://docs.rs/clawhdf5-accel/badge.svg)](https://docs.rs/clawhdf5-accel)
+distance, norms and int8 dot products, dispatched at run time to the best
 backend the CPU has, with a portable scalar fallback for every operation.
 [`clawhdf5-ann`](../clawhdf5-ann/README.md) and
 [`clawhdf5-agent`](../clawhdf5-agent/README.md) use it in their distance
 loops; it has nothing to do with HDF5 file I/O.
-SIMD-accelerated operations for clawhdf5.
+Not on crates.io yet; depend on it from git:
 ```toml
 [dependencies]
 clawhdf5-accel = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 ```
 ## API
 ```rust
 use clawhdf5_accel::{cosine_similarity, detect_backend, dot_i8, dot_product, l2_distance};
 let a = [1.0f32, 2.0, 3.0, 4.0];
 let b = [4.0f32, 3.0, 2.0, 1.0];
 assert_eq!(dot_product(&a, &b), 20.0);
 let _cos = cosine_similarity(&a, &b);
 let _l2 = l2_distance(&a, &b);
 assert_eq!(dot_i8(&[1, -2, 3], &[4, 5, -6]), -24);
 println!("{:?}", detect_backend()); // e.g. Avx2 on x86-64, Neon on aarch64
 ```
 Also `vector_norm`, `batch_norms`, `batch_cosine`, `batch_cosine_prenorm`,
 `f16_to_f32_batch`, `checksum_fletcher32` and `align_to_cache_line`.
 ## Backends
 `detect_backend()` picks once per process: `Avx512` (with the `avx512`
 feature), `Avx2` (AVX2 + FMA), `Neon` (every aarch64 CPU), or `Scalar`.
 `Sse4` and `WasmSimd128` are reported when detected but run the scalar
 kernels.
 `dot_i8`, used by the agent's quantised (int8) HNSW index, runs on
 AVX2 and on NEON — with the `SDOT` instruction (through inline assembly,
 since the intrinsic is unstable) on cores that have dotprod, such as the
 Raspberry Pi 5, and plain NEON on older ones. At equal recall the int8
 index answers 1.63x the queries per second of the f32 one on x86-64
 (AVX2) and 1.18x on a Raspberry Pi 5 ([`BENCHMARKS.md`](../../BENCHMARKS.md)).
 The aarch64 code is compiled out on x86, so only the `test-arm64` CI job
 builds and tests it.
 ## Features
- AVX2 and NEON SIMD acceleration
+| Feature | Default | What | Builds C |
- AVX-512 support (`avx512` feature)
+|---|---|---|---|
- Float16 conversion (`float16` feature)
+| `avx512` | no | AVX-512F kernels | no |
- CRC32 checksum acceleration
+| `float16` | no | `f16_to_f32_batch` through the `half` crate (a software conversion otherwise) | no |
-## Usage
+The half-precision conversion used for stored embeddings is
-
+`clawhdf5_format::float16`, not this crate's.
 ```rust
 use clawhdf5_accel::checksum::crc32_simd;
 let crc = crc32_simd(&data);
 ```
 ## License
@@ -1,28 +1,124 @@
 # clawhdf5-agent
-[![crates.io](https://img.shields.io/crates/v/clawhdf5-agent.svg)](https://crates.io/crates/clawhdf5-agent)
+Persistent memory for AI agents in a single HDF5 file: text chunks with
-[![docs.rs](https://img.shields.io/docsrs/clawhdf5-agent)](https://docs.rs/clawhdf5-agent)
+embeddings and metadata, hybrid search (HNSW vector search + BM25 keyword
 search, fused), sessions, a knowledge graph, a write-ahead log for crash
 safety, and optionally Ed25519-signed checkpoints. Stores open in h5py like
 any other HDF5 file. Built on [`clawhdf5`](../clawhdf5/README.md),
 [`clawhdf5-ann`](../clawhdf5-ann/README.md) and
 [`clawhdf5-accel`](../clawhdf5-accel/README.md).
-HDF5-backed persistent memory store for on-device AI agents.
+It is a library: no agent framework integrates it (OpenClaw and ZeroClaw
 integration claims were withdrawn on 2026-09-25; see
 [`docs/openclaw.md`](../../docs/openclaw.md)). The command-line front end
 is [`clawhdf5-cli`](../clawhdf5-cli/README.md).
-Built on [clawhdf5](https://crates.io/crates/clawhdf5), clawhdf5-agent provides a vector-searchable memory backend optimized for edge AI workloads. Store embeddings, text chunks, and metadata in a single HDF5 file with SIMD-accelerated similarity search.
+Not on crates.io yet; depend on it from git:
 ## Features
 - Persistent vector store in HDF5 format
 - Cosine similarity and L2 distance search
 - SIMD-accelerated via clawhdf5-accel (AVX2, NEON)
 - Optional GPU acceleration via clawhdf5-gpu
 - Memory-mapped access for large stores
 - f16 storage support for compact embeddings
 ## Usage
 ```toml
 [dependencies]
-clawhdf5-agent = "2.1.0"
+clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 ```
 ## Usage
 ```rust,no_run
 use std::path::PathBuf;
 use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions};
 let config = MemoryConfig::new(PathBuf::from("agent.h5"), "my-agent", 384);
 let mut mem = HDF5Memory::create(config)?;
 mem.save(MemoryEntry {
    chunk: "The deploy key rotates every Monday.".into(),
    embedding: vec![0.01; 384], // from your embedding model
    source_channel: "chat".into(),
    timestamp: 1_790_000_000.0,
    session_id: "s1".into(),
    tags: "ops".into(),
 })?;
 let query = vec![0.01f32; 384];
 let hits = mem.search(&query, "deploy key", &SearchOptions::new(5).with_sources(["chat"]));
 for h in &hits {
    println!("{:.3} {}", h.score, h.chunk);
 }
 mem.flush_wal()?; // checkpoint now; otherwise one is made after 500 WAL entries (wal_max_entries)
 # Ok::<(), clawhdf5_agent::MemoryError>(())
 ```
 ## What is in it
 - **`HDF5Memory`** — `create`, `open` (single writer: an exclusive lock on
  `<store>.h5.lock`, a second opener gets `MemoryError::Locked`),
  `open_read_only` (no lock, never writes). Through the `AgentMemory`
  trait: `save`, `save_batch`, `delete`, `compact`, `count`, `snapshot`,
  sessions; also `save_or_update`, `delete_batch`, `flush_wal`.
 - **Search** — `search(query_embedding, text, &SearchOptions)`: optional
  source-channel filter applied before ranking, vector + BM25 fusion
  (weighted or RRF), Hebbian activation scaling, optional re-ranking
  (`reranker::ReRankConfig`) and confidence rejection
  (`confidence::ConfidenceConfig`). `hybrid_search` and
  `hybrid_search_with` are thin wrappers. The vector stage uses the HNSW
  index (`hnsw` feature); its graph is saved to `<store>.h5.ann` at each
  checkpoint and reloaded on open (rebuilt if stale or damaged).
 - **Storage settings** (`MemoryConfig`, persisted with the store):
  `float16` embeddings (on by default for new stores; 48% smaller file at
  100K records, same retrieval on LongMemEval), `quantized_index` (int8
  copy of the vectors in the index, on by default; re-scored against the
  exact embeddings), `compression` (off by default), HNSW `m`/`ef`
  parameters, WAL settings (`wal_enabled`, on by default; `wal_max_entries`,
  500: the WAL is checkpointed into the `.h5` once it holds more).
 - **WAL** (`wal`) — every write is appended to `<store>.h5.wal` with a
  chained CRC32 per entry, so a corrupted, reordered or spliced entry stops
  replay. Recovers from a process crash at any point, including between a
  checkpoint and the WAL truncate. WAL appends are not fsynced: saves since
  the last checkpoint can be lost on power failure. An unreadable WAL is
  quarantined to `<store>.h5.wal.corrupt-<ts>`.
 - **Signed checkpoints** (`signing`) — `set_signing_key` signs a manifest
  (SHA-256 Merkle tree over records, plus settings, sessions and graph) at
  every checkpoint; `HDF5Memory::verify(path, &public_key)` checks it and
  locates edits. WAL entries after the checkpoint are not covered.
 - **Knowledge graph** (`knowledge`, `entity_extract`) — `add_entity`,
  `add_entity_alias`, `add_relation`, `extract_and_store_entities`,
  traversal and spreading activation.
 - **Also:** sessions (`session`), temporal index (`temporal`),
  consolidation tiers (`consolidation`), an in-memory TTL tier
  (`ephemeral`), multi-modal embeddings (`multimodal`), `AGENTS.md`
  generation (`agents_md`), query expansion, and a session-scoped
  provenance ledger and write-anomaly detector on every save
  (`take_anomaly_alerts`; alerts never block a save, and the source is
  inferred from `source_channel`, not authenticated).
 - `openclaw::ClawhdfBackend` is `search` with re-ranking and confidence
  on, plus Markdown import/export. The module name is historical: it is not
  an OpenClaw plugin.
 ## Features
 | Feature | Default | What | Builds C |
 |---|---|---|---|
 | `hnsw` | yes | HNSW vector index (`clawhdf5-ann`); without it the vector stage is an exact linear cosine scan | no |
 | `parallel` | yes | build the HNSW index on a rayon pool (same graph either way) | no |
 | `float16` | yes | f16 helpers in `vector_search` (`half`). Stores' `MemoryConfig::float16` works without it. | no |
 | `fast-math` | no | `matrixmultiply` batch distances in `strategy` | no |
 | `accelerate` | no | Apple Accelerate BLAS in `strategy` (macOS) | links a system framework |
 | `openblas` | no | OpenBLAS in `strategy` | yes (`openblas-src`) |
 | `gpu` | no | `gpu_search` through [`clawhdf5-gpu`](../clawhdf5-gpu/README.md) (wgpu), used by `strategy`, not by `HDF5Memory::search` | no, but needs GPU drivers |
 | `zstd` | no | Zstd instead of deflate when `MemoryConfig::compression` is on | yes (libzstd) |
 | `async` | no | `async_memory` wrapper on tokio | no |
 `--no-default-features --features float16` forces the exact linear scan.
 ## Measurements and limits
 - Search recall and latency, file size, LongMemEval and MemoryArena
  retrieval numbers: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with
  the `clawhdf5-bench` binaries (`search_harness`, `longmemeval_bench`,
  `footprint_bench`, ...).
 - Known issues and their history: [`docs/known-issues.md`](../../docs/known-issues.md).
 - Migrating a SQLite memory database:
  [`clawhdf5-migrate`](../clawhdf5-migrate/README.md).
 ## License
 MIT
@@ -0,0 +1,44 @@
 # clawhdf5-android
 A C ABI over [`clawhdf5-agent`](../clawhdf5-agent/README.md) for Android
 apps: a `cdylib` exporting `extern "C"` functions (`edgehdf5_*`, a name
 kept from the project's earlier "edgehdf5" days) that manage an
 `HDF5Memory` through an opaque handle.
 The functions are plain C symbols, not JNI-mangled `Java_...` entry points:
 a Kotlin/Java app calls them through a thin JNI shim or JNA of its own. No
 such shim, Gradle project or AAR is in this repository, and the crate is
 not built for an Android target in CI (only its host-side unit tests run
 with the workspace).
 ## Functions
 | Function | What |
 |---|---|
 | `edgehdf5_create(path, agent_id, embedding_dim)` / `edgehdf5_open(path)` | a handle, or null on failure |
 | `edgehdf5_close(handle)` | drop the store; what is not yet checkpointed stays in its WAL, as with any `HDF5Memory` |
 | `edgehdf5_save(handle, ...)` | save one entry; the embedding length is checked against the store's dimension before the pointer is read |
 | `edgehdf5_delete`, `edgehdf5_count`, `edgehdf5_count_active` | |
 | `edgehdf5_hybrid_search(handle, query, len, text, vector_weight, keyword_weight, max_results, out_indices, out_scores, out_chunks)` | results into caller-provided arrays; returns the number written |
 | `edgehdf5_add_session`, `edgehdf5_get_session_summary` | sessions |
 | `edgehdf5_add_entity`, `edgehdf5_add_relation` | knowledge graph |
 | `edgehdf5_free_string` | free a string this library returned |
 Every function is `unsafe`: the caller guarantees valid, NUL-terminated
 strings and correctly sized buffers (see each function's `# Safety`
 section), and serialises access to a handle; separate handles are
 independent.
 ## Build
 ```bash
 cargo build --release -p clawhdf5-android   # host build; for a device, add --target aarch64-linux-android with the NDK's linker configured
 ```
 It depends on `clawhdf5-agent` with **default features off**, so there is
 no HNSW index (the vector stage is an exact linear scan) and no rayon
 pool. No C is compiled.
 ## License
 MIT
@@ -1,25 +1,70 @@
 # clawhdf5-ann
-[![crates.io](https://img.shields.io/crates/v/clawhdf5-ann.svg)](https://crates.io/crates/clawhdf5-ann)
+An HNSW (Hierarchical Navigable Small World) approximate nearest-neighbour
-[![docs.rs](https://docs.rs/clawhdf5-ann/badge.svg)](https://docs.rs/clawhdf5-ann)
+index in pure Rust, with cosine or L2 distance, optional int8 storage of
 the vectors, deletions, and persistence as an HDF5 file. It is the vector
 stage of [`clawhdf5-agent`](../clawhdf5-agent/README.md)'s search (the
 agent's `hnsw` feature, on by default); distances run on
 [`clawhdf5-accel`](../clawhdf5-accel/README.md)'s SIMD kernels.
-HNSW approximate nearest neighbor index stored as HDF5.
+Neighbours are chosen with the HNSW paper's diversity heuristic, not plain
 closest-M (which capped recall on clustered data at 0.31 recall@10 at 100K
 vectors).
-## Features
+Not on crates.io yet; depend on it from git:
- Build and query HNSW indexes persisted in HDF5 format
+```toml
- Pure Rust, no C dependencies
+[dependencies]
- Efficient similarity search for high-dimensional vectors
+clawhdf5-ann = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 ```
 ## Usage
 ```rust
-use clawhdf5_ann::HnswIndex;
+use clawhdf5_ann::{DistanceMetric, HnswIndex, Storage};
-let index = HnswIndex::from_hdf5("vectors.h5").unwrap();
+let vectors: Vec<Vec<f32>> = (0..500)
-let neighbors = index.search(&query, 10);
+    .map(|i| (0..16).map(|j| ((i * 31 + j * 7) % 97) as f32 / 97.0).collect())
    .collect();
 // m = 16 connections per node, ef_construction = 200
 let mut index = HnswIndex::build_with(&vectors, 16, 200, DistanceMetric::Cosine, Storage::Int8);
 let hits = index.search(&vectors[42], 10, 64); // (id, distance), closest first; ef >= k
 assert!(hits[0].1 < 1e-3); // vector 42 itself (or an identical one)
 let id = index.insert(vec![0.5; 16]);
 index.mark_deleted(id);
 // Persist as HDF5 (a self-contained file: graph and vectors) and load it back
 let bytes = index.to_hdf5_bytes().unwrap();
 let loaded = HnswIndex::load_from_hdf5(&bytes).unwrap();
 assert_eq!(loaded.len(), index.len());
 ```
 - `HnswIndex::build` (L2), `build_with_metric`, `build_with` (metric and
  storage); `new`/`new_with` plus `insert` for an index built
  incrementally.
 - `Storage::Int8` keeps each vector as `i8`, a quarter of the memory; it
  applies to `Cosine` only (an L2 index keeps `Float32`). Distances are then
  approximate, so a caller that needs exact ranking re-scores the
  candidates, as the agent does.
 - `mark_deleted`, `is_deleted`, `deleted_count`, `active_len`, `compact`
  (returns the old-to-new id map).
 - `save_to_hdf5(&mut writer)` / `to_hdf5_bytes` / `load_from_hdf5` store
  the whole index; `graph_to_bytes` / `from_graph_bytes` store only the
  graph (with a CRC32) for a caller that keeps the vectors elsewhere — the
  agent's `<store>.h5.ann` sidecar.
 ## Features
 | Feature | Default | What | Builds C |
 |---|---|---|---|
 | `parallel` | no | build the graph on a rayon pool; the graph is identical with or without it | no |
 Recall and speed against exact search, for the index alone and in the
 agent: [`BENCHMARKS.md`](../../BENCHMARKS.md), measured with
 `cargo run --release -p clawhdf5-bench --bin search_harness`.
 ## License
 MIT
@@ -0,0 +1,49 @@
 # clawhdf5-bench
 The measurement harnesses behind [`BENCHMARKS.md`](../../BENCHMARKS.md):
 HDF5 read and write speed (against libhdf5 and h5py where noted) and the
 agent store's search, footprint and retrieval quality. Not meant for
 publishing; nothing else in the workspace depends on it. Run everything with
 `--release`, and quote numbers with the machine, date and command, as
 `BENCHMARKS.md` does.
 ## Binaries
 | Binary | Measures |
 |---|---|
 | `read_harness` | full reads vs hyperslab selections of a chunked 2-D dataset (compressed and not) and a contiguous one: does a selection cost scale with the selection or the dataset? (`-- --large` for 512 MB) |
 | `concurrent_read` | decoded read throughput vs threads on one open `File`; `scripts/concurrent_read_h5py.py` runs the same workload with h5py (threads and processes) and `scripts/compare_concurrent_read.py` tabulates both |
 | `search_harness` | HNSW recall@10 vs exact search, QPS and latency per `ef`, and end-to-end `HDF5Memory` ingest/checkpoint/open/search at 1K–100K (`--full`); studies: `--float16-study`, `--options-study`, `--signing-study`, `--ann-only --uniform` |
 | `longmemeval_bench` | LongMemEval retrieval recall (turn and session Hit@k, MRR) — **retrieval, not QA accuracy**. Oracle or full `longmemeval_s` haystack; `--features embeddings` (or `embeddings-cuda`) embeds with MiniLM, otherwise the vector stage is inert and the run is BM25-only |
 | `memory_arena` | a deterministic multi-session retrieval benchmark (BM25-only) |
 | `footprint_bench` | file size and bytes per record at 100–100K records, float16 or `--f32`, WAL on/off, compressed or not |
 | `consolidation_efficiency` | retrieval before and after consolidation on signal + noise records |
 | `ephemeral_perf` | the in-memory ephemeral tier's set/get latency |
 | `mpi_io_bench` | `clawhdf5-io`'s `MpiVol` (root-read + broadcast, not collective I/O); needs `--features mpi-io` and `mpirun` |
 ```bash
 cargo run --release -p clawhdf5-bench --bin search_harness -- --full
 cargo run --release -p clawhdf5-bench --bin read_harness
 ```
 ## Criterion benches and example
 - `cargo bench -p clawhdf5-bench` runs `h5bench_write`, `h5bench_read` and
  `h5bench_meta` (h5bench-style sequential, chunked, strided and metadata
  workloads). `--features libhdf5-compare` adds the same workloads through
  libhdf5 (the `hdf5-metno` crate; needs a system libhdf5 1.14).
 - `examples/worldmodel_sampling.rs`: shuffled per-frame reads of a
  `(N, H, W, C)` `uint8` dataset, clawhdf5 against h5py on the same file.
 ## Features
 | Feature | What | Builds C |
 |---|---|---|
 | `libhdf5-compare` | libhdf5 variants of the Criterion benches | links the system libhdf5 |
 | `mpi-io` | `mpi_io_bench` | yes (`mpi-sys`; needs an MPI installation) |
 | `embeddings` | MiniLM embeddings for `longmemeval_bench` (candle) | yes (a `cc` build dependency in the candle/tokenizers tree) |
 | `embeddings-cuda` | the same on a CUDA GPU (minutes instead of hours on the full haystack) | yes (CUDA) |
 ## License
 MIT
@@ -0,0 +1,49 @@
 # clawhdf5-cli
 The `clawhdf5` command: create, fill, search and inspect a
 [`clawhdf5-agent`](../clawhdf5-agent/README.md) memory store from the
 shell. Output is JSON. (For general HDF5 files use `h5rs` from
 [`clawhdf5-tools`](../clawhdf5-tools/README.md).)
 ```bash
 cargo install --path crates/clawhdf5-cli     # installs `clawhdf5`; not on crates.io yet
 # or: cargo run -p clawhdf5-cli -- --help
 ```
 No C is compiled.
 ## Commands
 The store is `--path FILE` (or `CLAWHDF5_PATH`) before the subcommand.
 | Command | What |
 |---|---|
 | `create [--agent-id ID] [--dim N] [--wal] [--f32] [--f32-index]` | a new store (dimension 384 by default); float16 embeddings and an int8 index copy unless `--f32` / `--f32-index`. The WAL is off unless `--wal` (the library's default is on), so each save is checkpointed at once |
 | `save [--json '{...}']` | save one entry, from `--json` or stdin: `{"chunk", "embedding", "source_channel", "timestamp", "session_id", "tags"}` |
 | `search --embedding '[...]' [--query TEXT] [-k N] [--vector-weight W] [--keyword-weight W]` | hybrid search (defaults 5 results, weights 0.7 / 0.3) |
 | `recall INDEX` | one entry by index |
 | `stats` | counts and configuration |
 | `flush-wal` | checkpoint the WAL into the `.h5` |
 | `agents-md [--output FILE]` | generate an `AGENTS.md` from the store |
 | `export` | every entry as JSON lines |
 | `snapshot DEST` | a copy of the store's `.h5` file |
 | `keygen --out FILE` | a new Ed25519 signing key (64 hex characters, created owner-only on Unix) |
 | `verify --public-key HEX_OR_FILE` | check a signed store; exit status 2 if it does not verify |
 `recall`, `stats`, `agents-md` and `export` open the store read-only
 (no lock, nothing written), so they work while another process has it
 open. `save`, `search` (which records activation boosts) and `flush-wal`
 open it for writing and take the store's lock. With
 `--signing-key FILE` (or `CLAWHDF5_SIGNING_KEY`) every checkpoint a command
 makes is signed; a signed store refuses to checkpoint without the key.
 ```bash
 clawhdf5 --path mem.h5 create --agent-id demo --dim 3
 echo '{"chunk":"hello","embedding":[0.1,0.2,0.3],"source_channel":"cli","timestamp":0,"session_id":"s1","tags":""}' \
  | clawhdf5 --path mem.h5 save
 clawhdf5 --path mem.h5 search --embedding '[0.1,0.2,0.3]' --query hello -k 3
 ```
 ## License
 MIT
@@ -1,28 +1,50 @@
 # clawhdf5-derive
-[![crates.io](https://img.shields.io/crates/v/clawhdf5-derive.svg)](https://crates.io/crates/clawhdf5-derive)
+`#[derive(H5Type)]`: maps a Rust struct with named fields to an HDF5
-[![docs.rs](https://docs.rs/clawhdf5-derive/badge.svg)](https://docs.rs/clawhdf5-derive)
+compound datatype. The derive generates three inherent methods:
-Derive macros for clawhdf5 HDF5 traits.
+- `hdf5_datatype() -> clawhdf5_format::datatype::Datatype` — the
  `Datatype::Compound` (members in field order, packed, little-endian);
 - `to_bytes(&self) -> Vec<u8>` — one element in that layout;
 - `from_bytes(&[u8]) -> Self` — the reverse (panics if the slice is shorter
  than the compound).
-## Features
+Supported field types: `f32`, `f64`, `i8`–`i64`, `u8`–`u64`, `bool`
 (stored as `u8`) and fixed-size arrays `[T; N]` of those numeric types.
 Tuple structs, enums and nested structs are refused at compile time.
- `#[derive(HDF5Type)]` for automatic HDF5 datatype mapping
+The generated code names `clawhdf5_format`, so the crate using the derive
- Struct-to-compound-type derivation
+must depend on [`clawhdf5-format`](../clawhdf5-format/README.md) too. Not
 on crates.io yet:
-## Usage
+```toml
 [dependencies]
 clawhdf5-derive = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 clawhdf5-format = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 ```
 ## Example
 ```rust
-use clawhdf5_derive::HDF5Type;
+use clawhdf5_derive::H5Type;
 use clawhdf5_format::datatype::Datatype;
-#[derive(HDF5Type)]
+#[derive(H5Type, Debug, PartialEq)]
 struct Point {
-    x: f64,
+    id: u32,
-    y: f64,
+    pos: [f64; 3],
-    z: f64,
+    valid: bool,
 }
 let p = Point { id: 7, pos: [1.0, 2.0, 3.0], valid: true };
 let bytes = p.to_bytes();
 assert_eq!(bytes.len(), 4 + 24 + 1);
 assert_eq!(Point::from_bytes(&bytes), p);
 assert!(matches!(Point::hdf5_datatype(), Datatype::Compound { size: 29, .. }));
 ```
 Tests: `crates/clawhdf5-format/tests/derive_tests.rs`.
 ## License
 MIT
@@ -1,27 +1,56 @@
 # clawhdf5-filters
-[![crates.io](https://img.shields.io/crates/v/clawhdf5-filters.svg)](https://crates.io/crates/clawhdf5-filters)
+Standalone deflate (zlib) compression and decompression with a choice of
-[![docs.rs](https://docs.rs/clawhdf5-filters/badge.svg)](https://docs.rs/clawhdf5-filters)
+backend: pure-Rust zlib-rs (default), zlib-ng, Apple's Compression
 framework, or miniz_oxide.
-Filter and compression pipeline for clawhdf5.
+This crate holds **deflate backends only**. The HDF5 filter pipeline, the
 filter registry and every other codec (shuffle, Fletcher-32, N-Bit,
 scale-offset, LZ4, Zstd, SZIP, pcodec, LZF, bitshuffle, bzip2, Blosc,
 Blosc2, ZFP) live in [`clawhdf5-format`](../clawhdf5-format/README.md),
 which calls flate2 itself and selects its deflate backend with its own
 features. No library crate of the workspace depends on this one (the
 `clawhdf5` facade uses it only in tests).
-## Features
+Not on crates.io yet; depend on it from git:
- DEFLATE compression/decompression
+```toml
- Pure-Rust deflate via zlib-rs (default, `zlib-rs` feature)
+[dependencies]
- zlib-ng instead, if you want it (`fast-deflate` feature; C, needs cmake)
+clawhdf5-filters = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
- Apple Compression framework support (`apple-compression` feature)
+```
-## Usage
+## API
 ```rust
-use clawhdf5_filters::{deflate_compress, deflate_decompress};
+use clawhdf5_filters::{deflate_backend, deflate_compress, deflate_decompress};
 let data: Vec<u8> = (0..10_000u32).map(|i| (i % 251) as u8).collect();
 let compressed = deflate_compress(&data, 6).unwrap();
 // The second argument bounds the output: the expected decompressed size.
 let decompressed = deflate_decompress(&compressed, data.len()).unwrap();
 assert_eq!(decompressed, data);
 println!("backend: {}", deflate_backend()); // "zlib-rs" by default
 ```
 Also `deflate_compress_miniz`/`deflate_decompress_miniz` (always
 miniz_oxide) and `fast_deflate::{compress, decompress, active_backend}`.
 ## Features
 Backend priority: `apple-compression` (macOS only) > zlib-ng > zlib-rs >
 miniz_oxide (with none enabled).
 | Feature | Default | Backend | Builds C |
 |---|---|---|---|
 | `zlib-rs` | yes | zlib-rs through flate2, with `runtime_detection` (needed for its SIMD) | no |
 | `fast-deflate` | no | zlib-ng through flate2 | yes (cmake) |
 | `system-zlib` | no | the system zlib through flate2 | yes (`libz-sys`) |
 | `apple-compression` | no | Apple Compression framework, macOS only (ignored elsewhere) | no (links a system framework) |
 zlib-rs matches zlib-ng on HDF5 reads and writes and produces
 byte-identical output: see "Deflate backend" in
 [`BENCHMARKS.md`](../../BENCHMARKS.md).
 ## License
 MIT
@@ -1,27 +1,106 @@
 # clawhdf5-format
-[![crates.io](https://img.shields.io/crates/v/clawhdf5-format.svg)](https://crates.io/crates/clawhdf5-format)
+The HDF5 file format in pure Rust: parsers and writers for every on-disk
-[![docs.rs](https://docs.rs/clawhdf5-format/badge.svg)](https://docs.rs/clawhdf5-format)
+structure, the filter pipeline and its codecs, and the shared type
 definitions the other crates use. Most users want the
 [`clawhdf5`](../clawhdf5/README.md) facade, which wraps this crate in an
 h5py-like API; use this one directly for low-level access or in `no_std`
 code.
-Pure-Rust HDF5 binary format parsing and writing — no C dependencies.
+Not on crates.io yet; depend on it from git:
 ```toml
 [dependencies]
 clawhdf5-format = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 ```
 ## What is in it
 - **Parsing:** superblock v0–v3 (`superblock`, with the superblock
  extension and metadata cache images, `superblock_ext`), object headers v1
  and v2 (`object_header`), every header message the readers use
  (`datatype`, `dataspace`, `data_layout` v1–v4 including virtual datasets,
  `fill_value`, `attribute`, `link_message`, `shared_message`, ...), groups
  old and new (`group_v1` symbol tables with local heaps, `group_v2` with
  fractal heaps and v2 B-trees), and every chunk index (v1 B-tree, single
  chunk, implicit, fixed array, extensible array, v2 B-tree).
 - **Reading data:** `data_read` (contiguous, compact, chunked),
  `partial_read` and `selection` (hyperslabs and points), `vl_data`
  (variable-length strings and sequences through the global heap),
  `chunk_cache`.
 - **Storage:** the `storage::Storage` trait (`read_at`, `read_ranges`,
  `len`, `hint`) that every read path goes through, so a file can be read
  from memory, a file handle or a remote backend
  ([`clawhdf5-remote`](../clawhdf5-remote/README.md)).
 - **Writing:** `file_writer::FileWriter` and the builders in
  `type_builders` (datasets, groups, attributes, compound and enum types,
  links, virtual datasets, creation-order tracking); chunk indexes and
  dense-storage B-trees of any size (`chunked_write`, `btree_v2_write`,
  `ea_writer`). Output is read by h5py and h5dump.
 - **Filters:** `filter_pipeline` and `filter_registry` (look up by ID; other
  IDs can be registered at run time with `register_filter`). Built in:
  deflate, shuffle, Fletcher-32, N-Bit, scale-offset; behind features LZ4,
  Zstd, SZIP (decode), pcodec, and the plugin filters LZF, bitshuffle,
  bzip2, Blosc 1 (read and write), Blosc2 and ZFP (read only).
 - **Shared pieces:** `float16` (the one IEEE half-precision conversion the
  workspace uses), `provenance` (SHA-256 dataset hashes), `checksum`
  (Jenkins lookup3 for v2+ structures).
 ## Example
 ```rust
 use clawhdf5_format::file_writer::{AttrValue, FileWriter};
 use clawhdf5_format::{group_v2, object_header, signature, superblock};
 // Write a file to memory
 let mut fw = FileWriter::new();
 fw.create_dataset("data")
    .with_f64_data(&[1.0, 2.0, 3.0])
    .with_shape(&[3])
    .set_attr("unit", AttrValue::String("m/s".into()));
 let bytes = fw.finish().unwrap();
 // Parse it back: superblock -> path -> object header
 let (_user_block, file) = signature::split_user_block(&bytes).unwrap();
 let sb = superblock::Superblock::parse(file, 0).unwrap();
 let addr = group_v2::resolve_path_any(file, &sb, "data").unwrap();
 let hdr = object_header::ObjectHeader::parse(file, addr as usize, sb.offset_size, sb.length_size)
    .unwrap();
 assert!(!hdr.messages.is_empty());
 ```
 ## Features
- Zero-copy superblock, object header, and B-tree parsing
+| Feature | Default | What | Builds C |
- Chunked dataset read/write with filter pipelines
+|---|---|---|---|
- `no_std` support (disable `std` feature)
+| `std` | yes | standard library; without it the crate is `no_std` + `alloc` (CI builds it for `thumbv7em-none-eabihf`) | no |
- Optional parallel reads via Rayon
+| `checksum` | yes | verify Jenkins lookup3 checksums | no |
- SHA-256 provenance tracking
+| `deflate` | yes | deflate through flate2 | no |
 | `zlib-rs` | yes | flate2's pure-Rust zlib-rs backend, with `runtime_detection` (without it zlib-rs loses SIMD and inflates 3.5x slower) | no |
 | `system-zlib-decompress` | yes | no effect (nothing reads it; kept so existing feature lists still build) | no |
 | `provenance` | yes | SHA-256 provenance hashes | no |
 | `lzf` | yes | LZF (32000) | no |
 | `parallel` | no | rayon-parallel chunk decoding | no |
 | `fast-checksum` | no | hardware CRC32 through `crc32fast` | no |
 | `lz4` | no | LZ4 (32004) | no |
 | `pcodec` | no | pcodec | no |
 | `bitshuffle`, `bzip2`, `blosc` | no | 32008, 307, 32001, read and write | no |
 | `blosc2`, `zfp` | no | 32026, 32013, read only | no |
 | `plugin-filters` | no | all six plugin filters above | no |
 | `lookup-stats` | no | counters for name-lookup benchmarks | no |
 | `zstd` | no | Zstandard (32015) | yes (libzstd) |
 | `szip` | no | SZIP (4) decoding | links the system libaec (`libaec-dev`) |
 | `fast-deflate` | no | zlib-ng | yes (cmake) |
 | `system-zlib` | no | the system zlib | yes (`libz-sys`) |
 | `blake3_hash` | no | `provenance::blake3_hash` | yes (`cc`) |
-## Usage
+## Robustness
-```rust
+Every parser is meant to return an error, never panic, on hostile input:
-use clawhdf5_format::Superblock;
+nine cargo-fuzz targets live in [`fuzz/`](fuzz/README.md), the conformance
-
+sweep includes the HDF Group's CVE corpus
-let data = std::fs::read("data.h5").unwrap();
+([`CONFORMANCE.md`](../../CONFORMANCE.md)), and header checks follow
-let sb = Superblock::from_bytes(&data).unwrap();
+libhdf5's. Open gaps are in [`docs/known-issues.md`](../../docs/known-issues.md).
 println!("HDF5 version {}.{}", sb.version_major(), sb.version_minor());
 ```
 ## License
@@ -51,10 +51,18 @@ done
 ## CI
-These targets are **not** run in CI (`.gitea/workflows/ci.yml`) — cargo-fuzz
+These targets are **not** run by the CI workflows (`.gitea/workflows/ci.yml`)
-requires nightly and each meaningful run takes minutes, which doesn't fit a
+— cargo-fuzz requires nightly and each meaningful run takes minutes, which
-per-PR gate. Run them manually on a schedule (e.g. before a release, or after
+doesn't fit a per-PR gate. Run them by hand before a release or after
-touching parser code) instead.
+touching parser code. `scripts/ci-test.sh` has an opt-in smoke run: with
 `CLAWHDF5_FUZZ_SECONDS=N` it runs every target of this crate and of
 `crates/clawhdf5-agent/fuzz` (the WAL parser) for N seconds each.
 Other robustness checks that do run: the nightly conformance sweep reads
 the HDF Group's CVE reproducers and fails on any panic, hang, crash or
 out-of-memory ([`conformance/README.md`](../../../conformance/README.md)),
 and `scripts/h5rs-fuzz.sh` runs every `h5rs` subcommand over them, optionally
 on byte-flipped copies.
 ## Reproducing Crashes
@@ -1,25 +1,62 @@
 # clawhdf5-gpu
-[![crates.io](https://img.shields.io/crates/v/clawhdf5-gpu.svg)](https://crates.io/crates/clawhdf5-gpu)
+GPU vector distance computation through [wgpu](https://wgpu.rs) and
-[![docs.rs](https://docs.rs/clawhdf5-gpu/badge.svg)](https://docs.rs/clawhdf5-gpu)
+hand-written WGSL compute shaders: upload a set of vectors once, then run
 cosine or L2 top-k searches, dot products, distance matrices and norms
 against them on Vulkan, Metal, DirectX 12 or OpenGL.
-GPU-accelerated vector operations for clawhdf5 using wgpu compute shaders.
+This crate does **not** read or write HDF5: dataset I/O in clawhdf5 is
 CPU-only. It is a vector-search accelerator used optionally by
 [`clawhdf5-agent`](../clawhdf5-agent/README.md) (its `gpu` feature exposes
 `gpu_search::GpuSearchBackend` and a GPU arm of `strategy::search_with_metrics`;
 `HDF5Memory::search` itself uses the HNSW index on the CPU).
-## Features
+Not on crates.io yet; depend on it from git:
- GPU-accelerated distance computations (L2, cosine)
+```toml
- wgpu-based compute shaders for cross-platform GPU support
+[dependencies]
- Float16 support via `half` crate
+clawhdf5-gpu = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 ```
 ## Usage
-```rust
+```rust,no_run
 use clawhdf5_gpu::GpuAccelerator;
-let accel = GpuAccelerator::new().unwrap();
+// Fall back to a CPU path when there is no usable GPU.
-let distances = accel.l2_distances(&query, &vectors).unwrap();
+let mut gpu = match GpuAccelerator::new() {
    Ok(g) => g,
    Err(_) => return,
 };
 let dim = 128;
 let vectors = vec![0.5f32; 1000 * dim]; // 1000 vectors, row-major
 gpu.upload_vectors(&vectors, dim).unwrap();
 let norms = gpu.compute_norms_gpu(&vectors, dim).unwrap();
 gpu.upload_norms(&norms).unwrap();
 let query = vec![1.0f32; dim];
 let top10 = gpu.cosine_search(&query, 10).unwrap(); // (index, similarity), best first
 let near10 = gpu.l2_search(&query, 10).unwrap();    // (index, distance), nearest first
 ```
 `GpuAccelerator` also has `is_available`, `device_info`,
 `batch_cosine_search`, `batch_dot_product`, `distance_matrix`,
 `compute_norms`, and `f16_to_f32_batch`/`f32_to_f16_batch`. Vector sets
 larger than the device's largest storage buffer binding are split into
 chunks and the results merged. A GPU→CPU readback waits at most 30 s and
 then fails with `GpuError::BufferMap` instead of hanging.
 ## Features
 | Feature | Default | What |
 |---|---|---|
 | `gpu-wgpu` | yes | the wgpu implementation. Without it `GpuAccelerator::new()` returns `GpuError::NotCompiled` and `is_available()` is `false`. |
 No C is compiled, but wgpu talks to the system's graphics drivers at run
 time; the crate is exempt from CI's "no C in the default build" check for
 that reason.
 ## License
 MIT
@@ -1,24 +1,54 @@
 # clawhdf5-io
-[![crates.io](https://img.shields.io/crates/v/clawhdf5-io.svg)](https://crates.io/crates/clawhdf5-io)
+I/O building blocks under [`clawhdf5`](../clawhdf5/README.md): the
-[![docs.rs](https://docs.rs/clawhdf5-io/badge.svg)](https://docs.rs/clawhdf5-io)
+`HDF5Read`/`HDF5ReadWrite` traits with in-memory, borrowed, file and
 memory-mapped readers, plus several experimental modules (async reads, an
 HSDS client, a VOL-style trait, sub-filing, prefetch, and an MPI connector).
 The facade uses it for memory-mapped reads (`MmapReader`, and the private
 copy-on-write mapping that applies a metadata cache image).
-I/O abstraction layer for clawhdf5.
+Remote files are **not** read through this crate: HTTP(S) and object
 stores go through `clawhdf5_format::storage::Storage` and
 [`clawhdf5-remote`](../clawhdf5-remote/README.md).
 Not on crates.io yet; depend on it from git:
 ```toml
 [dependencies]
 clawhdf5-io = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5", features = ["mmap"] }
 ```
 ## Main items
 | Item | What |
 |---|---|
 | `HDF5Read`, `HDF5ReadWrite` | byte-level read/write traits; `MemoryReader`, `BorrowedReader`, `FileReader`, `FileWriter` implement them |
 | `MmapReader`, `MmapReadWrite` (`mmap`) | memory-mapped files through `memmap2`; `HDF5Read::private_copy` gives a copy-on-write view |
 | `prefetch::PrefetchReader`, `sweep::SweepDetector` | read-ahead (`madvise(MADV_WILLNEED)` on mappings) and chunk-sweep prediction |
 | `ParallelConfig` | lane partitioning for parallel chunk decoding |
 | `vol::VirtualObjectLayer`, `vol::NativeVol` | a backend-agnostic object-layer trait (modelled on libhdf5's VOL) |
 | `async_read` (`async`) | tokio-based `AsyncHDF5Read` and `AsyncHDF5File` |
 | `hsds::HsdsClient` (`hsds`) | a REST client for an HSDS server |
 | `subfiling` | splitting one logical file across several physical files |
 | `mpi_vol::MpiVol` (`mpi-io`) | an MPI connector: see below |
 ### MPI (`mpi-io`)
 `MpiVol` is **not collective MPI-IO**. Reads are root-read + broadcast
 (rank 0 reads the file with `std::fs::read`, parses the dataset and
 broadcasts the bytes); writes gather every rank's shard to rank 0, which
 writes the merged dataset. It does not call `MPI_File_read_at_all` or any
 other MPI-IO routine. Collective I/O is on the [roadmap](../../ROADMAP.md).
 `clawhdf5-bench`'s `mpi_io_bench` binary exercises it.
 ## Features
- Memory-mapped file access (`mmap` feature)
+| Feature | Default | What | Builds C |
- Async I/O via Tokio (`async` feature)
+|---|---|---|---|
- HSDS remote access (`hsds` feature)
+| `mmap` | no (the `clawhdf5` facade turns it on) | `MmapReader`, `MmapReadWrite` | no |
- Prefetching and sweep optimizations
+| `async` | no | `async_read` (tokio) | no |
-
+| `hsds` | no | `hsds` (reqwest, and `async`) | yes: reqwest's default TLS is native-tls (OpenSSL on Linux) |
-## Usage
+| `mpi-io` | no | a real `MpiVol` (without it `MpiVol::new_world` returns an error) | yes: `mpi-sys` needs an MPI installation and libclang |
 ```rust
 use clawhdf5_io::MmapReader;
 let reader = MmapReader::open("data.h5").unwrap();
 ```
 ## License
@@ -1,11 +1,8 @@
 # clawhdf5-migrate
 [![crates.io](https://img.shields.io/crates/v/clawhdf5-migrate.svg)](https://crates.io/crates/clawhdf5-migrate)
 [![docs.rs](https://img.shields.io/docsrs/clawhdf5-migrate)](https://docs.rs/clawhdf5-migrate)
 CLI tool to migrate a SQLite agent-memory database in the `memory_chunks` / `sessions` / `entities` / `relations` layout (table and
 column names are configurable) to a
-[clawhdf5-agent](https://crates.io/crates/clawhdf5-agent) store. This is **not**
+[clawhdf5-agent](../clawhdf5-agent/README.md) store. This is **not**
 ZeroClaw's schema — ZeroClaw keeps memories in a single `memories` table and
 does not use clawhdf5.
@@ -15,10 +12,15 @@ the knowledge graph (entities and relations) are carried over.
 ## Installation
 Not on crates.io yet; install from a checkout:
 ```bash
-cargo install clawhdf5-migrate
+cargo install --path crates/clawhdf5-migrate
 ```
 It builds C: `rusqlite` is built with its `bundled` feature, which compiles
 SQLite (so no system libsqlite is needed, but a C compiler is).
 ## Usage
 ```bash
@@ -0,0 +1,38 @@
 # clawhdf5-napi
 > **Status: does not work end to end.** The TypeScript package built on
 > this crate (`packages/clawhdf5-node`) has never run successfully, is
 > unpublished, and is not built or tested in CI. See "The Node.js package
 > does not work" in [`docs/known-issues.md`](../../docs/known-issues.md).
 > Fix it and add CI, or remove it, before depending on it.
 A Node.js native addon ([napi-rs](https://napi.rs), N-API 9) exposing
 [`clawhdf5-agent`](../clawhdf5-agent/README.md) as a `ClawhdfMemory`
 class. It wraps `clawhdf5_agent::openclaw::ClawhdfBackend` (the agent's
 `search` with re-ranking and confidence on) and the consolidation engine.
 It was written for an OpenClaw integration that is not being pursued
 ([`docs/openclaw.md`](../../docs/openclaw.md)).
 ## What the addon exposes
 `ClawhdfMemory.create(path, dim)`, `.open(path)`, `.openOrCreate(path,
 dim)`, and on an instance: `search`, `get`, `write`, `ingestMarkdown`,
 `exportMarkdown`, `save`, `saveBatch`, `stats`, `compact`, `tickSession`,
 `flushWal`, `walPendingCount`, `runConsolidation`, and the ephemeral tier
 (`enableEphemeral`, `ephemeralSet`/`Get`/`Delete`, `ephemeralStats`,
 `promoteEphemeral`). napi-rs converts names and `#[napi(object)]` fields to
 camelCase.
 ## Build
 ```bash
 cargo build --release -p clawhdf5-napi     # the Rust cdylib
 # the .node package: npm install -g @napi-rs/cli; cd packages/clawhdf5-node; napi build --platform --release
 ```
 It links against Node's N-API through `napi-sys` (a `-sys` crate), so it is
 exempt from CI's "no C in the default build" check.
 ## License
 MIT
@@ -1,25 +1,53 @@
 # clawhdf5-netcdf4
-[![crates.io](https://img.shields.io/crates/v/clawhdf5-netcdf4.svg)](https://crates.io/crates/clawhdf5-netcdf4)
+Read NetCDF-4 files in pure Rust. NetCDF-4 files are HDF5 files with
-[![docs.rs](https://docs.rs/clawhdf5-netcdf4/badge.svg)](https://docs.rs/clawhdf5-netcdf4)
+conventions for dimensions, coordinate variables and attributes; this crate
 reads them through the [`clawhdf5`](../clawhdf5/README.md) facade, with no
 libnetcdf or libhdf5. Read-only: NetCDF-3 (classic) files are not HDF5 and
 are not supported.
-NetCDF-4 read support built on clawhdf5 — pure Rust, no C dependencies.
+Not on crates.io yet; depend on it from git:
-## Features
+```toml
-
+[dependencies]
- Read NetCDF-4 / HDF5-backed `.nc` files
+clawhdf5-netcdf4 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
- Dimension, variable, and CF convention support
+```
 - Climate and scientific data access
 ## Usage
-```rust
+```rust,no_run
 use clawhdf5_netcdf4::NetCDF4File;
-let nc = NetCDF4File::open("climate.nc").unwrap();
+let nc = NetCDF4File::open("climate.nc")?;
-let temp = nc.variable("temperature").unwrap();
+for dim in nc.dimensions()? {
    println!("{}: {} (unlimited = {})", dim.name, dim.size, dim.is_unlimited);
 }
 let mut temp = nc.variable("temperature")?;
 let dims: Vec<&str> = temp.dimensions().iter().map(|d| d.name.as_str()).collect();
 println!("{:?} over {:?}", temp.shape()?, dims);
 let cf = temp.cf_attributes()?;
 println!("units: {:?}", cf.units);
 // scale_factor/add_offset applied; _FillValue and missing_value become NaN
 let values: Vec<f64> = temp.read_f64()?;
 # Ok::<(), clawhdf5_netcdf4::Error>(())
 ```
 ## API
 | Item | What |
 |---|---|
 | `NetCDF4File` | `open`, `from_bytes`, `dimensions`, `variables`, `variable`, `global_attrs`, `group`, `group_names`, `nc_properties`, and `hdf5_file` for the underlying `clawhdf5::File` |
 | `NetCDF4Group` | the same for a sub-group (`dimensions`, `variables`, `attrs`, nested `group`) |
 | `Variable` | `name`, `shape`, `dimensions`, `nc_type`, `is_coordinate`, `attrs`, `cf_attributes`; `read_f64` (CF scale/offset and fill applied), `read_raw_f32`/`_f64`/`_i32`/`_i64`/`_u64`, `read_string`, `read_raw` |
 | `Dimension` | `name`, `size`, `is_unlimited` |
 | `CfAttributes` | CF convention attributes: `units`, `long_name`, `standard_name`, `fill_value` (`_FillValue`), `missing_value`, `scale_factor`, `add_offset`, `valid_range`, `calendar`, `axis` |
 | `NcType` | the NetCDF type of a variable |
 No cargo features. Tests compare against files written by netCDF4-python
 (`tests/interop_tests.rs`; the CI job requires them with
 `CLAWHDF5_REQUIRE_INTEROP=1`). What the HDF5 reader underneath cannot
 read is listed in [`docs/known-issues.md`](../../docs/known-issues.md).
 ## License
 MIT
@@ -1,8 +1,5 @@
 # clawhdf5-py
 [![crates.io](https://img.shields.io/crates/v/clawhdf5-py.svg)](https://crates.io/crates/clawhdf5-py)
 [![docs.rs](https://docs.rs/clawhdf5-py/badge.svg)](https://docs.rs/clawhdf5-py)
 Python bindings for clawhdf5 — a pure-Rust HDF5 library. The package is
 `clawhdf5` (`import clawhdf5`); it needs numpy and no libhdf5.
@@ -61,6 +58,8 @@ with clawhdf5.File("data.h5", "r") as f:
  `clawhdf5.InternalError`, a `RuntimeError`.
 - Attributes return what h5py returns; `clawhdf5.Empty` stands for a null
  dataspace (h5py's `Empty`).
 - Also as in h5py: `File.mode` (`'r'`, or `'r+'` for a writable file), `File.flush()` (a no-op:
  edits are already synced), `Dataset.chunks`.
 ## Remote files
@@ -114,3 +114,34 @@ let s = storage.stats(); // requests, bytes_fetched, hits, misses, cached_bytes,
 directory with range support (the server the tests use), and
 `cargo run -p clawhdf5-remote --example read_url -- URL [DATASET]` lists a
 file and prints what it cost.
 ## Other front ends
 - `h5rs` (built with `--features remote`, or `remote-https`) takes URLs as
  FILE arguments: [`clawhdf5-tools`](../clawhdf5-tools/README.md).
 - Python: `clawhdf5.File("http://…")` and `File.open_url(url, ...)` go
  through this crate: [`clawhdf5-py`](../clawhdf5-py/README.md).
 - The browser does **not** use this crate (its cache fetches by blocking);
  `clawhdf5-wasm`'s `openUrl` has its own restartable cache:
  [`examples/wasm-viewer`](../../examples/wasm-viewer/README.md).
 ## Limits
 Files a SWMR writer is still appending to cannot be followed remotely
 (the file is pinned at open, so growth is `RemoteError::FileChanged`); the
 block size is fixed rather than taken from a paged file's page size; the
 cloud backends are built and unit-tested but have not been run against a
 real bucket. The full list is under "Remote files (`clawhdf5-remote`)
 limits" in [`docs/known-issues.md`](../../docs/known-issues.md); the design
 is milestone M3 of [`docs/design/range-reads.md`](../../docs/design/range-reads.md).
 Not on crates.io yet; depend on it from git:
 ```toml
 [dependencies]
 clawhdf5-remote = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 ```
 ## License
 MIT
@@ -306,4 +306,15 @@ CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 cargo test -p clawhd
 The interop tests write their files with h5py and compare with h5ls, h5stat,
 h5dump and h5diff; each skips when what it needs is missing unless
-`CLAWHDF5_REQUIRE_INTEROP=1`.
+`CLAWHDF5_REQUIRE_INTEROP=1`. `tests/remote.rs` runs every subcommand on
 URLs against a local range server.
 This crate also holds the interop tests of the library's in-place editor
 (`clawhdf5::FileEditor`), since they use `h5rs check` and h5dump on every
 edited file and compare index and heap structures with what libhdf5 makes
 of the same edits:
 ```bash
 CLAWHDF5_PYTHON=.venv/bin/python CLAWHDF5_REQUIRE_INTEROP=1 \
  cargo test -p clawhdf5-tools --test edit_interop --test edit_coverage_interop
 ```
@@ -0,0 +1,51 @@
 # clawhdf5-wasm
 clawhdf5's HDF5 and NetCDF-4 reader compiled to WebAssembly with
 wasm-bindgen, for the browser (and Node). Read-only. Two ways in:
 - `open(bytes)` — a file already in memory (a dropped file, a fetched
  blob);
 - `openUrl(url, opts)` — a file on a web server, read by HTTP range
  requests as each call needs its bytes, without downloading it
  (range-read milestone M4, [`docs/design/range-reads.md`](../../docs/design/range-reads.md)).
 Both give `list`, `info`, `attrs`, `read` and `readHyperslab`; the remote
 file's methods return promises, and `stats()` counts requests and bytes.
 The JavaScript API, options, limits, package size and tests are documented
 with the demo page, [`examples/wasm-viewer/README.md`](../../examples/wasm-viewer/README.md).
 ## Layout
 - `src/core.rs` — the reader over any `clawhdf5_format::storage::Storage`
  (`Reader::open_storage`), plain Rust and tested natively.
 - `src/lazy.rs` — the restartable "NeedBytes" cache behind `openUrl`: a
  call runs as a pass over the blocks fetched so far; a pass that misses is
  abandoned, the missing (and hinted) blocks are fetched, and the pass is
  run again. No block is evicted while a call runs.
 - `js/remote.js` — the HTTP side: `fetch` with `Range`, checking every
  answer (a `206` with exactly the bytes asked for, same ETag/Last-Modified
  and length) so a call fails rather than return another file's bytes.
 - `src/lib.rs` — the wasm-bindgen exports.
 ## Build and test
 ```bash
 rustup target add wasm32-unknown-unknown
 cargo install wasm-bindgen-cli --version 0.2.129   # must equal the crate's wasm-bindgen
 bash examples/wasm-viewer/build.sh                  # -> examples/wasm-viewer/pkg/
 cargo test -p clawhdf5-wasm                         # native: h5py_interop, lazy, vl_strings
 bash examples/wasm-viewer/test/run.sh               # Node + headless Chromium (not in CI)
 ```
 `CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus cargo test -p
 clawhdf5-wasm --test lazy` compares every corpus file read lazily with the
 same file read from bytes.
 Built without `mmap` and `parallel` and without the Zstd and SZIP filters
 (they link C): such datasets fail with `unsupported filter`. No C is
 compiled; `publish = false` (it is distributed as the package
 `build.sh` makes).
 ## License
 MIT
@@ -1,27 +1,101 @@
 # clawhdf5
-[![crates.io](https://img.shields.io/crates/v/clawhdf5.svg)](https://crates.io/crates/clawhdf5)
+The main crate: a pure-Rust HDF5 reader, writer and in-place editor, with no
-[![docs.rs](https://docs.rs/clawhdf5/badge.svg)](https://docs.rs/clawhdf5)
+libhdf5 and, by default, no C code. It wraps
 [`clawhdf5-format`](../clawhdf5-format/README.md) (the binary format) and
 [`clawhdf5-io`](../clawhdf5-io/README.md) (memory-mapped reads) in an
 h5py-like API.
-Pure-Rust HDF5 reader/writer — no C dependencies.
+Not on crates.io yet; depend on it from git:
 ```toml
 [dependencies]
 clawhdf5 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
 ```
 ## Main types
 | Type | What it does |
 |---|---|
 | `File` | Opens a file (`open`, `open_buffered`, `from_bytes`, `open_storage` for any `Storage`), walks groups (`root`, `group`, `dataset`), lists `datasets`/`groups`/`attrs`. `File` is `Send + Sync`: several threads can read one open file. |
 | `Dataset` | `shape`, `dtype`, `max_dimensions`, `attrs`; reads `read_f64`/`read_f32`/`read_i32`/`read_i64`/`read_u64`, strings (`read_string`, `read_string_bytes`), variable-length data (`read_vlen`), hyperslabs and point selections (`read_selection`, `read_f64_selection`, ...), zero-copy views of contiguous data (`read_f64_zerocopy`, ...), `verify_provenance`. |
 | `FileBuilder` | Writes a new file: datasets of every numeric type, strings, compounds (`CompoundTypeBuilder`), enums, chunked and compressed layouts (deflate, shuffle, Fletcher-32, LZF, and with features LZ4, Zstd, bitshuffle, bzip2, Blosc, pcodec), nested groups, soft/hard/external links, virtual datasets, attribute creation order. Files open in h5py and h5dump. |
 | `FileEditor` | Changes an existing file in place without rewriting it: `write_values`/`write_selection`/`write_all`, `resize` of chunked datasets (every chunk index), `set_attr` (compact and dense storage). Anything it cannot do safely is `Error::Unsupported` before any write. |
 | `MmapFile`, `LazyFile` | Alternative readers: memory-mapped, and one that reads lazily and caches. |
 | `File::open_swmr` | Reads a file a libhdf5 SWMR writer is still appending to (`Dataset::refresh`, bounded retries), as h5py's `swmr=True` reader does. |
 ## Examples
 ```rust,no_run
 use clawhdf5::{AttrValue, File, FileBuilder, FileEditor, Selection};
 // Write
 let mut b = FileBuilder::new();
 b.create_dataset("sensors/temperature")
    .with_f64_data(&[20.5, 21.0, 21.5, 22.0])
    .with_shape(&[4])
    .with_maxshape(&[u64::MAX]) // unlimited, so it can grow
    .with_chunks(&[2])
    .with_deflate(4);
 b.set_attr("version", AttrValue::I64(1));
 b.write("data.h5")?;
 // Read
 let file = File::open("data.h5")?;
 let ds = file.dataset("sensors/temperature")?;
 assert_eq!(ds.shape()?, vec![4]);
 let values = ds.read_f64()?;
 // Edit in place: grow the dataset and fill the new tail
 let mut ed = FileEditor::open("data.h5")?;
 ed.resize("sensors/temperature", &[6])?;
 let tail = Selection::Hyperslab {
    start: vec![4],
    stride: vec![1],
    count: vec![2],
    block: vec![1],
 };
 ed.write_values("sensors/temperature", &tail, &[22.5f64, 23.0])?;
 # Ok::<(), clawhdf5::Error>(())
 ```
 Remote files (HTTP range requests, S3/GCS/Azure) are read through
 `File::open_storage`; [`clawhdf5-remote`](../clawhdf5-remote/README.md)
 provides the storage and its block cache.
 ## Features
- Read and write HDF5 files entirely in Rust
+| Feature | Default | What | Builds C |
- Memory-mapped I/O for large files (`mmap` feature, enabled by default)
+|---|---|---|---|
- Parallel chunk reads via Rayon (`parallel` feature)
+| `mmap` | yes | memory-mapped reads (`File::open` maps the file; `MmapFile`) | no |
- Lazy dataset access for minimal memory usage
+| `provenance` | yes | SHA-256 `_provenance_sha256` attributes (`DatasetBuilder::with_provenance`, `Dataset::verify_provenance`) | no |
- h5py-compatible file output
+| `lzf` | yes | LZF filter (32000), h5py's `compression="lzf"` | no |
 | `parallel` | no | chunk decoding on a rayon pool | no |
 | `lz4` | no | LZ4 filter (32004) | no |
 | `pcodec` | no | pcodec filter | no |
 | `bitshuffle`, `bzip2`, `blosc` | no | plugin filters 32008, 307, 32001 (read and write) | no (bzip2 uses the pure-Rust `libbz2-rs-sys`) |
 | `blosc2`, `zfp` | no | plugin filters 32026 and 32013, **read only** | no |
 | `plugin-filters` | no | `lzf`, `bitshuffle`, `bzip2`, `blosc`, `blosc2`, `zfp` | no |
 | `zstd` | no | Zstandard filter (32015) | yes (libzstd) |
 | `fast-deflate` | no | zlib-ng instead of the pure-Rust zlib-rs | yes (cmake) |
 | `blake3_hash` | no | `provenance::blake3_hash` helpers | yes (`cc`, for blake3's SIMD code) |
 | `apple-compression` | no | currently has no effect in this crate (it is not forwarded) | — |
-## Usage
+SZIP decoding is a `clawhdf5-format` feature (`szip`, links the system
 libaec); the facade does not forward it.
-```rust
+## Limits and further reading
 use clawhdf5::File;
-let file = File::open("data.h5").unwrap();
+- What is known not to work, and what was wrong in earlier releases:
-let dataset = file.dataset("/group/data").unwrap();
+  [`docs/known-issues.md`](../../docs/known-issues.md) (editor limits, range
-let values: Vec<f64> = dataset.read_1d().unwrap();
+  reads, external links and external raw data, which are explicit errors).
-```
+- Read coverage against libhdf5/h5py on eight public corpora:
  [`CONFORMANCE.md`](../../CONFORMANCE.md).
 - Read and write speed against libhdf5 and h5py:
  [`BENCHMARKS.md`](../../BENCHMARKS.md).
 - Range reads and SWMR design: [`docs/design/range-reads.md`](../../docs/design/range-reads.md),
  [`docs/design/swmr.md`](../../docs/design/swmr.md).
 - Changes: [`CHANGELOG.md`](../../CHANGELOG.md).
 ## License