docs: CLAUDE.md regrouped into rules, invariants and workflows
Standing rules (no C by default, h5py must read what we write, one float16 implementation, claims need evidence, OpenClaw/ZeroClaw withdrawn, the known-issues rule) are gathered in one place; library and agent-memory invariants are split; the CI section lists what ci-test.sh and the conformance workflow run now instead of a dated "two jobs, green" line. Adds the conformance, remote, wasm, Python and benchmark workflows (idle load below 2, dated records), and warns that scripts/run-benchmarks.sh is stale and overwrites BENCHMARKS.md. Crate roles corrected (clawhdf5-io is I/O adapters; codecs live in clawhdf5-format). Benchmark figures now live in BENCHMARKS.md only. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
@@ -1,253 +1,266 @@
|
||||
# clawhdf5
|
||||
|
||||
## Purpose
|
||||
Pure-Rust HDF5 format implementation with HNSW vector search, WAL-backed persistence, agent memory storage, and GPU-accelerated vector search. A standalone library. Its one verified consumer is ClawBrainHub (`.brain` files); no agent framework integrates it (OpenClaw and ZeroClaw claims were withdrawn on 2026-09-25 — neither was ever true).
|
||||
Pure-Rust HDF5 implementation (read, write, in-place edit, remote and browser
|
||||
reads) plus agent memory on top of it: HNSW vector search, a WAL-backed store,
|
||||
and GPU vector distances. A standalone library. Its one verified consumer is
|
||||
ClawBrainHub (`.brain` files); no agent framework integrates it (see
|
||||
*Standing rules*).
|
||||
|
||||
## Architecture
|
||||
|
||||
Cargo workspace with 19 crates under `crates/` (plus `libaec-sys`, an internal FFI bindings crate for the optional `szip` feature):
|
||||
Cargo workspace, 19 crates under `crates/` (plus `libaec-sys`, the FFI crate
|
||||
behind the optional `szip` feature). MSRV 1.92 (`rust-version`, checked in CI).
|
||||
|
||||
| Crate | Role |
|
||||
|-------|------|
|
||||
| `clawhdf5-format` | HDF5 binary spec parser (superblock, B-tree, heap) — also holds shared type definitions and physical constants |
|
||||
| `clawhdf5-io` | Read/write implementation |
|
||||
| `clawhdf5-filters` | Deflate backends (zlib-rs, zlib-ng, Apple Compression); the HDF5 filter pipeline, the filter registry (`clawhdf5_format::filter_registry`) and the other codecs (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec, and the pure-Rust plugin filters LZF, bitshuffle, bzip2, Blosc 1, and Blosc2 and ZFP read-only) live in `clawhdf5-format`. |
|
||||
| `clawhdf5-derive` | Proc-macro derive for HDF5-serializable structs |
|
||||
| `clawhdf5` | Main facade crate |
|
||||
| `clawhdf5-netcdf4` | NetCDF-4 compatibility layer |
|
||||
| `clawhdf5-ann` | HNSW approximate nearest-neighbor vector index |
|
||||
| `clawhdf5-agent` | Agent memory, session history, knowledge graph storage |
|
||||
| `clawhdf5-gpu` | GPU vector distance computation via wgpu (hand-written WGSL compute shaders) — not dataset I/O |
|
||||
| `clawhdf5-accel` | CPU SIMD acceleration path |
|
||||
| `clawhdf5-migrate` | SQLite → HDF5 agent-memory migration |
|
||||
| `clawhdf5-format` | The HDF5 format: parsers and writer (superblock, headers, B-trees, heaps, chunk indexes), the `Storage` trait, the filter pipeline and registry (`filter_registry`), every codec except deflate (LZ4, Zstd, SZIP, N-Bit, scale-offset, pcodec; pure-Rust LZF, bitshuffle, bzip2, Blosc 1; Blosc2 and ZFP read-only), `float16`, `checksum` |
|
||||
| `clawhdf5-filters` | Deflate backends (zlib-rs default, zlib-ng, Apple Compression) |
|
||||
| `clawhdf5-io` | I/O adapters (buffers, mmap, prefetch) |
|
||||
| `clawhdf5-derive` | `#[derive(H5Type)]` for compound types |
|
||||
| `clawhdf5` | Facade: `File`, `FileBuilder`, `Dataset`, `FileEditor` (`src/edit/`), SWMR reading (`src/swmr.rs`) |
|
||||
| `clawhdf5-netcdf4` | NetCDF-4 read support |
|
||||
| `clawhdf5-remote` | `open_url`: HTTP(S) range requests and object stores (S3, GCS, Azure) through `BlockCache` |
|
||||
| `clawhdf5-tools` | `h5rs`: `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` |
|
||||
| `clawhdf5-py` | PyO3 bindings (h5py-like API, remote files, `'r+'` editing) |
|
||||
| `clawhdf5-wasm` | wasm-bindgen browser reader (`open(bytes)`, `openUrl(url)`); demo in `examples/wasm-viewer/` |
|
||||
| `clawhdf5-ann` | HNSW index |
|
||||
| `clawhdf5-agent` | Agent memory store (`HDF5Memory`), sessions, knowledge graph, BM25 |
|
||||
| `clawhdf5-accel` | CPU SIMD kernels (AVX2, NEON) |
|
||||
| `clawhdf5-gpu` | wgpu vector distances (WGSL) — not dataset I/O; HDF5 I/O is CPU-only |
|
||||
| `clawhdf5-migrate` | SQLite → agent store migration |
|
||||
| `clawhdf5-cli` | Agent-memory CLI |
|
||||
| `clawhdf5-napi` | Node.js addon (the `packages/clawhdf5-node` wrapper is broken; `docs/known-issues.md`) |
|
||||
| `clawhdf5-android` | Android JNI bindings |
|
||||
| `clawhdf5-cli` | Command-line interface (agent memory) |
|
||||
| `clawhdf5-tools` | `h5rs`: pure-Rust HDF5 tools — `ls`, `dump` (DDL / hdf5-json), `stat`, `diff`, `check` (structural + checksum validator) |
|
||||
| `clawhdf5-napi` | Node.js native addon bindings |
|
||||
| `clawhdf5-py` | PyO3 Python bindings |
|
||||
| `clawhdf5-wasm` | WebAssembly (wasm-bindgen) reader for the browser; demo in `examples/wasm-viewer/` |
|
||||
| `clawhdf5-remote` | Remote files: `open_url` over HTTP(S) range requests and object stores (`object_store`: S3, GCS, Azure) through a mandatory block cache (`BlockCache`) |
|
||||
| `clawhdf5-bench` | Benchmark suite |
|
||||
| `clawhdf5-bench` | Benchmarks and harnesses (`search_harness`, `read_harness`, `concurrent_read`, `longmemeval_bench`, …) |
|
||||
|
||||
## Key Features
|
||||
- Zero-C-dependency HDF5 read/write: no libhdf5, and deflate defaults to
|
||||
pure-Rust zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake).
|
||||
`ci-test.sh` fails if a C-building crate enters the core crates' default
|
||||
tree. flate2 must keep `runtime_detection` with zlib-rs — without it zlib-rs
|
||||
loses SIMD and inflates 3.5x slower. MSRV is 1.92 (`rust-version`, checked
|
||||
in CI).
|
||||
- HNSW vector index for semantic similarity search over agent memories — the
|
||||
`clawhdf5-agent` `hnsw` feature is **on by default**, so `hybrid_search` uses
|
||||
the approximate `clawhdf5-ann` index for the vector stage (the index mirrors
|
||||
the cache and self-heals on drift). Build the agent with
|
||||
`--no-default-features --features float16` to force the exact linear cosine scan.
|
||||
The agent's `parallel` feature (also default) builds the index on a thread
|
||||
pool; the graph is identical with or without it.
|
||||
The index uses the HNSW paper's diversity heuristic for neighbour selection
|
||||
(plain closest-M capped recall on clustered data: 0.31 recall@10 at 100K). Its
|
||||
graph is saved to `<store>.h5.ann` at each checkpoint and reloaded by `open()`
|
||||
(tied to the checkpoint by a generation id; stale/damaged sidecars are
|
||||
ignored and the index rebuilt). `MemoryConfig::quantized_index` (**on by
|
||||
default** for new stores, persisted; stores predating the setting load as
|
||||
`false` and keep their f32 index — guarded by
|
||||
`tests/fixtures/store_v2_5_0.h5`; CLI opt-out is `create --f32-index`)
|
||||
stores the index's own copy of the embeddings as `i8`,
|
||||
which roughly halves a loaded store's memory (2.72x -> 1.74x the raw vectors
|
||||
at 100K); because quantised distances are approximate and `ef` cannot
|
||||
compensate, the query path then re-scores the candidate pool against the
|
||||
exact embeddings, which holds recall at the f32 index's level. It is also
|
||||
faster at equal recall: 1.63x the QPS on x86-64 (AVX2) and 1.18x on a
|
||||
Raspberry Pi 5 (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm since
|
||||
the intrinsic is unstable; plain NEON on pre-dotprod cores). The aarch64
|
||||
code is `cfg`'d out on x86, so x86 CI never compiles or lints it — test it
|
||||
on real ARM (`rpivision02`, 10.0.2.3, is a Pi 5). `hybrid_search` keeps one incremental BM25
|
||||
index for the life of the store and never writes the store: Hebbian
|
||||
activation boosts are persisted by the next checkpoint (or on drop), not per
|
||||
query. Measure any search-path change with
|
||||
`cargo run --release -p clawhdf5-bench --bin search_harness` (baselines in
|
||||
`BENCHMARKS.md`).
|
||||
- WAL (write-ahead log) for crash-safe persistence, with a chained CRC32
|
||||
trailer per entry (each entry's CRC folds in the previous entry's CRC) so a
|
||||
corrupted, reordered, duplicated, or spliced entry stops replay cleanly
|
||||
instead of loading bad or tampered data. The pre-chaining per-entry-CRC
|
||||
format (v2) is still fully readable; the oldest no-CRC format (v1) is only
|
||||
reachable through the one-time migration path in `HDF5Memory::open`, not
|
||||
through the public `WalFile::read_entries`.
|
||||
**What the WAL guarantees:** integrity, ordering, and recovery from a
|
||||
*process* crash at any point — including between a checkpoint and the WAL
|
||||
truncate (each checkpoint records a `WalMark` in `/meta`, and `open()` skips
|
||||
the WAL prefix the `.h5` already contains, so entries are never applied
|
||||
twice). Checkpoints and snapshots are made durable as a unit (temp file
|
||||
synced, renamed, directory synced). **What it does not guarantee:**
|
||||
individual WAL appends are *not* fsynced (a deliberate latency trade-off), so
|
||||
saves made since the last checkpoint can be lost on power failure or kernel
|
||||
panic. Current header version is 4 (adds the `Update` record used by
|
||||
`save_or_update`); v3 files are read and upgraded in place.
|
||||
- A store has a **single writer**: `HDF5Memory::create`/`open` hold an exclusive
|
||||
advisory lock on `<store>.h5.lock` and a second opener gets
|
||||
`MemoryError::Locked`. Use `HDF5Memory::open_read_only` for a lock-free,
|
||||
never-writing point-in-time view (the CLI's `recall`/`stats`/`agents-md`/
|
||||
`export` do). An unreadable WAL (torn header, bad magic) is quarantined to
|
||||
`<store>.h5.wal.corrupt-<ts>` rather than blocking `open()`; a WAL with an
|
||||
unknown *newer* version still fails and is left untouched.
|
||||
- `MemoryConfig::float16` (**on by default** for new stores, persisted;
|
||||
existing stores keep their recorded `false` — guarded by the v2.5.0
|
||||
fixture in `tests/float16_store.rs`; CLI opt-out is `create --f32`) writes
|
||||
`/memory/embeddings` as IEEE half precision (48% smaller file at 100K;
|
||||
LongMemEval with real MiniLM embeddings identical to f32).
|
||||
`MemoryCache::half_precision` rounds each embedding as it enters the cache (push, update, WAL replay, and on load of a store still
|
||||
`f32` on disk), so memory and file agree bit for bit; the conversions live
|
||||
in `clawhdf5_format::float16` and must stay the single implementation.
|
||||
Values beyond ±65504 are `MemoryError::InvalidEntry`. Interop: every file
|
||||
must open in h5py — `f32` datasets and empty datasets did not until
|
||||
2026-09-23 (see `docs/known-issues.md`); the agent's `h5py_interop` test
|
||||
guards a whole store.
|
||||
- `HDF5Memory::search(query_emb, text, &SearchOptions)` is the full search
|
||||
path: optional source-channel filter (applied before ranking; exact scan of
|
||||
the allowed records whenever cheaper than `pool × M` index distance
|
||||
Reference docs: `docs/known-issues.md` (open issues table first — check it
|
||||
before calling something a bug or a feature), `BENCHMARKS.md` (headline
|
||||
numbers first), `CONFORMANCE.md` (generated), `docs/design/range-reads.md`
|
||||
and `docs/design/swmr.md`, `CHANGELOG.md` (full detail of every fix).
|
||||
|
||||
## Standing rules
|
||||
|
||||
- **No C in the default build.** No libhdf5; deflate defaults to pure-Rust
|
||||
zlib-rs (`fast-deflate` opts into zlib-ng, which needs cmake). `ci-test.sh`
|
||||
fails if a C-building crate enters the core crates' default tree. Zstd,
|
||||
SZIP, `https` (ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are opt-in. flate2
|
||||
must keep `runtime_detection` with zlib-rs — without it zlib-rs loses SIMD
|
||||
and inflates 3.5x slower.
|
||||
- **Every file we write must open in h5py/libhdf5.** Interop tests compare
|
||||
against h5py and h5dump; `f32` and empty datasets did not open until
|
||||
2026-09-23.
|
||||
- **float16 has one implementation:** `clawhdf5_format::float16`.
|
||||
- **Claims need evidence.** Performance and integration claims in docs must
|
||||
be measured, dated (with machine and command), or withdrawn. Benchmark
|
||||
numbers are dated records: never edit a measured value, add a new dated
|
||||
section and mark the old one superseded.
|
||||
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not and
|
||||
never was an OpenClaw memory plugin; the old `memory.backend = "clawhdf5"`
|
||||
config was never valid. `docs/openclaw.md` records what a real plugin would
|
||||
need. The `openclaw` module's `ClawhdfBackend` is just `search` with
|
||||
re-rank + confidence on.
|
||||
- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream
|
||||
v0.8.5 and the `osobh/zeroclaw` fork and their history): its memory
|
||||
backends are its own; `clawhdf5-migrate`'s default SQLite layout is not
|
||||
ZeroClaw's schema. Don't reintroduce integration claims without an
|
||||
integration and a test against the real consumer.
|
||||
- **known-issues.md:** one entry per bug; when fixed, record it in
|
||||
`CHANGELOG.md` and move the entry to *Fixed (history)* with date, PR,
|
||||
affected releases and what users must do — never delete it.
|
||||
|
||||
## HDF5 library: invariants and gotchas
|
||||
|
||||
- **Remote/range reads** (`docs/design/range-reads.md`, M0-M5 merged in PRs
|
||||
#17-#21): every format-crate read path goes through `Storage`
|
||||
(`read_at`/`read_ranges`/`hint`). `File::open_storage` takes any
|
||||
`Storage`; `clawhdf5_remote::open_url` wraps HTTP (`HttpStorage`, ureq) or
|
||||
`ObjectStoreStorage` in `BlockCache` (1 MiB blocks, LRU budget, in-flight
|
||||
dedup, coalesced runs). Remote files are pinned by ETag/Last-Modified and
|
||||
length (`RemoteError::FileChanged`). Zero-copy APIs and `File::as_bytes`
|
||||
need an in-memory file. Parse through `File::storage()` and the `*_in`
|
||||
functions, not `as_bytes`, in new code (the Python bindings do).
|
||||
`ObjectStoreStorage` runs reads on its own small tokio runtime, so it
|
||||
works from any thread.
|
||||
- **SWMR** (`docs/design/swmr.md`): `File::open_swmr` reads a file a libhdf5
|
||||
SWMR writer is appending to — positioned reads, no chunk cache, bounded
|
||||
retries (100), `Dataset::refresh()`. clawhdf5 has no SWMR writer; remote
|
||||
SWMR is out of scope.
|
||||
- **Browser** (`clawhdf5-wasm`, read-only, no Zstd/SZIP): `openUrl` reads
|
||||
through the restartable "NeedBytes" cache (`src/lazy.rs`: a call is re-run
|
||||
after each wave of misses; no block is evicted while a call runs); the HTTP
|
||||
is JavaScript (`js/remote.js`).
|
||||
- **In-place editing** (`clawhdf5::FileEditor`): overwrites values, grows and
|
||||
shrinks chunked datasets (every chunk index) and sets attributes (compact
|
||||
and dense) without rewriting the file, changing indexes and heaps as
|
||||
libhdf5 does; freed space is reused within one editor. Anything it cannot
|
||||
do safely is `Error::Unsupported` before any write (limits in
|
||||
`docs/known-issues.md`). The algorithms follow libhdf5 `hdf5_1_14_6`
|
||||
(github.com/HDFGroup/hdf5). Test changes with `cargo test -p
|
||||
clawhdf5-tools --test edit_interop --test edit_coverage_interop`.
|
||||
- **Provenance:** `Dataset::verify_provenance()` (facade `provenance`
|
||||
feature, default) re-hashes a dataset against its `_provenance_sha256`
|
||||
attribute (`DatasetBuilder::with_provenance`). Opt-in per call; unkeyed
|
||||
hash — tamper-evident, not tamper-proof.
|
||||
|
||||
## Agent memory: invariants and gotchas
|
||||
|
||||
- **Search.** `HDF5Memory::search(query_emb, text, &SearchOptions)` is the
|
||||
full path: optional source-channel filter (before ranking; exact scan of
|
||||
the allowed records when cheaper than `pool × M` index distance
|
||||
evaluations, and as the fallback when the pool comes back short), fusion,
|
||||
activation scaling, optional re-ranking and confidence rejection.
|
||||
`hybrid_search`/`hybrid_search_with` are thin wrappers; `ClawhdfBackend`
|
||||
(the `openclaw` module) is `search` with re-rank + confidence on.
|
||||
- **OpenClaw is not supported** (decided 2026-09-25): clawhdf5 is not an
|
||||
OpenClaw memory plugin and never was — the old `memory.backend = "clawhdf5"`
|
||||
config was never valid. Don't reintroduce OpenClaw claims; `docs/openclaw.md`
|
||||
records what a real plugin would need.
|
||||
- **ZeroClaw does not use clawhdf5** (checked 2026-09-25 against upstream
|
||||
v0.8.5 and the `osobh/zeroclaw` fork, and their full history): no
|
||||
`clawhdf5` feature or backend exists; ZeroClaw's memory backends are
|
||||
sqlite/lucid/postgres/qdrant/markdown/none behind its own `Memory` trait.
|
||||
`clawhdf5-migrate`'s default SQLite layout (`memory_chunks`, `sessions`,
|
||||
`entities`, `relations`) is not ZeroClaw's schema either (ZeroClaw's is a
|
||||
`memories` table). Don't reintroduce integration claims without an
|
||||
integration and a test against the real consumer. Measure changes with
|
||||
`search_harness --options-study`.
|
||||
- `MemoryConfig::compression` is off by default; when on, embeddings are
|
||||
deflate-compressed, or Zstd with the agent's `zstd` feature (links libzstd).
|
||||
- Signed checkpoints (`clawhdf5-agent` `signing` module): with
|
||||
`HDF5Memory::set_signing_key` every checkpoint stores an Ed25519-signed
|
||||
manifest (SHA-256 per record in a Merkle tree + settings/sessions/graph
|
||||
hashes; per-record hashes in `/integrity/record_hashes`);
|
||||
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover exactly
|
||||
what the file persists in the form the loader returns it (strings lose
|
||||
trailing NULs; an empty WAL mark is not written) or untouched stores stop
|
||||
verifying — `tests/signed_store.rs` round-trips awkward strings. The key is
|
||||
never persisted; a signed store refuses to checkpoint without it
|
||||
(`MemoryError::SigningKeyRequired`, and `MemoryError` is `#[non_exhaustive]`).
|
||||
`hybrid_search`/`hybrid_search_with` are thin wrappers. It keeps one
|
||||
incremental BM25 index for the life of the store and never writes the
|
||||
store: Hebbian activation boosts are persisted by the next checkpoint (or
|
||||
on drop).
|
||||
- **HNSW** (`hnsw` feature, default): the approximate `clawhdf5-ann` index
|
||||
mirrors the cache and self-heals on drift; build the agent with
|
||||
`--no-default-features --features float16` for the exact linear scan.
|
||||
`parallel` (default) builds it on a thread pool with an identical graph.
|
||||
Neighbour selection uses the HNSW paper's diversity heuristic (closest-M
|
||||
capped recall at 0.31 recall@10 at 100K on clustered data). The graph is
|
||||
saved to `<store>.h5.ann` at each checkpoint, tied to it by a generation
|
||||
id; a stale or damaged sidecar is ignored and the index rebuilt.
|
||||
- **`MemoryConfig::quantized_index`** (default on for new stores, persisted;
|
||||
older stores load as `false` — guarded by `tests/fixtures/store_v2_5_0.h5`;
|
||||
CLI `create --f32-index`): the index's copy of the embeddings is `i8`, and
|
||||
the query path re-scores candidates against the exact embeddings. The
|
||||
aarch64 kernels (`clawhdf5_accel::dot_i8`, NEON `SDOT` via inline asm) are
|
||||
`cfg`'d out on x86, so x86 CI never compiles them — test on real ARM
|
||||
(`rpivision02`, 10.0.2.3, a Pi 5) or rely on the `test-arm64` job.
|
||||
- **`MemoryConfig::float16`** (default on for new stores, persisted; older
|
||||
stores keep `false` — guarded in `tests/float16_store.rs`; CLI `create
|
||||
--f32`): `/memory/embeddings` is IEEE half. `MemoryCache::half_precision`
|
||||
rounds each embedding as it enters the cache (push, update, WAL replay, and
|
||||
load of a store still `f32` on disk) so memory and file agree bit for bit.
|
||||
Values beyond ±65504 are `MemoryError::InvalidEntry`. The agent's
|
||||
`h5py_interop` test guards that a whole store opens in h5py.
|
||||
- **WAL.** Chained CRC32 per entry (a corrupted, reordered, duplicated or
|
||||
spliced entry stops replay cleanly). Header version 4 (`Update` record for
|
||||
`save_or_update`); v3 is upgraded in place, v2 read, v1 only through the
|
||||
one-time migration in `HDF5Memory::open`. Each checkpoint records a
|
||||
`WalMark` in `/meta` so `open()` never applies an entry twice; checkpoints
|
||||
and snapshots are durable as a unit (temp file synced, renamed, directory
|
||||
synced). Individual WAL appends are **not** fsynced (deliberate): saves
|
||||
since the last checkpoint can be lost on power failure or kernel panic.
|
||||
- **Single writer.** `create`/`open` hold an exclusive lock on
|
||||
`<store>.h5.lock` (`MemoryError::Locked` for a second opener);
|
||||
`open_read_only` is a lock-free point-in-time view (CLI `recall`/`stats`/
|
||||
`agents-md`/`export`). An unreadable WAL is quarantined to
|
||||
`<store>.h5.wal.corrupt-<ts>`; a WAL of an unknown newer version fails and
|
||||
is left untouched.
|
||||
- **Signed checkpoints** (`signing` module): with `set_signing_key` each
|
||||
checkpoint stores an Ed25519-signed manifest (per-record SHA-256 in a
|
||||
Merkle tree plus settings/sessions/graph hashes; `/integrity/record_hashes`);
|
||||
`HDF5Memory::verify(path, &pk)` locates edits. The hashes must cover
|
||||
exactly what the file persists in the form the loader returns it (strings
|
||||
lose trailing NULs; an empty WAL mark is not written) —
|
||||
`tests/signed_store.rs` round-trips awkward strings. The key is never
|
||||
persisted; a signed store refuses to checkpoint without it
|
||||
(`MemoryError::SigningKeyRequired`; `MemoryError` is `#[non_exhaustive]`).
|
||||
WAL entries after the checkpoint are not covered.
|
||||
- `Dataset::verify_provenance()` (clawhdf5 facade, `provenance` feature, on by
|
||||
default) recomputes a dataset's SHA-256 and compares it against the
|
||||
`_provenance_sha256` attribute written automatically on save when
|
||||
`DatasetBuilder::with_provenance` is used. It's opt-in per call, not run
|
||||
automatically on open — it decodes and hashes the whole dataset. The hash
|
||||
is unkeyed (tamper-*evident*, not tamper-*proof*): it detects accidental
|
||||
corruption, not a deliberate actor able to modify both the data and the
|
||||
stored hash.
|
||||
- `clawhdf5-agent`'s `HDF5Memory::save`/`save_batch`/`save_or_update` run every
|
||||
write through an in-memory (session-scoped, not persisted to disk)
|
||||
provenance ledger and write-anomaly detector: a content hash per record
|
||||
(`provenance.rs`) for detecting accidental mid-session corruption, plus
|
||||
rate-limit/injection-pattern/source-distribution checks (`anomaly.rs`).
|
||||
Alerts never block a save — drain them with `HDF5Memory::take_anomaly_alerts`.
|
||||
`MemorySource` for this bookkeeping is inferred from the caller-supplied
|
||||
`source_channel` string (a heuristic, not an authenticated trust boundary).
|
||||
- In-place modification: `clawhdf5::FileEditor` (`crates/clawhdf5/src/edit/`)
|
||||
overwrites values, grows and shrinks chunked datasets (every chunk index,
|
||||
version-2 B-trees included) and sets attributes (compact and dense
|
||||
storage) in existing files (h5py- or clawhdf5-written) without rewriting
|
||||
them, changing indexes and heaps as libhdf5 does (index shapes and heap
|
||||
bookkeeping are compared with libhdf5's in the tests); space an edit
|
||||
frees is reused by later edits of the same editor. Anything it cannot do
|
||||
safely is `Error::Unsupported` before any write (limits in
|
||||
`docs/known-issues.md`). Test changes with
|
||||
`cargo test -p clawhdf5-tools --test edit_interop --test
|
||||
edit_coverage_interop` (h5py, h5dump, `h5rs check`, structure comparisons
|
||||
with libhdf5; libhdf5 sources for the algorithms are at
|
||||
github.com/HDFGroup/hdf5, tag `hdf5_1_14_6`).
|
||||
- Remote files (`clawhdf5-remote`, range-read milestone M3 of
|
||||
`docs/design/range-reads.md`): `open_url("http://…")` gives a
|
||||
`clawhdf5::File` over `File::open_storage`, read through `BlockCache`
|
||||
(1 MiB blocks, LRU byte budget, per-block in-flight dedup across threads,
|
||||
runs coalesced into parallel requests). `HttpStorage` pins the file by
|
||||
ETag/Last-Modified and length (a change is `RemoteError::FileChanged`),
|
||||
refuses servers that ignore `Range` unless a full download is allowed,
|
||||
and retries transient failures. `ObjectStoreStorage` (feature
|
||||
`object-store`, pure Rust) runs each read on a small owned tokio
|
||||
runtime and waits on a channel, so it works from any thread, including
|
||||
inside `spawn_blocking` or another runtime. Default build is plain HTTP with
|
||||
no C; `https` (rustls + ring) and `s3`/`gcs`/`azure` (aws-lc-rs) are
|
||||
opt-in. Tests run a std-only HTTP server
|
||||
(`tests/common/server.rs`, also the `range_server` example);
|
||||
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus
|
||||
file over HTTP with `File::open`.
|
||||
- GPU-accelerated vector distance computation (`clawhdf5-gpu`, wgpu); HDF5 I/O itself is CPU-only
|
||||
- Browser: `clawhdf5-wasm` (wasm-bindgen, read-only; no Zstd/SZIP since
|
||||
they link C) and the `examples/wasm-viewer/` page. `open(bytes)` holds
|
||||
the file in memory; `openUrl(url)` (range-read M4) reads it by HTTP range
|
||||
requests through the restartable "NeedBytes" cache (`src/lazy.rs`: a
|
||||
call is re-run after each wave of misses; no block evicted while a call
|
||||
runs), the HTTP in `js/remote.js`. `examples/wasm-viewer/test/run.sh`
|
||||
builds the package (needs the `wasm-bindgen` CLI at the crate's exact
|
||||
version) and tests it under Node and headless Chromium (a Playwright
|
||||
download in `~/.cache/ms-playwright` on tank) against `test/serve.py`
|
||||
(range server with request counts, 200 MB budget file); the CI container
|
||||
has neither, so CI runs the native `h5py_interop` and `lazy` tests
|
||||
(`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus` for the corpus). Size
|
||||
numbers in the example's README predate `openUrl`.
|
||||
- Python and Node.js bindings for cross-language use
|
||||
- NetCDF-4 compatibility for scientific data interop
|
||||
- **Write bookkeeping.** `save`/`save_batch`/`save_or_update` feed an
|
||||
in-memory, session-scoped provenance ledger and anomaly detector
|
||||
(`provenance.rs`, `anomaly.rs`); alerts never block a save
|
||||
(`take_anomaly_alerts`). `MemorySource` is inferred from the caller's
|
||||
`source_channel` string — a heuristic, not a trust boundary.
|
||||
- `MemoryConfig::compression` is off by default (deflate, or Zstd with the
|
||||
agent's `zstd` feature, which links libzstd).
|
||||
|
||||
## Workflows
|
||||
|
||||
### Build
|
||||
Put `$HOME/.cargo/bin` on `PATH`. The h5py/netCDF4 interop tests find their
|
||||
Python through `CLAWHDF5_PYTHON` (or `.venv/bin/python`); create it with
|
||||
`python3 -m venv .venv && .venv/bin/pip install h5py numpy netCDF4 hdf5plugin`.
|
||||
Set `CLAWHDF5_REQUIRE_INTEROP=1` to make a missing interpreter a failure.
|
||||
|
||||
```bash
|
||||
cargo build --release
|
||||
```
|
||||
|
||||
### Test
|
||||
```bash
|
||||
cargo test --workspace
|
||||
bash scripts/ci-test.sh # everything CI runs (see below)
|
||||
```
|
||||
|
||||
### CI
|
||||
`.gitea/workflows/ci.yml` has two jobs, both green as of 2026-09-22:
|
||||
- **`test`** (`ubuntu-latest`, in `rust:latest`) runs `scripts/ci-test.sh` with
|
||||
the h5py/netCDF4 interop suites required (`CLAWHDF5_REQUIRE_INTEROP=1`).
|
||||
Served by the `tank` and `architect` runners.
|
||||
- **`test-arm64`** (`linux_arm64`) lints and tests the aarch64 code — the NEON
|
||||
kernels are `cfg`'d out on x86, so this is the only place they are built.
|
||||
Served by `vision-01` (host mode) and `vision-02` (Docker), so steps must
|
||||
work in both.
|
||||
### CI (`.gitea/workflows/`)
|
||||
- **`ci.yml` `test`** (`ubuntu-latest`, `rust:latest` container; runners
|
||||
`tank`, `architect`): installs h5py/netCDF4/xarray/hdf5plugin/maturin/pytest,
|
||||
`hdf5-tools` and `cmake`, then runs `scripts/ci-test.sh` with
|
||||
`CLAWHDF5_REQUIRE_INTEROP=1`. The script runs: fmt; clippy (workspace, the
|
||||
format feature matrix, each plugin filter alone, parallel, fast-deflate,
|
||||
remote with all backends, h5rs remote); "no C in the default build";
|
||||
wasm32 build and clippy; `check-32bit-casts.sh`; the wasm package under
|
||||
Node when `node` and `wasm-bindgen` exist (not in CI); the MSRV check;
|
||||
`cargo test` (workspace plus feature variants: format matrix, parallel,
|
||||
remote/object_store, h5rs URLs, ann parallel, fast-deflate); the h5py
|
||||
interop suites (`writer_h5py_tests --include-ignored`, plugin filters,
|
||||
ZFP); the Python package (clippy, `maturin build`, pytest vs h5py);
|
||||
`cargo bench --no-run`; `check-nostd.sh`; an optional fuzz smoke run
|
||||
(`CLAWHDF5_FUZZ_SECONDS`).
|
||||
- **`ci.yml` `test-arm64`** (`linux_arm64`; `vision-01` host mode,
|
||||
`vision-02` Docker — steps must work in both): clippy and tests of
|
||||
`clawhdf5-accel`, `-ann`, `-format`; the only place the NEON kernels build.
|
||||
- **`conformance.yml`** (nightly 03:17 UTC and manual): probe unit tests,
|
||||
`conformance/test_ref.py`, then `conformance/run.sh` (gate:
|
||||
`conformance/check.py` against `baseline.json`).
|
||||
|
||||
Keep workflows free of JavaScript actions (`actions/checkout`, `actions/cache`,
|
||||
…): `rust:latest` has no `node`, and not every runner reaches GitHub, where
|
||||
they are fetched from. Check out with plain `git` instead. The `test` job
|
||||
installs `cmake` for the opt-in `fast-deflate` (zlib-ng) steps; the default
|
||||
build needs no C toolchain, so `test-arm64` does not.
|
||||
All runners are on `gitea-runner` 3.5.0, from `docker.gitea.com/act_runner`
|
||||
— `gitea/act_runner:latest` on Docker Hub is frozen at 0.6.1.
|
||||
Keep workflows free of JavaScript actions (`actions/checkout`,
|
||||
`actions/cache`, …): `rust:latest` has no `node` and not every runner reaches
|
||||
GitHub. Check out with plain `git`. Runners are `gitea-runner` 3.5.0 from
|
||||
`docker.gitea.com/act_runner` (`gitea/act_runner:latest` on Docker Hub is
|
||||
frozen at 0.6.1).
|
||||
|
||||
### CLI
|
||||
### Conformance
|
||||
```bash
|
||||
cargo run -p clawhdf5-cli -- --help
|
||||
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot subcommands
|
||||
CLAWHDF5_PYTHON=.venv/bin/python bash conformance/run.sh --no-fetch # writes CONFORMANCE.md
|
||||
```
|
||||
Reads 697 files of eight pinned corpora with clawhdf5 and h5py and compares
|
||||
them object by object (602 ok as of 2026-09-27). `CONFORMANCE.md` is
|
||||
generated — never hand-edit it (its wording lives in `conformance/report.py`).
|
||||
Use `--update-baseline` only after an intended change in results.
|
||||
`CONFORMANCE_CACHE` points at an existing corpus cache (`conformance/.cache`,
|
||||
about 450 MB). See `conformance/README.md`.
|
||||
|
||||
### HDF5 tools (`h5rs`, crate `clawhdf5-tools`)
|
||||
### HDF5 tools (`h5rs`)
|
||||
```bash
|
||||
cargo run -p clawhdf5-tools -- ls -r file.h5 # also dump [--json], stat, diff, check
|
||||
bash scripts/h5rs-fuzz.sh # every subcommand over the CVE corpus: no panic/crash/hang
|
||||
bash scripts/h5rs-check-ok-files.sh --data # check passes every fully-read conformance file
|
||||
```
|
||||
Its interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian
|
||||
`hdf5-tools`, installed in CI); `dump` must stay byte-identical to h5dump on
|
||||
the test files.
|
||||
Interop tests compare against h5ls/h5stat/h5dump/h5diff (Debian `hdf5-tools`);
|
||||
`dump` must stay byte-identical to h5dump on the test files.
|
||||
|
||||
### Remote and browser tests
|
||||
- `clawhdf5-remote` tests run a std-only HTTP server
|
||||
(`tests/common/server.rs`, also the `range_server` example);
|
||||
`CLAWHDF5_REMOTE_CORPUS=conformance/.cache/corpus` compares every corpus
|
||||
file over HTTP with `File::open`.
|
||||
- wasm: `bash examples/wasm-viewer/test/run.sh` builds the package (needs the
|
||||
`wasm-bindgen` CLI at the crate's exact version) and tests it under Node and
|
||||
headless Chromium (Playwright's download in `~/.cache/ms-playwright` on
|
||||
tank) against `test/serve.py` (range server with request counts). CI has
|
||||
neither, so it runs the native `h5py_interop` and `lazy` tests
|
||||
(`CLAWHDF5_WASM_CORPUS=conformance/.cache/corpus` for the corpus).
|
||||
|
||||
### Python bindings
|
||||
```bash
|
||||
cd crates/clawhdf5-py
|
||||
maturin develop
|
||||
python -c "import clawhdf5; print(clawhdf5.__version__)"
|
||||
cd crates/clawhdf5-py && maturin develop
|
||||
python -m pytest crates/clawhdf5-py/tests # compares with h5py; editing tests want CLAWHDF5_H5RS=<path to h5rs>
|
||||
```
|
||||
|
||||
### Benchmarks
|
||||
- Search path: `cargo run --release -p clawhdf5-bench --bin search_harness`
|
||||
(`--full`, `--options-study`, `--footprint`, …); reads: `read_harness`,
|
||||
`concurrent_read`; criterion benches with `cargo bench -p <crate>`.
|
||||
- Run on an idle machine (1-minute load average below 2; wait otherwise),
|
||||
alternate base and candidate binaries for A/B comparisons, and record date,
|
||||
machine, commit and command with every number in `BENCHMARKS.md`.
|
||||
- `scripts/run-benchmarks.sh` is stale (it benchmarks `rustyhdf5-format` and
|
||||
overwrites `BENCHMARKS.md`) — do not run it.
|
||||
|
||||
### CLI
|
||||
```bash
|
||||
cargo run -p clawhdf5-cli -- --help
|
||||
# create, save, search, recall, stats, flush-wal, agents-md, export, snapshot
|
||||
```
|
||||
|
||||
## Integration
|
||||
@@ -255,8 +268,7 @@ python -c "import clawhdf5; print(clawhdf5.__version__)"
|
||||
verified consumer: `cbh-core` reads and writes `.brain` files through the
|
||||
facade (`File`, `FileBuilder`, `AttrValue`, `Selection`), `cbh-scanner`
|
||||
uses the facade, and `cbh-cli` uses `clawhdf5_agent::bm25::BM25Index`. It
|
||||
depends on this repo by path (`../clawhdf5`), so it builds against whatever
|
||||
is checked out — changes to those APIs reach it directly. Verified
|
||||
2026-09-25 against main: builds, and its 204 tests pass.
|
||||
- OpenClaw and ZeroClaw were both described as consumers; neither integrates
|
||||
clawhdf5 (see Key Features and `docs/openclaw.md`).
|
||||
depends on this repo by path (`../clawhdf5`), so changes to those APIs
|
||||
reach it directly. Verified 2026-09-25 against main: builds, and its 204
|
||||
tests pass.
|
||||
- OpenClaw and ZeroClaw integrate nothing (see *Standing rules*).
|
||||
|
||||
Reference in New Issue
Block a user