clawhdf5/docs/QUICKSTART.md

# clawhdf5 quick start

Short, working examples for each way in. Every snippet here was compiled
and run against the repository (2026-09-28); the Rust ones assume a
function returning `Result<_, Box<dyn std::error::Error>>`.

| You want to | Go to |
|---|---|
| Read or write HDF5 from Rust | [HDF5 in Rust](#1-hdf5-in-rust) |
| Read or edit HDF5 from Python without libhdf5 | [Python](#2-python) |
| Read NetCDF-4 files | [NetCDF-4](#3-netcdf-4) |
| Inspect or validate files on the command line | [h5rs](#4-h5rs) |
| Give an AI agent a memory store | [Agent memory](#5-agent-memory) |

What is and is not supported: the [feature matrix](../README.md#what-is-supported)
and [known-issues.md](known-issues.md).

---

## 1. HDF5 in Rust

### Install

Not on crates.io yet; depend on the repository (MSRV 1.92):

```toml
[dependencies]
clawhdf5 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
# every plugin filter (bitshuffle, bzip2, Blosc, Blosc2, ZFP; LZF is on by default):
# clawhdf5 = { git = "...", features = ["plugin-filters"] }
```

### Write a file

```rust
use clawhdf5::{AttrValue, FileBuilder};

let mut b = FileBuilder::new();
b.set_attr("title", AttrValue::String("run 42".into()));      // a root attribute

b.create_dataset("temperatures")                              // 1-D f64, contiguous
    .with_f64_data(&[22.5, 23.1, 21.8, 24.0]);
b.create_dataset("grid")                                      // 2-D f32, chunked + gzip
    .with_f32_data(&vec![1.5f32; 256 * 256])
    .with_shape(&[256, 256])
    .with_chunks(&[64, 64])
    .with_deflate(4)
    .with_fletcher32();
b.create_dataset("counts")                                    // LZF (default feature), as h5py's compression="lzf"
    .with_i32_data(&(0..10_000).collect::<Vec<i32>>())
    .with_chunks(&[1000])
    .with_lzf();
b.create_dataset("log")                                       // appendable: unlimited first axis
    .with_f64_data(&[])
    .with_shape(&[0])
    .with_maxshape(&[u64::MAX])
    .with_chunks(&[1024]);

let mut sensors = b.create_group("sensors");                  // groups nest; paths work too
sensors.set_attr("site", AttrValue::String("north".into()));
sensors.create_dataset("ids").with_i32_data(&[7, 8, 9]);
b.add_group(sensors.finish());
b.add_soft_link("latest", "/sensors");
b.write("example.h5")?;
```

h5py, h5dump and `h5rs check --data` read the result. `FileBuilder` holds
the file in memory and writes it once (atomically). Other data:
`with_f16_data`, `with_i64_data`, `with_u64_data`, `with_u8_data`,
`with_compound_data` (with `CompoundTypeBuilder`), enums, array types;
filters `with_shuffle`, `with_zstd`, `with_lz4`, `with_bitshuffle`,
`with_bzip2`, `with_blosc` (behind features); `with_fill_value`,
`track_order`, hard and external links, virtual datasets. The writer does
not write variable-length data.

### Read a file

```rust
use clawhdf5::{File, Selection};

let file = File::open("example.h5")?;
let root = file.root();
println!("datasets {:?}, groups {:?}", root.datasets()?, root.groups()?);
println!("attrs {:?}", root.attrs()?);

let grid = file.dataset("grid")?;
println!("{:?} {:?} {:?}", grid.shape()?, grid.dtype()?, grid.max_dimensions()?);
let values: Vec<f32> = grid.read_f32()?;                       // integers/floats convert as libhdf5 does
let window = grid.read_f32_selection(&Selection::Hyperslab {
    start: vec![0, 0], stride: vec![2, 2], count: vec![16, 16], block: vec![1, 1],
})?;                                                            // every other element of a 32x32 corner
let ids = file.group("sensors")?.dataset("ids")?.read_i64()?;
let same = file.dataset("latest/ids")?.read_i32()?;             // through the soft link
```

A selection whose bounding box covers at most half the dataset decodes only
the chunks it touches; a larger one decodes the whole dataset
([known-issues.md](known-issues.md#selection-reads-that-decode-more-than-the-selection)).
`File::open` maps the file (`mmap` feature, default); `File::open_buffered`
reads it into memory, `File::from_bytes` takes a buffer, and
`File::open_storage` any `Storage` backend. A `File` is `Send + Sync`:
share it between threads.

Strings and variable-length data:

```rust
let file = clawhdf5::File::open("strings.h5")?;                 // written by h5py
let names: Vec<String> = file.dataset("names")?.read_string()?; // fixed- or variable-length
```

`read_vlen::<T>()` reads variable-length sequences, and
`File::decode_strings` / `decode_vlen` decode such values inside compounds
and raw attributes.

### Edit a file in place

`FileEditor` changes an existing file (from h5py or clawhdf5) without
rewriting it: values, dataset extents, attributes. Here, appending batches
to the unlimited `log` dataset written above:

```rust
use clawhdf5::{FileEditor, Selection};

let mut ed = FileEditor::open("example.h5")?;
for batch in 0..3u64 {
    let rows = vec![batch as f64; 500];
    ed.resize("log", &[(batch + 1) * 500])?;
    let sel = Selection::Hyperslab {
        start: vec![batch * 500], stride: vec![1], count: vec![500], block: vec![1],
    };
    ed.write_values("log", &sel, &rows)?;
}
```

Each call is written and synced before it returns. The editor holds an
exclusive lock and has no journal: a crash in the middle of an edit can
leave the file inconsistent. What it refuses (before writing anything):
[known-issues.md § In-place modification](known-issues.md#in-place-modification-fileeditor-limits).

### Remote files and SWMR

```rust
// clawhdf5-remote = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
let file = clawhdf5_remote::open_url("http://127.0.0.1:8000/tall.h5")?;
let values = file.dataset("/g2/dset2.1")?.read_f64()?;
```

Serve a directory with range support to try it:
`cargo run -p clawhdf5-remote --example range_server -- crates/clawhdf5/tests/fixtures 127.0.0.1:8000`.
`https://` needs the `https` feature; `s3://`, `gs://`, `az://` the `s3`,
`gcs`, `azure` features (credentials from the environment).
See [crates/clawhdf5-remote/README.md](../crates/clawhdf5-remote/README.md).

A file an h5py/libhdf5 SWMR writer is still appending to:

```rust
use std::time::{Duration, Instant};

let file = clawhdf5::File::open_swmr("live.h5")?;
let mut ds = file.dataset("samples")?;
let (mut seen, mut last_growth) = (0, Instant::now());
// Stop when the writer closes the file, or when the dataset has not grown for
// a minute (a writer that died never clears the SWMR-write flag).
while file.swmr_writer_active()? && last_growth.elapsed() < Duration::from_secs(60) {
    ds.refresh()?;                               // h5py: ds.refresh()
    let n = ds.shape()?[0];
    if n > seen {
        // read rows seen..n ...
        (seen, last_growth) = (n, Instant::now());
    }
    std::thread::sleep(Duration::from_millis(100));
}
```

Design and limits: [design/swmr.md](design/swmr.md).

---

## 2. Python

Not on PyPI yet; build the package with maturin into a virtualenv:

```bash
python -m venv .venv && . .venv/bin/activate
pip install maturin numpy
maturin develop --release -m crates/clawhdf5-py/Cargo.toml
```

Reading follows h5py:

```python
import numpy as np
import clawhdf5

with clawhdf5.File("data.h5", "r") as f:
    print(list(f.keys()))              # member names, like h5py
    ds = f["group/temperatures"]       # relative or absolute paths
    print(ds.shape, ds.dtype, ds.chunks)
    block = ds[100:200, ::4]           # a small selection decodes only its chunks
    row = ds[-1]                       # integers drop the axis
    picked = ds[[1, 5, 9], :]          # one increasing index list per key
    units = ds.attrs["units"]          # attributes come back as h5py returns them
    everything = np.asarray(ds)
    ids = f["table"]["id"]             # compound -> structured array; one field
```

Editing an existing file in place (`'r+'`, through `FileEditor`), with
h5py's keys, broadcasting and numeric conversion; each edit is on disk when
the statement returns:

```python
with clawhdf5.File("data.h5", "r+") as f:
    f["group/temperatures"][100:200, ::4] = 0.0
    f["series"].resize(5000, axis=0)   # chunked datasets, within maxshape
    f["series"][4000:] = np.ones(1000)
    f["group"].attrs["calibrated"] = True
```

`'r+'` cannot create or delete datasets and groups, or delete attributes
(`NotImplementedError`, nothing written). New files (`'w'`) take numeric
arrays (`float64`, `float32`, `int64`, `int32`, `uint8`):

```python
with clawhdf5.File("new.h5", "w") as f:
    f.create_dataset("x", data=np.arange(1000.0), chunks=(100,), compression="gzip")
    f.create_group("meta").attrs["version"] = np.int64(2)
```

A URL opens a remote file read-only, by range requests (`http://` in the
default build; `https://` and `s3://`/`gs://`/`az://` with
`--features https` / `s3` / `gcs` / `azure`):

```python
with clawhdf5.File("http://data.example.org/run42.h5") as f:
    first = f["group/temperatures"][0]
f = clawhdf5.File.open_url("http://data.example.org/run42.h5", block_size=256 * 1024,
                           headers={"Authorization": "Bearer ..."})
print(f.remote_stats)
```

Types, keys and limits: [crates/clawhdf5-py/README.md](../crates/clawhdf5-py/README.md).

---

## 3. NetCDF-4

```rust
// clawhdf5-netcdf4 = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
use clawhdf5_netcdf4::NetCDF4File;

let nc = NetCDF4File::open("climate.nc")?;
let mut temp = nc.variable("temperature")?;
let values = temp.read_f64()?;                        // CF scale_factor/add_offset/_FillValue applied
println!("{:?} {:?}", temp.shape()?, temp.cf_attributes()?.units);
```

`dimensions()`, `variables()`, `global_attrs()` and `group(..)` walk the
rest of the file; `hdf5_file()` gives the underlying `clawhdf5::File`.

---

## 4. h5rs

```bash
cargo install --path crates/clawhdf5-tools    # --features remote for URLs
h5rs ls -r example.h5
h5rs dump example.h5          # DDL like h5dump; --json for hdf5-json
h5rs stat example.h5
h5rs diff a.h5 b.h5
h5rs check --data example.h5  # structure + checksums + every dataset decoded
```

See [crates/clawhdf5-tools/README.md](../crates/clawhdf5-tools/README.md).

---

## 5. Agent memory

```toml
[dependencies]
clawhdf5-agent = { git = "https://git.redclaw.dev/quantumclaw/clawhdf5" }
```

```rust
use clawhdf5_agent::{AgentMemory, HDF5Memory, MemoryConfig, MemoryEntry, SearchOptions};

// A new store: 384-dim embeddings (float16 on disk and an int8 HNSW index by default).
let mut memory = HDF5Memory::create(MemoryConfig::new("agent.h5".into(), "my-agent", 384))?;

memory.save(MemoryEntry {
    chunk: "User prefers dark mode and vim keybindings.".into(),
    embedding: embed("User prefers dark mode and vim keybindings."), // your embedder
    source_channel: "chat".into(),
    timestamp: now,
    session_id: "session-001".into(),
    tags: "preference".into(),
})?;

// Hybrid search: HNSW vector + BM25 keyword, fused 0.4 / 0.6 (the measured default).
let query = embed("what editor does the user like?");
for r in memory.search(&query, "editor preferences", &SearchOptions::new(5)) {
    println!("[{:.3}] {}", r.score, r.chunk);
}
memory.flush_wal()?; // checkpoint the WAL into agent.h5
```

`embed` is yours: clawhdf5 stores embeddings, it does not compute them.
Each agent gets its own store; a store has a single writer, and
`HDF5Memory::open_read_only` gives other processes a lock-free view.
Source filters, re-ranking, signed checkpoints, the knowledge graph,
consolidation and the rest: [agent-memory.md](agent-memory.md).

### CLI

`clawhdf5-cli` installs a binary named `clawhdf5`; output is JSON.

```bash
cargo install --path crates/clawhdf5-cli
clawhdf5 --path agent.h5 create --agent-id my-agent --dim 384 --wal
echo '{"chunk":"User prefers dark mode","embedding":[0.1, ...],"source_channel":"chat","timestamp":1700000000.0,"session_id":"s1","tags":"pref"}' \
  | clawhdf5 --path agent.h5 save
clawhdf5 --path agent.h5 search --embedding '[0.1, ...]' --query 'dark mode preferences' \
  --top-k 5 --vector-weight 0.4 --keyword-weight 0.6
clawhdf5 --path agent.h5 stats
clawhdf5 --path agent.h5 export > memories.jsonl
clawhdf5 --path agent.h5 snapshot backup.h5
```

The CLI's `search` defaults to weights 0.7 / 0.3, not the library's
0.4 / 0.6, so pass them.

---

## Next

- [USE_CASES.md](USE_CASES.md) — where clawhdf5 fits
- [CONFORMANCE.md](../CONFORMANCE.md), [BENCHMARKS.md](../BENCHMARKS.md) — the evidence
- [README.md](README.md) — every document