osobhandClaude Opus 5.5
e5d6f59e12
Read a file's metadata the way netCDF-C 4.9.3 does (libhdf5/hdf5open.c), for the whole file on first use (src/model.rs, replacing src/scope.rs): - links in creation order when the group tracks it, else name order; a group's datasets before its subgroups; dimension ids file-wide; - variables' dimensions from _Netcdf4Coordinates (file-wide ids), else the scales DIMENSION_LIST attaches when the first axis has one, else netCDF-C's phony dimensions phony_dim_<id> (create_phony_dims: shared by length and unlimitedness within a group, not between two axes of one variable, numbered subgroups first, a zero length unlimited); - datasets of types netCDF-C cannot represent are not variables (references, bit fields, time, arrays, compounds/enums/VLENs over them), replaying netCDF-C's file-wide type list, failed types included; - unlimited lengths as nc4_find_dim_len (its group and below). NcType gains Enum, Compound, VLen, Opaque and is #[non_exhaustive]; Variable::nc_type is netCDF-C's type (1-byte strings NC_CHAR). New clawhdf5_format::group_v2::links_in_creation_order_in. Tests compare with netCDF-C itself (tests/netcdf_c_view.py calls the libnetcdf netCDF4-python bundles through ctypes): new interop cases for h5py files without dimension scales, every type class, link order; and the gated corpus_vs_netcdf_c (CLAWHDF5_NETCDF_CORPUS): 420 of the 429 conformance-corpus files netCDF-C opens match (main: 68); the other 9 are explained in tests/corpus_known_differences.txt and known-issues. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
clawhdf5 documentation
Every document in the repository, one line each. Start with the README and the quick start.
Using clawhdf5
| Document | What it covers |
|---|---|
| README | What clawhdf5 is, the evidence, the feature matrix, install, quick starts, crate map |
| QUICKSTART.md | Working examples: HDF5 in Rust and Python, remote files, SWMR, NetCDF-4, h5rs, agent memory, CLI |
| USE_CASES.md | Where clawhdf5 fits, and when to use something else |
| agent-memory.md | The agent-memory store: search, durability, signing, modules, performance, schema, CLI, SQLite migration |
| known-issues.md | Open limits and fixed bugs, dated — read before relying on an edge case |
| openclaw.md | Why clawhdf5 is not an OpenClaw memory plugin, and what one would need |
| CHANGELOG.md | Every change by release, with upgrade notes; "Unreleased" is everything since v2.7.0 |
Evidence
| Document | What it covers |
|---|---|
| CONFORMANCE.md | Generated report: 697 public HDF5 files read by clawhdf5 and h5py and compared; the CVE corpus against h5dump and h5py |
| conformance/README.md | How the conformance sweep works and how to run it |
| BENCHMARKS.md | Every measurement with date, machine and command: HDF5 reads and writes, concurrency, deflate backends, search, LongMemEval, footprint |
| benchmarks/longmemeval/README.md | Downloading the LongMemEval data |
| benchmarks/2026-03-01-oracle-xeon.md | An early (March 2026) benchmark run on a Xeon server; superseded by BENCHMARKS.md |
Design
| Document | What it covers |
|---|---|
| design/range-reads.md | Reading through a Storage trait: milestones M0–M5 (indexed lookups, storage, raw data, remote files, the browser, SWMR) |
| design/swmr.md | Reading files a libhdf5 SWMR writer is appending to (M5) |
| design/tools/ | Scripts behind the range-read design's measurements (inventory.py, libhdf5_reads.py, range-trace) |
Crates and packages
| Document | What it covers |
|---|---|
| crates/clawhdf5 | The facade: File, FileBuilder, FileEditor |
| crates/clawhdf5-format | The format implementation and codecs; fuzzing |
| crates/clawhdf5-filters | Deflate backends |
| crates/clawhdf5-io | I/O helpers (mmap, async, HSDS, MPI) |
| crates/clawhdf5-remote | Remote files: HTTP(S), object stores, block cache |
| crates/clawhdf5-netcdf4 | NetCDF-4 layer |
| crates/clawhdf5-derive | Derive macros |
| crates/clawhdf5-tools | h5rs |
| crates/clawhdf5-py | Python bindings |
| crates/clawhdf5-wasm | The browser reader crate |
| examples/wasm-viewer | Browser viewer and the clawhdf5-wasm JavaScript API |
| crates/clawhdf5-napi, packages/clawhdf5-node | Node.js bindings and package (unpublished, does not work) |
| crates/clawhdf5-android | Android JNI bindings for the agent store |
| crates/clawhdf5-agent | Agent memory (full guide: agent-memory.md) |
| crates/clawhdf5-ann | HNSW index |
| crates/clawhdf5-accel | SIMD kernels |
| crates/clawhdf5-gpu | GPU vector distances |
| crates/clawhdf5-migrate | SQLite migration |
| crates/clawhdf5-cli | The agent-memory CLI |
| crates/clawhdf5-bench | Benchmarks and harnesses |
Project history and working notes
| Document | What it covers |
|---|---|
| ROADMAP.md | What has shipped (releases and PRs since v2.7.0) and what is next |
| CLAUDE.md | Architecture and workflow notes for contributors and coding agents |
| archive/IMPROVEMENT_LOG.md, archive/IMPROVEMENT_SCAN.md | Logs of earlier automated improvement passes (archived, historical) |
| archive/plans/ | Implementation plans from June 2026 (filter codecs, format write extensions, MPI-IO); archived, historical |
| research/ | Research briefs from August 2026 (performance, security, provenance) |