docs: refresh README, crate READMEs and reference docs; fact-check every claim #22

2026-09-28T16:38:55Z

osobh commented

2026-09-28 16:38:55 +00:00

Brings the README and docs up to date with everything merged through #21, and checks every claim against the code and the dated measurements.

Rewritten

README.md (1120 → 444 lines): what works, what's partial, what's unsupported, headline numbers with dates. Agent-memory detail moved to the new docs/agent-memory.md.
Every crate README; android, bench, cli, napi and wasm are new.
docs/known-issues.md restructured around an open-issues table (1207 → 808 lines, nothing deleted from the history).
BENCHMARKS.md gets a "Current headline numbers" section. Measured values are unchanged.
ROADMAP.md, QUICKSTART.md, USE_CASES.md, CLAUDE.md, conformance/README.md, and docs/README.md (now an index of every doc).

Archived: IMPROVEMENT_LOG, IMPROVEMENT_SCAN and the old plans, moved to docs/archive/.

Removed: scripts/run-benchmarks.sh. It benchmarked the pre-rename rustyhdf5-format and overwrote BENCHMARKS.md.

Fact-check (about 650 claims, 26 fixed):

Figures with no date now carry their dates.
The libhdf5 chunked-write figure uses the newest measurement: 35x from 2026-09-23, replacing 45.3x.
Conformance counts follow the 2026-09-28 run.
Claims that went beyond their evidence are now scoped: zlib-rs matching zlib-ng, and LZ4 being opt-in.
Wrong type and feature names are fixed.

New known issue: NetCDF-4 unlimited dimensions report size 0 (the fix will be separate).

Six Cargo.toml descriptions no longer mention rustyhdf5 or edgehdf5.

Checks:

The link check over all 52 tracked .md files is clean.
The README and QUICKSTART Rust examples compile and run, and the Python snippets run.
scripts/ci-test.sh: 26/27 steps pass. The one failure is edit_interop::editor_locks_the_file, a pre-existing fork/flock race that happens only under a full parallel run. It passes 15/15 on its own, and this PR changes no code. It will be fixed separately.

🤖 Generated with Claude Code

Brings the README and docs up to date with everything merged through #21, and checks every claim against the code and the dated measurements. **Rewritten** - `README.md` (1120 → 444 lines): what works, what's partial, what's unsupported, headline numbers with dates. Agent-memory detail moved to the new `docs/agent-memory.md`. - Every crate README; android, bench, cli, napi and wasm are new. - `docs/known-issues.md` restructured around an open-issues table (1207 → 808 lines, nothing deleted from the history). - `BENCHMARKS.md` gets a "Current headline numbers" section. Measured values are unchanged. - `ROADMAP.md`, `QUICKSTART.md`, `USE_CASES.md`, `CLAUDE.md`, `conformance/README.md`, and `docs/README.md` (now an index of every doc). **Archived:** `IMPROVEMENT_LOG`, `IMPROVEMENT_SCAN` and the old plans, moved to `docs/archive/`. **Removed:** `scripts/run-benchmarks.sh`. It benchmarked the pre-rename `rustyhdf5-format` and overwrote `BENCHMARKS.md`. **Fact-check** (about 650 claims, 26 fixed): - Figures with no date now carry their dates. - The libhdf5 chunked-write figure uses the newest measurement: 35x from 2026-09-23, replacing 45.3x. - Conformance counts follow the 2026-09-28 run. - Claims that went beyond their evidence are now scoped: zlib-rs matching zlib-ng, and LZ4 being opt-in. - Wrong type and feature names are fixed. **New known issue:** NetCDF-4 unlimited dimensions report size 0 (the fix will be separate). Six `Cargo.toml` descriptions no longer mention rustyhdf5 or edgehdf5. Checks: - The link check over all 52 tracked `.md` files is clean. - The README and QUICKSTART Rust examples compile and run, and the Python snippets run. - `scripts/ci-test.sh`: 26/27 steps pass. The one failure is `edit_interop::editor_locks_the_file`, a pre-existing fork/`flock` race that happens only under a full parallel run. It passes 15/15 on its own, and this PR changes no code. It will be fixed separately. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

osobh added 21 commits 2026-09-28 16:38:55 +00:00

docs: known issues split into open issues and a condensed fixed history 798331ddbc

An "Open issues" table at the top links each open entry; fixed entries
move to "Fixed (history)", newest first, keeping the date, PR, affected
releases and what users must do. Open entries re-checked against main
(9b5803f): the remaining audit gaps are gathered into one entry, the
range-read and remote limits no longer contradict themselves (Python and
the browser open URLs; SWMR reading is File::open_swmr), and the
nondeterministic damaged-chunk error, fixed in PR #19, has its own
history entry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: benchmark headline numbers, superseded sections labelled 06d8e2ee45

A "Current headline numbers" table gives each figure's newest dated
measurement with its machine, command and section. Sections a later run
replaced are marked superseded with a link to the newer one; stale
"still open" notes and cross-references now point at the fixes. No
measured value changed.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: design status notes reflect what is merged ac0020594b

range-reads.md opens with a table of milestones M0-M5 and the PRs that
merged them (#17-#21), replacing a header left garbled by earlier
merges, and each milestone's status names its PR. swmr.md says the
reader is merged (PR #19) and the writer does not exist. openclaw.md
links the Node package's known-issues entry.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: known issue for the flapping bad_nbit_parms_walk classification 24cc9d14d8

The committed CONFORMANCE.md counts bad_nbit_parms_walk.h5 as an
our-error because its six h5py reads agreed in that run; a rerun the
same day confirmed the over-read and counted it ref-bug. The ok count
(602 of 697) is the same either way.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: CLAUDE.md regrouped into rules, invariants and workflows 38107b90ed

Standing rules (no C by default, h5py must read what we write, one
float16 implementation, claims need evidence, OpenClaw/ZeroClaw
withdrawn, the known-issues rule) are gathered in one place; library and
agent-memory invariants are split; the CI section lists what
ci-test.sh and the conformance workflow run now instead of a dated
"two jobs, green" line. Adds the conformance, remote, wasm, Python and
benchmark workflows (idle load below 2, dated records), and warns that
scripts/run-benchmarks.sh is stale and overwrites BENCHMARKS.md. Crate
roles corrected (clawhdf5-io is I/O adapters; codecs live in
clawhdf5-format). Benchmark figures now live in BENCHMARKS.md only.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: agent memory guide in docs/agent-memory.md 4d1a43fd7a

The agent-memory detail that lived only in README.md (architecture,
modules, performance and footprint tables, LongMemEval, feature flags and
settings, file schema, SQLite migration, research foundation) moves to its
own page, so the README can lead with the HDF5 library. Code examples are
updated to the current API (MemoryConfig::new takes a PathBuf,
HDF5Memory::search with SearchOptions, consolidation with timestamps) and
were compiled and run against the workspace; the CLI section was run
against the built `clawhdf5` binary. New: the CLI's search defaults to
0.7/0.3, not the library's 0.4/0.6; the /integrity group of signed stores.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: README rewritten around the HDF5 library and its evidence a90373ca84

Leads with what clawhdf5 is today for an HDF5 reader: the conformance
result (602 of 697, 0 mismatches, no panic/hang/crash; CONFORMANCE.md of
2026-09-28), the CVE corpus against h5dump and h5py, concurrent reads
against h5py threads and processes (BENCHMARKS.md, 2026-09-26, c5334b1)
and the libhdf5 comparison with its date and caveat; then a feature matrix
(supported / read only / not supported), install from git and maturin,
Rust and Python quick starts, remote files, the browser, SWMR, h5rs, a
short agent-memory section, the crate map and a documentation table.

Removed: the unverifiable "1850+ tests" badge and "~86K lines" footer,
the "What's new v2.2 -> v2.7" list (it is CHANGELOG.md), the agent
comparison table with other products, the Phase 1/2 roadmap checklist,
and the long agent sections (now docs/agent-memory.md). Fixed: crates
listed as C-free, SZIP / N-Bit / scale-offset as read-only, virtual
datasets as written too, Python 'r+' can create attributes (it cannot
create or delete objects). Every code snippet was compiled and run
against the workspace, the Python ones against a wheel built from it.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: QUICKSTART and USE_CASES on the current APIs b0b4018919

QUICKSTART used APIs that do not exist (file.dataset_names(),
File::attr, AttrValue::Str, memory.search(&q, 5), MemoryConfig::new with
a &str, consolidation without timestamps), `clawhdf5 = "2.0"` from
crates.io, and "3-45x faster than libhdf5". It now covers HDF5 in Rust
(write, read, strings, in-place append, remote, SWMR), Python (read, r+,
w, URLs), NetCDF-4, h5rs, agent memory and the CLI, every snippet
compiled and run (Python against a wheel built from the tree).

USE_CASES dropped claims with no source (the agent crate adds ~2MB,
IVF-PQ under 1.2 ms on modest hardware, an OpenClaw scenario, a .brain
layout and `clawhub publish` commands) and now covers the HDF5 cases
(no-C builds, threads, remote data, untrusted files, SWMR, in-place
edits), the agent cases with measured numbers, and when to use
something else.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: docs/README.md indexes every document b660952421

One line per document: user guides, evidence (CONFORMANCE, BENCHMARKS,
the conformance README), design docs, crate and package READMEs, and the
historical notes (roadmap, improvement logs, June plans, research briefs).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: crate READMEs describe each crate as it is today b55b24b7ba

Every crate under crates/ now has a README (android, bench, cli, napi and
wasm had none), each saying what the crate is, its main types and
functions (names checked against the code), its cargo features with
defaults and which ones build C (checked with `cargo tree`), and links to
the top-level docs.

Corrections to the old stubs:
- clawhdf5-derive: the derive is `H5Type`, not `HDF5Type`, and it needs
  clawhdf5-format as a dependency.
- clawhdf5-filters: deflate backends only, and no library crate depends
  on it; the filter pipeline and every other codec are in -format.
- clawhdf5-gpu: vector distance compute, not I/O; not used by
  HDF5Memory::search.
- clawhdf5-io: MpiVol is root-read + broadcast, not collective MPI-IO.
- clawhdf5-ann: from_hdf5/search(q, k) did not exist; load_from_hdf5 and
  search(q, k, ef).
- clawhdf5-accel: checksum::crc32_simd did not exist; the SSE4 and wasm
  backends are reported but run the scalar kernels.
- clawhdf5-gpu: the old example called l2_distances, which does not
  exist (l2_search).
- clawhdf5-agent: it described a "vector store" with "GPU acceleration";
  it now covers HDF5Memory, search options, WAL, signing, the graph.
- crates.io/docs.rs badges removed and `cargo install <crate>` replaced:
  nothing is published; depend on git.
- fuzz: the opt-in CLAWHDF5_FUZZ_SECONDS smoke run in ci-test.sh.
- tools: the FileEditor interop tests that live in this crate.
- remote, py: license, other front ends, limits, File.mode/flush/chunks.

The Rust examples of the facade, format, filters, accel, ann, derive and
agent READMEs were compiled and run as tests (netcdf4, gpu and remote
compiled only) in a scratch crate; the CLI example was run.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs(wasm-viewer): package size re-measured after openUrl; round-trip costs 999cb86071

The size table predated openUrl (it said so). Re-measured on tank at
9b5803f with `bash examples/wasm-viewer/build.sh`, then `wc -c` and
`gzip -9 -n -c`: the wasm is 1,384,607 B (378,485 gzipped), was 627,501
(191,639); the glue 40,711 (8,181), was 21,826 (4,487); remote.js 9,326
(3,448). 476,062 B of the wasm is the function-name section. opt-level z
(CARGO_PROFILE_WASM_RELEASE_OPT_LEVEL=z) is now 1% smaller gzipped than
the profile's s; 3 is larger. h5wasm rows unchanged.

Also adds the measured cost of listing a large group and reading one
dataset by URL (passes / requests / bytes, from CHANGELOG, 2026-09-27).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs(conformance): file classes, ref-bug and ref_fix, latest result cab952bb00

Explains each class compare.py assigns, how ref_bugs.py confirms a
ref-bug in every run and how ref.py corrects h5py's big-endian VL values
(ref_fix), and records the latest result (602 of 697 ok, 3 ref-bug,
2026-09-27) with a link to CONFORMANCE.md. Mentions --no-fetch.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: ROADMAP rewritten from CHANGELOG, git log and known-issues 419cb52287

The old file was a tracker for the mid-2026 agent-memory tracks, last
updated 2026-08-05, with an OpenClaw track and "what's next" items that
have since shipped (CI, fuzz target, WAL checksums, HNSW parallel build).

Now: releases v2.0.0-v2.7.0 and every PR merged since (#3-#21, merge
dates from git log), range-read milestones M0-M5, a one-paragraph summary
of the agent-memory work, and what is genuinely next: crates.io and PyPI
publishing (plus the broken Node package), SWMR writing, MPI collective
I/O, paged-metadata single-request reads, Blosc2/ZFP encoders, and the
open items of docs/known-issues.md. OpenClaw and ZeroClaw are listed only
as withdrawn. No dates are given for future work.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: archive the improvement log/scan and the June superpowers plans 3c557c9f0a

Moved to docs/archive/ (kept for their history, not deleted), each with a
one-line header saying it is historical and what supersedes it:

- IMPROVEMENT_LOG.md: three automated-loop PRs from April-May 2026, on
  the earlier quantumclaw PR numbering, which now collides with this
  repo's #12-#15. Superseded by CHANGELOG.md and git log.
- IMPROVEMENT_SCAN.md: one scan's notes (2026-05-04) of changes merged
  long ago. Superseded by CHANGELOG.md and git log.
- docs/superpowers/plans/*.md -> docs/archive/plans/: agent pre-work
  plans for d6c4d4f (2026-06-30), already marked implemented. The MPI plan
  promised collective MPI-IO, which is not what shipped (MpiVol is
  root-read + broadcast); its header says so and points at ROADMAP.md,
  where collective I/O is an open item.

Nothing links to the old paths (research/*.md mention IMPROVEMENT_LOG.md
and ROADMAP.md in prose only).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Merge branch 'docs/refresh-reference' into docs/readme-refresh 14790487a7

Merge branch 'docs/refresh-crates' into docs/readme-refresh c53c43b14f

Merge branch 'docs/refresh-readme' into docs/readme-refresh 2bffd6b622

docs: fix cross-links after the refresh; remove the stale benchmark script 3100f0143b

- docs/README.md links the improvement logs and June plans where the
  refresh archived them (docs/archive/).
- clawhdf5-py README: 'r+' creates and replaces attributes (compact or
  dense); only deleting them is unsupported.
- scripts/run-benchmarks.sh benchmarked the pre-rename rustyhdf5-format
  and overwrote BENCHMARKS.md; nothing referenced it. Removed.
- Cargo.toml descriptions no longer name rustyhdf5/edgehdf5; clawhdf5-gpu
  says it is not HDF5 I/O.
- benchmarks/cross_platform.sh pointed at a ROADMAP section that no longer
  exists.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: known issue — NetCDF-4 unlimited dimensions report size 0 a48cb9f1a4

Found while checking the README refresh: clawhdf5-netcdf4 reads an
unlimited dimension's size from its dimension-scale dataset, which
netCDF-C never extends, so a dimension with 2 records reports 0.
Variable shapes and values are right. To be fixed separately.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: fact-check the refreshed documentation against its sources c27a478e44

Numbers, API names, feature defaults and PR references checked against
CONFORMANCE.md, BENCHMARKS.md, CHANGELOG.md, the code and git history.

- int8 index figures (1.74x memory, 1.63x QPS) carry the dates git gives
  them (2026-09-19/20, machine not recorded, not re-run) instead of none;
  the Pi 5 1.18x carries 2026-09-21.
- BENCHMARKS headline: the libhdf5 chunked-write figure is the newest
  measurement (35x, 2026-09-23), not 45.3x (2026-08-03).
- Conformance counts follow the 2026-09-28 run (1 our-error, 2 ref-bug)
  in conformance/README.md, ROADMAP.md and CLAUDE.md, with a pointer to
  the bad_nbit_parms_walk.h5 flip.
- README: LZ4 is opt-in; the browser refuses reference/opaque/bitfield/
  time datasets too; zlib-rs byte-identity scoped to what was measured;
  macOS default links the system libz for inflate.
- Crate READMEs: system-zlib-decompress does something (macOS), SweepDetector
  lives in prefetch, checkpoint after more than 500 WAL entries, NetCDF-4
  unlimited-dimension size warning.
- agent-memory.md: string-dataset compression threshold, agents-md prints
  Markdown, float16 file sizes linked to their study.
- known-issues.md: contiguous selection reads, 1.21x vs h5py threads.
- docs/README.md, USE_CASES.md, ROADMAP.md, CLAUDE.md: range-read
  milestones M0-M5 and PRs #17-#19, missing README rows, CLI keygen/verify,
  dated figures, fast-math is not BLAS.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs: README — szip is a clawhdf5-format feature

CI / test-arm64 (pull_request) Successful in 1m35s

Details

CI / test (pull_request) Successful in 15m45s

Details

d3d8d7ded3

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

osobh merged commit 5eae9ee60b into main

2026-09-28 17:24:26 +00:00

osobh deleted branch docs/readme-refresh

2026-09-28 17:24:30 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: quantumclaw/clawhdf5#22