docs: fact-check the refreshed documentation against its sources
Numbers, API names, feature defaults and PR references checked against CONFORMANCE.md, BENCHMARKS.md, CHANGELOG.md, the code and git history. - int8 index figures (1.74x memory, 1.63x QPS) carry the dates git gives them (2026-09-19/20, machine not recorded, not re-run) instead of none; the Pi 5 1.18x carries 2026-09-21. - BENCHMARKS headline: the libhdf5 chunked-write figure is the newest measurement (35x, 2026-09-23), not 45.3x (2026-08-03). - Conformance counts follow the 2026-09-28 run (1 our-error, 2 ref-bug) in conformance/README.md, ROADMAP.md and CLAUDE.md, with a pointer to the bad_nbit_parms_walk.h5 flip. - README: LZ4 is opt-in; the browser refuses reference/opaque/bitfield/ time datasets too; zlib-rs byte-identity scoped to what was measured; macOS default links the system libz for inflate. - Crate READMEs: system-zlib-decompress does something (macOS), SweepDetector lives in prefetch, checkpoint after more than 500 WAL entries, NetCDF-4 unlimited-dimension size warning. - agent-memory.md: string-dataset compression threshold, agents-md prints Markdown, float16 file sizes linked to their study. - known-issues.md: contiguous selection reads, 1.21x vs h5py threads. - docs/README.md, USE_CASES.md, ROADMAP.md, CLAUDE.md: range-read milestones M0-M5 and PRs #17-#19, missing README rows, CLI keygen/verify, dated figures, fast-math is not BLAS. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
@@ -57,7 +57,9 @@ the end of a chunk, short unfiltered chunks, an N-Bit parameter list one
|
||||
value short) that HDF5 2.0 returns only by reading past a buffer;
|
||||
clawhdf5 refuses them, as libhdf5's development branch and its own
|
||||
`test_filter_bad_params` do. Details and evidence in
|
||||
[CONFORMANCE.md § Reference bugs](CONFORMANCE.md#reference-bugs). The
|
||||
[CONFORMANCE.md § Reference bugs](CONFORMANCE.md#reference-bugs); the
|
||||
N-Bit file is counted as ref-bug or our-error depending on the run
|
||||
([known-issues](docs/known-issues.md#conformance-bad_nbit_parms_walkh5-flips-between-ref-bug-and-our-error)). The
|
||||
run is a nightly CI job (`.gitea/workflows/conformance.yml`) that fails on
|
||||
any panic, hang or crash, or on an ok file that stops being ok.
|
||||
|
||||
@@ -114,10 +116,10 @@ Limits and open issues, with dates, are in
|
||||
| **Groups and links** | Symbol-table, compact and dense groups (tested to 100 000 links), creation order, soft and hard links; writing external links | | Following external links (explicit error); user-defined links are skipped |
|
||||
| **Datatypes** | Integers and IEEE floats of every width and byte order (incl. `f16`), enums, compounds (every version, incl. HDF5 2.0's v5), arrays, fixed-length strings, opaque, complex (HDF5 2.0 class 11) | Variable-length strings and sequences, object references | Writing variable-length data; decoding region and attribute references; x87 long double and binary128 |
|
||||
| **Layouts and chunk indexes** | Compact, contiguous and chunked; chunk indexes single chunk, Fixed Array, Extensible Array and v2 B-tree (the writer picks one as libhdf5 does); fill values; resizable datasets; virtual datasets (read limits in known-issues) | Chunk indexes v1 B-tree and implicit (the editor also changes them) | External raw data files (explicit error) |
|
||||
| **Filters** | deflate (pure-Rust zlib-rs), shuffle, Fletcher-32, LZ4, Zstd (C, opt-in); plugins LZF, bitshuffle, bzip2, Blosc 1 | N-Bit, scale-offset, SZIP (C, opt-in); plugins Blosc2 and ZFP | Other filter IDs, unless you register a codec (`filter_registry::register_filter`) |
|
||||
| **Filters** | deflate (pure-Rust zlib-rs), shuffle, Fletcher-32, LZ4 (opt-in), Zstd (C, opt-in); plugins LZF, bitshuffle, bzip2, Blosc 1 | N-Bit, scale-offset, SZIP (C, opt-in); plugins Blosc2 and ZFP | Other filter IDs, unless you register a codec (`filter_registry::register_filter`) |
|
||||
| **Editing in place** | `FileEditor`: overwrite values, grow and shrink chunked datasets (every index), set attributes (compact and dense), in files from h5py or clawhdf5 | | Creating or deleting objects in an existing file; deleting attributes; new chunks in implicit indexes; VL data; filters this build cannot encode (refused before any write) |
|
||||
| **Access** | Local files (mmap or buffered), bytes in memory, any `Storage` backend, HTTP(S) and S3/GCS/Azure via `clawhdf5-remote`, SWMR reading (`File::open_swmr`, `Dataset::refresh`) | Remote files and the browser are read-only | SWMR writing; remote SWMR; MPI collective I/O (`clawhdf5-io`'s `mpi-io` reads on one rank and broadcasts) |
|
||||
| **Bindings** | Python (read, `'w'` for numeric arrays, `'r+'` editing, URLs), NetCDF-4 (CF scale/offset/fill) | WebAssembly (`open(bytes)`, `openUrl`); no Zstd/SZIP/pcodec, no compound or VL-sequence datasets | Node.js (the package does not work; see known-issues) |
|
||||
| **Bindings** | Python (read, `'w'` for numeric arrays, `'r+'` editing, URLs), NetCDF-4 (CF scale/offset/fill) | WebAssembly (`open(bytes)`, `openUrl`); no Zstd/SZIP/pcodec, no compound, reference, opaque, bitfield, time or VL-sequence datasets | Node.js (the package does not work; see known-issues) |
|
||||
|
||||
Plugin filters other than LZF are cargo features (`bitshuffle`, `bzip2`,
|
||||
`blosc`, `blosc2`, `zfp`, or `plugin-filters` for all of them), all pure
|
||||
@@ -127,13 +129,15 @@ that only clawhdf5 reads.
|
||||
|
||||
**C dependencies, precisely.** The core crates build no C by default: no
|
||||
libhdf5, and deflate is [zlib-rs](https://github.com/trifectatechfoundation/zlib-rs),
|
||||
which produces output byte-identical to zlib-ng and matches its HDF5 read and
|
||||
write speed within 6%
|
||||
([BENCHMARKS.md](BENCHMARKS.md#deflate-backend-zlib-rs-vs-zlib-ng)). CI
|
||||
whose output was byte-identical to zlib-ng's at levels 1, 6 and 9 on the
|
||||
benchmark inputs and which matched its HDF5 read and write speed within 6%
|
||||
(tank, 2026-09-23, [BENCHMARKS.md](BENCHMARKS.md#deflate-backend-zlib-rs-vs-zlib-ng)). CI
|
||||
fails if a C-building crate enters their default dependency tree. C comes in
|
||||
only when you ask: `fast-deflate` (zlib-ng, needs cmake), `zstd`, `szip`,
|
||||
`https` and the cloud stores (ring / aws-lc-rs), the BLAS backends,
|
||||
`clawhdf5-migrate` (bundled SQLite) and the Node.js bindings.
|
||||
`clawhdf5-migrate` (bundled SQLite) and the Node.js bindings. One exception
|
||||
links rather than builds C: on macOS the default `system-zlib-decompress`
|
||||
feature inflates with the system libz first.
|
||||
|
||||
## Install
|
||||
|
||||
@@ -334,10 +338,12 @@ a knowledge graph — in one HDF5 file (readable by h5py), with:
|
||||
MiniLM embeddings) turn-level Hit@5 is **81.4%** — retrieval recall, not
|
||||
the official QA-accuracy metric (tank, re-run 2026-09-27,
|
||||
[BENCHMARKS.md](BENCHMARKS.md#longmemeval-results)).
|
||||
- **Compact by default**: float16 embeddings on disk (48% smaller at 100K)
|
||||
and an int8 index copy with exact re-scoring — 1.74x the raw vectors in
|
||||
memory at 100K instead of 2.72x, and 1.63x the QPS at equal recall on
|
||||
AVX2 (paired runs; see BENCHMARKS.md for which rows were re-run).
|
||||
- **Compact by default**: float16 embeddings on disk (48% smaller at 100K;
|
||||
tank, 2026-09-23) and an int8 index copy with exact re-scoring — 1.74x
|
||||
the raw vectors in memory at 100K instead of 2.72x, and 1.63x the QPS at
|
||||
equal recall on AVX2 (int8 side measured 2026-09-19/20, machine not
|
||||
recorded, not re-run since; see
|
||||
[BENCHMARKS.md](BENCHMARKS.md#quantising-the-index-copy-quantized_index)).
|
||||
- **Durability**: a write-ahead log with a chained CRC per entry,
|
||||
crash-safe checkpoints, a single-writer lock and a read-only open. WAL
|
||||
appends are not fsynced: saves since the last checkpoint can be lost on
|
||||
|
||||
Reference in New Issue
Block a user