docs: fact-check the refreshed documentation against its sources

Numbers, API names, feature defaults and PR references checked against
CONFORMANCE.md, BENCHMARKS.md, CHANGELOG.md, the code and git history.

- int8 index figures (1.74x memory, 1.63x QPS) carry the dates git gives
  them (2026-09-19/20, machine not recorded, not re-run) instead of none;
  the Pi 5 1.18x carries 2026-09-21.
- BENCHMARKS headline: the libhdf5 chunked-write figure is the newest
  measurement (35x, 2026-09-23), not 45.3x (2026-08-03).
- Conformance counts follow the 2026-09-28 run (1 our-error, 2 ref-bug)
  in conformance/README.md, ROADMAP.md and CLAUDE.md, with a pointer to
  the bad_nbit_parms_walk.h5 flip.
- README: LZ4 is opt-in; the browser refuses reference/opaque/bitfield/
  time datasets too; zlib-rs byte-identity scoped to what was measured;
  macOS default links the system libz for inflate.
- Crate READMEs: system-zlib-decompress does something (macOS), SweepDetector
  lives in prefetch, checkpoint after more than 500 WAL entries, NetCDF-4
  unlimited-dimension size warning.
- agent-memory.md: string-dataset compression threshold, agents-md prints
  Markdown, float16 file sizes linked to their study.
- known-issues.md: contiguous selection reads, 1.21x vs h5py threads.
- docs/README.md, USE_CASES.md, ROADMAP.md, CLAUDE.md: range-read
  milestones M0-M5 and PRs #17-#19, missing README rows, CLI keygen/verify,
  dated figures, fast-math is not BLAS.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
osobh
2026-09-28 11:23:51 -05:00
co-authored by Claude Opus 5.5
parent a48cb9f1a4
commit c27a478e44
14 changed files with 85 additions and 62 deletions
+21 -14
View File
@@ -278,11 +278,14 @@ N = 100K, M = 16, ef_construction = 64, ef = 64, recall against an exact scan
| `f32` | 0.9945 | 13 399 | 3.2 s |
| `i8` + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
A paired comparison (medians of alternating runs, same binary), not re-run
on 2026-09-24: a single `f32` run that day measured recall 0.9945, 19 001
QPS and a 2.7 s build, so the 1.63x ratio has not been re-checked. On a
Raspberry Pi 5 (NEON `SDOT`) the int8 index is 1.18x the `f32` QPS at equal
recall. Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was
A paired comparison (medians of alternating runs, same binary: the int8
index answers 1.63x the queries per second at equal recall), recorded
2026-09-20 with the machine not recorded, and not re-run since: a single
`f32` run on 2026-09-24 (tank) measured recall 0.9945, 19 001 QPS and a
2.7 s build, so the 1.63x ratio has not been re-checked. On a Raspberry
Pi 5 (NEON `SDOT`) the int8 index is 1.18x the `f32` QPS at equal recall
(2026-09-21; [§ On ARM](../BENCHMARKS.md#on-arm-raspberry-pi-5-cortex-a76)).
Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was
0.31.
**Operations:**
@@ -344,9 +347,11 @@ The benchmark's vector stage needs `clawhdf5-bench`'s `embeddings` feature.
text, `footprint_bench`: 810.4 KB at 1K records, 7.8 MB at 10K, 76.7 MB at
100K (803–829 bytes per record). The synthetic text is far more repetitive
than real text (40 distinct strings, deflated), so real records will be
larger; the embeddings alone are 768 B per record. On the same data, 100K ×
384 takes 80.8 MiB as `float16` and 154.0 MiB as `f32`
([§ Memory Footprint](../BENCHMARKS.md#memory-footprint-1)).
larger; the embeddings alone are 768 B per record
([§ Memory Footprint](../BENCHMARKS.md#memory-footprint-1), 2026-09-24). In the
float16 study (clustered data, 2026-09-23), 100K × 384 takes 80.8 MiB as
`float16` and 154.0 MiB as `f32`
([§ float16 embedding storage](../BENCHMARKS.md#float16-embedding-storage-memoryconfigfloat16)).
**In memory** — a store reopened from disk, counting allocator
([§ Memory footprint](../BENCHMARKS.md#memory-footprint)):
@@ -357,7 +362,9 @@ larger; the embeddings alone are 768 B per record. On the same data, 100K ×
| 10K | 15 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
| 100K | 146 MiB | 399 MiB (2.72x) | 256 MiB (1.74x) |
The `f32` column was re-measured on 2026-09-24; the `i8` column was not.
The `f32` column was re-measured on 2026-09-24 (tank); the `i8` column was
first measured 2026-09-19 (commit c0a9206, machine not recorded) and not
re-run ([§ Quantising the index copy](../BENCHMARKS.md#quantising-the-index-copy-quantized_index)).
## Feature flags and settings
@@ -385,8 +392,8 @@ Settings stored in the file (`MemoryConfig`):
above. Opt out with `quantized_index = false` or `create --f32-index`.
- `hnsw_m`, `hnsw_ef_construction`, `hnsw_ef_search`: 16 / 64 / scaled with
`k` by default.
- `compression` (off): deflate (or Zstd) for embeddings; text of 4 KiB or
more is always deflated.
- `compression` (off): deflate (or Zstd) for embeddings; string datasets
(text, channels, tags, ...) of 4 KiB or more are always deflated.
- `wal_enabled` (on), `wal_max_entries`, `hebbian_boost`, `decay_factor`.
## File schema
@@ -442,9 +449,9 @@ clawhdf5 --path agent.h5 snapshot backup.h5
clawhdf5 keygen --out signing.key # then --signing-key signing.key; verify --public-key <hex>
```
Output is JSON. The CLI's `search` defaults to weights 0.7 / 0.3, not the
library's 0.4 / 0.6, so pass them. `recall`, `stats`, `agents-md` and
`export` open the store read-only.
Output is JSON (Markdown for `agents-md`). The CLI's `search` defaults to
weights 0.7 / 0.3, not the library's 0.4 / 0.6, so pass them. `recall`,
`stats`, `agents-md` and `export` open the store read-only.
## Migrating from SQLite