docs: fact-check the refreshed documentation against its sources
Numbers, API names, feature defaults and PR references checked against CONFORMANCE.md, BENCHMARKS.md, CHANGELOG.md, the code and git history. - int8 index figures (1.74x memory, 1.63x QPS) carry the dates git gives them (2026-09-19/20, machine not recorded, not re-run) instead of none; the Pi 5 1.18x carries 2026-09-21. - BENCHMARKS headline: the libhdf5 chunked-write figure is the newest measurement (35x, 2026-09-23), not 45.3x (2026-08-03). - Conformance counts follow the 2026-09-28 run (1 our-error, 2 ref-bug) in conformance/README.md, ROADMAP.md and CLAUDE.md, with a pointer to the bad_nbit_parms_walk.h5 flip. - README: LZ4 is opt-in; the browser refuses reference/opaque/bitfield/ time datasets too; zlib-rs byte-identity scoped to what was measured; macOS default links the system libz for inflate. - Crate READMEs: system-zlib-decompress does something (macOS), SweepDetector lives in prefetch, checkpoint after more than 500 WAL entries, NetCDF-4 unlimited-dimension size warning. - agent-memory.md: string-dataset compression threshold, agents-md prints Markdown, float16 file sizes linked to their study. - known-issues.md: contiguous selection reads, 1.21x vs h5py threads. - docs/README.md, USE_CASES.md, ROADMAP.md, CLAUDE.md: range-read milestones M0-M5 and PRs #17-#19, missing README rows, CLI keygen/verify, dated figures, fast-math is not BLAS. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
This commit is contained in:
+21
-14
@@ -278,11 +278,14 @@ N = 100K, M = 16, ef_construction = 64, ef = 64, recall against an exact scan
|
||||
| `f32` | 0.9945 | 13 399 | 3.2 s |
|
||||
| `i8` + exact re-score (**default for new stores**) | 0.9940 | **21 848** | **1.8 s** |
|
||||
|
||||
A paired comparison (medians of alternating runs, same binary), not re-run
|
||||
on 2026-09-24: a single `f32` run that day measured recall 0.9945, 19 001
|
||||
QPS and a 2.7 s build, so the 1.63x ratio has not been re-checked. On a
|
||||
Raspberry Pi 5 (NEON `SDOT`) the int8 index is 1.18x the `f32` QPS at equal
|
||||
recall. Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was
|
||||
A paired comparison (medians of alternating runs, same binary: the int8
|
||||
index answers 1.63x the queries per second at equal recall), recorded
|
||||
2026-09-20 with the machine not recorded, and not re-run since: a single
|
||||
`f32` run on 2026-09-24 (tank) measured recall 0.9945, 19 001 QPS and a
|
||||
2.7 s build, so the 1.63x ratio has not been re-checked. On a Raspberry
|
||||
Pi 5 (NEON `SDOT`) the int8 index is 1.18x the `f32` QPS at equal recall
|
||||
(2026-09-21; [§ On ARM](../BENCHMARKS.md#on-arm-raspberry-pi-5-cortex-a76)).
|
||||
Before the v2.4.0 neighbour-selection fix, recall@10 at 100K was
|
||||
0.31.
|
||||
|
||||
**Operations:**
|
||||
@@ -344,9 +347,11 @@ The benchmark's vector stage needs `clawhdf5-bench`'s `embeddings` feature.
|
||||
text, `footprint_bench`: 810.4 KB at 1K records, 7.8 MB at 10K, 76.7 MB at
|
||||
100K (803–829 bytes per record). The synthetic text is far more repetitive
|
||||
than real text (40 distinct strings, deflated), so real records will be
|
||||
larger; the embeddings alone are 768 B per record. On the same data, 100K ×
|
||||
384 takes 80.8 MiB as `float16` and 154.0 MiB as `f32`
|
||||
([§ Memory Footprint](../BENCHMARKS.md#memory-footprint-1)).
|
||||
larger; the embeddings alone are 768 B per record
|
||||
([§ Memory Footprint](../BENCHMARKS.md#memory-footprint-1), 2026-09-24). In the
|
||||
float16 study (clustered data, 2026-09-23), 100K × 384 takes 80.8 MiB as
|
||||
`float16` and 154.0 MiB as `f32`
|
||||
([§ float16 embedding storage](../BENCHMARKS.md#float16-embedding-storage-memoryconfigfloat16)).
|
||||
|
||||
**In memory** — a store reopened from disk, counting allocator
|
||||
([§ Memory footprint](../BENCHMARKS.md#memory-footprint)):
|
||||
@@ -357,7 +362,9 @@ larger; the embeddings alone are 768 B per record. On the same data, 100K ×
|
||||
| 10K | 15 MiB | 44 MiB (3.03x) | 27 MiB (1.81x) |
|
||||
| 100K | 146 MiB | 399 MiB (2.72x) | 256 MiB (1.74x) |
|
||||
|
||||
The `f32` column was re-measured on 2026-09-24; the `i8` column was not.
|
||||
The `f32` column was re-measured on 2026-09-24 (tank); the `i8` column was
|
||||
first measured 2026-09-19 (commit c0a9206, machine not recorded) and not
|
||||
re-run ([§ Quantising the index copy](../BENCHMARKS.md#quantising-the-index-copy-quantized_index)).
|
||||
|
||||
## Feature flags and settings
|
||||
|
||||
@@ -385,8 +392,8 @@ Settings stored in the file (`MemoryConfig`):
|
||||
above. Opt out with `quantized_index = false` or `create --f32-index`.
|
||||
- `hnsw_m`, `hnsw_ef_construction`, `hnsw_ef_search`: 16 / 64 / scaled with
|
||||
`k` by default.
|
||||
- `compression` (off): deflate (or Zstd) for embeddings; text of 4 KiB or
|
||||
more is always deflated.
|
||||
- `compression` (off): deflate (or Zstd) for embeddings; string datasets
|
||||
(text, channels, tags, ...) of 4 KiB or more are always deflated.
|
||||
- `wal_enabled` (on), `wal_max_entries`, `hebbian_boost`, `decay_factor`.
|
||||
|
||||
## File schema
|
||||
@@ -442,9 +449,9 @@ clawhdf5 --path agent.h5 snapshot backup.h5
|
||||
clawhdf5 keygen --out signing.key # then --signing-key signing.key; verify --public-key <hex>
|
||||
```
|
||||
|
||||
Output is JSON. The CLI's `search` defaults to weights 0.7 / 0.3, not the
|
||||
library's 0.4 / 0.6, so pass them. `recall`, `stats`, `agents-md` and
|
||||
`export` open the store read-only.
|
||||
Output is JSON (Markdown for `agents-md`). The CLI's `search` defaults to
|
||||
weights 0.7 / 0.3, not the library's 0.4 / 0.6, so pass them. `recall`,
|
||||
`stats`, `agents-md` and `export` open the store read-only.
|
||||
|
||||
## Migrating from SQLite
|
||||
|
||||
|
||||
Reference in New Issue
Block a user