Two jobs, which runners serve them, and the two constraints that kept
the x86 job failing on every push until today: no JavaScript actions
(`rust:latest` has no `node`, and GitHub is not reachable from every
runner) and `cmake` for libz-ng-sys. Also notes that the Docker Hub
`latest` tag for the runner is frozen at 0.6.1, so it is not a way to
stay current.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`MemoryConfig::quantized_index` now defaults to `true`. It holds a
quarter of the index memory and, with the exact re-score, is faster at
equal recall on every configuration measured: 1.63x the queries per
second on x86-64 (AVX2) and 1.18x on a Raspberry Pi 5 (NEON SDOT), with
builds 1.8x and 2.3x faster. The one argument for keeping it off — that
int8 search was slower on ARM — did not survive being measured.
Existing stores do not change. A store written with v2.6.0 or later
keeps its persisted setting. One written before the setting existed has
no stored value, and it loads as `false` rather than as the new default,
so reopening it never changes how its index is held. That case is
guarded by a real store written with the v2.5.0 CLI, committed as
`tests/fixtures/store_v2_5_0.h5` (6.8 KB): the test asserts it reopens
with an f32 index and still searches, and it fails if the load default
is changed to `true`.
The CLI needed more than a new default. `create --quantized-index`
assigned its value straight into the config, so under the new default
every CLI-created store would have been forced back to f32 unless the
caller knew to ask for int8. It is replaced by `--f32-index`, which only
ever switches the default off; `--quantized-index` is still accepted,
hidden, as a no-op, and the two conflict.
The whole agent suite passes under the new default, including the
brute-force recall oracle, now running on int8 plus re-score without
being asked to.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
On a Raspberry Pi 5 at N = 100 000 and equal recall (0.9940 vs 0.9945),
medians of three runs:
f32 33 413 ms build 6 164 QPS
int8 scalar 18 950 ms ~6 190 QPS (what v2.7.0 shipped)
int8 NEON ~17 000 ms 6 640 QPS
int8 SDOT 14 464 ms 7 267 QPS 1.18x f32, 2.3x build
The docs said quantised search stayed off by default because aarch64
"falls back to the scalar loop, where the original trade still
applies" — that it was ~13% slower than f32 there, as on x86. That was
extrapolated rather than measured, and it was wrong: x86's portable
baseline is SSE2 against hand-written AVX2 f32 kernels, but on aarch64
NEON is the baseline and the scalar loop vectorises well, so it already
matched f32. Corrected in BENCHMARKS.md, README.md and CLAUDE.md; the
released v2.7.0 changelog entry is left as it was and the correction is
recorded in a new one.
Labelled as Pi 5 figures throughout — a Pi's memory bandwidth and cache
are far below an M-series or flagship phone, so the ratios will move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Measured with `clawhdf5_accel::dot_i8` in place, medians of three
alternating runs at N = 100 000 x 384, same binary:
build f32 3197 ms int8 1778 ms int8+re-score 1826 ms
ef = 64 f32 13 399 QPS @ 0.9945 int8+re-score 21 848 QPS @ 0.9940
So at equal recall the quantised index is 1.63x the queries per second
and 1.8x the build speed, holding a quarter of the vectors. The earlier
"~13% of QPS" figure compared a scalar int8 loop against hand-written
AVX2 f32 kernels and was measuring the missing kernel; it is kept in
BENCHMARKS.md with that explanation rather than quietly replaced.
Still off by default, now for portability rather than performance: the
kernel is AVX2-only and aarch64 falls back to scalar, where the original
trade applies. A NEON kernel would settle it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`MemoryConfig::quantized_index` stores the HNSW index's own copy of the
embeddings as i8 rather than f32. At 100k x 384 that takes the index from
266 to 123 MiB and the whole reopened store from 399 to 256 MiB — 2.72x
to 1.74x the raw vectors, the largest remaining item in the footprint.
Quantised distances are approximate and `ef` cannot compensate, because
the loss is in the distances rather than in the graph: recall@10 tops out
at 0.967 against f32's 0.9995 and does not move between ef=128 and
ef=256. The store already holds the exact embeddings, though, so when the
index is quantised the query path re-scores the candidate pool against
them before fusion. That restores recall (0.9940 vs 0.9945 at ef=64) and
costs about 13% of QPS.
Off by default: it trades query speed for memory and which side is worth
more depends on the deployment. The flag is persisted in `/meta`, so a
reopened store does not silently revert to four times the index memory,
and the sidecar graph is rehydrated into the configured storage.
Also on the CLI as `create --quantized-index`.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`parallel` joins the agent's default features, so the HNSW bulk build uses the
thread pool: cold index build at 10K records 1152 -> ~380 ms in a same-moment
A/B (the graph is identical either way). Nothing else on the measured paths
changes — ingest, checkpoint, open and steady-state query times are the same
with the feature on or off. Adds rayon to the default dependency set; opt out
with `--no-default-features --features float16,hnsw`.
Harness: `--e2e-only` runs the end-to-end section without the index
benchmarks. Note for anyone comparing numbers: this machine's absolute timings
drifted ~1.5x over a long session, so only same-moment A/B runs are
comparable.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Two related gaps in the WAL format, both closed:
1. Each entry's CRC32 covered only its own bytes, with no sequence number
or chaining — entries could be reordered, duplicated, or spliced (e.g.
a Tombstone moved before/after its target Save) while every individual
entry still passed its own CRC check, silently changing replayed cache
state. Bump to WAL_VERSION 3: each entry's CRC32 trailer is now computed
over its own bytes chained with the previous entry's stored CRC
(crc32(entry_bytes ++ prev_crc)), seeded at 0 after a truncation. Moving,
duplicating, or reordering an entry breaks the chain at that point, and
replay stops there — same handling as a bit-flip or truncation. The
previous per-entry-CRC-only format becomes WAL_VERSION_CRC_UNCHAINED (2)
and remains fully readable (not restricted, since it still verifies each
entry); WalFile::open migrates it to v3 by recreating the file fresh,
same as the existing v1 migration.
WalFile::open() on an existing v3 file scans it once to resume the CRC
chain correctly for further appends — required because a process
restart without an intervening flush reopens the same (non-truncated)
WAL and keeps appending to it, so new entries must chain against the
real last entry already on disk, not restart from 0.
2. WAL_VERSION_LEGACY_NO_CRC (v1, no integrity verification at all) was
reachable through the public WalFile::read_entries — a version byte
flipped from 2/3 down to 1 silently downgraded every entry to the
fully-unverified pre-hardening parser for any caller, not just the
one-time migration path. Split into WalFile::read_entries (rejects v1
with a typed error; still reads v2/v3) and the pub(crate)
read_entries_for_migration (accepts v1 too), used exclusively by
HDF5Memory::open's migration flow.
INT-09
verify_dataset existed and was tested, but was only ever called from
clawhdf5-format's own test files — no reader path in clawhdf5-io or the
clawhdf5 facade called it, so a corrupted dataset was silently readable
even though the write-side SHA-256 hash machinery (gated on the
provenance feature) had already written what it needed to detect that.
Add Dataset::verify_provenance() to the clawhdf5 facade, gated behind a
new `provenance` feature (on by default, forwarding to
clawhdf5-format/provenance which is already default-on). It surfaces a
typed VerifyResult (Ok/Mismatch/NoHash) via the existing Error type
rather than panicking. Deliberately NOT called automatically on
open()/dataset() — it decodes and hashes the entire dataset, which would
regress every read path (including the zero-copy/mmap ones) if run
unconditionally; callers opt in per dataset where the cost is
acceptable (e.g. a periodic integrity sweep).
Also re-export clawhdf5_format::provenance from the facade crate so
VerifyResult is reachable without depending on clawhdf5-format directly.
INT-08
ProvenanceStore, WriteAnomalyDetector, and their check_*/verify_integrity
methods had zero callers outside their own module/tests — lib.rs only
declared the modules. The 15 injection-pattern checks, rate limiting, and
content-hash integrity verification described as shipped in ROADMAP.md
Track 5 never executed during normal library usage.
HDF5Memory::save/save_batch/save_or_update now record a MemoryProvenance
entry (content hash, inferred MemorySource, session) for every write, run
check_rate_anomaly/check_pattern_anomaly/check_source_anomaly against it,
and queue any triggered AnomalyAlert for the caller to drain via the new
take_anomaly_alerts(). save_or_update's update path additionally verifies
the existing record's content against its last recorded hash before
overwriting, catching accidental in-session corruption.
Scope notes, stated plainly rather than overclaimed:
- There is no on-disk provenance ledger (see the CLAUDE.md note added
here) — this is session-scoped bookkeeping, not a disk-integrity
control. open() starts the store empty; there's no historical hash to
verify loaded records against, so "verify on load" is implemented as
"populate the store so subsequent updates in this session are
checkable" rather than a check against nothing.
- MemorySource is inferred from source_channel via a plain string match
(infer_memory_source) — a heuristic for bookkeeping, not the gated
trust-boundary construction INT-05 asks for. That remains open.
- Alerts never block a save; this only makes detection real instead of
dead code. Whether writes should ever be blocked is a policy decision
left to the caller/a follow-up item.
INT-04
README.md:
- Fix badly stale LongMemEval numbers (badge said Hit@5 46%, table showed
fabricated ~46%/~0.34/~72% figures that never matched BENCHMARKS.md's
actual results of Hit@5 100% session / 84.4% turn-level, MRR 1.0/0.6597)
- Remove clawhdf5-types from the Crate Map — that crate was removed in an
earlier cleanup pass but the README diagram was never updated; fix the
crate count (16, not 17) and stale line-of-code figures (72,087/84K -> ~92K)
- Fix a dead #benchmarks badge anchor (no such heading exists) -> #performance
- Document the new clawhdf5-ann `parallel` feature (had no Feature Flags entry)
- Note WAL's CRC32 per-entry check, link the new tank LongMemEval/SIMD/
vector-search reproduction section, update stale test-count comment
(417+ -> 1,650+) and Phase 2 roadmap blurb (LongMemEval is now done)
ROADMAP.md:
- Check off "Academic benchmark cross-validation" (done via the tank
LongMemEval re-run) and add a new "Recently closed out" section
summarizing the Tier 3-4 hardening pass (Android JNI validation, pyo3
bump, WAL CRC32, bounds-check audit + fuzz harness that found 3 real
bugs, HNSW optional parallel feature, workspace.dependencies)
- Update stale test count (1,546 -> 1,650+) and last-updated date
CLAUDE.md: mention WAL's per-entry CRC32 check
CHANGELOG.md: add Security/Performance/Architecture/Documentation entries
under Unreleased summarizing all of Tiers 1-4 (this had not been touched
since 2026-06-04, predating the entire hardening pass)
- Fix version skew: clawhdf5-py (pyproject.toml 1.93.0 -> 2.1.0) and
packages/clawhdf5-node (package.json 2.0.0 -> 2.1.0) were both behind
the actual crate version.
- Correct stale ROADMAP.md claims: the TypeScript bridge already has a
complete napi-rs package (not "no package.json"); CI/CD is now wired
up via .gitea/workflows/ci.yml.
- Fix CLAUDE.md: clawhdf5-gpu uses wgpu with hand-written WGSL compute
shaders, not CubeCL.
- chunked_read.rs: drop 12 unnecessary chunk_dimensions[..rank].to_vec()
allocations — all three callees already accept &[u32].
- btree_v1.rs: add an overflow-safe ensure_len(data, offset, needed)
helper (checked_add) and use it at the two plain-arithmetic bounds
guards, closing a usize-overflow edge case reachable from a crafted
near-usize::MAX B-tree offset. Add a regression test.
- Clarify that the integrity hashes in clawhdf5-agent/provenance.rs
(FNV-1a) and clawhdf5-format/provenance.rs (SHA-256) are unkeyed and
only detect accidental corruption, not tampering — doc-only change.
- README.md: document that the mpi-io feature's read/write paths are
root-read+broadcast / gather-to-rank-0, not true collective I/O.
- Remove clawhdf5-types (empty 1-line stub crate; type defs already live in
clawhdf5-format). Update workspace Cargo.toml and CLAUDE.md accordingly.
- Implement HDF5 superblock v4 (page-buffer mode) read and write support in
clawhdf5-format: Superblock::parse_v4, page_size field, v4 serialize
branch, and FileWriter::with_page_size. This was the one task left
unimplemented from docs/superpowers/plans/2026-06-29-format-write-extensions.md.
- Reconcile the three docs/superpowers/plans/*.md docs (filter codecs,
format write extensions, MPI-IO VOL) against actual shipped code: they
were pre-work plans for d6c4d4f (2026-06-30) committed to git late on
2026-08-03 with all checkboxes still unchecked. Mark completed tasks done
and add a status note so they read as historical records, not open work.
- Refresh ROADMAP.md's "What's Next" section against current repo state.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Sweep of the docs after the v2.0.0 rename and recent changes:
- Per-crate READMEs (13 files): rename leftover rustyhdf5-*/edgehdf5-*
package names and badges to clawhdf5-*, bump usage versions to 2.1.0.
- README: update stale test badge (417 -> 1500+), workspace stats
(15 crates/72K -> 17 crates/84K), agent crate stats (20.7K/32 modules),
and add the missing clawhdf5-napi and clawhdf5-bench crates to the tree.
- CLAUDE.md: correct the CLI subcommand list (inspect/dump/index/search ->
the actual create/save/search/recall/stats/flush-wal/agents-md/export/snapshot).
No code changes. Verified there are zero todo!()/unimplemented!() macros and
no TODO/FIXME comments in the tree.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
